```
%load_ext autoreload
```

```
autoreload 2
```

```
%matplotlib inline
```

```
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym

import inputs
import models
import solvers
```

# Example:

## Worker skill and firm productivity are $\sim U[a, b]$...

```
# define some workers skill
x, a, b = sym.var('x, a, b')
skill_cdf = (x - a) / (b - a)
skill_params = {'a': 1.0, 'b': 2.0}
skill_bounds = [skill_params['a'], skill_params['b']]

workers = inputs.Input(var=x,
                       cdf=skill_cdf,
                       params=skill_params,
                       bounds=skill_bounds,
                       )

# define some firms
y = sym.var('y')
productivity_cdf = (y - a) / (b - a)
productivity_params = skill_params
productivity_bounds = skill_bounds

firms = inputs.Input(var=y,
                     cdf=productivity_cdf,
                     params=productivity_params,
                     bounds=productivity_bounds,
                     )
```

## ...and production function is multiplicatively separable.
A particularly attractive case arises under multiplicative separability of the form

$$F(x, y, l, r) = A(x, y)B(l, r)$$

In this case the condition for positive assortative matching can be written as

$$\frac{AA_{xy}}{A_xA_y}\frac{BB_{lr}}{B_lB_r} \ge 1$$

If $B$ has constant elasticity of substitution form, then we obtain an even simpler condition

$$\frac{AA_{xy}}{A_xA_y} \frac{1}{\sigma_{lr}}\ge 1$$

where $\sigma_{lr}$ is the elasticity of substitution between $l$ and $r$.

Finally, if $A$ also has constant elasticity of substitution form, then we obtain an even simpler condition

$$\frac{1}{\sigma_{xy}\sigma_{lr}}\ge 1 \iff \sigma_{xy}\sigma_{lr} \le 1$$

where $\sigma_{xy}$ is the elasticity of substitution between $x$ and $y$.

```
# define symbolic expression for CES between x and y
omega_A, sigma_A = sym.var('omega_A, sigma_A')
A = ((omega_A * x**((sigma_A - 1) / sigma_A) +
     (1 - omega_A) * y**((sigma_A - 1) / sigma_A))**(sigma_A / (sigma_A - 1)))

# define symbolic expression for Cobb-Douglas between l and r
l, r, omega_B, sigma_B = sym.var('l, r, omega_B, sigma_B')
B = l**omega_B * r**(1 - omega_B)

F = A * B
```

Create an instance of the `models.Model` class and an instance of the `solvers.ShootingSolver` class...

### Positive assortative matching

```
F_params = {'omega_A':0.5, 'omega_B':0.45, 'sigma_A':0.95, 'sigma_B':1.0}

model = models.Model(assortativity='positive',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)

solver = solvers.ShootingSolver(model=model)
```

```
solver.solve(1e1, tol=1e-6, number_knots=100, atol=1e-15, rtol=1e-12)
```

    Success! All workers and firms are matched

```
# examine the solution attribute
solver.solution
```

      firm productivity
      firm size
      wage
      profit

      x

      1.000000
       0.999999
       1.089348
       0.429310
       0.571594

      1.010101
       1.009285
       1.086340
       0.434131
       0.576417

      1.020202
       1.018595
       1.083403
       0.438957
       0.581249

      1.030303
       1.027931
       1.080534
       0.443788
       0.586090

      1.040404
       1.037292
       1.077730
       0.448625
       0.590940

      1.050505
       1.046676
       1.074990
       0.453467
       0.595799

      1.060606
       1.056084
       1.072310
       0.458313
       0.600666

      1.070707
       1.065516
       1.069689
       0.463165
       0.605541

      1.080808
       1.074970
       1.067126
       0.468022
       0.610425

      1.090909
       1.084447
       1.064617
       0.472884
       0.615316

      1.101010
       1.093946
       1.062161
       0.477751
       0.620214

      1.111111
       1.103466
       1.059756
       0.482622
       0.625120

      1.121212
       1.113009
       1.057401
       0.487499
       0.630033

      1.131313
       1.122572
       1.055094
       0.492380
       0.634954

      1.141414
       1.132156
       1.052834
       0.497266
       0.639881

      1.151515
       1.141760
       1.050619
       0.502157
       0.644815

      1.161616
       1.151384
       1.048447
       0.507053
       0.649755

      1.171717
       1.161028
       1.046318
       0.511953
       0.654702

      1.181818
       1.170692
       1.044229
       0.516858
       0.659656

      1.191919
       1.180374
       1.042180
       0.521767
       0.664615

      1.202020
       1.190076
       1.040170
       0.526681
       0.669580

      1.212121
       1.199796
       1.038197
       0.531600
       0.674551

      1.222222
       1.209535
       1.036260
       0.536523
       0.679528

      1.232323
       1.219291
       1.034359
       0.541451
       0.684511

      1.242424
       1.229066
       1.032491
       0.546383
       0.689499

      1.252525
       1.238857
       1.030657
       0.551320
       0.694493

      1.262626
       1.248667
       1.028854
       0.556260
       0.699491

      1.272727
       1.258493
       1.027083
       0.561206
       0.704495

      1.282828
       1.268336
       1.025343
       0.566155
       0.709504

      1.292929
       1.278195
       1.023632
       0.571109
       0.714518

      ...
      ...
      ...
      ...
      ...

      1.707071
       1.694467
       0.971475
       0.777619
       0.923313

      1.717172
       1.704869
       0.970515
       0.782733
       0.928466

      1.727273
       1.715282
       0.969566
       0.787849
       0.933621

      1.737374
       1.725706
       0.968626
       0.792969
       0.938778

      1.747475
       1.736139
       0.967697
       0.798093
       0.943937

      1.757576
       1.746582
       0.966778
       0.803219
       0.949098

      1.767677
       1.757035
       0.965869
       0.808349
       0.954262

      1.777778
       1.767498
       0.964970
       0.813482
       0.959427

      1.787879
       1.777970
       0.964079
       0.818619
       0.964594

      1.797980
       1.788453
       0.963198
       0.823759
       0.969763

      1.808081
       1.798944
       0.962326
       0.828901
       0.974934

      1.818182
       1.809445
       0.961463
       0.834048
       0.980108

      1.828283
       1.819956
       0.960609
       0.839197
       0.985282

      1.838384
       1.830476
       0.959763
       0.844350
       0.990459

      1.848485
       1.841005
       0.958926
       0.849505
       0.995638

      1.858586
       1.851543
       0.958097
       0.854664
       1.000818

      1.868687
       1.862090
       0.957276
       0.859826
       1.006000

      1.878788
       1.872647
       0.956463
       0.864992
       1.011184

      1.888889
       1.883212
       0.955657
       0.870160
       1.016369

      1.898990
       1.893786
       0.954860
       0.875331
       1.021556

      1.909091
       1.904369
       0.954070
       0.880506
       1.026745

      1.919192
       1.914961
       0.953287
       0.885683
       1.031935

      1.929293
       1.925561
       0.952512
       0.890864
       1.037127

      1.939394
       1.936170
       0.951744
       0.896048
       1.042321

      1.949495
       1.946787
       0.950983
       0.901235
       1.047516

      1.959596
       1.957413
       0.950229
       0.906424
       1.052713

      1.969697
       1.968047
       0.949481
       0.911617
       1.057911

      1.979798
       1.978690
       0.948740
       0.916813
       1.063110

      1.989899
       1.989341
       0.948006
       0.922012
       1.068311

      2.000000
       2.000000
       0.947279
       0.927213
       1.073514

100 rows × 4 columns

```
def plot_residual_mu(ax, x, deg, normed=False):
    residual = solver.evaluate_residual(x, k=deg)
    residual_mu = np.abs(residual[:,0])
    residual_mu_line, = ax.plot(x, residual_mu)
    ax.set_xlim(solver.model.workers.lower, solver.model.workers.upper)
    ax.set_yscale('log')
    ax.grid('on')

    return [residual_mu_line]

def plot_residual_theta(ax, x, deg, normed=False):
    residual = solver.evaluate_residual(x, k=deg)
    residual_theta, = ax.plot(x, np.abs(residual[:,1]))
    ax.set_xlim(solver.model.workers.lower, solver.model.workers.upper)
    ax.set_yscale('log')
    ax.grid('on')

    return [residual_theta]
```

```
fig, ax = plt.subplots(1,1, figsize=(8,6))
plot_residual_mu(ax, np.linspace(1, 2, 1000), deg=5)
plt.show()
```

```
fig, ax = plt.subplots(1,1, figsize=(8,6))
plot_residual_theta(ax, np.linspace(1, 2, 1000), deg=5)
plt.show()
```

```
solver.evaluate_rhs(solver.solution.index.values, solver.solution[['firm productivity', 'firm size']].values.T)
```

    array([ 0.91798057,  0.92052183,  0.92301727,  0.92546817,  0.92787578,
            0.93024131,  0.93256592,  0.93485074,  0.93709686,  0.93930532,
            0.94147715,  0.94361332,  0.94571478,  0.94778246,  0.94981723,
            0.95181997,  0.95379149,  0.9557326 ,  0.95764408,  0.95952669,
            0.96138115,  0.96320816,  0.96500842,  0.96678258,  0.96853128,
            0.97025515,  0.97195478,  0.97363076,  0.97528365,  0.97691401,
            0.97852236,  0.98010922,  0.98167509,  0.98322045,  0.98474577,
            0.98625151,  0.98773812,  0.98920602,  0.99065562,  0.99208735,
            0.99350158,  0.9948987 ,  0.99627908,  0.99764309,  0.99899107,
            1.00032337,  1.00164031,  1.00294222,  1.00422941,  1.00550219,
            1.00676084,  1.00800567,  1.00923695,  1.01045496,  1.01165995,
            1.01285219,  1.01403193,  1.01519941,  1.01635488,  1.01749856,
            1.01863069,  1.01975147,  1.02086114,  1.02195989,  1.02304793,
            1.02412547,  1.02519269,  1.02624978,  1.02729693,  1.02833431,
            1.02936211,  1.0303805 ,  1.03138964,  1.03238969,  1.03338082,
            1.03436317,  1.03533691,  1.03630218,  1.03725912,  1.03820788,
            1.03914859,  1.04008138,  1.0410064 ,  1.04192377,  1.0428336 ,
            1.04373604,  1.04463119,  1.04551918,  1.04640012,  1.04727412,
            1.04814129,  1.04900173,  1.04985557,  1.05070288,  1.05154379,
            1.05237838,  1.05320675,  1.054029  ,  1.05484522,  1.05565549,
           -0.30129466, -0.2942018 , -0.28736374, -0.28076888, -0.27440625,
           -0.26826547, -0.26233671, -0.25661066, -0.2510785 , -0.24573187,
           -0.24056282, -0.23556384, -0.23072776, -0.2260478 , -0.2215175 ,
           -0.21713074, -0.21288167, -0.20876475, -0.20477469, -0.20090646,
           -0.19715528, -0.19351657, -0.18998598, -0.18655937, -0.18323277,
           -0.18000241, -0.17686467, -0.17381612, -0.17085346, -0.16797355,
           -0.16517338, -0.16245008, -0.15980089, -0.15722319, -0.15471446,
           -0.15227228, -0.14989434, -0.14757843, -0.14532243, -0.14312431,
           -0.14098211, -0.13889397, -0.13685808, -0.13487273, -0.13293626,
           -0.13104708, -0.12920367, -0.12740456, -0.12564834, -0.12393365,
           -0.1222592 , -0.12062372, -0.11902602, -0.11746493, -0.11593934,
           -0.11444817, -0.11299038, -0.11156498, -0.110171  , -0.10880752,
           -0.10747364, -0.1061685 , -0.10489128, -0.10364116, -0.10241738,
           -0.10121919, -0.10004587, -0.09889673, -0.09777108, -0.09666829,
           -0.09558772, -0.09452877, -0.09349086, -0.09247341, -0.09147589,
           -0.09049775, -0.08953849, -0.08859762, -0.08767465, -0.08676911,
           -0.08588057, -0.08500858, -0.08415272, -0.08331259, -0.08248779,
           -0.08167794, -0.08088266, -0.0801016 , -0.0793344 , -0.07858073,
           -0.07784026, -0.07711267, -0.07639766, -0.07569492, -0.07500416,
           -0.0743251 , -0.07365747, -0.073001  , -0.07235543, -0.07172051])

```
fig, ax = plt.subplots(1, 1, figsize=(8,6))
solver.plot_equilibrium_firm_size(ax, 'firm_productivity')
plt.show()
```

### Negative assortative matching

```
# negative assortativity requires that sigma_A * sigma_B > 1
F_params = {'omega_A':0.25, 'omega_B':0.5, 'sigma_A':1.5, 'sigma_B':1.0}
```

```
model = models.Model(assortativity='negative',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)

solver = solvers.ShootingSolver(model=model)
```

```
solver.solve(2e0, tol=1e-6, number_knots=100, atol=1e-15, rtol=1e-12)
```

    Exhausted firms: initial guess of 1.0 for firm size is too low.
    Exhausted firms: initial guess of 1.5 for firm size is too low.
    Exhausted firms: initial guess of 1.75 for firm size is too low.
    Exhausted workers: initial guess of 1.875 for firm size is too high!
    Exhausted firms: Initial guess of 1.8125 for firm size was too low!
    Exhausted workers: initial guess of 1.84375 for firm size is too high!
    Exhausted firms: Initial guess of 1.828125 for firm size was too low!
    Exhausted workers: initial guess of 1.8359375 for firm size is too high!
    Exhausted workers: initial guess of 1.83203125 for firm size is too high!
    Exhausted workers: initial guess of 1.830078125 for firm size is too high!
    Exhausted firms: Initial guess of 1.8291015625 for firm size was too low!
    Exhausted firms: Initial guess of 1.82958984375 for firm size was too low!
    Exhausted workers: initial guess of 1.82983398438 for firm size is too high!
    Exhausted workers: initial guess of 1.82971191406 for firm size is too high!
    Exhausted workers: initial guess of 1.82965087891 for firm size is too high!
    Exhausted firms: Initial guess of 1.82962036133 for firm size was too low!
    Exhausted firms: Initial guess of 1.82963562012 for firm size was too low!
    Exhausted workers: initial guess of 1.82964324951 for firm size is too high!
    Success! All workers and firms are matched

```
solver.solution
```

      firm productivity
      firm size
      wage
      profit

      x

      1.000000
       2.000000
       1.829639
       0.630705
       1.153963

      1.010101
       1.994455
       1.813970
       0.633368
       1.148910

      1.020202
       1.988863
       1.798401
       0.636024
       1.143826

      1.030303
       1.983222
       1.782931
       0.638673
       1.138710

      1.040404
       1.977532
       1.767557
       0.641317
       1.133564

      1.050505
       1.971792
       1.752276
       0.643954
       1.128386

      1.060606
       1.966003
       1.737087
       0.646586
       1.123175

      1.070707
       1.960162
       1.721986
       0.649212
       1.117933

      1.080808
       1.954271
       1.706973
       0.651832
       1.112659

      1.090909
       1.948327
       1.692044
       0.654446
       1.107352

      1.101010
       1.942331
       1.677197
       0.657056
       1.102012

      1.111111
       1.936282
       1.662431
       0.659660
       1.096639

      1.121212
       1.930179
       1.647743
       0.662259
       1.091233

      1.131313
       1.924021
       1.633132
       0.664853
       1.085793

      1.141414
       1.917808
       1.618594
       0.667443
       1.080319

      1.151515
       1.911540
       1.604130
       0.670027
       1.074810

      1.161616
       1.905214
       1.589736
       0.672607
       1.069268

      1.171717
       1.898832
       1.575410
       0.675183
       1.063690

      1.181818
       1.892391
       1.561152
       0.677754
       1.058076

      1.191919
       1.885891
       1.546958
       0.680321
       1.052428

      1.202020
       1.879331
       1.532828
       0.682883
       1.046743

      1.212121
       1.872711
       1.518759
       0.685442
       1.041022

      1.222222
       1.866029
       1.504751
       0.687997
       1.035264

      1.232323
       1.859285
       1.490800
       0.690548
       1.029469

      1.242424
       1.852478
       1.476906
       0.693095
       1.023637

      1.252525
       1.845606
       1.463067
       0.695639
       1.017767

      1.262626
       1.838670
       1.449281
       0.698180
       1.011858

      1.272727
       1.831667
       1.435546
       0.700716
       1.005911

      1.282828
       1.824597
       1.421862
       0.703250
       0.999924

      1.292929
       1.817458
       1.408225
       0.705781
       0.993898

      ...
      ...
      ...
      ...
      ...

      1.707071
       1.446606
       0.871852
       0.808071
       0.704518

      1.717172
       1.434933
       0.858825
       0.810563
       0.696132

      1.727273
       1.423081
       0.845771
       0.813056
       0.687660

      1.737374
       1.411045
       0.832690
       0.815552
       0.679102

      1.747475
       1.398818
       0.819576
       0.818049
       0.670454

      1.757576
       1.386393
       0.806429
       0.820548
       0.661714

      1.767677
       1.373764
       0.793244
       0.823050
       0.652879

      1.777778
       1.360923
       0.780018
       0.825554
       0.643947

      1.787879
       1.347862
       0.766748
       0.828061
       0.634914

      1.797980
       1.334573
       0.753431
       0.830571
       0.625778

      1.808081
       1.321046
       0.740063
       0.833083
       0.616534

      1.818182
       1.307271
       0.726639
       0.835599
       0.607179

      1.828283
       1.293240
       0.713157
       0.838119
       0.597710

      1.838384
       1.278940
       0.699610
       0.840642
       0.588122

      1.848485
       1.264360
       0.685996
       0.843170
       0.578411

      1.858586
       1.249486
       0.672308
       0.845701
       0.568572

      1.868687
       1.234306
       0.658542
       0.848237
       0.558600

      1.878788
       1.218804
       0.644692
       0.850778
       0.548489

      1.888889
       1.202965
       0.630751
       0.853324
       0.538235

      1.898990
       1.186770
       0.616714
       0.855875
       0.527830

      1.909091
       1.170201
       0.602573
       0.858432
       0.517268

      1.919192
       1.153236
       0.588320
       0.860995
       0.506541

      1.929293
       1.135854
       0.573948
       0.863565
       0.495641

      1.939394
       1.118029
       0.559447
       0.866141
       0.484560

      1.949495
       1.099734
       0.544807
       0.868725
       0.473288

      1.959596
       1.080937
       0.530019
       0.871317
       0.461814

      1.969697
       1.061606
       0.515068
       0.873917
       0.450127

      1.979798
       1.041702
       0.499944
       0.876526
       0.438214

      1.989899
       1.021183
       0.484630
       0.879145
       0.426060

      2.000000
       1.000000
       0.469110
       0.881774
       0.413649

100 rows × 4 columns

Now we can do all the cool things that one can do with Pandas...

```
solver.solution['firm size'].plot()
```

```

```

# Example:

Multiplicative separability with log-normal worker skill and firm productivity distributions...

```
# define some workers skill
mu1, sigma1 = sym.var('mu1, sigma1')
skill_cdf = 0.5 + 0.5 * sym.erf((sym.log(x) - mu1) / sym.sqrt(2 * sigma1**2))
skill_params = {'mu1': 0.0, 'sigma1': 1.0}
skill_bounds = [1e-3, 5e1]

workers = inputs.Input(var=x,
                       cdf=skill_cdf,
                       params=skill_params,
                       bounds=skill_bounds,
                       )

# define some firms
mu2, sigma2 = sym.var('mu2, sigma2')
productivity_cdf = 0.5 + 0.5 * sym.erf((sym.log(y) - mu2) / sym.sqrt(2 * sigma2**2))
productivity_params = {'mu2': 0.0, 'sigma2': 1.0}
productivity_bounds = [1e-3, 5e1]

firms = inputs.Input(var=y,
                     cdf=productivity_cdf,
                     params=productivity_params,
                     bounds=productivity_bounds,
                     )
```

### Positive assortative matching

```
# define symbolic expression for CES between x and y
omega_A, sigma_A = sym.var('omega_A, sigma_A')
A = ((omega_A * x**((sigma_A - 1) / sigma_A) +
     (1 - omega_A) * y**((sigma_A - 1) / sigma_A))**(sigma_A / (sigma_A - 1)))

# define symbolic expression for Cobb-Douglas between l and r
l, r, omega_B, sigma_B = sym.var('l, r, omega_B, sigma_B')
B = l**omega_B * r**(1 - omega_B)

F = A * B

F_params = {'omega_A':0.5, 'omega_B':0.45, 'sigma_A':0.5, 'sigma_B':1.0}

model = models.Model('positive',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)

solver = solvers.ShootingSolver(model=model)
```

```
solver.solve(5e-1, tol=1e-6, number_knots=5000, integrator='lsoda',
             with_jacobian=True, atol=1e-12, rtol=1e-9)
```

    Exhausted firms: initial guess of 0.25 for firm size is too low.
    Exhausted firms: initial guess of 0.375 for firm size is too low.

    /Users/drpugh/anaconda/lib/python2.7/site-packages/scipy/integrate/_ode.py:1127: UserWarning: lsoda: Excess work done on this call (perhaps wrong Dfun type).
      'Unexpected istate=%s' % istate))

```
solver.solution
```

      firm productivity
      firm size
      wage
      profit

      x

      0.001000
        0.001067
       3.549875
        0.000232
        0.001005

      0.051049
        0.043892
       1.585344
        0.016516
        0.032003

      0.101098
        0.086925
       1.429779
        0.034621
        0.060501

      0.151147
        0.129960
       1.345459
        0.053521
        0.088013

      0.201196
        0.172995
       1.288533
        0.072958
        0.114900

      0.251245
        0.216032
       1.245988
        0.092806
        0.141332

      0.301294
        0.259069
       1.212244
        0.112987
        0.167405

      0.351343
        0.302107
       1.184414
        0.133450
        0.193185

      0.401392
        0.345146
       1.160814
        0.154158
        0.218716

      0.451441
        0.388186
       1.140382
        0.175083
        0.244030

      0.501490
        0.431227
       1.122406
        0.196201
        0.269155

      0.551540
        0.474268
       1.106387
        0.217496
        0.294109

      0.601589
        0.517310
       1.091959
        0.238953
        0.318910

      0.651638
        0.560353
       1.078853
        0.260558
        0.343572

      0.701687
        0.603397
       1.066857
        0.282303
        0.368105

      0.751736
        0.646441
       1.055808
        0.304177
        0.392519

      0.801785
        0.689487
       1.045576
        0.326172
        0.416824

      0.851834
        0.732533
       1.036054
        0.348282
        0.441026

      0.901883
        0.775580
       1.027156
        0.370501
        0.465131

      0.951932
        0.818627
       1.018809
        0.392822
        0.489146

      1.001981
        0.861676
       1.010952
        0.415241
        0.513076

      1.052030
        0.904725
       1.003536
        0.437754
        0.536924

      1.102079
        0.947775
       0.996515
        0.460356
        0.560696

      1.152128
        0.990826
       0.989852
        0.483043
        0.584395

      1.202177
        1.033878
       0.983515
        0.505813
        0.608025

      1.252226
        1.076930
       0.977474
        0.528662
        0.631588

      1.302275
        1.119983
       0.971705
        0.551587
        0.655087

      1.352324
        1.163038
       0.966186
        0.574586
        0.678525

      1.402373
        1.206093
       0.960898
        0.597655
        0.701905

      1.452422
        1.249148
       0.955823
        0.620793
        0.725228

      ...
      ...
      ...
      ...
      ...

      48.548578
       47.782881
       0.626701
       28.025228
       21.466424

      48.598627
       47.856867
       0.626869
       28.057248
       21.496718

      48.648676
       47.931017
       0.627039
       28.089273
       21.527071

      48.698725
       48.005333
       0.627209
       28.121305
       21.557486

      48.748774
       48.079815
       0.627381
       28.153343
       21.587963

      48.798823
       48.154466
       0.627554
       28.185387
       21.618502

      48.848872
       48.229287
       0.627729
       28.217437
       21.649103

      48.898921
       48.304279
       0.627905
       28.249494
       21.679767

      48.948970
       48.379445
       0.628082
       28.281558
       21.710495

      48.999019
       48.454784
       0.628260
       28.313628
       21.741288

      49.049068
       48.530300
       0.628440
       28.345705
       21.772145

      49.099117
       48.605993
       0.628621
       28.377788
       21.803067

      49.149166
       48.681864
       0.628803
       28.409878
       21.834055

      49.199215
       48.757917
       0.628987
       28.441975
       21.865110

      49.249264
       48.834153
       0.629172
       28.474079
       21.896232

      49.299313
       48.910573
       0.629359
       28.506189
       21.927422

      49.349362
       48.987179
       0.629547
       28.538307
       21.958680

      49.399411
       49.063971
       0.629736
       28.570432
       21.990006

      49.449460
       49.140903
       0.629926
       28.602557
       22.021386

      49.499510
       49.218053
       0.630118
       28.634693
       22.052845

      49.549559
       49.295391
       0.630311
       28.666836
       22.084373

      49.599608
       49.372918
       0.630506
       28.698986
       22.115971

      49.649657
       49.450632
       0.630702
       28.731141
       22.147637

      49.699706
       49.528536
       0.630900
       28.763304
       22.179374

      49.749755
       49.606634
       0.631099
       28.795473
       22.211182

      49.799804
       49.684927
       0.631299
       28.827648
       22.243062

      49.849853
       49.763416
       0.631501
       28.859831
       22.275013

      49.899902
       49.842100
       0.631704
       28.892020
       22.307037

      49.949951
       49.920980
       0.631909
       28.924216
       22.339133

      50.000000
       50.000000
       0.632115
       28.956412
       22.371282

1000 rows × 4 columns

```
solver.solution['firm size'].plot()
```

### Negative assortative matching

```
F_params = {'omega_A':0.5, 'omega_B':0.45, 'sigma_A':1.5, 'sigma_B':1.0}

model = models.Model('negative',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)

solver = solvers.ShootingSolver(model=model)
```

```
solver.solve(1e3, tol=1e-6, number_knots=1000, atol=1e-12, rtol=1e-9)
```

    Exhausted firms: initial guess of 500.0 for firm size is too low.
    Exhausted workers: initial guess of 750.0 for firm size is too high!
    Exhausted firms: initial guess of 625.0 for firm size is too low.
    Exhausted firms: initial guess of 687.5 for firm size is too low.
    Exhausted workers: initial guess of 718.75 for firm size is too high!
    Exhausted workers: initial guess of 703.125 for firm size is too high!
    Exhausted firms: initial guess of 695.3125 for firm size is too low.
    Exhausted workers: initial guess of 699.21875 for firm size is too high!
    Exhausted firms: initial guess of 697.265625 for firm size is too low.
    Exhausted workers: initial guess of 698.2421875 for firm size is too high!
    Exhausted workers: initial guess of 697.75390625 for firm size is too high!
    Exhausted firms: initial guess of 697.509765625 for firm size is too low.
    Exhausted workers: initial guess of 697.631835938 for firm size is too high!
    Exhausted workers: initial guess of 697.570800781 for firm size is too high!
    Exhausted workers: initial guess of 697.540283203 for firm size is too high!
    Exhausted workers: initial guess of 697.525024414 for firm size is too high!
    Exhausted firms: initial guess of 697.51739502 for firm size is too low.
    Exhausted workers: initial guess of 697.521209717 for firm size is too high!
    Exhausted firms: initial guess of 697.519302368 for firm size is too low.
    Exhausted firms: initial guess of 697.520256042 for firm size is too low.
    Exhausted firms: initial guess of 697.52073288 for firm size is too low.
    Exhausted workers: initial guess of 697.520971298 for firm size is too high!
    Exhausted firms: initial guess of 697.520852089 for firm size is too low.
    Exhausted workers: initial guess of 697.520911694 for firm size is too high!
    Exhausted workers: initial guess of 697.520881891 for firm size is too high!
    Exhausted firms: initial guess of 697.52086699 for firm size is too low.
    Exhausted workers: initial guess of 697.520874441 for firm size is too high!
    Exhausted firms: initial guess of 697.520870715 for firm size is too low.
    Exhausted firms: initial guess of 697.520872578 for firm size is too low.
    Exhausted workers: initial guess of 697.520873509 for firm size is too high!
    Exhausted firms: initial guess of 697.520873044 for firm size is too low.
    Exhausted workers: initial guess of 697.520873277 for firm size is too high!
    Exhausted firms: initial guess of 697.52087316 for firm size is too low.
    Exhausted firms: initial guess of 697.520873218 for firm size is too low.
    Exhausted firms: initial guess of 697.520873247 for firm size is too low.
    Exhausted firms: Initial guess of 697.520873262 for firm size was too low!
    Exhausted firms: Initial guess of 697.520873269 for firm size was too low!
    Exhausted workers: initial guess of 697.520873273 for firm size is too high!
    Exhausted firms: Initial guess of 697.520873271 for firm size was too low!
    Exhausted firms: Initial guess of 697.520873272 for firm size was too low!
    Exhausted firms: Initial guess of 697.520873272 for firm size was too low!
    Exhausted workers: initial guess of 697.520873273 for firm size is too high!
    Exhausted firms: Initial guess of 697.520873273 for firm size was too low!
    Exhausted firms: Initial guess of 697.520873273 for firm size was too low!
    Success! All workers and firms are matched

```
solver.solution
```

    array([[  1.00000000e-04,   5.00000000e+01,   6.97520873e+02,
              7.96982563e-02,   6.79447968e+01],
           [  1.00200000e-01,   4.38421936e+01,   2.76462641e+02,
              1.62305818e-01,   5.48429383e+01],
           [  2.00300000e-01,   3.01915720e+01,   1.25086665e+02,
              1.99932868e-01,   3.05664771e+01],
           ...,
           [  9.97998000e+01,   2.00562063e-02,   4.40863126e-04,
              4.66712075e+02,   2.51479732e-01],
           [  9.98999000e+01,   1.70657087e-02,   4.33023425e-04,
              4.67696609e+02,   2.47528829e-01],
           [  1.00000000e+02,   9.99753364e-04,   3.61574332e-04,
              4.68687387e+02,   2.07124291e-01]])

```
plt.plot(solver.solution[:, 0], solver.solution[:, 2])
plt.xscale('log')
plt.yscale('log')
plt.show()
```

# Example:

More sophisticated production function that is not multiplicatively separable...

```
# define some workers skill
x, mu1, sigma1 = sym.var('x, mu1, sigma1')
skill_cdf = 0.5 + 0.5 * sym.erf((sym.log(x) - mu1) / sym.sqrt(2 * sigma1**2))
skill_params = {'mu1': 0.0, 'sigma1': 1e0}
skill_bounds = [1e-3, 50.0]

workers = inputs.Input(var=x,
                       cdf=skill_cdf,
                       params=skill_params,
                       bounds=skill_bounds,
                       )

# define some firms
y, mu2, sigma2 = sym.var('y, mu2, sigma2')
productivity_cdf = 0.5 + 0.5 * sym.erf((sym.log(y) - mu2) / sym.sqrt(2 * sigma2**2))
productivity_params = {'mu2': 0.0, 'sigma2': 1e0}
productivity_bounds = [1e-3, 50.0]

firms = inputs.Input(var=y,
                     cdf=productivity_cdf,
                     params=productivity_params,
                     bounds=productivity_bounds,
                     )

# define some valid model params
F_params = {'eta': 0.89, 'kappa': 1.0, 'gamma': 0.54, 'rho': 0.24, 'A': 1.0, 'k': 1.0}

# define a valid production function
A, k, kappa, eta, rho, l, gamma, r = sym.var('A, k, kappa, eta, rho, l, gamma, r')
F = r * A * kappa * eta * ((k * x)**rho + (1 - eta) * (y * (l / r))**rho)**(gamma / rho)

model = models.Model('negative',
                     workers=workers,
                     firms=firms,
                     production=F,
                     params=F_params)
```

```
solver = solvers.ShootingSolver(model=model)
```

```
solver.solve(1e0, tol=1e-4, number_knots=10000, integrator='vode',
             atol=1e-9, rtol=1e-6, check=True)
```

    Exhausted firms: initial guess of 0.5 for firm size is too low.

```
solver.integrator.successful()
```

    False

```
solver._solution
```

    array([[  1.00000000e-03,   5.00000000e+01,   7.50000000e-01,
              6.25292214e-02,   1.02985470e-01],
           [  6.00040004e-03,   4.91355141e+01,   2.93698995e-02,
              6.50895813e-01,   1.02614135e-01],
           [  1.10008001e-02,   1.87728995e+01,   6.10881218e-04,
              8.73207163e+00,   9.34543423e-02],
           [  1.34286568e-02,   2.61227847e+06,   1.98653860e+04,
              1.04151668e-01,   1.79568793e+03]])

```
integrat
```

    50.0

```

```


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,y = symbols("x h y")
Image(url= "https://i.imgur.com/bNY57ZF.png")
```

```python
expr = 4/(x-2)
def F(x):
    return expr
F(x)
#first step is to find dF(x)
#then we do slope point form with dF(x) as m   so y = y0 + dF(x)(x-x0)
```

$\displaystyle \frac{4}{x - 2}$

```python
dF = diff(F(x))
dF
```

$\displaystyle - \frac{4}{\left(x - 2\right)^{2}}$

```python
print(dF.subs(x,4))
```

    -1

```python
MofT = (dF.subs(x,4))
y = 2 + MofT*(x - 4)
y
```

$\displaystyle 6 - x$

```python
Image(url= "https://i.imgur.com/MJ6qRQd.png")
```

```python

```


# Bernstein-Vazirani algorithm（Overview）

We explain Bernstein-Vazirani algorithm.

In this algorithm, we consider the following function that takes $x$ as input and computes the output using the internal bit sequence $a$.

$$
f_a(x) = (a\cdot x) \bmod2 = (\sum_i a_i x_i) \bmod2
$$

The goal is to find the bit sequence $a$ from the output of $f_a(x)$.

$a, x$ are $n$-bit strings $a = a_0 a_1... a_{n-1}$, $x = x_0 x_1.... x_{n-1}$, respectively.

Consider the case where $a$ is a 4-bit sequence $1001$.
A foolproof way to find $x$ in classical calculations is to input $x$ in order, with only one bit set to 1 and the others to 0, as in $x=1000, 0100, 0010, 0001$.
From $0 \cdot 0 = 0 \cdot 1 = 0$, the bit with $x_i=0$ do not affect the result, so we can determine the value of $a_i$ one bit at a time.

Calculate $f_a(x)$ where $a=1001$ for four different $x$ as follows.

$(1001 \cdot 1000) \bmod2 = 1$
$(1001 \cdot 0100) \bmod2 = 0$
$(1001 \cdot 0010) \bmod2 = 0$
$(1001 \cdot 0001) \bmod2 = 1$

From the above output, we can get $a=1001$.

The Bernstein-Vazirani algorithm uses quantum superposition to find $a$ in a single calculation.

The specific circuit for the case $a=1001$ is as follows.

Let the $CX$ gate act on the qubits $i$, where $a_i=1$ are the control qubits, respectively.
All target qubits are the fourth auxiliary qubit.

Let's check the states.

$$
\begin{align}
\lvert \psi_1\rangle &= \biggl(\otimes^4 H\lvert 0\rangle \biggr)\otimes H \lvert 1\rangle \\
&= \frac{1}{\sqrt{2}}(\lvert 0\rangle + \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle + \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle + \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle + \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle - \lvert 1\rangle)
\end{align}
$$

Next, check the state of $\lvert \psi_2\rangle$
Let's focus only on the 0th and 4th qubits that the $CX$ gates act on.

$$
\begin{align}
\lvert \psi_1\rangle_{04} &= \frac{1}{\sqrt{2}}(\lvert 0\rangle_0 + \lvert 1\rangle_0) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle_4 - \lvert 1\rangle_4) \\
&= \lvert +\rangle_0 \otimes \lvert -\rangle_4
\end{align}
$$

$$
\begin{align}
\lvert \psi_2\rangle_{04} &= \frac{1}{\sqrt{2}}\lvert 0\rangle_0 \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle_4 - \lvert 1\rangle_4) + \frac{1}{\sqrt{2}}\lvert 1\rangle_0 \otimes \frac{1}{\sqrt{2}}(\lvert 1\rangle_4 - \lvert 0\rangle_4) \\
&= \frac{1}{\sqrt{2}}\lvert 0\rangle_0 \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle_4 - \lvert 1\rangle_4) - \frac{1}{\sqrt{2}}\lvert 1\rangle_0 \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle_4 - \lvert 1\rangle_4) \\
&= \frac{1}{\sqrt{2}}(\lvert 0\rangle_0 - \lvert 1\rangle_0 ) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle_4 - \lvert 1\rangle_4) \\
&=  \lvert -\rangle_0 \otimes \lvert -\rangle_4
\end{align}
$$

By acting on the 0th and 4th qubits with a $CX$ gate, we inverted the phase of the 0th qubit, changing it to $\lvert + \rangle \to \lvert - \rangle$.
Similarly, with $CX$ gates acting on the third and fourth qubits, the state $\lvert\psi_3\rangle$ is as follows

$$
\lvert \psi_3\rangle = \frac{1}{\sqrt{2}}(\lvert 0\rangle - \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle + \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle + \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle - \lvert 1\rangle) \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle - \lvert 1\rangle)
$$

Finally, let the H-gate act.

$$
\lvert \psi_4\rangle = \lvert 1\rangle \otimes \lvert 0\rangle \otimes \lvert 0\rangle \otimes \lvert 1\rangle \otimes \frac{1}{\sqrt{2}}(\lvert 0\rangle - \lvert 1\rangle)
$$

Measure the 0th to third qubits and we get $1001$ as the measurement result.
Now we can find $a$.

Let's implement this with blueqat.

```python
from blueqat import Circuit
import numpy as np
```

First, prepare a function to act oracle $U_f$.

```python
def Uf(c, a):
    N = len(a)
    for i, val in enumerate(list(a)):
        if val == '1':
            c.cx[i, len(a)]
```

The following is the main body of the algorithm.
First, determine the $a$ you want to find by a random number.

Find $a$ from the output result of the quantum circuit using the oracle, and check if the answer is correct.

```python
n = 4

a = ''
for i in range(n):
    a += str(np.random.randint(2))

c = Circuit(n + 1)
c.x[n].h[:]
Uf(c, a)
c.h[:].m[:]
res = c.run(shots = 1000)

print(res)

if [arr[:n] for arr in res.keys()] == [a]:
    print("OK")
else:
    print("incorrect")
```

    Counter({'11001': 1000})
    OK

From the above, we were able to find the $a$ that the oracle has internally by using the Bernstein-Vazirani algorithm.

```python

```


# Exercises

In this section we have two exercises:
1. Implement the polynomial kernel.
2. Implement the multiclass C-SVM.

## Polynomial kernel

You need to extend the ``build_kernel`` function and implement the polynomial kernel if the ``kernel_type`` is set to 'poly'. The equation that needs to be implemented:
\begin{equation}
K=(X^{T}*Y)^{d}.
\end{equation}

```python
def build_kernel(data_set, kernel_type='linear'):
    kernel = np.dot(data_set, data_set.T)
    if kernel_type == 'rbf':
        sigma = 1.0
        objects_count = len(data_set)
        b = np.ones((objects_count, 1))
        diag = np.diag(kernel)*np.ones((1, objects_count))
        kernel -= 0.5 * (np.dot(diag.T, b.T) + np.dot(b, diag))
        kernel = np.exp(kernel / (2. * sigma ** 2))
    elif kernel_type == 'poly':
        kernel = np.dot(np.dot(kernel, np.inv(data_set.T)).T, data_set)**2
    return kernel
```

## Implement a multiclass C-SVM

Use the classification method that we used in notebook 7.3 and IRIS dataset to build a multiclass C-SVM classifier. Most implementation is about a function that will return the proper data set that need to be used for the prediction. You need to implement:
- ``choose_set_for_label``
- ``get_labels_count``

```python
def choose_set_for_label(data_set, label):
    data_set_data = data_set.data
    labels = data_set.target
    # your code here
    data_set_data = data_set_data[labels!=label]
    labels = labels[labels!=label]

    train_data_set, test_data_set, train_labels, test_labels = train_test_split(
        data_set_data, labels, test_size=0.2, random_state=15)

    max_label = np.max(np.unique(labels))
    train_labels[train_labels<max_label] = -1
    train_labels[train_labels==max_label] = 1
    test_labels[test_labels<max_label] = -1
    test_labels[test_labels==max_label] = 1
    return train_data_set, test_data_set, train_labels, test_labels
```

```python
def get_labels_count(data_set):
    return labels_count
```

Use the code that we have implemented earlier:

```python
def train(train_data_set, train_labels, kernel_type='linear', C=10, threshold=1e-5):
    kernel = build_kernel(train_data_set, kernel_type=kernel_type)

    objects_count = len(train_labels)
    P = train_labels * train_labels.transpose() * kernel
    q = -np.ones((objects_count, 1))
    G = np.concatenate((np.eye(objects_count), -np.eye(objects_count)))
    h = np.concatenate((C * np.ones((objects_count, 1)), np.zeros((objects_count, 1))))

    A = train_labels.reshape(1, objects_count)
    A = A.astype(float)
    b = 0.0

    sol = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q), cvxopt.matrix(G), cvxopt.matrix(h), cvxopt.matrix(A), cvxopt.matrix(b))

    lambdas = np.array(sol['x'])

    support_vectors_id = np.where(lambdas > threshold)[0]
    vector_number = len(support_vectors_id)
    support_vectors = train_data_set[support_vectors_id, :]

    lambdas = lambdas[support_vectors_id]
    targets = train_labels[support_vectors_id]

    b = np.sum(targets)
    for n in range(vector_number):
        b -= np.sum(lambdas * targets * np.reshape(kernel[support_vectors_id[n], support_vectors_id], (vector_number, 1)))
    b /= len(lambdas)

    return lambdas, support_vectors, support_vectors_id, b, targets, vector_number

def build_kernel(data_set, kernel_type='linear'):
    kernel = np.dot(data_set, data_set.T)
    if kernel_type == 'rbf':
        sigma = 1.0
        objects_count = len(data_set)
        b = np.ones((len(data_set), 1))
        kernel -= 0.5 * (np.dot((np.diag(kernel)*np.ones((1, objects_count))).T, b.T)
                         + np.dot(b, (np.diag(kernel) * np.ones((1, objects_count))).T.T))
        kernel = np.exp(kernel / (2. * sigma ** 2))
    return kernel

def classify_rbf(test_data_set, train_data_set, lambdas, targets, b, vector_number, support_vectors, support_vectors_id):
    kernel = np.dot(test_data_set, support_vectors.T)
    sigma = 1.0
    #K = np.dot(test_data_set, support_vectors.T)
    #kernel = build_kernel(train_data_set, kernel_type='rbf')
    c = (1. / sigma * np.sum(test_data_set ** 2, axis=1) * np.ones((1, np.shape(test_data_set)[0]))).T
    c = np.dot(c, np.ones((1, np.shape(kernel)[1])))
    #aa = np.dot((np.diag(K)*np.ones((1,len(test_data_set)))).T[support_vectors_id], np.ones((1, np.shape(K)[0]))).T
    sv = (np.diag(np.dot(train_data_set, train_data_set.T))*np.ones((1,len(train_data_set)))).T[support_vectors_id]
    aa = np.dot(sv,np.ones((1,np.shape(kernel)[0]))).T
    kernel = kernel - 0.5 * c - 0.5 * aa
    kernel = np.exp(kernel / (2. * sigma ** 2))

    y = np.zeros((np.shape(test_data_set)[0], 1))
    for j in range(np.shape(test_data_set)[0]):
        for i in range(vector_number):
            y[j] += lambdas[i] * targets[i] * kernel[j, i]
        y[j] += b
    return np.sign(y)
```

```python
from sklearn.datasets import load_iris
import numpy as np
from sklearn.model_selection import train_test_split
import cvxopt

iris = load_iris()

train_data_set, test_data_set, train_labels, test_labels = choose_set_for_label(iris, 0)
# modify this part
lambdas, support_vectors, support_vectors_id, b, targets, vector_number = train(train_data_set, train_labels, kernel_type='poly')
predicted = classify_rbf(test_data_set, train_data_set, lambdas, targets, b, vector_number, support_vectors, support_vectors_id)
predicted = list(predicted.astype(int))

from sklearn.metrics import accuracy_score

print(accuracy_score(predicted, test_labels))
```

         pcost       dcost       gap    pres   dres
     0: -2.2893e+00 -1.7102e+03  4e+03  3e-01  8e-14
     1:  7.8020e-01 -1.0603e+02  2e+02  1e-02  7e-14
     2:  8.7753e-01 -2.1372e+00  4e+00  1e-04  7e-15
     3:  1.4420e-01 -2.3725e-01  4e-01  2e-07  3e-15
     4:  7.9258e-03 -4.6701e-02  5e-02  2e-16  2e-15
     5: -6.8219e-03 -1.3461e-02  7e-03  2e-16  5e-16
     6: -8.3051e-03 -1.1442e-02  3e-03  2e-16  3e-16
     7: -1.0328e-02 -1.1126e-02  8e-04  2e-16  2e-16
     8: -1.0560e-02 -1.0681e-02  1e-04  2e-16  2e-16
     9: -1.0620e-02 -1.0638e-02  2e-05  2e-16  2e-16
    10: -1.0628e-02 -1.0628e-02  2e-07  2e-16  3e-16
    11: -1.0628e-02 -1.0628e-02  2e-09  2e-16  3e-16
    Optimal solution found.
    0.6


# Finite Difference Methods for the Poisson Equation with Zero Boundary
This notebook will focus on numerically approximating a inhomogenous second order Poisson Equation with zero boundary conditions.
## The Differential Equation
The general two dimensional Poisson Equation is of the form:
\begin{equation} \frac{\partial^2 u}{\partial y^2} + \frac{\partial^2 u}{\partial x^2}=f(x,y), \ \ \ (x,y) \in \Omega=(0,1)\times (0,1),\end{equation}
with boundary conditions
\begin{equation}U(x,y) = g(x,y), \ \ \  (x,y)\in\delta\Omega\text{ - boundary}. \end{equation}
## Homogenous Poisson Equation
This notebook will implement a finite difference scheme to approximate the inhomogenous form of the Poisson Equation $f(x,y)=x^2+y^2$, with a zero boundary:
\begin{equation} \frac{\partial^2 u}{\partial y^2} + \frac{\partial^2 u}{\partial x^2}=x^2+y^2.\end{equation}
with the Boundary Conditions:
\begin{equation} u(x,0)=0, \ \ \ \ \ 0 \leq x \leq 1, \text{ lower},\end{equation}
\begin{equation} u(x,1)=0, \ \ \ \ \ 0 \leq x \leq 1, \text{ upper},\end{equation}
\begin{equation} u(0,y)=0, \ \ \ \ \ 0 \leq y \leq 1, \text{ left},\end{equation}
\begin{equation} u(1,y)=0, \ \ \ \ \ 0 \leq y \leq 1, \text{ right}.\end{equation}

```python
# LIBRARY
# vector manipulation
import numpy as np
# math functions
import math

# THIS IS FOR PLOTTING
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import warnings
warnings.filterwarnings("ignore")
from IPython.display import HTML
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt

```

## Discete Grid
The region $\Omega=(0,1)\times(0,1)$ is discretised into a uniform mesh $\Omega_h$. In the  $x$ and $y$ directions into $N$ steps giving a stepsize of
\begin{equation} h=\frac{1-0}{N},\end{equation}
resulting in
\begin{equation}x[i]=0+ih, \ \ \  i=0,1,...,N,\end{equation}
and
\begin{equation}x[j]=0+jh, \ \ \  j=0,1,...,N,\end{equation}
The Figure below shows the discrete grid points for $N=10$,  the known boundary conditions (green),  and the unknown values (red) of the Poisson Equation.

```python
N=10
h=1/N
x=np.arange(0,1.0001,h)
y=np.arange(0,1.0001,h)
X, Y = np.meshgrid(x, y)
fig = plt.figure()
plt.plot(x[1],y[1],'ro',label='unknown');
plt.plot(X,Y,'ro');
plt.plot(np.ones(N+1),y,'go',label='Boundary Condition');
plt.plot(np.zeros(N+1),y,'go');
plt.plot(x,np.zeros(N+1),'go');
plt.plot(x, np.ones(N+1),'go');
plt.xlim((-0.1,1.1))
plt.ylim((-0.1,1.1))
plt.xlabel('x')
plt.ylabel('y')
plt.gca().set_aspect('equal', adjustable='box')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title(r'Discrete Grid $\Omega_h,$ h= %s'%(h),fontsize=24,y=1.08)
plt.show();
```

## Boundary Conditions

The  discrete boundary conditions are
\begin{equation} w[i,0]=0, \text{ for } i=0,...,10, \text{ upper},\end{equation}
\begin{equation} w[i,N]=0, \text{ for } i=0,...,10,  \text{ lower},\end{equation}
\begin{equation} w[0,j]=0, \text{ for } j=0,...,10,   \text{ left},\end{equation}
\begin{equation} w[N,j]=0, \text{ for } i=0,...,10,\text{ right}. \end{equation}

The Figure below plots the boundary values of $w[i,j]$.

```python
w=np.zeros((N+1,N+1))

for i in range (0,N):
        w[i,0]=0 #left Boundary
        w[i,N]=0 #Right Boundary

for j in range (0,N):
        w[0,j]=0 #Lower Boundary
        w[N,j]=0 #Upper Boundary

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
# Plot a basic wireframe.
ax.plot_wireframe(X, Y, w,color='r', rstride=10, cstride=10)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('w')
plt.title(r'Boundary Values',fontsize=24,y=1.08)
plt.show()
```

## Numerical Method
The Poisson Equation  is discretised using
$\delta_x^2$ is the central difference approximation of the second derivative in the $x$ direction
\begin{equation}\delta_x^2=\frac{1}{h^2}(w_{i+1j}-2w_{ij}+w_{i-1j}), \end{equation}
and $\delta_y^2$ is the central difference approximation of the second derivative in the $y$ direction
\begin{equation}\delta_y^2=\frac{1}{h^2}(w_{ij+1}-2w_{ij}+w_{ij-1}). \end{equation}
The gives the Poisson Difference Equation,
\begin{equation}-(\delta_x^2w_{ij}+\delta_y^2w_{ij})=f_{ij} \ \ (x_i,y_j) \in \Omega_h, \end{equation}
\begin{equation}w_{ij}=g_{ij} \ \ (x_i,y_j) \in \partial\Omega_h, \end{equation}
where $w_ij$ is the numerical approximation of $U$ at $x_i$ and $y_j$.
Expanding the  the Poisson Difference Equation gives the five point method,
\begin{equation}-(w_{i-1j}+w_{ij-1}-4w_{ij}+w_{ij+1}+w_{i+1j})=h^2f_{ij} \end{equation}
for $i=1,...,N-1$ and $j=1,...,N-1.$

### Matrix form
This can be written as a system of $(N-1)\times(N-1)$ equations can be arranged in matrix form
\begin{equation} A\mathbf{w}=\mathbf{r},\end{equation}
where $A$ is an $(N-1)^2\times(N-1)^2$  matrix made up of the following block tridiagonal structure
\begin{equation}\left(\begin{array}{ccccccc}
T&I&0&0&.&.&.\\
I&T&I&0&0&.&.\\
.&.&.&.&.&.&.\\
.&.&.&0&I&T&I\\
.&.&.&.&0&I&T\\
\end{array}\right),
\end{equation}
where $I$ denotes an $N-1 \times N-1$ identity matrix and $T$ is the tridiagonal matrix of the form:
\begin{equation} T=\left(\begin{array}{ccccccc}
-4&1&0&0&.&.&.\\
1&-4&1&0&0&.&.\\
.&.&.&.&.&.&.\\
.&.&.&0&1&-4&1\\
.&.&.&.&0&1&-4\\
\end{array}\right).
\end{equation}
The plot below shows the matrix $A$ and its inverse $A^{-1}$ as a colourplot.

```python
N2=(N-1)*(N-1)
A=np.zeros((N2,N2))
## Diagonal
for i in range (0,N-1):
    for j in range (0,N-1):
        A[i+(N-1)*j,i+(N-1)*j]=-4

# LOWER DIAGONAL
for i in range (1,N-1):
    for j in range (0,N-1):
        A[i+(N-1)*j,i+(N-1)*j-1]=1
# UPPPER DIAGONAL
for i in range (0,N-2):
    for j in range (0,N-1):
        A[i+(N-1)*j,i+(N-1)*j+1]=1

# LOWER IDENTITY MATRIX
for i in range (0,N-1):
    for j in range (1,N-1):
        A[i+(N-1)*j,i+(N-1)*(j-1)]=1

# UPPER IDENTITY MATRIX
for i in range (0,N-1):
    for j in range (0,N-2):
        A[i+(N-1)*j,i+(N-1)*(j+1)]=1
Ainv=np.linalg.inv(A)
fig = plt.figure(figsize=(12,4));
plt.subplot(121)
plt.imshow(A,interpolation='none');
clb=plt.colorbar();
clb.set_label('Matrix elements values');
plt.title('Matrix A ',fontsize=24)
plt.subplot(122)
plt.imshow(Ainv,interpolation='none');
clb=plt.colorbar();
clb.set_label('Matrix elements values');
plt.title(r'Matrix $A^{-1}$ ',fontsize=24)

fig.tight_layout()
plt.show();
```

The vector $\mathbf{w}$ is of length $(N-1)\times(N-1)$ which made up of $N-1$ subvectors $\mathbf{w}_j$  of length $N-1$ of the form
\begin{equation}\mathbf{w}_j=\left(\begin{array}{c}
w_{1j}\\
w_{2j}\\
.\\
.\\
w_{N-2j}\\
w_{N-1j}\\
\end{array}\right).
\end{equation}
The vector $\mathbf{r}$ is of length $(N-1)\times(N-1)$ which made up of $N-1$ subvectors of the form $\mathbf{r}_j=-h^2\mathbf{f}_j-\mathbf{bx}_{j}-\mathbf{by}_j$. In this example the boundary is $0$ which means that
\begin{equation}\mathbf{bx}_j =0,\end{equation}
\begin{equation}
\mathbf{by}_{j} =0,
\end{equation}
and
\begin{equation}\mathbf{f}_j =\left(\begin{array}{c}
x_1^2+y_j^2\\
x_2^2+y_j^2\\
.\\
.\\
x_{N-2}^2+y_j^2\\
x_{N-1}^2+y_j^2\\
\end{array}\right)
\end{equation}
for $j=1,...,N-1$.

```python
r=np.zeros(N2)

# vector r
for i in range (0,N-1):
    for j in range (0,N-1):
        r[i+(N-1)*j]=h*h*(x[i]*x[i]+y[j]*y[j])
# Boundary
b_bottom_top=np.zeros(N2)
for i in range (0,N-1):
    b_bottom_top[i]=0 #Bottom Boundary
    b_bottom_top[i+(N-1)*(N-2)]=0# Top Boundary

b_left_right=np.zeros(N2)
for j in range (0,N-1):
    b_left_right[(N-1)*j]=0 # Left Boundary
    b_left_right[N-2+(N-1)*j]=0# Right Boundary

b=b_left_right+b_bottom_top
```

## Results

To solve the system for $\mathbf{w}$ invert the matrix $A$
\begin{equation} A\mathbf{w}=\mathbf{r},\end{equation}
such that
\begin{equation} \mathbf{w}=A^{-1}\mathbf{r}.\end{equation}
Lastly, as $\mathbf{w}$ is in vector it has to be reshaped into grid form to plot.

The figure below shows the numerical approximation of the homogeneous Equation.

```python
C=np.dot(Ainv,r-b)
w[1:N,1:N]=C.reshape((N-1,N-1))

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111, projection='3d');
# Plot a basic wireframe.
ax.plot_wireframe(X, Y, w,color='r');
ax.set_xlabel('x');
ax.set_ylabel('y');
ax.set_zlabel('w');
plt.title(r'Numerical Approximation of the Poisson Equation',fontsize=24,y=1.08);
plt.show();
```

# Consistency and Convergence
We now ask how well the grid function determined by the five point scheme approximates the exact solution of the Poisson problem.
## Consistency

### Consitency (Definition)
Let
\begin{equation}\nabla^2_h(\varphi)=-(\varphi_{i-1j}+\varphi_{ij-1}-4\varphi_{ij}+\varphi_{ij+1}+\varphi_{i+1j})\end{equation}
denote the finite difference approximation associated with the grid $\Omega_h$ having the mesh size $h$, to a partial differential operator
\begin{equation}\nabla^2(\varphi)=\frac{\partial^2 \varphi}{\partial x^2}+\frac{\partial^2 \varphi}{\partial y^2}\end{equation} defined on
a simply connected, open set $\Omega \subset R^2$. For a given function $\varphi\in C^{\infty}(\Omega)$,
the truncation error of $\nabla^2_h$ is
\begin{equation}\tau_{h}(\mathbf{x})=(\nabla^2-\nabla^2_h)\varphi(\mathbf{x}) \end{equation}
The approximation $\nabla^2_h$ is consistent with $\nabla^2$ if
\begin{equation}\lim_{h\rightarrow 0}\tau_h(\mathbf{x})=0,\end{equation}
for all $\mathbf{x} \in D$ and all $\varphi \in C^{\infty}(\Omega)$. The approximation is consistent to order $p$ if $\tau_h(\mathbf{x})=O(h^p)$.

_In other words a method is consistent with the differential equation it is approximating._

## Proof of Consistency
The five-point difference analog $\nabla^2_h$ is consistent to order 2 with $\nabla^2$.

__Proof__

Pick $\varphi \in C^{\infty}(D)$, and let $(x,y) \in \Omega$ be a point such that $(x\pm h, y),(x,y \pm h) \in \Omega\bigcup \partial\Omega$.  By the Taylor Theorem
\begin{eqnarray*}
\varphi(x\pm h,y)&=&\varphi(x,y) \pm h \frac{\partial \varphi}{\partial x}(x,y)+\frac{h^2}{2!}\frac{\partial^2 \varphi}{\partial x^2}(x,y) \pm\frac{h^3}{3!}\frac{\partial^3 \varphi}{\partial x^3}(x,y)+\frac{h^4}{4!}\frac{\partial^4 \varphi}{\partial x^4}(\zeta^{\pm},y)
\end{eqnarray*}
where $\zeta^{\pm} \in (x-h,x+h)$. Adding this pair of equation together and rearranging , we get
\begin{equation}\frac{1}{h^2}[\varphi(x+h,y)-2\varphi(x,y)+\varphi(x-h,y) ] -\frac{\partial^2 \varphi}{\partial x^2}(x,y)=\frac{h^2}{4!}\left[\frac{\partial^4 \varphi}{\partial x^4}(\zeta^{+},y)+
\frac{\partial^4 \varphi}{\partial x^4}(\zeta^{-},y)
 \right]
\end{equation}
By the intermediate value theorem
\end{equation}\left[\frac{\partial^4 \varphi}{\partial x^4}(\zeta^{+},y)+
\frac{\partial^4 \varphi}{\partial x^4}(\zeta^{-},y)
 \right]
=2\frac{\partial^4 \varphi}{\partial x^4}(\zeta,y),\end{equation}
for some $\zeta \in (x-h,x+h)$.  Therefore,
\begin{equation}\delta_x^2(x,y)
=\frac{\partial^2 \varphi}{\partial x^2}(x,y)+\frac{h^2}{2!}\frac{\partial^4 \varphi}{\partial x^4}(\zeta,y)\end{equation}
Similar reasoning shows that
\begin{equation}\delta_y^2(x,y)
=\frac{\partial^2 \varphi}{\partial y^2}(x,y)+\frac{h^2}{2!}\frac{\partial^4 \varphi}{\partial y^4}(x,\eta)
\end{equation}
for some $\eta \in (y-h,y+h)$. We conclude that $\tau_h(x,y)=(\nabla-\nabla_h)\varphi(x,y)=O(h^2).$

## Convergence
### Definition
Let $\nabla^2_hw(\mathbf{x}_j)=f(\mathbf{x}_j)$ be a finite difference approximation, defined on a grid mesh size $h$, to a PDE $\nabla^2U(\mathbf{x})=f(\mathbf{x})$ on a simply connected set $D \subset R^n$. Assume that $w(x,y)=U(x,y)$ at all points $(x,y)$ on the boundary $\partial\Omega$.  The finite difference scheme converges (or is convergent) if
\end{equation} \max_j|U(\mathbf{x}_j)-w(\mathbf{x}_j)| \rightarrow 0 \mbox{  as  } h \rightarrow 0.\end{equation}

### Theorem (DISCRETE MAXIMUM PRINCIPLE).
If $\nabla^2_hV_{ij}\geq 0$ for all points $(x_i,y_j) \in \Omega_h$, then
\begin{equation} \max_{(x_i,y_j)\in\Omega_h}V_{ij}\leq  \max_{(x_i,y_j)\in\partial\Omega_h}V_{ij},\end{equation}
If $\nabla^2_hV_{ij}\leq 0$ for all points $(x_i,y_j) \in \Omega_h$, then
\begin{equation} \min_{(x_i,y_j)\in\Omega_h}V_{ij}\geq  \min_{(x_i,y_j)\in\partial\Omega_h}V_{ij}.\end{equation}

### Propositions
1. The zero grid function for which $U_{ij}=0$ for all $(x_i,y_j) \in \Omega_h \bigcup \partial\Omega_h$
is the only solution to the finite difference problem
\begin{equation}\nabla_h^2U_{ij}=0 \mbox{ for }(x_i,y_j)\in\Omega_h,\end{equation}
\begin{equation}U_{ij}=0 \mbox{ for }(x_i,y_j)\in\partial\Omega_h.\end{equation}

2. For prescribed grid functions $f_{ij}$ and $g_{ij}$, there exists a unique solution to the problem
\begin{equation}\nabla_h^2U_{ij}=f_{ij} \mbox{ for }(x_i,y_j)\in\Omega_h,\end{equation}
\begin{equation}U_{ij}=g_{ij} \mbox{ for }(x_i,y_j)\in\partial\Omega_h.\end{equation}

### Definition
For any grid function $V:\Omega_h\bigcup\partial\Omega_h \rightarrow R$,
\begin{equation}||V||_{\Omega} =\max_{(x_i,y_j)\in\Omega_h}|V_{ij}|, \end{equation}
\begin{equation}||V||_{\partial\Omega} =\max_{(x_i,y_j)\in\partial\Omega_h}|V_{ij}|. \end{equation}

### Lemma
If the grid function $V:\Omega_h\bigcup\partial\Omega_h\rightarrow R$ satisfies the boundary condition $V_{ij}=0$ for $(x_i,y_j)\in \partial\Omega_h$, then
\begin{equation}||V_||_{\Omega}\leq \frac{1}{8}||\nabla_h^2V||_{\Omega}. \end{equation}

Given these Lemmas and Propositions, we can now prove that the solution to the five point scheme $\nabla^2_h$ is convergent to the exact solution of the Poisson Equation $\nabla^2$.
### Convergence Theorem
Let $U$ be a solution to the Poisson equation and let $w$ be the grid function
that satisfies the discrete analog
\begin{equation}-\nabla_h^2w_{ij}=f_{ij} \ \ \mbox{ for } (x_i,y_j)\in\Omega_h, \end{equation}
\begin{equation}w_{ij}=g_{ij} \ \ \mbox{ for } (x_i,y_j)\in\partial\Omega_h. \end{equation}
Then there exists a positive constant $K$ such that
\begin{equation}||U-w||_{\Omega}\leq KMh^2, \end{equation}
where
\begin{equation} M=\left\{
\left|\left|\frac{\partial^4 U}{\partial x^4} \right|\right|_{\infty},
\left|\left|\frac{\partial^4 U}{\partial y^4} \right|\right|_{\infty}
 \right\}\end{equation}

 __Proof__

 The statement of the theorem assumes that $U\in C^4(\bar{\Omega})$. This assumption
holds if $f$ and $g$ are smooth enough.
\begin{proof}
Following from the proof of the Proposition we have
\begin{equation} (\nabla_h^2-\nabla^2)U_{ij}=\frac{h^2}{12}\left[ \frac{\partial^4 U}{\partial x^4}(\zeta_i,y_j)+\frac{\partial^4 U}{\partial y^4}(x_i,\eta_j) \right],\end{equation}
for some $\zeta \in (x_{i-1},x_{i+1})$ and $\eta_j\in(y_{j-1},y_{j+1})$.  Therefore,
\begin{equation} -\nabla_h^2U_{ij}=f_{ij}-\frac{h^2}{12}\left[ \frac{\partial^4 U}{\partial x^4}(\zeta_i,y_j)+\frac{\partial^4 U}{\partial y^4}(x_i,\eta_j) \right].\end{equation}
If we subtract from this the identity equation $-\nabla_h^2w_{ij}=f_{ij}$ and note
that $U-w$ vanishes on $\partial\Omega_h$, we find that
\begin{equation} \nabla_h^2(U_{ij}-w_{ij})=\frac{h^2}{12}\left[ \frac{\partial^4 U}{\partial x^4}(\zeta_i,y_j)+\frac{\partial^4 U}{\partial y^4}(x_i,\eta_j) \right].\end{equation}
It follows that

\begin{equation} ||U-w||_{\Omega}\leq\frac{1}{8}||\nabla_h^2(U-w)||_{\Omega}\leq KMh^2.\end{equation}

```python

```

```python

```

Chapter 09 - Elliptic Equations/902_Poisson Equation-Zero Boundary Conditions.ipynb


# Extinction Efficiency Factor

Figure 6.5 from Chapter 6 of *Interstellar and Intergalactic Medium* by Ryden & Pogge, 2021,
Cambridge University Press.

Plot the efficiency factor Q$_{ext}$ for two values of the real index of refraction, $n_r=1.5$ (glass) and
$n_r=1.33$ (water ice).

Uses van de Hulst's method to compute Mie scattering in the astrophysically interesting limit that the
spherical scatterer is large compared to the wavelength of light ($2\pi a >> \lambda$) and only moderately
refractive and absorptive at wavelengths of interest ($|n-1|<1$).  In this limit, the solution for the pure
scattering case yeilds an efficiency factor
\begin{equation}
   Q_{ext} = Q_{sca}\approx 2-\frac{4}{\varrho}\sin\varrho + \frac{4}{\varrho^2}(1-\cos\varrho)
\end{equation}
where
\begin{equation}
   \varrho = 2\left(\frac{2\pi a}{\lambda}\right)|n_r-1|
\end{equation}
in the short-wavelength limit this function approaches the limiting value of $Q_{ext}=2.0$.

```python
%matplotlib inline

import os
import sys
import math
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import MultipleLocator, LogLocator, NullFormatter

import warnings
warnings.filterwarnings('ignore',category=UserWarning, append=True)
```

## Standard Plot Format

Setup the standard plotting format and make the plot. Fonts and resolution adopted follow CUP style.

```python
figName = 'Fig6_5'

# graphic aspect ratio = width/height

aspect = 4.0/3.0 # 4:3

# Text width in inches - don't change, this is defined by the print layout

textWidth = 6.0 # inches

# output format and resolution

figFmt = 'png'
dpi = 600

# Graphic dimensions

plotWidth = dpi*textWidth
plotHeight = plotWidth/aspect
axisFontSize = 10
labelFontSize = 6
lwidth = 0.5
axisPad = 5
wInches = textWidth
hInches = wInches/aspect

# Plot filename

plotFile = f'{figName}.{figFmt}'

# LaTeX is used throughout for markup of symbols, Times-Roman serif font

plt.rc('text', usetex=True)
plt.rc('font', **{'family':'serif','serif':['Times-Roman'],'weight':'bold','size':'16'})

# Font and line weight defaults for axes

matplotlib.rc('axes',linewidth=lwidth)
matplotlib.rcParams.update({'font.size':axisFontSize})

# axis and label padding

plt.rcParams['xtick.major.pad'] = f'{axisPad}'
plt.rcParams['ytick.major.pad'] = f'{axisPad}'
plt.rcParams['axes.labelpad'] = f'{axisPad}'
```

## Scattering Efficiency Factor

Do the computation for real two indices of refraction
 * $n_r=1.5$ - typical of glass (fused silica) at visible wavelengths
 * $n_r=1.33$ - typical of pure water ice at visible wavelenths

parameterize using $x=2\pi a/\lambda$.

```python
# Range of x

xMin = 0.01
xMax = 14.0
x = np.linspace(xMin,xMax,501)

yMin = 0.0
yMax = 4.0

# glass (fused silica)

nr = 1.5
rho = 2.0*x*(nr-1.0)
Qext1 = 2.0 - (4.0/rho)*np.sin(rho)+(4.0/(rho*rho))*(1-np.cos(rho))

# pure water ice

nr = 1.33
rho = 2.0*x*(nr-1.0)
Qext2 = 2.0 - (4.0/rho)*np.sin(rho)+(4.0/(rho*rho))*(1-np.cos(rho))
```

## Make the plot

Plot $n_r$=1.5 (glass) as a solid line, $n_r$=1.33 (ice) as a dotted line, with $Q_{ext}$=2.0 for reference
(see text).

```python
fig,ax = plt.subplots()

fig.set_dpi(dpi)
fig.set_size_inches(wInches,hInches,forward=True)

ax.tick_params('both',length=6,width=lwidth,which='major',direction='in',top='on',right='on')
ax.tick_params('both',length=3,width=lwidth,which='minor',direction='in',top='on',right='on')

# Limits

plt.xlim(xMin,xMax)
ax.xaxis.set_major_locator(MultipleLocator(2))
ax.xaxis.set_minor_locator(MultipleLocator(1))
plt.xlabel(r'$x=2\pi a/\lambda$',fontsize=axisFontSize)

plt.ylim(yMin,yMax)
ax.yaxis.set_major_locator(MultipleLocator(1))
ax.yaxis.set_minor_locator(MultipleLocator(0.5))
plt.ylabel(r'Q$_{\rm ext}$',fontsize=axisFontSize)

# glass (nr=1.5)

plt.plot(x,Qext1,'-',color='black',lw=1.0,zorder=10)
plt.text(4.0,3.2,r'$n_{r}$=1.5',fontsize=axisFontSize,ha='right',color='black')

# ice (nr=1.33)

plt.plot(x,Qext2,':',color='black',lw=1.0,zorder=10)
plt.text(7.5,3.0,r'$n_{r}$=1.33',fontsize=axisFontSize,ha='left',color='black')

# large x asymptote at Qext=2

plt.hlines(2.0,xMin,xMax,ls='--',color='black',lw=0.5,zorder=8)

plt.plot()
plt.savefig(plotFile,bbox_inches='tight',facecolor='white')
```


# EECS 445: Machine Learning

## Hands On 10: Bias Variance Tradeoff

Consider a sequence of IID random variable:
$$
X_i =
\begin{cases}
100 & \text{ with prob. } 0.02 \\
0 & \text{ with prob. } 0.97 \\
-100 & \text{ with prob. } 0.01 \\
\end{cases}
$$
The true mean of $X_i$ is
$$
0.02 \times 100 + 0.97 \times 0 + 0.01 \times -100 = 1
$$

We want to estimate the true mean of this distribution. We will consider two different estimators of the true mean.
Let's say you take three samples $X_1, X_2, X_3$, and you compute the **empirical mean** $Z=\frac{X_1 + X_2 + X_3}{3}$ and **empirical median** $Y$ of these three samples (recall that the median is obtained by sorting $X_1, X_2, X_3$ and then choosing the middle (2nd) entry).

### What is the bias-variance tradeoff of the  $Y$ and $Z$ for estimating the true mean of the above distribution?

* They are both unbiased estimators of the true mean, and have the same variance.
* The median has higher bias and higher variance.
* The mean has higher bias and higher variance.
* They both have no bias, but the mean has lower variance.
* The mean has no bias but some variance, and the median has non-zero bias but less variance

### Solution
> The last answer is correct.

> The empirical mean of a sample of random $n$ IID random variables is always an unbiased estimate of the true mean. However, the empirical mean estimator can have high variance. Here it is $ \text{Var}(Z) = \frac{\text{Var}(X_i)}{3} =  \frac{(100-1)^2 \times 0.02 + (-100 - 1)^2 \times 0.01 + (0-1)^2 \times 0.97}{3} = 99 \frac 2 3.$

>The median, on the other hand, is a biased estimator. It is a little bit hard to calculate exactly, but here goes:
$$
median = \begin{cases} 100 &  w.p. 0.02^3 + \binom{3}{1} 0.02^2 \times 0.98 \\
-100 & w.p. 0.01^3 + \binom{3}{1} 0.01^2 \times 0.99
\end{cases}
$$
If you work this out, you see that the median on average is $0.089$. This means that the $\text{bias}^2 \approx (1-0.089)^2$ which is no more than 1. Using a similar argument, you can check that the variance of the median is no more than 20. This can be checked experimentally!

## Derivation of Bias-Variance Tradeoff eqaution
Assume that we have noisy data, modeled by $f = y + \epsilon$, where $\epsilon \in \mathcal{N}(0,\sigma)$. Given an estimator $\hat{f}$, the squared error can be derived as follows:

$$
\begin{align}
\mathbb{E}\left[\left(\hat{f} - f\right)^2\right] &= \mathbb{E}\left[\hat{f}^2 - 2f\hat{f} + f^2\right]\\
&= \mathbb{E}\left[\hat{f}^2\right] + \mathbb{E}\left[f^2\right] - 2\mathbb{E}\left[f\hat{f}^2\right] \text{ By linearity of expectation} \\
\end{align}
$$
Now, by definition, $Var(x) = \mathbb{E}\left[x^2\right] - \left(\mathbb{E}\left[x\right]\right)^2$. Subsituting this definition into the eqaution above, we get:
$$
\begin{align}
\mathbb{E}\left[\hat{f}^2\right] + \mathbb{E}\left[f^2\right] - 2\mathbb{E}\left[f\hat{f}^2\right] &= Var(\hat{f}) + \left(\mathbb{E}[\hat{f}]\right)^2  + Var(f) + \left(\mathbb{E}[f]\right)^2 - 2f\mathbb{E}[\hat{F}^2] \\
&= Var(\hat{f}) + Var(f) + \left(\mathbb{E}[\hat{f}] - f\right)^2\\
&= \boxed{\sigma + Var(\hat{f}) + \left(\mathbb{E}[\hat{f}] - f\right)^2}
\end{align}
$$

The first term $\sigma$ is the irreducible error due to the noise in the data (from the distribution of $\epsilon$). The second term is the **variance** of the estimator $\hat{f}$ and the final term is the **bias** of the estimator. There is an inherent tradeoff between the bias and variance of an estimator. Generally, more complex estimators (think of high-degree polynomials as an example) will have a low bias since they will fit the sampled data really well. However, this accuracy will not be maintained if we continued to resample the data, which implies that the variance of this estimator is high.

## Activity 1: Bias Variance Tradeoff
We will now see try to see the inherent tradeoff between bias and variance of estimators through linear regression. Consider the following dataset.

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy.matlib import repmat

from sklearn
degrees = [1,2,3,4,5]

#define data
n = 20
sub = 1000
mean = 0
std = 0.25

#define test set
Xtest = np.random.random((n,1))*2*np.pi
ytest = np.sin(Xtest) + np.random.normal(mean,std,(n,1))

#pre-allocate variables
preds = np.zeros((n,sub))
bias = np.zeros(len(degrees))
variance = np.zeros(len(degrees))
mse = np.zeros(len(degrees))
values = np.expand_dims(np.linspace(0,2*np.pi,100),1)

```

Let's try several polynomial fits to the data:

```python
for j,degree in enumerate(degrees):

    for i in range(sub):

        #create data - sample from sine wave
        x = np.random.random((n,1))*2*np.pi
        y = np.sin(x) + np.random.normal(mean,std,(n,1))

        #TODO
        #create features corresponding to degree - ex: 1, x, x^2, x^3...
        A =

        #TODO:
        #fit model using least squares solution (linear regression)
        #later include ridge regression/normalization
        coeffs =

        #store predictions for each sampling
        preds[:,i] = poly.fit_transform(Xtest).dot(coeffs)[:,0]

        #plot 9 images
        if i < 9:
            plt.subplot(3,3,i+1)
            plt.plot(values,poly.fit_transform(values).dot(coeffs),x,y,'.b')

    plt.axis([0,2*np.pi,-2,2])
    plt.suptitle('PolyFit = %i' % (degree))
    plt.show()

    #TODO
    #Calculate mean bias, variance, and MSE (UNCOMMENT CODE BELOW!)
    #bias[j] =
    #variance[j] =
    #mse[j] =

```

Let's plot the data with the estimators!

```python
plt.subplot(3,1,1)
plt.plot(degrees,bias)
plt.title('bias')
plt.subplot(3,1,2)
plt.plot(degrees,variance)
plt.title('variance')
plt.subplot(3,1,3)
plt.plot(degrees,mse)
plt.title('MSE')
plt.show()
```


# Rayleigh damping in multi-degree-of-freedom (MDOF) systems

[Introduction](#Introduction)

[Dynamic equilibrium equation](#Dynamic-equilibrium-equation)

[Computational lab](#Computational-lab)

[Mass proportional damping](#Mass-proportional-damping)

[Stiffness proportional damping](#Stiffness-proportional-damping)

[Rayleigh damping](#Rayleigh-damping)

[Example](#Example)

[References](#References)

[Odds and ends](#Odds-and-ends)

## Introduction

In structural dynamics, mass and stiffness can be computed from the geometric characteristics and material properties of a structure but damping can only be estimated based on the fact that structural dynamic responses are, well, [damped](http://www.edwilson.org/BOOK-Wilson/19-DAMP.pdf). It is usually assumed that such damping is viscous, [in the absence of more accurate information](http://www-g.eng.cam.ac.uk/dv_library/Theses/sondiponthesis.pdf), which fits nicely in the solution of the [dynamic equilibrium equation](#Dynamic-equilibrium-equation).

This python notebook will explore the influence of the following special cases in the computation of the dynamic response of MDOF systems:

1. [Mass proportional damping](#Mass-proportional-damping)
2. [Stiffness proportional damping](#Stiffness-proportional-damping)
3. [Rayleigh damping](#Rayleigh-damping)

What makes these cases special is the fact that the damping matrix is orthogonal to the modal matrix and, therefore, it is diagonalizable. This constitutes a clear advantage when it comes to solving the dynamic equilibrium equation system for MDOF systems.

[Back to top](#top)

## Dynamic equilibrium equation

In structural dynamics the second order differential dynamic equilibrium equation can be written in terms of generalized coordinates (d[isplacement]) and their first (v[elocity]) and second (a[cceleration]) time derivatives:

\begin{equation}
\mathbf{M} \times \mathbf{a(t)} + \mathbf{C} \times \mathbf{v(t)} + \mathbf{K} \times \mathbf{d(t)} = \mathbf{F(t)}
\end{equation}

where:

> $\mathbf{M}$ is the mass matrix
> $\mathbf{C}$ is the damping matrix
> $\mathbf{K}$ is the stiffness matrix
> $\mathbf{a(t)}$ is the acceleration vector
> $\mathbf{v(t)}$ is the velocity vector
> $\mathbf{d(t)}$ is the displacement vector
> $\mathbf{F(t)}$ is the force input vector

In a MDOF systems all these matrices are of size $NDOF \times NDOF$, where $NDOF$ is the number of generalized degrees of freedom. Carrying out the usual coordinate transformation from generalized coordinates to modal coordinates, $\mathbf{d(t)} = \mathbf{\Phi} \times \mathbf{q(t)}$, and pre-multiplying by the transpose of the modal matrix, $\mathbf{\Phi}^T$, one obtains the following:

\begin{equation}
\mathbf{\Phi}^T \times \mathbf{M} \times \mathbf{\Phi} \times \mathbf{\ddot q(t)} + \mathbf{\Phi}^T \times \mathbf{C} \cdot \mathbf{\Phi} \times \mathbf{\dot q(t)} + \mathbf{\Phi}^T \times \mathbf{K} \times \mathbf{\Phi} \times \mathbf{q(t)} = \mathbf{\Phi}^T \times \mathbf{F(t)}
\end{equation}

where:

> $\mathbf{\Phi}^T \times \mathbf{M} \times \mathbf{\Phi} = \mathbf{M_n}$ is a diagonal matrix (with modal mass in the main diagonal)
> $\mathbf{\Phi}^T \times \mathbf{K} \times \mathbf{\Phi} = \mathbf{K_n}$ is a diagonal matrix (with modal stiffness in the main diagonal)
> $\mathbf{\Phi}^T \times \mathbf{F(t)} = \mathbf{F_n(t)}$ is a column vector (with modal excitation functions)

In what concerns the product $\mathbf{\Phi}^T \times \mathbf{C} \times \mathbf{\Phi}$ it will be a diagonal matrix (with modal damping in the main diagonal) only in certain circunstances. This is achieved when damping is proportional to either the mass or the stiffness or a combination of both, which is usually referred to as [Rayleigh damping](http://www.ejge.com/2003/Ppr0318/Ppr0318.pdf). In its generalised form it is represented by the [Caughey series](http://engweb.swan.ac.uk/~adhikaris/fulltext/conference/ftc27.pdf) (see [References](#References) section for more information):

\begin{equation}
\mathbf{C} = \mathbf{M} \times \sum_{j=0}^{N-1}{\alpha_j \cdot \left[ \mathbf{M}^{-1} \times \mathbf{K} \right]^j}
\end{equation}

When this happens, the previous dynamic equilibrium equation system transforms into a set of $NDOF$ one-degree-of-freedom independent dynamic equilibrium equations (modal equations):

\begin{equation}
\mathbf{M_n} \times \mathbf{\ddot q(t)} + \mathbf{C_n} \times \mathbf{\dot q(t)} + \mathbf{K_n} \times \mathbf{q(t)} = \mathbf{F_n(t)}
\end{equation}

The $NDOF$ independent modal equilibrium equations can be rewritten as:

\begin{equation}
\mathbf{\ddot q_n(t)} + \mathbf{2 \cdot \zeta_n \cdot \omega_n} \cdot \mathbf{\dot q_n(t)} + \mathbf{\omega_n^2} \cdot \mathbf{q_n(t)} = \mathbf{a_n(t)}
\end{equation}

where:

> $\zeta_n$ is the modal damping coefficient of mode $N$
> $\omega_n$ is the modal angular frequency of mode $N$
> $a_n(t)$ is the modal excitation of mode $N$

The solution of these $NDOF$ independent dynamic equilibrium equations will follow the standard procedures for the one-degree-of-freedom case. The final solution for the MDOF system ($d(t)$) is obtained by superposing the $NDOF$ modal solutions $q_n(t)$.

We will look now at three different cases where the damping matrix is diagonalizable.

[Back to top](#top)

## Computational lab

Before proceeding any further, let us set the computational lab for this Python notebook:

```python
import sys
import math
import numpy as np
import matplotlib as mpl
print('System: {}'.format(sys.version))
for package in (np, mpl):
    print('Package: {} {}'.format(package.__name__, package.__version__))
```

    System: 3.5.2 |Anaconda custom (64-bit)| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)]
    Package: numpy 1.11.2
    Package: matplotlib 1.5.3

We will produce some plots based on a frequency range to illustrate the concepts:

```python
import matplotlib.pyplot as plt
%matplotlib inline

ff = np.linspace(0.01, 6., num=600)
wn = 2.*np.pi*ff
```

[Back to top](#top)

## Mass proportional damping

Mass proportional damping means that the damping matrix is somehow a multiple of the mass matrix:

\begin{equation}
\mathbf{C} = \alpha \cdot \mathbf{M}
\end{equation}

where $\alpha$ is the constant of mass proportionality. In these circunstances, the dynamic equilibrium equation can be written as:

\begin{equation}
\mathbf{M} \times \mathbf{a(t)} + \alpha \cdot \mathbf{M} \times \mathbf{v(t)} + \mathbf{K} \times \mathbf{d(t)} = \mathbf{F(t)}
\end{equation}

Proceeding the same as above, one obtains the $NDOF$ independent modal equilibrium equations:

\begin{equation}
\mathbf{M_n} \times \mathbf{\ddot q_n(t)} + \alpha \cdot \mathbf{M_n} \times \mathbf{\dot q_n(t)} + \mathbf{K_n} \times \mathbf{q_n(t)} = \mathbf{F_n(t)}
\end{equation}

or, equivalently:

\begin{equation}
\mathbf{\ddot q_n(t)} + \alpha \cdot \mathbf{\dot q_n(t)} + \mathbf{\omega_n^2} \cdot \mathbf{q_n(t)} = \mathbf{a_n(t)}
\end{equation}

Comparing expressions, one obtains

\begin{equation}
\alpha = 2 \cdot \zeta_n \cdot \omega_n \Leftrightarrow \zeta_n = \frac{\alpha}{2 \cdot \omega_n}
\end{equation}

from where it can be seen that the mass proportional damping is a hyperbolic function of the vibration frequency $\omega_n$.

```python
alpha = 0.1
zn_a = alpha/(2.*wn)
plt.plot(wn, zn_a, label='mass proportional')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.grid(True)
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()
```

[Back to top](#top)

## Stiffness proportional damping

Stiffness proportional damping means that damping matrix is somehow a multiple of the stiffness matrix:

\begin{equation}
\mathbf{C} = \beta \cdot \mathbf{K}
\end{equation}

where $\beta$ is the constant of stiffness proportionality. In these circunstances, the dynamic equilibrium equation can be written as:

\begin{equation}
\mathbf{M} \times \mathbf{a(t)} + \beta \cdot \mathbf{K} \times \mathbf{v(t)} + \mathbf{K} \times \mathbf{d(t)} = \mathbf{F(t)}
\end{equation}

Proceeding the same as above, one obtains the $NDOF$ independent modal equilibrium equations:

\begin{equation}
\mathbf{M_n} \times \mathbf{\ddot q_n(t)} + \beta \cdot \mathbf{K_n} \times \mathbf{\dot q_n(t)} + \mathbf{K_n} \times \mathbf{q_n(t)} = \mathbf{F_n(t)}
\end{equation}

or, equivalently:

\begin{equation}
\mathbf{\ddot q_n(t)} + \beta \cdot \mathbf{\omega_n^2} \cdot \mathbf{\dot q_n(t)} + \mathbf{\omega_n^2} \cdot \mathbf{q_n(t)} = \mathbf{a_n(t)}
\end{equation}

Comparing expressions, one obtains

\begin{equation}
\beta \cdot \omega_n^2 = 2 \cdot \zeta \cdot \omega_n \Leftrightarrow \zeta_n = \frac{\omega_n \cdot \beta}{2}
\end{equation}

from where it can be seen that the stiffness proportional damping is a linear function of the vibration frequency $\omega_n$.

```python
beta = 0.005
zn_b = (beta*wn)/2.
plt.plot(wn, zn_b, label='stiffness proportional')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.grid(True)
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()
```

[Back to top](#top)

## Rayleigh damping

When Rayleigh damping is considered it means that the damping coefficient is a combination of the two previous ones, that is, it is a multiple of mass and stifnness:

\begin{equation}
\mathbf{C} = \alpha \cdot \mathbf{M} + \beta \cdot \mathbf{K}
\end{equation}

where $\alpha$ and $\beta$ have the previous meanings. In these circunstances, the dynamic equilibrium equation can be written as:

\begin{equation}
\mathbf{M} \times \mathbf{a(t)} + \left[ \alpha \cdot \mathbf{M} + \beta \cdot \mathbf{K} \right] \times \mathbf{v(t)} + \mathbf{K} \times \mathbf{d(t)} = \mathbf{F(t)}
\end{equation}

Proceeding the same as above, one obtains the $NDOF$ independent modal equilibrium equations:

\begin{equation}
\mathbf{M_n} \times \mathbf{\ddot q_n(t)} + \left[ \alpha \cdot \mathbf{M_n} + \beta \cdot \mathbf{K_n} \right] \times \mathbf{\dot q_n(t)} + \mathbf{K_n} \times \mathbf{q_n(t)} = \mathbf{F_n(t)}
\end{equation}

or, equivalently:

\begin{equation}
\mathbf{\ddot q_n(t)} + \left[ \alpha + \beta \cdot \mathbf{\omega_n^2} \right] \cdot \mathbf{\dot q_n(t)} + \mathbf{\omega_n^2} \cdot \mathbf{q_n(t)} = \mathbf{a_n(t)}
\end{equation}

Comparing expressions, one obtains

\begin{equation}
\alpha + \beta \cdot \omega_n^2 = 2 \cdot \zeta \cdot \omega_n \Leftrightarrow \zeta_n = \frac{\alpha}{2 \cdot \omega_n} + \frac{\omega_n \cdot \beta}{2}
\end{equation}

from where it can be seen that the Rayleigh damping is the sum of the previous linear and hyperbolic functions of the vibration frequency $\omega_n$.

```python
plt.hold(True)
plt.plot(wn, zn_a+zn_b, label='Rayleigh damping')
plt.plot(wn, zn_a, label='mass proportional')
plt.plot(wn, zn_b, label='stiffness proportional')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.grid(True)
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()
```

When the Rayleigh damping is used in MDOF systems, the coefficients $\alpha$ and $\beta$ can be computed in order to give an appropriate damping coefficient value for a given frequency range, related to the vibration modes of interest for the dynamic analysis. This is achieved by setting a simple two equation system whose solution yields the values of $\alpha$ and $\beta$:

$$
\left[\begin{array}{cc}
\zeta_1 \\ \zeta_2
\end{array}\right]
=
\left[\begin{array}{cc}
\frac{1}{2 \cdot \omega_1} && \frac{\omega_1}{2} \\ \frac{1}{2 \cdot \omega_2} && \frac{\omega_2}{2}
\end{array}\right]
\times
\left[\begin{array}{cc}
\alpha \\ \beta
\end{array}\right]
$$

[Back to top](#top)

## Example

Let us consider a MDOF system where there are several vibration modes of interest, ranging from 1 to 4 Hz, and that we want compute the dynamic response considering a damping coefficient of 2% for the first mode and 5% for the last mode.

```python
f1, f2 = 1., 4.
z1, z2 = 0.02, 0.05
w1 = 2.*np.pi*f1
w2 = 2.*np.pi*f2
alpha, beta = np.linalg.solve([[1./(2.*w1), w1/2.], [1./(2.*w2), w2/2.]], [z1, z2])
print('Alpha={:.6f}\nBeta={:.6f}'.format(alpha, beta))
```

    Alpha=0.100531
    Beta=0.003820

We can check that the Rayleigh damping assumes the required values at the desired frequencies, although may vary considerably for other frequencies:

```python
zn_a = alpha/(2.*wn)
zn_b = (beta*wn)/2.
plt.hold(True)
plt.plot(wn, zn_a+zn_b, label='Rayleigh damping')
plt.plot(wn, zn_a, label='mass proportional')
plt.plot(wn, zn_b, label='stiffness proportional')
plt.plot(w1, z1, 'o')
plt.plot(w2, z2, 'o')
plt.axvline(w1, ls=':')
plt.axhline(z1, ls=':')
plt.axvline(w2, ls=':')
plt.axhline(z2, ls=':')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()
```

[Back to top](#top)

## References

Caughey, T. K. and O’Kelly, M. E. J. (1965). “Classical normal modes in damped linear dynamic systems.” Transactions of ASME, Journal of Applied Mechanics, 32, 583–588.

Clough, Ray W., and Penzien, Joseph, Dynamics of Structures, 2nd ed. (revised), Computers and Structures, 2003.

[Back to top](#top)

## Odds and ends

This notebook was created by Paulo Xavier Candeias.

[Back to top](#top)