Chapter 7

< Polynomial Regression >
===============================

### Polynomial Features
- 1차 방정식을 고차다항식으로 변경하는 기법
- sklearn.preprocessing.PolynomialFeatures 사용
- PolynomialFeatures(degree(**차수**))

\begin{align}
X0 + X1 + X2 + X1^2 + X1X2 + X2^2
\end{align}

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
```

```python
X = np.arange(6).reshape(3, 2)
```

```python
X
```

    array([[0, 1],
           [2, 3],
           [4, 5]])

```python
poly = PolynomialFeatures(2)
```

```python
poly.fit_transform(X)
```

    array([[  1.,   0.,   1.,   0.,   0.,   1.],
           [  1.,   2.,   3.,   4.,   6.,   9.],
           [  1.,   4.,   5.,  16.,  20.,  25.]])

```python
ploy = PolynomialFeatures(interaction_only=True)
```

```python
poly.fit_transform(X)
```

    array([[  1.,   0.,   1.,   0.,   0.,   1.],
           [  1.,   2.,   3.,   4.,   6.,   9.],
           [  1.,   4.,   5.,  16.,  20.,  25.]])

### Example
- f 함수
\begin{align}
x * sin(x^2) + 1 반환
\end{align}

- sample 함수

    + 임의의 값을 넣어 랜덤 값 생성

- linspace 함수

    + a 부터 b까지 사이의 size가 n개인 수 균등하게 생성

```python
def f(size):
    x = np.linspace(0, 5, size)
    y = x * np.sin(x ** 2) + 1
    return (x, y)

def sample(size):
    x = np.linspace(0, 5, size)
    y = x * np.sin(x ** 2) \
        + 1 + np.random.randn(x.size) * 0.5
    return (x, y)
```

```python
f_x, f_y = f(1000)
plt.plot(f_x, f_y)
X, y = sample(1000)
plt.scatter(X, y, s=3, c='black')
plt.show()
```

```python
X
```

    array([ 0.        ,  0.00500501,  0.01001001,  0.01501502,  0.02002002,
            0.02502503,  0.03003003,  0.03503504,  0.04004004,  0.04504505,
            0.05005005,  0.05505506,  0.06006006,  0.06506507,  0.07007007,
            0.07507508,  0.08008008,  0.08508509,  0.09009009,  0.0950951 ,
            0.1001001 ,  0.10510511,  0.11011011,  0.11511512,  0.12012012,
            0.12512513,  0.13013013,  0.13513514,  0.14014014,  0.14514515,
            0.15015015,  0.15515516,  0.16016016,  0.16516517,  0.17017017,
            0.17517518,  0.18018018,  0.18518519,  0.19019019,  0.1951952 ,
            0.2002002 ,  0.20520521,  0.21021021,  0.21521522,  0.22022022,
            0.22522523,  0.23023023,  0.23523524,  0.24024024,  0.24524525,
            0.25025025,  0.25525526,  0.26026026,  0.26526527,  0.27027027,
            0.27527528,  0.28028028,  0.28528529,  0.29029029,  0.2952953 ,
            0.3003003 ,  0.30530531,  0.31031031,  0.31531532,  0.32032032,
            0.32532533,  0.33033033,  0.33533534,  0.34034034,  0.34534535,
            0.35035035,  0.35535536,  0.36036036,  0.36536537,  0.37037037,
            0.37537538,  0.38038038,  0.38538539,  0.39039039,  0.3953954 ,
            0.4004004 ,  0.40540541,  0.41041041,  0.41541542,  0.42042042,
            0.42542543,  0.43043043,  0.43543544,  0.44044044,  0.44544545,
            0.45045045,  0.45545546,  0.46046046,  0.46546547,  0.47047047,
            0.47547548,  0.48048048,  0.48548549,  0.49049049,  0.4954955 ,
            0.5005005 ,  0.50550551,  0.51051051,  0.51551552,  0.52052052,
            0.52552553,  0.53053053,  0.53553554,  0.54054054,  0.54554555,
            0.55055055,  0.55555556,  0.56056056,  0.56556557,  0.57057057,
            0.57557558,  0.58058058,  0.58558559,  0.59059059,  0.5955956 ,
            0.6006006 ,  0.60560561,  0.61061061,  0.61561562,  0.62062062,
            0.62562563,  0.63063063,  0.63563564,  0.64064064,  0.64564565,
            0.65065065,  0.65565566,  0.66066066,  0.66566567,  0.67067067,
            0.67567568,  0.68068068,  0.68568569,  0.69069069,  0.6956957 ,
            0.7007007 ,  0.70570571,  0.71071071,  0.71571572,  0.72072072,
            0.72572573,  0.73073073,  0.73573574,  0.74074074,  0.74574575,
            0.75075075,  0.75575576,  0.76076076,  0.76576577,  0.77077077,
            0.77577578,  0.78078078,  0.78578579,  0.79079079,  0.7957958 ,
            0.8008008 ,  0.80580581,  0.81081081,  0.81581582,  0.82082082,
            0.82582583,  0.83083083,  0.83583584,  0.84084084,  0.84584585,
            0.85085085,  0.85585586,  0.86086086,  0.86586587,  0.87087087,
            0.87587588,  0.88088088,  0.88588589,  0.89089089,  0.8958959 ,
            0.9009009 ,  0.90590591,  0.91091091,  0.91591592,  0.92092092,
            0.92592593,  0.93093093,  0.93593594,  0.94094094,  0.94594595,
            0.95095095,  0.95595596,  0.96096096,  0.96596597,  0.97097097,
            0.97597598,  0.98098098,  0.98598599,  0.99099099,  0.995996  ,
            1.001001  ,  1.00600601,  1.01101101,  1.01601602,  1.02102102,
            1.02602603,  1.03103103,  1.03603604,  1.04104104,  1.04604605,
            1.05105105,  1.05605606,  1.06106106,  1.06606607,  1.07107107,
            1.07607608,  1.08108108,  1.08608609,  1.09109109,  1.0960961 ,
            1.1011011 ,  1.10610611,  1.11111111,  1.11611612,  1.12112112,
            1.12612613,  1.13113113,  1.13613614,  1.14114114,  1.14614615,
            1.15115115,  1.15615616,  1.16116116,  1.16616617,  1.17117117,
            1.17617618,  1.18118118,  1.18618619,  1.19119119,  1.1961962 ,
            1.2012012 ,  1.20620621,  1.21121121,  1.21621622,  1.22122122,
            1.22622623,  1.23123123,  1.23623624,  1.24124124,  1.24624625,
            1.25125125,  1.25625626,  1.26126126,  1.26626627,  1.27127127,
            1.27627628,  1.28128128,  1.28628629,  1.29129129,  1.2962963 ,
            1.3013013 ,  1.30630631,  1.31131131,  1.31631632,  1.32132132,
            1.32632633,  1.33133133,  1.33633634,  1.34134134,  1.34634635,
            1.35135135,  1.35635636,  1.36136136,  1.36636637,  1.37137137,
            1.37637638,  1.38138138,  1.38638639,  1.39139139,  1.3963964 ,
            1.4014014 ,  1.40640641,  1.41141141,  1.41641642,  1.42142142,
            1.42642643,  1.43143143,  1.43643644,  1.44144144,  1.44644645,
            1.45145145,  1.45645646,  1.46146146,  1.46646647,  1.47147147,
            1.47647648,  1.48148148,  1.48648649,  1.49149149,  1.4964965 ,
            1.5015015 ,  1.50650651,  1.51151151,  1.51651652,  1.52152152,
            1.52652653,  1.53153153,  1.53653654,  1.54154154,  1.54654655,
            1.55155155,  1.55655656,  1.56156156,  1.56656657,  1.57157157,
            1.57657658,  1.58158158,  1.58658659,  1.59159159,  1.5965966 ,
            1.6016016 ,  1.60660661,  1.61161161,  1.61661662,  1.62162162,
            1.62662663,  1.63163163,  1.63663664,  1.64164164,  1.64664665,
            1.65165165,  1.65665666,  1.66166166,  1.66666667,  1.67167167,
            1.67667668,  1.68168168,  1.68668669,  1.69169169,  1.6966967 ,
            1.7017017 ,  1.70670671,  1.71171171,  1.71671672,  1.72172172,
            1.72672673,  1.73173173,  1.73673674,  1.74174174,  1.74674675,
            1.75175175,  1.75675676,  1.76176176,  1.76676677,  1.77177177,
            1.77677678,  1.78178178,  1.78678679,  1.79179179,  1.7967968 ,
            1.8018018 ,  1.80680681,  1.81181181,  1.81681682,  1.82182182,
            1.82682683,  1.83183183,  1.83683684,  1.84184184,  1.84684685,
            1.85185185,  1.85685686,  1.86186186,  1.86686687,  1.87187187,
            1.87687688,  1.88188188,  1.88688689,  1.89189189,  1.8968969 ,
            1.9019019 ,  1.90690691,  1.91191191,  1.91691692,  1.92192192,
            1.92692693,  1.93193193,  1.93693694,  1.94194194,  1.94694695,
            1.95195195,  1.95695696,  1.96196196,  1.96696697,  1.97197197,
            1.97697698,  1.98198198,  1.98698699,  1.99199199,  1.996997  ,
            2.002002  ,  2.00700701,  2.01201201,  2.01701702,  2.02202202,
            2.02702703,  2.03203203,  2.03703704,  2.04204204,  2.04704705,
            2.05205205,  2.05705706,  2.06206206,  2.06706707,  2.07207207,
            2.07707708,  2.08208208,  2.08708709,  2.09209209,  2.0970971 ,
            2.1021021 ,  2.10710711,  2.11211211,  2.11711712,  2.12212212,
            2.12712713,  2.13213213,  2.13713714,  2.14214214,  2.14714715,
            2.15215215,  2.15715716,  2.16216216,  2.16716717,  2.17217217,
            2.17717718,  2.18218218,  2.18718719,  2.19219219,  2.1971972 ,
            2.2022022 ,  2.20720721,  2.21221221,  2.21721722,  2.22222222,
            2.22722723,  2.23223223,  2.23723724,  2.24224224,  2.24724725,
            2.25225225,  2.25725726,  2.26226226,  2.26726727,  2.27227227,
            2.27727728,  2.28228228,  2.28728729,  2.29229229,  2.2972973 ,
            2.3023023 ,  2.30730731,  2.31231231,  2.31731732,  2.32232232,
            2.32732733,  2.33233233,  2.33733734,  2.34234234,  2.34734735,
            2.35235235,  2.35735736,  2.36236236,  2.36736737,  2.37237237,
            2.37737738,  2.38238238,  2.38738739,  2.39239239,  2.3973974 ,
            2.4024024 ,  2.40740741,  2.41241241,  2.41741742,  2.42242242,
            2.42742743,  2.43243243,  2.43743744,  2.44244244,  2.44744745,
            2.45245245,  2.45745746,  2.46246246,  2.46746747,  2.47247247,
            2.47747748,  2.48248248,  2.48748749,  2.49249249,  2.4974975 ,
            2.5025025 ,  2.50750751,  2.51251251,  2.51751752,  2.52252252,
            2.52752753,  2.53253253,  2.53753754,  2.54254254,  2.54754755,
            2.55255255,  2.55755756,  2.56256256,  2.56756757,  2.57257257,
            2.57757758,  2.58258258,  2.58758759,  2.59259259,  2.5975976 ,
            2.6026026 ,  2.60760761,  2.61261261,  2.61761762,  2.62262262,
            2.62762763,  2.63263263,  2.63763764,  2.64264264,  2.64764765,
            2.65265265,  2.65765766,  2.66266266,  2.66766767,  2.67267267,
            2.67767768,  2.68268268,  2.68768769,  2.69269269,  2.6976977 ,
            2.7027027 ,  2.70770771,  2.71271271,  2.71771772,  2.72272272,
            2.72772773,  2.73273273,  2.73773774,  2.74274274,  2.74774775,
            2.75275275,  2.75775776,  2.76276276,  2.76776777,  2.77277277,
            2.77777778,  2.78278278,  2.78778779,  2.79279279,  2.7977978 ,
            2.8028028 ,  2.80780781,  2.81281281,  2.81781782,  2.82282282,
            2.82782783,  2.83283283,  2.83783784,  2.84284284,  2.84784785,
            2.85285285,  2.85785786,  2.86286286,  2.86786787,  2.87287287,
            2.87787788,  2.88288288,  2.88788789,  2.89289289,  2.8978979 ,
            2.9029029 ,  2.90790791,  2.91291291,  2.91791792,  2.92292292,
            2.92792793,  2.93293293,  2.93793794,  2.94294294,  2.94794795,
            2.95295295,  2.95795796,  2.96296296,  2.96796797,  2.97297297,
            2.97797798,  2.98298298,  2.98798799,  2.99299299,  2.997998  ,
            3.003003  ,  3.00800801,  3.01301301,  3.01801802,  3.02302302,
            3.02802803,  3.03303303,  3.03803804,  3.04304304,  3.04804805,
            3.05305305,  3.05805806,  3.06306306,  3.06806807,  3.07307307,
            3.07807808,  3.08308308,  3.08808809,  3.09309309,  3.0980981 ,
            3.1031031 ,  3.10810811,  3.11311311,  3.11811812,  3.12312312,
            3.12812813,  3.13313313,  3.13813814,  3.14314314,  3.14814815,
            3.15315315,  3.15815816,  3.16316316,  3.16816817,  3.17317317,
            3.17817818,  3.18318318,  3.18818819,  3.19319319,  3.1981982 ,
            3.2032032 ,  3.20820821,  3.21321321,  3.21821822,  3.22322322,
            3.22822823,  3.23323323,  3.23823824,  3.24324324,  3.24824825,
            3.25325325,  3.25825826,  3.26326326,  3.26826827,  3.27327327,
            3.27827828,  3.28328328,  3.28828829,  3.29329329,  3.2982983 ,
            3.3033033 ,  3.30830831,  3.31331331,  3.31831832,  3.32332332,
            3.32832833,  3.33333333,  3.33833834,  3.34334334,  3.34834835,
            3.35335335,  3.35835836,  3.36336336,  3.36836837,  3.37337337,
            3.37837838,  3.38338338,  3.38838839,  3.39339339,  3.3983984 ,
            3.4034034 ,  3.40840841,  3.41341341,  3.41841842,  3.42342342,
            3.42842843,  3.43343343,  3.43843844,  3.44344344,  3.44844845,
            3.45345345,  3.45845846,  3.46346346,  3.46846847,  3.47347347,
            3.47847848,  3.48348348,  3.48848849,  3.49349349,  3.4984985 ,
            3.5035035 ,  3.50850851,  3.51351351,  3.51851852,  3.52352352,
            3.52852853,  3.53353353,  3.53853854,  3.54354354,  3.54854855,
            3.55355355,  3.55855856,  3.56356356,  3.56856857,  3.57357357,
            3.57857858,  3.58358358,  3.58858859,  3.59359359,  3.5985986 ,
            3.6036036 ,  3.60860861,  3.61361361,  3.61861862,  3.62362362,
            3.62862863,  3.63363363,  3.63863864,  3.64364364,  3.64864865,
            3.65365365,  3.65865866,  3.66366366,  3.66866867,  3.67367367,
            3.67867868,  3.68368368,  3.68868869,  3.69369369,  3.6986987 ,
            3.7037037 ,  3.70870871,  3.71371371,  3.71871872,  3.72372372,
            3.72872873,  3.73373373,  3.73873874,  3.74374374,  3.74874875,
            3.75375375,  3.75875876,  3.76376376,  3.76876877,  3.77377377,
            3.77877878,  3.78378378,  3.78878879,  3.79379379,  3.7987988 ,
            3.8038038 ,  3.80880881,  3.81381381,  3.81881882,  3.82382382,
            3.82882883,  3.83383383,  3.83883884,  3.84384384,  3.84884885,
            3.85385385,  3.85885886,  3.86386386,  3.86886887,  3.87387387,
            3.87887888,  3.88388388,  3.88888889,  3.89389389,  3.8988989 ,
            3.9039039 ,  3.90890891,  3.91391391,  3.91891892,  3.92392392,
            3.92892893,  3.93393393,  3.93893894,  3.94394394,  3.94894895,
            3.95395395,  3.95895896,  3.96396396,  3.96896897,  3.97397397,
            3.97897898,  3.98398398,  3.98898899,  3.99399399,  3.998999  ,
            4.004004  ,  4.00900901,  4.01401401,  4.01901902,  4.02402402,
            4.02902903,  4.03403403,  4.03903904,  4.04404404,  4.04904905,
            4.05405405,  4.05905906,  4.06406406,  4.06906907,  4.07407407,
            4.07907908,  4.08408408,  4.08908909,  4.09409409,  4.0990991 ,
            4.1041041 ,  4.10910911,  4.11411411,  4.11911912,  4.12412412,
            4.12912913,  4.13413413,  4.13913914,  4.14414414,  4.14914915,
            4.15415415,  4.15915916,  4.16416416,  4.16916917,  4.17417417,
            4.17917918,  4.18418418,  4.18918919,  4.19419419,  4.1991992 ,
            4.2042042 ,  4.20920921,  4.21421421,  4.21921922,  4.22422422,
            4.22922923,  4.23423423,  4.23923924,  4.24424424,  4.24924925,
            4.25425425,  4.25925926,  4.26426426,  4.26926927,  4.27427427,
            4.27927928,  4.28428428,  4.28928929,  4.29429429,  4.2992993 ,
            4.3043043 ,  4.30930931,  4.31431431,  4.31931932,  4.32432432,
            4.32932933,  4.33433433,  4.33933934,  4.34434434,  4.34934935,
            4.35435435,  4.35935936,  4.36436436,  4.36936937,  4.37437437,
            4.37937938,  4.38438438,  4.38938939,  4.39439439,  4.3993994 ,
            4.4044044 ,  4.40940941,  4.41441441,  4.41941942,  4.42442442,
            4.42942943,  4.43443443,  4.43943944,  4.44444444,  4.44944945,
            4.45445445,  4.45945946,  4.46446446,  4.46946947,  4.47447447,
            4.47947948,  4.48448448,  4.48948949,  4.49449449,  4.4994995 ,
            4.5045045 ,  4.50950951,  4.51451451,  4.51951952,  4.52452452,
            4.52952953,  4.53453453,  4.53953954,  4.54454454,  4.54954955,
            4.55455455,  4.55955956,  4.56456456,  4.56956957,  4.57457457,
            4.57957958,  4.58458458,  4.58958959,  4.59459459,  4.5995996 ,
            4.6046046 ,  4.60960961,  4.61461461,  4.61961962,  4.62462462,
            4.62962963,  4.63463463,  4.63963964,  4.64464464,  4.64964965,
            4.65465465,  4.65965966,  4.66466466,  4.66966967,  4.67467467,
            4.67967968,  4.68468468,  4.68968969,  4.69469469,  4.6996997 ,
            4.7047047 ,  4.70970971,  4.71471471,  4.71971972,  4.72472472,
            4.72972973,  4.73473473,  4.73973974,  4.74474474,  4.74974975,
            4.75475475,  4.75975976,  4.76476476,  4.76976977,  4.77477477,
            4.77977978,  4.78478478,  4.78978979,  4.79479479,  4.7997998 ,
            4.8048048 ,  4.80980981,  4.81481481,  4.81981982,  4.82482482,
            4.82982983,  4.83483483,  4.83983984,  4.84484484,  4.84984985,
            4.85485485,  4.85985986,  4.86486486,  4.86986987,  4.87487487,
            4.87987988,  4.88488488,  4.88988989,  4.89489489,  4.8998999 ,
            4.9049049 ,  4.90990991,  4.91491491,  4.91991992,  4.92492492,
            4.92992993,  4.93493493,  4.93993994,  4.94494494,  4.94994995,
            4.95495495,  4.95995996,  4.96496496,  4.96996997,  4.97497497,
            4.97997998,  4.98498498,  4.98998999,  4.99499499,  5.        ])

```python
X.shape, y.shape
```

    ((1000,), (1000,))

```python
X = X.reshape(-1, 1)
y = y.reshape(-1, 1)
X.shape, y.shape
```

    ((1000, 1), (1000, 1))

```python
from sklearn.linear_model import LinearRegression

lr = LinearRegression()
lr.fit(X, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

```python
f_x, f_y = f(1000)
plt.plot(f_x, f_y)
plt.scatter(X.flatten(), y.flatten(), s=3, c='black')
plt.plot(X.flatten(), lr.predict(X).flatten())
plt.show()
```

```python
poly_features = PolynomialFeatures(degree=2)
X_poly = poly_features.fit_transform(X)
X_poly[:10]
```

    array([[  1.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [  1.00000000e+00,   5.00500501e-03,   2.50500751e-05],
           [  1.00000000e+00,   1.00100100e-02,   1.00200300e-04],
           [  1.00000000e+00,   1.50150150e-02,   2.25450676e-04],
           [  1.00000000e+00,   2.00200200e-02,   4.00801202e-04],
           [  1.00000000e+00,   2.50250250e-02,   6.26251878e-04],
           [  1.00000000e+00,   3.00300300e-02,   9.01802704e-04],
           [  1.00000000e+00,   3.50350350e-02,   1.22745368e-03],
           [  1.00000000e+00,   4.00400400e-02,   1.60320481e-03],
           [  1.00000000e+00,   4.50450450e-02,   2.02905608e-03]])

```python
lr = LinearRegression(fit_intercept=False)
lr.fit(X_poly, y)
```

    LinearRegression(copy_X=True, fit_intercept=False, n_jobs=1, normalize=False)

```python
f_x, f_y = f(1000)
plt.plot(f_x, f_y)
plt.scatter(X.flatten(), y.flatten(), s=3, c='black')
plt.plot(X.flatten(), lr.predict(X_poly).flatten())
plt.show()
```

```python
poly_features = PolynomialFeatures(degree=16)
X_poly = poly_features.fit_transform(X)
X_poly[:3]
```

    array([[  1.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              0.00000000e+00,   0.00000000e+00],
           [  1.00000000e+00,   5.00500501e-03,   2.50500751e-05,
              1.25375751e-07,   6.27506263e-10,   3.14067198e-12,
              1.57190790e-14,   7.86740691e-17,   3.93764110e-19,
              1.97079134e-21,   9.86382051e-24,   4.93684710e-26,
              2.47089445e-28,   1.23668391e-30,   6.18960915e-33,
              3.09790248e-35,   1.55050174e-37],
           [  1.00000000e+00,   1.00100100e-02,   1.00200300e-04,
              1.00300601e-06,   1.00401002e-08,   1.00501504e-10,
              1.00602106e-12,   1.00702808e-14,   1.00803612e-16,
              1.00904517e-18,   1.01005522e-20,   1.01106629e-22,
              1.01207837e-24,   1.01309146e-26,   1.01410556e-28,
              1.01512068e-30,   1.01613682e-32]])

```python
lr = LinearRegression(fit_intercept=False)
lr.fit(X_poly, y)
f_x, f_y = f(1000)
plt.plot(f_x, f_y)
plt.scatter(X.flatten(), y.flatten(), s=3, c='black')
plt.plot(X.flatten(), lr.predict(X_poly).flatten())
plt.show()
```

### 하고있는 작업
- parameter search
    + degree를 바꾸며 진행
    + 실험적으로 접근

#### rmse(Root Mean Square Error)
- 추정 값 또는 모델이 예측한 값과 실제 환경에서

관찰되는 값의 차이를 다룰 때 흔히 사용하는 측도

```python
def rmse(predctions, targets):
    return np.sqrt(((predctions - targets) ** 2).mean())
```

- Lasso : L1

- Ridge : L2

- 가장 성능이 좋은 모델 : rmse의 값이 가장 작은 것

```python
poly_range = list(range(10, 50))
rmse_lr_list = []
rmse_lasso_list = []
rmse_ridge_list = []

from sklearn.linear_model import Lasso
from sklearn.linear_model import Ridge

for poly_value in poly_range:
    poly_features = PolynomialFeatures(degree=poly_value)
    X_poly = poly_features.fit_transform(X)

    lr = LinearRegression(fit_intercept=False)
    lr.fit(X_poly, y)

    rmse_lr_list.append(rmse(lr.predict(X_poly), y))

    lasso = Lasso(fit_intercept=False)
    lasso.fit(X_poly, y)
    rmse_lasso_list.append(rmse(lasso.predict(X_poly), y))

    ridge = Ridge(fit_intercept=False)
    ridge.fit(X_poly, y)
    rmse_ridge_list.append(rmse(ridge.predict(X_poly), y))
```

    C:\Users\Beginning\AppData\Roaming\Python\Python36\site-packages\sklearn\linear_model\coordinate_descent.py:479: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
      ConvergenceWarning)
    C:\Users\Beginning\AppData\Roaming\Python\Python36\site-packages\sklearn\linear_model\ridge.py:112: LinAlgWarning: scipy.linalg.solve
    Ill-conditioned matrix detected. Result is not guaranteed to be accurate.
    Reciprocal condition number4.855329e-18
      overwrite_a=True).T
    C:\Users\Beginning\AppData\Roaming\Python\Python36\site-packages\sklearn\linear_model\ridge.py:112: LinAlgWarning: scipy.linalg.solve
    Ill-conditioned matrix detected. Result is not guaranteed to be accurate.
    Reciprocal condition number2.072980e-19
      overwrite_a=True).T
    C:\Users\Beginning\AppData\Roaming\Python\Python36\site-packages\sklearn\linear_model\ridge.py:112: LinAlgWarning: scipy.linalg.solve
    Ill-conditioned matrix detected. Result is not guaranteed to be accurate.
    Reciprocal condition number9.431182e-21
      overwrite_a=True).T
    C:\Users\Beginning\AppData\Roaming\Python\Python36\site-packages\sklearn\linear_model\ridge.py:112: LinAlgWarning: scipy.linalg.solve
    Ill-conditioned matrix detected. Result is not guaranteed to be accurate.
    Reciprocal condition number3.824242e-22
      overwrite_a=True).T
    C:\Users\Beginning\AppData\Roaming\Python\Python36\site-packages\sklearn\linear_model\ridge.py:112: LinAlgWarning: scipy.linalg.solve
    Ill-conditioned matrix detected. Result is not guaranteed to be accurate.
    Reciprocal condition number1.676899e-23
      overwrite_a=True).T

```python
from pandas import DataFrame

data = {
    "poly_range":poly_range,
    "lr_rmse":rmse_lr_list,
    "lasso_rmse":rmse_lasso_list,
    "ridge_rmse":rmse_ridge_list,
}

df = DataFrame(data).set_index("poly_range")
df
```

      lasso_rmse
      lr_rmse
      ridge_rmse

      poly_range

      10
      2.299570
      1.879458
      1.905629

      11
      2.301829
      1.871697
      1.887834

      12
      2.302391
      1.618999
      1.887285

      13
      2.301838
      1.351464
      1.863351

      14
      2.300668
      1.338428
      1.717291

      15
      2.299291
      0.931010
      1.525134

      16
      2.297974
      0.721666
      1.484689

      17
      2.296860
      0.684943
      1.292028

      18
      2.296006
      0.710934
      0.887707

      19
      2.295418
      0.738678
      0.762333

      20
      2.295072
      0.664958
      0.755168

      21
      2.294948
      0.634458
      0.620734

      22
      2.295040
      0.542938
      0.525130

      23
      2.295328
      0.607375
      0.532148

      24
      2.295784
      0.987866
      0.794248

      25
      2.296395
      1.169196
      1.252629

      26
      2.297141
      1.245403
      1.912797

      27
      2.298009
      1.291127
      1.896643

      28
      2.298988
      1.341025
      1.549787

      29
      2.300073
      1.358899
      1.552103

      30
      2.301266
      1.337711
      1.568422

      31
      2.302576
      1.294514
      1.536809

      32
      2.304012
      1.611622
      1.769376

      33
      2.305588
      1.655436
      2.084459

      34
      2.307319
      1.687042
      1.896240

      35
      2.309216
      1.693862
      1.807973

      36
      2.311286
      2.100174
      2.116398

      37
      2.313529
      1.663860
      1.878996

      38
      2.315937
      1.643938
      1.998969

      39
      2.318500
      1.883064
      2.081083

      40
      2.321200
      1.761776
      1.799921

      41
      2.324016
      1.636209
      1.874346

      42
      2.326916
      1.694193
      2.197637

      43
      2.329862
      1.715541
      1.776886

      44
      2.332818
      1.743614
      1.875378

      45
      2.335748
      1.738181
      1.909581

      46
      2.338622
      1.970173
      1.752807

      47
      2.341408
      1.693169
      1.908255

      48
      2.344081
      2.008898
      1.748193

      49
      2.346612
      1.840254
      1.768137

### 어떻게 최적화 하는가?
- **RMSE**의 최소값을 찾자
- **Ridge**, **Lasso**, **LR** 모두 써보자
- Degree를 10 ~ 50까지 써보자
- 결과를 **한눈에 정리**해보자

```python
plt.plot(poly_range, df["ridge_rmse"], label="ridge")
plt.plot(poly_range, df["lr_rmse"], label="lr")
plt.plot(poly_range, df["lasso_rmse"], label="lasso")
plt.legend()
plt.show()
```

### L1(Lasso)
- Unstable Solution

- Always on Solution

- Sparse Solution

- Feature Selection

### L2(Ridge)
- Stable Solution

- Only one Solution

- Non-sparse Solution

### L1의 값이 비슷한 이유
- L1(Lasso)의 경우 의미없는 데이터들을 0으로 바꾸기 때문

```python
df.min()
```

    lasso_rmse    2.294948
    lr_rmse       0.542938
    ridge_rmse    0.525130
    dtype: float64

확인 결과 **Ridge**가 가장 작다.

```python
df["ridge_rmse"].sort_values().head()
```

    poly_range
    22    0.525130
    23    0.532148
    21    0.620734
    20    0.755168
    19    0.762333
    Name: ridge_rmse, dtype: float64

가장 작은 Ridge의 **최솟값** 정렬하여 확인

`degree`가 `22`일때 가장 값이 적다.

```python
poly_features = PolynomialFeatures(degree=22)
X_poly = poly_features.fit_transform(X)
ridge = Ridge(fit_intercept=False)
ridge.fit(X_poly, y)
```

    Ridge(alpha=1.0, copy_X=True, fit_intercept=False, max_iter=None,
       normalize=False, random_state=None, solver='auto', tol=0.001)

```python
f_x, f_y = f(1000)
plt.plot(f_x, f_y)
plt.scatter(X.flatten(), y.flatten(), s=3, c="black")
plt.plot(X.flatten(), ridge.predict(X_poly).flatten())
plt.show()
```

### 언제 사용하는가?
- 한개 변수가 `Y`값과 **비선형적인 관계**가 있다고 의심될 때
- **주기적인 패턴**을 보이는 Series 데이터일 때
- 모델 자체가 복잡해지면 해결가능한 부분이 많다.
    + SVM, Tree-based Models

### Challenge

```python
df = pd.read_csv('yield.csv', sep='\t')
df.head()
```

      i
      Temp
      Yield

      0
      1
      50
      3.3

      1
      2
      50
      2.8

      2
      3
      50
      2.9

      3
      4
      70
      2.3

      4
      5
      70
      2.6

```python
plt.scatter(df['Temp'], df['Yield'])
plt.show()
```


```python
%matplotlib inline
```

# Numerical instabilities of the Nonlinear Schrödinger Equation

This example demonstrates numerical instabilities that build up during
simulations of the standard nonlinear Schrödinger equation when the
$z$-increment used for propagtion is too small.

Here, the simple split-step Fourier method is used and the instabilities
develop from round-off errors during the propagation of a fundamental soliton.

The numerical instabilities of the nonlinear Schrödinger equation developing on
top of  plain wave solutions where first studied in Ref. [WH1986]_.
An in-depth study of instabilities of the split-step Fourier method for the
simulation of the nonlinear Schrödinger equation, developing on a background
given by a soliton, is provided by Ref. [L2012]_.

.. codeauthor:: Oliver Melchert <melchert@iqo.uni-hannover.de>

We first import the functionality needed to perform the sequence of numerical
experiments:

```python
import sys
import numpy as np
import numpy.fft as nfft
from fmas.models import ModelBaseClass
from fmas.config import FTFREQ, FT, IFT, C0
from fmas.solver import SiSSM, SySSM, IFM_RK4IP, LEM_SySSM, CQE
from fmas.grid import Grid
```

Next, we implement a model for the nonlinear Schrödinger equation.  In
particular, we here consider the standard nonlinear Schrödinger equation,
given by

\begin{align}\partial_z u = -i \frac{\beta_2}{2}\partial_t^2 u + i\gamma |u|^2 u,\end{align}

wherein $u = u(z, t)$ represents the slowly varying pulse envelope,
$\beta_2=-1$ is the second order dispersion parameter, and
$\gamma=1$ is the nonlinear parameter:

```python
class NSE(ModelBaseClass):

    def __init__(self, w, b2 = -1.0, gamma = 1.):
        super().__init__(w, 0.5*b2*w*w)
        self.gamma = gamma

    @property
    def Lw(self):
        return 1j*self.beta_w

    def Nw(self, uw):
        ut = IFT(uw)
        return 1j*self.gamma*FT(np.abs(ut)**2*ut)
```

Next, we initialize the computational domain and use a simple split-step
Fourier method to propagate a single fundamental soliton for ten soliton
periods.

```python
# -- INITIALIZATION STAGE
# ... COMPUTATIONAL DOMAIN
grid = Grid( t_max = 30., t_num = 2**10)
t, w = grid.t, grid.w
# ... NSE MODEL
model = NSE(w, b2=-1., gamma=1.)
# ... INITIAL CONDITION
u_0t = 1./np.cosh(t)
```

In a first numerical experiment, the stepsize is intentionally kept very
large in order to allow the numerical istabilities to build up.

```python
solver = SySSM(model.Lw, model.Nw)
solver.set_initial_condition(w, FT(u_0t))
solver.propagate(z_range = 10*np.pi, n_steps = 511, n_skip = 1)
z, utz = solver.z_, solver.utz
```

In this case, instabilities are expected to build up since the
$z$-increment $\Delta z$, used by the propagation algorithm,
exceeds the threshold increment $\Delta
z_{\mathrm{T}}=2\pi/\mathrm{max}(\omega)$ (both increments are displayed
below).

```python
# -- MAXIMUM FREQUENCY SUPPORTED ON COMPUTATIONAL GRID
w_max = np.pi/(t[1]-t[0])
# -- THRESHOLD INCREMENT
dz_T = np.pi*2/w_max**2

print("Increment dz =", z[1]-z[0])
print("Threshold increment dz_T =", dz_T)
```

In a second numerical experiment, the stepsize is set small enough to shift
the resonance outside the computational domain.

```python
solver = SySSM(model.Lw, model.Nw)
solver.set_initial_condition(w, FT(u_0t))
solver.propagate(z_range = 10*np.pi, n_steps = 15000, n_skip = 1)
z2, utz2 = solver.z_, solver.utz

print("Increment dz =", z2[1]-z2[0])
print("Threshold increment dz_T =", dz_T)
```

Next, we prepare a figure that shows the results of the above to experiments.
The left subfigure shows the results of the first simulation run in which the
numerical instabilities and their predicted locations are shown.
The right subfigure shows the results of second simulation run in which the
$z$-increment was small enough to shift the instabilities outside the
computational domain.

```python
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.colors as col

f, (ax,ax2) = plt.subplots(1, 2, figsize=(8,4))

# -- EXAMPLE WITH INSTABILITIES

dz = z[1]-z[0]
w_I = lambda n: np.sqrt(n*2*np.pi*2./dz)

shift=nfft.fftshift
Iw_ini = np.abs(FT(utz[0]))**2
Iw_fin = np.abs(FT(utz[-1]))**2
ax.plot(shift(w), shift(Iw_ini)/np.max(Iw_ini), color='gray', dashes=[2,2], lw=1., label='$z=0$')
ax.plot(shift(w), shift(Iw_fin)/np.max(Iw_ini), color='k', lw=1., label='$z=10\pi$')

for n in range(1,20,1):
    ax.axvline(w_I(n), lw=0.75)
    ax.axvline(-w_I(n), lw=0.75)

dw_lim = (-35,35)
dw_ticks = (-30, -15, 0, 15, 30)
ax.tick_params(axis='x', length=2., pad=2, top=False)
ax.set_xlim(dw_lim)
ax.set_xticks(dw_ticks)
ax.set_xlabel(r"Detuning $\omega$")

y_lim = (1e-35,10)
y_ticks = (1e-30,1e-20,1e-10,1)
ax.tick_params(axis='y', length=2., pad=2, top=False)
ax.set_yscale('log')
ax.set_ylim(y_lim)
ax.set_yticks(y_ticks)
ax.set_ylabel(r"Spectral intensity $I_\omega(z)/\mathrm{max}(I_\omega(z=0)}$")
ax.set_title(r"Numerical instabilities build up")

ax.legend()

# -- EXAMPLE WITHOUT INSTABILITIES

Iw_ini = np.abs(FT(utz2[0]))**2
Iw_fin = np.abs(FT(utz2[-1]))**2
ax2.plot(shift(w), shift(Iw_ini)/np.max(Iw_ini), color='gray', dashes=[2,2], lw=1., label='$z=0$')
ax2.plot(shift(w), shift(Iw_fin)/np.max(Iw_ini), color='k', lw=1., label='$z=10\pi$')

ax2.tick_params(axis='x', length=2., pad=2, top=False)
ax2.set_xlim(dw_lim)
ax2.set_xticks(dw_ticks)
ax2.set_xlabel(r"Detuning $\omega$")

ax2.tick_params(axis='y', length=2., pad=2, top=False)
ax2.set_yscale('log')
ax2.set_ylim(y_lim)
ax2.set_yticks(y_ticks)
ax2.set_title(r"No numerical instabilities")

ax2.legend()

plt.show()
```

**References:**

.. [WH1986] J.A.C. Weideman, B.M. Herbst, Split-step methods for the solution
            of the nonlinear Schrödinger equation, SIAM J. Numer. Anal., 23
            (1986) 485, http://www.jstor.org/stable/2157521.

.. [L2012] T.I. Lakoba, Instability Analysis of the Split-Step Fourier Method
           on the Background of a Soliton of the Nonlinear Schrödinger
           Equation, Numerical Methods for Partial Differential Equations 28
           (2012) 641, https://doi.org/10.1002/num.20649


```{warning}
This book is a work in progress and should be considered currently to be in a
**pre**draft state. Work is actively taking place in preparation for October
2020.

If you happen to find this and notice any typos and/or have any suggestions
please open an issue on the github repo: <https://github.com/drvinceknight/pfm>
```

# Python for Mathematics

## Introduction

This book aims to introduce readers to programming for mathematics.

It is assumed that readers are used to solving high school mathematics problems
of the form:

---

```{admonition} Problem
Given the function $f:\mathbb{R}\to\mathbb{R}$ defined by
$f(x) = x ^ 2 - 3 x + 1$ obtain the global minima of the function.
```

```{admonition} Solution
:class: tip

To solve this we need to apply our **mathematical knowledge** which tells us to:

1. Differentiate $f(x)$ to get $\frac{df}{dx}$;
2. Equate $\frac{df}{dx}=0$;
3. Use the second derivative test on the solution to the previous equation.

For each of those 3 steps we will usually make use of our **mathematical
techniques**:

1. Differentiate $f(x)$:

   $$\frac{df}{dx} = 2 x - 3$$

2. Equate $\frac{df}{dx}=0$:

   $$2x-3 =0 \Rightarrow x = 3/2$$

3. Use the second derivative test on the solution:

   $$\frac{d^2f}{dx^2} = 2 > 0\text{ for all values of }x$$

   Thus $x=2/3$ is the global minima of the function.
```

```{attention}
As we progress as mathematicians **mathematical knowledge** is more prominent
than **mathematical technique**: often knowing what to do is the real problem as
opposed to having the technical ability to do it.
```

This is what this book will cover: **programming** allows us to instruct a
computer to carry out mathematical techniques.

We will for example learn how to solve the above problem by instructing a
computer which **mathematical technique** to carry out.

**This book will teach us how to give the correct instructions to a
computer.**

The following is an example, do not worry too much about the specific code used
for now:

1. Differentiate $f(x)$ to get $\frac{df}{dx}$;

```python
import sympy as sym

x = sym.Symbol("x")
sym.diff(x ** 2 - 3 * x + 1, x)
```

$\displaystyle 2 x - 3$

2. Equate $\frac{df}{dx}=0$:

```python
sym.solveset(2 * x - 3, x)
```

$\displaystyle \left\{\frac{3}{2}\right\}$

3. Use the second derivative test on the solution:

```python
sym.diff(x ** 2 - 3 * x + 1, x, 2)
```

$\displaystyle 2$

{ref}`Knowledge versus technique <fig:knowledge_vs_technique>` is a brief summary.

```{figure} ./img/knowledge_vs_technique/main.png
---
width: 50%
name: fig:knowledge_vs_technique
---
Knowledge versus technique in this book.
```

## How this book is structured

Most programming texts introduce readers to the building blocks of
programming and build up to using more sophisticated tools for a specific
purpose.

This is akin to teaching someone how to forge metal so as to make a nail and
then slowly work our way to using more sophisticated tools such as power tools
to build a house.

This book will do thing in a different way: we will start with using and
understanding tools that are helpful to mathematicians. In the later part of the
book we will cover the building blocks and you will be able to build your own
sophisticated tools.

The book is in two parts:

1. Tools for mathematics;
2. Building tools.

The first part of the book will not make use of any novel mathematics.
Instead we will consider a number of mathematics problem that are often covered
in secondary school.

- Algebraic manipulation
- Calculus (differentiation and integration)
- Permutations and combinations
- Probability
- Linear algebra

The questions we will tackle will be familiar in their presentation and
description. **What will be different** is that no **by hand** calculations will
be done. We will instead carry them all out using a programming language.

In the second part of the book you will be encouraged to build your own tools
to be able to tackle a problem type of your choice.

```{attention}
Every chapter will have 4 parts:

- A tutorial: you will be walked through solving a problem. You will be
  specifically told what to do and what to expect.
- A how to section: this will be a shorter more succinct section that will
  detail how to carry out specific things.
- A reference section: this will be a section with references to further
  resources as well as background information about specific things in the
  chapter.
- An exercise section: this will be a number of exercises that you can work on.
```


# A Simple Tutorial of Parareal Algorithm

 --- Demonstration with Jupyter notebook

 Xiaozhou Li (University of Electronic Science and Technology of China)

## History
The parareal algorithm was first presented by Lions, Maday, Turinici in 2001, but unfortunately, the paper was written in French.  The term "parareal" comes from words "parallel" and "real" (real time problem).

## Problem
### The Initial Value Problem
Consider the initial value problem:
\begin{align}
    u'(t) & = f(t,u(t)),\qquad t \in [0, T], \\
    u(0) & = u_0.
\end{align}
- It is clear that the solution $u$ later in time is affected (determined in this IVP).
- It is actual a serial problem.
- The purpose of parallel in time algorithms is to obtain the solution faster than sequential, but not necessarily scalability or efficiency.

### Notations
To describe numerical approximations of the IVP, in general, we divide the time interval
$[0,T]$ into $N$ subintervals such that
$$
    0 = t_0 < t_1 < \cdots < t_N = T,
$$
and $\Delta t_n = t_{n+1} - t_n = T/N$.

To describe the parareal algorithm, we denote the two numerical approximation methods by $\mathcal{G}$ (coarse) propagator and $\mathcal{F}$ (fine propagator).  Both $\mathcal{G}$ and $\mathcal{F}$ propagate an initial value $u_n$ to $u_{n+1}$ by approximating the solution of the IVP from $t_n$ to $t_{n+1}$.  That is
$$
    u_{n+1} = \mathcal{G}(t_n, t_{n+1}, u_n)\qquad or \qquad u_{n+1} = \mathcal{F}(t_n, t_{n+1}, u_n),
$$
by solving
\begin{align}
    u'(t) & = f(t,u(t)),\qquad t \in [t_n, t_{n+1}], \\
    u(t_n) & = u_n,
\end{align}
with propagator $\mathcal{G}$ or $\mathcal{F}$.

The propagators need satisfy the basic rules:
- coarse propagator $\mathcal{G}$: inaccurate, computationally very cheap.
- fine propagator $\mathcal{F}$: very accurate, but computationally very expensive.
- of course, propagator $\mathcal{G}$ and $\mathcal{F}$ have to satisfy certain relations for stability purpose.

#### Example 1
Considering a simple IVP problem that
\begin{align}
        u'(t) & = (1 - 2t) u(t),\qquad 0 \leq t \leq 3,\\
        u(0) & = 1.
\end{align}
The exact solution is $u(t) = e^{t - t^2}$.

- First, we divide interval $[0, 3]$ into $5$ equal subinterval $0 = t_0 < \cdots < t_5 = 3$, $t_n = n$ for $n = 0,\ldots,5$.
- Then, from $t_n$ to $t_{n+1}$ we can choose
    - $1$ step Euler method as the coarse propagator $\mathcal{G}$
    - $20$ steps Euler method as the fine propagator $\mathcal{F}$.
- Clearly, the fine propagator $\mathcal{F}$ is $20$ times more expensive than the coarse propagator $\mathcal{G}$, but much more accurate.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from IPython.display import clear_output, display
```

```python
def fun(t, u):
    return (1 - 2*t)*u

def u_ex(t):
    return np.exp(t - t**2)

def ExEuler(u0, t0, tn, n, RHS):
    # Explicit Euler methods from t0 to tn with initial value u0, number of steps = n, RHS(t,u) = fun(t,u)
    dt = (tn - t0)/n
    t = np.linspace(t0, tn, n+1)
    u = np.zeros(n+1) + u0
    for j in range(n):
        u[j+1] = u[j] + dt*RHS(t[j], u[j])
    return u[n]

def ImEuler(u0, t0, tn, n, RHS):
    dt = (tn - t0)/n
    t = np.linspace(t0, tn, n+1)
    u = np.zeros(n+1) + u0
    for j in range(n):
        u[j+1] = u[j]/(1 - dt*(1 - 2*t[j]))
    return u[n]

def draw_fun(t0, tn, N, fun):
    t = np.linspace(t0,tn,N+1)
    u = np.zeros(N+1)
    for i in range(N+1):
        u[i] = fun(t[i])
    plt.plot(t, u, 'k-', linewidth=2, label='exact')

u0 = 1.
t0 = 0.
tN = 3.
N = 5

t = np.linspace(t0, tN, N+1)
u = np.zeros(N+1) + u0

def draw_propagators(n_coarse, n_fine, RHS):
    fig=plt.figure(figsize=(12,9))
    plt.ylim(-0.2, 1.8)
    draw_fun(t0, tN, 100, u_ex)

    err = np.zeros(N+1)
    for j in range (N):
        u[j+1] = ExEuler(u[j], t[j], t[j+1], n_coarse, fun)
        #u[j+1] = ImEuler(u[j],t[j],t[j+1],n_coarse,fun)
        err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    plt.plot(t,u,'*',linewidth=2,label='coarse')
    print ('coarse propagator error: ', np.max(err))

    for j in range(N):
        u[j+1] = ExEuler(u[j],t[j],t[j+1],n_fine,fun)
        #u[j+1] = ImEuler(u[j],t[j],t[j+1],n_fine,fun)
        err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    plt.plot(t,u,'o',linewidth=2,label='fine')
    print ('fine propagator error:   ', np.max(err))
    plt.legend(fontsize=15)
    plt.show()

# interact or interactive with display
#interact(draw_propagators, n_coarse=(1,5), n_fine=(5,40))
w = interactive(draw_propagators, n_coarse=widgets.IntSlider(min=1,max=5,value=1), \
                n_fine=widgets.IntSlider(min=5,max=100,step=5,value=20), RHS=fixed(fun))
display(w)
```

    interactive(children=(IntSlider(value=1, description='n_coarse', max=5, min=1), IntSlider(value=20, descriptio…

## The Parareal Algorithm
### The Basic Idea
The basic idea of many parallel in time algorithms, since Nievergelt (1964), is
- prediction
- correction
and so does the parareal algorithem.

### Parareal Algorithm
1. __(SERIAL)__ Initialize the perdiction $u^0_n$ by computing an approximation in serial with the coarse propagator $\mathcal{G}$,
$$
    u^0_{n+1} = \mathcal{G}(t_n, t_{n+1}, u_n^0),
$$
from $n = 0,\ldots,N-1$ with $u_0^0 = u_0$ __(SERIAL)__.
2. __(Parallel)__ Computing the correction term by using the fine propagator $\mathcal{F}$ with the initial values given by previous iteration:
$$
    \mathcal{F}(t_n, t_{n+1}, u^k_n)
$$
in parallel with $N$ cores. __(Parallel)__
3. __(SERIAL)__ Computing the new value with the correction term,
$$
    u^{k+1}_{n+1} = \mathcal{G}(t_n, t_{n+1}, u_n^{k+1}) + \mathcal{F}(t_n, t_{n+1}, u^k_n) - \mathcal{G}(t_n, t_{n+1}, u^k_n)
$$
in serial. __(SERIAL)__
4. Repeating step 2 and 3 for $k = 0,\ldots$ until the desired accuracy is achieved.

### A Stey by Stey Demonstration
Considering the previous Example 1, but now using the parareal algorithm.

#### Step 1: Perdiction (in serial)
- For each subinterval $[t_n, t_{n+1}]$ for $n = 0,\ldots,4$, using $M_c$ steps explicit Euler Method as the coarse propagator, that is, start with the initial value, $u_0^0 = u_0$.
- For $n = 0,\ldots, N-1$:
    - $u_{n,0} = u_n^0$;
    - For $m = 0, \ldots, M_c-1$:
        - $t_{n,m} = t_n + \frac{m\Delta t_n}{M_c}$,
        - $u_{n,m+1} = u_{n,m} + \frac{\Delta t_n}{M_c}f(t_{n,m},u_{n,m+1}).$
    - $u_{n+1}^0 = u_{n,M}$.

```python
n_coarse = 0
def draw_propagator_prediction(Mc, RHS):
    fig=plt.figure(figsize=(12,9))
    plt.ylim(-0.2, 1.8)
    draw_fun(t0, tN, 100, u_ex)

    err = np.zeros(N+1)
    for j in range (N):
        u[j+1] = ExEuler(u[j], t[j], t[j+1], Mc, fun)
        #u[j+1] = ImEuler(u[j],t[j],t[j+1],n,fun)
        err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    plt.plot(t,u,'b*',linewidth=2)
    print ('prediction error: ', np.max(err))
    plt.show()
    return Mc

w = interactive(draw_propagator_prediction, Mc=widgets.IntSlider(min=1,max=5,value=1), RHS=fixed(fun))
display(w)
Mc = w.result

print ("Now we have the prediction values can be used as initial values for the parallel computing!")
print (u)
```

    interactive(children=(IntSlider(value=1, description='Mc', max=5, min=1), Output()), _dom_classes=('widget-int…

    Now we have the prediction values can be used as initial values for the parallel computing!
    [ 1.         1.6        1.408      0.22528   -0.1261568  0.1614807]

#### Step 2: Computing the Fine Propagator  (in parallel)
- Distribute the prediction values $u_n^k$ ($n = 0,\ldots, N-1$) to $N$ different cores.
- Each core only needs to compute the fine propagator $\mathcal{F}$ for subinterval $[t_n, t_{n+1}]$ with inital value $u_{n,0} = u_n^k$.
- Here, for the fine propagator $\mathcal{F}$, we use $M_f$ (of course, $M_f > M_c$, or better $M_f >> M_c$) steps explicit Euler Method.
- Then, computing $\mathcal{F}(t_n, t_{n+1}, u^k_n)$ in parallel.  That is
    - For $m = 0, \ldots, M_f-1$:
         - $t_{n,m} = t_n + \frac{m\Delta t_n}{M_f}$,
         - $u_{n,m+1} = u_{n,m} + \frac{\Delta t_n}{M_f}f(t_{n,m},u_{n,m+1}).$
    - $\mathcal{F}(t_n, t_{n+1}, u^k_n) = u_{n,M}$.
- __Note: __ In this step, we can actually compute $\mathcal{G}(t_n, t_{n+1}, u^k_n)$ in parallel too, but the difference is quite slight.

```python
u_fine = np.zeros([N+1,6])
def draw_propagator_parallel(process, Mf, RHS):
    # parameter i, [0%, 20%, 40%, 60%, 80%, 100%], describes the process
    # It is only for demonstration purpose, no needs for real implementation.
    fig=plt.figure(figsize=(12,9))
    plt.ylim(-0.2, 1.8)
    draw_fun(t0, tN, 100, u_ex)
    plt.plot(t,u,'b*',linewidth=2)
    i = round(process*5)
    u_old = np.copy(u)
    for j in range (N):
        u_fine[j,0] = u_old[j]
        tm = np.linspace(t[j], t[j+1], 6)
        for m in range (i):
            u_fine[j,m+1] = ExEuler(u_fine[j,m], tm[m], tm[m+1], round(Mf*0.2), fun)
            #u_fine[j,m+1] = ImEuler(u_fine[j,m], tm[m], tm[m+1], round(n*process), fun)
        if (i > 0): plt.plot(tm[1:i+1], u_fine[j,1:i+1], 'ro', linewidth=2)
    #if (i == 5)
        #print ('prediction error: ', np.max(err))
    plt.show()
    return Mf

w = interactive(draw_propagator_parallel, process=widgets.FloatSlider(min=0,max=1,step=0.2, value=1),\
                Mf=widgets.IntSlider(min=5,max=100,step=5, value=20), RHS=fixed(fun))
display(w)
Mf = w.result
```

    interactive(children=(FloatSlider(value=1.0, description='process', max=1.0, step=0.2), IntSlider(value=20, de…

#### Step 3: Computing the Correction  (in serial)
- Computing the new value by adding the correction between the coarse propagator $\mathcal{G}$ and the fine propagator $\mathcal{F}$:
$$
    u^{k+1}_{n+1} = \mathcal{G}(t_n, t_{n+1}, u_n^{k+1}) + \mathcal{F}(t_n, t_{n+1}, u^k_n) - \mathcal{G}(t_n, t_{n+1}, u^k_n)
$$

```python
def draw_propagator_correction():
    fig=plt.figure(figsize=(12,9))
    plt.ylim(-0.2, 1.8)
    draw_fun(t0, tN, 100, u_ex)
    plt.plot(t,u,'b*',linewidth=2)
    u_old = np.copy(u)
    err = np.zeros(N+1)
    for j in range (N):
        tm = np.linspace(t[j], t[j+1], 6)
        u[j+1] = ExEuler(u[j], t[j], t[j+1], Mc, fun) + u_fine[j,5] - ExEuler(u_old[j], t[j], t[j+1], Mc, fun)
        plt.plot(tm[1:], u_fine[j,1:], 'ro', linewidth=2)
        err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    print ('corrected error: ', np.max(err))
    plt.plot(t,u,'k*',linewidth=2)
    plt.show()

draw_propagator_correction()
```

#### Step 4: Repeating Step 2 and Step 3.
- Repeating step 2 and step 3 for $k = 1, \ldots,$ until the desired accuracy is obtained.
- Note: the final accuracy is bounded by the accuracy of the fine propagator.

```python
def draw_parareal_iteration(iters):
    fig=plt.figure(figsize=(12,9))
    plt.ylim(-0.2, 1.8)
    draw_fun(t0, tN, 100, u_ex)
    err = np.zeros(N+1)
    if (iters == 1):
        plt.plot(t,u_fine[:,0],'b*',linewidth=2)
        for j in range (N):
            tm = np.linspace(t[j], t[j+1], 6)
            plt.plot(tm[1:], u_fine[j,1:], 'ro', linewidth=2)
            err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    else:
        plt.plot(t,u,'b*',linewidth=2)
        u_old = np.copy(u)

        # computing the fine approximation in parallel
        for j in range (N):
            u_fine[j,0] = u_old[j]
            tm = np.linspace(t[j], t[j+1], 6)
            for m in range (5):
                u_fine[j,m+1] = ExEuler(u_fine[j,m], tm[m], tm[m+1], round(Mf*0.2), fun)
            plt.plot(tm[1:], u_fine[j,1:], 'ro', linewidth=2)

        # correction in serial
        for j in range (N):
            u[j+1] = ExEuler(u[j], t[j], t[j+1], Mc, fun) + u_fine[j,5] - ExEuler(u_old[j], t[j], t[j+1], Mc, fun)
            err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    print ('corrected error: ', np.max(err))
    plt.plot(t,u,'k*',linewidth=2)
    plt.show()

w = interactive(draw_parareal_iteration, iters=widgets.IntSlider(min=1,max=10,step=1, value=1))
display(w)
```

    interactive(children=(IntSlider(value=1, description='iters', max=10, min=1), Output()), _dom_classes=('widget…

### A Clear Code

```python
def parareal(N, Mc, Mf, iters):
    fig=plt.figure(figsize=(4,3))
    plt.ylim(-0.2, 1.8)
    draw_fun(t0, tN, 100, u_ex)

    u = np.zeros(N+1) + u0
    t = np.linspace(t0, tN, N+1)
    u_fine = np.zeros([N+1,6])
    err = np.zeros(N+1)
    # fine propagator
    for j in range(N):
        u[j+1] = ExEuler(u[j],t[j],t[j+1],Mf,fun)
        #u[j+1] = ImEuler(u[j],t[j],t[j+1],n_fine,fun)
        err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    plt.plot(t,u,'o',linewidth=2,label='fine')
    print ('fine propagator error:   ', np.max(err))

    # prediction
    for j in range (N):
        u[j+1] = ExEuler(u[j], t[j], t[j+1], Mc, fun)
        #u[j+1] = ImEuler(u[j],t[j],t[j+1],n,fun)
        err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    plt.plot(t,u,'b*',linewidth=2, label='prediction')
    print ('prediction error: ', np.max(err))

    for i in range (iters):
        u_old = np.copy(u)

        # computing the fine approximation in parallel
        for j in range (N):
            u_fine[j,0] = u_old[j]
            tm = np.linspace(t[j], t[j+1], 6)
            for m in range (5):
                u_fine[j,m+1] = ExEuler(u_fine[j,m], tm[m], tm[m+1], round(Mf*0.2), fun)
            #plt.plot(tm[1:], u_fine[j,1:], 'ro', linewidth=2)

        # correction in serial
        for j in range (N):
            u[j+1] = ExEuler(u[j], t[j], t[j+1], Mc, fun) + u_fine[j,5] - ExEuler(u_old[j], t[j], t[j+1], Mc, fun)
            err[j+1] = np.abs(u_ex(t[j+1]) - u[j+1])
    plt.plot(t,u,'*',linewidth=2, label='parareal')
    print ('parareal error: ', np.max(err))
    plt.legend()
    plt.show()

w = interactive(parareal, N=widgets.IntSlider(min=5,max=100,step=5,value=5), \
                Mc=widgets.IntSlider(min=1,max=5,value=1), \
                Mf=widgets.IntSlider(min=5,max=100,step=5,value=20),\
                iters=widgets.IntSlider(min=0,max=10,value=0))
display(w)
```

    interactive(children=(IntSlider(value=5, description='N', min=5, step=5), IntSlider(value=1, description='Mc',…

### Features
- Arbitrary order of accuracy, depends on the total number of iterations and the accuracy order of the coarse propagator G.
- It is a predictor corrector parallel algorithm.
- It can be combined with other iterative process, such as domain decomposition, multigrid, etc.
- Ideally, the algorithm needs for one processor for each subinterval.
- Stability issue becomes more complicated, the stability of the coarse and fine propagators need satisfy some restrictions.

```python

```


## SurfinPy

#### Tutorial 1 - Generating a phase diagram as a function of chemical potential and surface energy at 0 K

In this tutorial we will learn how to generate a basic phase diagram from DFT energies. This example will consider a series of surfaces that contain differing amounts of surface oxygen and adsorbed water species.
The physical quantity that is used to define the stability of a surface with a given composition is its surface energy $\gamma$ (J $m^{-2}$). Going forward in this tutorial we will use the example of water adsorbing on to defective Ti$O_2$ surfaces.

\begin{align}
\gamma_{Surf} & = \frac{1}{2A} \Bigg( E_{TiO_2}^{slab} - \frac{nTi_{slab}}{nTi_{Bulk}} E_{TiO_2}^{Bulk} \Bigg) - \Gamma_O \mu_O - \Gamma_{H_2O} \mu_{H_2O} ,
\end{align}

where A is the surface area, $E_{TiO_2}^{slab}$ is the DFT energy of the slab, $nTi_{Slab}$ is the number of cations in the slab, $nTi_{Bulk}$ is the number of cations in the bulk unit cell, $E_{TiO_2}^{Bulk}$ is the DFT energy of the bulk unit cell and

\begin{align}
\Gamma_O & = \frac{1}{2A} \Bigg( nO_{Slab} - \frac{nO_{Bulk}}{nTi_{Bulk}}nTi_{Slab}  \Bigg) ,
\end{align}

\begin{align}
\Gamma_{H_2O} & = \frac{nH_2O}{2A} ,
\end{align}

where $nO_{Slab}$ is the number of anions in the slab, $nO_{Bulk}$ is the number of anions in the bulk and $nH_2O$ is the number of adsorbing water molecules. $\Gamma_O$ / $\Gamma_{H_2O}$ is the excess oxygen / water at the surface and $\mu_O$ / $\mu_{H_2O}$ is the oxygen / water chemcial potential. Clearly $\Gamma$ and $\mu$ will only matter when the surface is non stoichiometric.

So now lets work through an example

```python
import matplotlib.pyplot as plt
from surfinpy import mu_vs_mu
from surfinpy import data
```

The first thing to do is input the data that we have generated from our DFT simulations. The input data needs to be contained within a `surfinpy.data` object. First we have created a `surfinpy.data.ReferenceDataSet` object for the bulk data (reference data), where `cation` is the number of cations, `anion` is the number of anions, `energy` is the DFT energy and `funits` is the number of formula units.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 2, energy = -780.0, funits = 4)
```

Next we create the surface `surfinpy.data.DataSet` objects - one for each surface or "phase". `cation` is the number of cations, `x` is in this case the number of oxygen species (corresponding to the X axis of the phase diagram), `y` is the number of in this case water molecules (corresponding to the Y axis of our phase diagram), `area` is the surface area, `energy` is the DFT energy, `label` is the label for the surface (appears on the phase diagram) and finally `nSpecies` is the number of adsorbin species.

```python
pure = data.DataSet(cation = 24, x = 48, y = 0, area = 60.0, energy = -575.0, label = "0.00 $TiO_2$", nspecies = 1)
H2O = data.DataSet(cation = 24, x = 48, y = 2, area = 60.0, energy = -612.0, label = "0.16 $TiO_2$", nspecies = 1)
H2O_2 = data.DataSet(cation = 24, x = 48, y = 4, area = 60.0, energy = -640.0, label = "0.32 $TiO_2$", nspecies = 1)
H2O_3 = data.DataSet(cation = 24, x = 48, y = 8, area = 60.0, energy = -676.0, label = "0.64 $TiO_2$", nspecies = 1)
Vo = data.DataSet(cation = 24, x = 46, y = 0, area = 60.0, energy = -558.0, label = "0.00 $TiO_1.9$", nspecies = 1)
H2O_Vo_1 = data.DataSet(cation = 24, x = 46, y = 2, area = 60.0, energy = -594.0, label = "0.00 $TiO_1.9$", nspecies = 1)
H2O_Vo_2 = data.DataSet(cation = 24, x = 46, y = 4, area = 60.0, energy = -624.0, label = "0.16 $TiO_1.9$", nspecies = 1)
H2O_Vo_3 = data.DataSet(cation = 24, x = 46, y = 6, area = 60.0, energy = -640.0, label = "0.32 $TiO_1.9$", nspecies = 1)
H2O_Vo_4 = data.DataSet(cation = 24, x = 46, y = 8, area = 60.0, energy = -670.0, label = "0.64 $TiO_1.9$", nspecies = 1)
```

Next we need to create a list of our data. Don't worry about the order, surfinpy will sort that out for you.

```python
data = [pure, Vo, H2O, H2O_Vo_1, H2O_2, H2O_Vo_2, H2O_3, H2O_Vo_3,  H2O_Vo_4]
```

We now need to generate our X and Y axis, or more appropriately, our chemical potential values. Again these exist in a dictionary. 'Range' corresponds to the range of chemcial potential values to be considered and 'Label' is the axis label.
```
    deltaX = {'Range': Range of Chemical Potential,
              'Label': Species Label}

```

```python
deltaX = {'Range': [ -12, -6],  'Label': 'O'}
deltaY = {'Range': [ -19, -12], 'Label': 'H_2O'}
```

And finally we can generate our plot using these 4 variables of data.

```python
system = mu_vs_mu.calculate(data, bulk, deltaX, deltaY)
ax = system.plot_phase(cbar_title="$H_2O$ $/$ $nm^2$", figsize=(6, 5))
plt.savefig("../../../docs/source/Figures/Surfaces_1.png", dpi=600)
```

This plot is a good start in that the relative stability of the surfaces has been evaluated. However the chemical potential values are essentially meaningless and can be dependent on the pseudopotentials. The first thing that we can do is add in the DFT energy of species X/Y, this will give us a 0K phase diagram.

```python
Zero_K = mu_vs_mu.calculate(data, bulk, deltaX, deltaY, x_energy=-4.54, y_energy=-14.84)
ax = Zero_K.plot_phase(colourmap="RdYlBu", cbar_title="$H_2O$ $/$ $nm^2$", figsize=(6, 5))
plt.savefig("../../../docs/source/Figures/Surfaces_2.png", dpi=600)
```

```python

```


```python
!pip install qiskit
```

    Requirement already satisfied: qiskit in /usr/local/lib/python3.7/dist-packages (0.36.1)
    Requirement already satisfied: qiskit-terra==0.20.1 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.20.1)
    Requirement already satisfied: qiskit-aer==0.10.4 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.10.4)
    Requirement already satisfied: qiskit-ignis==0.7.0 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.7.0)
    Requirement already satisfied: qiskit-ibmq-provider==0.19.1 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.19.1)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.4->qiskit) (1.7.3)
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.4->qiskit) (1.21.6)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (2.23.0)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (1.24.3)
    Requirement already satisfied: websocket-client>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (1.3.2)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (1.1.0)
    Requirement already satisfied: websockets>=10.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (10.3)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.19.1->qiskit) (2.8.2)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Requirement already satisfied: retworkx>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (0.11.0)
    Requirement already satisfied: ply>=3.10 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (3.11)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (1.7.1)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (0.3.4)
    Requirement already satisfied: stevedore>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (3.5.0)
    Requirement already satisfied: tweedledum<2.0,>=1.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (1.1.1)
    Requirement already satisfied: symengine>=0.9 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (0.9.2)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (5.4.8)
    Requirement already satisfied: python-constraint>=1.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.20.1->qiskit) (1.4.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.19.1->qiskit) (1.15.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.19.1->qiskit) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.19.1->qiskit) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.19.1->qiskit) (2021.10.8)
    Requirement already satisfied: cryptography>=1.3 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.19.1->qiskit) (37.0.2)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.19.1->qiskit) (1.5.0)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.19.1->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.19.1->qiskit) (2.21)
    Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (5.9.0)
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (4.11.3)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (3.8.0)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.20.1->qiskit) (4.2.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.20.1->qiskit) (1.2.1)

```python
from qiskit import QuantumCircuit, Aer, assemble, execute
```

```python
qc = QuantumCircuit(3,3)
qc.x(0)
qc.barrier()
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7fea77573280>

```python
qc.h(1)
qc.cx(1,2)
qc.draw()
```

     ┌───┐ ░
q_0: ┤ X ├─░───────────
     └───┘ ░ ┌───┐
q_1: ──────░─┤ H ├──■──
           ░ └───┘┌─┴─┐
q_2: ──────░──────┤ X ├
           ░      └───┘
c: 3/══════════════════

```python
qc.cx(0,1)
qc.h(0)
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7fea77498aa0>

```python
qc.barrier()
qc.measure([0,1],[0,1])
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7fea77429730>

```python
qc.draw()
```

     ┌───┐ ░                ┌───┐ ░ ┌─┐
q_0: ┤ X ├─░─────────────■──┤ H ├─░─┤M├───
     └───┘ ░ ┌───┐     ┌─┴─┐└───┘ ░ └╥┘┌─┐
q_1: ──────░─┤ H ├──■──┤ X ├──────░──╫─┤M├
           ░ └───┘┌─┴─┐└───┘      ░  ║ └╥┘
q_2: ──────░──────┤ X ├───────────░──╫──╫─
           ░      └───┘           ░  ║  ║
c: 3/════════════════════════════════╩══╩═
                                     0  1

```python
qc.barrier()
qc.cx(1,2)
qc.cz(0,2)
qc.draw()
```

     ┌───┐ ░                ┌───┐ ░ ┌─┐    ░
q_0: ┤ X ├─░─────────────■──┤ H ├─░─┤M├────░───────■─
     └───┘ ░ ┌───┐     ┌─┴─┐└───┘ ░ └╥┘┌─┐ ░       │
q_1: ──────░─┤ H ├──■──┤ X ├──────░──╫─┤M├─░───■───┼─
           ░ └───┘┌─┴─┐└───┘      ░  ║ └╥┘ ░ ┌─┴─┐ │
q_2: ──────░──────┤ X ├───────────░──╫──╫──░─┤ X ├─■─
           ░      └───┘           ░  ║  ║  ░ └───┘
c: 3/════════════════════════════════╩══╩════════════
                                     0  1

```python
qc.measure(2,2)
sim = Aer.get_backend('qasm_simulator')
result = execute(qc,backend=sim,shots=1024).result()
counts = result.get_counts()
from qiskit.visualization import plot_histogram
plot_histogram(counts)

```

```python
qc.draw()
```

     ┌───┐ ░                ┌───┐ ░ ┌─┐    ░
q_0: ┤ X ├─░─────────────■──┤ H ├─░─┤M├────░───────■────
     └───┘ ░ ┌───┐     ┌─┴─┐└───┘ ░ └╥┘┌─┐ ░       │
q_1: ──────░─┤ H ├──■──┤ X ├──────░──╫─┤M├─░───■───┼────
           ░ └───┘┌─┴─┐└───┘      ░  ║ └╥┘ ░ ┌─┴─┐ │ ┌─┐
q_2: ──────░──────┤ X ├───────────░──╫──╫──░─┤ X ├─■─┤M├
           ░      └───┘           ░  ║  ║  ░ └───┘   └╥┘
c: 3/════════════════════════════════╩══╩═════════════╩═
                                     0  1             2

```python
print(counts)
```

    {'111': 210, '100': 277, '101': 297, '110': 240}

```python

```


```python
import tensorflow as tf
```

```python
from pycalphad import Database, Model, variables as v
from pycalphad.codegen.sympydiff_utils import build_functions
from sympy import lambdify
import numpy as np

dbf = Database('Al-Cu-Zr_Zhou.tdb')
mod = Model(dbf, ['AL', 'CU', 'ZR'], 'LIQUID')
```

```python
mod.variables
```

    [T, LIQUID0AL, LIQUID0CU, LIQUID0ZR]

# Expression Building

```python
%time bfr = build_functions(mod.GM, mod.variables, include_grad=True)
```

    Wall time: 377 ms

```python
%%timeit
tf_func = lambdify(mod.variables, mod.GM, modules='tensorflow')
tf_grads = lambdify(mod.variables, [mod.GM.diff(x) for x in mod.variables], modules='tensorflow')
```

    632 ms ± 20.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
tf_func = lambdify(mod.variables, mod.GM, modules='tensorflow')
tf_grads = lambdify(mod.variables, [mod.GM.diff(x) for x in mod.variables], modules='tensorflow')
```

```python
%%timeit
func_xla = tf.function(experimental_compile=True)(tf_func)
grad_xla = tf.function(experimental_compile=True)(tf_grads)
```

    93 µs ± 3.83 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
func_xla = tf.function(experimental_compile=True)(tf_func)
grad_xla = tf.function(experimental_compile=True)(tf_grads)
```

# Function Evaluation

```python
out = np.array([0.])
%timeit bfr.func.unsafe_real(np.array([300., 0.3, 0.3, 0.4]), out)
print(out)
```

    3.94 µs ± 189 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
    [-41178.9428136]

```python
out = np.zeros(4)
%timeit bfr.grad.unsafe_real(np.array([300., 0.3, 0.3, 0.4]), out)
print(out)
```

    4.27 µs ± 84.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
    [-3.39000632e+01 -6.67993032e+04 -2.12186404e+04 -3.12432907e+04]

```python
%timeit func_xla(300.0, 0.3, 0.3, 0.4)
```

    202 µs ± 76.5 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%timeit grad_xla(300.0, 0.3, 0.3, 0.4)
```

    299 µs ± 71.8 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
temps = tf.constant(np.linspace(300., 2000., num=1000, dtype=np.float32))
%timeit grad_xla(temps, 0.3, 0.3, 0.4)
```

    480 µs ± 76.4 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python

```


# Introduction

This workbook is an introduction to the basic concepts and designs relating to the paper

**Fast estimation of sparse quantum noise** by *Harper, Yu and Flammia* (in production)

This workbook is going to go through the basic ideas behind the Walsh Hadamard Transform for a 2 qubit system. I will focus on the type of transforms, and how it relates eigenvalues (fidelities) and probabilities and measurement outcomes.

Copyright 2020: Robin Harper

# Software needed

For this introductory notebook, we need minimal software. All these packages should be available through the Julia package manager.

If you get an error trying to "use" them the error message tells you how to load them.

```julia
using Hadamard

# convenience (type /otimes<tab>) - <tab> is the "tab" key.
⊗ = kron
# type /oplus<tab>
⊕ = (x,y)->mod.(x+y,2)

```

    #3 (generic function with 1 method)

# Some preliminary information

## Conventions

### What's in a name?

There are a number of conventions as to where which qubit should be. Here we are going to adopt a least significant digit approach - which is different from the normal 'ket' approach.

So for example: $IZ$ means an $I$ 'Pauli' on qubit = 2 and a $Z$ 'Pauli' on qubit = 1 (indexing off 1).

### Arrays indexed starting with 1.

For those less familiar with Julia, unlike - say - python all arrays and vectors are indexed off 1. Without going into the merits or otherwise of this, we just need to keep it in mind. With our bitstring the bitstring 0000 represents the two qubits $II$, it has value 0, but it will index the first value in our vector i.e. 1.

### Representing Paulis with bitstrings.

There are many ways to represent Paulis with bit strings, including for instance the convention used in **Improved Simulation of Stabilizer Circuits**, *Scott Aaronson and Daniel Gottesman*, [arXiv:quant-ph/0406196v5](https://arxiv.org/abs/quant-ph/0406196v5).

Here we are going to use one that allows us to naturally translate the Pauli to its position in our vector of Pauli eigenvalues (of course this is arbitrary, we could map them however we like).

The mapping I am going to use is this (together with the least significant convention):

---

- I $\rightarrow$ 00
- X $\rightarrow$ 01
- Y $\rightarrow$ 10
- Z $\rightarrow$ 11

---

This then naturally translates as below:

### SuperOperator  - Pauli basis

We have defined our SuperOperator basis to be as below, which means with the julia vector starting at 1 we have :

    Pauli
    Vector Index
    ---
    Binary
    Integer

    II
    1
    →
    0000
    0

    IX
    2
    →
    0001
    1

    IY
    3
    →
    0010
    2

    IZ
    4
    →
    0011
    3

    XI
    5
    →
    0100
    4

    XX
    6
    →
    0101
    5

    XY
    7
    →
    0110
    6

    XZ
    8
    →
    0111
    7

    YI
    9
    →
    1000
    8

    YX
    10
    →
    1001
    9

    YY
    11
    →
    1010
    10

    YZ
    12
    →
    1011
    11

    ZI
    13
    →
    1100
    12

    ZX
    14
    →
    1101
    13

    ZY
    15
    →
    1110
    14

    ZZ
    16
    →
    1111
    15

I have set this out in painful detail, because its important to understand the mapping for the rest to make sense.

So for instance in our EIGENVALUE vector (think superoperator diagonal, or what we used to call the Pauli fidelities), the PAULI $YX$ has binary representation $Y$=10 $X$=01, therefore 1001, has "binary-value" 9 and is the 10th (9+1) entry in our eigenvalue vector.

## What's in a Walsh Hadamard Transform?

The standard Walsh-Hadmard transform is based off tensor (Kronecker) products of the following matrix:

$$\left(\begin{array}{cc}1 & 1\\1 & -1\end{array}\right)^{\otimes n}$$

### WHT_natural

So for one qubit it would be:

$\begin{array}{cc}
& \begin{array}{cccc} \quad 00 &  \quad 01 & \quad 10  &\quad 11 \end{array}\\
\begin{array}{c} 00\\ 01\\ 10\\11\end{array} &
  \left(\begin{array}{cccc}
   \quad 1&\quad 1& \quad 1 &\quad 1\\
   \quad 1 &\quad -1 & \quad 1 &\quad -1\\   \quad 1&\quad 1 & \quad -1 &\quad -1\\   \quad 1 &\quad -1 & \quad -1 &\quad 1\\\end{array}\right)
\end{array}$

where I have included above (and to the left) of the transform matrix the binary representations of the position and the matrix can also be calculated as $(-1)^{\langle i,j\rangle}$ where the inner product here is the binary inner product $i,j\in\mathbb{F}_2^n$ as $\langle i,j\rangle=\sum^{n-1}_{t=0}i[t]j[t]$ with arithmetic over $\mathbb{F}_2$

### WHT_Pauli

In the paper we use a different form of the Walsh-Hadamard transform. In this case we use the inner product of the Paulis, not the 'binary bitstring' inner product. The matrix is subtly different some rows or, if you prefer, columns are swapped:

$\begin{array}{cc}
& \begin{array}{cccc} I(00) &  X(01) & Y(10)  & Z(11) \end{array}\\
\begin{array}{c} I(00)\\ X(01)\\ Y(10)\\Z(11)\end{array} &
  \left(\begin{array}{cccc}
   \quad 1&\quad 1& \quad 1 &\quad 1\\
   \quad 1 &\quad 1 & \quad -1 &\quad -1\\   \quad 1&\quad -1 & \quad 1 &\quad -1\\   \quad 1 &\quad -1 & \quad -1 &\quad 1\\\end{array}\right)
\end{array}$

To calculate this one, we can use the 'symplectic inner product' to work it out. In particular for one of the indices you just need to swap the X and Y components. A matrix of the form

$$\begin{align}
    J_n = \mathbf{I}_n\otimes X = \left(\begin{smallmatrix}
    0 & 1 & & & \\
    1 & 0 & & & \\
      & & \ddots & & \\
      & & & 0 & 1  \\
      & & & 1 & 0
    \end{smallmatrix}\right),
\end{align}$$

will do this and then use the binary inner product between the index and the transformed index. (The binary inner is detailed above).

### Which one to use

When transforming Pauli eigenvalue to the probability of a particular error occuring, there are distinct advantages in using the WHT_Pauli transform. The order of the Pauli errors and the order of the Pauli eigenvalues remains the same. However, most common packages (including the one we are going to use here in Julia) don't support this type of transform, rather they implement the WHT_natural transform. The WHT_natural transform also makes the peeling algorithm slightly less fiddly. **However** it does mean we need to be very careful about the order of things. If we use the WHT_natural transform then the following relationship holds - *note the indices (labels) of the Paulis in the probability vector*:

So the natural translation (labelling of Paulis) then becomes as follows:

### Eigenvalue vector space ###

- I $\rightarrow$ 00
- X $\rightarrow$ 01
- Y $\rightarrow$ 10
- Z $\rightarrow$ 11

### Probability vector space ###

- I $\rightarrow$ 00
- X $\rightarrow$ 10
- Y $\rightarrow$ 01
- Z $\rightarrow$ 11

---

The numbers here are the numbers from a previous example (with sparse errors) - for the purposes of demonstrating the next step, I'll use a denser probability distribution.

# Set up some Pauli Errors to demonstrate

```julia
# TO demonstrate the peeling decoder we are going to set up a sparse (fake) distribution.
# This is going to mirror the example shown above -
# NOTE Julia indexing starts at 1, so we add 1 to the integers in the above table
# TO demonstrate the peeling decoder we are going to set up a sparse (fake) distribution.
# This is going to be a 4 qubit (so 4^4 = 256 possible probabilities)

dist = zeros(16)
cl = zeros(2,4)

# Qubit 1
cl[1,2] = 0.01 #y1
cl[1,3] = 0.004 #x1
cl[1,4] = 0.001 #z1
cl[1,1] = 1-sum(cl[1,:])

# Qubit 2
cl[2,2] = 0.03
cl[2,3] = 0.02
cl[2,4]=  0.005
cl[2,1] = 1-sum(cl[2,:])

for q1 in 1:4
    for q2 in 1:4
      dist[(q2-1)*4+q1] = cl[1,q1]*cl[2,q2]
    end
end

# rounding to 10 digits just to get rid of some annoying numerical instability
dist = round.(dist,digits=10)
basicSuperOpDiagonal = round.(ifwht_natural(dist),digits=10)
print("Basic SuperOperator (diagonal): $basicSuperOpDiagonal\n")
print("Distribution: $(dist)\n")
```

    Basic SuperOperator (diagonal): [1.0, 0.978, 0.99, 0.972, 0.93, 0.90954, 0.9207, 0.90396, 0.95, 0.9291, 0.9405, 0.9234, 0.9, 0.8802, 0.891, 0.8748]
    Distribution: [0.930825, 0.00945, 0.00378, 0.000945, 0.02955, 0.0003, 0.00012, 3.0e-5, 0.0197, 0.0002, 8.0e-5, 2.0e-5, 0.004925, 5.0e-5, 2.0e-5, 5.0e-6]

## So why do we need the transform.

There are two main reasons:

- We can determine the eigenvalues in a SPAM (state preperation and measurement) error free way. This uses the technicques of randomised benchmarking and is listed later.
- We can't actually deteremine the probabilities directly - but we can determine the eigenvalues.

Why do I say we can't determine the probabilities directly?

## How to work out what we "measure"

Well we can only measure a commuting set of probabilities simultaneously (as otherwise the measurement alters the state). So for instance if we prepare in the computational basis $|00\rangle\langle00|$, and measure the overlap between the noise altered state and the different computational basis states we get a probability distribution of the chance of *observing* an error.

So our measurements here are a projective basis.

$M_{00}=|00\rangle\langle00|,M_{01}=|01\rangle\langle01|,M_{10}=|10\rangle\langle10|,M_{11}=|11\rangle\langle11|$

Which you should be able to convince yourself form the following density matrices:

$M_{00}=\left(\begin{array}{cc}1&0&0&0\\0&0&0&0\\0&0&0&0\\0&0&0&0\end{array}\right), M_{01}=\left(\begin{array}{cc}0&0&0&0\\0&1&0&0\\0&0&0&0\\0&0&0&0\end{array}\right),M_{10}=\left(\begin{array}{cc}0&0&0&0\\0&0&0&0\\0&0&1&0\\0&0&0&0\end{array}\right),M_{11}=\left(\begin{array}{cc}0&0&0&0\\0&0&0&0\\0&0&0&0\\0&0&0&1\end{array}\right)$

(note they sum to the identity matrix - see Neilson & Chuang as to why this is a POVM).

And if we vectorise these in the Pauli basis we get the following:

$(M_{00}|=\left(\begin{array}{*{11}c} 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 1 &\end{array}\right)$

$(M_{01}|=\left(\begin{array}{*{11}c} 1 & 0 & 0 & -1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & -1  \end{array}\right)$

$(M_{10}|=\left(\begin{array}{*{11}c} 1 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & -1 & 0 & 0 & -1  \end{array}\right)$

$(M_{11}|=\left(\begin{array}{*{11}c} 1 & 0 & 0 & -1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & -1 & 0 & 0 & 1  \end{array}\right)$

Where I have ignored the normalisation factor of 0.25 in front of each of these (Sometimes you will see the normalisation split evenly betweeen the state prep and measurment - i.e 0.5 on both). Basically I'll just divide everything by 4!

The state preperation vector is just

$|\rho_{00})=||00\rangle\langle00|)=\left(\begin{array}{*{11}c} 1 \\ 0 \\ 0 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 1 \\ 0 \\ 0 \\ 1 \\ \end{array}\right)$

, so we can see in the absence of noise

$(M_{00}|\rho_{00}) = 1$,$(M_{01}|\rho_{00}) = 0$,$(M_{10}|\rho_{00}) = 0$,$(M_{11}|\rho_{00}) = 0$

which (since they form a basis) is what we would expect.

So how does noise affect this?

```julia
zs =
 [[1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1],
 [1, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, -1],
 [1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, -1],
 [1, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, 1]]
```

    4-element Array{Array{Int64,1},1}:
     [1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1]
     [1, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, -1]
     [1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, -1]
     [1, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, 1]

```julia
# Side note if you have installed my Juqst package (see https://github.com/rharper2/Juqst.jl), then
# You can generate these for any size using Juqst.genZs(2) (for 2 qubits).
```

```julia
using LinearAlgebra

superOperatorNoise = diagm(0=>basicSuperOpDiagonal)

```

    16×16 Array{Float64,2}:
     1.0  0.0    0.0   0.0    0.0   …  0.0     0.0  0.0     0.0    0.0
     0.0  0.978  0.0   0.0    0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.99  0.0    0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.972  0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.93     0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0   …  0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0   …  0.0     0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.9234  0.0  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.9  0.0     0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.0  0.8802  0.0    0.0
     0.0  0.0    0.0   0.0    0.0      0.0     0.0  0.0     0.891  0.0
     0.0  0.0    0.0   0.0    0.0   …  0.0     0.0  0.0     0.0    0.8748

## If we measure this noise using these measurement operators then we get the following:

```julia
# The measurement outcomes are then:
print("M_00 = ↑↑ = $(round(0.25*zs[1]'*superOperatorNoise*zs[1],digits=10))\n")
print("M_01 = ↑↓ = $(round(0.25*zs[2]'*superOperatorNoise*zs[1],digits=10))\n")
print("M_10 = ↓↑ = $(round(0.25*zs[3]'*superOperatorNoise*zs[1],digits=10))\n")
print("M_11 = ↓↓ = $(round(0.25*zs[4]'*superOperatorNoise*zs[1],digits=10))\n")
outcomes = 0.25.*[x'*superOperatorNoise*zs[1] for x in zs]
print("total probability = $(sum(outcomes))\n")

```

    M_00 = ↑↑ = 0.9367
    M_01 = ↑↓ = 0.0133
    M_10 = ↓↑ = 0.0493
    M_11 = ↓↓ = 0.0007
    total probability = 1.0

### What do these probabilities represent?

Basically if we took a quantum device, prepared the |00$\rangle$ state, subjected it to that noisy channel and then measured the resulting state, we would expect to see the above probability distribution on the measurement results.

One question we might ask then is what would we see if we ran it through the noisy channel twice. Well because we know the noisy channel we can simply apply it twice in simulation and see the new probability distribution.

On an actual device we can only see the probability distribution for $2^n$ commuting measurements at a time - and that makes it a lot more difficult to work the global probability distribution - as we explore now:

## So how does this relate to the acutal global distribution we had?

Well at first blush it doesn't if we remember the actual probability distribution looks like this:

```julia
# Some function to give us labels:
function probabilityLabels(x;qubits=2)
    str = string(x,base=4,pad=qubits)
    paulis = ['I','Y','X','Z']
    return map(x->paulis[parse(Int,x)+1],str)
end

function fidelityLabels(x;qubits=2)
    str = string(x,base=4,pad=qubits)
    paulis = ['I','X','Y','Z']
    return map(x->paulis[parse(Int,x)+1],str)
end
```

    fidelityLabels (generic function with 1 method)

```julia

# Print label and probability
for i in 1:16
    print("$i: $(probabilityLabels(i-1)): $(dist[i])\n")
end
```

    1: II: 0.930825
    2: IY: 0.00945
    3: IX: 0.00378
    4: IZ: 0.000945
    5: YI: 0.02955
    6: YY: 0.0003
    7: YX: 0.00012
    8: YZ: 3.0e-5
    9: XI: 0.0197
    10: XY: 0.0002
    11: XX: 8.0e-5
    12: XZ: 2.0e-5
    13: ZI: 0.004925
    14: ZY: 5.0e-5
    15: ZX: 2.0e-5
    16: ZZ: 5.0e-6

So we can see the IZ pauli error rate is 0.00945, yet we measured updown as 0.0133, what gives?

Well the probability of "seeing" an error in the computational basis is NOT the IZ error rate. In fact we can't see a Z error at all. We can only *see* an X or a Y error.

Therefore we will *see* an error on qubit 1 (only) if there is no error *seen* on qubit 2, and an error seen on qubit 1. The pauli errors that will give us *only* an error on qubit 1 are:

IX, IY, ZY and ZX (recall qubit 1 is the right most digit).

Those correspond to indices 2,3, 14 and 15 above:

```julia
print("Sum of relevant probabilities: $(dist[2]+dist[3]+dist[14]+dist[15])\n")
print("Expected measurement outcome: M_01 = ↑↓ = $(round(0.25*zs[2]'*superOperatorNoise*zs[1],digits=10))\n")
```

    Sum of relevant probabilities: 0.0133
    Expected measurement outcome: M_01 = ↑↓ = 0.0133

Similarly if we wanted the probability of *seeing* an error on qubit 2,

XI, YI, YZ, XZ
(9,5,8 and 12)

```julia
print("Sum of relevant probabilities: $(round(dist[9]+dist[5]+dist[8]+dist[12],digits=10))\n")
print("Expected measurement outcome: M_10 = ↓↑ = $(round(0.25*zs[3]'*superOperatorNoise*zs[1],digits=10))\n")
```

    Sum of relevant probabilities: 0.0493
    Expected measurement outcome: M_10 = ↓↑ = 0.0493

And finally the chance of seeing an error on BOTH qubits is
YX, YY, XX, XY
(7,6,11,10)

```julia
print("Sum of relevant probabilities: $(round(dist[7]+dist[6]+dist[11]+dist[10],digits=10))\n")
print("Expected measurement outcome: M_11 = ↓↓ = $(round(0.25*zs[4]'*superOperatorNoise*zs[1],digits=10))\n")
```

    Sum of relevant probabilities: 0.0007
    Expected measurement outcome: M_11 = ↓↓ = 0.0007

## But what is the important thing to keep in mind?

Well its that these probabilities (which are made up combinations of the Pauli error rates - transform directly to the appropriate eigenvalue entry). So if we set out the probabilities in a vector and then transform them: We get the entries in the eigenvalue vector.

So the 'measured' proabilities in the computational basis are:

$\left(\begin{array}{c}0.9367\\0.0133\\0.0493\\0.0007\end{array}\right)$

We can use the WHT (natural) transform on this:$\qquad \left(\begin{array}{*{11}c} 1 & 1 & 1 & 1 \\ 1 & -1 & 1 & -1 \\ 1 & 1 & -1 & -1 \\ 1 & -1 & -1 & 1 \\ \end{array}\right)$

To get
$\qquad\left(\begin{array}{c}1.0\\0.972\\0.9\\ 0.8748\end{array}\right)$

```julia
round.(Hadamard.hadamard(4)*[0.9367, 0.0133, 0.0493, 0.0007],digits=10)
```

    4-element Array{Float64,1}:
     1.0
     0.972
     0.9
     0.8748

And those are exactly the entries (for II, IZ, ZI and ZZ)  in the diagonal of the super operator

```julia
# Print label and probability
for i in 1:16
    print("$i: $(fidelityLabels(i-1)): $(basicSuperOpDiagonal[i])\n")
end
```

    1: II: 1.0
    2: IX: 0.978
    3: IY: 0.99
    4: IZ: 0.972
    5: XI: 0.93
    6: XX: 0.90954
    7: XY: 0.9207
    8: XZ: 0.90396
    9: YI: 0.95
    10: YX: 0.9291
    11: YY: 0.9405
    12: YZ: 0.9234
    13: ZI: 0.9
    14: ZX: 0.8802
    15: ZY: 0.891
    16: ZZ: 0.8748

## So this gives us the way we can reconstruct the Global probability distribution

(which in turn tells us the chance of an X error or Y error or Z error on each qubit, which in turn lets us e.g. fine-tune our error correction decoders)

We can measure each different set of stabiliser measurements until we have reconstructed the entire superoperator (in fact we don't need to do this much work, but that is for another time!)

One final example:

We should be able to see that II, ZX, YZ and XY are all mutually commuting - that means we can in fact measure them all at the same time.

How would that work? Well we just have to find the correct basis. (As always), there are several ways of thinking of this. We can imagine that if we could prepare the following state:

```julia
req = [1,0,0,0,0,0,1,0,0,0,0,1,0,1,0,0]
for i in 1:16
    print("$(string(i,pad=2)) $(fidelityLabels(i-1)): $(req[i])\n")
end
```

    01 II: 1
    02 IX: 0
    03 IY: 0
    04 IZ: 0
    05 XI: 0
    06 XX: 0
    07 XY: 1
    08 XZ: 0
    09 YI: 0
    10 YX: 0
    11 YY: 0
    12 YZ: 1
    13 ZI: 0
    14 ZX: 1
    15 ZY: 0
    16 ZZ: 0

Then that is going to pick up the relevant entries from the SuperOperator representation. If we then measure in the same basis we are going to get the correct probabilities, which we can transform back to get the eigenvalues.

Of course because we are just manipulating matrices it all feels a bit 'circular'. In a device where we don't know the eigenvalues, what we have to do is actually run the circuits.

So the circuit that transforms into various stabiliser groups look like this

Here we are interested in (a).

**Note**

Here I am going to use some code that is available at:
[github - rharper2/Juqst.jl](https://github.com/rharper2/Juqst.jl)

```julia
using Juqst
# Set up the relevant gates
cnot21 = [1 0;0 1]⊗[1 0;0 0]+ [0 1;1 0]⊗[0 0;0 1]
superCnot21 = makeSuper(cnot21)
superPI = makeSuper([1 0;0 im]⊗[1 0;0 1])
superIP = makeSuper([1 0;0 1]⊗[1 0;0 im])
superHH = makeSuper((1/sqrt(2)*[1 1;1 -1])⊗(1/sqrt(2)*[1 1;1 -1]));
```

```julia
# Circuit (a) is then just these gates applied (here circuit is from left to right, but matrices the other way!)

circuit = superCnot21*superPI*superCnot21*superIP*superHH
```

\begin{equation*} \left(\begin{array}{*{16}c} 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ \end{array}\right)\\\end{equation*}

And we can apply this to the $|00\rangle$ state, to generate the correct state:

```julia
newB = circuit*zs[1]
for i in 1:16
    print("$(string(i,pad=2)) $(fidelityLabels(i-1)): $(newB[i])\n")
end
```

    01 II: 1.0
    02 IX: 0.0
    03 IY: 0.0
    04 IZ: 0.0
    05 XI: 0.0
    06 XX: 0.0
    07 XY: 1.0
    08 XZ: 0.0
    09 YI: 0.0
    10 YX: 0.0
    11 YY: 0.0
    12 YZ: 1.0
    13 ZI: 0.0
    14 ZX: -1.0
    15 ZY: 0.0
    16 ZZ: 0.0

And the reverse circuit takes it back into the computational basis.

Here I was just going to be lazy and use the transpose of the superoperator gate that represents the circuit, but lets show they are the same. Reverse circuit - just apply inverse gates in reverse order one at a time (recall CNOT is self inverting, and three phases = phase$^\dagger$

```julia
Rcircuit = superHH*superIP*superIP*superIP * superCnot21*superPI*superPI*superPI*superCnot21
```

\begin{equation*} \left(\begin{array}{*{16}c} 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 \\ 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & -1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ \end{array}\right)\\\end{equation*}

```julia
Rcircuit *circuit
```

\begin{equation*} \left(\begin{array}{*{16}c} 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & 0.0 \\ 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 \\ \end{array}\right)\\\end{equation*}

```julia
# And just a little sanity check, that the transpose of superoperator is indeed its inverse
Rcircuit == transpose(circuit)
```

    true

### We can work out what stats we would get if we ran this experiment

(this is what we would see in an actual device -- assuming the only noise is our superOperatorNoise)

```julia
# The measurement outcomes are then:
print("M_00 = ↑↑ = $(round(0.25*zs[1]'*Rcircuit*superOperatorNoise*circuit*zs[1],digits=10))\n")
print("M_01 = ↑↓ = $(round(0.25*zs[2]'*Rcircuit*superOperatorNoise*circuit*zs[1],digits=10))\n")
print("M_10 = ↓↑ = $(round(0.25*zs[3]'*Rcircuit*superOperatorNoise*circuit*zs[1],digits=10))\n")
print("M_11 = ↓↓ = $(round(0.25*zs[4]'*Rcircuit*superOperatorNoise*circuit*zs[1],digits=10))\n")
outcomes = 0.25.*[x'*Rcircuit*superOperatorNoise*circuit*zs[1] for x in zs]
print("total probability = $(sum(outcomes))\n")
```

    M_00 = ↑↑ = 0.931075
    M_01 = ↑↓ = 0.030625
    M_10 = ↓↑ = 0.009025
    M_11 = ↓↓ = 0.029275
    total probability = 1.0

## Then we transform back to the eigenvalues:

```julia
round.(Hadamard.hadamard(4)*outcomes,digits=10)
```

    4-element Array{Float64,1}:
     1.0
     0.8802
     0.9234
     0.9207

```julia
# Recall the eigenvalues:
# Print label and probability
for i in 1:16
    print("$i: $(fidelityLabels(i-1)): $(basicSuperOpDiagonal[i])\n")
end
```

    1: II: 1.0
    2: IX: 0.978
    3: IY: 0.99
    4: IZ: 0.972
    5: XI: 0.93
    6: XX: 0.90954
    7: XY: 0.9207
    8: XZ: 0.90396
    9: YI: 0.95
    10: YX: 0.9291
    11: YY: 0.9405
    12: YZ: 0.9234
    13: ZI: 0.9
    14: ZX: 0.8802
    15: ZY: 0.891
    16: ZZ: 0.8748

And we can see we have correctly 'reconstructed'

- ZX
- YZ
- XY

As predicted.

We can change the circuits and reconstruct all of the eigenvalues, then we can generate the 'global' probability distribution. There are however otherways of doing this if we can assume e..g the distribution is sparse.

```julia

```