```python
import numpy as np
import matplotlib.pyplot as plt

from sympy.solvers import solve
from sympy import Symbol

from math import e, log
```

# 4.1 Functions

```python
def f(x):
    return x ** 2

print(f(2))
print(f(-3))
```

    4
    9

## 4.1.4 The Plot of a Function

```python
x = np.linspace(-10, 10, 1000)
x
```

    array([-10.        ,  -9.97997998,  -9.95995996,  -9.93993994,
            -9.91991992,  -9.8998999 ,  -9.87987988,  -9.85985986,
            -9.83983984,  -9.81981982,  -9.7997998 ,  -9.77977978,
            -9.75975976,  -9.73973974,  -9.71971972,  -9.6996997 ,
            -9.67967968,  -9.65965966,  -9.63963964,  -9.61961962,
            -9.5995996 ,  -9.57957958,  -9.55955956,  -9.53953954,
            -9.51951952,  -9.4994995 ,  -9.47947948,  -9.45945946,
            -9.43943944,  -9.41941942,  -9.3993994 ,  -9.37937938,
            -9.35935936,  -9.33933934,  -9.31931932,  -9.2992993 ,
            -9.27927928,  -9.25925926,  -9.23923924,  -9.21921922,
            -9.1991992 ,  -9.17917918,  -9.15915916,  -9.13913914,
            -9.11911912,  -9.0990991 ,  -9.07907908,  -9.05905906,
            -9.03903904,  -9.01901902,  -8.998999  ,  -8.97897898,
            -8.95895896,  -8.93893894,  -8.91891892,  -8.8988989 ,
            -8.87887888,  -8.85885886,  -8.83883884,  -8.81881882,
            -8.7987988 ,  -8.77877878,  -8.75875876,  -8.73873874,
            -8.71871872,  -8.6986987 ,  -8.67867868,  -8.65865866,
            -8.63863864,  -8.61861862,  -8.5985986 ,  -8.57857858,
            -8.55855856,  -8.53853854,  -8.51851852,  -8.4984985 ,
            -8.47847848,  -8.45845846,  -8.43843844,  -8.41841842,
            -8.3983984 ,  -8.37837838,  -8.35835836,  -8.33833834,
            -8.31831832,  -8.2982983 ,  -8.27827828,  -8.25825826,
            -8.23823824,  -8.21821822,  -8.1981982 ,  -8.17817818,
            -8.15815816,  -8.13813814,  -8.11811812,  -8.0980981 ,
            -8.07807808,  -8.05805806,  -8.03803804,  -8.01801802,
            -7.997998  ,  -7.97797798,  -7.95795796,  -7.93793794,
            -7.91791792,  -7.8978979 ,  -7.87787788,  -7.85785786,
            -7.83783784,  -7.81781782,  -7.7977978 ,  -7.77777778,
            -7.75775776,  -7.73773774,  -7.71771772,  -7.6976977 ,
            -7.67767768,  -7.65765766,  -7.63763764,  -7.61761762,
            -7.5975976 ,  -7.57757758,  -7.55755756,  -7.53753754,
            -7.51751752,  -7.4974975 ,  -7.47747748,  -7.45745746,
            -7.43743744,  -7.41741742,  -7.3973974 ,  -7.37737738,
            -7.35735736,  -7.33733734,  -7.31731732,  -7.2972973 ,
            -7.27727728,  -7.25725726,  -7.23723724,  -7.21721722,
            -7.1971972 ,  -7.17717718,  -7.15715716,  -7.13713714,
            -7.11711712,  -7.0970971 ,  -7.07707708,  -7.05705706,
            -7.03703704,  -7.01701702,  -6.996997  ,  -6.97697698,
            -6.95695696,  -6.93693694,  -6.91691692,  -6.8968969 ,
            -6.87687688,  -6.85685686,  -6.83683684,  -6.81681682,
            -6.7967968 ,  -6.77677678,  -6.75675676,  -6.73673674,
            -6.71671672,  -6.6966967 ,  -6.67667668,  -6.65665666,
            -6.63663664,  -6.61661662,  -6.5965966 ,  -6.57657658,
            -6.55655656,  -6.53653654,  -6.51651652,  -6.4964965 ,
            -6.47647648,  -6.45645646,  -6.43643644,  -6.41641642,
            -6.3963964 ,  -6.37637638,  -6.35635636,  -6.33633634,
            -6.31631632,  -6.2962963 ,  -6.27627628,  -6.25625626,
            -6.23623624,  -6.21621622,  -6.1961962 ,  -6.17617618,
            -6.15615616,  -6.13613614,  -6.11611612,  -6.0960961 ,
            -6.07607608,  -6.05605606,  -6.03603604,  -6.01601602,
            -5.995996  ,  -5.97597598,  -5.95595596,  -5.93593594,
            -5.91591592,  -5.8958959 ,  -5.87587588,  -5.85585586,
            -5.83583584,  -5.81581582,  -5.7957958 ,  -5.77577578,
            -5.75575576,  -5.73573574,  -5.71571572,  -5.6956957 ,
            -5.67567568,  -5.65565566,  -5.63563564,  -5.61561562,
            -5.5955956 ,  -5.57557558,  -5.55555556,  -5.53553554,
            -5.51551552,  -5.4954955 ,  -5.47547548,  -5.45545546,
            -5.43543544,  -5.41541542,  -5.3953954 ,  -5.37537538,
            -5.35535536,  -5.33533534,  -5.31531532,  -5.2952953 ,
            -5.27527528,  -5.25525526,  -5.23523524,  -5.21521522,
            -5.1951952 ,  -5.17517518,  -5.15515516,  -5.13513514,
            -5.11511512,  -5.0950951 ,  -5.07507508,  -5.05505506,
            -5.03503504,  -5.01501502,  -4.99499499,  -4.97497497,
            -4.95495495,  -4.93493493,  -4.91491491,  -4.89489489,
            -4.87487487,  -4.85485485,  -4.83483483,  -4.81481481,
            -4.79479479,  -4.77477477,  -4.75475475,  -4.73473473,
            -4.71471471,  -4.69469469,  -4.67467467,  -4.65465465,
            -4.63463463,  -4.61461461,  -4.59459459,  -4.57457457,
            -4.55455455,  -4.53453453,  -4.51451451,  -4.49449449,
            -4.47447447,  -4.45445445,  -4.43443443,  -4.41441441,
            -4.39439439,  -4.37437437,  -4.35435435,  -4.33433433,
            -4.31431431,  -4.29429429,  -4.27427427,  -4.25425425,
            -4.23423423,  -4.21421421,  -4.19419419,  -4.17417417,
            -4.15415415,  -4.13413413,  -4.11411411,  -4.09409409,
            -4.07407407,  -4.05405405,  -4.03403403,  -4.01401401,
            -3.99399399,  -3.97397397,  -3.95395395,  -3.93393393,
            -3.91391391,  -3.89389389,  -3.87387387,  -3.85385385,
            -3.83383383,  -3.81381381,  -3.79379379,  -3.77377377,
            -3.75375375,  -3.73373373,  -3.71371371,  -3.69369369,
            -3.67367367,  -3.65365365,  -3.63363363,  -3.61361361,
            -3.59359359,  -3.57357357,  -3.55355355,  -3.53353353,
            -3.51351351,  -3.49349349,  -3.47347347,  -3.45345345,
            -3.43343343,  -3.41341341,  -3.39339339,  -3.37337337,
            -3.35335335,  -3.33333333,  -3.31331331,  -3.29329329,
            -3.27327327,  -3.25325325,  -3.23323323,  -3.21321321,
            -3.19319319,  -3.17317317,  -3.15315315,  -3.13313313,
            -3.11311311,  -3.09309309,  -3.07307307,  -3.05305305,
            -3.03303303,  -3.01301301,  -2.99299299,  -2.97297297,
            -2.95295295,  -2.93293293,  -2.91291291,  -2.89289289,
            -2.87287287,  -2.85285285,  -2.83283283,  -2.81281281,
            -2.79279279,  -2.77277277,  -2.75275275,  -2.73273273,
            -2.71271271,  -2.69269269,  -2.67267267,  -2.65265265,
            -2.63263263,  -2.61261261,  -2.59259259,  -2.57257257,
            -2.55255255,  -2.53253253,  -2.51251251,  -2.49249249,
            -2.47247247,  -2.45245245,  -2.43243243,  -2.41241241,
            -2.39239239,  -2.37237237,  -2.35235235,  -2.33233233,
            -2.31231231,  -2.29229229,  -2.27227227,  -2.25225225,
            -2.23223223,  -2.21221221,  -2.19219219,  -2.17217217,
            -2.15215215,  -2.13213213,  -2.11211211,  -2.09209209,
            -2.07207207,  -2.05205205,  -2.03203203,  -2.01201201,
            -1.99199199,  -1.97197197,  -1.95195195,  -1.93193193,
            -1.91191191,  -1.89189189,  -1.87187187,  -1.85185185,
            -1.83183183,  -1.81181181,  -1.79179179,  -1.77177177,
            -1.75175175,  -1.73173173,  -1.71171171,  -1.69169169,
            -1.67167167,  -1.65165165,  -1.63163163,  -1.61161161,
            -1.59159159,  -1.57157157,  -1.55155155,  -1.53153153,
            -1.51151151,  -1.49149149,  -1.47147147,  -1.45145145,
            -1.43143143,  -1.41141141,  -1.39139139,  -1.37137137,
            -1.35135135,  -1.33133133,  -1.31131131,  -1.29129129,
            -1.27127127,  -1.25125125,  -1.23123123,  -1.21121121,
            -1.19119119,  -1.17117117,  -1.15115115,  -1.13113113,
            -1.11111111,  -1.09109109,  -1.07107107,  -1.05105105,
            -1.03103103,  -1.01101101,  -0.99099099,  -0.97097097,
            -0.95095095,  -0.93093093,  -0.91091091,  -0.89089089,
            -0.87087087,  -0.85085085,  -0.83083083,  -0.81081081,
            -0.79079079,  -0.77077077,  -0.75075075,  -0.73073073,
            -0.71071071,  -0.69069069,  -0.67067067,  -0.65065065,
            -0.63063063,  -0.61061061,  -0.59059059,  -0.57057057,
            -0.55055055,  -0.53053053,  -0.51051051,  -0.49049049,
            -0.47047047,  -0.45045045,  -0.43043043,  -0.41041041,
            -0.39039039,  -0.37037037,  -0.35035035,  -0.33033033,
            -0.31031031,  -0.29029029,  -0.27027027,  -0.25025025,
            -0.23023023,  -0.21021021,  -0.19019019,  -0.17017017,
            -0.15015015,  -0.13013013,  -0.11011011,  -0.09009009,
            -0.07007007,  -0.05005005,  -0.03003003,  -0.01001001,
             0.01001001,   0.03003003,   0.05005005,   0.07007007,
             0.09009009,   0.11011011,   0.13013013,   0.15015015,
             0.17017017,   0.19019019,   0.21021021,   0.23023023,
             0.25025025,   0.27027027,   0.29029029,   0.31031031,
             0.33033033,   0.35035035,   0.37037037,   0.39039039,
             0.41041041,   0.43043043,   0.45045045,   0.47047047,
             0.49049049,   0.51051051,   0.53053053,   0.55055055,
             0.57057057,   0.59059059,   0.61061061,   0.63063063,
             0.65065065,   0.67067067,   0.69069069,   0.71071071,
             0.73073073,   0.75075075,   0.77077077,   0.79079079,
             0.81081081,   0.83083083,   0.85085085,   0.87087087,
             0.89089089,   0.91091091,   0.93093093,   0.95095095,
             0.97097097,   0.99099099,   1.01101101,   1.03103103,
             1.05105105,   1.07107107,   1.09109109,   1.11111111,
             1.13113113,   1.15115115,   1.17117117,   1.19119119,
             1.21121121,   1.23123123,   1.25125125,   1.27127127,
             1.29129129,   1.31131131,   1.33133133,   1.35135135,
             1.37137137,   1.39139139,   1.41141141,   1.43143143,
             1.45145145,   1.47147147,   1.49149149,   1.51151151,
             1.53153153,   1.55155155,   1.57157157,   1.59159159,
             1.61161161,   1.63163163,   1.65165165,   1.67167167,
             1.69169169,   1.71171171,   1.73173173,   1.75175175,
             1.77177177,   1.79179179,   1.81181181,   1.83183183,
             1.85185185,   1.87187187,   1.89189189,   1.91191191,
             1.93193193,   1.95195195,   1.97197197,   1.99199199,
             2.01201201,   2.03203203,   2.05205205,   2.07207207,
             2.09209209,   2.11211211,   2.13213213,   2.15215215,
             2.17217217,   2.19219219,   2.21221221,   2.23223223,
             2.25225225,   2.27227227,   2.29229229,   2.31231231,
             2.33233233,   2.35235235,   2.37237237,   2.39239239,
             2.41241241,   2.43243243,   2.45245245,   2.47247247,
             2.49249249,   2.51251251,   2.53253253,   2.55255255,
             2.57257257,   2.59259259,   2.61261261,   2.63263263,
             2.65265265,   2.67267267,   2.69269269,   2.71271271,
             2.73273273,   2.75275275,   2.77277277,   2.79279279,
             2.81281281,   2.83283283,   2.85285285,   2.87287287,
             2.89289289,   2.91291291,   2.93293293,   2.95295295,
             2.97297297,   2.99299299,   3.01301301,   3.03303303,
             3.05305305,   3.07307307,   3.09309309,   3.11311311,
             3.13313313,   3.15315315,   3.17317317,   3.19319319,
             3.21321321,   3.23323323,   3.25325325,   3.27327327,
             3.29329329,   3.31331331,   3.33333333,   3.35335335,
             3.37337337,   3.39339339,   3.41341341,   3.43343343,
             3.45345345,   3.47347347,   3.49349349,   3.51351351,
             3.53353353,   3.55355355,   3.57357357,   3.59359359,
             3.61361361,   3.63363363,   3.65365365,   3.67367367,
             3.69369369,   3.71371371,   3.73373373,   3.75375375,
             3.77377377,   3.79379379,   3.81381381,   3.83383383,
             3.85385385,   3.87387387,   3.89389389,   3.91391391,
             3.93393393,   3.95395395,   3.97397397,   3.99399399,
             4.01401401,   4.03403403,   4.05405405,   4.07407407,
             4.09409409,   4.11411411,   4.13413413,   4.15415415,
             4.17417417,   4.19419419,   4.21421421,   4.23423423,
             4.25425425,   4.27427427,   4.29429429,   4.31431431,
             4.33433433,   4.35435435,   4.37437437,   4.39439439,
             4.41441441,   4.43443443,   4.45445445,   4.47447447,
             4.49449449,   4.51451451,   4.53453453,   4.55455455,
             4.57457457,   4.59459459,   4.61461461,   4.63463463,
             4.65465465,   4.67467467,   4.69469469,   4.71471471,
             4.73473473,   4.75475475,   4.77477477,   4.79479479,
             4.81481481,   4.83483483,   4.85485485,   4.87487487,
             4.89489489,   4.91491491,   4.93493493,   4.95495495,
             4.97497497,   4.99499499,   5.01501502,   5.03503504,
             5.05505506,   5.07507508,   5.0950951 ,   5.11511512,
             5.13513514,   5.15515516,   5.17517518,   5.1951952 ,
             5.21521522,   5.23523524,   5.25525526,   5.27527528,
             5.2952953 ,   5.31531532,   5.33533534,   5.35535536,
             5.37537538,   5.3953954 ,   5.41541542,   5.43543544,
             5.45545546,   5.47547548,   5.4954955 ,   5.51551552,
             5.53553554,   5.55555556,   5.57557558,   5.5955956 ,
             5.61561562,   5.63563564,   5.65565566,   5.67567568,
             5.6956957 ,   5.71571572,   5.73573574,   5.75575576,
             5.77577578,   5.7957958 ,   5.81581582,   5.83583584,
             5.85585586,   5.87587588,   5.8958959 ,   5.91591592,
             5.93593594,   5.95595596,   5.97597598,   5.995996  ,
             6.01601602,   6.03603604,   6.05605606,   6.07607608,
             6.0960961 ,   6.11611612,   6.13613614,   6.15615616,
             6.17617618,   6.1961962 ,   6.21621622,   6.23623624,
             6.25625626,   6.27627628,   6.2962963 ,   6.31631632,
             6.33633634,   6.35635636,   6.37637638,   6.3963964 ,
             6.41641642,   6.43643644,   6.45645646,   6.47647648,
             6.4964965 ,   6.51651652,   6.53653654,   6.55655656,
             6.57657658,   6.5965966 ,   6.61661662,   6.63663664,
             6.65665666,   6.67667668,   6.6966967 ,   6.71671672,
             6.73673674,   6.75675676,   6.77677678,   6.7967968 ,
             6.81681682,   6.83683684,   6.85685686,   6.87687688,
             6.8968969 ,   6.91691692,   6.93693694,   6.95695696,
             6.97697698,   6.996997  ,   7.01701702,   7.03703704,
             7.05705706,   7.07707708,   7.0970971 ,   7.11711712,
             7.13713714,   7.15715716,   7.17717718,   7.1971972 ,
             7.21721722,   7.23723724,   7.25725726,   7.27727728,
             7.2972973 ,   7.31731732,   7.33733734,   7.35735736,
             7.37737738,   7.3973974 ,   7.41741742,   7.43743744,
             7.45745746,   7.47747748,   7.4974975 ,   7.51751752,
             7.53753754,   7.55755756,   7.57757758,   7.5975976 ,
             7.61761762,   7.63763764,   7.65765766,   7.67767768,
             7.6976977 ,   7.71771772,   7.73773774,   7.75775776,
             7.77777778,   7.7977978 ,   7.81781782,   7.83783784,
             7.85785786,   7.87787788,   7.8978979 ,   7.91791792,
             7.93793794,   7.95795796,   7.97797798,   7.997998  ,
             8.01801802,   8.03803804,   8.05805806,   8.07807808,
             8.0980981 ,   8.11811812,   8.13813814,   8.15815816,
             8.17817818,   8.1981982 ,   8.21821822,   8.23823824,
             8.25825826,   8.27827828,   8.2982983 ,   8.31831832,
             8.33833834,   8.35835836,   8.37837838,   8.3983984 ,
             8.41841842,   8.43843844,   8.45845846,   8.47847848,
             8.4984985 ,   8.51851852,   8.53853854,   8.55855856,
             8.57857858,   8.5985986 ,   8.61861862,   8.63863864,
             8.65865866,   8.67867868,   8.6986987 ,   8.71871872,
             8.73873874,   8.75875876,   8.77877878,   8.7987988 ,
             8.81881882,   8.83883884,   8.85885886,   8.87887888,
             8.8988989 ,   8.91891892,   8.93893894,   8.95895896,
             8.97897898,   8.998999  ,   9.01901902,   9.03903904,
             9.05905906,   9.07907908,   9.0990991 ,   9.11911912,
             9.13913914,   9.15915916,   9.17917918,   9.1991992 ,
             9.21921922,   9.23923924,   9.25925926,   9.27927928,
             9.2992993 ,   9.31931932,   9.33933934,   9.35935936,
             9.37937938,   9.3993994 ,   9.41941942,   9.43943944,
             9.45945946,   9.47947948,   9.4994995 ,   9.51951952,
             9.53953954,   9.55955956,   9.57957958,   9.5995996 ,
             9.61961962,   9.63963964,   9.65965966,   9.67967968,
             9.6996997 ,   9.71971972,   9.73973974,   9.75975976,
             9.77977978,   9.7997998 ,   9.81981982,   9.83983984,
             9.85985986,   9.87987988,   9.8998999 ,   9.91991992,
             9.93993994,   9.95995996,   9.97997998,  10.        ])

```python
plt.plot(x, x + 1)
plt.show()
```

# 4.3 Equations

## 4.3.3 Using Python

```python
x = Symbol('x')
x
```

$\displaystyle x$

```python
solve(x ** 3 - 7 * x ** 2 + 15 * x - 9, x)
```

    [1, 3]

```python
solve(3 * e ** (x + 2) + 3 - 2 * (e ** x + 100), x)
```

    [2.27914777845756]

```python
log(197 / (3 * e ** 2 - 2))
```

    2.279147778457572

# 4.4 Systems of Equations

# 4.4.2 Systems of Non-Linear Equations

```python
x = Symbol('x')
y = Symbol('y')

solve(
    [x + y - 5, x ** 2 - x + 2 * y - 8],
    x, y
)
```

    [(1, 4), (2, 3)]

```python
solve(
    [x ** 2 - 2 * x - y ** 2 + 1, x ** 2 - 2 * y - 1],
    x, y
)
```

    [(-3, 4), (1, 0)]

```python

```


```
from printer import  Format
from ga import Ga
from sympy import symbols
```

```
Format(ipy=True)
```

```
coords = (x,y,z) = symbols('x,y,z',real=True)
(o3d,ex,ey,ez) = Ga.build('e_x e_y e_z',g=[1,1,1],coords=coords)
```

```
v = o3d.mv('v','vector')
```

```
v
```

```
V = o3d.mv('V','vector',f=True)
```

```
V
```

```
gradV = o3d.grad*V
```

```
gradV
```

\begin{equation*} \left ( \partial_{x} V^{x}  + \partial_{y} V^{y}  + \partial_{z} V^{z} \right )  + \left ( - \partial_{y} V^{x}  + \partial_{x} V^{y} \right ) \boldsymbol{e_{x}\wedge e_{y}} + \left ( - \partial_{z} V^{x}  + \partial_{x} V^{z} \right ) \boldsymbol{e_{x}\wedge e_{z}} + \left ( - \partial_{z} V^{y}  + \partial_{y} V^{z} \right ) \boldsymbol{e_{y}\wedge e_{z}} \end{equation*}

```

```

```
lap = o3d.grad|o3d.grad
```

```
lap
```

    <mv.Dop at 0x7f5a659c6a90>

```
A = o3d.lt('A')
```

```
A
```

    <lt.Lt at 0x7f5a659c6bd0>

```

```


# Playing around

```python
from IPython.display import Math
from sympy import *
XS = range(0, 1000)
x = Symbol('x')
f = 2*x + 5
f_real = lambdify(x, f, 'numpy')
f
```

$\displaystyle 2 x + 5$

```python
from matplotlib.pyplot import plot, show
ys = [f_real(x) for x in XS]
%matplotlib inline
plot(XS, ys)
show()
```

```latex
%%latex
\begin{align}
\int_{-\infty}^{+\infty} 2x + 5 \ dx = x^2 + 5x + C
\end{align}
```

\begin{align}
\int 2x + 5 \ dx = x^2 + 5x + C
\end{align}

```python
from matplotlib.pyplot import plot, show
fint = integrate(f)
fint_real = lambdify(x, fint, 'numpy')
ys = [fint_real(x) for x in XS]
%matplotlib inline
plot(XS, ys)
show()
```


# Path Planning for a Simple Car

The following equations describe a simple model of a car

\begin{align}
\frac{dx}{dt} & = u \cos(\theta) \\
\frac{dy}{dt} & = u \sin(\theta) \\
\frac{d\theta}{dt} & = \frac{u}{L}\tan(\phi) \\
\end{align}

where $x$ and $y$ denote the position of the center of the rear axle, $\theta$ is the angle of the car axis to the horizontal, and $phi$ is the angle of the front steering wheels to the car axis. The length $L$ is the distance from the center of the rear axle to the center of the front axle.

The velocity $u$ is controlled by acceleration of the car, the position of the wheels is controlled by the rate limited steering input $v$.

\begin{align}
\frac{du}{dt} & = a \\
\frac{d\phi}{dt} & = v
\end{align}

The state of the car is determined by the value of the five state variables $x$, $y$, $\theta$, $u$, and $\phi$.

The path planning problem is to find find values of the functions $a(t)$ and $v(t)$ on a time interval $0 \leq t \leq t_f$ to steer the car from an initial condition $\left[x(0), y(0), \theta(0), u(0), \phi(0)\right]$ to a specified final condition $\left[x(t_f), y(t_f), \theta(t_f), u(t_f), \phi(t_f)\right]$ that minimizes an objective function:

\begin{align}
J  = \min \int_0^{t_f} \left( \phi(t)^2 + \alpha a(t)^2 + \beta v(t)^2\right)\,dt
\end{align}

```
!pip install -q pyomo
!wget -N -q "https://ampl.com/dl/open/ipopt/ipopt-linux64.zip"
!unzip -o -q ipopt-linux64

ipopt_executable = '/content/ipopt'
```

## Pyomo Model

```
from pyomo.environ import *
from pyomo.dae import *

L = 2
tf = 50

# create a model object
m = ConcreteModel()

# define the independent variable
m.time = ContinuousSet(bounds=(0, tf))

# define control inputs
m.a = Var(m.time)
m.v = Var(m.time, domain=Reals, bounds=(-0.1,0.1))

# define the dependent variables
m.x = Var(m.time)
m.y = Var(m.time)
m.t = Var(m.time)
m.u = Var(m.time)
m.p = Var(m.time, domain=Reals, bounds=(-0.5,0.5))

m.dxdt = DerivativeVar(m.x)
m.dydt = DerivativeVar(m.y)
m.dtdt = DerivativeVar(m.t)
m.dudt = DerivativeVar(m.u)
m.dpdt = DerivativeVar(m.p)

# define the differential equation as a constraint
m.ode_x = Constraint(m.time, rule=lambda m, time: m.dxdt[time] == m.u[time]*cos(m.t[time]))
m.ode_y = Constraint(m.time, rule=lambda m, time: m.dydt[time] == m.u[time]*sin(m.t[time]))
m.ode_t = Constraint(m.time, rule=lambda m, time: m.dtdt[time] == m.u[time]*tan(m.p[time])/L)
m.ode_u = Constraint(m.time, rule=lambda m, time: m.dudt[time] == m.a[time])
m.ode_p = Constraint(m.time, rule=lambda m, time: m.dpdt[time] == m.v[time])

# path constraints
m.path_x1 = Constraint(m.time, rule=lambda m, time: m.x[time] >= 0)
m.path_y1 = Constraint(m.time, rule=lambda m, time: m.y[time] >= 0)

# initial conditions
m.ic = ConstraintList()
m.ic.add(m.x[0]==0)
m.ic.add(m.y[0]==0)
m.ic.add(m.t[0]==0)
m.ic.add(m.u[0]==0)
m.ic.add(m.p[0]==0)

# final conditions
m.fc = ConstraintList()
m.fc.add(m.x[tf]==0)
m.fc.add(m.y[tf]==20)
m.fc.add(m.t[tf]==0)
m.fc.add(m.u[tf]==0)
m.fc.add(m.p[tf]==0)

# define the optimization objective
m.integral = Integral(m.time, wrt=m.time,
                      rule=lambda m, time: 0.2*m.p[time]**2 + m.a[time]**2 + m.v[time]**2)
m.obj = Objective(expr=m.integral)

# transform and solve
TransformationFactory('dae.collocation').apply_to(m, wrt=m.time, nfe=3, ncp=12, method='BACKWARD')
solver = SolverFactory('ipopt', executable=ipopt_executable)
solver.solve(m).write()
```

    # ==========================================================
    # = Solver Results                                         =
    # ==========================================================
    # ----------------------------------------------------------
    #   Problem Information
    # ----------------------------------------------------------
    Problem:
    - Lower bound: -inf
      Upper bound: inf
      Number of objectives: 1
      Number of constraints: 449
      Number of variables: 444
      Sense: unknown
    # ----------------------------------------------------------
    #   Solver Information
    # ----------------------------------------------------------
    Solver:
    - Status: ok
      Message: Ipopt 3.12.8\x3a Optimal Solution Found
      Termination condition: optimal
      Id: 0
      Error rc: 0
      Time: 1.157731294631958
    # ----------------------------------------------------------
    #   Solution Information
    # ----------------------------------------------------------
    Solution:
    - number of solutions: 0
      number of solutions displayed: 0

## Accessing Solution Data

```
# access the results
time = [time for time in m.time]

a = [m.a[time]() for time in m.time]
v = [m.v[time]() for time in m.time]

x = [m.x[time]() for time in m.time]
y = [m.y[time]() for time in m.time]
t = [m.t[time]() for time in m.time]
u = [m.u[time]() for time in m.time]
p = [m.p[time]() for time in m.time]
```

## Visualizing Car Path

```
% matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

scl=0.3

def draw_car(x=0, y=0, theta=0, phi=0):
    R = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
    car = np.array([[0.2, 0.5], [-0.2, 0.5], [0, 0.5], [0, -0.5],
                    [0.2, -0.5], [-0.2, -0.5], [0, -0.5], [0, 0], [L, 0], [L, 0.5],
                    [L + 0.2*np.cos(phi), 0.5 + 0.2*np.sin(phi)],
                    [L - 0.2*np.cos(phi), 0.5 - 0.2*np.sin(phi)], [L, 0.5],[L, -0.5],
                    [L + 0.2*np.cos(phi), -0.5 + 0.2*np.sin(phi)],
                    [L - 0.2*np.cos(phi), -0.5 - 0.2*np.sin(phi)]])
    carz= scl*R.dot(car.T)
    plt.plot(x + carz[0], y + carz[1], 'k', lw=2)
    plt.plot(x, y, 'k.', ms=10)

plt.figure(figsize=(10,10))
for xs,ys,ts,ps in zip(x,y,t,p):
    draw_car(xs, ys, ts, scl*ps)
plt.plot(x, y, 'r--', lw=0.8)
plt.axis('square')
```

```
plt.figure(figsize=(10,8))
plt.subplot(311)
plt.plot(time, a, time, v)
plt.legend(['Acceleration','Steering Input'])

plt.subplot(312)
plt.plot(time, p, time, t)
plt.legend(['Wheel Position','Car Direction'])

plt.subplot(313)
plt.plot(time, u)
plt.legend(['Velocity'])
```

```

```


# Exercises

This chapter is an intermezzo that allows us to check and have a deeper understanding of the concepts seen so far by means of exercises. We will see how the code shown can be rewritten to take advantage of battle-tested solutions and idioms that emerges from daily practice.

First of all, we import some modules (be free to skim the corresponding documentation for each one of them),

```python
import functools, operator, math, itertools, random, collections, statistics, bisect, operator, heapq
```

that contains useful definitions for the code that we are going to write. Moreover, an utility for generators,

```python
def take(iterable, n):
    return map(lambda p: p[1], zip(range(n), iterable))
```

that consumes an iterable and return a generator that will yield $n$ objects at most. For the sake of clarity,

```python
taken = take(itertools.count(), 50)
taken
```

    <generator object take at 0x7f9dbe3b65f0>

is a actually generator and its content equals

```python
assert list(taken) == list(range(50))
```

Before starting, we initialize the random generator with a nice prime

```python
random.seed(11)
```

## Intersection

```python
A = list(range(10000))
B = list(range(10000))

random.shuffle(A)
random.shuffle(B)
```

```python
def intersection(A, B):

    B = set(B)
    return (a for a in A if a in B)
```

```python
%timeit list(intersection(A, B))
```

    1.73 ms ± 48.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%timeit list(zip(A, set(B)))
```

    1.44 ms ± 206 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
def intersection(A, B):

    A, B = iter(sorted(A)), iter(sorted(B))

    a, b = next(A), next(B)

    while True:

        try:
            if a == b:
                yield a
                a, b = next(A), next(B)
            elif a < b:
                a = next(A)
            else:
                b = next(B)
        except StopIteration:
            break
```

```python
%timeit list(intersection(A, B))
```

    6.47 ms ± 1.09 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

## (Pythagorean) tuples

Let

```python
def tuples(*slices):
    return itertools.product(*map(lambda s: range(s.start, s.stop), slices))
```

**INTERMEZZO**

```python
def A(a, b, c, d):
    pass
```

```python
def A(*args):
    return list(map(lambda i: i + 4, args))
```

```python
def AA(args):
    return list(map(lambda i: i + 4, args))
```

```python
def B(a, b, *args):
    return [a, b] + list(map(lambda i: i + 4, args))
```

```python
A(1, 2, 3)
```

```python
A([1, 2, 3])
```

```python
AA([1, 2, 3])
```

```python
B(1,)
```

```python
B(1, 2)
```

```python
B(1, 2, 3)
```

```python
A()
```

```python
A(1, 2, 3)
```

```python
A(1, 2, 3, 4, 5, 6, 7)
```

```python
container = range(5)
A( *container  )
```

---

where

```python
help(itertools.product)
```

Consider the application to an empty sequence of `slide`s,

```python
units = tuples()
units
```

then saturate it

```python
list(units)
```

Now, build tuples using just a `slide` object,

```python
singletons = tuples(slice(5, 11))
singletons
```

then saturate it

```python
list(singletons)
```

Now, build tuples using a twin `slide` object,

```python
s = slice(5, 11)
pairs = tuples(s, s)
pairs
```

then saturate it

```python
list(pairs)
```

Now, build tuples using a three different `slide` objects (taking into account of splitting the returned generator),

```python
triples_a, triples_b = itertools.tee(tuples(slice(5, 11), slice(6, 13), slice(7, 14)))
```

where

```python
help(itertools.tee)
```

then saturate it

```python
list(triples_a)
```

Now a corner case, but still interesting for ensuring a sound behavior,

```python
triples = tuples(slice(5, 11), slice(6, 6), slice(7, 14)) # ouch!
```

```python
L = [1, 2, 3, 4]
L[2:2]
```

```python
L[slice(2, 2)]
```

then saturate it

```python
list(triples) # who we have to blame?
```

Finally, let

```python
type(True)
```

```python
def is_pythagorean(tup: tuple, n=2) -> bool: # is_pythagorean is a *predicate*
    '''A Pythagorean triple consists of three positive integers a, b, and c, such that a^2 + b^2 = c^2.

    Such a triple is commonly written (a, b, c), and a well-known example is (3, 4, 5).
    If (a, b, c) is a Pythagorean triple, then so is (ka, kb, kc) for any positive integer k.

    A primitive Pythagorean triple is one in which a, b and c are coprime (that is,
    they have no common divisor larger than 1).

    See also https://en.wikipedia.org/wiki/Pythagorean_triple.
    '''
    a, b, c = tup # tuple unpacking
    return (a**n + b**n == c**n) if a <= b <= c else False
```

in

```python
filter(is_pythagorean, triples_b)
```

```python
list(filter(is_pythagorean, triples_b)) # do a selection
```

and

```python
help(is_pythagorean) # just to show that writing docstrings is cool and useful.
```

## `sum_upto`

Let

```python
def sum_upto(n):
    return functools.reduce(operator.add, range(n+1))
```

and test according to Euler's quicker formula

```python
n = 100
v = sum_upto(n)
gauss = (n*(n+1)/2)
assert v == gauss == 5050
```

where

```python
help(functools.reduce)
```

and

```python
help(operator.add)
```

## `sqrt`

Let

```python
def sqrt(n):

    refined = n
    while True:
        yield refined
        refined = (n/refined + refined)/2
```

to enumerate 15 approximation of the square root of 37

```python
n = 37
list(take(sqrt(37), 15))
```

and check with respect to

```python
math.sqrt(n)
```

where

```python
help(math.sqrt)
```

    Help on built-in function sqrt in module math:

    sqrt(x, /)
        Return the square root of x.

## $\pi$

According to https://en.wikipedia.org/wiki/Leibniz_formula_for_%CF%80, let

```python
def pi_Leibniz():

    d = 0
    for i, coeff in enumerate(itertools.count(1, step=2)):
        yield 4*d
        d += (-1)**i/coeff
```

in

```python
list(take(pi_Leibniz(), 1000))[-10:]
```

and check against the

```python
math.pi
```

where

```python
help(itertools.count)
```

## The Collatz's conjecture

Consider the following operation on an arbitrary positive integer:

    If the number is even, divide it by two.
    If the number is odd, triple it and add one.

See also https://en.wikipedia.org/wiki/Collatz_conjecture. Let

```python
def collatz(n):

    while True:

        yield n
        n = 3*n + 1 if n % 2 else n // 2 # be aware that we lose track of the original `n`!
```

in

```python
[list(take(collatz(n), 15)) for n in range(1, 20)]
```

## Fibonacci numbers

Directly from https://docs.python.org/3/library/functools.html#functools.cache:

```python
@functools.lru_cache()
def factorial(n):
    print('•', end='')
    return n * factorial(n-1) if n else 1
```

no previously cached result, makes 11 recursive calls (count the • symbols)

```python
factorial(10)
```

just looks up cached value result

```python
factorial(5)
```

makes two new recursive calls, the other 10 are cached

```python
factorial(12)
```

## Uniform `random` on segmented interval

The problem here reads as follow: sample uniformly from $[a, b)$ and $[c, d)$ where $b <= c$. <br>Eventually, try to generate to an arbitrary sequence of `slice`s, assuming they are fed in sorted order with respect to `<`.

```python
help(random.random)
```

    Help on built-in function random:

    random() method of random.Random instance
        random() -> x in the interval [0, 1).

```python
def samples(*slices):

    step = 1/len(slices)

    steps = itertools.count(step, step)
    bins = [(s, sl) for sl, s in zip(slices, steps)]

    while True:
        r = random.random()
        i = bisect.bisect_left(bins, (r, None))
        sl = slices[i]
        yield abs(sl.stop - sl.start) * (r - (i*step))/step + sl.start
```

```python
samples(slice(10, 20), slice(35, 40))
```

    <generator object samples at 0x7f94b236eba0>

Then define the generator with respect to $[10, 20)$ and $[35, 40)$

```python
observations = take(samples(slice(10, 20), slice(35, 40)), 1_000_000)
observations
```

    <generator object take at 0x7f94b2336970>

have a look at some observations

```python
sorted([i for _, i in zip(range(100), observations)])
```

then observe the quantiles:

```python
statistics.quantiles(observations)
```

it looks uniform. By the way, use different intervals, $[14, 20)$ and $[35,40)$,

```python
observations = take(samples(slice(14, 20), slice(35, 40)), 1_000_000)
```

look again at some observations,

```python
sorted([i for _, i in zip(range(100), observations)])
```

and check the corresponding quantiles

```python
statistics.quantiles(observations)
```

it should be uniform too. Finally, we test the corner case where $b=c$, so let $[10, 20)$ and $[20,40)$,

```python
observations = take(samples(slice(10, 20), slice(20, 40)), 1_000_000)
```

look again at some observations,

```python
sorted([i for _, i in zip(range(100), observations)])
```

and check the corresponding quantiles

```python
statistics.quantiles(observations)
```

it should be uniform either. Finally, attempt a sampling from `4` slices,

```python
observations = take(samples(slice(0, 5), slice(10, 15), slice(20, 25), slice(30, 35)), 1_000_000)
```

look again at some observations,

```python
sorted([i for _, i in zip(range(100), observations)])
```

and check the corresponding quantiles

```python
statistics.quantiles(observations)
```

it should be uniform either.

## Bernoulli random variable

```python
int(True) # this is a very quick check to see if a Boolean can be used as integer
```

```python
def Bernoulli(p):
    'This is a generator for a Bernoulli random variable of parameter `p` for success.'

    while True:              # forever we loop
        r = random.random()  # get a sample
        yield int(r < p)     # if that sample denotes a success or a failure we *yield* that outcome
```

```python
B = Bernoulli(p=0.6) # B is our random variable
B
```

```python
next(B)
```

```python
next(B)
```

```python
next(B)
```

```python
next(B)
```

```python
list(take(B, 20))
```

```python
C = collections.Counter(take(B, 1_000_000))
C
```

```python
C[1]/(C[0]+C[1])
```

where

```python
print(collections.Counter.__doc__)
```

## Russian Peasant Multiplication

Let

```python
def halves_doubles(n, m):
    halving = n
    doubling = m
    acc = 0
    while halving:
        digit = halving % 2
        acc = acc + digit * doubling
        yield (digit, halving, doubling, acc)
        halving = halving >> 1 # int(halving / 2)
        doubling = doubling << 1
```

in

```python
list(halves_doubles(89, 18))
```

    [(1, 89, 18, 18),
     (0, 44, 36, 18),
     (0, 22, 72, 18),
     (1, 11, 144, 162),
     (1, 5, 288, 450),
     (0, 2, 576, 450),
     (1, 1, 1152, 1602)]

see https://en.wikipedia.org/wiki/Ancient_Egyptian_multiplication and also https://www.cut-the-knot.org/Curriculum/Algebra/PeasantMultiplication.shtml. Then,

```python
def rpm(n, m):
    *prefix, (b, h, d, s) = halves_doubles(n, m)
    return s
```

so the check passes,

```python
assert rpm(89, 18) == 89 * 18 == 1602
```

because

```python
bin(89)
```

    '0b1011001'

Of course, it works too when the first number is even,

```python
rpm(6, 100)
```

    600

Of course our implementation

```python
%timeit rpm(293819385789379687596845, 921038209831568476843584365)
```

    33.2 µs ± 111 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

is *slower* than the primitive one

```python
%timeit 293819385789379687596845 * 921038209831568476843584365
```

    98.8 ns ± 0.164 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

because arithmetic is performed in the virtual machine.

Let us give a strict version also,

```python
def rpm_strict(n, m):
    halving = n
    doubling = m
    acc = 0
    while halving:
        digit = halving % 2
        acc = acc + digit * doubling
        halving = halving >> 1
        doubling = doubling << 1
    return acc
```

check that it is correct,

```python
rpm_strict(89, 18)
```

and observe that it is a little bit *faster* than our former implementation

```python
%timeit rpm_strict(293819385789379687596845, 921038209831568476843584365)
```

## Fixed sum

```python
def subarrays(L):
   return (L[i:j] for i in range(len(L)) for j in range(i, len(L)+1))
```

```python
L = [-1, 5, 8, -9, 4, 1]
```

```python
list(subarrays(L))
```

```python
def fixed_sum(L, n):
    return filter(lambda s: sum(s)==n, subarrays(L))
```

```python
list(fixed_sum(L, 10))
```

```python
def partial_sums(L):
    g = itertools.accumulate(subarrays(L), lambda s, each: s + each[-1] if each else 0, initial=0)
    next(g) # to ignore the initial 0 given above
    return g
```

```python
list(partial_sums(L))
```

Toward an optimization...

```python
def subarrays_rev(L):
   return (tuple(L[i:j]) for i in range(len(L)-1, -1, -1) for j in range(i+1, len(L)+1))
```

```python
list(subarrays_rev(L))
```

```python
def fixed_sum_rev(L, n, cache={}):
    for tup in subarrays_rev(L):
        rest = tup[1:]
        s = tup[0] + cache.get(rest, 0)
        cache[tup] = s
        if s == n: yield tup
```

```python
cache = {}
list(fixed_sum_rev(L, 10, cache))
```

```python
cache # have a look at the collected values
```

```python
def sample(n):
    O, b, *rest = bin(random.getrandbits(n)) # because `string`s are iterable objects indeed.
    return list(map(int, rest))
```

where

```python
help(random.getrandbits)
```

```python
LL = sample(1000)
```

```python
assert set(map(tuple, fixed_sum(LL, 10))) == set(fixed_sum_rev(LL, 10))
```

```python
%timeit list(fixed_sum(LL, 10))
```

```python
%timeit list(fixed_sum_rev(LL, 10))
```

**INTERMEZZO**

```python
if 4 < 8:
    print('a')
else:
    pass
```

```python
b = if 4 < 8:
       '''

       lots of code

       '''
    else:
       6
```

```python
b = 5 if 4 < 8 else 6
```

```python
b
```

## Some strange uses of recursion

For more on this recursion schemata see https://www.cs.ox.ac.uk/people/ralf.hinze/publications/ICFP09.pdf and also https://www.sciencedirect.com/science/article/pii/S1571066104809721.

### Constants

```python
def const(n):

    yield n

    yield from const(n)
```

```python
const(1)
```

    <generator object const at 0x7f425c332970>

```python
ones = const(1)
```

```python
list(take(ones, 10))
```

    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

### Nats

```python
def nats():

    yield 0

    g = nats() # !!

    yield from map(lambda n: n + 1, g)
```

```python
list(take(nats(), 10))
```

    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

### Primes

Consider the following functional specification for the naturals that are also *primes*
```haskell
primes = filterPrime [2..]
  where filterPrime (p:xs) =
          p : filterPrime [x | x <- xs, x `mod` p /= 0]
```

```python
def primes():

    def P(numbers):

        prime = next(numbers) # get the next prime from the iterator `it`.

        yield prime # yield the next prime number

        def not_divisible_by_prime(n):  # a mnemonic predicate.
            q, r = divmod(n, prime)
            return r != 0

        yield from P(filter(not_divisible_by_prime, numbers)) # `numbers` has been advanced before.

    yield from P(itertools.count(2))
```

```python
list(take(primes(), 20))
```

    [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71]

### Fibonacci, again

Remember,
$$
    f_{n+2} = f_{n+1} + f_{n}, \quad \text{where} \quad f_{0} = 0 \wedge f_{1} = 1
$$

```python
def fibs(first=0, second=1):

    yield first  # the first number in the Fibonacci series,
    yield second # ... and the second one.

    f, ff = itertools.tee(fibs(first, second)) # duplicate the stream of fibonacci numbers.

    next(ff) # advance just one of them

    yield from map(operator.add, f, ff) # according to the Fibonacci rule, yield all the rest.
```

```python
list(take(fibs(), 20))
```

    [0,
     1,
     1,
     2,
     3,
     5,
     8,
     13,
     21,
     34,
     55,
     89,
     144,
     233,
     377,
     610,
     987,
     1597,
     2584,
     4181]

#### ...and again

```python
from sympy import IndexedBase, init_printing # SymPy for symbolic computation

init_printing() # pretty printing math symbols and expressions
```

```python
x = IndexedBase('x')
x[1] # indexing as done in math.
```

```python
fibos = list(take(fibs(x[0], x[1]), 20)) # generate an abstract schema
fibos
```

```python
[expr.subs({x[0]:0, x[1]:1}) for expr in fibos] # Fibonacci numbers, as usual.
```

```python
[expr.subs({x[0]:2, x[1]:1}) for expr in fibos] # Lucas numbers, less usual.
```


```python
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
from OceanLab import OA
```

```python
x = Symbol('x')
y = Symbol('y')

amp = 1
L = 200
scalar = amp*sin(2*pi*x/L)*sin(2*pi*y/(0.5*L))
scalar
```

$\displaystyle \sin{\left(\frac{\pi x}{100} \right)} \sin{\left(0.02 \pi y \right)}$

```python
xi,yi = np.mgrid[0:100,0:100]

n = 300
np.random.seed(0)
ind = np.random.randint(0,xi.size-1,n)

iscalar = OA.scaloa(xi.ravel(),yi.ravel(),
          xi.ravel()[ind],
          yi.ravel()[ind],
          lambdify((x,y),scalar)(xi,yi).ravel()[ind],
          L,L/2,0.001)[0].reshape(xi.shape)

u = -Derivative(scalar,y).doit()
v = Derivative(scalar,x).doit()

ui = lambdify((x,y),u)(xi,yi)
vi = lambdify((x,y),v)(xi,yi)

iscalar2 = OA.vectoa(xi,yi,
          xi.ravel()[ind],yi.ravel()[ind],
          ui.ravel()[ind],vi.ravel()[ind],
          L,L/2,0.001)
iscalar2 = iscalar2-iscalar2.mean()+lambdify((x,y),scalar)(xi,yi).mean()
```

```python
kw = {
    'contourf':{
        'vmin':-amp,
        'vmax':amp,
        'levels':30,
        'cmap':'bwr',
    },
    'contour':{
        'levels':np.arange(-amp,amp+0.1,0.1),
        'colors':'k',
    },
    'scatter':{
        'vmin':-amp,
        'vmax':amp,
        'edgecolor':'k',
        'cmap':'bwr',
    },
    'quiver':{
        'scale':0.5,
    },
}

fig,ax = plt.subplots(2,3,figsize=(14,8))

C = ax[0,0].contourf(xi,yi,lambdify((x,y),scalar)(xi,yi),**kw['contourf'])
ax[0,0].scatter(xi.ravel()[ind],yi.ravel()[ind],c=lambdify((x,y),scalar)(xi,yi).ravel()[ind],**kw['scatter'])

ax[0,1].contourf(xi,yi,iscalar,**kw['contourf'])
ax[0,1].scatter(xi.ravel()[ind],yi.ravel()[ind],c=lambdify((x,y),scalar)(xi,yi).ravel()[ind],**kw['scatter'])

ax[0,2].contourf(xi,yi,iscalar-lambdify((x,y),scalar)(xi,yi),**kw['contourf'])
Ct = ax[0,2].contour(xi,yi,iscalar-lambdify((x,y),scalar)(xi,yi),**kw['contour'])
plt.clabel(Ct,fmt='%.1f')

C = ax[1,0].contourf(xi,yi,lambdify((x,y),scalar)(xi,yi),**kw['contourf'])
ax[1,0].quiver(xi.ravel()[ind],yi.ravel()[ind],ui.ravel()[ind],vi.ravel()[ind],**kw['quiver'])

ax[1,1].contourf(xi,yi,iscalar,**kw['contourf'])
ax[1,1].scatter(xi.ravel()[ind],yi.ravel()[ind],c=lambdify((x,y),scalar)(xi,yi).ravel()[ind],**kw['scatter'])

ax[1,2].contourf(xi,yi,iscalar2-lambdify((x,y),scalar)(xi,yi),**kw['contourf'])
Ct = ax[1,2].contour(xi,yi,iscalar2-lambdify((x,y),scalar)(xi,yi),**kw['contour'])
plt.clabel(Ct,fmt='%.1f')

fig.colorbar(C,ax=ax)

_ = ax[0,0].set(title='Data')
_ = ax[0,1].set(title='Interpolation')
_ = ax[0,2].set(title='Absolute Error')

_ = ax[0,0].set(ylabel='SCALOA')
_ = ax[1,0].set(ylabel='VECTOA')
```


# Tutorial Brief

SymPy is symbolic mathematics library written completely in Python and doesn't require any dependencies.

Finding Help:

- http://docs.sympy.org/latest/index.html
- http://sympy.org/en/features.html
- http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-5-Sympy.ipynb

NumPy
 Base N-dimensional array package

SciPy
 Fundamental library for scientific computing

Matplotlib
 Comprehensive 2D Plotting

IPython
 Enhanced Interactive Console

SymPy
 Symbolic mathematics

Pandas
 Data structures & analysis

> ###SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python and does not require any external libraries.
> **http://sympy.org/**

# Import SymPy

```python
from sympy import *
```

```python
3 + math.sqrt(3)
```

    4.732050807568877

```python
expr = 3 * sqrt(3)
expr
```

    3*sqrt(3)

```python
init_printing(use_latex='mathjax')
```

```python
expr
```

$$3 \sqrt{3}$$

```python
expr = sqrt(8)
expr
```

$$2 \sqrt{2}$$

# `symbols()` & `Symbol()`

```python
x, y = symbols("x y")
```

```python
expr = x**2 + y**2
expr
```

$$x^{2} + y^{2}$$

```python
expr = (x+y)**3
expr
```

$$\left(x + y\right)^{3}$$

## Assumptions for symbols

```python
a = Symbol("a")
```

```python
a.is_imaginary
```

```python
b = Symbol("b", integer=True)
```

```python
b.is_imaginary
```

    False

```python
c = Symbol("c", positive=True)
```

```python
c.is_positive
```

    True

```python
c.is_imaginary
```

    False

# Imaginary Numbers

```python
I
```

$$i$$

```python
I ** 2
```

$$-1$$

# `Rational()`

```python
Rational(1,3)
```

$$\frac{1}{3}$$

```python
Rational(1,3) + Rational(1,2)
```

$$\frac{5}{6}$$

# Numerical evaluation

```python
expr = Rational(1,3) + Rational(1,2)
N(expr)
```

$$0.833333333333333$$

```python
N(pi, 100)
```

$$3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068$$

```python
pi.evalf(100)
```

$$3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068$$

# `subs()`

```python
expr = x**2 + 2*x + 1
expr
```

$$x^{2} + 2 x + 1$$

```python
expr.subs(x, 1)
```

$$4$$

```python
expr = pi * x**2
expr
```

$$\pi x^{2}$$

```python
expr.subs(x, 3)
```

$$9 \pi$$

```python
N(_)
```

$$28.2743338823081$$

# `factor()` and `expand()`

```python
expr = (x + y) ** 2
expr
```

$$\left(x + y\right)^{2}$$

```python
expand(expr)
```

$$x^{2} + 2 x y + y^{2}$$

```python
factor(_)
```

$$\left(x + y\right)^{2}$$

# `simplify()`

```python
expr = (2*x + Rational(1,3)*x + 4) / x
expr
```

$$\frac{1}{x} \left(\frac{7 x}{3} + 4\right)$$

```python
simplify(expr)
```

$$\frac{7}{3} + \frac{4}{x}$$

```python
expr = "(2*x + 1/3*x + 4)/x"
simplify(expr)
```

$$\frac{7}{3} + \frac{4}{x}$$

```python
expr = sin(x)/cos(x)
expr
```

$$\frac{\sin{\left (x \right )}}{\cos{\left (x \right )}}$$

```python
simplify(expr)
```

$$\tan{\left (x \right )}$$

# `apart()` and `together()`

```python
expr = 1/(x**2 + 2*x)
expr
```

$$\frac{1}{x^{2} + 2 x}$$

```python
apart(expr)
```

$$- \frac{1}{2 x + 4} + \frac{1}{2 x}$$

```python
together(_)
```

$$\frac{1}{x \left(x + 2\right)}$$

# Calculus

```python
diff(sin(x), x)
```

$$\cos{\left (x \right )}$$

```python
diff(log(x**2 + 1) + 2*x, x)
```

$$\frac{2 x}{x^{2} + 1} + 2$$

```python
integrate(cos(x), x)
```

$$\sin{\left (x \right )}$$

```python
Integral(sin(x), (x,0,pi))
```

$$\int_{0}^{\pi} \sin{\left (x \right )}\, dx$$

```python
N(_)
```

$$2.0$$

# `Sum()`

```python
expr = Sum(1/(x**2 + 2*x), (x, 1, 10))
expr
```

$$\sum_{x=1}^{10} \frac{1}{x^{2} + 2 x}$$

```python
expr.doit()
```

$$\frac{175}{264}$$

# `Product()`

```python
expr = Product(1/(x**2 + 2*x), (x, 1, 10))
expr
```

$$\prod_{x=1}^{10} \frac{1}{x^{2} + 2 x}$$

```python
expr.doit()
```

$$\frac{1}{869100503040000}$$

# `Solve()`

```python
expr = 2*x + 1
solve(expr)
```

$$\begin{bmatrix}- \frac{1}{2}\end{bmatrix}$$

```python
expr = x**2 - 1
solve(expr)
```

$$\begin{bmatrix}-1, & 1\end{bmatrix}$$

```python
expr_1 = 2*x + y + 3
expr_2 = 2*y - x
solve([expr_1, expr_2],(x,y))
```

$$\begin{Bmatrix}x : - \frac{6}{5}, & y : - \frac{3}{5}\end{Bmatrix}$$

# Units

```python
from sympy.physics import units as u
```

```python
5. * u.milligram
```

$$5.0 \cdot 10^{-6} kg$$

```python
1./2 * u.inch
```

$$0.0127 m$$

```python
1. * u.nano
```

$$1.0 \cdot 10^{-9}$$

```python
u.watt
```

$$\frac{kg m^{2}}{s^{3}}$$

```python
u.ohm
```

$$\frac{kg m^{2}}{A^{2} s^{3}}$$

## Converting from Kilometers/hours to Miles/hours

```python
kmph = u.km / u.hour
mph = u.mile / u.hour

N(mph / kmph)
```

$$1.609344$$

```python
80 * N(mph / kmph)
```

$$128.74752$$

# Working with NumPy / Pandas and Matplotlib

```python
def sympy_expr(x_val):
    expr = x**2 + sqrt(3)*x - Rational(1,3)
    return expr.subs(x, x_val)
```

```python
sympy_expr(3)
```

$$3 \sqrt{3} + \frac{26}{3}$$

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
list1 = np.arange(1,1000)
list2 = pd.Series(list1)
```

```python
%timeit [sympy_expr(item) for item in list1]
%timeit [sympy_expr(item) for item in list2]
```

    1 loops, best of 3: 103 ms per loop
    10 loops, best of 3: 105 ms per loop

```python
%timeit np.vectorize(sympy_expr)(list1)
%timeit list2.apply(sympy_expr)
```

    10 loops, best of 3: 776 ms per loop
    1 loops, best of 3: 1.09 s per loop

```python
expr = x**2 + sqrt(3)*x - Rational(1,3)

lf = lambdify(x, expr)
```

```python
%timeit lf(list1)
%timeit lf(list2)
```

    1000 loops, best of 3: 242 µs per loop
    100 loops, best of 3: 4.8 ms per loop

```python
fig = plt.figure()
axes = fig.add_subplot(111)

x_vals = np.linspace(-5.,5.)
y_vals = lf(x_vals)

axes.grid()
axes.plot(x_vals, y_vals)

plt.show();
```


```python
import numpy as np
import sympy as smp
import scipy.io as sc
from scipy.integrate import odeint
import matplotlib.pyplot as plt
from matplotlib import animation
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.animation import PillowWriter
```

```python
t, g = smp.symbols('t g')
m1, m2 = smp.symbols('m1 m2')
L1, L2 = smp.symbols('L1, L2')
```

```python
the1, the2 = smp.symbols(r'\theta_1, \theta_2', cls=smp.Function)
```

```python
the1 = the1(t)
the2 = the2(t)

```

```python
the1
```

$\displaystyle \theta_{1}{\left(t \right)}$

```python

the1_d = smp.diff(the1, t)
the2_d = smp.diff(the2, t)
the1_dd = smp.diff(the1_d, t)
the2_dd = smp.diff(the2_d, t)
```

```python

x1 = L1*smp.sin(the1)
y1 = -L1*smp.cos(the1)
x2 = L1*smp.sin(the1)+L2*smp.sin(the2)
y2 = -L1*smp.cos(the1)-L2*smp.cos(the2)
```

```python

# Kinetic
T1 = 1/2 * m1 * (smp.diff(x1, t)**2 + smp.diff(y1, t)**2)
T2 = 1/2 * m2 * (smp.diff(x2, t)**2 + smp.diff(y2, t)**2)
T = T1+T2
# Potential
V1 = m1*g*y1
V2 = m2*g*y2
V = V1 + V2
# Lagrangian
L = T-V
```

```python
L
```

$\displaystyle L_{1} g m_{1} \cos{\left(\theta_{1}{\left(t \right)} \right)} - g m_{2} \left(- L_{1} \cos{\left(\theta_{1}{\left(t \right)} \right)} - L_{2} \cos{\left(\theta_{2}{\left(t \right)} \right)}\right) + 0.5 m_{1} \left(L_{1}^{2} \sin^{2}{\left(\theta_{1}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{1}{\left(t \right)}\right)^{2} + L_{1}^{2} \cos^{2}{\left(\theta_{1}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{1}{\left(t \right)}\right)^{2}\right) + 0.5 m_{2} \left(\left(L_{1} \sin{\left(\theta_{1}{\left(t \right)} \right)} \frac{d}{d t} \theta_{1}{\left(t \right)} + L_{2} \sin{\left(\theta_{2}{\left(t \right)} \right)} \frac{d}{d t} \theta_{2}{\left(t \right)}\right)^{2} + \left(L_{1} \cos{\left(\theta_{1}{\left(t \right)} \right)} \frac{d}{d t} \theta_{1}{\left(t \right)} + L_{2} \cos{\left(\theta_{2}{\left(t \right)} \right)} \frac{d}{d t} \theta_{2}{\left(t \right)}\right)^{2}\right)$

```python
LE1 = (smp.diff(L, the1) - (smp.diff(smp.diff(L, the1_d), t))).simplify()
LE2 = (smp.diff(L, the2) - (smp.diff(smp.diff(L, the2_d), t))).simplify()
```

```python
LE1
```

$\displaystyle - 1.0 L_{1} \left(L_{1} m_{1} \frac{d^{2}}{d t^{2}} \theta_{1}{\left(t \right)} + L_{1} m_{2} \frac{d^{2}}{d t^{2}} \theta_{1}{\left(t \right)} + L_{2} m_{2} \sin{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{2}{\left(t \right)}\right)^{2} + L_{2} m_{2} \cos{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \frac{d^{2}}{d t^{2}} \theta_{2}{\left(t \right)} + g m_{1} \sin{\left(\theta_{1}{\left(t \right)} \right)} + g m_{2} \sin{\left(\theta_{1}{\left(t \right)} \right)}\right)$

```python
LE2
```

$\displaystyle 1.0 L_{2} m_{2} \left(L_{1} \sin{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{1}{\left(t \right)}\right)^{2} - L_{1} \cos{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \frac{d^{2}}{d t^{2}} \theta_{1}{\left(t \right)} - L_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left(t \right)} - g \sin{\left(\theta_{2}{\left(t \right)} \right)}\right)$

```python
sols = smp.solve([LE1, LE2], (the1_dd, the2_dd),
                simplify=False, rational=False)
```

```python
dz1dt_f = smp.lambdify((t,g,m1,m2,L1,L2,the1,the2,the1_d,the2_d), sols[the1_dd])
dz2dt_f = smp.lambdify((t,g,m1,m2,L1,L2,the1,the2,the1_d,the2_d), sols[the2_dd])
dthe1dt_f = smp.lambdify(the1_d, the1_d)
dthe2dt_f = smp.lambdify(the2_d, the2_d)
```

```python
def dSdt(S, t, g, m1, m2, L1, L2):
    the1, z1, the2, z2 = S
    return [
        dthe1dt_f(z1),
        dz1dt_f(t, g, m1, m2, L1, L2, the1, the2, z1, z2),
        dthe2dt_f(z2),
        dz2dt_f(t, g, m1, m2, L1, L2, the1, the2, z1, z2),
    ]
```

```python
t = np.linspace(0, 40, 10001)
g = 9.81
m1=2
m2=2
L1 = 2
L2 = 1
ans = odeint(dSdt, y0=[np.pi-0.07, 0, 0, 0], t=t, args=(g,m1,m2,L1,L2))
t = np.reshape(t, (-1,1))
data = np.hstack([t, ans])
np.savetxt("dp.csv", data, delimiter=",")
```

```python
t.shape
```

    (10001, 1)

```python
ans.T
```

    array([[ 3.07159265e+00,  3.07158719e+00,  3.07157080e+00, ...,
            -1.42418506e+00, -1.41445660e+00, -1.40466985e+00],
           [ 0.00000000e+00, -2.73119796e-03, -5.46240519e-03, ...,
             2.41885754e+00,  2.44238666e+00,  2.44802582e+00],
           [ 0.00000000e+00, -1.08978506e-05, -4.35897561e-05, ...,
            -2.65149001e+01, -2.65626525e+01, -2.66103632e+01],
           [ 0.00000000e+00, -5.44886567e-03, -1.08968945e-02, ...,
            -1.19304009e+01, -1.19393661e+01, -1.19095540e+01]])

```python
the1 =  (np.arctan2(np.sin(ans.T[0]), np.cos(ans.T[0]))*180)/np.pi
the2 = (np.arctan2(np.sin(ans.T[2]), np.cos(ans.T[2]))*180)/np.pi
```

```python
plt.plot(t, the1)
plt.plot(t, the2)
plt.show()
```

```python
def get_x1y1x2y2(t, the1, the2, L1, L2):
    return (L1*np.sin(the1),
            -L1*np.cos(the1),
            L1*np.sin(the1) + L2*np.sin(the2),
            -L1*np.cos(the1) - L2*np.cos(the2))

x1, y1, x2, y2 = get_x1y1x2y2(t, ans.T[0], ans.T[2], L1, L2)
```

```python
def animate(i):
    ln1.set_data([0, x1[i], x2[i]], [0, y1[i], y2[i]])

fig, ax = plt.subplots(1,1, figsize=(8,8))
ax.set_facecolor('k')
ax.get_xaxis().set_ticks([])    # enable this to hide x axis ticks
ax.get_yaxis().set_ticks([])    # enable this to hide y axis ticks
ln1, = plt.plot([], [], 'ro--', lw=3, markersize=8)
ax.set_ylim(-4,4)
ax.set_xlim(-4,4)
ani = animation.FuncAnimation(fig, animate, frames=1000, interval=50)
ani.save('pen.gif',writer='pillow',fps=25)
```


```python
#from sympy import *
import sympy as sy
import numpy as np
import matplotlib.pyplot as plt

from IPython.display import display, Math

sy.init_printing()
```

The second command makes sure that formulas are presented in a graphical way, if
possible. Then, we generate a symbol and define the integrand:

```python
x = sy.symbols('x') #Python object of type Symbol
#f1 = sy.Lambda(x, (x**2)) # SymPy Lambda function
f1 = sy.lambdify(x, (x**2), "numpy")
#f2 = sy.Lambda(x, (4*x)) # SymPy Lambda function
f2 = sy.lambdify(x, (4*x), "numpy")
display(f1)
display(f2)
```

    <function _lambdifygenerated(x)>

    <function _lambdifygenerated(x)>

```python
p_range = (x,-1,5)
p = sy.plot(f1(x), f2(x), p_range, show=True)
```

Now we start with the symbolic computation of the integral:

```python
if1 = sy.Lambda(x, sy.integrate(f1(x),x))
if2 = sy.Lambda(x, sy.integrate(f2(x),x))
display(if1)
display(if2)
```

## Poiščemo presečišče krivulj

```python
presecisca = sy.solve(f1(x) - f2(x), x)
print(f'najdena presecisca so {presecisca}')
```

    najdena presecisca so [0, 4]

## narišemo iskane pološčine

```python
#x = symbols('x')
#f = sin(x)

x_array = np.linspace(0, 4, 10)
f1_array = f1(x_array)
f2_array = f2(x_array)

print(x_array)
print(f1_array)
#f_array = sy.lambdify(x, f)(x_array)

fig, ax = plt.subplots()

ax.plot(x_array, f1_array, color = 'r', label='f1')
ax.plot(x_array, f2_array, color = 'b', label='f2')

ax.fill_between(x_array, f1_array, f2_array, facecolor = 'grey', alpha = 0.5)

ax.legend()
plt.show()
```

## Izračunamo ploščino med krivuljama s uporabo določenega integrala

surface = definite integral in range x -> 0, 4

```python
s1 = if1(4) - if1(0)
s2 = if2(4) - if2(0)

print(f'ploščina med abciso in krivuljo je {s1}, med abciso in premico pa {s2}')

# ker je premica nad krivuljo, ima večjo površino in odštevamo od nje
s = s2 - s1
print(f'ploščina med krivuljama med x=0 in x=4 je {s}')
```

    ploščina med abciso in krivuljo je 64/3, med abciso in premico pa 32
    ploščina med krivuljama med x=0 in x=4 je 32/3

```python

```


Symbolic Algebra with `sympy`
====

```python
from sympy import *
```

```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.5.1 (Python 3.6.10-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

Basics
----

```python
from sympy.stats import *
E(Die('X', 6))
```

```python
sqrt(8)
```

```python
expr = x + 2*y
```

```python
expr2 = x*expr
```

```python
expr2
```

```python
expand(expr2)
```

```python
factor(expand(expr2))
```

```python
diff(sin(x) * exp(x), x)
```

```python
integrate(exp(x)*sin(x) + exp(x)*cos(x), x)
```

```python
integrate(sin(x**2), (x, -oo, oo))
```

```python
dsolve(Eq(f(t).diff(t, t) - f(t), exp(t)), f(t))
```

```python
Matrix([[1,2],[2,2]]).eigenvals()
```

```python
nu = symbols('nu')
besselj(nu, z).rewrite(jn)
```

```python
latex(Integral(cos(x)**2, (x, 0, pi)))
```

    '\\int\\limits_{0}^{\\pi} \\cos^{2}{\\left(x \\right)}\\, dx'

```python
expr = cos(x) + 1
```

```python
expr.subs(x, y)
```

```python
expr = x**y
expr = expr.subs(y, x**y)
expr = expr.subs(y, x**y)
expr = expr.subs(x, x**x)
```

```python
expr
```

```python
expr = sin(2*x) + cos(2*x)
expand_trig(expr)
```

```python
expr.subs(sin(2*x), 2*sin(x)*cos(x))
```

```python
expr = x**4 - 4*x**3 + 4*x**2 - 2*x +3
```

```python
replacements = [(x**i, y**i) for i in range(5) if i%2 == 0]
expr.subs(replacements)
```

```python
sexpr = "x**4 - 4*x**3 + 4*x**2 - 2*x +3"
sympify(sexpr)
```

```python
sqrt(8).evalf()
```

```python
pi.evalf(100)
```

```python
expr = cos(2*x)
expr.evalf(subs = {x: 2.4})
```

```python
expr = cos(x)**2 + sin(x)**2
expr.evalf(subs = {x: 1}, chop=True)
```

```python
a = range(10)
expr = sin(x)
f = lambdify(x, expr, "numpy")
f(a)
```

    array([ 0.        ,  0.84147098,  0.90929743,  0.14112001, -0.7568025 ,
           -0.95892427, -0.2794155 ,  0.6569866 ,  0.98935825,  0.41211849])

Simplify
----

```python
simplify(sin(x)**2 + cos(x)**2)
```

```python
simplify(gamma(x) / gamma(x-2))
```

```python
expand((x + y)**3)
```

```python
factor(x**3 - x**2 + x - 1)
```

```python
factor_list(x**3 - x**2 + x - 1)
```

```python
expand((cos(x) + sin(x))**2)
```

```python
expr = x*y + x - 3 + 2*x**2 - z*x**2 + x**3
cexpr = collect(expr, x)
cexpr
```

```python
cexpr.coeff(x**2)
```

```python
cancel((x**2 + 2*x + 1) / (x**2 + x))
```

```python
expr = (x*y**2 - 2*x*y*z + x*z**2 + y**2 - 2*y*z + z**2)/(x**2 - 1)
```

```python
cancel(expr)
```

```python
factor(expr)
```

```python
expr = (4*x**3 + 21*x**2 + 10*x + 12)/(x**4 + 5*x**3 + 5*x**2 + 4*x)
```

```python
apart(expr)
```

```python
trigsimp(sin(x)*tan(x)/sec(x))
```

```python
trigsimp(cosh(x)**2 + sinh(x)**2)
```

```python
expand_trig(sin(x + y))
```

```python
x, y = symbols('x y', positive=True)
a, b = symbols('a b', real=True)
z, t, c = symbols('z t c')
```

```python
powsimp(x**a*x**b)
```

```python
powsimp(x**a*y**a)
```

```python
powsimp(t**c*z**c)
```

```python
powsimp(t**c*z**c, force=True)
```

```python
expand_power_exp(x**(a + b))
```

```python
expand_power_base((x*y)**a)
```

```python
powdenest((x**a)**b)
```

```python
n = symbols('n', real=True)
```

```python
expand_log(log(x*y))
```

```python
expand_log(log(x**n))
```

```python
logcombine(n*log(x))
```

```python
x, y, z = symbols('x y z')
k, m, n = symbols('k m n')
```

```python
factorial(n)
```

```python
binomial(n, k)
```

```python
hyper([1,2], [3], z)
```

```python
factorial(x).rewrite(gamma)
```

```python
tan(x).rewrite(sin)
```

```python
expand_func(gamma(x + 3))
```

```python
hyperexpand(hyper([1, 1], [2], z))
```

```python
expr =  meijerg([[1],[1]], [[1],[]], -z)
```

```python
expr
```

```python
hyperexpand(expr)
```

```python
combsimp(binomial(n+1, k+1)/binomial(n, k))
```

```python
def list_to_frac(l):
    expr = Integer(0)
    for i in reversed(l[1:]):
        expr += i
        expr = 1/expr
    return l[0] + expr
```

```python
list_to_frac([1,2,3,4])
```

```python
syms = symbols('a0:5')
```

```python
syms
```

```python
frac = list_to_frac(syms)
```

```python
frac
```

```python
frac = cancel(frac)
```

```python
frac
```

```python
from sympy.printing import print_ccode
print_ccode(frac)
```

Calculus
----

```python
diff(cos(x), x)
```

```python
diff(x**4, x, 3)
```

```python
expr = exp(x*y*z)
diff(expr, x, y, 2, z, 4)
```

```python
deriv = Derivative(expr, x, y, 2, z, 4)
deriv
```

```python
deriv.doit()
```

```python
integrate(cos(x), x)
```

```python
integrate(exp(-x), (x, 0, oo))
```

```python
integrate(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
integrate(x**x, x)
```

```python
expr = Integral(log(x)**2, x)
expr
```

```python
expr.doit()
```

```python
integrate(sin(x**2), x)
```

```python
integrate(x**y*exp(-x), (x, 0, oo))
```

```python
limit(sin(x)/x, x, 0)
```

```python
expr = Limit((cos(x) - 1)/x, x, 0)
expr
```

```python
expr.doit()
```

```python
limit(1/x, x, 0, '-')
```

```python
expr = exp(sin(x))
expr.series(x, 0, 4)
```

```python
x + x**3 + x**6 + O(x**4)
```

```python
x * O(1)
```

```python
expr.series(x, 0, 4).removeO()
```

```python
exp(x - 6).series(x, 6)
```

```python
exp(x - 6).series(x, 6).removeO().subs(x, x - 6)
```

Working with matrices
----

```python
M = Matrix([[1,2,3],[3,2,1]])
P = Matrix([0,1,1])
M*P
```

```python
M
```

```python
M.shape
```

```python
M.col(-1)
```

```python
M = Matrix([[1,3], [-2,3]])
M**-1
```

```python
M.T
```

```python
eye(3)
```

```python
diag(1,2,3)
```

```python
M.det()
```

```python
M= Matrix([[1,0,1,3],[2,3,4,7],[-1,-3,-3,-4]])
M
```

```python
M.rref()
```

```python
M = Matrix([[1,2,3,0,0],[4,10,0,0,1]])
M
```

```python
M.nullspace()
```

```python
M = Matrix([[1,1,2],[1,4,1],[2,1,5]])
M
```

```python
L, U, P = M.LUdecomposition()
L, U, P
```

```python
L*U
```

```python
L, D = M.LDLdecomposition()
L, D
```

```python
L*D*L.T
```

```python
L = M.cholesky()
L
```

```python
L*L.T
```

```python
M = Matrix([[3, -2, 4, -2], [5,3,-3,-2], [5,-2,2,-2], [5,-2,-3,3]])
M
```

```python
M.eigenvals()
```

```python
M.eigenvects()
```

```python
P, D = M.diagonalize()
```

```python
P
```

```python
D
```

```python
P*D*P**-1
```

```python
lamda = symbols('lamda')
p = M.charpoly(lamda)
p
```

```python
factor(p)
```

Solving Algebraic and Differential Equations
----

```python
solve(x**2 - 1, x)
```

```python
solve((x - y + 2, x + y -3), (x, y))
```

```python
solve(x**3 - 6*x**2 + 9*x, x)
```

```python
roots(x**3 - 6*x**2 + 9*x, x)
```

```python
f, g = symbols('f g', cls=Function)
```

```python
f(x).diff(x)
```

```python
diffeq = Eq(f(x).diff(x, 2) - 2*f(x).diff(x) + f(x), sin(x))
```

```python
diffeq
```

```python
dsolve(diffeq, f(x))
```

```python
dsolve(f(x).diff(x)*(1 - sin(f(x))), f(x))
```

```python
a, t = symbols('a t')
f(t).diff(t)
diffeq = Eq(f(t).diff(t), a*t)
```

```python
diffeq
```

```python
dsolve(diffeq, f(t))
```

```python
x = symbols('x', cls=Function)
diffeq = Eq(x(t).diff(t), a*x(t))
diffeq
```

```python
dsolve(diffeq, x(t))
```

Numerics
----

```python
N(pi, 10)
```

```python
x = symbols('x')
```

```python
expr = Integral(sin(x)/(x**2), (x, 1, oo))
```

```python
expr.evalf()
```

```python
expr.evalf(maxn=20)
```

```python
expr.evalf(quad='osc')
```

```python
expr.evalf(20, quad='osc')
```

```python
expr = Integral(sin(1/x), (x, 0, 1))
expr
```

```python
expr.evalf()
```

```python
expr = expr.transform(x, 1/x)
expr
```

```python
expr.evalf(quad='osc')
```

```python
nsimplify(pi, tolerance=0.001)
```

```python
expr = sin(x)/x
```

```python
%timeit expr.evalf(subs={x: 3.14})
```

```python
f1 = lambdify(x, expr)
%timeit f1(3.14)
```

```python
f2 = lambdify(x, expr, 'numpy')
%timeit f2(3.14)
```

```python
%timeit f2(np.linspace(1, 10, 10000))
```

```python
%timeit [f1(x) for x in np.linspace(1, 10, 10000)]
```

```python
from mpmath import *
```

```python
f = odefun(lambda x, y: [-y[1], y[0]], 0, [1, 0])
for x in [0, 1, 2.5, 10]:
    nprint(f(x), 15)
    nprint([cos(x), sin(x)], 15)
```

```python
from sympy.plotting import plot
%matplotlib inline

plot(x*y**3 - y*x**3)
pass
```

```python
from sympy.plotting import plot3d_parametric_surface
from sympy import sin, cos
u, v = symbols('u v')
plot3d_parametric_surface(cos(u + v), sin(u - v), u-v, (u, -5, 5), (v, -5, 5))
pass
```

Statistics
----

```python
from sympy.stats import *
```

```python
k = Symbol("k", positive=True)
theta = Symbol("theta", positive=True)
z = Symbol("z")
X = Gamma("x", k, theta)
```

```python
D = density(X)(z)
D
```

```python
C = cdf(X, meijerg=True)(z)
C
```

```python
E(X)
```

```python
V = variance(X)
V
```

```python
simplify(V)
```

```python
N = Normal('Gaussian', 10, 5)
density(N)(z)
```

```python
density(N)(3).evalf()
```

```python
simplify(cdf(N)(z))
```

```python
P(N > 10)
```

```python
sample(N)
```


Simulación matemática 2018

Lázaro Alonso

Email:   `alonsosilva@iteso.mx, lazarus.alon@gmail.com`

### Por favor, den click al siguiente link. Será una forma fácil de que me hagan llegar sus dudas, además de que todos podremos estar al tanto de los problemas tanto de tarea como de clase.

https://join.slack.com/t/sm-grupo/shared_invite/enQtMzcxMzcxMjY1NzgyLWE5ZDlhYjg4OGJhMGE2ZmY2ZGUzZWIxYzQzOTUxZWU2ZGM5YjUyYWMyZGUzNzZjMDE5ZDIxYTA4YTI2ZWQ1NTU

```python
import sympy as sym; sym.init_printing(use_latex='mathjax')
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

### Máximos y mínimos

##### 1. Encuentre los valores máximo y mínimo locales de la función
$$g(x) = x + 2 \sin x$$

Elementos que debe de contener su respuesta:
   - Gráfica de la funcion $g(x)$
   - Gráfica de la primera y segunda derivada, ($g(x)'$, $g(x)''$)
   - Indicar en la gráfica los máximos y mínimos (_utilizar plt.scatter_)

```python
sym.var('x')
g = x + 2*sym.sin(x)
g_num = sym.lambdify([x], g, 'numpy')
x_vec = np.linspace(-4, 4, 100)
plt.figure(figsize = (4,4))
plt.plot(x_vec, g_num(x_vec), "-.")
plt.xlabel('$x$', fontsize = 18)
plt.ylabel('$g(x)$', fontsize = 18)
plt.savefig("fx_primero.jpeg", bbox_inches='tight')
plt.show()
```

##### 2. Discuta la curva $f(x) = x^4 - 4x^3$. Puntos de inflexión, máximos y mínimos. Su respuesta debe de incluir los mismos puntos que el caso anterior.

```python
sym.var('x')
f = x**4 - 4*x**3
f_num = sym.lambdify([x], f, 'numpy')
x_vec = np.linspace(-2, 4, 100)
plt.figure(figsize = (4,4))
plt.plot(x_vec, f_num(x_vec), "k-.")
plt.xlabel('$x$', fontsize = 18)
plt.ylabel('$f(x)$', fontsize = 18)
plt.xlim(xmax = 3.5, xmin = -1.5)
plt.ylim(ymax = 10, ymin= -32)
plt.scatter([3],[-27], s= 55, c = "r", label = "Min Global")
plt.text(2.2, -30, "Min Global", fontsize = 10)
plt.legend()
plt.savefig("fx_segundo.jpeg", bbox_inches='tight')
plt.show()
```

##### 3. Se va a fabricar una lata que ha de contaner 1L de aceite. Encuentre las dimensiones que debe de tener la lata de manera que minimicen el costo del metal para fabricarla.

- La solución debe de incluir los siguientes elementos
    - Ecuación del sistema
    - Números críticos
    - Dibujar la lata

```python
from matplotlib.patches import Circle, PathPatch
from mpl_toolkits.mplot3d import Axes3D
import mpl_toolkits.mplot3d.art3d as art3d
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
fig = plt.figure(figsize=(4,5))
ax = Axes3D(fig)
x = np.linspace(-1, 1, 100)
z = np.linspace(0, 4, 100)
X, Z = np.meshgrid(x, z)
Y = np.sqrt(1 - X**2)

# Draw parameters
rstride = 4
cstride = 4
### Cilindro
ax.plot_surface(X, Y, Z, alpha=0.5, cmap = "Blues", rstride=rstride, cstride=cstride)
ax.plot_surface(X, -Y, Z, alpha=0.5, cmap = "Blues", rstride=rstride, cstride=cstride)
# Tapa superior e inferior
up = Circle((0, 0), 1, alpha = .5, fc = "blue")
ax.add_patch(up)
art3d.pathpatch_2d_to_3d(up, z=4.1)
#inferior
down = Circle((0, 0), 1, alpha = .5, fc = "orange")
ax.add_patch(down)
art3d.pathpatch_2d_to_3d(down, z=-.1)
#Estilo
ax.set_xlabel("X")
ax.set_ylabel("Y")
ax.set_zlabel("Z")
ax.set_xticks([-1,1])
ax.set_yticks([-1,1])
ax.set_zticks([-2,2])
ax.set_facecolor("white")
ax.xaxis.pane.fill = False
ax.yaxis.pane.fill = False
ax.zaxis.pane.fill = False
ax.axis("off");
plt.savefig("lata.jpeg", bbox_inches='tight');
```

Created with Jupyter by Lázaro Alonso.
  Copyright:  Public Domain como en [CC](https://creativecommons.org/licenses/by/2.0/) (Exepto donde se indique lo contrario)


# Non-Linear Least Squares

We're now going to approach estimation with a non-linear state to measurement space mapping.

$
y  = h(x) + v
$

where $h(x)$ is a non-linear function and $v$ is a noise vector.

As presented in class we cannot apply recursive estimation to the problem in it's current non-linear form. However, we can *linearize* the problem, allowing application of recursive estimation:

$
h(x) \approx h(\hat{x}_t) + H_{\hat{x}_t}(x - \hat{x}_t)
$

where $H_{\hat{x}_t}$ is the Jacobian of h evaluated at $\hat{x}_t$:

This presents $h(x)$ as a linear function in the form of $Ax + b$ since $h(\hat{x}_t)$ and $H_{\hat{x}_t}$ are constant in this context. From here we can use recursive estimation the same as before. Note the *linearization* is only useful if $x$ is near $\hat{x}_t$, otherwise the approximation quickly breaks down. This is why it's important to update the Jacobian frequently.

```python
import numpy as np
import matplotlib.pyplot as plt
import numpy.linalg as LA

%matplotlib inline
```

We'll define $h(x)$ as:

$h(x) = (f_{range}(x), f_{bearing}(x))$

where

$
f_{range}(x) = sqrt({x_1}^2 + {x_2}^2) \\
f_{bearing}(x) = atan2(x_2, x_1)
$

```python
# TODO: complete implementation
def f_range(x):
    """
    Distance of x from the origin.
    """
    return LA.norm(x)

# TODO: complete implementation
def f_bearing(x):
    """
    atan2(x_2, x_1)
    """
    return np.arctan2(x[1], x[0])

def h(x):
    return np.array([f_range(x), f_bearing(x)])
```

### Linearize $h(x)$

In order to linearize $h(x)$ you'll need the Jacobian:

$
\begin{bmatrix}
\frac{\partial{f_{range}}}{\partial{x_1}} & \frac{\partial{f_{range}}}{\partial{x_2}} \\
\frac{\partial{f_{bearing}}}{\partial{x_1}} & \frac{\partial{f_{bearing}}}{\partial{x_2}} \\
\end{bmatrix}
$

Remember to swap the derivative results of atan2 to match the swapped inputs ($atan2(x, y)$ vs $atan2(y, x)$).

$
\begin{align}
\frac{\partial{f_{range}}}{\partial{x_1}} &= 2 x_1 \cdot \frac{1}{2} \left ( x_1^2 + x_2^2 \right )^{-\frac{1}{2}} \\
\frac{\partial{f_{range}}}{\partial{x_2}} &= 2 x_2 \cdot \frac{1}{2} \left ( x_1^2 + x_2^2 \right )^{-\frac{1}{2}}
\end{align}
$

```python
# TODO: complete jacobian of h(x)
def jacobian_of_h(x):
    x_mag = LA.norm(x)
    return np.array([
        [x[0] / x_mag, x[1] / x_mag],
        [-x[0] / x_mag**2, x[1] / x_mag**2]
    ]).squeeze()
```

Awesome! With the Jacobian of $h$ in your toolbox, you can plug it into recursive estimation.

The update functions should look familiar ($H_{\hat{x}_t}$ is the Jacobian of $\hat{x}_t$).

$
Q_{t+1} = (Q_{t}^{-1} + H_{\hat{x}_t}^T R^{-1} H_{\hat{x}_t})^{-1} \\
\hat{x_{t+1}} = \hat{x_t} + Q_{t+1} H_{\hat{x}_t}^{T} R^{-1} (\tilde{y_t} -  h(\hat{x_t}))
$

### Setup

```python
n_samples = 1000

# Covariance matrix
# added noise for range and bearing functions
#
# NOTE: these are set to low variance values
# to start with, if you increase them you
# might more samples to get
# a good estimate.
R = np.eye(2)
R[0, 0] = 0.01
R[1, 1] = np.radians(1)

# ground truth state
x = np.array([1.5, 1])
```

Initialize $\hat{x}_0$ and $Q_0$.

```python
x_hat0 = np.array([3., 3]).reshape(-1, 1)
Q0 = np.eye(len(x_hat0))
```

```python
# TODO: Recursive Estimation
def recursive_estimation(x_hat0, Q0, n_samples):
    x_hat = np.copy(x_hat0)
    Q = np.copy(Q0)

    for _ in range(n_samples):
        # print(f"x_hat: {x_hat}")
        # TODO: sample a measurement
        y_obs = (h(x) + np.random.multivariate_normal([0., 0.], R)).reshape(-1, 1)
        # print(f"y_obs: {y_obs}")

        # TODO: compute the jacobian of h(x_hat)
        H = jacobian_of_h(x_hat)
        # print(f"J(h): {H}")

        # print(f"h(x): {h(x_hat)}")
        # TODO: update Q and x_hat
        Q = LA.pinv(LA.pinv(Q) + H.T @ LA.pinv(R) @ H)
        # print(f"y_obs - h(x): {y_obs - h(x_hat).reshape(-1, 1)}")
        x_hat = x_hat + Q @ H.T @ LA.pinv(R) @ (y_obs - h(x_hat).reshape(-1, 1))

    return x_hat, Q
```

```python
print("x̂0 =", x_hat0.squeeze())

x_hat, Q = recursive_estimation(x_hat0, Q0, n_samples)

print("x =", x.squeeze())
print("x̂ =", x_hat.squeeze())
print("Hx =", h(x))
print("Hx̂ =", h(x_hat))
```

    x̂0 = [ 3.  3.]
    x = [ 1.5  1. ]
    x̂ = [ 1.49531679  1.00665381]
    Hx = [ 1.80277564  0.5880026 ]
    Hx̂ = [ 1.80258819  0.59251507]

### Error Curve

```python
errors = []
Ns = np.arange(0, 201, 5)
for n in Ns:
    x_hat, Q = recursive_estimation(x_hat0, Q0, n)
    errors.append(LA.norm(x.squeeze() - x_hat.squeeze()))

plt.plot(Ns, errors)
plt.xlabel('Number of samples')
plt.ylabel('Error')
```

[Solution](./Non-Linear-Least-Squares-Solution.ipynb)

```python

```


# Basic (Gaussian likelihood) GP regression model

This notebook shows the different steps for creating and using a standard GP regression model, including:
  - reading and formatting data
  - choosing a kernel function
  - choosing a mean function (optional)
  - creating the model
  - viewing, getting, and setting model parameters
  - optimizing the model parameters
  - making predictions

We focus here on the implementation of the models in GPflow; for more intuition on these models, see [A Practical Guide to Gaussian Processes](https://drafts.distill.pub/gp/).

```python
import gpflow
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from gpflow.utilities import print_summary

# The lines below are specific to the notebook format
%matplotlib inline
plt.rcParams["figure.figsize"] = (12, 6)
```

`X` and `Y` denote the input and output values. **NOTE:** `X` and `Y` must be two-dimensional NumPy arrays, $N \times 1$ or $N \times D$, where $D$ is the number of input dimensions/features, with the same number of rows as $N$ (one for each data point):

```python
data = np.genfromtxt("data/regression_1D.csv", delimiter=",")
X = data[:, 0].reshape(-1, 1)
Y = data[:, 1].reshape(-1, 1)

_ = plt.plot(X, Y, "kx", mew=2)
```

We will consider the following probabilistic model:
\begin{equation}
Y_i = f(X_i) + \varepsilon_i\,,
\end{equation}
where $f \sim \mathcal{GP}(\mu(\cdot), k(\cdot, \cdot'))$, and $\varepsilon \sim \mathcal{N}(0, \tau^2 I)$.

## Choose a kernel
Several kernels (covariance functions) are implemented in GPflow. You can easily combine them to create new ones (see [Manipulating kernels](../advanced/kernels.ipynb)). You can also implement new covariance functions, as shown in the [Kernel design](../tailor/kernel_design.ipynb) notebook. Here, we will use a simple one:

```python
k = gpflow.kernels.Matern52()
```

For more advanced kernels see the [advanced kernel notebook](../advanced/kernels.ipynb) (including kernels defined on subspaces). A summary of the kernel can be obtained by

```python
print_summary(k)
```

    ╒═══════════════════════╤═══════════╤═════════════╤═════════╤═════════════╤═════════╤═════════╤═════════╕
    │ name                  │ class     │ transform   │ prior   │ trainable   │ shape   │ dtype   │   value │
    ╞═══════════════════════╪═══════════╪═════════════╪═════════╪═════════════╪═════════╪═════════╪═════════╡
    │ Matern52.variance     │ Parameter │ Softplus    │         │ True        │ ()      │ float64 │       1 │
    ├───────────────────────┼───────────┼─────────────┼─────────┼─────────────┼─────────┼─────────┼─────────┤
    │ Matern52.lengthscales │ Parameter │ Softplus    │         │ True        │ ()      │ float64 │       1 │
    ╘═══════════════════════╧═══════════╧═════════════╧═════════╧═════════════╧═════════╧═════════╧═════════╛

The Matern 5/2 kernel has two parameters: `lengthscales`, which encodes the "wiggliness" of the GP, and `variance`, which tunes the amplitude. They are both set to 1.0 as the default value. For more details on the meaning of the other columns, see [Manipulating kernels](../advanced/kernels.ipynb).

## Choose a mean function (optional)
It is common to choose $\mu = 0$, which is the GPflow default.

However, if there is a clear pattern (such as a mean value of `Y` that is far away from 0, or a linear trend in the data), mean functions can  be beneficial. Some simple ones are provided in the `gpflow.mean_functions` module.

Here's how to define a linear mean function:

`meanf = gpflow.mean_functions.Linear()`

## Construct a model
A GPflow model is created by instantiating one of the GPflow model classes, in this case GPR. We'll make a kernel `k` and instantiate a GPR object using the generated data and the kernel. We'll also set the variance of the likelihood to a sensible initial guess.

```python
m = gpflow.models.GPR(data=(X, Y), kernel=k, mean_function=None)
```

A summary of the model can be obtained by

```python
print_summary(m)
```

    ╒═════════════════════════╤═══════════╤══════════════════╤═════════╤═════════════╤═════════╤═════════╤═════════╕
    │ name                    │ class     │ transform        │ prior   │ trainable   │ shape   │ dtype   │   value │
    ╞═════════════════════════╪═══════════╪══════════════════╪═════════╪═════════════╪═════════╪═════════╪═════════╡
    │ GPR.kernel.variance     │ Parameter │ Softplus         │         │ True        │ ()      │ float64 │       1 │
    ├─────────────────────────┼───────────┼──────────────────┼─────────┼─────────────┼─────────┼─────────┼─────────┤
    │ GPR.kernel.lengthscales │ Parameter │ Softplus         │         │ True        │ ()      │ float64 │       1 │
    ├─────────────────────────┼───────────┼──────────────────┼─────────┼─────────────┼─────────┼─────────┼─────────┤
    │ GPR.likelihood.variance │ Parameter │ Softplus + Shift │         │ True        │ ()      │ float64 │       1 │
    ╘═════════════════════════╧═══════════╧══════════════════╧═════════╧═════════════╧═════════╧═════════╧═════════╛

The first two lines correspond to the kernel parameters, and the third one gives the likelihood parameter (the noise variance $\tau^2$ in our model).

You can access those values and manually set them to sensible initial guesses. For example:

```python
m.likelihood.variance.assign(0.01)
m.kernel.lengthscales.assign(0.3)
```

    <tf.Variable 'UnreadVariable' shape=() dtype=float64, numpy=-1.0502256128148466>

## Optimize the model parameters

To obtain meaningful predictions, you need to tune the model parameters (that is, the parameters of the kernel, the likelihood, and the mean function if applicable) to the data at hand.

There are several optimizers available in GPflow. Here we use the `Scipy` optimizer, which by default implements the L-BFGS-B algorithm. (You can select other algorithms by using the `method=` keyword argument to its `minimize` method; see [the SciPy documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html) for details of available options.)

```python
opt = gpflow.optimizers.Scipy()
```

In order to train the model, we need to maximize the log marginal likelihood.
GPflow models define a `training_loss` that can be passed to the `minimize`
method of an optimizer; in this case it is simply the negative log marginal
likelihood. We also need to specify the variables to train with
`m.trainable_variables`, and the number of iterations.

```python
opt_logs = opt.minimize(m.training_loss, m.trainable_variables, options=dict(maxiter=100))
print_summary(m)
```

    ╒═════════════════════════╤═══════════╤══════════════════╤═════════╤═════════════╤═════════╤═════════╤═══════════╕
    │ name                    │ class     │ transform        │ prior   │ trainable   │ shape   │ dtype   │     value │
    ╞═════════════════════════╪═══════════╪══════════════════╪═════════╪═════════════╪═════════╪═════════╪═══════════╡
    │ GPR.kernel.variance     │ Parameter │ Softplus         │         │ True        │ ()      │ float64 │ 7.96581   │
    ├─────────────────────────┼───────────┼──────────────────┼─────────┼─────────────┼─────────┼─────────┼───────────┤
    │ GPR.kernel.lengthscales │ Parameter │ Softplus         │         │ True        │ ()      │ float64 │ 0.212416  │
    ├─────────────────────────┼───────────┼──────────────────┼─────────┼─────────────┼─────────┼─────────┼───────────┤
    │ GPR.likelihood.variance │ Parameter │ Softplus + Shift │         │ True        │ ()      │ float64 │ 0.0057594 │
    ╘═════════════════════════╧═══════════╧══════════════════╧═════════╧═════════════╧═════════╧═════════╧═══════════╛

Notice how the value column has changed.

The local optimum found by Maximum Likelihood might not be the one you want (for example, it might be overfitting or oversmooth). This depends on the initial values of the hyperparameters, and is specific to each dataset. As an alternative to Maximum Likelihood, [Markov Chain Monte Carlo (MCMC)](../advanced/mcmc.ipynb) is also available.

## Make predictions

We can now use the model to make some predictions at the new points `Xnew`. You might be interested in predicting two different quantities: the latent function values `f(Xnew)` (the denoised signal), or the values of new observations `y(Xnew)` (signal + noise). Because we are dealing with Gaussian probabilistic models, the predictions typically produce a mean and variance as output. Alternatively, you can obtain samples of `f(Xnew)` or the log density of the new data points `(Xnew, Ynew)`.

GPflow models have several prediction methods:

 - `m.predict_f` returns the mean and marginal variance of $f$ at the points `Xnew`.

 - `m.predict_f` with argument `full_cov=True` returns the mean and the full covariance matrix of $f$ at the points `Xnew`.

 - `m.predict_f_samples` returns samples of the latent function.

 - `m.predict_y` returns the mean and variance of a new data point (that is, it includes the noise variance).

 - `m.predict_log_density` returns the log density of the observations `Ynew` at `Xnew`.

We use `predict_f` and `predict_f_samples` to plot 95% confidence intervals and samples from the posterior distribution.

```python
## generate test points for prediction
xx = np.linspace(-0.1, 1.1, 100).reshape(100, 1)  # test points must be of shape (N, D)

## predict mean and variance of latent GP at test points
mean, var = m.predict_f(xx)

## generate 10 samples from posterior
tf.random.set_seed(1)  # for reproducibility
samples = m.predict_f_samples(xx, 10)  # shape (10, 100, 1)

## plot
plt.figure(figsize=(12, 6))
plt.plot(X, Y, "kx", mew=2)
plt.plot(xx, mean, "C0", lw=2)
plt.fill_between(
    xx[:, 0],
    mean[:, 0] - 1.96 * np.sqrt(var[:, 0]),
    mean[:, 0] + 1.96 * np.sqrt(var[:, 0]),
    color="C0",
    alpha=0.2,
)

plt.plot(xx, samples[:, :, 0].numpy().T, "C0", linewidth=0.5)
_ = plt.xlim(-0.1, 1.1)
```

## GP regression in higher dimensions

Very little changes when the input space has more than one dimension. By default, the `lengthscales` is an isotropic (scalar) parameter. It is generally recommended that you allow to tune a different lengthscale for each dimension (Automatic Relevance Determination, ARD): simply initialize `lengthscales` with an array of length $D$ corresponding to the input dimension of `X`.  See [Manipulating kernels](../advanced/kernels.ipynb) for further information.

## Further reading

  - [Stochastic Variational Inference for scalability with SVGP](../advanced/gps_for_big_data.ipynb) for cases where there are a large number of observations.
  - [Ordinal regression](../advanced/ordinal_regression.ipynb) if the data is ordinal.
  - [Multi-output models and coregionalisation](../advanced/coregionalisation.ipynb) if `Y` is multidimensional.


# Single-objective acquisition functions
> A comparison of single-objective acquisition functions in Bayesian optimization

- toc: true
- categories: [bayesian optimization]
- image: images/acquisition-single.png

In Bayesian optimization the acquisition $\alpha(x)$ quantifies how useful a point $x$ is towards finding the minimum $y^* = \min_{x \in \mathcal{X}} f(x)$ of some black-box function $f$.
Hence, we want to maximize the acquisition function at each step $t=1 ... T$.
$D_t = \{(x_1, y_1) ... (x_t, y_t)\}$ is the data, of which $x^+$ is the location of the best point $y^+$ observed so far.

Evaluations of $f$ are often modeled by $y_t = f(x_t) + \epsilon_t$ where the $\epsilon_t \sim N(0,\sigma)$ and $f$ is a probabilistic surrogate model. In case of a Gaussian Process the posterior predictive distribution can be expressed in terms of a predictive mean $\mu$ and variance $\sigma^2$.

> Note: This post is work in progress.

For illustrating the different acquisition functions, let's consider an 1-dimensional Forrester function as the unknown black-box function which we wish to minimize.
We already sampled 4 points and use a GP with a Matern 5/2 kernel as surrogate model.

```python
#collapse-hide
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('png')
import scipy.stats
import GPy
import GPyOpt

# set up unknown test function
σ_noise = 0.5
np.random.seed(1337)
f_true = GPyOpt.objective_examples.experiments1d.forrester()
f_noisy = GPyOpt.objective_examples.experiments1d.forrester(sd=σ_noise)

X = np.linspace(0, 1, 201).reshape(201, 1)
Y = f_true.f(X)

# observed data
Xd = np.array([0.1, 0.5, 0.63, 0.92]).reshape([4, 1])
Yd = f_noisy.f(Xd)
best = Yd.min()

# fit Gaussian process
gp = GPy.models.GPRegression(Xd, Yd, kernel=GPy.kern.Matern52(1))
gp.optimize_restarts(10, verbose=0)

# GP posterior
μ, σ = gp.predict(X)
σ = σ ** 0.5

def plot(plot_true=False, acquisition=None):
    fig, ax = plt.subplots(1, figsize=(8, 5))
    ax.plot(Xd, Yd, 'ko', label='data')
    if plot_true:
        ax.plot(X, Y, 'r', label='true')
    ax.plot(X, μ, 'C0', label='predictive mean')
    ax.fill_between(X[:,0], (μ - σ)[:,0], (μ + σ)[:,0], alpha=.15, color='C0', label='predictive 68% interval')
    ax.set(xlabel='x', ylabel='y', xlim=(0, 1), ylim=(-7, 12))
    ax.legend(loc='upper left')
    if acquisition is not None:
        acquisition -= acquisition.min()
        acquisition /= acquisition.max()
        ax2 = ax.twinx()
        ax2.plot(X, acquisition, 'C1', label='acquisition')
        ax2.legend(loc='upper right')

plot(True)
```

## Confidence bound
With confidence bound (CB) {% cite Auer2003 Srinivas2009 %} the idea is to be optimistic and simply look for the best value within $k-$sigma off the mean. Thus for a minimization problem, we maximize
\begin{align}
\alpha_\mathrm{CB}(x) = - (\mu(x) - k \sigma(x))
\end{align}
where $\mu$ and $\sigma$ are the predictive mean and standard deviation, and $k$ controls the exploration.

```python
plot(acquisition=-(μ - σ))
```

## Probability of improvement
Probability of improvement (PI), {% cite Mockus1974 %} selects the point with the highest probability of any improvement with respect to $f^+$ the best value observed so far.
\begin{align}
\alpha_\mathrm{PI}(x) &= \mathbb{P}(I(x) > 0) = \Phi(u) \\
I(x) &= \max(0, f^+ - f(x)) \\
u &= (f^+ - \mu(x)) / \sigma(x)
\end{align}
where $\Phi$ is the standard normal cdf.

```python
u = (best - μ) / σ
PI = scipy.stats.norm.cdf(u)
plot(acquisition=PI)
```

## Expected improvement
Instead of looking for any kind of improvement ($I(x) > 0$), expected improvement (EI), {% cite Kushner1964 %}) takes the magnitude of the improvement $I(x)$ into account and maximize its expectation value:
\begin{align}
\alpha_\mathrm{EI}(x) = \mathbb{E} [I(x)] = (f^+ - \mu(x)) \Phi(x) + \sigma(x) \phi(x)
\end{align}
where $\Phi$ and $\phi$ are the standard normal cdf and pdf, respectively.

```python
u = (best - μ) / σ
cdf = scipy.stats.norm.cdf(u)
pdf = scipy.stats.norm.pdf(u)
EI = (best - μ) * cdf + σ * pdf

plot(acquisition=EI)
```

## Scaled expected improvement
Scaled expected improvement (SEI) {% cite Noe2018 %} is a recent modification that takes the uncertainty of the improvement quantifier $I(x)$ into account:
\begin{align}
\mathbb{V}[I(x)] = σ_f^2(x)((u^2 + 1) \Phi(u) + u \phi(u)) − \mathrm{EI}(x)^2
\end{align}
Scaling the expectation value of $I(x)$ by its uncertainty, we arrive at
\begin{align}
\alpha_\mathrm{SEI} = \mathbb{E}[I(x)] / \sqrt{\mathbb{V}[I(x)]}
\end{align}

Comment: At least in this example SEI looks very similar to PI as dividing by the uncertainty of the improvement seems to cancel out the exploration part in EI. The authors claim superior empirical performance compared to both EI and PI on the standard set of low-dimensional optimization functions however ...

```python
u = (best - μ) / σ
cdf = scipy.stats.norm.cdf(u)
pdf = scipy.stats.norm.pdf(u)
EI = (best - μ) * cdf + σ * pdf
V = σ**2 * ((u**2 + 1) * cdf + u * pdf) - EI**2
SEI = EI / V**0.5

plot(acquisition=SEI)
```

## Top-two expected improvement
...

```python

```

## Uncertainty sampling

Uncertainty sampling chooses candidates based on where the model is most uncertain.
In the single-objective case, this the point with the greatest predictive variance $\sigma_f^2$.
For the multi-objective case {% cite Rosario2019 %} propose the sum of the coefficients of variation in all objectives.
\begin{align}
\alpha(x) = \sum_m \sigma_m / \mu_m
\end{align}
Alternatively, with all objectives normalized to the unit range, we can simply sum up the $\sigma_m$.

```python
plot(acquisition=σ)
```

## Thompson sampling
With Thompson sampling (TS) we simply draw a sample of the posterior predictive distribution and optimize it.
For Gaussian processes there is no known way of sampling *functions*.
Hence, either a certain number of *points* $x$ sampled from the predictive distribution are considered for the optimiziation, or an approximate *function* sample is used.

A limitation for the former is that all previously sampled points contribute to the GP posterior when sampling further points, thus the computational complexity for iterative optimization methods grows with $n^3$, where $n$ is the number of iteratiations / evaluations.
A simpler but non-optimal alternative is brute-force optimization over a fixed number of simultaneously sampled points.

For approximate function samples a method called [spectral sampling](https://davidwalz.github.io/blog/gaussian%20process/bayesian%20optimization/2020/05/03/gp-spectral-sampling.html) is proposed in {% cite Hernandez2014 Bradford2018 %}.
The advantage of this method is that the function sample can be evaluated at fixed cost and thus is amenable to optimization.

```python
# Thompson sampling using a finite number of sampled points
μ, cov = gp.predict(X, full_cov=True)
rng = np.random.default_rng(seed=42)
y_sample = rng.multivariate_normal(μ.ravel(), cov)
# we would pick the

plot()
plt.plot(X, y_sample, color="C1");
```

## Entropy search
In a nutshell entropy search approximates the distribution of the global minimum and tries to best decrease it's entropy.
$\alpha_{ES}(x) = H(p(x^*|\mathcal{D})) - E[H(p(x^*|\mathcal{D} \cup \{x, y\}))]$ where $x^*$ is the argmax of the global optimum. Since $x^*$ is unknown we need to derive it's distribution via sampling from the posterior.

## Predictive entropy search
$\alpha(x) = H(p(y|D_t,x)) - E[H(p(y|D_t, x, x_*))]$

## Max-value entropy search
The idea is to keep the information-theoretic approach, but reducing the computational complexity by searching for the max $y^*$ instead of the argmax $x^*$.
$\alpha_\mathrm{MES}(x) = H(p(y|\mathcal{D},x)) - E[H(p(y|\mathcal{D}, x, y_*))]$ where in the left hand term $p$ is a Gaussian, an in the right hand term $p$ is a truncated Gaussian fullfilling $y \leq y^*$
If the value of $y^*$ is known MES has closed form. If it is unknow it needs to be sampled. Next to sampling from the posterior the authors propose using a Gumbel distribution as approximation.
$\alpha(x) = (y^* + |\mathcal{N}(0, \sigma)| - \mu_f(x)) / \sigma_f(x)$

## Knowledge gradient
{% cite Frazier2009 Scott2011}

## Mutual information
{% cite Contal2013 %}

## GLASSES
{% cite Gonzales2015 %}

## Estimation strategy
{% cite Wang2015 %}

##
{% cite Jiang2020 %}

## References

{% bibliography --cited %}