# Spherical harmonic models 1

```python
# Import notebook dependencies

import os
import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

from src import sha, mag, IGRF13_FILE
```

## Spherical harmonics and representing the geomagnetic field

The north (X), east (Y) and vertical (Z) (downwards) components of the internally-generated geomagnetic field at colatitude $\theta$, longitude $\phi$ and radial distance $r$ (in geocentric coordinates with reference radius $a=6371.2$ km for the IGRF) are written as follows:

$$\begin{align}
&X= \sum_{n=1}^N\left(\frac{a}{r}\right)^{n+2}\left[ g_n^0X_n^0+\sum_{m=1}^n\left( g_n^m\cos m\phi+h_n^m\sin m\phi \right)X_n^m\right]\\[6pt]
&Y= \sum_{n=1}^N\left(\frac{a}{r}\right)^{n+2} \sum_{m=1}^n \left(g_n^m\sin m\phi-h_n^m\cos m\phi \right)Y_n^m \\[6pt]
&Z= \sum_{n=1}^N\left(\frac{a}{r}\right)^{n+2} \left[g_n^0Z_n^0+\sum_{m=1}^n\left( g_n^m\cos m\phi+h_n^m\sin m\phi \right)Z_n^m\right]\\[6pt]
\text{with}&\\[6pt]
&X_n^m=\frac{dP_n^n}{d\theta}\\[6pt]
&Y_n^m=\frac{m}{\sin \theta}P_n^m \kern{10ex} \text{(Except at the poles where $Y_n^m=X_n^m\cos \theta$.)}\\[6pt]
&Z_n^m=-(n+1)P_n^m
\end{align}$$

where $n$ and $m$ are spherical harmonic degree and order, respectively, and the ($g_n^m, h_n^m$) are the Gauss coefficients for a particular model (e.g. the IGRF) of maximum degree $N$.

The Associated Legendre functions of degree $n$ and order $m$ are defined, in Schmidt semi-normalised form by

$$P^m_n(x) = \frac{1}{2^n n!}\left[ \frac{(2-\delta_{0m})(n-m)!\left(1-x^2\right)^m}{(n+m)!} \right]^{1/2}\frac{d^{n+m}}{dx^{n+m}}\left(1-x^2\right)^{n},$$

where $x = \cos(\theta)$.

Referring to Malin and Barraclough (1981), the recurrence relations

$$\begin{align}
P_n^n&=\left(1-\frac{1}{2n}\right)^{1/2}\sin \theta \thinspace P_{n-1}^{n-1} \\[6pt]
P_n^m&=\left[\left(2n-1\right) \cos \theta \thinspace P_{n-1}^m-\left[ \left(n-1\right)^2-m^2\right]^{1/2}P_{n-2}^m\right]\left(n^2-m^2\right)^{-1/2},\\[6pt]
\end{align}$$

and

$$\begin{align}
X_n^n&=\left(1-\frac{1}{2n}\right)^{1/2}\left( \sin \theta \thinspace X_{n-1}^{n-1}+ \cos \theta \thinspace P_{n-1}^{n-1} \right)\\[6pt]
X_n^m&=\left[\left(2n-1\right) \cos \theta \thinspace X_{n-1}^m- \sin \theta \thinspace P_{n-1}^m\right] - \left[ \left(n-1\right)^2-m^2\right]^{1/2}X_{n-2}^m\left(n^2-m^2\right)^{-1/2}.
\end{align}$$

may be used to calculate the $X^m_n$, $Y^m_n$ and $Z^m_n$, given $P_0^0=1$, $P_1^1=\sin(\theta)$, $X_0^0=0$ and $X_1^1=\cos(\theta)$.

## Plotting $P_n^m$ and $X_n^m$
The $P_n^m(\theta)$ and $X_n^m(\theta)$ are building blocks for computing geomagnetic field models given a spherical harmonic model. It's instructive to visualise these functions and below you can experiment by setting different values of spherical harmonic degree ($n$) and order ($m \le n$). Note how the choice of $n$ and $m$ affects the number of zeroes of the functions.

The functions are plotted on a semi-circle representing the surface of the Earth, with the inner core added for cosmetic purposes only! Again, purely for cosmetic purposes, the functions are scaled to fit within $\pm$10% of the Earth's surface.

**>> USER INPUT HERE: Set the spherical harmonic degree and order for the plot**

```python
degree = 13
order  = 7
```

```python
# Calculate Pnm and Xmn values every 0.5 degrees
colat   = np.linspace(0,180,361)
pnmvals = np.zeros(len(colat))
xnmvals = np.zeros(len(colat))

idx     = sha.pnmindex(degree,order)
for i, cl in enumerate(colat):
    p,x = sha.pxyznm_calc(degree, cl)[0:2]
    pnmvals[i] = p[idx]
    xnmvals[i] = x[idx]

theta   = np.deg2rad(colat)
ct      = np.cos(theta)
st      = np.sin(theta)

# Numbers mimicking the Earth's surface and outer core radii
e_rad   = 6.371
c_rad   = 3.485

# Scale values to fit within 10% of "Earth's surface". Firstly the P(n,m),
shell   = 0.1*e_rad
pmax    = np.abs(pnmvals).max()
pnmvals = pnmvals*shell/pmax + e_rad
xp      = pnmvals*st
yp      = pnmvals*ct

# and now the X(n,m)
xmax    = np.abs(xnmvals).max()
xnmvals = xnmvals*shell/xmax + e_rad
xx      = xnmvals*st
yx      = xnmvals*ct

# Values to draw the Earth's and outer core surfaces as semi-circles
e_xvals = e_rad*st
e_yvals = e_rad*ct
c_xvals = e_xvals*c_rad/e_rad
c_yvals = e_yvals*c_rad/e_rad

# Earth-like background framework for plots
def eplot(ax):
    ax.set_aspect('equal')
    ax.set_axis_off()
    ax.plot(e_xvals,e_yvals, color='blue')
    ax.plot(c_xvals,c_yvals, color='black')
    ax.fill_between(c_xvals, c_yvals, y2=0, color='lightgrey')
    ax.plot((0, 0), (-e_rad, e_rad), color='black')

# Plot the P(n,m) and X(n,m)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(18, 8))
fig.suptitle('Degree (n) = '+str(degree)+', order (m) = '+str(order), fontsize=20)

axes[0].plot(xp,yp, color='red')
axes[0].set_title('P('+ str(degree)+',' + str(order)+')', fontsize=16)
eplot(axes[0])

axes[1].plot(xx,yx, color='red')
axes[1].set_title('X('+ str(degree)+',' + str(order)+')', fontsize=16)
eplot(axes[1])
```

**Try again!**

## The International Geomagnetic Reference Field
The latest version of the IGRF is IGRF13 which consists of a main-field model every five years from 1900.0 to 2020.0 and a secular variation model for 2020-2025. The main field models have (maximum) spherical harmonic degree (n) and order (m) 10 up to 1995 and n=m=13 from 2000 onwards. The secular variation model has n=m=8.

The coefficients are first loaded into a pandas database:

```python
igrf13 = pd.read_csv(IGRF13_FILE, delim_whitespace=True,  header=3)
igrf13.head()  # Check the values have loaded correctly
```

### a) Calculating geomagnetic field values using the IGRF
The function below calculates geomagnetic field values at a point defined by its colatitude, longitude and altitude, using a spherical harmonic model of maximum degree _nmax_ supplied as an array _gh_. The parameter _coord_ is a string specifying whether the input position is in geocentric coordinates (when _altitude_ should be the geocentric distance in km) or geodetic coordinates (when altitude is distance above mean sea level in km).

(It's unconventional, but I've chosen to include a monopole term, set to zero, at index zero in the _gh_ array.)

```python
def shm_calculator(gh, nmax, altitude, colat, long, coord):
    RREF     = 6371.2 #The reference radius assumed by the IGRF
    degree   = nmax
    phi      = long

    if (coord == 'Geodetic'):
        # Geodetic to geocentric conversion using the WGS84 spheroid
        rad, theta, sd, cd = sha.gd2gc(altitude, colat)
    else:
        rad   = altitude
        theta = colat

    # Function 'rad_powers' to create an array with values of (a/r)^(n+2) for n = 0,1, 2 ..., degree
    rpow = sha.rad_powers(degree, RREF, rad)

    # Function 'csmphi' to create arrays with cos(m*phi), sin(m*phi) for m = 0, 1, 2 ..., degree
    cmphi, smphi = sha.csmphi(degree,phi)

    # Function 'gh_phi_rad' to create arrays with terms such as [g(3,2)*cos(2*phi) + h(3,2)*sin(2*phi)]*(a/r)**5
    ghxz, ghy = sha.gh_phi_rad(gh, degree, cmphi, smphi, rpow)

    # Function 'pnm_calc' to calculate arrays of the Associated Legendre Polynomials for n (&m) = 0,1, 2 ..., degree
    pnm, xnm, ynm, znm = sha.pxyznm_calc(degree, theta)

    # Geomagnetic field components are calculated as a dot product
    X =  np.dot(ghxz, xnm)
    Y =  np.dot(ghy,  ynm)
    Z =  np.dot(ghxz, znm)

    # Convert back to geodetic (X, Y, Z) if required
    if (coord == 'Geodetic'):
        t = X
        X = X*cd + Z*sd
        Z = Z*cd - t*sd

    return((X, Y, Z))
```

**>>  >> USER INPUT HERE: Set the input parameters**

```python
location = 'Erehwon'
ctype    = 'Geocentric' # coordinate type
altitude = 6371.2      # in km above the spheroid if ctype = 'Geodetic', radial distance if ctype = 'Geocentric'
colat    = 35          # NB colatitude, not latitude
long     = -3          # longitude
NMAX     = 13          # Maxiimum spherical harmonic degree of the model
date     = 2020.0      # Date for the field estimates
```

Now calculate the IGRF geomagnetic field estimates.

```python
# Calculate the gh values for the supplied date
if date == 2020.0:
    gh = igrf13['2020.0']
elif date < 2020.0:
    date_1 = (date//5)*5
    date_2 = date_1 + 5
    w1 = date-date_1
    w2 = date_2-date
    gh = np.array((w2*igrf13[str(date_1)] + w1*igrf13[str(date_2)])/(w1+w2))
elif date > 2020.0:
    gh =np.array(igrf13['2020.0'] + (date-2020.0)*igrf13['2020-25'])

gh = np.append(0., gh) # Add a zero monopole term corresponding to g(0,0)

bxyz = shm_calculator(gh, NMAX, altitude, colat, long, ctype)
dec, hoz ,inc , eff = mag.xyz2dhif(bxyz[0], bxyz[1], bxyz[2])

print('\nGeomagnetic field values at: ', location+', '+ str(date), '\n')
print('Declination (D):', '{: .1f}'.format(dec), 'degrees')
print('Inclination (I):', '{: .1f}'.format(inc), 'degrees')
print('Horizontal intensity (H):', '{: .1f}'.format(hoz), 'nT')
print('Total intensity (F)     :', '{: .1f}'.format(eff), 'nT')
print('North component (X)     :', '{: .1f}'.format(bxyz[0]), 'nT')
print('East component (Y)      :', '{: .1f}'.format(bxyz[1]), 'nT')
print('Vertical component (Z)  :', '{: .1f}'.format(bxyz[2]), 'nT')
```

### b) Maps of the IGRF
Now draw maps of the IGRF at the date selected above. The latitude range is set at -85 degrees to +85 degrees and the longitude range -180 degrees to +180 degrees and IGRF values for (X, Y, Z) are calculated on a 5 degree grid (this may take a few seconds to complete).

**>>  >> USER INPUT HERE:  Set the element to plot**

Enter the geomagnetic element to plot below:

D = declination

H = horizontal intensity

I = inclination

X = north component

Y = east component

Z = vertically (downwards) component

F = total intensity.)

```python
el2plot = 'H'
```

```python
def IGRF_plotter(el_name, vals, date):
    if el_name=='D':
        cvals = np.arange(-25,30,5)
    else:
        cvals = 15
    fig, ax = plt.subplots(figsize=(16, 8))
    cplt = ax.contour(longs, lats, vals, levels=cvals)
    ax.clabel(cplt, cplt.levels, inline=True, fmt='%d', fontsize=10)
    ax.set_title('IGRF: '+ el_name + ' (' + str(date) + ')', fontsize=20)
    ax.set_xlabel('Longitude', fontsize=16)
    ax.set_ylabel('Latitude', fontsize=16)
```

```python
longs  = np.linspace(-180, 180, 73)
lats   = np.linspace(-85, 85, 35)
Bx, By, Bz = zip(*[sha.shm_calculator(gh,13,6371.2,90-lat,lon,'Geocentric') \
                 for lat in lats for lon in longs])
X = np.asarray(Bx).reshape(35,73)
Y = np.asarray(By).reshape(35,73)
Z = np.asarray(Bz).reshape(35,73)
D, H, I, F = [mag.xyz2dhif(X, Y, Z)[el] for el in range(4)]

el_dict={'X':X, 'Y':Y, 'Z':Z, 'D':D, 'H':H, 'I':I, 'F':F}
IGRF_plotter(el2plot, el_dict[el2plot], date)
```

### Exercise
Produce a similar plot for the secular variation from 2020 to 2025 using the IGRF.

### References

Malin, S. R. . and Barraclough, D., (1981). An algorithm for synthesizing the geomagnetic field, Computers & Geosciences. Pergamon, 7(4), pp. 401–405. doi: 10.1016/0098-3004(81)90082-0.


# "[Draft] Optimizer Code Supplement Using fastai"
> "Building the Optimizers explained by Sebastian Ruder using fastai"

- toc: true
- branch: master
- badges: true
- comments: true
- author: Kevin Bird
- categories: [fastai]

## Introduction

This post is meant to be a supporting guide that can be used with the incredibly detailed [post from Sebastian Ruder](https://ruder.io/optimizing-gradient-descent/index.html).  The goal is to implement each of these optimizers using fastai and run them on a small problem.

The sample problem that will be used is the one created by Jeremy and Rachel in the graddesc spreadsheet.

Because this post is generated using Jupyter Notebooks, you can also interact with any of these concepts and modify them as you see fit.  In as many places as possible, I will try to connect the code back to the concepts by utilizing the greek letters Sebastian (and usually the paper) uses

```python
#hide
from fastai.vision.all import *
```

```python
a_real = 2
b_real = 30
```

`a_real` is a weight and `b_real` is a bias they form the equation $a\_real*x + b\_real= y$
Both a and b are parameters and will be learned by our optimizer through the loss function.
In a real world problem, these values would be unknown and there would typically be a lot more weights, but the concept will remain the same whether there are 2 parameters or [175 billion](https://www.google.com/search?client=firefox-b-1-d&q=how+many+parameters+does+gpt+3+have).

```python
#hide
class MySimpleModel(torch.nn.Module):
    def __init__(self):
        super(MySimpleModel, self).__init__()
        self.linear = nn.Linear(1,1)
        self.linear.weight.data = nn.Parameter(torch.tensor([[1.0]]))
        self.linear.bias.data = nn.Parameter(torch.tensor([1.0]))
        self.a = self.linear.weight
        self.b = self.linear.bias

    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.linear(x)
        return x
```

```python
#hide
class OptimizerInsight(Callback):
    def before_batch(self):
        if self.training:
            self.a_guess+=[float(self.model.a)]
            self.b_guess+=[float(self.model.b)]
            #plt.plot(theta_real,b_real, 'ro', markersize=5)
            #print(self.model.theta.tolist())
    def before_epoch(self):
        try:
            self.a_guess = [self.a_guess[-1]]
        except:
            self.a_guess = []
        try:
            self.b_guess = [self.b_guess[-1]]
        except:
            self.b_guess = []

    def after_epoch(self):
        plt.plot(self.a_guess, self.b_guess)
```

```python
#hide
x_list = torch.tensor([14.0,86,28,51,28,29,72,62,84,15,42,62,47,35,9,38,44,99,13,21,28,20,8,64,99,70,27,17,8])
y_list = torch.tensor([58.0,202,86,132,86,88,174,154,198,60,114,154,124,100,48,106,118,228,56,72,86,70,46,158,228,170,84,64,46])
b_list = [(x,y) for x,y in zip(x_list, y_list)]
```

```python
#hide
dl = DataLoader(b_list, bs=4)
#setting validation set equal to training dataset.  For this example, we are only interested in the training dataset
dls = DataLoaders(dl,dl)
```

## Gradient Descent Variants

### [Batch Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.html#batchgradientdescent)

Forumula:
$\theta = \theta - \eta \cdot \nabla_\theta J( \theta)$

Code:
```python
def sgd_step(p, lr, **kwargs):
    p.data.add_(p.grad.data, alpha=-lr)
```

Translation:
  $\theta$ = p
  $\eta$ = lr
  $\nabla_\theta J( \theta)$ = p.grad.data

```python
def _do_epoch_BGD(self):
    self._do_epoch_train()
    self._step()
    self('after_step')
    self.opt.zero_grad()
    self._do_epoch_validate()
```

```python
def _do_one_batch_BGD(self):
    self.pred = self.model(*self.xb)
    self('after_pred')
    if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    self('after_loss')
    if not self.training or not len(self.yb): return
    self('before_backward')
    self._backward()
    self('after_backward')
```

```python
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.00005
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_BGD,learn)
learn._do_one_batch = partial(_do_one_batch_BGD,learn)
```

```python
learn.fit(10)
```

### [Stochastic Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.html#stochasticgradientdescent)

Formula:
$\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i)}; y^{(i)})$

Code:
```python
def sgd_step(p, lr, **kwargs):
    p.data.add_(p.grad.data, alpha=-lr)
```

Translation:
  $\theta$ = p
  $\eta$ = lr
  $\nabla_\theta J( \theta; x^{(i)}; y^{(i)})$ = p.grad.data

```python
def _do_epoch_SGD(self):
    self._do_epoch_train()
    self._do_epoch_validate()
```

```python
def _do_one_batch_SGD(self):
    #pdb.set_trace()
    for i,b in enumerate(zip(self.xb[0][None].T,self.yb[0][None].T)):
        x,y = b
        self.pred = self.model(x)
        self('after_pred')
        if len(y): self.loss = self.loss_func(self.pred, y)
        self('after_loss')
        if not self.training or not len(self.yb): return
        self('before_backward')
        self._backward()
        self('after_backward')
        self._step()
        self('after_step')
        self.opt.zero_grad()
```

```python
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.0001
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_SGD,learn)
learn._do_one_batch = partial(_do_one_batch_SGD,learn)
```

```python
learn.fit(10)
```

### [Mini-batch Gradient Descent](https://ruder.io/optimizing-gradient-descent/index.html#minibatchgradientdescent)

$\theta = \theta - \eta \cdot \nabla_\theta J( \theta; x^{(i:i+n)}; y^{(i:i+n)})$

Comments about Batch Gradient Descent, Stochastic Gradient Descent, and Mini-batch Gradient Descent:
All three of these use itentical code for the optimizer itself.  The huge difference between these three Optimizers is when the step actually occurs.

```python
def _do_epoch_MBGD(self):
    self._do_epoch_train()
    self._do_epoch_validate()
```

```python
def _do_one_batch_MBGD(self):
    self.pred = self.model(*self.xb)
    self('after_pred')
    if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
    self('after_loss')
    if not self.training or not len(self.yb): return
    self('before_backward')
    self._backward()
    self('after_backward')
    self._step()
    self('after_step')
    self.opt.zero_grad()
```

```python
model = MySimpleModel()
loss_func = mse #F.mse_loss
𝜂 = 0.0001
partial_learner = partial(Learner,dls, model, loss_func, cbs=[OptimizerInsight])
learn = partial_learner(SGD, lr=𝜂)
learn._do_epoch = partial(_do_epoch_MBGD,learn)
learn._do_one_batch = partial(_do_one_batch_MBGD,learn)
```

```python
learn.fit(10)
```

The SGD Optimizer can do all three SGD($\theta$, $\eta$) above with the same parameters.

**Note**: `mom`, `wd`, and `decouple_wd` are all 0 or don't influence things with their default values.

## Beyond Vanilla Gradient Descent

### [SGD with Momentum](https://ruder.io/optimizing-gradient-descent/index.html#momentum)

$
\begin{align}
\begin{split}
v_t &= \gamma v_{t-1} + \eta \nabla_\theta J( \theta) \\
\theta &= \theta - v_t
\end{split}
\end{align}
$

```python

```

### [Nesterov Accelerated Gradient](https://ruder.io/optimizing-gradient-descent/index.html#nesterovacceleratedgradient)

$
\begin{align}
\begin{split}
v_t &= \gamma v_{t-1} + \eta \nabla_\theta J( \theta - \gamma v_{t-1} ) \\
\theta &= \theta - v_t
\end{split}
\end{align}
$

```python

```

### [Adagrad](https://ruder.io/optimizing-gradient-descent/index.html#adagrad)

$g_{t, i} = \nabla_\theta J( \theta_{t, i} )$
$\theta_{t+1, i} = \theta_{t, i} - \eta \cdot g_{t, i}$
$\theta_{t+1, i} = \theta_{t, i} - \dfrac{\eta}{\sqrt{G_{t, ii} + \epsilon}} \cdot g_{t, i}$
$\theta_{t+1} = \theta_{t} - \dfrac{\eta}{\sqrt{G_{t} + \epsilon}} \odot g_{t}$

```python

```

### [Adadelta](https://ruder.io/optimizing-gradient-descent/index.html#adadelta)

$E[g^2]_t = \gamma E[g^2]_{t-1} + (1 - \gamma) g^2_t$
$
\begin{align}
\begin{split}
\Delta \theta_t &= - \eta \cdot g_{t, i} \\
\theta_{t+1} &= \theta_t + \Delta \theta_t \end{split}
\end{align}
$
$\Delta \theta_t = - \dfrac{\eta}{\sqrt{G_{t} + \epsilon}} \odot g_{t}$
$\Delta \theta_t = - \dfrac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_{t}$
$\Delta \theta_t = - \dfrac{\eta}{RMS[g]_{t}} g_t$
$E[\Delta \theta^2]_t = \gamma E[\Delta \theta^2]_{t-1} + (1 - \gamma) \Delta \theta^2_t$
$RMS[\Delta \theta]_{t} = \sqrt{E[\Delta \theta^2]_t + \epsilon}$
$
\begin{align}
\begin{split}
\Delta \theta_t &= - \dfrac{RMS[\Delta \theta]_{t-1}}{RMS[g]_{t}} g_{t} \\
\theta_{t+1} &= \theta_t + \Delta \theta_t
\end{split}
\end{align}
$

```python

```

### [RMSprop](https://ruder.io/optimizing-gradient-descent/index.html#rmsprop)

```python

```

```python

```

### [Adam](https://ruder.io/optimizing-gradient-descent/index.html#adam)

```python

```

```python

```

### [AdaMax](https://ruder.io/optimizing-gradient-descent/index.html#adamax)

```python

```

```python

```

### [Nadam](https://ruder.io/optimizing-gradient-descent/index.html#nadam)

```python

```

```python

```

### [AMSGrad](https://ruder.io/optimizing-gradient-descent/index.html#amsgrad)

```python

```

```python

```

### AdamW

```python

```

```python

```

### QHAdam

```python

```

```python

```

### AggMo

```python

```

```python

```


# SymPy - Symbolic algebra in Python

J.R. Johansson (jrjohansson at gmail.com)

The latest version of this [IPython notebook](http://ipython.org/notebook.html) lecture is available at [http://github.com/jrjohansson/scientific-python-lectures](http://github.com/jrjohansson/scientific-python-lectures).

The other notebooks in this lecture series are indexed at [http://jrjohansson.github.io](http://jrjohansson.github.io).

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

## Introduction

There are two notable Computer Algebra Systems (CAS) for Python:

* [SymPy](http://sympy.org/en/index.html) - A python module that can be used in any Python program, or in an IPython session, that provides powerful CAS features.
* [Sage](http://www.sagemath.org/) - Sage is a full-featured and very powerful CAS enviroment that aims to provide an open source system that competes with Mathematica and Maple. Sage is not a regular Python module, but rather a CAS environment that uses Python as its programming language.

Sage is in some aspects more powerful than SymPy, but both offer very comprehensive CAS functionality. The advantage of SymPy is that it is a regular Python module and integrates well with the IPython notebook.

In this lecture we will therefore look at how to use SymPy with IPython notebooks. If you are interested in an open source CAS environment I also recommend to read more about Sage.

To get started using SymPy in a Python program or notebook, import the module `sympy`:

```python
from sympy import *
```

To get nice-looking $\LaTeX$ formatted output run:

```python
init_printing()

# or with older versions of sympy/ipython, load the IPython extension
#%load_ext sympy.interactive.ipythonprinting
# or
#%load_ext sympyprinting
```

## Symbolic variables

In SymPy we need to create symbols for the variables we want to work with. We can create a new symbol using the `Symbol` class:

```python
x = Symbol('x')
```

```python
(pi + x)**2
```

```python
# alternative way of defining symbols
a, b, c = symbols("a, b, c")
```

```python
type(a)
```

    sympy.core.symbol.Symbol

We can add assumptions to symbols when we create them:

```python
x = Symbol('x', real=True)
```

```python
x.is_imaginary
```

    False

```python
x = Symbol('x', positive=True)
```

```python
x > 0
```

### Complex numbers

The imaginary unit is denoted `I` in SymPy.

```python
1+1*I
```

```python
I**2
```

```python
(x * I + 1)**2
```

### Rational numbers

There are three different numerical types in SymPy: `Real`, `Rational`, `Integer`:

```python
r1 = Rational(4,5)
r2 = Rational(5,4)
```

```python
r1
```

```python
r1+r2
```

```python
r1/r2
```

## Numerical evaluation

SymPy uses a library for arbitrary precision as numerical backend, and has predefined SymPy expressions for a number of mathematical constants, such as: `pi`, `e`, `oo` for infinity.

To evaluate an expression numerically we can use the `evalf` function (or `N`). It takes an argument `n` which specifies the number of significant digits.

```python
pi.evalf(n=50)
```

```python
y = (x + pi)**2
```

```python
N(y, 5) # same as evalf
```

When we numerically evaluate algebraic expressions we often want to substitute a symbol with a numerical value. In SymPy we do that using the `subs` function:

```python
y.subs(x, 1.5)
```

```python
N(y.subs(x, 1.5))
```

The `subs` function can of course also be used to substitute Symbols and expressions:

```python
y.subs(x, a+pi)
```

We can also combine numerical evaluation of expressions with NumPy arrays:

```python
import numpy
```

```python
x_vec = numpy.arange(0, 10, 0.1)
```

```python
y_vec = numpy.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

```python
fig, ax = plt.subplots()
ax.plot(x_vec, y_vec);
```

However, this kind of numerical evaluation can be very slow, and there is a much more efficient way to do it: Use the function `lambdify` to "compile" a SymPy expression into a function that is much more efficient to evaluate numerically:

```python
f = lambdify([x], (x + pi)**2, 'numpy')  # the first argument is a list of variables that
                                         # f will be a function of: in this case only x -> f(x)
```

```python
y_vec = f(x_vec)  # now we can directly pass a numpy array and f(x) is efficiently evaluated
```

The speedup when using "lambdified" functions instead of direct numerical evaluation can be significant, often several orders of magnitude. Even in this simple example we get a significant speed up:

```python
%%timeit

y_vec = numpy.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

    10 loops, best of 3: 28.2 ms per loop

```python
%%timeit

y_vec = f(x_vec)
```

    The slowest run took 8.86 times longer than the fastest. This could mean that an intermediate result is being cached
    100000 loops, best of 3: 2.93 µs per loop

## Algebraic manipulations

One of the main uses of an CAS is to perform algebraic manipulations of expressions. For example, we might want to expand a product, factor an expression, or simplify an expression. The functions for doing these basic operations in SymPy are demonstrated in this section.

### Expand and factor

The first steps in an algebraic manipulation

```python
(x+1)*(x+2)*(x+3)
```

```python
expand((x+1)*(x+2)*(x+3))
```

The `expand` function takes a number of keywords arguments which we can tell the functions what kind of expansions we want to have performed. For example, to expand trigonometric expressions, use the `trig=True` keyword argument:

```python
sin(a+b)
```

```python
expand(sin(a+b), trig=True)
```

See `help(expand)` for a detailed explanation of the various types of expansions the `expand` functions can perform.

The opposite of product expansion is of course factoring. To factor an expression in SymPy use the `factor` function:

```python
factor(x**3 + 6 * x**2 + 11*x + 6)
```

### Simplify

The `simplify` tries to simplify an expression into a nice looking expression, using various techniques. More specific alternatives to the `simplify` functions also exists: `trigsimp`, `powsimp`, `logcombine`, etc.

The basic usages of these functions are as follows:

```python
# simplify (sometimes) expands a product
simplify((x+1)*(x+2)*(x+3))
```

```python
# simplify uses trigonometric identities
simplify(sin(a)**2 + cos(a)**2)
```

```python
simplify(cos(x)/sin(x))
```

### apart and together

To manipulate symbolic expressions of fractions, we can use the `apart` and `together` functions:

```python
f1 = 1/((a+1)*(a+2))
```

```python
f1
```

```python
apart(f1)
```

```python
f2 = 1/(a+2) + 1/(a+3)
```

```python
f2
```

```python
together(f2)
```

Simplify usually combines fractions but does not factor:

```python
simplify(f2)
```

## Calculus

In addition to algebraic manipulations, the other main use of CAS is to do calculus, like derivatives and integrals of algebraic expressions.

### Differentiation

Differentiation is usually simple. Use the `diff` function. The first argument is the expression to take the derivative of, and the second argument is the symbol by which to take the derivative:

```python
y
```

```python
diff(y**2, x)
```

For higher order derivatives we can do:

```python
diff(y**2, x, x)
```

```python
diff(y**2, x, 2) # same as above
```

To calculate the derivative of a multivariate expression, we can do:

```python
x, y, z = symbols("x,y,z")
```

```python
f = sin(x*y) + cos(y*z)
```

$\frac{d^3f}{dxdy^2}$

```python
diff(f, x, 1, y, 2)
```

## Integration

Integration is done in a similar fashion:

```python
f
```

```python
integrate(f, x)
```

By providing limits for the integration variable we can evaluate definite integrals:

```python
integrate(f, (x, -1, 1))
```

and also improper integrals

```python
integrate(exp(-x**2), (x, -oo, oo))
```

Remember, `oo` is the SymPy notation for inifinity.

### Sums and products

We can evaluate sums using the function `Sum`:

```python
n = Symbol("n")
```

```python
Sum(1/n**2, (n, 1, 10))
```

```python
Sum(1/n**2, (n,1, 10)).evalf()
```

```python
Sum(1/n**2, (n, 1, oo)).evalf()
```

Products work much the same way:

```python
Product(n, (n, 1, 10)) # 10!
```

## Limits

Limits can be evaluated using the `limit` function. For example,

```python
limit(sin(x)/x, x, 0)
```

We can use 'limit' to check the result of derivation using the `diff` function:

```python
f
```

```python
diff(f, x)
```

$\displaystyle \frac{\mathrm{d}f(x,y)}{\mathrm{d}x} = \frac{f(x+h,y)-f(x,y)}{h}$

```python
h = Symbol("h")
```

```python
limit((f.subs(x, x+h) - f)/h, h, 0)
```

OK!

We can change the direction from which we approach the limiting point using the `dir` keywork argument:

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

## Series

Series expansion is also one of the most useful features of a CAS. In SymPy we can perform a series expansion of an expression using the `series` function:

```python
series(exp(x), x)
```

By default it expands the expression around $x=0$, but we can expand around any value of $x$ by explicitly include a value in the function call:

```python
series(exp(x), x, 1)
```

And we can explicitly define to which order the series expansion should be carried out:

```python
series(exp(x), x, 1, 10)
```

The series expansion includes the order of the approximation, which is very useful for keeping track of the order of validity when we do calculations with series expansions of different order:

```python
s1 = cos(x).series(x, 0, 5)
s1
```

```python
s2 = sin(x).series(x, 0, 2)
s2
```

```python
expand(s1 * s2)
```

If we want to get rid of the order information we can use the `removeO` method:

```python
expand(s1.removeO() * s2.removeO())
```

But note that this is not the correct expansion of $\cos(x)\sin(x)$ to $5$th order:

```python
(cos(x)*sin(x)).series(x, 0, 6)
```

## Linear algebra

### Matrices

Matrices are defined using the `Matrix` class:

```python
m11, m12, m21, m22 = symbols("m11, m12, m21, m22")
b1, b2 = symbols("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]])
A
```

```python
b = Matrix([[b1], [b2]])
b
```

With `Matrix` class instances we can do the usual matrix algebra operations:

```python
A**2
```

```python
A * b
```

And calculate determinants and inverses, and the like:

```python
A.det()
```

```python
A.inv()
```

## Solving equations

For solving equations and systems of equations we can use the `solve` function:

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 - x**2 - 1, x)
```

System of equations:

```python
solve([x + y - 1, x - y - 1], [x,y])
```

In terms of other symbolic expressions:

```python
solve([x + y - a, x - y - c], [x,y])
```

## Further reading

* http://sympy.org/en/index.html - The SymPy projects web page.
* https://github.com/sympy/sympy - The source code of SymPy.
* http://live.sympy.org - Online version of SymPy for testing and demonstrations.

## Versions

```python
%reload_ext version_information

%version_information numpy, matplotlib, sympy
```

Software	Version

Python	2.7.10 64bit [GCC 4.2.1 (Apple Inc. build 5577)]

IPython	3.2.1

OS	Darwin 14.1.0 x86_64 i386 64bit

numpy	1.9.2

matplotlib	1.4.3

sympy	0.7.6

Sat Aug 15 11:37:37 2015 JST


# Exercício 2.1
Repetindo o meu conselho do capítulo anterior, sempre que você aprender um recurso novo, você deve testá-lo no modo interativo e fazer erros de propósito para ver o que acontece.

Vimos que n = 42 é legal. E 42 = n?

```python
n = 42
```

```python
print(n)
```

    42

```python
42 = n
```

Ou x = y = 1?

```python
x = y = 1
```

```python
print(x)
print(y)
```

    1
    1

Em algumas linguagens, cada instrução termina em um ponto e vírgula ;. O que acontece se você puser um ponto e vírgula no fim de uma instrução no Python?

```python
n = 42;
print(type(n))
```

    <class 'int'>

```python
n = 42.
print(type(n))
```

    <class 'float'>

E se puser um ponto no fim de uma instrução?

```python
n = '42'.
print(type(n))
```

Em notação matemática é possível multiplicar x e y desta forma: xy. O que acontece se você tentar fazer o mesmo no Python?

```python
x = 4
y = 2
print(xy)
```

# Exercício 2.2
Pratique o uso do interpretador do Python como uma calculadora:

O volume de uma esfera com raio r é:
\begin{align}
\frac{4}{3} \pi r³
\end{align} Qual é o volume de uma esfera com raio 5?

```python
from math import pi
r = 5
volume_esfera = (4/3) * r * (pow(pi, 3))
print(f'{volume_esfera:.2f}')
```

    206.71

Suponha que o preço de capa de um livro seja R\\$ 24,95, mas as livrarias recebem um desconto de 40 \%. O transporte custa R$ 3,00 para o primeiro exemplar e 75 centavos para cada exemplar adicional. Qual é o custo total de atacado para 60 cópias?

```python
preco_de_capa = 24.95
desconto = .40
exemplar_unitario = 3.
exemplar_adicional = .75

numero_de_copias = 60

total = (preco_de_capa * numero_de_copias * desconto) + (exemplar_unitario + (exemplar_adicional * numero_de_copias - 1))

print(f'{total:.2f}')
```

    645.80

Se eu sair da minha casa às 6:52 e correr 1 quilômetro a um certo passo (8min15s por quilômetro), então 3 quilômetros a um passo mais rápido (7min12s por quilômetro) e 1 quilômetro no mesmo passo usado em primeiro lugar, que horas chego em casa para o café da manhã?

```python
velocidade_1 = ((8 * 60) + 15) * 2
velocidade_2 = ((7 * 60) + 12) * 3

tempo = (velocidade_1 + velocidade_2) / 60
horario_de_saida = (6 * 60) + 52

horario_de_chegada = (horario_de_saida + tempo) / 60
h, m = f'{horario_de_chegada:.2f}'.split('.')
print(f'Horário do café: {h}:{m}')
```

    Horário do café: 7:50

```python

```


# SymPy tutorial

**SymPy** is a Python package for performing **symbolic mathematics**
which can perform algebra, integrate and differentiate equations,
find solutions to differential equations, and *numerically solve
messy equations* -- along other uses.

CHANGE LOG

    2017-06-12  First revision since 2015-12-26.

Let's import sympy and initialize its pretty print functionality
which will print equations using LaTeX.
Jupyter notebooks uses Mathjax to render equations
so we specify that option.

```python
import sympy as sym
sym.init_printing(use_latex='mathjax')

#  If you were not in a notebook environment,
#  but working within a terminal, use:
#
#  sym.init_printing(use_unicode=True)
```

## Usage

These sections are illustrated with examples drawn from
[rlabbe](https://github.com/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/Appendix-A-Installation.ipynb) from his appendix for Kalman Filters.

It is important to distinguish a Python variable
from a **declared symbol** in sympy.

```python
phi, x = sym.symbols('\phi, x')

#  x here is a sympy symbol, and we form a list:
[ phi, x ]
```

$$\left [ \phi, \quad x\right ]$$

Notice how we used a LaTeX expression for the symbol `phi`.
This is not necessary, but if you do the output will render nicely as LaTeX.

Also notice how $x$ did not have a numerical value for the list to evaluate.

So what is the **derivative** of $\sqrt{\phi}$ ?

```python
sym.diff('sqrt(phi)')
```

$$\frac{1}{2 \sqrt{\phi}}$$

We can **factor** equations:

```python
sym.factor( phi**3 - phi**2 + phi - 1 )
```

$$\left(\phi - 1\right) \left(\phi^{2} + 1\right)$$

and we can **expand** them:

```python
((phi+1)*(phi-4)).expand()
```

$$\phi^{2} - 3 \phi - 4$$

You can also use strings for equations that use symbols that you have not defined:

```python
x = sym.expand('(t+1)*2')
x
```

$$2 t + 2$$

## Symbolic solution

Now let's use sympy to compute the **Jacobian** of a matrix.
Suppose we have a function,

$$h=\sqrt{(x^2 + z^2)}$$

for which we want to find the Jacobian with respect to x, y, and z.

```python
x, y, z = sym.symbols('x y z')

H = sym.Matrix([sym.sqrt(x**2 + z**2)])

state = sym.Matrix([x, y, z])

H.jacobian(state)
```

$$\left[\begin{matrix}\frac{x}{\sqrt{x^{2} + z^{2}}} & 0 & \frac{z}{\sqrt{x^{2} + z^{2}}}\end{matrix}\right]$$

Now let's compute the discrete process noise matrix $\mathbf{Q}_k$ given the continuous process noise matrix
$$\mathbf{Q} = \Phi_s \begin{bmatrix}0&0&0\\0&0&0\\0&0&1\end{bmatrix}$$

and the equation

$$\mathbf{Q} = \int_0^{\Delta t} \Phi(t)\mathbf{Q}\Phi^T(t) dt$$

where
$$\Phi(t) = \begin{bmatrix}1 & \Delta t & {\Delta t}^2/2 \\ 0 & 1 & \Delta t\\ 0& 0& 1\end{bmatrix}$$

```python
dt = sym.symbols('\Delta{t}')

F_k = sym.Matrix([[1, dt, dt**2/2],
                  [0,  1,      dt],
                  [0,  0,      1]])

Q = sym.Matrix([[0,0,0],
                [0,0,0],
                [0,0,1]])

sym.integrate(F_k*Q*F_k.T,(dt, 0, dt))
```

$$\left[\begin{matrix}\frac{\Delta{t}^{5}}{20} & \frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{6}\\\frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{3} & \frac{\Delta{t}^{2}}{2}\\\frac{\Delta{t}^{3}}{6} & \frac{\Delta{t}^{2}}{2} & \Delta{t}\end{matrix}\right]$$

## Numerical solution

You can find the *numerical value* of an equation by substituting in a value for a variable:

```python
x = sym.symbols('x')

w = (x**2) - (3*x) + 4
w.subs(x, 4)
```

$$8$$

Typically we want a numerical solution where the analytic solution is messy,
that is, we want a **solver**.
This is done by specifying a sympy equation, for example:

```python
LHS = (x**2) - (8*x) + 15
RHS = 0
#  where both RHS and LHS can be complicated expressions.

solved = sym.solveset( sym.Eq(LHS, RHS), x, domain=sym.S.Reals )
#  Notice how the domain solution can be specified.

solved
#  A set of solution(s) is returned.
```

$$\left\{3, 5\right\}$$

```python
#  Testing whether any solution(s) were found:
if solved != sym.S.EmptySet:
    print("Solution set was not empty.")
```

    Solution set was not empty.

```python
#  sympy sets are not like the usual Python sets...
type(solved)
```

    sympy.sets.sets.FiniteSet

```python
#  ... but can easily to converted to a Python list:
l = list(solved)
print( l, type(l) )
```

    ([3, 5], <type 'list'>)

```python
LHS = (x**2)
RHS = -4
#  where both RHS and LHS can be complicated expressions.

solved = sym.solveset( sym.Eq(LHS, RHS), x )
#  Leaving out the domain will include the complex domain.

solved
```

$$\left\{- 2 i, 2 i\right\}$$

## Application to financial economics

We used sympy to deduce parameters of Gaussian mixtures
in module `lib/ys_gauss_mix.py` and the explanatory notebook
is rendered at https://git.io/gmix


# Solid Angle subtended by a sphere Approximation

We want to approximate the value of the solid angle subtended by a sphere as seen from any point in space outside of that sphere. This expression will be used in ToFu to compute the radiated power received by a particle of arbitrary radius (small vs plasma volume discretization) from the whole plasma. The expression will allow faster computation.

## Notations

Let’s consider the case of a spherical particle of radius $r$, observed from point $M$ located at a distance $d$ from the center $C$ of the particle, as illustrated in the figure above. By definition, the solid angle $\Omega = \dfrac{S}{d^2}$ , where $S$ is the surface on the sphere of center $M$ intersecting the particle center $C$ and limited by its radius, as represented in the figure below.

## Solid Angle approximation
In our case, we get

$$\Omega = 2\pi \left( 1 - \sqrt{1-\left(\dfrac{r}{d}\right)^2}\right)$$

However, the particle radius is almost always much smaller than the distance between the particle and the observation point $M$. Thus, often $$\dfrac{r}{d} = X \xrightarrow[]{} 0$$

The taylor series of the function $\Omega(X) = 2\pi \left( 1 - \sqrt{1-X^2}\right)$ at $X=0$ is given by

$$\Omega(X) = \Omega(0) + X\Omega'(0) + \dfrac{X^2}{2}\Omega''(0) + \dfrac{X^3}{6}\Omega^{(3)}(0)+ \dfrac{X^4}{24}\Omega^{(4)}(0) + O(x^4)$$

where

$$
\begin{align}
\Omega(X) &= 2\pi \left( 1 - \sqrt{1-X^2}\right)\\
\Omega'(X) &= 2\pi X \left( 1 - X^2\right)^{-\dfrac{1}{2}}\\
\Omega''(X) &= 2\pi  \left( 1 - X^2\right)^{-\dfrac{3}{2}}\\
\Omega^{(3)}(X) &= 6 \pi X \left( 1 - X^2\right)^{-\dfrac{5}{2}}\\
\Omega^{(4)}(X) &= 6 \pi  \left(4X^2 + 1 \right)\left( 1 - X^2\right)^{-\dfrac{7}{2}}
\end{align}
$$

Thus, we get

$$ \Omega(X) = \pi x^2 + \dfrac{x^4 \pi}{4} + O(x^4) $$

Replacing the variable back

$$ \Omega \approx \pi \left(\dfrac{r}{d}\right)^2 + \dfrac{\pi}{4}\left(\dfrac{r}{d}\right)^4$$

And to the 9-th degree

$$ \Omega \approx \pi \left(\dfrac{r}{d}\right)^2 + \dfrac{\pi}{4}\left(\dfrac{r}{d}\right)^4 + \dfrac{\pi}{8}\left(\dfrac{r}{d}\right)^6 + \dfrac{5 \pi}{64}\left(\dfrac{r}{d}\right)^8$$

## Computation

```python
%matplotlib widget
import ipywidgets as widgets
import matplotlib.pyplot as plt
import numpy as np
```

```python
# set up plot
fig, ax = plt.subplots(figsize=(6, 4))
ax.grid(True)

def exact(r, d):
    """
    Return a sine for x with angular frequeny w and amplitude amp.
    """
    return 2*np.pi*(1-np.sqrt(1-(r/d)**2))

def approx(r,d):
    """
    Return a sine for x with angular frequeny w and amplitude amp.
    """
    x = r/d
    return np.pi*(x**2 + x**4/4)

# generate x values
d = np.linspace(1, 10, 100)

maxdiff = 0.
for r in np.linspace(0.1,0.8,8):
    diff = abs(exact(r, d) - approx(r,d))
    if r < 0.5:
        maxdiff = max(np.max(diff), maxdiff)
    ax.plot(d, diff, label=str(r))

ax.set_ylim([0, maxdiff])
ax.legend()
ax.set_title("Error with respect to distance for different radius")

```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Text(0.5, 1.0, 'Error with respect to distance for different radius')


## Задание 1

Даны два вектора в трехмерном пространстве: (10,10,10) и (0,0,-10)
- Найдите их сумму. (на листочке)
- Напишите код на Python, реализующий расчет длины вектора, заданного его координатами. (в программе)

len = x1(10,10,10) + x2(0,0,-10) = (10,10,0)

```python
def lvec(x: list):
    t = 0
    for i in x:
        t += i**2
    return t**0.5
```

```python
lvec([3, 4])
```

    5.0

## Задание 2

Почему прямые не кажутся перпендикулярными?

```python
import numpy as np
from matplotlib import pyplot as plt
```

```python
from pylab import rcParams
```

```python
x = np.linspace(-5, 5, 21)
y1 = 3*x+1
y2 = (1/3)*x+1
plt.plot(x, y1)
plt.plot(x, y2)
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

Ответ: гипотетически, из за разных масштабов осей

```python
x = np.linspace(-5, 5, 21)
rcParams["figure.figsize"] = (6,6)
y1 = 3*x + 1
y2 = (1/3)*x + 1
plt.axis('equal')
plt.grid()
plt.xlim((-5, 5))
plt.ylim((-5, 5))
plt.plot(x, y1)
plt.plot(x, y2)
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

с осями теперь все ок, но прямые все равно не перпендикулярны ¯\\_(ツ)_/¯

## Задание 3

Напишите код на Python, реализующий построение графиков:
- окружности,
- эллипса,
- гиперболы.

```python
t = np.arange(0, 2*np.pi, 0.01)
r = 4
plt.grid()
plt.plot(r*np.sin(t), r*np.cos(t), lw=3)
plt.axis('equal')
plt.show()
```

```python
t = np.arange(0, 2*np.pi, 0.01)
r = 4
plt.grid()
plt.plot(r*np.sin(t), 0.5*r*np.cos(t), lw=3)
plt.axis('equal')
plt.show()
```

```python
def hypY(x: int, a: int, b: int):
    return np.sqrt(np.abs(b**2*(x**2/a**2 - 1)))
```

```python
x1 = np.linspace(-10, -1, 100)
x2 = np.linspace(1, 10, 100)
plt.grid()
plt.plot(x1, hypY(x1, -1, 1))
plt.plot(x1, -hypY(x1, -1, 1))
plt.plot(x2, hypY(x2, -1, 1))
plt.plot(x2, -hypY(x2, -1, 1))
```

## Задание 4

### 1. Пусть задана плоскость: Ax + By + Cz + D = 0. Напишите уравнение плоскости, параллельной данной и проходящей через начало координат

Плоскость: Ax + By + Cz = 0 ( D = 0 )

### 2. Пусть задана плоскость:

$A_1x + B_1y + C_1z + D_1 = 0$

и прямая:

\begin{equation}
\frac{x - x_1}{x_2 - x_1} = \frac{y-y_1}{y_2 - y_1} = \frac{z-z_1}{z_2 - z_1}
\end{equation}

Как узнать, принадлежит прямая плоскости или нет?

Ответ: подставить координаты прямой в уравнение плоскости

$A_1x_1 + B_1y_1 + C_1z_1 + D_1 = A_1x_2 + B_1y_2 + C_1z_2 + D_1$

Если равенство выполняется, значит прямая принадлежит плоскости

### Задание 5

- Нарисуйте трехмерный график двух параллельных плоскостей.
- Нарисуйте трехмерный график двух любых поверхностей второго порядка.

```python
from pylab import *
from mpl_toolkits.mplot3d import Axes3D
```

```python
fig = figure()
ax = Axes3D(fig)
X = np.arange(-5, 5, 0.5)
Y = np.arange(-5, 5, 0.5)
X, Y = np.meshgrid(X, Y)
Z1 = 2*X + 5*Y
Z2 = 2*X + 5*Y + 10
ax.plot_wireframe(X, Y, Z1)
ax.plot_wireframe(X, Y, Z2)
show()
```

```python
from matplotlib import cm
```

```python
fig = figure()
ax = Axes3D(fig)
X = np.arange(-5, 5, 0.2)
Y = np.arange(-5, 5, 0.2)
X, Y = np.meshgrid(X, Y)
Z = 0.5*sin(0.1*X**2) + 0.09*sin(0.15*Y**2)
ax.plot_surface(X, Y, Z, cmap=cm.terrain)
show()
```

## Задание 6

Нарисуйте график функции:
y(x) = k∙cos(x – a) + b
для некоторых (2-3 различных) значений параметров k, a, b

```python
x = np.linspace(-5, 5, 100)
plt.grid()
plt.plot(x, np.cos(x - 1) + 1)
plt.plot(x, 2*np.cos(x - 1) + 1)
plt.plot(x, np.cos(x + 1) -1)
plt.show()
```

## Задание 7

Докажите, что при ортогональном преобразовании сохраняется расстояние между точками

¯\\_(ツ)_/¯

## Задание 8

- Напишите код, который будет переводить полярные координаты в декартовы.
- Напишите код, который будет рисовать график окружности в полярных координатах.
- Напишите код, который будет рисовать график отрезка прямой линии в полярных координатах.

```python
def convPolar(r, a):
    return r*np.cos(a), r*np.sin(a)
```

```python
convPolar(1, np.pi/2)
```

    (6.123233995736766e-17, 1.0)

```python
x = np.linspace(0, 2*np.pi, 100)
y = np.empty(100)
y.fill(0.5)
plt.polar(x, y)
r = np.array([0, 0.2])
a = np.array([1, 1])
b = np.array([1+np.pi, 1+np.pi])
plt.polar(a, r)
plt.polar(b, r)
plt.show()
```

## Задание 9

### 1. Решите систему уравнений

\begin{equation}
\left\{ \begin{array}{ll}
y = x^2 - 1\\
e^x + x (1-y) = 1
\end{array} \right.
\end{equation}

\begin{equation}
\left\{ \begin{array}{ll}
y = x^2 - 1\\
y = 1 - \frac{1 - e^x}{x}
\end{array} \right.
\end{equation}

```python
def fy1(x):
    return x**2 - 1

def fy2(x):
    return 1 - (1 - np.exp(x))/x
```

```python
xx1 = np.linspace(-5, -0.01, 100)
xx2 = np.linspace(0.01, 5, 100)
plt.plot(xx1, fy1(xx1))
plt.plot(xx1, fy2(xx1))
plt.plot(xx2, fy1(xx2))
plt.plot(xx2, fy2(xx2))
plt.grid()
plt.show()
```

```python
from scipy.optimize import fsolve
```

```python
def equations(p):
    x, y = p
    return (y - x**2 + 1, y-1+(1-np.exp(x))/x)
```

```python
x1, y1 = fsolve(equations, (-4, 5))
x2, y2 = fsolve(equations, (3, 3))
x3, y3 = fsolve(equations, (5, 5))
print(x1, y1)
print(x2, y2)
print(x3, y3)
```

    -1.5818353528959466 1.50220308367128
    2.618145573085482 5.854686241867206
    4.200105841166561 16.640889076993545

```python
xx1 = np.linspace(-5, -0.01, 100)
xx2 = np.linspace(0.01, 5, 100)
plt.plot(xx1, fy1(xx1))
plt.plot(xx1, fy2(xx1))
plt.plot(xx2, fy1(xx2))
plt.plot(xx2, fy2(xx2))
plt.scatter([x1, x2, x3],[y1, y2, y3])
plt.grid()
plt.show()
```

### 1. Решите систему уравнений и неравенств

\begin{equation}
\left\{ \begin{array}{ll}
y = x^2 - 1\\
e^x + x (1-y) > 1
\end{array} \right.
\end{equation}

\begin{equation}
\left\{ \begin{array}{ll}
y = x^2 - 1\\
y > 1 - \frac{1 - e^x}{x}
\end{array} \right.
\end{equation}

¯\\_(ツ)_/¯ ¯\\_(ツ)_/¯  ¯\\_(ツ)_/¯


A normal distribution, sometimes called the bell curve, is a distribution that occurs naturally in many situations. For example, the bell curve is seen in tests like the SAT and GRE. The bulk of students will score the average (C), while smaller numbers of students will score a B or D. An even smaller percentage of students score an F or an A. This creates a distribution that resembles a bell (hence the nickname). The bell curve is symmetrical. Half of the data will fall to the left of the mean; half will fall to the right.
Many groups follow this type of pattern. That’s why it’s widely used in business, statistics and in government bodies like the FDA:

Heights of people.

Measurement errors.

Blood pressure.

Points on a test.

IQ scores.

Salaries.

The standard deviation controls the spread of the distribution.
A smaller standard deviation indicates that the data is tightly clustered around the mean; the normal distribution will be taller. A larger standard deviation indicates that the data is spread out around the mean; the normal distribution will be flatter and wider.

Properties of a normal distribution

The mean, mode and median are all equal.

The curve is symmetric at the center (i.e. around the mean, μ).

Exactly half of the values are to the left of center and exactly half the values are to the right.

The total area under the curve is 1.

The Gaussian distribution is a continuous family of curves, all shaped like a bell. In other words, there are endless possibilities for the number of possible distributions, given the limitless possibilities for standard deviation measurements (which could be from 0 to infinity). The standard Gaussian distribution  has a mean of 0 and a standard deviation of 1. The larger the standard deviation, the flatter the curve. The smaller the standard deviation, the higher the peak of the curve.

A Gaussian distribution function can be used to describe physical events if the number of events is very large (Central Limit Theorem(CLT)). In simple terms, the CLT says that while you may not be able to predict what one item will do, if you have a whole ton of items, you can predict what they will do as a whole. For example, if you have a jar of gas at a constant temperature, the Gaussian distribution will enable you to figure out the probability that one particle will move at a certain velocity.

The Probability Distribution Function (PDF) of Normal or Gaussian or Bell Shaped Distribution is given by:

\begin{equation}
P(X=k) = \frac{1}{\sqrt{2\pi}\sigma} \mathrm{e}^{\frac{-(k-\mu)^2}{2\sigma^2}}\
\end{equation}

Also called Normal PDF

```python
import scipy.stats as S
import numpy as np
import seaborn as sns
```

We are going to withdraw some numbers from Normal Distribution having mean = 65 and standard deviation, sigma = 10

The concept of probability being limiting case of relative frequency can be demonstrated here also. Let's first calculate the true probability (theoretical value) of probability of Random Variable, X taking value k=45 calculated from the Normal PDF given mean = 60 and sigma = 10. So, the theoretical value of probability of Random Variable, X taking value k=45 is given by:

\begin{equation}
P(X=45) = \frac{1}{\sqrt{2\pi} 10} \mathrm{e}^{\frac{-(45-60)^2}{2*10^2}}\ = 0.0129
\end{equation}

Which means that the frequency of value 45 is 1.29 % of the size of population.

Let's calculate the value of probability of value 60.

\begin{equation}
P(X=60) = \frac{1}{\sqrt{2\pi} 10} = 0.0399
\end{equation}

Which means that the frequency of value 60 is 3.99 % of the size of population.

Let's generate a sample size of 100 observations.

```python
sigma = 10
mean = 60
sample = np.random.normal(loc=mean,scale=sigma,size=100)
```

```python
sample.shape
```

    (100,)

```python
sns.distplot(sample,bins='rice',kde=True)
```

Let's generate a sample size of 1000 observations.

```python
sample = np.random.normal(loc=mean,scale=sigma,size=1000)
```

```python
sns.distplot(sample,bins='rice',kde=True)
```

Now, let's generate a sample size of 10000 observations.

```python
sample = np.random.normal(loc=mean,scale=sigma,size=10000)
```

```python
sns.distplot(sample,bins='rice',kde=True)
```

Now, for one more last time, let's generate a sample size of 100000 observations.

```python
sample = np.random.normal(loc=mean,scale=sigma,size=100000)
```

```python
sns.distplot(sample,bins='rice',kde=True)
```

All of the plots which we can see above are frequency distributions. In the first plot of frequency distribution, the bulge is coming is coming nearby 60, approximately 55.

In the second frequency distribution where sample size is increased with respect to the previous case i.e. 1000, the distribution is approaching towards the Normal Distribution with the mean of 60. Although, the bulge is still at the value of 55, approximately but this time it's more refined in shape.

In the third frequency distribution where the sample size is further increased i.e. 10000, the distribution is approaching more closely towards actual normal probability distribution with the mean=60 and sigma=10, getting finer.

Finally, if we look at the last plotted frequency distribution with the sample size of 100000, we see that it's not that much different from the previous plotted frequency distribution and so, we can say that the frequency of each of the observation is stablizing and stopped changing much and hence approaching towards probability but how ????

We know that the relative frequency of Random Variable value, X=60 is:

According to First Frequency Distribution, it's approximately 13/100 = 0.13.

According to Second Frequency Distribution, It's approximately 260/1000 = 0.26.

According to Third Frequency Distribution, it's approximately 3250/10000 = 0.3250.

According to Fourth Frequency Distribution, it's approximately 35000/100000 = 0.35.

4. Normal or Gaussian Distribution.ipynb


# The Problem of Overfitting

Consider the problem of predicting $y$ from $x \in R$. The leftmost figure below shows the result of fitting a $y = \theta_0+\theta_1x$ to a dataset. We see that the data doesn’t really lie on straight line, and so the fit is not very good.

Instead, if we had added an extra feature $x^2$ , and fit $y=\theta_0+\theta_1x+\theta_2x^2$ , then we obtain a slightly better fit to the data (See middle figure). Naively, it might seem that the more features we add, the better.

However, there is also a **danger** in adding too many features: The rightmost figure is the result of fitting a 5th order polynomial $y = \sum_{j=0} ^5 \theta_j x^j$. We see that even though the fitted curve passes through the data *perfectly*, we would not expect this to be a very good predictor of, say, housing prices (y) for different living areas (x).

Without formally defining what these terms mean, we’ll say the figure on the left shows an instance of **underfitting**—in which the data clearly shows structure not captured by the model—and the figure on the right is an example of **overfitting**.

**Underfitting**, or high bias, is when the form of our hypothesis function h maps poorly to the trend of the data. It is usually caused by a function that is too simple or uses too few features.

At the other extreme, **overfitting**, or high variance, is caused by a hypothesis function that fits the available data but does not generalize well to predict new data. It is usually caused by a complicated function that creates a lot of unnecessary curves and angles unrelated to the data.

This terminology is applied to both linear and logistic regression. There are two main options to address the issue of overfitting:

### วิธีการแก้ปัญหา Overfitting ทำได้โดย

1) Reduce the number of features:
- Manually select which features to keep.
- Use a model selection algorithm (studied later in the course).

2) Regularization
- Keep all the features, but reduce the magnitude of parameters $\theta_j$.
- Regularization works well when we have a lot of slightly useful features.

# Cost Function

If we have overfitting from our hypothesis function, we can reduce the weight that some of the terms in our function carry by increasing their cost.

Say we wanted to make the following function more quadratic:

$\theta_0 + \theta_1x + \theta_2x^2 + \theta_3x^3 + \theta_4x^4$

We'll want to eliminate the influence of $\theta_3x^3$ and $\theta_4x^4$ . Without actually getting rid of these features or changing the form of our hypothesis, we can instead modify our **cost function**:

$$min_\theta\ \dfrac{1}{2m}\sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + 1000\cdot\theta_3^2 + 1000\cdot\theta_4^2$$

We've added two extra terms at the end to inflate the cost of $\theta_3$ and $\theta_4$. Now, in order for the cost function to get close to zero, we will have to reduce the values of $\theta_3$ and $\theta_4$ to near zero. This will in turn greatly reduce the values of $\theta_3x^3$ and $\theta_4x^4$ in our hypothesis function. As a result, we see that the new hypothesis (depicted by the pink curve) looks like a quadratic function but fits the data better due to the extra small terms $\theta_3x^3$ and $\theta_4x^4$.

We could also regularize all of our theta parameters in a single summation as:

$$min_\theta\ \dfrac{1}{2m}\  \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 + \lambda\ \sum_{j=1}^n \theta_j^2$$

The $\lambda$, or lambda, is the **regularization parameter**. It determines how much the costs of our theta parameters are inflated.

Using the above cost function with the extra summation, we can smooth the output of our hypothesis function to reduce overfitting. **If lambda is chosen to be too large, it may smooth out the function too much and cause underfitting.** Hence, what would happen if λ=0 or is too small ? --> Overfitting

> มันเป็นการ trade-off ระหว่างจะ underfitting หรือ overfitting ถ้า $\lambda$ ใหญ่จะ under ตรงกันข้ามถ้า $\lambda$ เล็กก็จะ over

# Regularized Linear Regression
We can apply regularization to both linear regression and logistic regression. We will approach linear regression first.

### Gradient Descent
We will modify our gradient descent function to **separate out $\theta_0$ from the rest of the parameters** because we do not want to penalize $\theta_0$.

$\begin{align*} & \text{Repeat}\ \lbrace \newline & \ \ \ \ \theta_0 := \theta_0 - \alpha\ \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_0^{(i)} \newline & \ \ \ \ \theta_j := \theta_j - \alpha\ \left[ \left( \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) + \frac{\lambda}{m}\theta_j \right] &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \rbrace \end{align*}$

(ลอง Prove ด้วย Calculus ดู)

The term $\frac{\lambda}{m}\theta_j$ performs our regularization. With some manipulation our update rule can also be represented as:

$\theta_j := \theta_j(1 - \alpha\frac{\lambda}{m}) - \alpha\frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)}$

The first term in the above equation,** $1 - \alpha\frac{\lambda}{m}$ will always be less than 1**. Intuitively you can see it as reducing the value of $\theta_j$ by some amount on every update. Notice that the second term is now exactly the same as it was before.

### Normal Equation

Now let's approach regularization using the alternate method of the non-iterative normal equation.

To add in regularization, the equation is the same as our original, except that we add another term inside the parentheses:

$\begin{align*}& \theta = \left( X^TX + \lambda \cdot L \right)^{-1} X^Ty \newline& \text{where}\ \ L = \begin{bmatrix} 0 & & & & \newline & 1 & & & \newline & & 1 & & \newline & & & \ddots & \newline & & & & 1 \newline\end{bmatrix} \in \mathbb{R}^{(n+1)x(n+1)}\end{align*}$

L is a matrix with 0 at the top left and 1's down the diagonal, with 0's everywhere else. It should have dimension (n+1)×(n+1). Intuitively, this is the identity matrix (though we are not including x0), multiplied with a single real number $λ$.

Recall that if m < n, then $X^TX$ is non-invertible. However, when we add the term $λ⋅L$, then $X^TX + λ⋅L$ becomes invertible.

# Regularized Logistic Regression
We can regularize logistic regression in a similar way that we regularize linear regression. As a result, we can avoid overfitting. The following image shows how the regularized function, displayed by the pink line, is less likely to overfit than the non-regularized function represented by the blue line:

### Cost Function
Recall that our cost function for logistic regression was:

$J(\theta) = - \frac{1}{m} \sum_{i=1}^m \large[ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)})) \large]$

We can regularize this equation by adding a term to the end:

$J(\theta) = - \frac{1}{m} \sum_{i=1}^m \large[ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)}))\large] + \frac{\lambda}{2m}\sum_{j=1}^n \theta_j^2$

*Note : Prove หน่อย*

The second sum, $\sum_{j=1}^n \theta_j^2$ **means to explicitly** exclude the bias term, $\theta_0$. I.e. the $\theta$ vector is indexed from 0 to n (holding n+1 values, $\theta_0$ through $\theta_n$), and this sum explicitly skips $\theta_0$, by running from 1 to n, skipping 0. Thus, when computing the equation, we should continuously update the two following equations:

### Prove + Code ส่วนนี้ด้วย

# ====================== CODE =========================

 ข้อมูลชิพที่ผ่าน และไม่ผ่าน คุณภาพการผลิต

```python
import pandas as pd
import matplotlib.pyplot as plt
import scipy.optimize as opt
import numpy as np

data2 = pd.read_csv('programing/machine-learning-ex2/ex2/ex2data2.txt',names=['Test 1','Test 2','Accepted'])

positive = data2[data2['Accepted'].isin([1])]
negative = data2[data2['Accepted'].isin([0])]

fig, ax = plt.subplots(figsize=(8,5))
ax.scatter(positive['Test 1'], positive['Test 2'], s=50, c='b', marker='o', label='Accepted')
ax.scatter(negative['Test 1'], negative['Test 2'], s=50, c='r', marker='x', label='Rejected')
ax.legend()
ax.set_xlabel('Test 1 Score')
ax.set_ylabel('Test 2 Score')
plt.show()
```

จากรูป ดูทรงแล้ว Decision Boundary น่าจะเป็นสมการกำลังเลขคู่ (2,4,6) ถ้าลองที่กำลัง 6 จะได้

$$
\begin{align}
z = \theta_0 + \theta_1x_1 + \theta_2x_2 + \theta_3x_1^2 + \theta_4x_1x_2 + \theta_5x_2^2 + \theta_6x_1^3 + \theta_7x_1^2x_2 + \theta_8x_1x_2^2 + \theta_9x_2^3 + \theta_{10}x_1^4 + \theta_{11}x_1^3x_2 + \theta_{12}x_1^2x_2^2 + \theta_{13}x_1x_2^3 + \theta_{14}x_2^4 + \theta_{15}x_1^5 + \theta_{16}x_1^4x_2^1 + \theta_{17}x_1^3x_2^2 + \theta_{18}x_1^2x_2^3 + \theta_{19}x_1x_2^4 + \theta_{20}x_2^5 + \theta_{21}x_1^6 + \theta_{22}x_1^5x_2^1 + \theta_{23}x_1^4x_2^2 + \theta_{24}x_1^3x_2^3 + \theta_{25}x_1^2x_2^4 + \theta_{26}x_1x_2^5 + \theta_{27}x_2^6
\end{align}
$$

จะเห็นว่าสมการมันไม่ linear อยู่ แปลงให้เป็น linear จะได้

$$
\begin{align}
z = \theta_0 + \theta_1x_1 + \theta_2x_2 + \theta_3x_3 + \theta_4x_4 + \theta_5x_5 + \theta_6x_6 + \theta_7x_7 + \theta_8x_8 + \theta_9x_9 + \theta_{10}x_{10} + \theta_{11}x_{11} + \theta_{12}x_{12} + \theta_{13}x_{13} + \theta_{14}x_{14} + \theta_{15}x_{15} + \theta_{16}x_{16} + \theta_{17}x_{17} + \theta_{18}x_{18} + \theta_{19}x_{19} + \theta_{20}x_{20} + \theta_{21}x_{21} + \theta_{22}x_{22} + \theta_{23}x_{23} + \theta_{24}x_{24} + \theta_{25}x_{25} + \theta_{26}x_{26} + \theta_{27}x_{27}
\end{align}
$$

ดังนั้นจาก ค่า $x_1,x_2$ ที่เรามีอยู่แล้ว เราต้องหาค่า $x_3 - x_{27}$ เพิ่มด้วย

สร้างฟังก์ชั่นสำหรับแปลง $x_1,x_2$ เป็น $x_1 - x_n$ (จำนวน $n$ ขึ้นกับ degree ของสมการ เช่นที่ degree 6 $n$ คือ 27)

```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
```

```python
def mapFeature(degree,x1,x2):
    #   Returns a new feature array with more features, comprising of X1, X2, X1.^2, X2.^2, X1*X2, X1*X2.^2, etc..
    #   Inputs X1, X2 must be the same size

    df = pd.DataFrame()
    df['Ones'] = np.ones(len(x1))

    for i in range(1, degree+1):
        for j in range(0, i+1):
            df['F' + str(i) + str(j)] = np.power(x1, i-j) * np.power(x2, j)
    return df
```

```python
x1 = data2['Test 1']
x2 = data2['Test 2']
features = mapFeature(6,x1,x2)
features.head()
```

      Ones
      F10
      F11
      F20
      F21
      F22
      F30
      F31
      F32
      F33
      ...
      F53
      F54
      F55
      F60
      F61
      F62
      F63
      F64
      F65
      F66

      0
      1.0
      0.051267
      0.69956
      0.002628
      0.035864
      0.489384
      0.000135
      0.001839
      0.025089
      0.342354
      ...
      0.000900
      0.012278
      0.167542
      1.815630e-08
      2.477505e-07
      0.000003
      0.000046
      0.000629
      0.008589
      0.117206

      1
      1.0
      -0.092742
      0.68494
      0.008601
      -0.063523
      0.469143
      -0.000798
      0.005891
      -0.043509
      0.321335
      ...
      0.002764
      -0.020412
      0.150752
      6.362953e-07
      -4.699318e-06
      0.000035
      -0.000256
      0.001893
      -0.013981
      0.103256

      2
      1.0
      -0.213710
      0.69225
      0.045672
      -0.147941
      0.479210
      -0.009761
      0.031616
      -0.102412
      0.331733
      ...
      0.015151
      -0.049077
      0.158970
      9.526844e-05
      -3.085938e-04
      0.001000
      -0.003238
      0.010488
      -0.033973
      0.110047

      3
      1.0
      -0.375000
      0.50219
      0.140625
      -0.188321
      0.252195
      -0.052734
      0.070620
      -0.094573
      0.126650
      ...
      0.017810
      -0.023851
      0.031940
      2.780914e-03
      -3.724126e-03
      0.004987
      -0.006679
      0.008944
      -0.011978
      0.016040

      4
      1.0
      -0.513250
      0.46564
      0.263426
      -0.238990
      0.216821
      -0.135203
      0.122661
      -0.111283
      0.100960
      ...
      0.026596
      -0.024128
      0.021890
      1.827990e-02
      -1.658422e-02
      0.015046
      -0.013650
      0.012384
      -0.011235
      0.010193

5 rows × 28 columns

เมื่อ regularize cost function เป็นแบบนี้

$J(\theta) = - \frac{1}{m} \sum_{i=1}^m \large[ y^{(i)}\ \log (h_\theta (x^{(i)})) + (1 - y^{(i)})\ \log (1 - h_\theta(x^{(i)}))\large] + \frac{\lambda}{2m}\sum_{j=1}^n \theta_j^2$

จะได้

```python
def costReg(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    first = np.multiply(-y, np.log(sigmoid(X * theta.T)))
    second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))
    reg = (learningRate / 2 * len(X)) * np.sum(np.power(theta[:,1:theta.shape[1]], 2))
    return np.sum(first - second) / (len(X)) + reg
```

```python
def gradientReg(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)

    parameters = int(theta.ravel().shape[1])
    grad = np.zeros(parameters)

    error = sigmoid(X * theta.T) - y

    for i in range(parameters):
        term = np.multiply(error, X[:,i])

        if (i == 0):
            grad[i] = np.sum(term) / len(X)
        else:
            grad[i] = (np.sum(term) / len(X)) + ((learningRate / len(X)) * theta[:,i])
    return grad
```

เตรียม Data ให้ format ถูก ลองใช้ `costReg`

```python
# set X and y
X2 = features.iloc[:,:]
y2 = data2.iloc[:,2:3]

# convert to numpy arrays and initalize the parameter array theta
X2 = np.array(X2.values)
y2 = np.array(y2.values)
theta2 = np.zeros(len(X2[0]))

learningRate = 1

costReg(theta2, X2, y2, learningRate)
```

    0.6931471805599454

หาพารามิเตอร์ของ Decision Boundary จาก `fmin_tnc`

```python
result2 = opt.fmin_tnc(func=costReg, x0=theta2, fprime=gradientReg, args=(X2, y2, learningRate))
theta_min = result2[0]
```

จากพารามิเตอร์ที่ได้มา ลอง plot ดู decision boundary ดังนี้ (ที่ power = 6)

$$
\begin{align}
z = \theta_0 + \theta_1x_1 + \theta_2x_2 + \theta_3x_1^2 + \theta_4x_1x_2 + \theta_5x_2^2 + \theta_6x_1^3 + \theta_7x_1^2x_2 + \theta_8x_1x_2^2 + \theta_9x_2^3 + \theta_{10}x_1^4 + \theta_{11}x_1^3x_2 + \theta_{12}x_1^2x_2^2 + \theta_{13}x_1x_2^3 + \theta_{14}x_2^4 + \theta_{15}x_1^5 + \theta_{16}x_1^4x_2^1 + \theta_{17}x_1^3x_2^2 + \theta_{18}x_1^2x_2^3 + \theta_{19}x_1x_2^4 + \theta_{20}x_2^5 + \theta_{21}x_1^6 + \theta_{22}x_1^5x_2^1 + \theta_{23}x_1^4x_2^2 + \theta_{24}x_1^3x_2^3 + \theta_{25}x_1^2x_2^4 + \theta_{26}x_1x_2^5 + \theta_{27}x_2^6
\end{align}
$$

ดูจากทรงมันยากที่เราจะวาดเส้น decision boundary จาก การแก้สมการนี้ คิดว่าที่เป็นไปได้ก็คือ แทน x1,x2 ไปเลยในช่วงทั้งหมด แล้วน่าจะเห็นเส้นที่ z = 0 เส้นนั้นแหละคือ decision boundary

```python
def plotDecisionBoundary(theta):
    # Here is the grid range
    test1 = np.arange(-1,1.5,0.1)
    test2 = np.arange(-1,1.5,0.1)

    z = np.zeros((len(test1),len(test2)))

    # Evaluate z = theta*x over the grid
    for t1 in range(len(test1)):
        for t2 in range(len(test2)):
            z[t1,t2] = mapFeature(6,np.array([test1[t1]]),np.array([test2[t2]]) ).values.dot(theta)[0]

    T1, T2 = np.meshgrid(test1, test2)
    fig, ax = plt.subplots(figsize=(8,5))

    # Data Plot
    ax.scatter(positive['Test 1'], positive['Test 2'], s=50, c='b', marker='o', label='Accepted')
    ax.scatter(negative['Test 1'], negative['Test 2'], s=50, c='r', marker='x', label='Rejected')
    # Decision Boundary
    CS = plt.contour(T1, T2, z,0.00000000,colors='y')

    ax.legend()
    ax.set_xlabel('Test 1 Score')
    ax.set_ylabel('Test 2 Score')
    plt.show()
```

```python
plotDecisionBoundary(theta_min)
```

สรุปคือ ที่ lambda = 1 decision boundary จะเป็นดังรูปข้างบน

## Predict
เมื่อได้ parameter ของ decision boundary มาแล้ว นำมาทดลองทำนายผลว่าจะเป็น 0 หรือ 1 ดังนี้

$
h_{\theta}(x) = g(z) = \frac{1}{1 + e^{-z}}
$

ถ้า $z>0$ จะได้ว่า $g(z)$ ลู่เข้า 1 ตรงกันข้าม ถ้า $z<0$ จะได้ว่า $g(z)$ ลู่เข้า 0

```python
def predict(theta, X):
    z = X.dot(theta.T)
    predict = (z>=0)
    return predict
```

```python
theta_min = np.matrix(result2[0])
predictions = predict(theta_min, X2)
correct = (y2 == predictions)
accuracy = sum(correct)[0,0]%len(correct)
print('accuracy = {0}%'.format(accuracy))
```

    accuracy = 95%

จากโปรแกรมข้างบนเราทดลองที่ lambda = 1 อย่างเดียว อยากรู้ว่า decision boundary จะเป็นอย่างไรเมื่อ lambda เป็นค่าอื่นๆ

ก่อนอื่นเราฟังชั่นเก่ามาโมก่อน

```python
def plotDecisionBoundaryVaryLambda(X,y,lamb):
    theta2 = np.zeros(len(X[0]))

    result2 = opt.fmin_tnc(func=costReg, x0=theta2, fprime=gradientReg, args=(X, y, lamb))
    theta_min = result2[0]

    # Here is the grid range
    test1 = np.arange(-1,1.5,0.1)
    test2 = np.arange(-1,1.5,0.1)

    z = np.zeros((len(test1),len(test2)))

    # Evaluate z = theta*x over the grid
    for t1 in range(len(test1)):
        for t2 in range(len(test2)):
            z[t1,t2] = mapFeature(6,np.array([test1[t1]]),np.array([test2[t2]]) ).values.dot(theta_min)[0]

    T1, T2 = np.meshgrid(test1, test2)
    fig, ax = plt.subplots(figsize=(8,5))

    # Data Plot
    ax.scatter(positive['Test 1'], positive['Test 2'], s=50, c='b', marker='o', label='Accepted')
    ax.scatter(negative['Test 1'], negative['Test 2'], s=50, c='r', marker='x', label='Rejected')
    # Decision Boundary
    CS = plt.contour(T1, T2, z,0.00000000,colors='y')

    ax.legend()
    ax.set_xlabel('Test 1 Score')
    ax.set_ylabel('Test 2 Score')
    plt.show()
```

ที่ lambda = 0

```python
plotDecisionBoundaryVaryLambda(X2,y2,0)
```

จะเห็นว่า Overfitting หน่อยๆ

ที่ lambda = 100

```python
plotDecisionBoundaryVaryLambda(X2,y2,100)
```

จะเห็นว่า Underfitting มากๆ


## Importando uma função

```python
import numpy as np
```

## Usando uma função/como gerar um array

```python
a = np.array([1, 2, 3])
print(a)
```

    [1 2 3]

```python
a = np.array([[1, 2, 3],
              [4, 5, 6],
              [1, 1, 1]])
print(a)
```

    [[1 2 3]
     [4 5 6]
     [1 1 1]]

## Como inverter matriz

```python
i = np.linalg.inv(a)
print(i)
```

    [[-1.2009599e+16  1.2009599e+16 -3.6028797e+16]
     [ 2.4019198e+16 -2.4019198e+16  7.2057594e+16]
     [-1.2009599e+16  1.2009599e+16 -3.6028797e+16]]

## Arrendondando matriz

```python
b = np.arange(1, 10.1, 1.332)
print(b)
b = np.around(b, 2)
print(b)
```

    [1.    2.332 3.664 4.996 6.328 7.66  8.992]
    [1.   2.33 3.66 5.   6.33 7.66 8.99]

## Arange vs Linspace

```python
# Voçê decide o passo e a função escolhe o tamanho
x = np.arange(step=2, stop=4, start=0)
print(x)
```

    [0 2]

```python
# Voçê decide o tamanho e a função escolhe o passo
y = np.linspace(0, 155, 7)
print(y)
```

    [  0.          25.83333333  51.66666667  77.5        103.33333333
     129.16666667 155.        ]

## Plotando grafico

```python
import matplotlib.pyplot as plt
```

```python
x = np.arange(10)
y = 2*x
z = 3*x
plt.plot(x, y, label='2x')
plt.plot(x, z, label='3x')
plt.xlabel('x')
plt.legend()
plt.show()
#show() plota o grafico com tds os comandos plt acima dele até chegar num outro show()
plt.plot(x, y, label='2x')
plt.plot(x, z, label='3x')
plt.xlabel('x')
plt.legend()
plt.show()
```

```python
a = np.array([1, 2, 3, 4, 5, 6])
print(f'{a[0]}\n{a[-1]}\n{a[2:5]}')
```

    1
    6
    [3 4 5]

```python
a = a.reshape([2, 3])
print(a)
```

    [[1 2 3]
     [4 5 6]]

```python
a = np.eye(3)
print(a)
```

    [[1. 0. 0.]
     [0. 1. 0.]
     [0. 0. 1.]]

```python
from scipy.linalg import toeplitz
e = toeplitz([1, 0, -1, -2], [1, 2, 3, 4])    #Escreve uma matriz([d, d-1, d-2, ...], [d(n), d+1, d+2, ...])
print(e)
```

    [[ 1  2  3  4]
     [ 0  1  2  3]
     [-1  0  1  2]
     [-2 -1  0  1]]

```python
a = np.array([[1, 2, 3],
              [2, 1, 2],
              [3, 2, 1]])
a = np.linalg.inv(a)
print(a)
```

    [[-0.375  0.5    0.125]
     [ 0.5   -1.     0.5  ]
     [ 0.125  0.5   -0.375]]

```python
a = np.array([[1, 1, 1],
                  [2, 2, 2],
                  [3, 3, 3]])
b = np.transpose(a)
print(a)
print(b)
```

    [[1 1 1]
     [2 2 2]
     [3 3 3]]
    [[1 2 3]
     [1 2 3]
     [1 2 3]]

```python
a = np.array([[1, 2, 3],
              [2, 1, 2],
              [3, 2, 1]])
b = np.array([[1, 1, 1],
              [2, 2, 2],
              [3, 3, 3]])
c = np.dot(a, b)
print(c)
```

    [[14 14 14]
     [10 10 10]
     [10 10 10]]

```python
a = np.array([[2, 1],
              [3, 4]])
p = np.array([[10],
              [20]])
from scipy.linalg import solve

s = solve(a, p)
print(s)
```

    [[4.]
     [2.]]

```python
from scipy.linalg import lu_factor, lu_solve
lu, piv  = lu_factor(a)
s = lu_solve((lu, piv), p)
print(s)
```

    [[4.]
     [2.]]

```python
from sympy import *
```

```python
f = lambda x: 1/x
x = symbols ('x')
```

```python
l = limit(f(x), x, 0)
print(l)
```

    oo

```python
print(l + 1)
```

    oo

```python
from scipy.misc import derivative
d = derivative(f, 1, dx=1e-10, n=1)
print(d)
```

    -1.000000082740371

```python

print(f'{d:.2f}')
```

    -1.00

```python
y = lambda x: x**2
v = symbols('v')
i = integrate(y(v), (v , 1, 2))
print(i)
print(type(i))
a = float(i)
print(a)
print(type(a))
```

    7/3
    <class 'sympy.core.numbers.Rational'>
    2.3333333333333335
    <class 'float'>

```python
i = float(integrate(f(v), (v , 1, 2)))
print(i)
```

    0.6931471805599453

```python
print(type(i))
```

    <class 'float'>

```python
import matplotlib.pyplot as plt
D = lambda x: 12 - 2*x
S = lambda x: 20*x
x = [i for i in range(0, 1000)]
dx = [D(i) for i in range(0, 1000)]
sx = [S(i) for i in range(1000, 0, -1)]
plt.plot(dx, x)
plt.plot(sx, x)
plt.show()
```

```python
import matplotlib.pyplot as plt
import numpy as np

def f(x):
    return x

def g(x):
    return x**2

def h(x):
    return 2*x

x = np.arange(-5, 6, 1)
y = [f(i) for i in x]
z = [g(i) for i in x]
w = [h(i) for i in x]
#cria uma lista com os valores de y para cada valor de x

plt.plot(x, y, label='y = x')
plt.plot(x, z, 'r--', label='y = x**2')
plt.plot(x, w, 'g-o', label='y = 2x')
plt.title('Titulo desejado')
plt.xlabel('Eixo X')
plt.ylabel('Eixo Y')
plt.legend(loc='best')
plt.tight_layout()
plt.style.use('ggplot')
plt.grid()
plt.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt

preco = np.array([8, 7, 6, 5, 4, 3, 2, 1, 0])
quant = np.array([0, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000])

plt.plot(quant, preco)
plt.ylabel('Preço(R$)')
plt.xlabel('Quantidade comprada')
plt.title('Quantidade x Preço')
plt.tight_layout()
plt.style.use('ggplot')
plt.show()
```

```python
pontos = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I']

```

```python
def elasticidade(P, PN, Q, QN):
    Num = abs((Q - QN)/((Q + QN)/2))
    Dem = abs((P - PN)/((P + PN)/2))
    return Num/Dem
```

```python
for i in range(len(pontos)-1):
    r = elasticidade(preco[i], preco[i+1], quant[i], quant[i+1])
    print(f'Entre os pontos {pontos[i]} e {pontos[i+1]} a elasticidade vale {r:.2f}')
```

    Entre os pontos A e B a elasticidade vale 15.00
    Entre os pontos B e C a elasticidade vale 4.33
    Entre os pontos C e D a elasticidade vale 2.20
    Entre os pontos D e E a elasticidade vale 1.29
    Entre os pontos E e F a elasticidade vale 0.78
    Entre os pontos F e G a elasticidade vale 0.45
    Entre os pontos G e H a elasticidade vale 0.23
    Entre os pontos H e I a elasticidade vale 0.07

```python

```


# The 24 Game

The Python program below finds solutions to the 24 game: use four numbers and any of the four basic arithmetic operations (multiplication, division, addition and subtraction) to produce the number 24 (or any number you choose).

Execute the program, choose four numbers (separated by commas) and the target number (24 by default).

```python
from sympy import symbols, parsing
from sympy.parsing.sympy_parser import parse_expr
from itertools import permutations, combinations_with_replacement
# list the default numbers to use here
numbahs = [2,4,6,8]
# enter the default number to calculate here
ansah = 24

numberlist = input("Enter a list of four integers, separated by commas [default: 2,4,6,8]: ")
if numberlist:
    numbahs = [int(n) for n in numberlist.split(',')]
answer = input("Enter a number compute to [default: 24]: ")
if answer:
    ansah = int(answer)

# generate all possible arrangements of parentheses and operators
# ways of combining arguments with parenthesis
pgroups = [
    "{0}{1}{2}{3}{4}{5}{6}",
    "({0}{1}{2}){3}{4}{5}{6}",
    "({0}{1}{2}{3}{4}){5}{6}",
    "(({0}{1}{2}){3}{4}){5}{6}",
    "({0}{1}({2}{3}{4})){5}{6}",
    "{0}{1}({2}{3}{4}){5}{6}",
    "{0}{1}({2}{3}({4}{5}{6}))",
    "{0}{1}(({2}{3}{4}){5}{6})",
    "{0}{1}({2}{3}{4}{5}{6})",
    "{0}{1}{2}{3}({4}{5}{6})"
]

# available operators
operators = ['*','/','+','-']
# available symbols
variables = ['a','b','c','d']
# symbols for sympy to work with
a, b, c, d = symbols('a b c d')

# does a file exist with unique expressions?
try:
    with open("uniqueexpressions.txt") as f:
        expressions = [parse_expr(s) for s in f.readlines()]

# no file.. generate one
except:
    # collect unique expressions in a set
    expressions = set()
    for p in pgroups: # for every combination of parenthesis
        for combo in combinations_with_replacement(operators, 3): # and every combination of operators
            for o in permutations(combo):  # permuted
                for v in permutations(variables): # and every permutation of variables
                    s = p.format(v[0],o[0],v[1],o[1],v[2],o[2],v[3]) # construct the expression and...
                    s = parse_expr(s)
                    # add to expressions set -- this will drop equivalent expressions
                    expressions.add(s)
    with open("uniqueexpressions.txt", "w") as f:
        [f.write(str(s)+'\n') for s in expressions]

# for every unique expression, substitute the values and evaluate
for x in expressions:
    try:
        val = x.subs({a:numbahs[0],b:numbahs[1],c:numbahs[2],d:numbahs[3]})
        if ansah == val:
            s = str(x)
            for sym, num in zip(variables, numbahs):
                s = s.replace(sym, str(num))
            print(s)
    except:
        pass
#print(expressions)
```

    Enter a list of four integers, separated by commas [default: 2,4,6,8]:  1,2,3,4
    Enter a number compute to [default: 24]:

    2*3*4/1
    1*2*3*4
    4*(1 + 2 + 3)


# Lecture 10: Expectation Continued

## A Proof of Linearity (discrete case)

Let $T = X + Y$, and show that $\mathbb{E}(T) = \mathbb{E}(X) + \mathbb{E}(Y)$.

We will also show that $\mathbb{E}(cX) = c \mathbb{E}(X)$.

In general, we'd like to be in a position where

\begin{align}
  \sum_{t} t P(T=t) \stackrel{?}{=} \sum_{x} x P(X=x) + \sum_{y} y P(Y=y)
\end{align}

so, let's try attacking this from the l.h.s.

Considering the image above of a discrete r.v. in Pebble World, note that

\begin{align}
  \mathbb{E}(X) &= \sum_{x} x P(X=x) & &\text{grouping the pebbles per X value; weighted average} \\
  &= \sum_{s}X(s)P(\{s\}) & &\text{ungrouped; sum each pebble separately} \\
  \\
  \\
  \Rightarrow \mathbb{E}(T) &= \sum_{s} (X+Y)(s)P(\{s\}) \\
  &= \sum_{s}X(s)P(\{s\}) + \sum_{s}Y(s)P(\{s\}) \\
  &= \sum_{x} x P(X=x) + \sum_{y} y P(Y=y) \\
  &= \mathbb{E}(X) + \mathbb{E}(Y) ~~~~ \blacksquare \\
  \\
  \\
  \Rightarrow \mathbb{E}(cX) &= \sum_{x} cx P(X=x) \\
  &= c \sum_{x} x P(X=x) \\
  &= c \mathbb{E}(X) ~~~~ \blacksquare
\end{align}

----

## Negative Binomial Distribution

### Description

A misnomer: this distribution is actually non-negative, and not binomial, either.

The Negative Binomial is a generalization of the Geometric distribution, where we have a series of independent $Bern(p)$ trials and we want to know # failures before the $r^{\text{th}}$ success.

We can codify this using a bit string:

\begin{align}
  & \text{1000100100001001} & \text{0 denotes failure, 1 denotes success} & \\
  & r = 5 \\
  & n = 11 & \text{failures}
\end{align}

Note that the very last bit position is, of course, a success.

Note also that we can permutate the preceding $r-1$ successes amongst the $n+r-1$ slots that come before that final $r^{\text{th}}$ success.

### Notation

$X \sim NB(r,p)$

### Parameters

* $r$ - the total number of successes before we stop counting
* $p$ - probability of success

### Probability mass function

\begin{align}
  P(X=n) &= \binom{n+r-1}{r-1}  p^r (1-p)^n & &\text{for } n = 0,1,2,\dots\\
  &= \binom{n+r-1}{n}  p^r (1-p)^n & &\text{or conversely}\\
\end{align}

### Expected value

Let $X_j$ be the # failures before the $(j-1)^{\text{st}}$ and $j^{\text{th}}$ success. Then we could write

\begin{align}
  \mathbb{E}(X) &= \mathbb{E}(X_1 + X_2 + \dots + X_r) \\
                &= \mathbb{E}(X_1) + \mathbb{E}(X_2) + \dots + \mathbb{E}(X_r) & &\text{by Linearity} \\
                &= r \mathbb{E}(X_1) & &\text{by symmetry} \\
                &= r \frac{q}{p} ~~~~ \blacksquare
\end{align}

----

## Revisting the Geometric: the First Success Distribution

$X \sim FS(p)$ is the geometric distribution that counts the trials until first success, *including that first success*.

Let $Y = X - 1$.

Then $Y \sim Geom(p)$

Expected value of $FS(p)$ is

\begin{align}
  \mathbb{E}(X) &= E(Y) + 1 \\
                &= \frac{q}{p} + 1 \\
                &= \boxed{\frac{1}{p}}
\end{align}

----

## Putnam Problem

Consider a random permutation of $1, 2, 3, \dots , n$, where $n \ge 2$.

Find expected # local maxima. For example, given the permuation $\boxed{3} ~~ 2 ~~ 1 ~~ 4 ~~ \boxed{7} ~~ 5 ~~ \boxed{6}$ we have 3 local maxima:

- $\boxed{3} \gt 2$
- $4 \lt \boxed{7} \gt 5$
- $ 5 \lt \boxed{6}$

Now, there are 2 kinds of cases we need to consider:

- non-edge case: $4 ~~ \boxed{7} ~~ 5$ has probability of $\frac{1}{3}$ that the largest number is in the middle position
- edge case: in both left-edge $\boxed{3} ~~ 2$ and right-edge $5 ~~ \boxed{6}$, the probability that the larger number is in the right position is $\frac{1}{2}$

Let $I_j$ be the indicator r.v. of position $j$ having a local maximum, $1 \le j \le n$.

Using Linearity, we can say that the expected number of local maxima is given by

\begin{align}
  \mathbb{E}(I_j) &= \mathbb{E}(I_1 + I_2 + \dots + I_n) \\
                  &= \mathbb{E}(I_1) + \mathbb{E}(I_2) + \dots + \mathbb{E}(I_n) & &\text{by Linearity} \\
                  &= (n-2) \frac{1}{3} + 2 \frac{1}{2} \\
                  &= \boxed{\frac{n+1}{3}}
\end{align}

Idiot-checking this, we have:

\begin{align}
  \mathbb{E}(I_{n=2}) &= \frac{2+1}{3} & &\text{... case where } n=2 \\
  &= 1 \\
  \\
  \\
  \mathbb{E}(I_{n=\infty}) &= \frac{\infty+1}{3} & &\text{... case where } n= \infty \\
  &= \infty \\
\end{align}

----

## St. Petersburg Paradox

Consider a game of chance involving a fair coin. We will flip the coin until the very first heads shows (hypergeometric distribution).

- If heads shows on the very first flip, you get $\$2$.
- If the first heads shows on the second flip, you get $\$4$.
- If the first heads shows on the third flip, you get $\$8$.

So you will get $\$2^n$ if the first heads shows up on the $n^\text{th}$ trial, including the heads flip.

_How much would you be willing to play this game?_

Let's tackle this by thinking about the expected number of $\$\$\$$ we stand to make.

Given $Y = 2^n$, find $\mathbb{E}(Y)$:

\begin{align}
  \mathbb{E}(Y) &= \sum_{k=1}^\infty 2^k \frac{1}{2^{k-1}} ~ \frac{1}{2}\\
                &= \sum_{k=1}^\infty 2^k \frac{1}{2^k}\\
                &= \sum_{k=1}^\infty 1\\
  \\
  \\
  \mathbb{E}(Y_{k=40}) &= \sum_{k=1}^{40} 1 \\
                       &= 40
\end{align}

So, the "paradox" here is that even if we capped the payout to $2^{40} \approx \$1000000000$, Linearity shows us we would only pay $40. It is very hard to grasp this, but the truth is that if you were offered this game at any price, you should take it.

----