EE 502 P: Analytical Methods for Electrical Engineering

# Homework 1: Python Setup
## Due October 10, 2021 by 11:59 PM
### Mayank Kumar

Copyright © 2021, University of Washington

**Instructions**: Please use this notebook as a template. Answer all questions using well formatted Markdown with embedded LaTeX equations, executable Jupyter cells, or both. Submit your homework solutions as an `.ipynb` file via Canvas.

Although you may discuss the homework with others, you must turn in your own, original work.

**Things to remember:**
- Use complete sentences. Equations should appear in text as grammatical elements.
- Comment your code.
- Label your axes. Title your plots. Use legends where appropriate.
- Before submitting a notebook, choose Kernel -> Restart and Run All to make sure your notebook runs when the cells are evaluated in order.

Note : Late homework will be accepted up to one week after the due date and will be worth 50% of its full credit score.

### 0. Warmup (Do not turn in)

- Get Jupyter running on your computer, or learn to use Google Colab's Jupyter environment.
- Make sure you can click through the Lecture 1 notes on Python. Try changing some of the cells to see the effects.
- If you haven't done any Python, follow one of the links in Lecture 1 to a tutorial and work through it.
- If you haven't done any Numpy or Sympy, read through the linked documentation and tutorials for those too.

### 1. Complex Numbers
Write a function `rand_complex(n)` that returns a list of `n` random complex numbers uniformly distributed in the unit circle (i.e., the magnitudes of the numbers are all between 0 and 1). Give the function a docstring. Demonstrate the function by making a list of 25 complex numbers.

```python
def rand_complex(n):
    """

    n : number of complex number to be generated
    function during call imports random and numpy libraries for processing.

    This function prints the complex number uniformly distributed in the unit circle.

    """
    import random        # import "random" library to generate random numbers.
    import numpy as np   # import numpy to process mathematical expression

    a = [ ]                    # declare an empty list
    for i in range(n):
        x = random.uniform(-1,1)    # generate random number between -1 and 1.
        y = random.uniform(-(np.sqrt(1-(x*x))),np.sqrt(1-(x*x)))   # generating random value of y between -max and max.
        # max implies corresponding point on the circle for with y will lie on the circle.

        a.append(complex(x,y)) # append complex number x+yj to the list a.

       # print("Complex %d: %f + %fj" %(i+1,x, y))
       # print("Value %d: %f" %((i+1),np.sqrt(x*x+y*y)))
    return a

#function call for 25 complex numbers
rand_complex(25)

```

    [(-0.23291727633158343-0.03223906965648793j),
     (0.1075264193171328-0.6287554693178661j),
     (-0.9648352685957642-0.09836768328893047j),
     (0.9729332666351493-0.07563889894437151j),
     (-0.31264579365485834-0.6952447113453855j),
     (-0.7543046289338111-0.2007322497775722j),
     (0.35390768875343515+0.24066078970197824j),
     (0.40023028389808935-0.4322478564776867j),
     (0.7575837355779895-0.42474223841029735j),
     (-0.017343727286792454-0.9318922926373809j),
     (0.9893226815073348-0.07429562306514148j),
     (-0.7557002838583444+0.5007781393244392j),
     (0.14229428659162435-0.05580365383869268j),
     (0.34332631754502363+0.7610185062340256j),
     (-0.6663127004466431-0.43778123350460907j),
     (0.698490378322481+0.33122161883243073j),
     (0.3875019475298742+0.5520329887602473j),
     (-0.7285191732411849+0.20066390378294496j),
     (-0.6876991691513978-0.10572637939038865j),
     (0.2895621483951627-0.14222573877072409j),
     (0.559677483878626+0.18654677359454308j),
     (0.8954820520485438-0.2273070618668761j),
     (0.4062147904057407+0.3113418259043812j),
     (0.5540089171896212-0.42022748042357205j),
     (-0.8270129193619533-0.27250082404388287j)]

### 2. Hashes
Write a function `to_hash(L) `that takes a list of complex numbers `L` and returns an array of hashes of equal length, where each hash is of the form `{ "re": a, "im": b }`. Give the function a docstring and test it by converting a list of 25 numbers generated by your `rand_complex` function.

```python
def to_hash(L):
    """
    L: It is a list of complex numbers.

    With a list as input, the function returns an array of hashes in form {"re": a, "im": b}
    """
    b = [] #declare an empty list
    for i in range(len(L)):
        x1 = L[i].real #extract real part of the (i)th value of list L
        y1 = L[i].imag #extract imaginary part of the (i)th value of list L
        b.append({"re": x1, "im": y1}) #append the values in hash form.

    return b

#function call using the function from Question 1.
to_hash(rand_complex(25))

#extra line of code to verify the outcome.
#d = to_hash(rand_complex(25))
#print(d[1])
#d[1]["re"]

```

### 3. Matrices

Write a function `lower_traingular(n)` that returns an $n \times n$ numpy matrix with zeros on the upper diagonal, and ones on the diagonal and lower diagonal. For example, `lower_triangular(3)` would return

```python
array([[1, 0, 0],
       [1, 1, 0],
       [1, 1, 1]])
```

```python
import numpy as np # variable np is used inside the function to generate array.

#function defination starts here.
def lower_triangular(n):
    """

    function takes integer n as input and returns lower triangular matrix of dimension nxn.
    import module "numpy as np" to use this function without any errors.

    """
    c = np.ones(n*n)      # create an array of nxn elements with value = 1.
    c = c.reshape (n,n)   # Reshape the array to a matrix of dimension nxn.

    for i in range (n):
        for j in range (n):
            if j > i:        # checks for the elements where we have to change it to zero.
                c[i][j] = 0  # Replace the elements in upper triangle other than diagonal elements with 0.

    return c
print("lower triangular matrix :")
lower_triangular(3)
```

    lower triangular matrix :

    array([[1., 0., 0.],
           [1., 1., 0.],
           [1., 1., 1.]])

### 4. Numpy

Write a function `convolve(M,K)` that takes an $n \times m$ matrix $M$ and a $3 \times 3$ matrix $K$ (called the kernel) and returns their convolution as in [this diagram](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcTYo2_VuAlQhfeEGJHva3WUlnSJLeE0ApYyjw&usqp=CAU).

Please do not use any predefined convolution functions from numpy or scipy. Write your own. If the matrix $M$ is too small, your function should return a exception.

You can read more about convolution in [this post](https://setosa.io/ev/image-kernels/).

The matrix returned will have two fewer rows and two fewer columns than $M$. Test your function by making a $100 \times 100$ matrix of zeros and ones that as an image look like the letter X and convolve it with the kernel

$$
K = \frac{1}{16} \begin{pmatrix}
1 & 2 & 1 \\
2 & 4 & 2 \\
1 & 2 & 1
\end{pmatrix}
$$

Use `imshow` to display both images using subplots.

```python
# import numpy as np #import numpy library into variable np

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

#define kernel
K = np.array ([[1,2,1],
               [2,4,2],
               [1,2,1]])

K = K/16 # divide all elements of matrix by 16.

#create image for testing as per the question
m = 100
n = 100
im = np.ones(m*n)     #created an array of length mn.
im = im.reshape(m,n)  #reshaped the array in matrix of dimension mxn.

#creating image in form X.
for i in range (m):
    for j in range (n):
        if ((i==j) or (i+j == n-1)):    #select all the elements where manipulation is required.
            im[i][j] = 0                #replace 1 with 0 at all the locations selected above.

fig = plt.figure(figsize = (8,8))       #define figure size
fig.add_subplot(1,2,1)                  #add a subplot to the figure
plt.imshow(im)                          #show input image which was created earlier.
plt.axis('off')                         #axis are turned off
plt.title("Original")                   #adding title to the image

#function defination starts here
def convolve (M,K):
    """
    function take two inputs.
    M: it is the original image on which convolution should be performed.
    K: it is the kernel which will be applied to the input matrix.

    it returns matrix with (m-2) x (n-2)
    """
    im_out = np.zeros(M.shape[0]* M.shape[1]).reshape(M.shape[0], M.shape[1]) #declare a matrix with same dimension as M

    if (M.shape[0] < 3 or M.shape[1] < 3): #raising exception for the matrices with either if dimensions < 3.
        raise Exception("Convolution of matrix M can't be calculated. Check input matrix for dimension.")
    else:
        for i in range(1, M.shape[0]-1):
            for j in range(1, M.shape[1]-1):
                im_out[i][j] = np.sum(K * M[(i-1):(i+2),(j-1):(j+2)]) #performing sum on matrix multiplication with kernal and current pixels of the image.
                if (im_out[i][j]) > 255:  #checking for the pixels which get more than 255.
                    im_out[i][j] = 255
                elif (im_out[i][j] < 0):  #checking for the pixels which get less than 0.
                    im_out[i][j] = 0

    im_out = im_out[1:(M.shape[0]-1),1:(M.shape[1]-1)] #removed the blank pixels where convolution was not possible.
    return im_out

out_image = convolve(im,K)              #function call, output is saved for further usage
#print(len(out_image[0]))

fig.add_subplot(1,2,2)                  #add a subplot to the figure
plt.imshow(out_image)                   #Show the output image to the location
plt.axis('off')                         #axis is turned off as it is not required
plt.title("Output")                     #adding title to the output image

```

### 5. Symbolic Manipulation

Use sympy to specify and solve the following equations for $x$.

- $x^2 + 2x - 1 = 0$
- $a x^2 + bx + c = 0$

Also, evaluate the following integrals using sympy

- $\int x^2 dx$
- $\int x e^{6x} dx$
- $\int (3t+5)\cos(\frac{t}{4}) dt$

```python
#Importing related libraries
import math
from sympy import *
init_printing(use_latex='mathjax')

#solving First equation i.e., x^2 + 2x - 1 = 0
x = symbols("x")
expr_1 = (x**2) + (2*x) - 1
result_1 = solve(expr_1,x)
print("solution of equation x^2 + 2x - 1 = 0 is :")
result_1
```

```python
#solving second equation i.e., ax^2 + bx + c = 0
a,b,c = symbols("a b c")
x = symbols("x")
expr_2 = (a*(x**2)) + b*x + c
result_2 = solve(expr_2,x)
print("solution of equation ax^2 + bx + c = 0 is : " )
result_2
```

    solution of equation ax^2 + bx + c = 0 is :

$\displaystyle \left[ \frac{- b + \sqrt{- 4 a c + b^{2}}}{2 a}, \  - \frac{b + \sqrt{- 4 a c + b^{2}}}{2 a}\right]$

```python
#evaluating integral 1
x = symbols("x")
expr_3 = x**2
integrate(expr_3,x)
```

```python
#evaluating integral 2
x = symbols("x")
expr_4 = x * exp(6*x)
integrate(expr_4)
```

```python
#evaluating integral 3
t = symbols("t")
expr_5 = ((3*t) + 5) * cos(t/4)
integrate(expr_5)
```

### 6. Typesetting

Use LaTeX to typeset the following equations.

## 17 Equations that changed the world
###   by Ian Stewart
#### Typesetting starts Now
---
**1. Pythagoras theorm:**
\begin{align}
a^2 + b^2 = c^2
\end{align}
___
**2. Logrithms:**
\begin{align}
\log xy = \log x +\log y
\end{align}
___

**3. Calculus:**
\begin{align}
\frac{df}{dt} = \lim_{h \rightarrow 0} \frac {f(t+h) - f(t)}{f(h)}
\end{align}
I have made some modification in the formula written above. as it was not conveying intended meaning.
___

**4. Law of Gravity:**
\begin{align}
F = G\frac{m_1 m_2}{r^2}
\end{align}
___

**5. Square root of minus one**
\begin{align}
i^2 = -1
\end{align}
___

**6. The Euler's formula for polyhedra**
\begin{align}
V - E + F = 2
\end{align}
___

**7. Normal Distribution**
\begin{align}
\phi(x) = \frac {1}{\sqrt{2\pi\rho}} e^\frac{(x-\mu)^2}{2\rho^2}
\end{align}
___

**8. Wave Equation**
\begin{align}
\frac{\partial^2 u}{\partial t^2} = c^2\frac{\partial^2 u}{\partial x^2}
\end{align}
___

**9. Fourier Transform**
\begin{align}
f(w) = \int_{-\infty}^{\infty} f(x)e^{-2\pi i x w} dx
\end{align}
___

**10. Navier-Stokes Equation**
\begin{align}
\rho(\frac{\partial v}{\partial t} + v.\nabla v ) = - \nabla p + \nabla.T + f
\end{align}
___

**11. Maxwell's Equation**
\begin{align}
\nabla . E = 0 \hspace{50 pt}\nabla.H = 0 \\
\nabla \times E = -\frac{1}{c} \frac{\partial H}{\partial t} \hspace{30 pt}\nabla \times H = \frac{1}{c} \frac{\partial E}{\partial t}
\end{align}
___

**12. Second Law of Thermodynamics**
\begin{align}
dS \geq 0
\end{align}
___

**13. Relativity**
\begin{align}
E = mc^2
\end{align}
___

**14. Schrodinger's Equation**
\begin{align}
i\hbar \frac{\partial}{\partial t} \Psi = H \Psi
\end{align}
___

**15. Information Theory**
\begin{align}
H = -\sum p(x)\log p(x)
\end{align}
___

**16. Chaos Theory**
\begin{align}
x_{t + 1} = k x_t(1 - x_t)
\end{align}
___

**17.Black-Scholes Equation**

\begin{align}
\frac{1}{2}\sigma S^2 \frac{\partial^2 V}{\partial S^2} + r S \frac{\partial V}{\partial S} + \frac{\partial V}{\partial t} - r V = 0
\end{align}
___

```python

```


# Flow in a nano-porous material

In this notebook we'll explore flow induced in a nano-porous material. To do this we introduce a force $\mathbf{F}_x$ acting on each particle $i$ in the nano-porous material.
In the case of a gravitational field, we can formulate Darcy's law for flow as
\begin{align}
    \mathbf{U} = \frac{k}{\mu}\left( \nabla P - \rho \mathbf{a} \right).
\end{align}
Here $\mathbf{U}$ is the _Darcy velocity_, $k$ the permeabillity, $\mu$ the fluid viscosity, $P$ the externally applied pressure, $\rho$ the density and $\mathbf{a}$ the acceleration from the applied force. In our case $\nabla P = 0$ as we have no externally applied pressure. Looking at Newton's second law for a single particle we have
\begin{align}
    \mathbf{F}_x = m_i \mathbf{a}_i.
\end{align}
For all particles of the same mass with the same applied force this yields the total applied for to the system
\begin{align}
    N\mathbf{F}_x = N m \mathbf{a},
\end{align}
where $N$ is the number of particles. Dividng by the volume of the system we find
\begin{gather}
    \frac{N}{V}\mathbf{F}_x = \frac{N m}{V} \mathbf{a}
    \implies
    n \mathbf{F}_x = \rho \mathbf{a},
\end{gather}
where $n$ is the number density and $\rho$ the density of the system. Thus we see that we can replace $\rho \mathbf{a}$ in Darcy's law with $n\mathbf{F}_x$ and we write
\begin{align}
    \mathbf{U} = \frac{k}{\mu}\left( \nabla P - n \mathbf{F}_x \right).
\end{align}

```python
import os
import sys

sys.path.append("..")

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from read_lammps_dump import read_dump
from read_lammps_log import read_log, get_temp_lognames

sns.set(color_codes=True)
```

```python
%%writefile scripts/flow_profile.in

# 3d Lennard-Jones gas
units lj
dimension 3
# Periodic boundiaries
boundary p p p
atom_style atomic

variable seed equal 87287
variable sigma equal 3.405
variable b equal 5.72
variable reduced_density equal 4/((${b}/${sigma})^3)
variable temperature equal 1.5
variable num_box equal 20
variable radius_min equal 20/${sigma}
variable radius_max equal 30/${sigma}

# Set fcc lattice with specified density
lattice fcc ${reduced_density}
region simbox block 0 ${num_box} 0 ${num_box} 0 ${num_box}
create_box 2 simbox
create_atoms 1 box

variable box_len equal lx

mass * 1.0
velocity all create ${temperature} ${seed}

pair_style lj/cut 3.0
pair_coeff * * 1.0 1.0

fix 1 all nvt temp ${temperature} ${temperature} 0.5

thermo 100
run 1000

# Use a radius of 2 nm (equal to 20 Å)
variable radius equal 20/${sigma}
variable centre_x equal lx/2
variable centre_y equal ly/2
variable centre_z equal lz/2
region cylinder_reg cylinder x ${centre_y} ${centre_z} ${radius} EDGE EDGE units box

# Delete half of the atoms in the region
delete_atoms porosity cylinder_reg 0.5 ${seed}

group pore_group region cylinder_reg
set group pore_group type 2

# Create another group from all the remaining atoms except for the ones in the cylinder
group frozen subtract all pore_group
# Set the velocity of the atoms outside the cylinder to zero
velocity frozen set 0 0 0
# Avoid integrating all the particles
unfix 1

# Run pore_group using NVE
fix 1 pore_group nve

dump 1 all custom 10 dat/flow_profile.lammpstrj id type x y z vx vy vz

# Compute the porosity
variable porosity equal count(pore_group)/count(all)
print ${porosity} file dat/flow_profile_porosity.dat

reset_timestep 0
compute pore_temp pore_group temp
compute msd pore_group msd
variable time equal dt*step

thermo_style custom step v_time c_pore_temp etotal press c_msd[4]

fix force pore_group addforce 0.1 0.0 0.0

thermo 10
log dat/flow_profile.log

run 2000
```

    Overwriting scripts/flow_profile.in

```python
!export OMP_NUM_THREADS=4 && mpirun -np 4 lmp -in scripts/flow_profile.in
```

    LAMMPS (4 Jan 2019)
      using 4 OpenMP thread(s) per MPI task
    Lattice spacing in x,y,z = 1.67988 1.67988 1.67988
    Created orthogonal box = (0 0 0) to (33.5977 33.5977 33.5977)
      1 by 2 by 2 MPI processor grid
    Created 32000 atoms
      Time spent = 0.000936326 secs
    Neighbor list info ...
      update every 1 steps, delay 10 steps, check yes
      max neighbors/atom: 2000, page size: 100000
      master list distance cutoff = 3.3
      ghost atom cutoff = 3.3
      binsize = 1.65, bins = 21 21 21
      1 neighbor lists, perpetual/occasional/extra = 1 0 0
      (1) pair lj/cut, perpetual
          attributes: half, newton on
          pair build: half/bin/atomonly/newton
          stencil: half/bin/3d/newton
          bin: standard
    Setting up Verlet run ...
      Unit style    : lj
      Current step  : 0
      Time step     : 0.005
    Per MPI rank memory allocation (min/avg/max) = 11.78 | 11.78 | 11.78 Mbytes
    Step Temp E_pair E_mol TotEng Press
           0          1.5   -6.9322187            0    -4.682289   -5.2444747
         100   0.83086252   -5.8374799            0    -4.591225   0.44095734
         200   0.89207531   -5.7494799            0   -4.4114088   0.96545454
         300   0.95531327   -5.6359652            0   -4.2030401    1.6073066
         400    1.0310145   -5.5207836            0   -3.9743102    2.2626592
         500    1.1268149   -5.4204835            0    -3.730314    2.8367065
         600    1.2236803   -5.3148949            0   -3.4794318       3.4178
         700    1.3223056   -5.2158884            0    -3.232492    3.9589791
         800    1.4216384    -5.138538            0   -3.0061471    4.4004075
         900    1.4970804   -5.0685191            0   -2.8229687    4.7892089
        1000    1.5389536    -5.026557            0   -2.7181987    5.0230683
    Loop time of 10.6031 on 16 procs for 1000 steps with 32000 atoms

    Performance: 40742.857 tau/day, 94.312 timesteps/s
    93.5% CPU use with 4 MPI tasks x 4 OpenMP threads

    MPI task timing breakdown:
    Section |  min time  |  avg time  |  max time  |%varavg| %total
    ---------------------------------------------------------------
    Pair    | 7.2892     | 7.3881     | 7.4796     |   2.5 | 69.68
    Neigh   | 1.6673     | 1.6971     | 1.7218     |   1.6 | 16.01
    Comm    | 1.1018     | 1.2206     | 1.32       |   7.3 | 11.51
    Output  | 0.00041037 | 0.00055746 | 0.0009981  |   0.0 |  0.01
    Modify  | 0.24716    | 0.254      | 0.2575     |   0.8 |  2.40
    Other   |            | 0.04266    |            |       |  0.40

    Nlocal:    8000 ave 8012 max 7992 min
    Histogram: 2 0 0 0 0 1 0 0 0 1
    Nghost:    10569.8 ave 10586 max 10553 min
    Histogram: 2 0 0 0 0 0 0 0 0 2
    Neighs:    507169 ave 508901 max 505973 min
    Histogram: 1 1 0 0 1 0 0 0 0 1

    Total # of neighbors = 2028676
    Ave neighs/atom = 63.3961
    Neighbor list builds = 100
    Dangerous builds = 100
    Deleted 1515 atoms, new total = 30485
    1563 atoms in group pore_group
    Setting atom values ...
      1563 settings made for type
    28922 atoms in group frozen
    0.0512711169427587
    Setting up Verlet run ...
      Unit style    : lj
      Current step  : 0
      Time step     : 0.005
    Per MPI rank memory allocation (min/avg/max) = 14.51 | 14.51 | 14.51 Mbytes
    Step v_time c_pore_temp TotEng Press c_msd[4]
           0            0     1.556341   -4.7673613    3.5222692            0
          10         0.05    1.6226809   -4.7673334    3.4915247  0.011425426
          20          0.1    1.6655718   -4.7671312    3.4526968  0.042844607
          30         0.15     1.713296   -4.7667633    3.4119839  0.089785272
          40          0.2    1.7214711   -4.7663075    3.3903432   0.15034512
          50         0.25    1.7011665   -4.7657659    3.3860801   0.22240328
          60          0.3    1.6816469   -4.7651098    3.3891821   0.30244957
          70         0.35    1.6895606   -4.7643422     3.388494   0.38599542
          80          0.4    1.6966579   -4.7635004    3.3901041   0.47248322
          90         0.45    1.7025103   -4.7625217    3.3936222   0.56190999
         100          0.5    1.7026666   -4.7614323    3.3996442   0.65228415
         110         0.55    1.7225509   -4.7602455    3.3996811   0.74369183
         120          0.6    1.7698201   -4.7589869    3.3932972   0.84058649
         130         0.65    1.7450442   -4.7576001    3.4080245   0.94601375
         140          0.7    1.7751328    -4.756129    3.4064509    1.0553715
         150         0.75    1.8123885   -4.7545247    3.4014138    1.1672432
         160          0.8    1.8148844   -4.7528427     3.407761    1.2818844
         170         0.85    1.8189045   -4.7510512    3.4151745     1.397624
         180          0.9     1.880289   -4.7491948    3.4065472    1.5168278
         190         0.95    1.8956455   -4.7472306     3.409284    1.6417874
         200            1    1.9244998   -4.7452009    3.4086737    1.7738104
         210         1.05    1.9053031   -4.7430871    3.4221637    1.9141912
         220          1.1    1.9401364   -4.7409184    3.4205936    2.0581843
         230         1.15    1.9809537    -4.738688    3.4171726    2.2038772
         240          1.2    2.0247656   -4.7363801    3.4140281    2.3536711
         250         1.25    2.0537087   -4.7340105    3.4154858    2.5104525
         260          1.3    2.0673613   -4.7315587    3.4220972    2.6757947
         270         1.35    2.0966925   -4.7290595    3.4257685    2.8483904
         280          1.4    2.1266947   -4.7264723    3.4302612    3.0304433
         290         1.45    2.1793948   -4.7238233    3.4271879    3.2222913
         300          1.5    2.1782783   -4.7210866    3.4391421    3.4257233
         310         1.55    2.2429504   -4.7183069    3.4321996    3.6365847
         320          1.6    2.2769612   -4.7154596    3.4333721    3.8567057
         330         1.65    2.3373949    -4.712578    3.4292491    4.0873989
         340          1.7    2.3482396   -4.7095801    3.4388622    4.3288371
         350         1.75    2.3912769    -4.706572    3.4406851    4.5825644
         360          1.8    2.4335239   -4.7035319    3.4426313    4.8484216
         370         1.85    2.4392775   -4.7003916    3.4556431    5.1261019
         380          1.9    2.4967827   -4.6972225    3.4528619    5.4132875
         390         1.95    2.5343972     -4.69399    3.4544748    5.7143598
         400            2    2.5811021   -4.6906839    3.4551043    6.0325748
         410         2.05    2.6073461   -4.6873208    3.4611769    6.3709376
         420          2.1    2.6672956   -4.6838565    3.4572212    6.7220757
         430         2.15    2.7217369   -4.6803161    3.4550452     7.088626
         440          2.2    2.7728282   -4.6766935    3.4552192    7.4768602
         450         2.25    2.8125359   -4.6729978    3.4608268    7.8858543
         460          2.3    2.8392562   -4.6692151    3.4687243    8.3099824
         470         2.35    2.9059135   -4.6653987    3.4682056    8.7507891
         480          2.4    2.9219086   -4.6614783    3.4825516    9.2076049
         490         2.45    2.9956589   -4.6574832    3.4810218    9.6769098
         500          2.5    3.0409905   -4.6533953    3.4868586    10.169484
         510         2.55    3.1103074   -4.6492661    3.4845959     10.68513
         520          2.6    3.1228467   -4.6450388    3.4965075    11.222315
         530         2.65    3.1640407   -4.6408176    3.5011616    11.782757
         540          2.7    3.2157045   -4.6365925    3.5048666    12.367732
         550         2.75    3.2638322   -4.6323625    3.5093234    12.979374
         560          2.8    3.3063086   -4.6280637    3.5118129    13.612547
         570         2.85    3.3836587   -4.6237059    3.5040435     14.26135
         580          2.9    3.4552339   -4.6193082    3.4994723    14.937564
         590         2.95    3.5108801   -4.6148805    3.5012002    15.647481
         600            3    3.5311193   -4.6103609    3.5157058    16.392895
         610         3.05    3.5464745   -4.6058049    3.5300573    17.168815
         620          3.1    3.6188034   -4.6011895    3.5257833    17.970289
         630         3.15    3.6595983   -4.5965362    3.5304719    18.791107
         640          3.2    3.7340594   -4.5918326    3.5260769    19.634251
         650         3.25    3.7783189   -4.5870306     3.531801    20.520442
         660          3.3    3.8638215    -4.582158    3.5279484    21.442286
         670         3.35    3.9037622   -4.5772234    3.5385869    22.395289
         680          3.4      3.97255   -4.5722393     3.541191    23.376589
         690         3.45    4.0052179   -4.5671354    3.5525372    24.388756
         700          3.5    4.1409667   -4.5620386    3.5377442    25.435826
         710         3.55    4.1350852   -4.5568611     3.564173    26.524506
         720          3.6    4.2289219   -4.5516577     3.561721    27.644657
         730         3.65    4.2729326   -4.5463693    3.5741881    28.792112
         740          3.7    4.3117577   -4.5410201    3.5867274    29.971253
         750         3.75     4.413177   -4.5356307    3.5786625    31.177907
         760          3.8    4.4460504   -4.5301691    3.5902842    32.420125
         770         3.85    4.5246194   -4.5246956    3.5913368    33.703625
         780          3.9    4.6414188   -4.5191244    3.5826175    35.032894
         790         3.95    4.6870589    -4.513518     3.594336    36.399381
         800            4    4.7286654   -4.5078875    3.6066993    37.796534
         810         4.05    4.8013106   -4.5022191    3.6085756    39.228591
         820          4.1    4.8916862   -4.4965008    3.6029174    40.703725
         830         4.15    4.9599314   -4.4907202    3.6033377    42.235911
         840          4.2    5.0219392   -4.4848984    3.6075539    43.816934
         850         4.25    5.1062076   -4.4791214    3.6097204    45.429532
         860          4.3    5.1020194   -4.4733336    3.6358367    47.089802
         870         4.35    5.2087081   -4.4674839    3.6296674    48.796464
         880          4.4    5.2683748   -4.4615735    3.6365082    50.546853
         890         4.45    5.3466911   -4.4556667    3.6401483    52.333698
         900          4.5    5.4202749   -4.4496857    3.6453895    54.159416
         910         4.55     5.442964   -4.4436122    3.6638307    56.029053
         920          4.6    5.5736588   -4.4374595    3.6532684    57.948436
         930         4.65    5.6373394   -4.4312087    3.6611902    59.918958
         940          4.7    5.7047759   -4.4250029    3.6709245    61.939679
         950         4.75     5.809896   -4.4186667    3.6695655    64.007749
         960          4.8    5.8607493   -4.4122757      3.68209    66.126796
         970         4.85    5.9937632   -4.4058615    3.6718414    68.308833
         980          4.9    5.9903261   -4.3993673    3.7003435    70.545818
         990         4.95    6.0405795   -4.3928669    3.7167414    72.804312
        1000            5    6.1547772   -4.3864834     3.712641    75.088874
        1010         5.05     6.169011   -4.3800108    3.7326623    77.404419
        1020          5.1     6.292757   -4.3735769    3.7231717    79.736061
        1030         5.15    6.3682259   -4.3670396     3.723185     82.11999
        1040          5.2    6.4206349    -4.360483    3.7319124    84.558484
        1050         5.25    6.4999371   -4.3538959    3.7333883    87.085958
        1060          5.3    6.6407645   -4.3471855    3.7218887    89.672039
        1070         5.35     6.694731   -4.3404438    3.7378542    92.315095
        1080          5.4    6.8082153   -4.3337804    3.7373684    94.989193
        1090         5.45    6.8669947   -4.3271236     3.750399     97.71253
        1100          5.5    6.9254099   -4.3205158    3.7660717     100.5007
        1110         5.55    6.9522028     -4.31381    3.7857955    103.33737
        1120          5.6    7.0895692   -4.3070086    3.7769121    106.21248
        1130         5.65     7.153069   -4.3001777    3.7878875    109.16447
        1140          5.7    7.2607358   -4.2932809    3.7835553    112.20519
        1150         5.75     7.365323   -4.2863778    3.7807459     115.2981
        1160          5.8    7.3789323   -4.2793241      3.80624    118.41961
        1170         5.85    7.5344185   -4.2722096    3.7947038    121.55417
        1180          5.9     7.603916   -4.2651101    3.8044999    124.76891
        1190         5.95    7.6493534   -4.2579034    3.8215811     128.0403
        1200            6    7.7243514   -4.2506917    3.8279771    131.35705
        1210         6.05    7.8957988   -4.2436022    3.8127795    134.70828
        1220          6.1    7.8639726   -4.2364641    3.8504726    138.10576
        1230         6.15    7.9890518   -4.2293299    3.8459574    141.52985
        1240          6.2    7.9959484   -4.2221758    3.8757307    144.99745
        1250         6.25    8.1612554    -4.215057    3.8641086    148.53775
        1260          6.3    8.1904204   -4.2078877     3.885333    152.14397
        1270         6.35    8.3317835   -4.2007707    3.8744512    155.79355
        1280          6.4    8.4156884    -4.193553    3.8790428    159.48706
        1290         6.45    8.5277462   -4.1863578    3.8811522    163.22179
        1300          6.5    8.5690486   -4.1791639    3.8997742    167.01224
        1310         6.55    8.6897743   -4.1718686    3.8985563    170.88247
        1320          6.6    8.7752418   -4.1646888    3.9011567    174.82238
        1330         6.65    8.8612533   -4.1573431    3.9009047    178.80225
        1340          6.7    8.9370504   -4.1500639    3.9073067    182.82034
        1350         6.75    8.9387275   -4.1426495    3.9354374    186.89551
        1360          6.8    9.0360622   -4.1352794    3.9384127    191.01318
        1370         6.85    9.2185395   -4.1280723    3.9256637    195.18953
        1380          6.9    9.1624731   -4.1206426    3.9743679    199.39962
        1390         6.95    9.3537523   -4.1132829      3.95501     203.6827
        1400            7    9.4709815   -4.1059197    3.9526517    208.04778
        1410         7.05    9.5148779   -4.0982914    3.9678767    212.46407
        1420          7.1    9.4800288   -4.0908445     4.005119    216.94792
        1430         7.15    9.5583137   -4.0834148    4.0197104      221.412
        1440          7.2    9.6839914   -4.0758483    4.0145303    225.90504
        1450         7.25    9.8125949   -4.0683846    4.0053541    230.49875
        1460          7.3    9.9640461   -4.0607265     3.993496    235.15464
        1470         7.35    10.070616   -4.0534286    3.9980634    239.84238
        1480          7.4     10.02006   -4.0460085    4.0444117    244.58398
        1490         7.45    10.112509   -4.0385112    4.0508279    249.32992
        1500          7.5     10.26053   -4.0309783    4.0422123    254.06603
        1510         7.55    10.295687   -4.0235474    4.0670172    258.89719
        1520          7.6    10.438377   -4.0160952    4.0600412    263.81716
        1530         7.65    10.534836   -4.0085217    4.0622085    268.83565
        1540          7.7    10.545572   -4.0010135    4.0894785    273.90054
        1550         7.75    10.654369   -3.9935985    4.0894985    278.95691
        1560          7.8    10.739603   -3.9859427    4.1001853    284.06874
        1570         7.85    10.829569   -3.9786615    4.1104049    289.20265
        1580          7.9    10.907113   -3.9711701    4.1144077    294.38617
        1590         7.95    11.051152   -3.9637244    4.1082897    299.64862
        1600            8    11.043228   -3.9562872    4.1444703    304.98818
        1610         8.05    11.221035   -3.9489497    4.1240523     310.3174
        1620          8.1    11.354103   -3.9415996    4.1142658     315.7089
        1630         8.15    11.279482   -3.9340781    4.1646239    321.20789
        1640          8.2    11.495513     -3.92657    4.1374229    326.75232
        1650         8.25    11.513198   -3.9189756    4.1593595    332.31365
        1660          8.3    11.609601   -3.9112872    4.1617421     337.9165
        1670         8.35    11.735535   -3.9039064    4.1585564    343.53149
        1680          8.4    11.835482   -3.8963475    4.1584949     349.1453
        1690         8.45    11.820409   -3.8886367     4.190376    354.83638
        1700          8.5    12.010618   -3.8811582    4.1760646    360.54281
        1710         8.55    11.907459   -3.8734353    4.2393973    366.33858
        1720          8.6     12.12771   -3.8661854     4.215323    372.16206
        1730         8.65    12.069916   -3.8583743    4.2614665    377.97333
        1740          8.7    12.292248   -3.8510463    4.2353259    383.88094
        1750         8.75    12.429439    -3.843436    4.2307765    389.87615
        1760          8.8    12.470699   -3.8360189     4.247228     395.8929
        1770         8.85    12.559041   -3.8287621    4.2500861    401.93598
        1780          8.9    12.606793   -3.8211243    4.2659459    407.98863
        1790         8.95    12.669929   -3.8134716    4.2739709    414.06308
        1800            9    12.930027   -3.8064286    4.2304414    420.18528
        1810         9.05    12.909803   -3.7987114    4.2594595     426.5081
        1820          9.1    13.036669   -3.7911148    4.2559203     432.9299
        1830         9.15    13.100235   -3.7836055    4.2758046    439.41459
        1840          9.2    13.146261   -3.7758944    4.3019927    446.04417
        1850         9.25    13.188015   -3.7683086    4.3271723    452.70219
        1860          9.3    13.345279   -3.7609373     4.319715     459.2758
        1870         9.35    13.434558   -3.7531937    4.3194898    465.91237
        1880          9.4     13.44966   -3.7455567    4.3435133    472.63881
        1890         9.45    13.693175   -3.7376995    4.3138342    479.28206
        1900          9.5    13.658181   -3.7298423    4.3542712    485.91225
        1910         9.55    13.760613    -3.722533    4.3552248    492.60663
        1920          9.6    13.773225   -3.7149874    4.3802429    499.29751
        1930         9.65    13.771261   -3.7076571    4.4086344    505.93416
        1940          9.7    13.982591   -3.7001272    4.3717909     512.4918
        1950         9.75    14.100515   -3.6931397    4.3733475    519.07428
        1960          9.8    14.120054   -3.6859854     4.404141    525.67556
        1970         9.85    14.240262   -3.6785779    4.4048824    532.32624
        1980          9.9    14.306705   -3.6717108    4.4181481    538.89121
        1990         9.95    14.264276   -3.6646102    4.4525492    545.43917
        2000           10    14.524766   -3.6572645    4.4161283    551.90178
    Loop time of 22.7507 on 16 procs for 2000 steps with 30485 atoms

    Performance: 37976.921 tau/day, 87.910 timesteps/s
    92.4% CPU use with 4 MPI tasks x 4 OpenMP threads

    MPI task timing breakdown:
    Section |  min time  |  avg time  |  max time  |%varavg| %total
    ---------------------------------------------------------------
    Pair    | 12.964     | 13.258     | 13.726     |   7.8 | 58.28
    Neigh   | 2.9157     | 2.97       | 3.0289     |   2.3 | 13.05
    Comm    | 2.5761     | 3.0318     | 3.3308     |  16.0 | 13.33
    Output  | 3.29       | 3.2929     | 3.3008     |   0.3 | 14.47
    Modify  | 0.086288   | 0.091819   | 0.097341   |   1.4 |  0.40
    Other   |            | 0.1058     |            |       |  0.47

    Nlocal:    7621.25 ave 7644 max 7602 min
    Histogram: 1 0 1 0 0 1 0 0 0 1
    Nghost:    9735.25 ave 9768 max 9698 min
    Histogram: 1 0 0 0 1 0 1 0 0 1
    Neighs:    468020 ave 472222 max 462849 min
    Histogram: 1 0 0 0 1 0 0 1 0 1

    Total # of neighbors = 1872079
    Ave neighs/atom = 61.4098
    Neighbor list builds = 200
    Dangerous builds = 200
    Total wall time: 0:00:33

```python
df, num_atoms, bounds = read_dump("dat/flow_profile.lammpstrj")
```

```python
centre_bounds = [np.sum(bound) / 2 for bound in bounds]
```

```python
df.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>type</th>
      <th>x</th>
      <th>y</th>
      <th>z</th>
      <th>vx</th>
      <th>vy</th>
      <th>vz</th>
      <th>timestep</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>1</td>
      <td>1.317050</td>
      <td>0.539350</td>
      <td>1.321310</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>1</td>
      <td>0.730629</td>
      <td>0.913395</td>
      <td>0.438076</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>1</td>
      <td>1.489180</td>
      <td>0.110520</td>
      <td>0.252968</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>1</td>
      <td>2.822600</td>
      <td>0.398555</td>
      <td>0.965105</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>1</td>
      <td>2.492060</td>
      <td>1.469520</td>
      <td>0.833761</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_flowing = df[df.type == 2]
```

```python
df_flowing.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>type</th>
      <th>x</th>
      <th>y</th>
      <th>z</th>
      <th>vx</th>
      <th>vy</th>
      <th>vz</th>
      <th>timestep</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>5456</th>
      <td>5457</td>
      <td>2</td>
      <td>0.047327</td>
      <td>14.4051</td>
      <td>11.5221</td>
      <td>0.261905</td>
      <td>1.547730</td>
      <td>-0.142932</td>
      <td>0</td>
    </tr>
    <tr>
      <th>5461</th>
      <td>5462</td>
      <td>2</td>
      <td>4.995970</td>
      <td>14.3578</td>
      <td>11.5652</td>
      <td>-0.814679</td>
      <td>0.088776</td>
      <td>2.151870</td>
      <td>0</td>
    </tr>
    <tr>
      <th>5471</th>
      <td>5472</td>
      <td>2</td>
      <td>7.869410</td>
      <td>14.3234</td>
      <td>11.7257</td>
      <td>-0.107854</td>
      <td>-1.664720</td>
      <td>-0.679219</td>
      <td>0</td>
    </tr>
    <tr>
      <th>5476</th>
      <td>5477</td>
      <td>2</td>
      <td>9.216250</td>
      <td>14.1895</td>
      <td>11.6376</td>
      <td>2.152910</td>
      <td>-1.234740</td>
      <td>-0.632470</td>
      <td>0</td>
    </tr>
    <tr>
      <th>5490</th>
      <td>5491</td>
      <td>2</td>
      <td>32.132100</td>
      <td>16.2043</td>
      <td>16.1695</td>
      <td>-1.195920</td>
      <td>-0.712419</td>
      <td>-1.389600</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>

```python
df_flowing.x -= centre_bounds[0]
df_flowing.y -= centre_bounds[1]
df_flowing.z -= centre_bounds[2]
```

    /home/schoyen/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py:5096: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      self[name] = value

```python
df_flowing.loc["r"] = np.sqrt(df_flowing["y"] ** 2 + df_flowing["z"] ** 2)
```

    /home/schoyen/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      """Entry point for launching an IPython kernel.

```python
df_flowing.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>type</th>
      <th>x</th>
      <th>y</th>
      <th>z</th>
      <th>vx</th>
      <th>vy</th>
      <th>vz</th>
      <th>timestep</th>
      <th>r</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>5456</th>
      <td>5457.0</td>
      <td>2.0</td>
      <td>-16.751499</td>
      <td>-2.393725</td>
      <td>-5.276725</td>
      <td>0.261905</td>
      <td>1.547730</td>
      <td>-0.142932</td>
      <td>0.0</td>
      <td>5.794286</td>
    </tr>
    <tr>
      <th>5461</th>
      <td>5462.0</td>
      <td>2.0</td>
      <td>-11.802855</td>
      <td>-2.441025</td>
      <td>-5.233625</td>
      <td>-0.814679</td>
      <td>0.088776</td>
      <td>2.151870</td>
      <td>0.0</td>
      <td>5.774897</td>
    </tr>
    <tr>
      <th>5471</th>
      <td>5472.0</td>
      <td>2.0</td>
      <td>-8.929415</td>
      <td>-2.475425</td>
      <td>-5.073125</td>
      <td>-0.107854</td>
      <td>-1.664720</td>
      <td>-0.679219</td>
      <td>0.0</td>
      <td>5.644850</td>
    </tr>
    <tr>
      <th>5476</th>
      <td>5477.0</td>
      <td>2.0</td>
      <td>-7.582575</td>
      <td>-2.609325</td>
      <td>-5.161225</td>
      <td>2.152910</td>
      <td>-1.234740</td>
      <td>-0.632470</td>
      <td>0.0</td>
      <td>5.783323</td>
    </tr>
    <tr>
      <th>5490</th>
      <td>5491.0</td>
      <td>2.0</td>
      <td>15.333275</td>
      <td>-0.594525</td>
      <td>-0.629325</td>
      <td>-1.195920</td>
      <td>-0.712419</td>
      <td>-1.389600</td>
      <td>0.0</td>
      <td>0.865743</td>
    </tr>
  </tbody>
</table>
</div>

```python
r = df_flowing[df_flowing.timestep == np.max(df_flowing.timestep)].r

hist, bins = np.histogram(
    np.abs(df_flowing[df_flowing.timestep == np.max(df_flowing.timestep)].vx),
    np.linspace(0, np.max(r), 101)
)
```

```python
fig = plt.figure(figsize=(14, 10))

plt.hist(hist, bins)
plt.show()
```

```python
fig = plt.figure(figsize=(14, 10))

plt.scatter(
    df_flowing[df_flowing.timestep == np.max(df_flowing.timestep)].r,
    np.abs(df_flowing[df_flowing.timestep == np.max(df_flowing.timestep)].vx)
)
plt.show()
```


# Symbolic Mathematics in Python

There are times when you need to solve a difficult problem symbollically or analytically. If you have ever used Wolfram Alpha, then you have already done this. Sympy is a python library that allows you to do symbolic mathematics in python.

```python
import sympy as sym
```

## 1. Introduction

### Example 1.1

If you try to write the follwing in python by itself, you will get an error telling you x is undefined:
$$x-x$$

```python
x-x
```

(The error above is on purpose). Variables in python need to be defined before you can say something specific about them

```python
x=102
x-x
```

If you are trying to show that $x-x=0$ is true for any $x$, the above answer would not be valid. Instead you can use a symbolic expression to show that it is true

**First we define the variable as a symmbolic expression**

```python
x = sym.symbols('x')
```

**Now we can use the variable in a symbolic expression**

```python
x-x
```

### Example 1.2

Sympy can be used to perform algebreic operations (among other things). Consider the following expression: $$(3a-4b)^3$$

We can use symppy to expand the expression algebraically.

**First we need to define the variables as symbolic expressions**

```python
a,b = sym.symbols('a,b')
```

**Side note** Notice that the left hand side of the epression has two variables being defined. Python can define more than one variable at a time:

```python
x1,y1 =10,20
print(x1)
print(y1)
```

    10
    20

**Back to the expression** We can define an expression using the variables $a$ and $b$.

```python
expr = (3*a-4*b)**3
print(expr)
```

    (3*a - 4*b)**3

We can also make it look nicer in our notebook. This doesn't affect the math, but it makes our notebook more readable.

```python
sym.init_printing()
```

```python
expr
```

**Now we expand the function algebreically**

```python
expr.expand()
```

Sympy can also factor the equation

```python
sym.factor(26*a**3-108*a**2*b+144*a*b**2-64*b**3)
```

If you want to copy and paste a result, you print the result.

```python
print(sym.factor(26*a**3-108*a**2*b+144*a*b**2-64*b**3))
```

    2*(a - 2*b)*(13*a**2 - 28*a*b + 16*b**2)

You can also chain together functions

```python
expr.expand().factor()
```

### Exercise 1.1

Show that the following two expressions are true.
$$(2w-3z)(2w+3z)=4w^2-9z^2$$
$$(2w-3z)^2\ne4w^2-9z^2$$

```python
w,z = sym.symbols('w,z')
expr = (2*w-3*z)*(2*w+3*z)
```

```python
expr
```

```python
sym.expand(expr).factor(expr)
```

```python
expr1 = (2*w-3*z)**2
```

```python
expr1
```

```python
sym.expand(expr1).factor(expr1)
```

## 2. Solving Equations

Sympy can be used to symbolilically solve equations. As before, you need to define which variables are symbols

### Example 2.1

Use sympy to solve the following equation
$$ax^3+bx^2+cx+d=0$$

```python
# Define the variables
a,b,c,d,x = sym.symbols('a,b,c,d,x')
```

```python
# Define the expression
expr=a*x**3+b*x**2+c*x+d
expr
```

We can use the `solvset` function to solve this equation

```python
solutions=sym.solveset(expr,x)
```

```python
print(solutions)
```

    {-(-3*c/a + b**2/a**2)/(3*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a), -(-3*c/a + b**2/a**2)/(3*(-1/2 - sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (-1/2 - sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a), -(-3*c/a + b**2/a**2)/(3*(-1/2 + sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)) - (-1/2 + sqrt(3)*I/2)*(sqrt(-4*(-3*c/a + b**2/a**2)**3 + (27*d/a - 9*b*c/a**2 + 2*b**3/a**3)**2)/2 + 27*d/(2*a) - 9*b*c/(2*a**2) + b**3/a**3)**(1/3)/3 - b/(3*a)}

```python
solutions
```

What if I need help. You can do this with any python function. `function?`

```python
# Run this command to see a help box
sym.solveset?
```

### Exercise 2.1

Use the `solveset` function to solve the following chemical problem.

Phosgene gas, $\text{COCl}_2$, dissociates at high temperatures according to the following equilibrium:

$$ \text{COCl}_2 \rightleftharpoons  \text{CO} + \text{Cl}_2 $$

At $\text{400 C}$, the equilibrium constant $K_c=8.05$.

If you start with a  $\text{0.250 M}$ phosgene sample at $\text{400 C}$, determine the concentrations of all species at equilibrium.

```python
sym.symbols('x')
expr2=x**2-((8.05*0.250))
solutions=sym.solveset(expr2,x)
```

```python
solutions
```

Why did you pick your answer?

## 3. Calculus

We can use also Sympy to differentiate and integrate. Let us experiment with differentiating the following expression:

$$x ^ 2 - \cos(x)$$

```python
sym.diff(x ** 2 - sym.cos(x), x)
```

Similarly we can integrate:

```python
sym.integrate(x ** 2 - sym.cos(x), x)
```

We can also carry out definite integrals:

```python
sym.integrate(x ** 2 - sym.cos(x), (x, 0, 5))
```

### Exercise 3.1

Use Sympy to calculate the following:

1. $\frac{d(x ^2 + xy - \ln(y))}{dy}$
1. $\int_0^5 e^{2x}\;dx$

### Exercise 3.2
Solve the following definate integral
$$\int\limits_{ - \infty }^\infty  {\frac{1}{{\sigma \sqrt {2\pi } }}{e^{ - \frac{1}{2}{{\left( {\frac{{x - \mu }}{\sigma }} \right)}^2}}}}$$

Hint, the sympy symbol for infinity is `oo`

```python
# Your code here
```

Lookup Gaussian functions: https://en.wikipedia.org/wiki/Gaussian_function
Does your answer maake sense?

## 4. Plotting with Sympy

Finally Sympy can be used to plot functions. Note that this makes use of [matplotlib](http://matplotlib.org/).

Let us plot $x^2$:

```python
expr = x ** 2
p = sym.plot(expr)
```

    <Figure size 640x480 with 1 Axes>

### Exercise 4.1 Plot the following function:

1. $y=x + cos(x)$
1. ${\frac{1}{{ \sqrt {2\pi } }}{e^{ - \frac{x^2}{2}}}}$

```python
# Your code here
```

# Lecture

## L1. Hydrogen Atom

Sympy has built in modules for the eigenfunctions of the hydrogen atom.

```python
import sympy.physics.hydrogen
import numpy as np
```

You can caluclate the eigenvalues ($E$) in Hartrees

`sym.physics.hydrogen.E_nl(n,Z)`

```python
sym.physics.hydrogen.E_nl(1,1)
```

We can use a loop to print out many energies

```python
for n in range(1,5):
    print(sym.physics.hydrogen.E_nl(n,1))
```

    -1/2
    -1/8
    -1/18
    -1/32

We can plot the hydrogen radial wavefunction (1s orbital)

```python
r = sympy.symbols('r')
sympy.physics.hydrogen.R_nl(1, 0, r, 1)
```

```python
sym.plot(sympy.physics.hydrogen.R_nl(1, 0, r, 1),(r,0,10.50))
```

And the probablity distribution function

```python
sympy.symbols('r')
prob_1s=sympy.physics.hydrogen.R_nl(1, 0, r, 1)*sympy.physics.hydrogen.R_nl(1, 0, r, 1)
prob_1s
```

```python
sym.plot(prob_1s,(r,0,10))
```

Plot a 2s orbital

```python
sympy.symbols('r')
prob_2s=sympy.physics.hydrogen.R_nl(2, 0, r, 1)*sympy.physics.hydrogen.R_nl(2, 0, r, 1)
prob_2s
```

```python
sym.plot(prob_2s,(r,0,10))
```

We can change the range to see the node better.

```python
sym.plot(prob_2s,(r,1,8))
```

Notice the node!

### Exercise L1.1

Plot the radial distriubution function for a 2p, 3s, 4s, and 3d orbital.

```python
# Your code here
```

```python

```


# Software profesional en Acústica 2020-21 (M2i)

*This notebook is inspired from Chapter 10 "Timoshenko beams", included in the book Ferreira, A. J., Fantuzzi, N. (2020). MATLAB codes for finite element analysis, Springer. Available in https://doi.org/10.1007/978-3-030-47952-7*

# Free vibrations of a clamped Timoshenko beam in FEniCS

This notebook compute the eigenfrequencies and eigenmodes associated with a Timoshenko beam with clamped endpoints using a standard finite element discretization with *FEniCS*. Consider a beam of length $L$, the the system of partial differential equations are given by
\begin{align}
    -\omega^2\rho A u_1-\frac{\kappa A E}{2(1+\nu)}(u_1''-\beta_1')=0& \qquad\text{in}\quad (0,L),\\
    -\omega^2\rho J_{1}\beta_1-\rho J_{1}\beta''-v(u_1'-\beta_1)=0 & \qquad\text{in}\quad (0,L),\\
    -\omega^2\rho A u_2-\frac{\kappa A E}{2(1+\nu)}(u_2''-\beta_2')=0& \qquad\text{in}\quad (0,L),\\
    -\omega^2\rho J_{2}\beta_2-\rho J_{2}\beta_2''-v(u_2'-\beta_2)=0 & \qquad\text{in}\quad (0,L),\\
    -\omega^2\rho Au_3-AEu_3''=0 & \qquad\text{in}\quad (0,L),\\
\end{align}
with clamped boundary conditions, i.e., $u_1(0)=u_1(L)=u_2(0)=u_2(L)=u_3(0)=u_3(L)=\beta_2(0)=\beta_2(L)=0$. In the equation written above,
 - $A$: area of the cross-section
 - $J_1$, $J_2$: second-moment of inertia with respect to the tranverse directions
 - $\rho$: mass density
 - $E$: Young's elastic modulus
 - $\nu$: Poisson coefficient
 - $\kappa$: shear correction factor

The eigenvalues to be computed are given by $\lambda=\omega^2$.

#### Import Python modules

```python
from dolfin import *
import numpy as np
import matplotlib.pyplot as plt

# Setting matplotlib
plt.style.use('seaborn-poster')
%matplotlib inline
```

#### Creation of the mesh

```python
# Create mesh of the interval [0,L] with n elements
L = 1.0; n=50
mesh = IntervalMesh(n,0.,L)
```

#### Definition of the constants in the model

```python
# Define constants in the model for a beam with rectangular cross section of b x h
h = 0.01; b = 1.
rhos=1. #  densidade de masa
A=h*b #  area seccion transversal
E=1e8 #  modulo de elongacion
G=E/(2.0*(1.0+nu)) #  modulo de cizallamento
nu=0.3 #  coeficiente de Poisson
Ix=b*h**3/12 #  segundo momento de inercia xx
Iy=h*b**3/12 #  segundo momento de inercia yy
kappa=5/6 #  factor corrector
rho0=Constant(rhos*A)
rho1x=Constant(rhos*Ix)
rho1y=Constant(rhos*Iy)
k0=Constant(kappa*A*E/(2.0*(1.0+nu)))
k1x=Constant(E*Ix)
k1y=Constant(E*Iy)
k3=Constant(A*E)
```

#### Define the function space and the test/trial functions

```python
CG1  = FiniteElement("CG", mesh.ufl_cell(), 1)
W = FunctionSpace(mesh, MixedElement([CG1, CG1, CG1, CG1, CG1]))

# Define the test and trial functions
(v1, phi1, v2, phi2, u3) = TestFunctions(W)
w = TrialFunction(W)
u1, beta1, u2, beta2, v3 = split(w)
```

#### Variational formulation

```python
# Define the forms
a = (k0*u1.dx(0)*v1.dx(0)-k0*beta1*v1.dx(0)+k1x*beta1.dx(0)*phi1.dx(0)-k0*u1.dx(0)*phi1+k0*beta1*phi1)*dx \
    +(k0*u2.dx(0)*v2.dx(0)-k0*beta2*v2.dx(0)+k1y*beta2.dx(0)*phi2.dx(0)-k0*u2.dx(0)*phi2+k0*beta2*phi2)*dx \
    +(k3*u3.dx(0)*v3.dx(0))*dx
b = (rho0*u1*v1+rho1x*beta1*phi1+rho0*u2*v2+rho1y*beta2*phi2+rho0*u3*v3)*dx
```

#### Boundary conditions

```python
# Null boundary data
zero = Constant(0.)
bc_u1 = DirichletBC(W.sub(0), zero, "on_boundary")
bc_u2 = DirichletBC(W.sub(2), zero, "on_boundary")
bc_beta2 = DirichletBC(W.sub(3), zero, "on_boundary")
bc_u3 = DirichletBC(W.sub(4), zero, "on_boundary")
bcs=[bc_u1, bc_u2, bc_beta2, bc_u3]
```

#### Assemble and apply the Dirichlet boundary conditions for the clamped case

```python
# Assemble the stiffness matrix (A) and mass matrix (B)
A = PETScMatrix()
B = PETScMatrix()
assemble(a, tensor=A)
assemble(b, tensor=B)
for bc in bcs:
    bc.apply(A)
    bc.apply(B)
```

#### Compute the smallest eigenvalues

```python
# Solve the eigensystem
esolver = SLEPcEigenSolver(A, B)
esolver.parameters["spectrum"] = "smallest real"
esolver.solve()
```

#### Plot the first three eigenmodes

```python
# Extract largest (first) eigenpair
cont = 0; tol = 1.
for i in range(25):
    r, c, xr, xc = esolver.get_eigenpair(i)
    eigenfreq = np.sqrt(r)/2.0/pi
    if eigenfreq > tol:
        print("Eigenfrequency (",i,"):", eigenfreq)
```

    Eigenfrequency ( 8 ): 68.51114751953378
    Eigenfrequency ( 9 ): 274.1757439547364
    Eigenfrequency ( 10 ): 617.3872163063614
    Eigenfrequency ( 11 ): 1098.8011486314717
    Eigenfrequency ( 12 ): 1719.335007145995
    Eigenfrequency ( 13 ): 2480.1675885896566
    Eigenfrequency ( 14 ): 2724.908932011892
    Eigenfrequency ( 15 ): 3382.738015945998
    Eigenfrequency ( 16 ): 4428.744049849873
    Eigenfrequency ( 17 ): 5000.822507570756
    Eigenfrequency ( 18 ): 5023.04038314834
    Eigenfrequency ( 19 ): 5620.139405284401
    Eigenfrequency ( 20 ): 6364.947063810959
    Eigenfrequency ( 21 ): 6959.129696003808
    Eigenfrequency ( 22 ): 8184.399486320883
    Eigenfrequency ( 23 ): 8448.16652378254
    Eigenfrequency ( 24 ): 10006.58102865365

```python
# Plot eigenmode 9th
r, c, xr, xc = esolver.get_eigenpair(8)
print("Eigenfrequency to plot:", np.sqrt(r)/2.0/pi)
# Initialize function and assign eigenvector
w = Function(W)
w.vector()[:] = xr
(u1, beta1, u2, beta2, u3) = w.split(True)
# Plot eigenfunction
plt.figure()
plot(u1, label=r"$Re(u_1)$")
plot(u2, label=r"$Re(u_2)$")
plot(u3, label=r"$Re(u_3)$")
plt.axis('tight')
plt.legend()
plt.figure()
plot(beta1, label=r"$Re(\beta_1)$")
plot(beta2, label=r"$Re(\beta_2)$")
plt.axis('tight')
plt.legend()
```

### Exercise
Using an analogous code, compute the eigenfrequencies of the Helmholtz problem with sound speed $c=340$ and Dirichlet boundary conditions $u(0)=u(L)=0$.

```python
## YOUR CODE HERE
```


```python
import numpy as np
import pandas as pd
import numba
from sympy import isprime
from math import sqrt
```

```python
cities = pd.read_csv('../data/raw/cities.csv', index_col=['CityId'])
```

```python
XY = np.stack((cities.X.astype(np.float32), cities.Y.astype(np.float32)), axis=1)
is_not_prime = np.array([not isprime(city_id) for city_id in cities.index], dtype=np.int32)
```

```python
@numba.jit('f8(i8[:])', nopython=True, parallel=False)
def pure_score(path):
    '''Pure path score without penalties.'''
    dist = 0.0
    for i in numba.prange(path.shape[0] - 1):
        a, b = XY[path[i]], XY[path[i+1]]
        dx, dy = a[0] - b[0], a[1] - b[1]
        dist += sqrt(dx * dx + dy * dy)
    return dist
```

```python
@numba.jit('f8(i8[:])', nopython=True, parallel=False)
def pure_score_v2(path):
    '''Pure path score without penalties.'''
    dist = 0.0
    for i in numba.prange(path.shape[0] - 1):
        a, b = XY[path[i]], XY[path[i+1]]
        dx, dy = a[0] - b[0], a[1] - b[1]
        dist += sqrt(dx * dx + dy * dy)
    return dist
```

```python
path = np.concatenate([cities.index, [0]])
```

```python
%timeit pure_score(path)
```

    1.13 ms ± 9.48 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
pure_score(path)
```

    443430860.1241331

```python
pure_score_v2(path)
```

    560957677.8028173

```python

```


+ This notebook is part of lecture 23 *Differential equations and exponent A* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols, exp
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

```python
u, u1, u2, t, a, b, c = symbols('u u1 u2 t a b c')
```

# Differential equations
# Exponential  eAt of a matrix

## Differential equations (ordinary only)

* A differential equation moves on from the previous lecture's difference equation which had finite steps to continuously changing systems (here in the time parameter *t*)
* A differential equation included a function and its derivative(s)
* It has and order based on the highest derivative that appears
* Here we are only concerned with differential equation with constant coefficients
* The simplest differential equation is the following
$$ \frac{dy}{dt}={a}{y}\left(t\right) $$
* It is simply solved in the following manner (which gives us some insight into the general solution for these equations)
$$ \frac { dy }{ dt } =ay\\ \frac { 1 }{ y } dy=adt\\ \int { \frac { 1 }{ y }  } dy=a\int {  } dt\\ \ln { \left| y \right|  } =a\left( t+{ c }_{ 1 } \right) \\ { e }^{ \ln { \left| y \right|  }  }={ e }^{ at+a{ c }_{ 1 } }\\ y={ e }^{ at }{ e }^{ a{ c }_{ 1 } }\\ y=c{ e }^{ at } $$
* We can solve for the constant(s) if we have values for the initial condition (usually *t*=0) for *y*(t) and all its derivatives (called *initial value problems*)
* We can write a system of differential equations in matrix form
$$ { y }_{ 1 }^{ ' }=3{ y }_{ 1 }\\ { y }_{ 2 }^{ ' }=-2{ y }_{ 2 }\\ { y }_{ 3 }^{ ' }=6{ y }_{ 3 }\\ \therefore \quad \begin{bmatrix} { y }_{ 1 }^{ ' } \\ { y }_{ 2 }^{ ' } \\ { y }_{ 3 }^{ ' } \end{bmatrix}=\begin{bmatrix} 3 & 0 & 0 \\ 0 & -2 & 0 \\ 0 & 0 & 6 \end{bmatrix}\begin{bmatrix} { y }_{ 1 } \\ { y }_{ 2 } \\ { y }_{ 3 } \end{bmatrix} $$

* Rewriting the above, we consider the following differential equations in this lecture
$$ \frac { d\underline { u }  }{ dt } =A\underline { u }  $$

* So suppose we have these two differential equations
$$ \frac{{d}\underline{u}_{1}}{{d}{t}} = -\underline{u}_{1}+2\underline{u}_{2} \\ \frac{{d}\underline{u}_{2}}{{d}{t}} = \underline{u}_{1}-2\underline{u}_{2} $$
* The intial consitions are given by the following
$$ \underline{u}\left({0}\right)=\begin{bmatrix}1\\0\end{bmatrix} $$

* We can write it as A**u**

```python
A = Matrix([[-1, 2], [1, -2]])
A
```

```python
u_vect = Matrix([u1, u2]) # u is now a sympy mathematical symbol
# Have to use another variable name, i.e. u_vect
u_vect
```

* Multiplying this A**u** brings you back to the two linear equations

```python
A * u_vect
```

```python
A.eigenvals()
```

```python
A.eigenvects() # The results give the two eigenvectors in the following format
# (eigenvalue, no of eigenvectors, eigenvector)
```

```python
S, D = A.diagonalize() # For interest sake we get the matrix of eigenvectors and the diagonal matrix
S, D
```

* To complete the solution now, we note that there are two eigenvalues, which will give us the following
    * Two constants
    * Two exponent to the power eigenvalue times *t*
    * Two eigenvectors
* It is written like this, with **x**i denoting an eigenvector
$$ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ { \lambda  }_{ 1 }t }{ \underline { x }  }_{ 1 }+{ c }_{ 2 }{ e }^{ { \lambda  }_{ 2 }t }{ \underline { x }  }_{ 2 } $$
* This makes our solution as follows
$$ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ -3t }\begin{bmatrix} -1 \\ 1 \end{bmatrix}+{ c }_{ 2 }{ e }^{ \left( 0 \right) t }\begin{bmatrix} 2 \\ 1 \end{bmatrix}\\ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ -3t }\begin{bmatrix} -1 \\ 1 \end{bmatrix}+{ c }_{ 2 }\begin{bmatrix} 2 \\ 1 \end{bmatrix} $$
* There is clearly a constant term and a term that approaches zero at *t* approaches infinity

* Writing this as two separate equations we have the following
$$ { u }_{ 1 }\left( t \right) =-{ c }_{ 1 }{ e }^{ -3t }+2{ c }_{ 2 }\\ { u }_{ 2 }\left( t \right) ={ c }_{ 1 }{ e }^{ -3t }+{ c }_{ 2 }  $$

* Using the initial conditions we can solve for *c*i
$$ { u }_{ 1 }\left( 0 \right) =-{ c }_{ 1 }{ e }^{ -3\left( 0 \right)  }+2{ c }_{ 2 }=1\\ { u }_{ 2 }\left( 0 \right) ={ c }_{ 1 }{ e }^{ -3\left( 0 \right)  }+{ c }_{ 2 }=0\\ -{ c }_{ 1 }+2{ c }_{ 2 }=1\\ { c }_{ 1 }+{ c }_{ 2 }=0 $$

* Let's use an augmented matrix and Gauss-Jordan elimination to solve for the two constants (or at least the python™ equivalent)

```python
C = Matrix([[-1, 2, 1], [1, 1, 0]])
C.rref()
```

* Which gives us the final solution
$$  { u }_{ 1 }\left( t \right) =\frac { 1 }{ 3 } { e }^{ -3t }+\frac { 2 }{ 3 } \\ { u }_{ 2 }\left( t \right) =\frac { -1 }{ 3 } { e }^{ -3t }+\frac { 1 }{ 3 }   $$

* Just to remind ourselves about the previous lecture where we had difference equations and had something like the following
$$ \underline{u}={c}_{1}{\lambda}_{1}^{k}{\underline{x}}_{1}+{c}_{2}{\lambda}_{2}^{k}{\underline{x}}_{2} $$
* Which is for finite steps, i.e. stepping by one
$$ \underline{u}_{k+1}={A}\underline{u}_{k} $$

## What can the eigenvalues tell us about these equations as *t* approaches ∞

* If both eigenvalues (real parts) are negative, the equation **t**(t) approaches **0** (called *stability*)
* If one eigenvalue (real part) is zero and the others (real parts) are less than one, the equations reach is specific value (called a *steady state*)
* If any eigenvalue (real parts) is larger than zero, the equations approach ±∞

## What can the matrix A2×2 tell us about the eigenvalues (and then what happens when *t* approaches ∞)

* The trace is equal to the sum of the eigenvalues
* The determinant is the product of the eigenvalues

* If the trace is negative **and** the determinant positive, we will have stability

## Using diagonalization

* Consider the following derivation
$$ \frac { d\underline { u }  }{ dt } =A\underline { u } \\ \because \quad A\underline { u } =S\underline { v } \\ S\frac { d\underline { v }  }{ dt } =S\underline { v } \\ \frac { d\underline { v }  }{ dt } =S\underline { v } { S }^{ -1 }=\Lambda \underline { v }  $$

* From this we have the following
$$ \underline { v } \left( t \right) ={ e }^{ \Lambda t }\underline { v } \left( 0 \right) \\ \underline { u } \left( t \right) =S{ e }^{ \Lambda t }{ S }^{ -1 }\underline { u } \left( 0 \right) \\ {e}^{At}={S}{e}^{\Lambda{t}}{S}^{-1} $$

## Matrix exponential *e*At

* How do we calculate a matrix as a power?
* Consider Taylor series expansion
$$ {e}^{At}={I}+{At} + \frac{{\left(At\right)}^{2}}{2!}+\frac{{\left(At\right)}^{3}}{3!}+\dots+\frac{{\left(At\right)}^{n}}{n!} $$
* This comes from the following
$$ { e }^{ x }=\sum _{ n=0 }^{ \infty  }{ \frac { { x }^{ n } }{ n! }  }  $$
* As the denominator increases the *n*th term approaches 0
* Remember also this (geometric) series (just for fun)
$$ \frac { 1 }{ 1-x } =\sum _{ 0 }^{ \infty  }{ { x }^{ n } } \\ { \left( I-At \right)  }^{ -1 }=I+At+{ \left( At \right)  }^{ 2 }+\dots { \left( At \right)  }^{ n } $$
* This will blow up unless then eigenvalues of A are less than 1

* Now, let calculate *e*At, remembering the following
$$ {A}^{k}={S}{\Lambda}^{k}{S}^{-1} $$
$$ { e }^{ At }=\sum _{ n=0 }^{ \infty  }{ \frac { \left( S{ \Lambda  }^{ n }{ S }^{ -1 } \right) ^{ n }{ t }^{ n } }{ n! }  }  $$

* We thus have the following
$$ {e}^{At}={S}{e}^{\Lambda{t}}{S}^{-1} $$
* This is with the assumption that A can be diagonalized (otherwise we will have to use the infinite series (above)
$$ {e}^{At}={I}+{At} + \frac{{\left(At\right)}^{2}}{2!}+\frac{{\left(At\right)}^{3}}{3!}+\dots+\frac{{\left(At\right)}^{n}}{n!} $$

* Remember that Λ is a diagonal matrix and therefor we would have the following
$$ { e }^{ \Lambda t }=\begin{bmatrix} { e }^{ { \lambda  }_{ 1 }t } & 0 & 0 & 0 \\ 0 & { e }^{ { \lambda  }_{ 2 }t } & 0 & 0 \\ \vdots  & \vdots  & \dots  & \vdots  \\ 0 & 0 & 0 & { e }^{ { \lambda  }_{ n }t } \end{bmatrix} $$

* The S and S-1 matrices are stable, it is therefor the Λ matrix that provides an approach to zero as *t* approaches ∞
* This is achieved by every λi having a real part less than zero

* The powers of A go to zero if the absolute value of the real part of all the λi-values is less than 1

+ Let's consider this example

$$ \underline { y } ''+b\underline { y } '+{k}\underline{y}=\underline { 0 }  $$

* We have to create a system of two first-order equations
$$ \underline{u}=\begin{bmatrix} y' \\ y \end{bmatrix} \\ \underline{u}'=\begin{bmatrix}{y}''\\{y}'\end{bmatrix}=\begin{bmatrix} -{b} & -{k} \\ 1 & 0 \end{bmatrix}\begin{bmatrix} {y}' \\ {y} \end{bmatrix}$$

## Example problems

### Example problem 1

* Find the general solutions, the matrix A, and the first column of *e*At of the following third-order, ordinary, homogeneous differential equation with constant coefficients
$$ \frac { { d }^{ 3 }y }{ d{ t }^{ 3 } } +2\frac { { d }^{ 2 }y }{ d{ t }^{ 2 } } -\frac { dy }{ dt } -2y=0 $$

#### Solution

* Since the differential equation is third order, we need to create three first-order differential equations
$$ \frac { { d }^{ 3 }y }{ d{ t }^{ 3 } } +2\frac { { d }^{ 2 }y }{ d{ t }^{ 2 } } -\frac { dy }{ dt } -2y=0\\ \therefore \quad y'''=-2y''+y'+2\\ \underline { u } =\begin{bmatrix} y'' \\ y' \\ y \end{bmatrix}\\ \underline { u } '=\begin{bmatrix} y''' \\ y'' \\ y' \end{bmatrix}\\ \underline { u } '=A\underline { u } =\begin{bmatrix} -2 & 1 & 2 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{bmatrix}\begin{bmatrix} y'' \\ y' \\ y \end{bmatrix} $$

* Notice that when we do the last matrix multiplication, we get exactly what we need
$$  \underline { u } '=\begin{bmatrix} y''' \\ y'' \\ y' \end{bmatrix}=\begin{bmatrix} -2y''+y'+2y \\ y'' \\ y' \end{bmatrix} $$

* We now have the matrix A and we can calculate *e*At if A is diagonalizable

```python
A = Matrix([[-2, 1, 2], [1, 0, 0], [0, 1, 0]])
A
```

```python
S, D = A.diagonalize()
S, D
```

* We can calculate *e*At from the following
$$ {e}^{At}={S}{e}^{\Lambda{t}}{S}^{-1} $$

* Remember that Λ is a diagonal matrix and therefor we would have the following
$$ { e }^{ \Lambda t }=\begin{bmatrix} { e }^{ { \lambda  }_{ 1 }t } & 0 & 0 & 0 \\ 0 & { e }^{ { \lambda  }_{ 2 }t } & 0 & 0 \\ \vdots  & \vdots  & \dots  & \vdots  \\ 0 & 0 & 0 & { e }^{ { \lambda  }_{ n }t } \end{bmatrix} $$

```python
A.eigenvals()
```

* This gives us the three eigenvalues from which we can create the diagonal matrix *e*Λt

```python
e_Lamda_t = Matrix([[exp(-2 * t), 0, 0], [0, exp(-t), 0], [0, 0, exp(t)]])
e_Lamda_t
```

* We have to multiply the following three matrices (with which python is not comfortable, so I'll do it in two steps)

```python
S, e_Lamda_t, S.inv()
```

```python
first_part = S * e_Lamda_t
first_part
```

```python
first_part * S.inv()
```

* We still need to write our general solution
$$ \underline { u } \left( t \right) ={ c }_{ 1 }{ e }^{ { \lambda  }_{ 1 }t }{ \underline { x }  }_{ 1 }+{ c }_{ 2 }{ e }^{ { \lambda  }_{ 2 }t }{ \underline { x }  }_{ 2 }+{ c }_{ 3 }{ e }^{ { \lambda  }_{ 3 }t }{ \underline { x }  }_{ 3 } $$

```python
A.eigenvects()
```

* With these eigenvalues and eigenvectors we get
$$ \underline { u } \left( t \right) =\begin{bmatrix} { y }'' \\ y' \\ y \end{bmatrix}={ c }_{ 1 }{ e }^{ -2t }\begin{bmatrix} 4 \\ -2 \\ 1 \end{bmatrix}+{ c }_{ s }{ e }^{ -t }\begin{bmatrix} 1 \\ -1 \\ 1 \end{bmatrix}+{ c }_{ 3 }{ e }^{ t }\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}\\ \therefore \quad y\left( t \right) ={ c }_{ 1 }{ e }^{ -2t }+{ c }_{ 2 }{ e }^{ -t }+{ c }_{ 3 }{ e }^{ t } $$

### Example problem 2

* Solve the following second-order ordinary differential equation
$$ y''-y'-6y=0 $$

#### Solution

* We need to create two first-order equations
$$ \therefore \quad y''=y'+6y\\ \because \quad \underline { u } \left( t \right) =\begin{bmatrix} y' \\ y \end{bmatrix}\\ \underline { u } '\left( t \right) =\begin{bmatrix} y'' \\ y' \end{bmatrix}\\ \therefore \quad \underline { u } '\left( t \right) =\begin{bmatrix} 1 & 6 \\ 1 & 0 \end{bmatrix}\underline { u } \left( t \right)  $$

```python
A = Matrix([[1, 6], [1, 0]])
A
```

```python
A.eigenvects()
```

* The solution is thus as follows
$$ {y}\left({t}\right)={c}_{1}{e}^{-2t}+{c}_{2}{e}^{3t} $$

```python

```


# vector
vector 具有两个属性, magnitude & direction. 因此可以表达任何可以由这两个属性
决定的量. 由此, 它不仅仅可以表达常见的各种物理量, 例如力, 场强等; 它还可以表达
平面. 因为平面就有两个量, 面积和方向. 令矢量的 magnitude 等于平面的面积, 矢量
的方向为平面的法向即可.

# tensor
张量是标量和矢量的一般化. 张量的阶表示一个张量的任一 component 需要几个基矢量
来表示. 表达某一分量的各个基矢量, 可以具有不同含义, 即源于不同含义的座标系.
或者说, 一个 $N$ 阶张量, 往往是代表由 $N$ 个不同含义的简单座标系组合起来的一个
复合座标系统.

例如, 压强二阶张量 $T$ 的一个 component $T_{ij}$, 表示 $i$ 方向表面上的 $j$ 方向
的压强分量. 这个张量就是两个座标系统的复合下生成的.

需要不同阶的张量, 是因为我们在考察对象时, 对它的组成单元在不同情况下需要不同的假设.
一个作用可以是作用在点上的, 矢量即可描述; 若是作用在有限大小的体积上, 则需要 2 阶
张量去描述.

# vector-valued function

值域为多维矢量的集合的函数. 注意对定义域没有要求, 可以是标量, 也可以是任意维度的矢量.

例子:
-   自变量为标量, 值为矢量: 参数曲线.
    $$\vec{r}(t) = f(t)\hat{i}+g(t)\hat{j}+h(t)\hat{k}$$
-   自变量为 2 维矢量, 值为 3 维矢量: 参数曲面.
    $$(x,y,z) = (f(s,t), g(s,t), h(s,t)) \equiv F(s,t)$$
-   自变量为 n 维矢量, 值为 m 维矢量. 矩阵.
    $$\mathbf{y} = \mathbf{A}\mathbf{x}+\mathbf{b}$$

# Jacobian matrix and determinant

The matrix of all first-order partial derivatives of a
vector-valued function.
对于函数 $\mathbf{T}(\mathbf{x}): \mathbb{R}^n \rightarrow \mathbb{R}^m$,
Jacobian matrix 定义为:
$$\mathbf{J}_\mathbf{T} = \frac{\partial (T_1,\dots,T_m)}{\partial (x_1,\dots,x_n)} = \begin{bmatrix}
    \dfrac{\partial \mathbf{T}}{\partial x_1} & \cdots & \dfrac{\partial \mathbf{T}}{\partial x_n} \end{bmatrix}
= \begin{bmatrix}
    \dfrac{\partial T_1}{\partial x_1} & \cdots & \dfrac{\partial T_1}{\partial x_n}\\
    \vdots & \ddots & \vdots\\
    \dfrac{\partial T_m}{\partial x_1} & \cdots & \dfrac{\partial T_m}{\partial x_n} \end{bmatrix}$$
Jacobian $\mathbf{J}_\mathbf{T}(\mathbf{x})$ 是 $\mathbf{T}$ 在 $\mathbf{x}$ 点的全导数.

the Jacobian for a scalar-valued multivariate function is the gradient and that of a scalar-valued function of single variable is simply its derivative.

## Jacobian derminant and variable substitution in multiple integral
当 $m=n$ 时, Jacobian matrix 的行列式称为 Jacobian determinant.
Jacobian 行列式在多元积分换元计算时出现. 具体讲,
若 $\mathbf{T}$ 是两个座标系之间点的变换关系, 设 $\mathbf{x}$ 为第一个座标系中的点, $\mathbf{y}$ 为第二个座标系中的点, 而 $\mathbf{y} = (T_1(\mathbf{x}),\dots,T_n(\mathbf{x}))$, 则任意函数 $f$ 在第二个座标系中的 $R$ 区域上的积分, 可在第一个座标系中的 $S$ 区域中等价表示:
$$\int_R f(\mathbf{y})\operatorname d y_1 \cdots \operatorname d y_n = \int_S f(\mathbf{T}(\mathbf{x})) \left|\mathbf{J}_\mathbf{T}\right|\operatorname d x_1\cdots \operatorname d x_n$$

该式成立是因为, 在 $\mathbf{y}$ 座标系中的积分体积元 $\operatorname d y_1\cdots\operatorname d y_n$, 在 $x$ 座标系中
的等价表示为 $\left|\mathbf{J}_\mathbf{T}\right|\operatorname d x_1\cdots \operatorname d x_n$.

Proof: 体积元 $\operatorname d V$,
的一条边为 $\operatorname d y_1$, 由多元函数的全微分, $\operatorname d y_1$ 边的矢量在 $x$ 座标系表示为
$$\left<\frac{\partial y_1}{\partial x_1}\operatorname d x_1,\dots,\frac{\partial y_1}{\partial x_n}\operatorname d x_n\right>$$, 这些矢量围成的体积就是体积元, 由于任意维度的平行六面体的体积等于它的边矢量组成的
矩阵的行列式值, 故它在两座标系下分别表示为:
$$A =
\begin{bmatrix}
\operatorname d y_1 & &\\
& \ddots &\\
& & \operatorname d y_n
\end{bmatrix}, \operatorname d V = \det(A) = \operatorname d y_1\cdots\operatorname d y_n
$$
$$B =
\begin{bmatrix}
\frac{\partial y_1}{\partial x_1}\operatorname d x_1 & \cdots & \frac{\partial y_n}{\partial x_1} \operatorname d x_1 \\
\vdots & & \vdots \\
\frac{\partial y_1}{\partial x_n}\operatorname d x_n & \cdots & \frac{\partial y_n}{\partial x_n} \operatorname d x_n
\end{bmatrix}
, \operatorname d V = \det(B) = \left|\mathrm{J}\right|\operatorname d x_1\cdots\operatorname d x_n
$$
$\square$

# coordinate systems
## 曲线座标系概述

-   空间中一点的位置是客观的, 与坐标系的选择无关;
而该点的位置在不同坐标系中又有不同的表象. 现在取两坐标系, 一为 Cartesian
直角坐标, 二为任一曲线坐标, 由于空间中各点的客观性, 设某点在直角系中表示为 $(x,
y, z)$, 在曲线系中表示为 $(q_1, q_2, q_3)$, 则所有的 $(x, y, z)$ 与所有的
$(q_1, q_2, q_3)$ 是一一对应的关系. 这就是说, 可以找到一组函数
$$
\begin{align}
q_1 &= q_1(x,y,z) \\
q_2 &= q_2(x,y,z) \\
q_3 &= q_3(x,y,z) \\
\end{align}
$$
与反函数
$$
\begin{align}
x &= x(q_1, q_2, q_3) \\
y &= y(q_1, q_2, q_3) \\
z &= z(q_1, q_2, q_3) \\
\end{align}
$$
-   曲线坐标系的坐标面, 即为某一参量为常数的面. 三个坐标面分别为
$$
\begin{align}
q_1 &= C_1 \\
q_2 &= C_2 \\
q_3 &= C_3 \\
\end{align}
$$

-   曲线座标系的座标线即是两两座标面的交线.

## 正交曲线座标系
即为各座标线上的单位矢相互正交的座标系. 注意 $\hat{e_1}, \hat{e_2}, \hat{e_3}$
符合右手螺旋.

Lamé 系数:
$$
H_i = \sqrt{\left(\frac{dx}{dq_i}\right)^2+\left(\frac{dy}{dq_i}\right)^2+\left(\frac{dz}{dq_i}\right)^2}, i = 1,2,3
$$

## 一些公式

-   正交曲线座标系下算符的一般形式

-   Lamé 系数

## 正交曲线座标系的单位矢求导
$$
\begin{align}
\frac{\partial \hat{e}_1}{\partial q_1}&=-\frac{\hat{e}_2}{H_2}\frac{\partial H_1}{\partial q_2}-\frac{\hat{e}_3}{H_3}\frac{\partial H_1}{\partial
q_3}=-\nabla q_2\frac{\partial H_1}{\partial q_2}-\nabla q_3\frac{\partial H_1}{\partial q_3}\\
\frac{\partial \hat{e}_1}{\partial q_2}&=\frac{\hat{e}_2}{H_1}\frac{\partial H_2}{\partial q_1}=\frac{H_2}{H_1}\nabla q_2\frac{\partial H_2}{\partial
q_1}\\
\frac{\partial \hat{e}_1}{\partial q_3}&=\frac{\hat{e}_3}{H_1}\frac{\partial H_3}{\partial q_1}=\frac{H_3}{H_1}\nabla q_3\frac{\partial H_3}{\partial
q_1}\\
\end{align}
$$

```python

```


# 第五部分：支持向量机

接下来的几个掌机会介绍支持向量机学习算法（SVM），它通常被认为是最好的现成监督学习算法之一（很多人认为它是最好的）。为了详细介绍支持向量机，我们需要先了解间隔（margin）的概念，以及使用大间隙（gap）分割数据的想法。接着我们将介绍优化间隔分类器，进而讨论一下语言歧义（language duality）现象。然后介绍支持向量机的核心——如何在一个极高维特征值向量空间（比如无限维）上高效的应用支持向量机算法。最后我们将介绍序列最小优化算法（SMO: sequential minimal optimization）算法——一个更优化的支持向量机的实现。

## 1. 间隔：直观概念

我们从间隔开始讨论支持向量机算法，本节将给出“间隔”以及算法对预测的“信心”的直观概念，在第三节我们会进一步形式化这些概念。

回顾前面的逻辑回归，假设函数的模型$h_\theta(x)=g\left(\theta^Tx\right)$将给出关于$p(y=1\mid x;\theta)$的概率预测。当且仅当$h_\theta(x)\geq 0.5$即输入满足$\theta^Tx\geq 0$时，模型给出关于输入的预测为$1$。对于一个正训练样本（即$y=1$），$\theta^Tx$越大，则$h_\theta(x)=p(y=1\mid x;w,b)$，也就意味着我们将其预测为$1$的“信心”越高。因此，严谨的说，如果$\theta^Tx\gg 0$，则算法预测其为$1$的信心就会非常大。同样的，对于逻辑回归，如果$\theta^Tx\ll 0$，则算法预测其为$0$的信心就会非常大。对于给定的训练集，我们可以说，如果拟合参数$\theta$能够使得所有$y^{(i)}=1$的样本满足$\theta^Tx^{(i)}\gg0$，使所有$y^{(i)}=0$的样本满足$y^{(i)}=0$，则称这是一个良好的拟合，因为这反映出该拟合对分类结果的“信心”很足。所以，“信心”是一个很好的指标，后面我们将使用函数间隔形式化这个指标。

另一种直观的表达，如下图，其中x代表正训练样本，o代表负训练样本，直线就是判别边界（由$\theta^Tx=0$给出，也叫**分类超平面（separating hyperplane）**）。

图中的三个点A、B、C，点A距离判别边界非常远，如果求关于点A的预测，我们会非常确定其值为$y=1$。相反，点C距离判别边界很近，虽然它在判别边界$y=1$一侧，但是如果判别边界稍有变动，它就有可能被放在$y=0$一侧。因此，我们对点A预测的信心强于点C。而点B则在两者之间，通常来说，如果点距离判别边界越远，模型做出预测的信心就越强。也就是说，如果对于给定训练集，可以找到一条能够准确并可信（即样本距离边界很远）的预测所有训练样本的判别边界，则称这个拟合是良好的。我们在后面将使用几何间隔形式化这个概念。

## 2. 标记法

为了更加简洁的介绍支持向量机，我们需要先引入一种新的标记。考虑使用线性分类器解决“特征为$x$目标为$y$”的二元分类问题。这次，我们使用$y\in\{-1,1\}$来标记两个分类（而不是之前的$y\in\{0,1\}$），再使用参数向量$w,b$代替之前的参数向量$\theta$，于是我们现在将分类器写为：

$$h_{w,b}(x)=g\left(w^Tx+b\right)$$

此处，$g(z)=\begin{cases}1 &z\gt 0\\-1 &z\lt 0\end{cases}$，而$w,b$的记法可以将截距项与特征值的系数分开来记（也不再向特征值向量$x$中添加$x_0=1$分量），即$b$用来代替原来的$\theta_0$，而$w$用来代替原来的$\begin{bmatrix}\theta_1&\cdots&\theta_n\end{bmatrix}^T$。

还需要注意的是，根据函数$g$的定义，分类器将直接给出$-1$或$1$的结果（类似感知算法），省略了估计$y=1$的概率的步骤。

## 3. 函数间隔及几何间隔

本节将形式化函数间隔及几何间隔。对于给定的训练样本$\left(x^{(i)},y^{(i)}\right)$，我们定义关于此训练样本函数间隔的超平面$(w,b)$为：

$$\hat{\gamma}^{(i)}=y^{(i)}\left(w^Tx^{(i)}+b\right)$$

上式可以解释为，当$y^{(i)}=1$时，为了得到一个较大的函数间隔（即为了使预测有较高的的可信度及准确性），我们需要$w^Tx+b$取一个较大的正数（$w^Tx+b\gg 0$）；反之，当$y^{(i)}=-1$时，我摸需要$w^Tx+b$取一个较大的负数（$w^Tx+b\ll 0$）。此外，如果$y^{(i)}\left(w^Tx+b\right)\gt 0$，则说明关于此样本的预测是正确的。因此，较大的函数间隔意味着较高的可信度和准确性。

对于一个在$g\in\{-1,1\}$取值的线性分类器，有一个性质导致其函数间隔不能有效的反映预测的可信度：对于给定的$g$，如果将$(w,b)$替换为$(2w,2b)$，即$g\left(w^Tx+b\right)$变为$g\left(2w^Tx+2b\right)$，我们会发现分类超平面$h_{w,b}(x)$并不会改变，也就是说$h_{w,b}(x)$只关心$w^Tx+b$的正负，而不关心其大小。但是将$(w,b)$变为$(2w,2b)$相当于给函数间隔乘了系数$2$，于是我们发现，如果通过改变$w,b$的取值，我们可以让函数间隔变得很大，然而分类超平面并没有改变，所以单纯的通过这种方式改变函数间隔的大小没有什么实质意义。直觉告诉我们，应该引入一种标准化条件，比如令$\lVert w\rVert_2=1$，即把$(w,b)$变为$\left(\frac{w}{\lVert w\rVert_2},\frac{b}{\lVert w\rVert_2}\right)$，我们在后面会继续讨论这种方法。

对于给定的训练集$S=\left\{\left(x^{(i)},y^{(i)}\right);i=1,\cdots,m\right\}$，关于$S$以$(w,b)$为参数的函数间隔$\hat\gamma$定义为取所以独立的训练样本中最小的那个函数间隔（即取最坏的一个样本的情况）：

$$\hat\gamma=\operatorname*{min}_{i=1,\cdots,m}\hat\gamma^{(i)}$$

接下来我们讨论几何间隔，考虑下图：

直线为$(w,b)$确定的判定边界（即超平面$w^Tx+b=0$），向量$w$正交于分类超平面（在超平面上任取两点$P_1=(x_1,\cdots,x_n), P_2==(x'_1,\cdots,x'_n)$，则两点满足平面方程组$\begin{cases}w^TP_1+b=0\\w^TP_2+b=0\end{cases}$，两式相减得$w^T(P_1-P_2)=0$，即$w^T$正交于该超平面上任意向量，所以$w^T$为超平面法向量）。观察点$A$，设点$A$为训练样本$(x^{(i)},y^{(i)}=1)$，它到判定边界的距离$\gamma^{(i)}$即为线段$AB$的长度。

如何计算$\gamma^{(i)}$？注意到$\frac{w}{\lVert w\rVert}$是标准化后的$w$向量，点$A$为$x^{(i)}$，则点$B$为$x^{(i)}-\gamma^{(i)}\cdot\frac{w}{\lVert w\rVert}$。点$B$在判定边界上，而判定边界上的所有点都满足方程$w^Tx+b=0$，则：

$$w^T\left(x^{(i)}-\gamma^{(i)}\frac{w}{\lVert w\rVert}\right)+b=0$$

解出$\gamma^{(i)}$得：（注：$w^Tw=\lVert w\rVert^2$）

$$\gamma^{(i)}=\frac{w^Tx^{(i)}+b}{\lVert w\rVert}=\left(\frac{w}{\lVert w\rVert}\right)^Tx^{(i)}+\frac{b}{\lVert w\rVert}$$

这就是正训练样本$A$被正确的分在判别边界$y=1$一侧的情形，更一般的，我们定义关于样本$\left(x^{(i)},y^{(i)}\right)$以$(w,b)$为参数的函数间隔为：

$$\gamma^{(i)}=y^{(i)}\left(\left(\frac{w}{\lVert w\rVert}\right)^Tx^{(i)}+\frac{b}{\lVert w\rVert}\right)$$

可以看出，如果$\lVert w\rVert=1$，则函数间隔等于几何间隔（$\hat\gamma^{(i)}=\gamma^{(i)}$），这就是两种间隔的关系（$\hat\gamma^{(i)}=\frac{\gamma^{(i)}}{\Vert w\Vert}$）。与函数间隔一样，如果改变参数$(w,b)$为$(2w,2b)$，则几何间隔不会改变。这个性质在后面会很方便，利用改变参数$(w,b)$的大小不影响间隔，我们可以在拟合参数时引入$w$的限制条件，比如令$\lVert w\rVert=1$，或$\lvert w_1\rvert=5$，或$\lvert w+b\rvert+\lvert b\rvert=2$，诸如这种限制条件都可以通过给参数$(w,b)$乘以一个适当的系数得到。

对于给定的训练集$S=\left\{\left(x^{(i)},y^{(i)}\right);i=1,\cdots,m\right\}$，也有关于$S$以$(w,b)$为参数的几何间隔$\gamma$定义为取所以独立的训练样本中最小的那个几何间隔（即取最坏的一个样本的情况）：

$$\gamma=\operatorname*{min}_{i=1,\cdots,m}\gamma^{(i)}$$

另外，几何间隔实际上就是点到超平面的距离，高中时学过的点$\left(x^{(i)},y^{(i)}\right)$到直线$ax+by+c=0$的距离为：

$$d\left(x^{(i)},y^{(i)}\right)=\frac{\lvert ax^{(i)}+by^{(i)}+c\rvert}{\sqrt{a^2+b^2}}$$

推广到高维就是上面的几何间隔，而函数间隔就是未标准化的几何间隔。

最后，最大间隔分类器（maximum margin classifier，可以被看做是支持向量机的前身），实际上就选择特定的$w,b$使几何间隔最大化：

$$\begin{align}\displaystyle\operatorname*{max}_{w,b}&\quad\gamma\\\mathrm{s.t.}&\quad y^{(i)}\left(\left(\frac{w}{\lVert w\rVert}\right)^Tx^{(i)}+\frac{b}{\lVert w\rVert}\right)\end{align}$$

注：$\mathrm{s.t.}$是“subject to”的缩写，意为“受限于”。


Based on: https://github.com/Harvard-IACS/2018-CS109A/

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
import pandas as pd
%matplotlib inline
```

### Part 1: Neural Networks 101: Starting with a Single Node

The simplest way to describe a neural network is that we have some inputs $x$, which get combined into an auxilliary variable $z$.  The auxilliary variable is passed through the activation function $\sigma\left(z\right)$ and the result is the output.

Here is another image showing each step.

Notice that the inputs are linearly combined according to some weights $w$ and a bias $b$.  This transformation is also sometimes called an **affine** transformation.  The perceptron transforms the weighted inputs according to the rule of the activation function.  For a single perceptron, the output $y$ is just the output from the perceptron.  The linear transformation and activation of the neuron occurs within a single **layer** of the network (shown in the dotted box).

Let's see what the single-layer, single neuron network give us.  We have a couple of choices to make:
1. We must choose some weights and some biases
2. We must choose an activation function

For now, we will manually specify the weights and biases.

We choose a *sigmoid* activation function $$\sigma\left(z\right) = \dfrac{1}{1 + e^{-z}}.$$  What are the limits $\displaystyle\lim_{z\to\infty}\sigma\left(z\right)$ and $\displaystyle\lim_{z\to-\infty}\sigma\left(z\right)$?  Actually, the sigmoid we have here is called the *logistic* function.  Sigmoids are really a family of functions and the logistic function is just one member in that family.

In class exercise : Plot the sigmoid

Plot the sigmoid

```python
# your code here
```

```python
# %load solutions/sigmoid.py
def sigmoid(z):
    return 1/ (1 + np.e**(-z))

```

Generate a list of 500 $x$ points from -5 to 5 and plot both the sigmoid and the tanh (for tanh you may use `np.tanh`)

What do you observe?

```python
# %load solutions/plot_sig.py
x = np.linspace(-5.0, 5.0, 500) # input points
print(x[:5])
### plot
plt.figure(figsize = (9, 3))

plt.subplot(121)
plt.plot(x, [sigmoid(i) for i in x], label = 'sigmoid')
plt.title('sigmoid')

plt.subplot(122)
plt.plot(x, [np.tanh(i) for i in x], label = 'tanh')
plt.title('tanh')

plt.show()
```

##### Comments
* We say that the activation occurs when $\sigma = \dfrac{1}{2}$.  We can show that this corresponds to $x = -\dfrac{b}{w}$.
* The "steepness" of the sigmoid is controlled by $w$.

In class exercise: Approximate a Gaussian function using a node

The task is to approximate (or learn) a function $f\left(x\right)$ given some input $x$.  For demonstration purposes, the function we will try to learn is a Gaussian function
\begin{align}
f\left(x\right) = e^{-x^{2}}
\textrm{}
\end{align}

Even though we represent the input $x$ as a vector on the computer, you should think of it as a single input.

Start by plotting the above function using the $x$ dataset you created earlier

```python
f = np.exp(-x*x) # The real function, x = np.linspace(-5.0, 5.0, 500) input points

plt.plot(x, f, label='gaussian')
plt.legend()
```

Now, let's code the single node as per the image above. Write a function named `affine` that does the linear transformation.

```python
# your code here
def affine(x, w, b):
    """Return affine transformation of x

    INPUTS
    ======
    x: A numpy array of points in x
    w: A float representing the weight of the perceptron
    b: A float representing the bias of the perceptron

    RETURN
    ======
    z: A numpy array of points after the affine transformation
       z = wx + b
    """

    # Code goes here
    z =  w*x + b
    return z
```

```python
# %load solutions/affine.py
```

```python
# %load solutions/sigmoid.py
```

```python
# your code here
w = -4.5
b = 4.

h = sigmoid(affine(x,w,b))
```

```python
h;
```

```python
# %load solutions/perceptron.py
w = -2.5 # weight
b = 1.0 # bias

# Perceptron output
z = affine(x, w, b) # Affine transformation
h = sigmoid(z) # Sigmoid activation

```

And now we plot the activation function and the true function.

```python
fig, ax = plt.subplots(1,1, figsize=(11,7)) # create axes object

SIZE = 16
# Plot
ax.plot(x, f, ls='-.', lw=4, label=r'True function')
ax.plot(x, h, lw=4, label=r'Single Neuron')

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc=1) # Create a legend and make it big enough to read
```

The single perceptron simply turns the output on and off at some point, but that's about it.  We see that the neuron is on until about $x=0$ at which point it abruptly turns off.  It's able to get "close" to the true function.  Otherwise, it has nothing in common with the true function.

What do you think will happen if you change $w$ and $b$?

#### Important Observation
Notice that we wrote the output as `sigmoid(affine(x))`.  This was not a coincidence.  It looks like a composition of functions.  In fact, that is what a neural network is doing.  It's building up an approximation to a function by creating a composition of functions.  For example, a composition of three functions would be written as $$\varphi_{3}\left(\varphi_{2}\left(\varphi_{1}\left(x\right)\right)\right).$$

What happens if we play with the weights and biases?

```python
w = [-5.0, 0.1, 5.0] # Create a list of weights
b = [0.0, -1.0, 1.0] # Create a list of biases

fig, ax = plt.subplots(1,1, figsize=(15,10))
ax.plot(x, f, lw=4, ls='-.', label='True function')
for wi, bi in zip(w, b):
    h = sigmoid(affine(x, wi, bi))
    ax.plot(x, h, lw=4, label=r'$w = {0}$, $b = {1}$'.format(wi,bi))
ax.set_title('Single neuron network', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

We didn't do an exhaustive search of the weights and biases, but it sure looks like this single perceptron is never going to match the actual function.  Again, we shouldn't be suprised about this.  The output layer of the network is simple the logistic function, which can only have so much flexibility.

Let's try to make our network more flexible by using **more nodes**!

## Multiple Perceptrons in a Single Layer
It appears that a single neuron is somewhat limited in what it can accomplish.  What if we expand the number of nodes/neurons in our network?  We have two obvious choices here.  One option is to add depth to the network by putting layers next to each other.  The other option is to stack neurons on top of each other in the same layer.  Now the network has some width, but is still only one layer deep.

The following figure shows a single-layer network with two nodes in one layer.

#### Some observations
1. We still have a single input in this case.  Note that this is not necessary in general.  We're just keeping things simple with a single input for now. If we have more inputs we will have a matrix for $X$.
2. Each node (or neuron) has a weight and bias associated with it.  An affine transformation is performed for each node.
3. Both nodes use the same activation function form $\sigma\left(\cdot\right)$ on their respective inputs.
4. The outputs of the nodes must be combined to give the overall output of the network.  There are a variety of ways of accomplishing this.  In the current example, we just take a linear combination of the node outputs to produce the actual prediction.  Notice that now we have weights and biases at the output too.

Let's see what happens in this case.  First, we just compute the outputs of each neuron.

```python
x = np.linspace(-5.0, 5.0, 500) # input points
f = np.exp(-x*x) # data

w = np.array([3.5, -3.5])
b = np.array([3.5, 3.5])

# Affine transformations
z1 = w[0] * x + b[0]
z2 = w[1] * x + b[1]

# Node outputs
h1 = 1.0 / (1.0 + np.exp(-z1))
h2 = 1.0 / (1.0 + np.exp(-z2))
```

Now let's plot things and see what they look like.

```python
fig, ax = plt.subplots(1,1, figsize=(14,10))

ax.plot(x, f, lw=4, ls = '-.', label='True function')
ax.plot(x, h1, lw=4, label='First neuron')
ax.plot(x, h2, lw=4, label='Second neuron')

# Set title
ax.set_title('Comparison of Neuron Outputs', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

Just as we expected.  Some sigmoids.  Of course, to get the network prediction we must combine these two sigmoid curves somehow.  First we'll just add $h_{1}$ and $h_{2}$ without any weights to see what happens.

#### Note
We are **not** doing classification here.  We are trying to predict an actual function.  The sigmoid activation is convenient when doing classification because you need to go from $0$ to $1$.  However, when learning a function, we don't have as good of a reason to choose a sigmoid.

```python
# Network output
wout = np.ones(2) # Set the output weights to unity to begin
bout = 0 # No bias yet
yout = wout[0] * h1 + wout[1] * h2 + bout
```

And plot.

```python
fig, ax = plt.subplots(1,1, figsize=(14,10))

ax.plot(x, f, ls='-.', lw=4, label=r'True function')
ax.plot(x, yout, lw=4, label=r'$y_{out} = h_{1} + h_{2}$')

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

#### Observations
* The network prediction is still not good. *But*, it is pretty sophisticated.  We just have two neurons, but we get some pretty interesting behavior. We didn't do anything with the output weights.  Those are probably important.  Now let's see what happens when we change the weights on the output.

```python
# Network output
wout = np.array([-1.5, -1.5])
bout = np.array(1.5)

yout = wout[0] * h1 + wout[1] * h2 + bout
```

```python
fig, ax = plt.subplots(1,1, figsize=(14,10))

ax.plot(x, f, lw=4, ls = '-.', label='True function')
ax.plot(x, yout, lw=4, label=r'$y_{out} = w_{1}^{o}h_{1} + w_{2}^{o}h_{2} + b^{o}$')

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc=1) # Create a legend and make it big enough to read
```

Very cool!  The two nodes interact with each other to produce a pretty complicated-looking function.  It still doesn't match the true function, but now we have some hope.  In fact, it's starting to look a little bit like a Gaussian!

We can do better.  There are three obvious options at this point:
1. Change the number of nodes
2. Change the activation functions
3. Change the weights

### Some Mathematical Notation
Before proceeding, let's learn a more succint way of doing the calculations.  If you have a network with a lot of nodes, then you probably don't want to manually determine the output of each node.  It will take forever.  Instead, you can package the computations up using a more compact notation.  We'll illustrate the ideas with the two-node network.

Suppose we have a single input $x$ to a single-layer two-node network.  We can store the weights from each node in a vector $\mathbf{w} \in \mathbb{R}^{2}$.  Similarly, we can store the biases from each node in a vector $\mathbf{b} \in \mathbb{R}^{2}$.  The affine transformation is then written as $$\mathbf{z} = \mathbf{w}x + \mathbf{b}$$ where the usual laws of vector addition and multiplication by a scalar apply.  Of course, we have that $\mathbf{z} \in \mathbb{R}^{2}$ as well.  Next we evaluate the output from each node.  Formally, we write $$\mathbf{h} = \sigma\left(\mathbf{z}\right)$$ where, once again, $\mathbf{h}\in\mathbb{R}^{2}$.  Moreover, it is *understood* that $\sigma$ operates on each individual element of $\mathbf{z}$ separately.  If we denote each component of $\mathbf{z}$ by $z_{j}, \quad j = 1, 2$ then we can write $$h_{j} = \sigma\left(z_{j}\right), \quad j = 1, 2.$$

Lastly, we must do something about the output layer.  Mathematically we write $$y_{out} = \mathbf{w}_{out} \cdot \mathbf{h} + b_{out}$$ where $\mathbf{w}_{out} \in \mathbb{R}^{2}$ and $b_{out} \in \mathbb{R}$.

## Backpropagation

У даному пункті виконайте backpropagation для мережі з
- одним входом х
- одним прихованим шаром, що має 2 нейрони, з активацією сигмоїд
- одним виходом (z3) з лінійною активацією

```python
def affine(x, w, b):
    return w * x + b
def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))
def gaussian(x):
    return np.exp(-x*x)
def sigmoid_derivative(z):
    return sigmoid(z) * (1 - sigmoid(z))
def mse(y,yout):
    return np.mean((y-yout)*(y-yout))
```

```python
x = 1.

##### initialization

w_1 = np.array([1., -1.])
b_1 = np.array([1., 1.])

w_2 = np.array([1., -1.])
b_2 = np.array([1.])
```

```python
### do the forward pass of the network
### out = z3
z_1_2, z_3 = 0,0
def forward():
    global z_1_2, z_3
    z_1_2 = w_1 * x + b_1 # working with vector [z1, z2]
    h_1_2 = sigmoid(z_1_2)

    z_3 =  h_1_2 @ w_2 + b_2
```

```python
z_3
```

    0

```python
actual = gaussian(x)
y = actual
actual
```

    0.36787944117144233

Is the output of the network close to the actual value?

```python
def backprop(lr=1):
    global w_1, w_2, b_1, b_2
    #### DERIVATIVES CALCULATION
    dL_dw1 = 2 * (z_3 - y) * w_2 * sigmoid_derivative(z_1_2) * x
    dL_dw2 = 2 * (z_3 - y) * sigmoid(z_1_2)

    dL_db1 = 2 * (z_3 - y) * w_2 * sigmoid_derivative(z_1_2)
    dL_db2 = 2 * (z_3 - y)

    ##### WEIGHTS UPDATE

    w_1 -= lr * dL_dw1
    w_2 -= lr * dL_dw2
    b_1 -= lr * dL_db1
    b_2 -= lr * dL_db2
```

```python
backprop()
```

```python
### do again the forward pass of the network
forward()
z_3
```

    array([2.72790137])

Did output become closer to the actual value of the function?
Nope

Now let's write it into a loop

```python
N = 50000
lr = 0.01
```

```python
X = np.linspace(-5.0, 5.0, 500)

##### initialization

w_1 = np.array([1., -1.])
b_1 = np.array([1., 1.])

w_2 = np.array([1., -1.])
b_2 = np.array([1.])

for i in range(N):
    x = np.random.choice(X)
    y = gaussian(x)

    forward()
    backprop(lr=lr)
```

Now let's plot things and see what they look like.

```python
z1 = w_1[0] * X + b_1[0]
z2 = w_1[1] * X + b_1[1]

# Node outputs
h1 = 1.0 / (1.0 + np.exp(-z1))
h2 = 1.0 / (1.0 + np.exp(-z2))

z3 = w_2[0] * h1 + w_2[1] * h2 + b_2[0]
out = z3

SIZE = 15

fig, ax = plt.subplots(1,1, figsize=(14,10))

ax.plot(X, gaussian(X), lw=4, ls = '-.', label='True function')
ax.plot(X, out, lw=4, label='Perceptron')
#ax.plot(x, h2, lw=4, label='Second neuron')

# Set title
ax.set_title('Comparison of Neuron Outputs', fontsize=SIZE)

# Create labels (very important!)
ax.set_xlabel('$x$', fontsize=SIZE) # Notice we make the labels big enough to read
ax.set_ylabel('$y$', fontsize=SIZE)

ax.tick_params(labelsize=SIZE) # Make the tick labels big enough to read

ax.legend(fontsize=SIZE, loc='best') # Create a legend and make it big enough to read
```

### Classification problem

```python
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
from matplotlib import pyplot
from sklearn.metrics import log_loss, accuracy_score
from pandas import DataFrame
```

```python
# generate 2d classification dataset
X, y = make_blobs(n_samples=500, centers=2, n_features=2, random_state=0)
# scatter plot, dots colored by class value
df = DataFrame(dict(x=X[:,0], y=X[:,1], label=y))
colors = {0:'red', 1:'blue'}
fig, ax = pyplot.subplots()
grouped = df.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
pyplot.show()
```

```python
X.shape
```

    (500, 2)

```python
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2, random_state=42)
```

```python
x = X_test
```

```python
##### initialization

w_1 = np.array([1,1])
b_1 = np.array([1])
```

```python
z1 = w_1.dot(x.T) + b_1

# Node outputs
h1 = sigmoid(z1)
```

```python
pred = [int(score > 0.5) for score in h1]
sum(pred == y_test) / len(y_test)
```

    0.45

```python
### Loop
```

```python
N = 10000
```

```python
batch_size = 10

##### initialization

w_1 = np.array([1.,1.])
b_1 = np.array([1.])
lr=0.01

for i in range(N):
    perm = np.random.permutation(len(X_train))
    X_train = X_train[perm]
    y_train = y_train[perm]

    for batch_id in range(int(len(X_train)/batch_size)):
        x = X_train[batch_id*batch_size:(batch_id+1)*batch_size]
        y = y_train[batch_id*batch_size:(batch_id+1)*batch_size]

        m = x.shape[0]
        #### FORWARD PASS
        z_1 = x @ w_1 + b_1
        h_1 = sigmoid(z_1)

        #### DERIVATIVES CALCULATION
        dL_dw1 = (h_1 - y) @ x
        dL_db1 = sum(h_1 - y)

        ##### WEIGHTS UPDATE
        w_1 -= lr * dL_dw1
        b_1 -= lr * dL_db1

```

```python
### check the accuracy on the test set
y_pred = sigmoid(X_test @ w_1 + b_1)
accuracy_score(y_test, y_pred > 0.5)

```

    0.95

```python

```