# Pre-Calculus Notebook

```python
# Begin with some useful imports
import numpy as np
import pylab
from fractions import Fraction
from IPython.display import display, Math
from sympy import init_printing, N, pi, sqrt, symbols, Symbol
from sympy.abc import x, y, theta, phi
init_printing(use_latex='mathjax')  # do this to allow sympy to print mathy stuff
%matplotlib inline
```

## Trig Functions

This notebook will introduce basic trigonometric functions.

Given an angle, θ, there are two units used to indicate the measure of the angle: degrees and radians.

### First make a simple function to convert from degrees to radians

```python
def deg_to_rad(angle, as_expr=False):
    """ Convert degrees to radians
        param: theta: degrees as a float or numpy array of values that can be cast to a float
        param: as_text: True/False output as text with multiple of 'π' (Default = False)

        The formula for this is:
        2πθ/360°

        Note: it's usually bad form to allow a function argument to change an output type.  Oh, well.
    """
    radians = (angle * 2 * np.pi) / 360.0

    if as_expr:
        # note: this requires sympy
        radians = 2*pi*angle/360

    return radians
```

```python
# Test our function
# Note: normally you would print results, but with sympy expressions,
#       the `display` function of jupyter notebooks is used to print using MathJax.

display(deg_to_rad(210, as_expr=True))
display(deg_to_rad(360, as_expr=True))

```

$$\frac{7 \pi}{6}$$

$$2 \pi$$

### Now, let's make a list of angles around a unit circle and convert them to radians

```python
base_angles = [0, 30, 45, 60]
unit_circle_angles = []
for i in range(4):
    unit_circle_angles.extend(np.array(base_angles)+(i*90))
unit_circle_angles.append(360)
```

```python
# make a list of the angles converted to radians (list items are a sympy expression)
unit_circle_angles_rad = [deg_to_rad(x, False) for x in unit_circle_angles]
unit_circle_angles_rad_expr = [deg_to_rad(x, True) for x in unit_circle_angles]
```

```python
unit_circle_angles_rad
```

$$\left [ 0.0, \quad 0.523598775598, \quad 0.785398163397, \quad 1.0471975512, \quad 1.57079632679, \quad 2.09439510239, \quad 2.35619449019, \quad 2.61799387799, \quad 3.14159265359, \quad 3.66519142919, \quad 3.92699081699, \quad 4.18879020479, \quad 4.71238898038, \quad 5.23598775598, \quad 5.49778714378, \quad 5.75958653158, \quad 6.28318530718\right ]$$

```python
unit_circle_angles_rad_expr
```

$$\left [ 0, \quad \frac{\pi}{6}, \quad \frac{\pi}{4}, \quad \frac{\pi}{3}, \quad \frac{\pi}{2}, \quad \frac{2 \pi}{3}, \quad \frac{3 \pi}{4}, \quad \frac{5 \pi}{6}, \quad \pi, \quad \frac{7 \pi}{6}, \quad \frac{5 \pi}{4}, \quad \frac{4 \pi}{3}, \quad \frac{3 \pi}{2}, \quad \frac{5 \pi}{3}, \quad \frac{7 \pi}{4}, \quad \frac{11 \pi}{6}, \quad 2 \pi\right ]$$

```python
# show the list of angles in degrees
unit_circle_angles
```

$$\left [ 0, \quad 30, \quad 45, \quad 60, \quad 90, \quad 120, \quad 135, \quad 150, \quad 180, \quad 210, \quad 225, \quad 240, \quad 270, \quad 300, \quad 315, \quad 330, \quad 360\right ]$$

### So, what are radians

Radians are a unit of angle measure (similar to degrees).  Radians correspond to the chord length that an angle sweeps out on the unit circle.  A complete circle with a radius of 1 (a unit circle) has a circumference of $2\pi$ (from the formula for circumference: $C = 2\pi r$ )

Let's import some things to help us create a plot to see this in action.

```python
from bokeh.plotting import figure, show, output_notebook
output_notebook()
```

    Loading BokehJS ...

If you were to map out our angle values on the circle drawn on a cartesian coordinate grid, it might look like this:

```python
p = figure(plot_width=600, plot_height=600, match_aspect=True)

# draw angle lines
x = np.cos(unit_circle_angles_rad)
y = np.sin(unit_circle_angles_rad)
origin = np.zeros(x.shape)

for i in range(len(unit_circle_angles_rad)):
    #print (x[i], y[i])
    p.line((0, x[i]), (0, y[i]))

# draw unit circle
p.circle(0,0, radius=1.0, color="white", line_width=0.5, line_color='black',
         alpha=0.0, line_alpha=1.0)

```

GlyphRenderer(

id = 'fc41fd69-f9b1-4846-b0a5-77bea065dad0', …)

data_source = ColumnDataSource(id='d578de05-1c57-4568-bf26-45bbd7e91071', ...),

glyph = Circle(id='c04870c5-f214-4e3c-8b2a-9ca556962ac3', ...),

hover_glyph = None,

js_event_callbacks = {},

js_property_callbacks = {},

level = 'glyph',

muted = False,

muted_glyph = None,

name = None,

nonselection_glyph = Circle(id='e17f1a11-aaf3-46df-8b18-671e96ef5549', ...),

selection_glyph = None,

subscribed_events = [],

tags = [],

view = CDSView(id='86474633-4aeb-4e71-bd74-6de289e9ebd8', ...),

visible = True,

x_range_name = 'default',

y_range_name = 'default')

```python
show(p)
```

```python
import bokeh
```

```python
bokeh.__version__
```

    u'0.12.7'

```python
    from bokeh.plotting import figure, output_file, show

    output_file("circle_v1.html")

    p = figure(plot_width=400, plot_height=400, match_aspect=True, tools='save')
    p.circle(0, 0, radius=1.0, fill_alpha=0)
    p.line((0, 1), (0, 0))
    p.line((0, -1), (0, 0))
    p.line((0, 0), (0, 1))
    p.line((0, 0), (0, -1))
    #p.rect(0, 0, 2, 2, fill_alpha=0)

    show(p)

```

```python
import sys, IPython, bokeh
print "Python: ", sys.version
print "IPython: ", IPython.__version__
print "Bokeh: ", bokeh.__version__
```

    Python:  2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08)
    [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
    IPython:  5.3.0
    Bokeh:  0.12.10

```python
%reload_ext version_information
```

```python
%version_information
```

Software	Version

Python	2.7.13 64bit [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]

IPython	5.3.0

Bokeh	0.12.10

OS	Darwin 17.2.0 x86_64 i386 64bit

Wed Nov 15 13:14:25 2017 CST

```python
from bokeh.plotting import figure, output_file, show
from bokeh.layouts import layout

p1 = figure(match_aspect=True, title="Circle touches all 4 sides of square")
#p1.rect(0, 0, 300, 300, line_color='black')
p1.circle(x=0, y=0, radius=10, line_color='black', fill_color='grey',
          radius_units='data')

def draw_test_figure(aspect_scale=1, width=300, height=300):
    p = figure(
        plot_width=width,
        plot_height=height,
        match_aspect=True,
        aspect_scale=aspect_scale,
        title="Aspect scale = {0}".format(aspect_scale),
        toolbar_location=None)
    p.circle([-1, +1, +1, -1], [-1, -1, +1, +1])
    return p

aspect_scales = [0.25, 0.5, 1, 2, 4]
p2s = [draw_test_figure(aspect_scale=i) for i in aspect_scales]

sizes = [(100, 400), (200, 400), (400, 200), (400, 100)]
p3s = [draw_test_figure(width=a, height=b) for (a, b) in sizes]

layout = layout(children=[[p1], p2s, p3s])

output_file("aspect.html")
show(layout)
```

```python

```


# Problem 4.15 Additive Noise Models (ANMs)

## a) ANM I

Given the SCM
\begin{align}
X:=&N_X\\
Y:=&2X+N_Y
\end{align}
with $N_X\sim U(1,3)$ and $N_Y\sim U(-0.5,0.5)$.

### Support of $P_{X,Y}$

```python
%matplotlib inline
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
```

```python
n = 3000
x = np.random.uniform(1, 3, n)
y = 2 * x + np.random.uniform(-.5, .5, n)
```

```python
def plot_support(x, y):
    fig = plt.figure()
    ax=fig.add_axes([0, 0, 2, 2])
    ax.scatter(x, y, color='b')
    ax.set_xlabel('$X$')
    ax.set_ylabel('$Y$')
    ax.set_title('$P_{X,Y}$')
    plt.show()
```

```python
plot_support(x, y)
```

There is no ANM from $Y$ to $X$ because the noise shape depends on $Y$, when the plot _is flipped by 90 degrees_.

## b) ANM II

ANM with the same noise as ANM I, but the causal assignments
\begin{align}
X:=&N_X\\
Y:=&X^2+N_Y
\end{align}

```python
x = np.random.uniform(1, 3, n)
y = x**3 + np.random.uniform(-.5, .5, n)
```

```python
plot_support(x, y)
```

Visually, the thickness seems to vary because me measure orthogonal to the tangent. However, as seen from the $X$ axis, the thickness remains the same.

Again, the height of the _noise bar_ in $Y$ direction is always the same regardless of $X$. When flipping the plot, i.e. changing the causal direction, the noise thickness depends on $Y$.


# Laboratory 09: Matrices and Rotations (Crystallography)

## Background

The purpose of this laboratory is to understand and apply the concept of an agebraic group to produce one of the plane group patterns below.  In this context, algebraic groups are mathematical objects that represent a collection of geometric transformations and are used in the study of crystallography.  For example, when you say that is crystal is "face centered cubic" what you are really saying is that an atomic motif, when transformed by all the algebraic elements of a particular group, will be identical to the face centered cubic structure.  The group is the mathematical machinary of crystal structures.  The group itself is an abstract idea that is embodied in a crystal structure, a geometric design, a geometric series, a set of matrices, or a set of any kind where the rules that govern the set meet the criteria of a group.  The idea of a group exists beyond the example I present here.

## What Skills Will I Learn?

Using the context of two-dimensional plane groups, in this assignment you will practice the following skills:

* Identifying a symmetry operation (rotation, translation, mirror) and determining the associated matrix representation.
* Applying a symmetry operation to a position vector to transform the vector (or collection of vectors).
* Seeing the inner workings of a class object and its attributes for the purpose of simplifying a computing task.
* Using lists to collect and organize objects (such as transformation matrices)

## What Steps Should I Take?

1.  Review the idea of "symmetry operations" and familarize yourself with mirror, translational, and rotational symmetry operations.
1.  Review the idea of matrix transformations and how they can be used to represent symmetry operations and how those symmetry operations can be represented as a matrix/vector dot product.
1.  Read about the metric tensor below and practice a few calculations of lengths and angles in unit cell coordinates.
1.  Read and practice writing down 2D, 3D and 3D augmented transformation matrices and learn how Euler angles are defined.
1.  For each case above, use a position vector pointing in a direction you select and apply a transformation using each matrix in turn.  Does the resultant vector match your anticipated transformation?  (e.g. If you take the position vector pointing at the [0,0,1] position in a cubic lattice and apply a 90 degree rotation about the x-axis, where should the resultant vector be pointing?  Repeat this for mirror planes and translations.
1. Review the concept of a "class" in object oriented programming and familarize yourself with the `Point` and `Triangle` classes I have defined below.  Practice creating these objects and accessing their attributes.
1. Review the code that accesses the Python Imaging Library (PIL) and look at the `polygon` function call and the `Triangle` points method.  You can call the points method from within the polygon function to simplify the drawing of triangles.
1.  Using the `Point` and `Triangle` classes and your knowledge of symmetry transformations, reproduce one of the plane groups from the figure below.  My suggestion is to start with a single triangle and apply a set of operations (transformation matrices) and collect new triangles into a list.  Sketch a flowchart for how you might solve this problem.
1.  Review the very last block of code in this notebook and modify it to complete the assignment.  You will need to use combinations of translations, reflections and rotations to accomplish this.
1.  Prepare a new notebook (not just modifications to this one) that describes your approach to reproducing one or more of the plane groups.

You may discover that a unique set of matrices will reproduce the whole pattern.  This small set of matrices are an algebraic structure known as a **group**.  So - the "plane group" is the group of symmetry operations that reproduces the structure in the plane starting from a single motif.  The plane group representations are reproduced below for reference; this image comes from Hammond's book on crystallography, cited above.

## A Sucessful Jupyter Notebook Will

* Present a description of the essential elements of plane group symmetry, matrix algebra, and group theory to understand how these are related;
* Identify the audience for which the work is intended;
* Run the code necessary to draw one of the plane groups;
* Provide a narrative and equations to explain why your approach is relevant to solving the problem;
* Provide references and citations to any others' work you use to complete the assignment;
* Be checked into your GitHub repository by the due date (one week from assignment).

A high quality communication provides an organized, logically progressing, blend of narrative, equations, and code that teaches the reader a particular topic or idea.  You will be assessed on:
* The functionality of the code (i.e. it should perform the task assigned).
* The narrative you present.  I should be able to read and learn from it.  Choose your audience wisely.
* The supporting equations and figures you choose to include.

If your notebook is just computer code your assignment will be marked incomplete.

## Reading and Reference

* M. De Graef and M. McHenry, Structure of Materials, Cambridge University Press, 2nd ed.
* C. Hammond, The Basics of Crystallography and Diffraction, Oxford Science Publications, 4th ed.

## The Plane Groups

### Introduction
----

Operations using vector-like data structures are an essential component of numerical computing, mathematics, science and engineering.  In the field of crystallography the use of vectors and rotations in real and reciprocal space helps to simplify the description of and quantitative operations on crystal structures.  Matrix operations are used in the solution of partial differential equations.  The vector algebra and rotations are most easily performed using matrix tools and representations.  The concepts and their mathematical properties will be reviewed and demonstrated using symbolic algebra and numerical methods.  A review of matrix algebra will be helpful.

### Rotations
----

A vector can be transformed by translations, rotations, and stretching/shrinking.  A matrix multiplication operation can be used to define each individual operation.  We can use matrix multiplication to perform combinations of these operations and then this composite operator can be applied to a vector.  In general these types of transformations are called Affine Transformations.  A rotation in two dimensions is given by:

\begin{equation*}
\left[\begin{matrix}\cos{\left (\theta \right )} & - \sin{\left (\theta \right )}\\\sin{\left (\theta \right )} & \cos{\left (\theta \right )}\end{matrix}\right]
\end{equation*}

We can rotate a vector $x\mathbf{i} + y\mathbf{j}$ to the position $x'\mathbf{i} + y'\mathbf{j}$ using the following matrix operation:

\begin{equation*}
\left( \begin{matrix} x' \\ y' \end{matrix} \right) = \left[\begin{matrix}\cos{\left (\theta \right )} & - \sin{\left (\theta \right )}\\\sin{\left (\theta \right )} & \cos{\left (\theta \right )}\end{matrix}\right] \cdot \left( \begin{matrix} x \\ y \end{matrix} \right)
\end{equation*}

```python
%matplotlib inline

import numpy as np
import sympy as sp
sp.init_session(quiet=True)
```

```python
?sp.rot_axis3
```

```python
x = sp.symbols('\\theta')
sp.rot_axis3(x)
```

We can look up definitions, but we can also do some simple tests to see which way things rotate.  Let us take a vector pointing in the $\mathbf{x}$ direction and rotate it about $\mathbf{z}$ by 90 degrees and see what happens:

```python
xUnit = sp.Matrix([1,0,0])
zRotation = sp.rot_axis3(sp.rad(90))
```

```python
xUnit
```

```python
# Each column can be viewed as where the unit vectors are moved to in the new space.
zRotation
```

```python
zRotation*xUnit
```

```python
# This should not work.
xUnit*zRotation
```

```python
xUnit.T*zRotation
```

What can we learn from this result?  It is now known that:

* The convention for positive angles is a *counterclockwise* rotation.
* The rotation axis function in `sympy` as defined rotates *clockwise*
* There are conventions about active and passive rotations.
* Don't assume module functions will do what you want - always check.

Rather than rely on module functions, we can define our own rotation function.  Using a function called "isclose" or Boolean indexing it is possible to clean up the arrays and remove small numerical values that should be zeros.

```python
def rotation2D(theta):
    return np.array([[np.cos(theta), np.sin(theta)],
                     [-np.sin(theta),  np.cos(theta)]])
```

```python
testArray = rotation2D(0)
testArray
```

```python
np.dot(np.array([1,0]),testArray)
```

### DIY:  Computing Rotation Matrices
----

Compute the following rotation matrices:

* A rotation of 0$^\circ$ about the origin.
* A rotation of 45$^\circ$ about the origin.
* A rotation of 60$^\circ$ about the origin.
* A rotation of 90$^\circ$ about the origin.
* A rotation of 180$^\circ$ about the origin.
* A rotation of -270$^\circ$ about the origin.

```python
# Put your code here.
```

### Cleaning up the Small Values
---

Sometimes Numpy returns very small numbers instead of zeros.

One strategy is to remove small numbers less than some tolerance and set them equal to zero.  Algorithms like these where you compare your data to a tolerance and then operate on the entries that meet certain criteria are not uncommon in numerical methods.  This is the tradeoff between symbolic computation and numeric computation.

```python
testArray[np.abs(testArray) < 1e-5] = 0

testArray
```

The key is in the Boolean comparision using the `<` symbol.  The expression returns a `numpy` array of `dtype=bool`.  Let me say here that it is good to check the results of expressions if you are unsure.

```python
np.abs(testArray) < 1e-5
```

We can write a function to do this that is a bit more robust.  Modifications are done in-place (by reference) so we just return the array passed to the function after some manipulation that we do by Boolean indexing.

```python
def zeroArray(testArray):
    testArray[np.isclose(testArray, np.zeros_like(testArray))] = 0.0
    return testArray
```

```python
modifiedArray = rotation2D(np.pi/2)
modifiedArray
```

```python
modifiedArray = zeroArray(rotation2D(np.pi/2))
modifiedArray
```

### Rotations (Continued)
----

Using the new `rotation2D` function and `zeroArray` function we can now write:

```python
zeroArray(np.dot(np.array([1,0]),rotation2D(np.pi/2)))
```

A collection of functions for performing transformations is available at http://www.lfd.uci.edu/~gohlke/.  This can be imported and the functions used in your code.  The fastest way to explore what is available is to import the file and then use autocomplete and docstring viewing functions from the Jupyter notebook.

```python
import transformations as tfm
```

```python
tfm.rotation_matrix(np.pi/2, [0,1,0])
```

```python
zeroArray(tfm.rotation_matrix(np.pi/2, [0,1,0]))
```

### Symmetry Operations and Translations in Crystals
---

A generalized affine transformation in two dimensions can be thought of as an augmented matrix as:

$$\begin{bmatrix}
r_1 & r_2 & t_x\\
r_3 & r_4 & t_y\\
0 & 0 & 1\\
\end{bmatrix}$$

so you could imagine the following:

$$\begin{bmatrix} x'\\ y'\\ 1\\ \end{bmatrix} =
\begin{bmatrix} 1 & 0 & t_x\\ 0 & 1 & t_y\\ 0 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix}x\\ y\\ 1\\ \end{bmatrix} $$

expanding to:

$$x' = x + t_x $$

and

$$y' = y + t_y $$

We can explicitly write the rotation components as listed earlier:

$$\begin{bmatrix} x'\\ y'\\ 1\\ \end{bmatrix} =
\begin{bmatrix} \cos{\theta} & \sin{\theta} & t_x\\ -\sin{\theta} & \cos{\theta} & t_y\\ 0 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix}x\\ y\\ 1\\ \end{bmatrix} $$

where the $r_i$ represent the rotation matrix components and the $t_i$ represent the translations components.  This can be thought of as a shearing operation in 3D.  The Wikipedia article on this [topic](https://en.wikipedia.org/wiki/Affine_transformation) expands this idea a bit more.

In this format we can use a point description that looks like $(x, y, t)$ and matrix algebra to generate our transformed points.  Using SymPy:

```python
alpha, t_x, t_y, x, y = sp.symbols('alpha t_x t_y x y')
```

```python
sa = sp.sin(alpha)
ca = sp.cos(alpha)
M = sp.Matrix([[ca, sa, t_x], [-sa, ca, t_y], [0, 0, 1]])
V = sp.Matrix([x, y, 1])

M*V
```

### Helper Classes, Drawing and Plane Groups
----

Let us explore a bit of how we can draw an image - and then we have everything we need to start building the plane group representations.  To build pictoral representations of plane groups, two classes have been created to simplify the organization of the motif.

Below are the `Point` and `Triangle` classes for your use.  A `Point` has storage for an $(x,y)$ position.  Rather than building arrays and referencing specific positions within the array a more natural referencing of points is possible.  If we define a point `p1 = Point(10,20)` we can access the points by `p1.x` and `p1.y`.  The code is more easily read and debugged with this syntax.

Building on the `Point` class we define a `Triangle` class.  The `Triangle` permits access of each point and defines an `affine()` method that will take a `Numpy` array that represents a transformation matrix.  This method returns a new instance of a `Triangle` and preserves the original points.  Writing the code this way avoids explicit handling of the transformation matrices.

These two classes and methods are demonstrated in the second and third code blocks.  Building on this demonstration you will be able to complete the homework.

```python
# Class definitions

from math import sqrt
import numpy as np

class Point:
    """
    A Point object to simplify storage of (x,y) positions.
    p1.x, p1.y, etc...
    """
    def __init__(self,x_init,y_init):
        self.x = x_init
        self.y = y_init

    def shift(self, x, y):
        self.x += x
        self.y += y

    def __repr__(self):
        return "".join(["Point(", str(self.x), ",", str(self.y), ")"])

class Triangle:
    """
    A Triangle class constructed from points.  Helps organize information on
    triangles.  Has a points() method for returning points in a form that
    can be used with polygon drawing from Python Image Library (PIL) and a
    method, affine(), that applies a matrix transformation to the points.
    """
    def __init__(self, p1_init, p2_init, p3_init):
        self.p1 = p1_init
        self.p2 = p2_init
        self.p3 = p3_init

    def points(self):
        x1, y1 = self.p1.x, self.p1.y
        x2, y2 = self.p2.x, self.p2.y
        x3, y3 = self.p3.x, self.p3.y

        return [(x1,y1),(x2,y2),(x3,y3)]

    def affine(self, affineMatrix):
        """
        Applies an affine transformation to a triangle and changes the
        points of the triangle.  This code returns a new triangle.  Uses
        Points to simplify augmenting the matrix and dot products.
        """
        x1, y1 = self.p1.x, self.p1.y
        x2, y2 = self.p2.x, self.p2.y
        x3, y3 = self.p3.x, self.p3.y

        p1Vector = np.array([[x1, y1, 1]])
        p2Vector = np.array([[x2, y2 , 1]])
        p3Vector = np.array([[x3, y3, 1]])

        p1New = np.dot(affineMatrix, p1Vector.T)
        p2New = np.dot(affineMatrix, p2Vector.T)
        p3New = np.dot(affineMatrix, p3Vector.T)

        # This line needs to be cleaned up.
        newTriangle = Triangle(Point(p1New[0,0],p1New[1,0]),Point(p2New[0,0],p2New[1,0]),Point(p3New[0,0],p3New[1,0]))

        return newTriangle
```

### Using the Python Imaging Library
----

In order that our transformations can be visualized, we will use the Python Imaging Library and the `polygon()` method.  The `polygon()` method takes a list of points in the format returned by our `Triangle` class method `points()`.  The code below is a very simple starting point for the student to begin building a representation of the plane groups.

```python
# Class demonstrations

%matplotlib inline

import numpy as np
import math
from PIL import Image, ImageDraw

image = Image.new('RGB', (500, 500), 'white')
draw = ImageDraw.Draw(image)

t1 = Triangle(Point(10,10),Point(40,10),Point(10,50))
t2 = Triangle(Point(10,10),Point(40,10),Point(10,50))
translationMatrix = np.array([[1,0,50],[0,1,0],[0,0,1]])
reflectionMatrix = np.array([[-1,0,0],[0,1,0],[0,0,1]])

t2 = t1.affine(reflectionMatrix*translationMatrix)

print(t1.points(), t2.points())

draw.polygon(t1.points(), outline='black', fill='red')
draw.polygon(t2.points(), outline='black', fill='green')

image
```

```python
# A slightly more advanced demonstration

%matplotlib inline

import numpy as np
import math
from PIL import Image, ImageDraw

image = Image.new('RGB', (500, 500), 'white')
draw = ImageDraw.Draw(image)

t1 = Triangle(Point(70,10),Point(100,10),Point(70,50))

triangleList = [t1]

translationMatrix = np.array([[1,0,10],[0,1,0],[0,0,1]])
reflectionMatrixY = np.array([[-1,0,0],[0,1,0],[0,0,1]])
reflectionMatrixX = np.array([[1,0,0],[0,-1,0],[0,0,1]])

r90 = np.array([[0,-1,0],[1,0,0],[0,0,1]])

triangleList.append(t1.affine(r90))
triangleList.append((t1.affine(r90)).affine(r90))
triangleList.append((t1.affine(r90)).affine(r90).affine(r90))

tempList = [triangle.affine(reflectionMatrixX) for triangle in triangleList]
triangleList.extend(tempList)

# Using an affine transformation to center the triangles in the drawing
# as canvas coordinates are (0,0) at the top left.
centerMatrix = np.array([[1,0,250],[0,1,250],[0,0,1]])
drawList = [triangle.affine(centerMatrix) for triangle in triangleList]
for triangle in drawList:
    draw.polygon(triangle.points(), outline='black', fill='red')

image
```

# Advanced Topics:  The Metric Tensor

Students who learn crystallography for the first time are introduced to the topic through study of cubic crystal structures.  This permits an appeal to our intuition about orthonormal (Euclidian) coordinate systems.  This approach misses out on a more general method for teaching the topic where the d-spacing and the angle between directions can be worked out for any general crystal system.  The method for describing distances in a general reference frame involves the **metric tensor**.  The metric tensor defines how distances are measured in every direction and its components are the dot product of every combination of basis vector in the system of interest.  We use the standard lattice parameter designations:

$$
[a, b, c, \alpha, \beta, \gamma]
$$

where $\gamma$ is the angle between $\mathbf{a}$ and $\mathbf{b}$, etc.  This general system has basis vectors:

$$
\mathbf{a},\mathbf{b}, \mathbf{c}
$$

The standard geometric interpretation of an inner product of vectors is:

$$
\mathbf{a} \cdot \mathbf{b} = |a| \; |b| \; \cos{\gamma}
$$

so that the metric tensor is:

$$
g_{ij} =
\begin{bmatrix}
\mathbf{a} \cdot \mathbf{a} &
\mathbf{a} \cdot \mathbf{b} &
\mathbf{a} \cdot \mathbf{c} \\
\mathbf{b} \cdot \mathbf{a} &
\mathbf{b} \cdot \mathbf{b} &
\mathbf{b} \cdot \mathbf{c} \\
\mathbf{c} \cdot \mathbf{a} &
\mathbf{c} \cdot \mathbf{b} &
\mathbf{c} \cdot \mathbf{c}
\end{bmatrix}
$$

Committing this to memory is simple and it has an intuitive meaning in that each entry measures a different projection of a vector component onto an axis in a general crystal system within lattice coordinates.  It is possible to write a small function in SymPy to illustrate this.

```python
def metricTensor(a=1, b=1, c=1, alpha=sp.pi/2, beta=sp.pi/2, gamma=sp.pi/2):
    return sp.Matrix([[a*a, a*b*sp.cos(gamma), a*c*sp.cos(beta)], \
                     [a*b*sp.cos(gamma), b*b, b*c*sp.cos(alpha)], \
                     [a*c*sp.cos(beta), b*c*sp.cos(alpha), c*c]])
```

```python
sp.var('a b c alpha beta gamma u v w h k l')
```

```python
M = metricTensor(a=a,
                 b=b,
                 c=c,
                 alpha=alpha,
                 beta=beta,
                 gamma=gamma
                )
M
```

### Two Common Computations
----

Using the Einstein summation convention, the square of the distance between two points located at the end of vectors $\mathbf{p}$ and $\mathbf{q}$ is given by:

$$
D^2 = (\mathbf{q} - \mathbf{p})_i \; g_{ij} \; (\mathbf{q} - \mathbf{p})_j
$$

The dot product between two vectors is given by:

$$
\mathbf{p} \cdot \mathbf{q} = p_i \; g_{ij} \; q_j
$$

Note that the vectors $\mathbf{p}$ and $\mathbf{q}$ are in lattice coordinates.

### DIY:  Angle Between Two Vectors
---

Find the expression for and compute the angle between two vectors in a general coordinate system.  Use the function defined above or write a new one.  You are encouraged to use SymPy or Numpy in your calculations.  Refer to the earlier lectures that cover these topics for the implementation and technical details.

```python
# Put your code or markdown here.
```

### DIY:  Compute the angle between the $(123)$ plane and the $(112)$ plane in a trigonal crystal system
---

The trigonal crystal system has the least symmetry.  Refer to standard texts for the pattern of lattice parameters.

```python
vectorOne = np.array([1,2,3])
vectorTwo = np.array([1,1,2])

# Put your code here.
```

# Advanced Topics: Euler Angles

This discussion is derived primarily from Arfken, Chapter 3.3.  The figure below is from Arfken:

There are three successive rotations used in the Euler angle formalism - the product of these three rotations is the single operation that transforms one set of coordinates $(x,y,z)$ into another, $(x',y',z')$.  The order is important and is the difference between active and passive rotations.

In steps as shown in Figure 3.7 from Arfken:

1. The first rotation is about $x_3$.  In this case the $x'_3$ and $x_3$ axes coincide.  The angle $\alpha$ is taken to be positive (counterclockwise).  Our new coordinate system is $(x'_1,x'_2,x'_3)$.
1. The coordinates are now rotated through an angle $\beta$ around the $x'_2$ axis.  Our new coordinate system is now $(x''_1,x''_2,x''_3)$.
1. The final rotation is through the angle $\gamma$ about the $x'''_3$ axis.  Our coordinate system is now the $(x'''_1,x'''_2,x'''_3)$.  In the case pictured above the $x''_3$ and $x'''_3$ axes coincide.

For example:

```python
alpha, beta, gamma = sp.symbols('alpha beta gamma')

def rZ(angle):
    sa = sp.sin(angle)
    ca = sp.cos(angle)
    M = sp.Matrix([[ca, sa, 0],
                  [-sa, ca, 0],
                  [0, 0, 1]])
    return M

def rY(angle):
    sb = sp.sin(angle)
    cb = sp.cos(angle)
    M = sp.Matrix([[cb, 0, -sb],
                  [0, 1, 0],
                  [sb, 0, cb]])
    return M

```

```python
(rZ(alpha), rY(beta), rZ(gamma))
```

You'll find that the symbolic triple matrix product matches up with the results in Arfken for the definition of Euler angles $\alpha$, $\beta$, $\gamma$.  Note also that this is much easier to compute than by hand!  Also - less likely to result in errors.

```python
rZ(gamma)*rY(beta)*rZ(alpha)
```

To convert this symbolic expression to a numerical expression, one option is to use `lambdafy` from SymPy:

```python
eulerAngles = sp.lambdify((alpha,beta,gamma), rZ(gamma)*rY(beta)*rZ(alpha), "numpy")
```

```python
np.array([1,0,0]).dot(eulerAngles(np.pi/2.0,0,0))
```


## CCNSS 2018 Module 1: Neurons, synapses and networks
# Tutorial 1: Wilson-Cowan equations
[source](https://colab.research.google.com/drive/16strzPZxTEqR2owgSh6NNLlj2j7MNOQb)

Please execute the cell below to initalise the notebook environment.

```
import matplotlib.pyplot as plt    # import matplotlib
import numpy as np                 # import numpy
import scipy as sp                 # import scipy
import math                        # import basic math functions
import random                      # import basic random number generator functions

fig_w, fig_h = (6, 4)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
```

## Objectives
In this notebook we will introduce the *Wilson-Cowan* rate model, and use it to learn more about phase planes, nullclines, and attractors.

** Background paper:**
* Wilson H and Cowan J (1972) Excitatory and inhibitory interactions in localized populations of model neurons. Biophysical Journal 12.

## Background

The Wilson-Cowan equations model the mean-field (i.e., average across the population) dynamics of two coupled populations of excitatory (E) and inhibitory (I) neurons:

\begin{align}
&\tau_E \frac{dE}{dt} = -E + (1 - r E) F(w_{EE}E -w_{EI}I + I_{ext};a,\theta)\\
&\tau_I \frac{dI}{dt} = -I + (1 - r I) F(w_{IE}E -w_{II}I;a,\theta)
\end{align}

$E(t)$ represents the average activation of the excitatory population, and $I(t)$ the activation of the inhibitory population. The parameters $\tau_E$ and $\tau_I$ control the timescales of each population. The connection strengths are given by: $w_{EE}$ (E to E), $w_{EI}$ (I to E), $w_{IE}$ (E to I), and $w_{II}$ (I to I). Refractory effects are modelled through the parameter $r$, and $I_{ext}$ represents external input to the excitatory population.

The function F describes the population activation function. We assume F to be a sigmoidal function, which is parameterized by its gain $a$ and threshold $\theta$.

$$ F(x;a,\theta) = \frac{1}{1+\exp\{-a(x-\theta)\}} - \frac{1}{1+\exp\{a\theta\}}$$

The argument $x$ represents the input to the population. Note that the the second term is chosen so that $F(0;a,\theta)=0$.

To start, execute the cell below to initialise the simulation parameters.

```
dt = 0.1

# Connection weights
wEE = 12
wEI = 4
wIE = 13
wII = 11

# Refractory parameter
r = 1

# External input
I_ext = 0

# Excitatory parameters
tau_E = 1       # Timescale of excitatory population
a_E = 1.2       # Gain of excitatory population
theta_E = 2.8   # Threshold of excitatory population

# Inhibitory parameters
tau_I = 1       # Timescale of inhibitory population
a_I = 1         # Gain of inhibitory population
theta_I = 4     # Threshold of inhibitory population
```

**EXERCISE 1**

Fill in the function below to define the activation function F as a function of its input x, and arguments a, and $\theta$. Verify your function by evaluating the excitatory activation function for $x = 0,3,6$. Then plot F for both E and I population parameters over $0 \leq x \leq 10$.

```
def F(x,a,theta):
    """Population activation function.

    Arguments:
    x -- the population input
    a -- the gain of the function
    theta -- the threshold of the function

    Returns:
    y -- the population activation response
    """
    # insert your code here

    return y

# insert your code here
```

**EXPECTED OUTPUT**

```
0.0
0.5261444259857104
0.9453894296980492
```

**Exercise 2:** Fill in the function below to simulate the dynamics of the Wilson-Cowan equation for  up to $t_{max}=15$ with steps of $dt$. Remember from the LIF tutorial that we can numerically integrate the ODEs by replacing the derivatives with their discretized approximations:

\begin{align}
&\frac{dE}{dt} \to \frac{E[k+\Delta t]-E[k]}{\Delta t} \hspace{5 mm}\text{ and }\hspace{5mm}\frac{dI}{dt} \to \frac{I[k+\Delta t]-I[k]}{\Delta t}\\
\end{align}

Then simulate the dynamics of the population starting from initial condition $E_0=I_0=0.2$ and plot the results. What is the steady state solution? Then, also plot the dynamics starting from $E_0=I_0=0.25$ and plot the solution (in dashed lines). Now what is the steady state solution?

```
def simulate_wc(t,E0,I0):
    """Simulate the Wilson-Cowan equations.

    Arguments:
    t -- time (vector)
    E0 -- initial condition weeof the excitatory population
    I0 -- initial condition of the inhibitory population

    Returns:
    E -- Activity of excitatory population (vector)
    I -- Activity of inhibitory population (vector)
    """
    # insert your code here

    return E,I

# insert your code here
```

**EXPECTED OUTPUT**

**Exercise 3:** Now use the same function to simulate the Wilson Cowan equations for different initial conditions from $0.01 \leq E_0 \leq 1$ and $0.01 \leq I_0 \leq 1$ with stepsize 0.1. For each initial condition, find the steady state value to which $E$ and $I$ converge. There are several ways to do this. A simple way to do this is to check, for each initial condition, that the last two points in the simulation are within 1% of each other:

$$ \frac{E(t_{max})-E(t_{max}-dt)}{E(t_{max})} \leq 0.01 $$

Use the following code within your for loops to throw an error in case the trajectories have not converged:
``raise ValueError('Has not converged.')``

Then you can just keep increasing $t_{max}$ until every initial condition converges. Plot the steady state values ($E$ vs. $I$)  What fixed points do you see?

```
# insert your code here
```

**EXPECTED OUTPUT**

**Exercise 4**: To make the phase plane plot, we first need to determine the inverse of F. To calculate the inverse, set y = F(x), and then solve for x. Then, fill out the function below to define the inverse activation function $F^{-1}$. Check that this is the correct inverse function by testing $F^{-1}$ for $x=0,0.5,0.9$, and then plotting x against $F^{-1}(F(x))$ for $0\leq x\leq1$ (use the excitatory population parameters).

```
def F_inv(x,a,theta):
    """Define the inverse of the population activation function.

    Arguments:
    x -- the population input
    a -- the gain of the function
    theta -- the threshold of the function

    Returns:
    y -- value of the inverse function
    """
    # insert your code here

    return y

# insert your code here
```

**EXPECTED OUTPUT**

```
0.0
2.9120659956266
5.002378884081663
```

**Exercise 5:** Now, derive the E and I nullclines, in terms of the inverse function $F^{-1}$. To do this, set $\frac{dE}{dt}=0$ and solve for $I$, then set $\frac{dI}{dt}=0$ and solve for $E$. Then, fill out the two functions below to calculate the I nullcline (over $-0.01 \leq I \leq 0.3$) and E nullcline (over $-0.01 \leq E \leq 0.48$). First test the value of the I nullcline for $I=0.1$, then test the E nullcline for $E=0.1$. Then use these functions to plot the nullclines in phase space (E vs. I). What fixed points do you see? Compare the intersections of the nullclines with the steady state values you observed numerically in Exercise 3.

```
def get_E_nullcline(E):
    """Solve for I along the E nullcline (dE/dt = 0).

    Arguments:
    E -- values of E over which the nullcline is computed

    Returns:
    I -- values of I along the nullcline for each E
    """
    # insert your code here

    return I

def get_I_nullcline(I):
    """Solve for E along the I nullcline (dI/dt = 0).

    Arguments:
    I -- values of I over which the nullcline is computed

    Returns:
    E -- values of E along the nullcline for each I
    """
    # insert your code here

    return E

# insert your code here
```

**EXPECTED OUTPUT**
```
0.24546433162390224
-0.029802383619274175
```

**Exercise 6:** Now, on top of the nullclines, plot some sample trajectories starting with different initial conditions, for $0 \leq E_0 \leq 1$ and $0 \leq I_0 \leq 1$. How many attractors do you see?

```
# insert your code here
```

**EXPECTED OUTPUT**

**Exercise 7:** Repeat the previous exercise while varying the recurrent excitatory connectivity over the following values: $w_{EE}=5,10,12,15$. What is happening? Can you find a value of wEE where a qualitative transformation occurs? What does this tell you about how increasing recurrent connectivity affects the dynamics?

```
# insert your code here
```

**EXPECTED OUTPUT**


# 14 Partial Differential Equations — 1

## Solving Laplace's or Poisson's equation

**Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

Solutions depend on the **boundary conditions**:

* the *value of the potential* on the *boundary* or
* the *electric field* (i.e. the derivative of the potential, $\mathbf{E} = -\nabla\Phi$ *normal to the surface* ($\mathbf{n}\cdot\mathbf{E}$), which directly follows from the charge distribution).

### Example: 2D Laplace equation
$$
\frac{\partial^2 \Phi(x,y)}{\partial x^2} + \frac{\partial^2 \Phi(x,y)}{\partial y^2} = 0
$$
("elliptic PDE")

Boundary conditions:
* square area surrounded by wires
* three wires at ground (0 V), one wire at 100 V

## Finite difference algorithm for Poisson's equation
Discretize space on a lattice (2D) and solve for $\Phi$ on each lattice site.

Taylor-expansion of the four neighbors of $\Phi(x, y)$:

\begin{align}
\Phi(x \pm \Delta x, y) &= \Phi(x, y) \pm \Phi_x \Delta x + \frac{1}{2} \Phi_{xx} \Delta x^2 + \dots\\
\Phi(x, y \pm \Delta y) &= \Phi(x, y) \pm \Phi_y \Delta x + \frac{1}{2} \Phi_{yy} \Delta x^2 + \dots\\
\end{align}

Add equations in pairs: odd terms cancel, and **central difference approximation** for 2nd order partial derivatives (to $\mathcal{O}(\Delta^4)$):

\begin{align}
\Phi_{xx}(x,y) = \frac{\partial^2 \Phi}{\partial x^2} & \approx
  \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) - 2\Phi(x,y)}{\Delta x^2} \\
\Phi_{yy}(x,y) = \frac{\partial^2 \Phi}{\partial y^2} &\approx
  \frac{\Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) - 2\Phi(x,y)}{\Delta y^2}
\end{align}

Take $x$ and $y$ grids of equal spacing $\Delta$: Discretized Poisson equation

$$
\begin{split}
\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) &+ \\
   +\, \Phi(x,y-\Delta y) - 4\Phi(x,y) &= -4\pi\rho(x,y)\,\Delta^2
   \end{split}
$$

Or written for lattice sites $(i, j)$ where

$$
x = x_0 + i\Delta\quad\text{and}\quad y = y_0 + j\Delta, \quad 0 \leq i,j < N_\text{max}
$$

$$
\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1} - 4 \Phi_{i,j} = -4\pi\rho_{ij} \Delta^2
$$

Defines a system of $N_x \times N_y$ simultaneous algebraic equations for $\Phi_{ij}$ to be solved.

Can be solved directly via matrix approaches (and then is the best solution) but can be unwieldy for large grids.

Alternatively: **iterative solution**:

$$
\begin{split}
4\Phi(x,y) &= \Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\\
 &+ \Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) + 4\pi\rho(x,y)\,\Delta^2
\end{split}
$$

$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

* Converged solution at $(i, j)$ will be the average potential from the four neighbor sites + charge density contribution.
* *Not a direct solution*: iterate and hope for convergence.

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

#### Gauss-Seidel method

$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

### Solution via relaxation (Gauss-Seidel)

Solve the box-wire problem on a lattice: The wire at $x=0$ (the $y$-axis) is at 100 V, the other three sides of the box are grounded (0 V).

Note: $\rho=0$ inside the box.

Note for Jupyter notebook use:
* For interactive 3D plots, select
  ```
  %matplotlib widget
  ```
  or if the above fails, try
  ```
  %matplotlib notebook
  ```
* For standard inline figures (e.g. for exporting the notebook to LaTeX/PDF or html) use
  ```
  %matplotlib inline
  ```

Enable a matplotlib-Jupyter integration that works for you (try `conda install ipympl` or `pip install ipympl` first to get `%matplotlib widget` working).

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
# for interactive work

%matplotlib widget
#%matplotlib inline
#%matplotlib notebook
```

```python
# for plotting/saving
%matplotlib inline
```

#### Wire on a box: Solution of Laplace's equation with the Gauss-Seidel algorithm

* set boundary conditions
* Implement Gauss-Seidel algorithm
* visualize solution
* does it make sense?
* try higher `Max_iter`

```python
Nmax = 100
Max_iter = 70
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

Nx, Ny = Phi.shape

for n_iter in range(Max_iter):
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
```

#### Visualization of the potential

```python
# plot Phi(x,y)
x = np.arange(Phi.shape[0])
y = np.arange(Phi.shape[1])
X, Y = np.meshgrid(x, y)

Z = Phi[X, Y]
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')

ax.view_init(elev=40, azim=-65)
```

### Surfaces and 2D contours

Nicer plot (use this code for other projects):

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
surf = ax.plot_surface(X, Y, Z, cmap=plt.cm.coolwarm, alpha=0.6)
cset = ax.contourf(X, Y, Z, zdir='z', offset=-50, cmap=plt.cm.coolwarm)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')
ax.set_zlim(-50, 100)
ax.view_init(elev=40, azim=-65)

cb = fig.colorbar(surf, shrink=0.5, aspect=5)
cb.set_label(r"potential $\Phi$ (V)")
```

(Note that the calculation above is is *not converged* ... see next lecture.)


# 习题

- Q-1

```python
from sympy import *

x = symbols('x')
f_x = x**3 - 3 * x**2 - x - 1
g_x = 3 * x**2 - 2 * x + 1
q_x, r_x = div(f_x, g_x)
q_x.as_poly()
```

```python
r_x.as_poly()
```

- Q-2

```python
from sympy import *

x, m, p, q = symbols('x, m, p, q')
f_x = x**3 + p * x + q
g_x = x**2 + m * x - 1
q_x, r_x = div(f_x, g_x)
r_x.as_poly(x)
```

> 说明=>r_x = 0, 由此得出各项系数为 0

- Q-3

```python
from sympy import *

x = symbols('x')
f_x = x**3 - x**2 - x
g_x = x - 1 + 2 * I
q_x, r_x = div(f_x, g_x, domain='ZZ')
q_x
```

```python
r_x
```

- Q-4
- Q-4-3

```python
from sympy import *

x = symbols('x')
f_x = x**5
x0 = 1
series(f_x, x=x, x0=x0)
```

- Q-4-3

```python
from sympy import *

x = symbols('x')
f_x = x**4 + 2 * I * x**3 - (1 + I) * x**2 - 3 * x + 7 + I
series(f_x, x=x, x0=-I)
```

- Q-5

```python
from sympy import *

x = symbols('x')
f_x = x**4 - 10 * x**2 + 1
g_x = x**4 - 4 * sqrt(2) * x**3 + 6 * x**2 + 4 * sqrt(2) * x + 1
gcd(f_x.as_poly(x), g_x.as_poly(x))

```

$\displaystyle \operatorname{Poly}{\left( x^{2} -  2 \sqrt{2} x - 1, x, domain=\mathtt{\text{EX}} \right)}$

- Q-6

```python
from sympy import *

x = symbols('x')
f_x = x**4 - x**3 - 4 * x**2 + 4 * x + 1
g_x = x**2 - x - 1
gcdex(f_x, g_x)
```

```python

```


## Import libraries

```python
from matplotlib import pyplot as plt
import numpy as np

%matplotlib inline
```

## Define function
The goal is to create a plot of the following function
\begin{equation}
f(x)=0.2+0.4x^2+0.3x\cdot\sin(15x)+0.05\cos(50x)
\end{equation}

```python
x = np.linspace(0, 1, 100)
y = 0.2+0.4*x**2+0.3*x*np.sin(15*x)+0.05*np.cos(50*x)
```

## Produce figure

```python
plt.figure(figsize=(6, 6))
plt.plot(x, y)
plt.show()
```

```python

```


```python
from sympy import *
import pandas as pd
import numpy as np
init_printing(use_unicode=True)

x, y = symbols('x y')
```

```python
def check_maximum(f,interval,symbol):
    possiveis_max = []
    borda1 = (f.subs(symbol,interval.args[0]).evalf())
    borda2 = (f.subs(symbol,interval.args[1]).evalf())

    possiveis_max.append(borda1)
    possiveis_max.append(borda2)
    f_ = diff(f)
    zeros = solve(f_)
    for zero in zeros:
        if str(type(zero)) == "<class 'sympy.core.add.Add'>":
            zero = zero.evalf()
        if zero in interval:
            possiveis_max.append(f.subs(symbol,zero).evalf())

    possiveis_sem_complex = []
    for ele in possiveis_max:
        if str(type(ele)) != "<class 'sympy.core.add.Add'>":
            possiveis_sem_complex.append(float(ele))
    return Matrix(possiveis_sem_complex)

def df_from_M(M, func = None, symb = symbols('x')):
    x = symb
    M = transpose(M)
    M = np.array(M).astype(np.float64)
    try:
        df = pd.DataFrame(M, columns=['x', 'f(x)'])
    except:
        df = pd.DataFrame(M, columns=['x'])
        df['f(x)'] = ''
        for i in range(df.shape[0]):
            df.loc[i, 'f(x)'] = Rational(func.subs(x, Rational(df.loc[i, 'x'])))
    return df

class f_newton:
    def __init__(self, ind, xlist, ylist):
        n = len(xlist)
        xlist = [Rational(n) for n in xlist]
        ylist = [Rational(n) for n in ylist]
        self.n = n
        name = 'f['
        for i in range(n):
            name += 'x{},'.format(ind + i)
        name = name[:-1]
        name += ']'
        self.name = name
        self.xlist = xlist
        self.buffer = np.array([ylist,[0 for i in range(len(ylist))]]).transpose()
        self.list_ = xlist
        self.nivel = 0
        self.acha_val()

    def acha_val(self):
        while self.buffer.shape[0] >1:
            self.nivel += 1
            xlist = self.xlist
            buffer = self.buffer
            for i in range(buffer.shape[0]-1):
                buffer[i,1] = (buffer[i+1,0] - buffer[i,0])/(xlist[i+self.nivel]-xlist[i])
            buffer = np.hstack([buffer[:-1,1:],np.zeros(buffer[:-1,1:].shape)])
            self.buffer = buffer
        self.val = self.buffer[0,0]
        return self.val

class interpolador():
    def __init__(self, matrix, func=None,symb = symbols('x')):
        df = df_from_M(matrix, func, symb)
        self.df = df
        self.symb = symb
        min_ = df['x'].min()
        max_ = df['x'].max()
        Inter = Interval(min_,max_)
        self.min_ = min_
        self.max_ = max_
        self.Inter = Inter
        self.func = func

    def lagrange(self):
        df = self.df
        x = self.symb

        df['Li(x)'] = ''
        p = 0
        for i in range(df.shape[0]):
            up = 1
            down = 1
            for j in range(df.shape[0]):
                if i != j:
                    up *= (x-Rational(df.loc[j,'x']))
                    down *= (Rational(df.loc[i,'x'])-Rational(df.loc[j,'x']))
            df.loc[i, 'Li(x)'] = simplify(up/down)
            p += (up/down)*Rational(df.loc[i, 'f(x)'])
            p = simplify(p)
        self.df = df
        self.p_lagr = p

    def newton(self):
        x = symbols('x')
        df =  self.df
        xlist = df['x'].to_list()
        ylist = df['f(x)'].to_list()
        names = ['x','f(x)']
        n = len(xlist)
        arr = np.full((n,n-1), Rational(0))
        arr_ = np.full((n,n+1), Rational(0))
        for j in range(n):
            for i in range(n-j-1):
                if i == 0:
                    names.append(f_newton(i, xlist[i:i+j+2],ylist[i:i+j+1]).name)
                arr[i,j] = Rational(f_newton(i, xlist[i:i+j+2],ylist[i:i+j+2]).acha_val())
        arr_[:,2:] = arr
        arr_[:,0:1] = np.array([xlist]).transpose()
        arr_[:,1:2] = np.array([ylist]).transpose()
        df = pd.DataFrame(arr_, columns=[names])
        p_new = 0
        termo = 1
        for i in range(arr_.shape[1]-1):
            p_new += Rational(arr_[0,i+1])*termo
            termo *= (x - Rational(xlist[i]))
        self.df = df
        self.p_new = simplify(p_new)

    def Erro(self):
        x = symbols('x')
        func = self.func
        df = self.df
        Inter = self.Inter

        if func != None:
            Erro = 1
            n = df.shape[0]
            func___ = func
            for i in range(n):
                try:
                    Erro *= (x-Rational(df.loc[i,'x']))
                except:
                    Erro *= (x-Rational(df.loc[i,'x'].values[0]))
                func___ = diff(func___)
            # Erro = abs(Erro)
            Erro /= Rational(factorial(n+1))
            maxi = max(abs(check_maximum(func___,Inter, x)))
            Erro *= maxi
            Erro = simplify(Erro)/2
            self.Erro = Erro
            return Erro

class romberg:
    def __init__(self, Ts):
        h = symbols('h')
        cols = ['h','T(h)','S(h)','W(h)']
        df = pd.DataFrame(columns = cols)
        df['T(h)'] = Ts
        for i in range(df.shape[0]):
            df.loc[i, 'h'] = h
            h *= 1/Rational(2)
            if i != df.shape[0] - 1:
                i += 1
                df.loc[i, 'S(h)'] = (4*df.loc[i, 'T(h)'] - df.loc[i-1, 'T(h)'])/Rational(3)
                df.loc[i, 'W(h)'] = (16*df.loc[i, 'S(h)'] - df.loc[i-1, 'S(h)'])/Rational(15)
        self.df = df

class gauss:
    def __init__(self, grau, Inter,func,symb = symbols('x')):
        x = symb
        t = symbols('t')
        cnj = {
          2:{
              0:1,
              1:1
          },
          3:{
            0:0.5555555555555555555555,
            1:0.8888888888888888888888,
            2:0.5555555555555555555555
          },
          4:{
            0:0.3478548451,
            1:0.6521451549,
            2:0.6521451549,
            3:0.3478548451
          }
        }
        xnj = {
          2:{
              0:0.5773502692,
              1:-0.5773502692
          },
          3:{
            0:0.7745966692,
            1:0,
            2:-0.7745966692
          },
          4:{
            0:0.8611363116,
            1:0.3399810436,
            2:-0.3399810436,
            3:-0.8611363116
          }
        }
        n = 0
        while 2*n-1 < grau:
            n +=1
        self.n = n
        res = 0
        a = Inter.args[0]
        b = Inter.args[1]
        var = ((b-a)*t + a + b)/2

        var_ = diff(var, t)
        func = func.subs(x, var)
        for i in range(n):
            res += cnj[n][i]*func.subs(t, xnj[n][i])
        res *= var_
        self.res = res

class euler1l:
    def __init__(self, x0, y0, h, func):
        x, y = symbols('x y')
        xlist = []
        ylist = []
        xlist.append(x0)
        ylist.append(y0)
        for i in range(1,11):
            ylist.append(ylist[-1] + h*func.subs(x, xlist[-1]).subs(y, ylist[-1]))
            xlist.append(xlist[-1] + h)
        df = pd.DataFrame()
        df['x'] = xlist
        df['y'] = ylist
        self.df = df
        self.xs = xlist
        self.ys = ylist

class eulermod:
    def __init__(self, x0, y0, h, func):
        x, y = symbols('x y')
        xlist = []
        ylist = []
        xlist.append(x0)
        ylist.append(y0)
        for i in range(1,10):
            ylist.append(ylist[-1] + (h/2)*(func.subs(x, xlist[-1]).subs(y, ylist[-1]) + func.subs(x, xlist[-1] + h).subs(y, ylist[-1] + h*func.subs(x, xlist[-1]).subs(y, ylist[-1]))))
            xlist.append(xlist[-1] + h)
        df = pd.DataFrame()
        df['x'] = xlist
        df['y'] = ylist
        self.df = df
        self.xs = xlist
        self.ys = ylist

class eulerM:
    def __init__(self, x0, y0, h,coef):
        xlist = []
        ylist = []
        xlist.append(x0)
        ylist.append(y0)
        dlist = []
        for i in range(1,11):
            dlist.append(ylist[-1]*coef)
            ylist.append(ylist[-1] + h*dlist[-1])
            xlist.append(xlist[-1] + h)
        df = pd.DataFrame()
        df['x'] = xlist
        df['y'] = ylist
        self.df = df
        self.xs = xlist
        self.ys = ylist
```

```python
print('-----------------------------------------------------')
print('Q2')
A=[2/Rational(3), -5/Rational(3)]
Y0=[-7, -8]
h=1/Rational(10)

X0 = [0, 0]
a0 = eulerM(X0[0], Y0[0], h, A[0])
n1 = a0.ys[2]

a1 = eulerM(X0[1], Y0[1], h, A[1])
n2 = a1.ys[2]

print('Resp', n1+n2)

erros = []
a0.ys[-1] + 7*exp(2/3)
# for i in range(2):
#     segderi = Y0[i] * A[i]**2 * exp(A[i]*x)
#     Msegderi = max(abs(check_maximum(segderi,Interval(0,1),x)))
#     L = abs(A[i])
#     erro = h*Msegderi/(2*L)*(exp(L*1) - 1)
#     erros.append(erro.evalf())
# print(erros)
```

```python
print('-----------------------------------------------------')
print('Q3')

y0 = symbols('y0')
func = y*x**(-2)
x0 = 1/Rational(5)
xf = 3/Rational(10)
yf = -27/Rational(8)
h = (xf - x0)/2

a = euler1l(x0, y0,h,func)
y0_ = solve(a.df.loc[2,'y'] - yf)[0]
print(y0_)
a = eulermod(x0, y0_,h,func)
print(a.df.loc[2,'y'])
```

    -5/6
    -1919/480

```python
print('-----------------------------------------------------')
print('Q4')
M = Matrix([
    [0, 1, 2, 3],
])
sube = 1/Rational(6) #Valor que deve ser substituido em X para obter a fração desejada
a = interpolador(M, (Rational(8)/Rational(5))**x)

a.newton()
print()
print('Primeira resposta é a primeira linha do dataframe abaixo.\n', a.df)
print()
print()
print('Fração que é a resposta da questão do meio:')
print()
pprint(a.p_new.subs(x,Rational(sube)))

print()
print()
b = a.Erro()
print('Erro no ponto x = 1/6:           ', abs(b.subs(x, sube))*10)
print('Majoramento do erro no intervalo:', max(abs(check_maximum(b, Interval(M[0], M[-1]), x)))*10 )

```

    Primeira linha df
        x f(x) f[x0,x1] f[x0,x1,x2] f[x0,x1,x2,x3]
    0 -2  4/9      2/9        1/18          1/108
    1 -1  2/3      1/3        1/12              0
    2  0    1      1/2           0              0
    3  1  3/2        0           0              0
    Fracao 353/288
    Debaixo 0.00168925015274316

```python
print('-----------------------------------------------------')
print('Q5')

x = symbols('x')
func = Rational(1)/(Rational(16)/Rational(7) *x + Rational(6)/Rational(5))
Inter = Interval(8/3,49/18)
func__ = func
pprint(func)
for i in range(3):
    func__ = diff(func__)
check_maximum(func__, Inter, x)/(factorial(3))
```

       1
    ────────
    16⋅x   6
    ──── + ─
     7     5

$\displaystyle \left[\begin{matrix}-0.00421607128879581\\-0.0039348666208735\end{matrix}\right]$

```python
print('-----------------------------------------------------')
print('Q6')
h, fa, fb, fx1, fx2, fx3 = symbols('h fa fb fx1, fx2, fx3')

resps = [Rational(5)/2, Rational(5)/3, Rational(493)/336]
a = 1/Rational(2)

Eq1e = (fa + fb)*h
Eq1d = 2*a*resps[0]

Eq2e = h*(fa + fb) + h*2*fx2
Eq2d = 4*a*resps[1]

Eq3e = h*(fa + fb)+h*(+ 2*fx1 + 2*fx2+ 2*fx3)
Eq3d = 8*a*resps[2]

Eq3e *= 2
Eq3d *= 2

Eq3e += - Eq2e
Eq3d += - Eq2d

Eq3e *= 1/Rational(6)
Eq3d *= 1/Rational(6)

Eq3e = simplify(Eq3e)
pprint(Eq3d)
a = romberg(resps)
a.df

```

```python
print('-----------------------------------------------------')
print('Q7')
x = symbols('x')
Inter = Interval(-3,7)
M = Matrix([
    [-3, 2, 7],
    [2/5, -2, 4/9]
])

b = interpolador(M)
b.lagrange()
func = b.p_lagr
grau = 2

a = gauss(grau, Inter, func)
print(a.res)
t = symbols('t')
a = Inter.args[0]
b = Inter.args[1]
var = (2*x - a - b)/(b-a)
var_ = diff(var, t)
Legendre = t**3 - (3/5)*t
print(solve(Legendre.subs(t, var)))
```

    -11.9259259256358
    [-1.87298334620742, 2.00000000000000, 5.87298334620742]


# **1.3 Calculate elastic stiffness of a given composite**

[](https://moodle.rwth-aachen.de/mod/page/view.php?id=551801)

## Task
Predict the tensile stiffness of a reinforced concrete cross section shown in the Figure with the thickness of 10~mm and width of 100 mm.
The cross-section is reinforced with 6 layers of textile fabrics made of CAR-EP3300 specified in the table below

## Theory
The derived mixture rule will be used to solve the task

## Data

\begin{array}{|c|c|c|c|c|c|}
    \mathrm{Label}
    & \mathrm{Material}
    & \mathrm{Area }
    & \mathrm{Grid\; spacing}
    & \mathrm{Stiffness}
    & \mathrm{Strength\; (characteristic)}
    \\
  \hline
    &
    & [\mathrm{mm}^2]
    & [\mathrm{mm}]
    & [\mathrm{MPa}]
    & [\mathrm{Mpa}]
    \\
   \hline
    \mathrm{CAR-EP3300}
    & \mathrm{carbon/proxy}
    & 1.84
    & 25.77
    & 240000
    & 3500
    \\
   \hline
    \mathrm{solidian\; GRID \; Q95}
    & \mathrm{carbon/proxy}
    & 3.62
    & 36.0
    & 240000
    & 3200
    \\
\end{array}

# How to evaluate an expression?

**Calculator:** Let us use Python language as a calculator and evalute the mixture rule for the exemplified cross-section.m

```python
A_roving = 1.84 # [mm**2]
n_layers = 6 # -
spacing = 25.77 # [mm]
thickness = 10 # [mm]
width = 100 # [mm]
E_carbon = 240000 # [MPa]
E_concrete = 28000 # [MPa]
```

```python
A_composite = width * thickness
n_rovings = width / spacing
A_layer = n_rovings * A_roving
A_carbon = n_layers * A_layer
A_concrete = A_composite - A_carbon
E_composite = (E_carbon * A_carbon + E_concrete * A_concrete) / A_composite
E_composite
```

    37082.18859138533

Thus, the composite has an effective stiffness of 37 GPa.

# How to construct a model?

Once we have derived the formula capturing the design question, i.e. "what is the stiffness of the new composite?" we want to learn from the model and develop a new intuition. To explore the possible composite designs let us rewrite the above equations as symbolic expressions. Then, we can construct a model which can be interactively used to study the available design options.

In contrast to the previously performed numerical evaluation, we now express the derived equations as mathematical symbols instead of numbers. To do this, let us use a Python package `sympy` to do symbolic algebra.

```python
import sympy as sp
```

The parameters of the model are now introduced as `sympy.symbols`. To distinguish the symbols from numbers introduced above, let us name the Python variables referring to the mathematical symbols with a trailing underscore `_`

```python
A_roving_ = sp.Symbol('A_r')
n_layers_ = sp.Symbol('n_l')
spacing_ = sp.Symbol('d')
thickness_ = sp.Symbol('h')
width_ = sp.Symbol('b')
E_carbon_ = sp.Symbol('E_car')
E_concrete_ = sp.Symbol('E_c')
```

To see the difference between a variable referring to a number and to a symbol, let us display the variables `A_roving` and `A_roving_`

```python
display(A_roving, A_roving_)
```

    1.84

$\displaystyle A_{r}$

Let us now rephrase the the above derived equations in the symbolic form, i.e. using the symbols with the trailing underscore `_`

```python
A_composite_ = width_ * thickness_
n_rovings_ = width_ / spacing_
A_layer_ = n_rovings_ * A_roving_
A_carbon_ = n_layers_ * A_layer_
A_concrete_ = A_composite_ - A_carbon_
E_composite_ = (E_carbon_ * A_carbon_ + E_concrete_ * A_concrete_) / A_composite_
sp.simplify(E_composite_)
```

$\displaystyle \frac{A_{r} E_{car} n_{l} - E_{c} \left(A_{r} n_{l} - d h\right)}{d h}$

**The power of modeling**

 - Instead of a number, we have now a symbolic expression showing the influence of the individual parameters on a design property, i.e. on the material stiffness $\bar{E}$.

 - This expression represents the first model that we constructuted in the course using the conditions of compatibility, equilibrium and constitutive laws.

 - Using a model, we can explore the behavior of the composite, develop a design intuition, optimize the design.
   We learn how to construct simple models describing the mechanisms governing the behavior of a composite.

 - We will construct simplified analytical models for pull-out, that will help us to understand elementary types of
   material behavior.

# Next steps

 - Login to jupyter.rwth-aachen.de
 - Navigate to this mixture rule example
 - Evaluate the cells by issueing the [Shift+Return] key combination

# Why Jupyter Lab? Why Python?
 - [Jupyterlab introduction](https://youtu.be/A5YyoCKxEOU) [7 mins] video explaining the basic features of jupyter notebook within jupyter lab
 - [Most Popular Programming Languages 1965 - 2019](https://www.youtube.com/watch?v=Og847HVwRSI) Check this race of programming languages over the last 50 years and wait till the end of it ;-)
 - Useful packages
   - `matplotlib` - how to plot fancy diagrams
   - `sympy` - how to perform algebraic manipulations
   - `numpy` - how to manipulate data - beyond `Excel`
   will be shortly explained.

      1.2 Jupyter basics

    2.1 Pullout of elastic fiber from rigid matrix

```python

```


```python
from sympy import Matrix, symbols, sqrt, asin

x1, x2, x3, x4, K, r, Re, alpha0, theta = symbols('x1 x2 x3 x4 K r Re alpha0 theta')

H = Matrix([[sqrt(x1**2 + x3**2)]])
display(H)

x = Matrix([[x1],
            [x2],
            [x3]])
display(x)

dH = H.jacobian(x)
display(dH)
```

$\displaystyle \left[\begin{matrix}\sqrt{x_{1}^{2} + x_{3}^{2}}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}\frac{x_{1}}{\sqrt{x_{1}^{2} + x_{3}^{2}}} & 0 & \frac{x_{3}}{\sqrt{x_{1}^{2} + x_{3}^{2}}}\end{matrix}\right]$

```python
x = Matrix([[x1],
            [x2],
            [x3],
            [x4]])
display(x)

F = Matrix([[x2],
            [x1*x4**2 - K/x1**2],
            [x4],
            [-2*x2*x4/x1]])
display(F)

dF = F.jacobian(x)
display(dF)

H = Matrix([[asin(Re/x1)],
            [alpha0 - x3]])
display(H)

dH = H.jacobian(x)
display(dH)
```

$\displaystyle \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\\x_{4}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}x_{2}\\- \frac{K}{x_{1}^{2}} + x_{1} x_{4}^{2}\\x_{4}\\- \frac{2 x_{2} x_{4}}{x_{1}}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}0 & 1 & 0 & 0\\\frac{2 K}{x_{1}^{3}} + x_{4}^{2} & 0 & 0 & 2 x_{1} x_{4}\\0 & 0 & 0 & 1\\\frac{2 x_{2} x_{4}}{x_{1}^{2}} & - \frac{2 x_{4}}{x_{1}} & 0 & - \frac{2 x_{2}}{x_{1}}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}\operatorname{asin}{\left(\frac{Re}{x_{1}} \right)}\\\alpha_{0} - x_{3}\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}- \frac{Re}{x_{1}^{2} \sqrt{- \frac{Re^{2}}{x_{1}^{2}} + 1}} & 0 & 0 & 0\\0 & 0 & -1 & 0\end{matrix}\right]$


# ODE initial value problems

This notebook will explore how oridinary differential equations may be solved using Julia and the DifferentialEquations.jl package. We will specifically focus on intial value problems; boundary value problems will be examined in another notebook.

```julia
#using Pkg
#Pkg.add( "DifferentialEquations" )
```

```julia
using DifferentialEquations
```

    ┌ Info: Precompiling DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
    └ @ Base loading.jl:1260

Lets solve the equation
$$
\frac{d u}{d t} = p u
$$
on the interval $t \in [0,1]$ with the initial condition $u(0)=u_0$ where $p$ is a constant. The exact solution is $u(t)=u_0 e^{p t}$.

```julia
function simple_function!(du, u, p, t)
    du[1] = p[1] * u[1]
end

p = [1.1]
u0 = [1.0]
tspan = (0.0,1.0)
prob = ODEProblem(simple_function!,u0,tspan,p)
sol = solve(prob, Tsit5(), reltol=1e-8, abstol=1e-8)

using Plots
plot(sol,linewidth=5, xaxis="t",yaxis="u(t)",label="Numerical solution") # legend=false
plot!(sol.t, t->u0[1]*exp(p[1]*t),lw=3,ls=:dash,label="Exact Solution")
```

Here Tsit5() is the standard non-stiff solver. Now lets solve a system of ODEs such as the lorenz equations
$$
\begin{align}
\frac{dx}{dt} &= \sigma (y-x), \\
\frac{dy}{dt} &= x (\rho-z) - y, \\
\frac{dz}{dt} &= xy - \sigma z.
\end{align}
$$
This simplified model for atmospheric convection has chaotic solutions when the parameters have the values $\sigma = 10$, $\rho = 28$ and $\beta = 8/3$. We choose the initial conditions $x(0)=1$, $y(0)=0$ and $z(0)=0$.

```julia
function lorenz!(du,u,p,t)
  x,y,z = u
  σ,ρ,β = p
  du[1] = dx = σ*(y-x)
  du[2] = dy = x*(ρ-z) - y
  du[3] = dz = x*y - β*z
end
```

    lorenz! (generic function with 1 method)

```julia
p = [10.0,28.0,8/3]
u0 = [1.0;0.0;0.0]
tspan = (0.0,100.0)
prob = ODEProblem(lorenz!,u0,tspan,p)
sol = solve(prob)
plot(sol,vars=(1,2,3), title="Lorenz attractor", xaxis="x", yaxis="y",
     zaxis="z")
```

We can also plot the time series for each variable individually.

```julia
plot(sol, vars=(0,1), label="x(t)")
plot!(sol, vars=(0,2), label="y(t)")
plot!(sol, vars=(0,3), label="z(t)")
```

```julia

```


# Kullback-Leibler Divergence

The Kullback-Leibler divergence (KLD) measures the distance between two probability distributions, $Q$ and $P$. KLD between $Q$ and $P$ is defined as follows.

* $D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \, \log\frac{P(i)}{Q(i)}$
* $D_{\mathrm{KL}}(P\|Q) \geq 0$

The way to interpret the value of KLD is

* as the KLD is closer to zero, $P$ and $Q$ are more similar
* as the KLD is moves away from zero, $P$ and $Q$ are more dissimilar (diverging, more distant)

In the example below, we will calculate KLD against three distributions, each associated with different models. Model 1 takes on the following form.

* $X_1 \sim \mathcal{N}(0, 1)$
* $X_2 \sim \mathcal{N}(1, 1)$
* $X_3 \sim \mathcal{N}(2 + 0.8x_1 - 0.2x_2, 1)$

Model 2 takes on the following form.

* $X_1 \sim \mathcal{N}(0.85, 1)$
* $X_2 \sim \mathcal{N}(1.05, 1)$
* $X_3 \sim \mathcal{N}(2 + 0.9x_1 - 0.25x_2, 1)$

Model 3 takes on the following form.

* $X_1 \sim \mathcal{N}(2, 1)$
* $X_2 \sim \mathcal{N}(5, 1)$
* $X_3 \sim \mathcal{N}(4 + 0.8x_1 - 0.8x_2, 1)$

Note how Models 1 and 2 were constructed to be very similar, and Model 3 to be very dissimilar to Models 1 and 2.

## Generate data

```python
%matplotlib inline
import matplotlib.pylab as plt
import numpy as np
import seaborn as sns
from numpy.random import normal
from scipy.stats import multivariate_normal, norm, entropy

np.random.seed(37)
sns.set_style('whitegrid')
num_samples = 1000

x1 = normal(0, 1, num_samples)
x2 = normal(1, 1, num_samples)
x3 = normal(2 + 0.8 * x1 - 0.2 * x2, 1, num_samples)

data1 = data = np.column_stack((x1, x2, x3))
means1 = data1.mean(axis=0)
covs1 = np.cov(data1, rowvar=False)

x1 = normal(0.85, 1, num_samples)
x2 = normal(1.05, 1, num_samples)
x3 = normal(2 + 0.9 * x1 - 0.25 * x2, 1, num_samples)

data2 = np.column_stack((x1, x2, x3))
means2 = data2.mean(axis=0)
covs2 = np.cov(data2, rowvar=False)

x1 = normal(2, 1, num_samples)
x2 = normal(5, 1, num_samples)
x3 = normal(4 + 0.8 * x1 - 0.8 * x2, 1, num_samples)

data3 = np.column_stack((x1, x2, x3))
means3 = data3.mean(axis=0)
covs3 = np.cov(data3, rowvar=False)

print('means_1 = {}'.format(means1))
print('covariance_1')
print(covs1)
print('')
print('means_2 = {}'.format(means2))
print('covariance_2')
print(covs2)
print('')
print('means_3 = {}'.format(means3))
print('covariance_3')
print(covs3)
```

    means_1 = [0.01277839 0.9839153  1.80334137]
    covariance_1
    [[ 0.9634615  -0.00371354  0.76022725]
     [-0.00371354  0.97865653 -0.25181086]
     [ 0.76022725 -0.25181086  1.63064517]]

    means_2 = [0.85083876 1.07957661 2.46909572]
    covariance_2
    [[ 1.00406579  0.03774339  0.91788487]
     [ 0.03774339  1.00889847 -0.21973076]
     [ 0.91788487 -0.21973076  1.94124604]]

    means_3 = [2.00362816 4.97508849 1.65194765]
    covariance_3
    [[ 1.01322936  0.0112429   0.75369598]
     [ 0.0112429   0.96736793 -0.76265399]
     [ 0.75369598 -0.76265399  2.14695264]]

Note how we estimate the means and covariance matrix of Models 1 and 2 from the sampled data. For any observation, ${\mathbf X} = (x_{1}, \ldots, x_{k})$, we can compute the probablity of such data point according to the following probability density function.

$\begin{align}
f_{\mathbf X}(x_1,\ldots,x_k)
& = \frac{\exp\left(-\frac 1 2 ({\mathbf x}-{\boldsymbol\mu})^\mathrm{T}{\boldsymbol\Sigma}^{-1}({\mathbf x}-{\boldsymbol\mu})\right)}{\sqrt{(2\pi)^k|\boldsymbol\Sigma|}}
\end{align}$

## Visualize

Let's visualize the density curves of each variable in the models.

```python
fig, ax = plt.subplots(1, 1, figsize=(10, 5))
ax.set_title('Model 1')
ax.set_xlim([-4, 8])
sns.kdeplot(data1[:,0], bw=0.5, ax=ax, label=r'$x_{1}$')
sns.kdeplot(data1[:,1], bw=0.5, ax=ax, label=r'$x_{2}$')
sns.kdeplot(data1[:,2], bw=0.5, ax=ax, label=r'$x_{3}$')

fig, ax = plt.subplots(1, 1, figsize=(10, 5))
ax.set_title('Model 2')
ax.set_xlim([-4, 8])
sns.kdeplot(data2[:,0], bw=0.5, ax=ax, label=r'$x_{1}$')
sns.kdeplot(data2[:,1], bw=0.5, ax=ax, label=r'$x_{2}$')
sns.kdeplot(data2[:,2], bw=0.5, ax=ax, label=r'$x_{3}$')

fig, ax = plt.subplots(1, 1, figsize=(10, 5))
ax.set_title('Model 3')
ax.set_xlim([-4, 8])
sns.kdeplot(data3[:,0], bw=0.5, ax=ax, label=r'$x_{1}$')
sns.kdeplot(data3[:,1], bw=0.5, ax=ax, label=r'$x_{2}$')
sns.kdeplot(data3[:,2], bw=0.5, ax=ax, label=r'$x_{3}$')
```

## Measure divergence

Now that we have estimated the parameters (means and covariance matrix) of the models, we can plug these back into the density function above to estimate the probability of each data point in the data simulated from Model 1. Note that $P$ is the density function associated with Model 1, $Q1$ is the density function associated with Model 2, and $Q2$ is the density function associated with Model 3. Also note

* $D_{\mathrm{KL}}(P\|P) = 0$
* $D_{\mathrm{KL}}(P\|Q) \neq D_{\mathrm{KL}}(Q\|P)$ (the KLD is asymmetric)

```python
P = multivariate_normal.pdf(data1, mean=means1, cov=covs1)
Q1 = multivariate_normal.pdf(data1, mean=means2, cov=covs2)
Q2 = multivariate_normal.pdf(data1, mean=means3, cov=covs3)

print(entropy(P, P))
print(entropy(P, Q1))
print(entropy(P, Q2))
```

    0.0
    0.17877316564929496
    6.628549732040807

This time around, $P$ is the density function associated with Model 2 and $Q1$ is the density function associated with Model 1 and $Q2$ with Model 3.

```python
P = multivariate_normal.pdf(data2, mean=means2, cov=covs2)
Q1 = multivariate_normal.pdf(data2, mean=means1, cov=covs1)
Q2 = multivariate_normal.pdf(data2, mean=means3, cov=covs3)

print(entropy(P, P))
print(entropy(P, Q1))
print(entropy(P, Q2))
```

    0.0
    0.18572628345083425
    5.251771615080081

Finally, $P$ is the density function associated with Model 3 and $Q1$ is the density function associated with Model 1 and $Q2$ with Model 2.

```python
P = multivariate_normal.pdf(data3, mean=means3, cov=covs3)
Q1 = multivariate_normal.pdf(data3, mean=means1, cov=covs1)
Q2 = multivariate_normal.pdf(data3, mean=means2, cov=covs2)

print(entropy(P, P))
print(entropy(P, Q1))
print(entropy(P, Q2))
```

    0.0
    4.964071493531684
    4.154646297473461

Since Models 1 and 2 are very similar (as can be seen by how we constructed them), their KLD is closer to zero. On the other hand the KLDs between these two models and Model 3 are farther from zero. Though, it is interesting to note, that Model 2 is closer to Model 3 than Model 1 is to Model 3.


# Linear regression fitting noise versus structure

Let
$$
y = y^\ast + e, \quad y^\ast = X \theta^\ast
$$
where $X$ is a fat matrix. Consider optimizing the least squares objective
$$
\frac{1}{2}
\| X \theta - y \|^2.
$$
The gradient is
$$
\nabla f(\theta) = X^T X \theta - X^T y.
$$
We have
$$
\begin{align}
\theta_{k+1} - \theta^\ast
&=
\theta_k - \alpha(X^T X \theta - X^T y) - \theta^\ast \\
&=
\theta_k - \alpha(X^T X \theta - X^T X \theta^\ast - X^T e) - \theta^\ast \\
&=
(I - \alpha X^T X) (\theta_k - \theta^\ast) + \alpha X^T e
\end{align}
$$
The difference in terms of residual therefore becomes
$$
X\theta_{k+1} - X\theta^\ast
=
(I - \alpha XX^T) (X\theta_k - X\theta^\ast) + \alpha X X^T e \\
$$

```python
import matplotlib.pyplot as plt
#%matplotlib notebook
#import matplotlib.pyplot as plt
from numpy import *
import numpy as np
```

```python
def gradient_descent(A,b,niter = 10000,ytarget=None,stepsize=None):

    def f(x):
        return linalg.norm(dot(A,x) - b)**2

    def gradf(x):
        return 0.5*(dot(Q,x) - dot(b,A))

    Q = dot(A.T,A)
    eigenvalues = linalg.eigvals(Q)
    M = max(eigenvalues)
    m = min(eigenvalues)

    xopt = dot( linalg.inv( dot(A.T,A) ), dot( A.T , b ) )

    print("optimal errors: ", linalg.norm( y -  dot(A,xopt) ), linalg.norm( ytarget - dot(A,xopt) ) )

    if stepsize==None:
        stepsize = 2/(M+m)

    print("minimal and maximal eigenvalues: ", m, M)
    print("stepsize: ", stepsize)

    residuals = []
    gradients = []
    residual_target = []
    distances = []
    xk = zeros(n) #random.randn(n) # random initializer
    for k in range(niter):
        xk = xk - stepsize*gradf(xk)
        residuals.append( linalg.norm( y -  dot(A,xk) ) )
        gradients.append( linalg.norm(gradf(xk)) )
        residual_target.append( linalg.norm( ytarget - dot(A,xk) ) )
        distances.append( linalg.norm(xk) )
    return array(residuals), array(gradients), array(residual_target),array(distances)
```

```python
# generate a problem instance
n = 100
A = random.randn(n,n)
U,S,VT = linalg.svd(A)

ytarget = U[:, int(0)]
perturbation = random.randn(n) #U[:,n-10]
perturbation = perturbation/linalg.norm(perturbation)
#perturbation = U[:,n-10]

newS = np.array([s**3 for s in S])
newS = newS/np.max(newS)
plt.plot(newS)
plt.title("spectrum")
plt.show()

S = np.diag(newS)
A = U @ S @ VT

y = ytarget + perturbation

print(linalg.norm(y), linalg.norm(ytarget), linalg.norm(perturbation), dot(ytarget,perturbation)  )

steps = 1000
residuals,gradients,residual_target,distances = gradient_descent(A,y,niter=steps,ytarget=ytarget,stepsize=0.25)

print("logarithmic")
plt.plot( log(residuals) )
plt.show()
plt.plot( log(residual_target) )
plt.show()

print("non logarithmic")
plt.plot( residuals )
plt.show()
plt.plot( residual_target )
plt.show()

ks = np.array( [i for i in range(steps)] )
np.savetxt("ls_residuals.dat", np.vstack([ ks ,np.array(residuals),np.array(residual_target) ] ).T , delimiter="\t")

ns = np.array( [i for i in range(n)] )
np.savetxt("ls_spectrum.dat", np.vstack([ ns ,np.array(newS)] ).T , delimiter="\t")

```


# COURSE: A deep understanding of deep learning
## SECTION: Math prerequisites
### LECTURE: Derivatives: intuition and polynomials
#### TEACHER: Mike X Cohen, sincxpress.com
##### COURSE URL: udemy.com/course/dudl/?couponCode=202201

```python
import numpy as np
import matplotlib.pyplot as plt

# sympy = symbolic math in Python
import sympy as sym
import sympy.plotting.plot as symplot
```

```python
# create symbolic variables in sympy
x = sym.symbols('x')

# create a function
fx = 2*x**2

# compute its derivative
df = sym.diff(fx,x)

# print them
print(fx)
print(df)
```

```python
# plot them
symplot(fx,(x,-4,4),title='The function')
plt.show()

symplot(df,(x,-4,4),title='Its derivative')
plt.show()

```

```python
# repeat with relu and sigmoid

# create symbolic functions
relu = sym.Max(0,x)
sigmoid = 1 / (1+sym.exp(-x))

# graph the functions
p = symplot(relu,(x,-4,4),label='ReLU',show=False,line_color='blue')
p.extend( symplot(sigmoid,(x,-4,4),label='Sigmoid',show=False,line_color='red') )
p.legend = True
p.title = 'The functions'
p.show()

# graph their derivatives
p = symplot(sym.diff(relu),(x,-4,4),label='df(ReLU)',show=False,line_color='blue')
p.extend( symplot(sym.diff(sigmoid),(x,-4,4),label='df(Sigmoid)',show=False,line_color='red') )
p.legend = True
p.title = 'The derivatives'
p.show()

```


```python
%matplotlib inline

import numpy as np
import scipy as sc
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sympy as sp

import itertools

sns.set();
```

# Euler's method

Given a first-order explicit ODE (autonomous or non-autonomous) on $\vec{y} \in \mathbb{R}^{n}$ (It is often the case where $n = 1$, so we could drop the vector notation) in the general form

$$\frac{d}{dt}\vec{y}(t) = \vec{f}(t, y(t))$$

the Euler's method is a recursive method with the initial values given by the IVP's initial conditions. It attempts to approximate the solution by tracking its trajectory as ditacted by the ODE.

The approximate solution at time $t_{k+1} = t_{k} + h_{k}$ is given by the equation

$$\vec{y}_{k+1} = \vec{y}_{k} + h_{k} \vec{f}(t_{k}, \vec{y}_{k})$$

where

$$\vec{y}_{k}' = \vec{f}(t_{k}, \vec{y}_{k})$$

given the initial condition

$$\vec{y}_{0} = \vec{y}(0)$$

Note that it is a first-order ODE and it would be great if we could use Euler's method to solve any order ODE. In fact, we can. The way to do that is to use some auxiliary variables as follows:

$$
  \begin{cases}
    \vec{u}_{1} = \vec{y} \\
    \vec{u}_{2} = \vec{y}' \\
    \vdots \\
    \vec{u}_{m} = \vec{y}^{(m-1)}
  \end{cases}
$$

Then

$$
  \begin{cases}
    \vec{u}_{1}' = \vec{y}' = \vec{u}_{2}\\
    \vec{u}_{2}' = \vec{y}'' = \vec{u}_{3}\\
    \vdots \\
    \vec{u}_{m}' = \vec{y}^{(m)} = \vec{f}(t, u_{1}, u_{2}, \cdots, u_{m})
  \end{cases}
$$

And so, we have a system of $m$ fist-order ODE. We can solve it using any method, eg. Euler's method, for first-order ODE.

Below we write a simple still general generator function that yields at each time the tuple $(t_{k}, y_{k})$ given the initial conditions $(t_{0}, \vec{y}_{0})$, the function $\vec{f}$, the step size $h_{k}$ and the maximum number of iterations. Note that both $\vec{y}_{0}$ and $\vec{f}$ can be list-like data structures since they represent objects in $\mathbb{R}^{n}$.

```python
def euler(x_0, y_0, f, step=0.001, k_max=None):
    r"""
    Euler's method for solving first-order ODE.

    The function computes `k_max` iterations from the initial conditions `x_0` and `y_0` with
    steps of size `step`. It yields a total of `k_max` + 1 values. The recorrent equation is:

    y_{k+1} = y_{k} + h_{k} * f(x_{k}, y_{k})

    Parameters
    ----------
    x_0 : float
        The initial value for the independent variable.
    y_0 : array_like
        1-D array of initial values for the dependente variable evaluated at `x_0`.
    f : callable
        The function that represents the first derivative of y with respect to x.
        It must accept two arguments: the point x at which it will be evaluated and
        the value of y at this point.
    step : float, optional
        The size step between each iteration.
    k_max : number
        The maximum number of iterations.

    Yields
    ------
    x_k : float
        The point at which the function was evaluated in the last iteration.
    y_k : float
        The value of the function in the last iteration.
    """

    if k_max is None: counter = itertools.count()
    else: counter = range(k_max)
    x_k = x_0
    y_k = y_0
    yield (x_k, y_k)
    for k in counter:
        y_k = y_k + step * f(x_k, y_k)
        x_k = x_k + step
        yield (x_k, y_k)
```

```python
def extract(it):
    r"""
    Extract the values from a iterable of iterables.

    The function extracts the values from a iterable of iterables (eg. a list of tuples) to a list
    of coordinates. For example,

    [(1, 10), (2, 20), (3, 30), (4, 40)] -> [[1, 2, 3, 4], [10, 20, 30, 40]]

    If `it` is a list of M tuples each one with N elements, then `extract` returns
    a list of N lists each one with M elements.

    Parameters
    ----------
    it : iterable
        An iterable of iterables.

    Returns
    ------
    A list with the lists of first-elements, second-elements and so on.
    """

    return list(zip(*it))
```

## Example 1

The following example was taken from [Guidi], section 8.2, example 37.

The IVP is

$$y' = x^{2} + y^{2}$$

with initial conditions

$$y(0) = 0$$

By Euler's method

$$y_{k+1} = y_{k} + h_{k} f(t, y_{k})$$

Since $x_{k} = x_{0} + h_{k} k$ and $x_{0} = 0$, we have

$$x_{k} = h_{k} k$$

Remember that $y_{k}' = f(t, y_{k})$ and so

$$y_{k+1} = y_{k} + h_{k} (h_{k}^{2} k^{2} + y_{k}^{2})$$

```python
def example1(x_k, y_k):
    return x_k**2 + y_k**2

results = euler(x_0=0.0, y_0=0.0, f=example1, step=0.001, k_max=1000)

results = [(x, y) for k, (x, y) in enumerate(results) if k in range(0, 1001, 100)]
x, y_euler = extract(results)

df1 = pd.DataFrame({'x': x, 'euler': y_euler})

df1 = df1[['x', 'euler']]

df1.head(15)
```

      x
      euler

      0
      0.0
      0.000000

      1
      0.1
      0.000328

      2
      0.2
      0.002647

      3
      0.3
      0.008958

      4
      0.4
      0.021279

      5
      0.5
      0.041664

      6
      0.6
      0.072263

      7
      0.7
      0.115402

      8
      0.8
      0.173730

      9
      0.9
      0.250438

      10
      1.0
      0.349605

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df1['x'], df1['euler'], label='Euler approximation with step 0.001')
plt.legend(loc='upper left', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Solutions for the ODE", xlabel="Time", ylabel="y");
```

## Example 2

The following example was taken from [Heath], section 9.3.1, example 9.8.

The IVP is

$$y'(t) = y(t)$$

with initial conditions

$$t_0 = 0$$

Remembering Euler's method

$$y_{k+1} = y_{k} + h_{k} y_{k}$$

```python
def example2(x_k, y_k):
    return y_k

results = euler(x_0=0.0, y_0=1.0, f=example2, step=0.5, k_max=15)
x, y_euler = extract(results)

df2 = pd.DataFrame({'x': x, 'euler': y_euler, 'actual': np.exp(np.arange(0, 8, 0.5))})
df2 = df2[['x', 'euler', 'actual']]

df2.head(15)
```

      x
      euler
      actual

      0
      0.0
      1.000000
      1.000000

      1
      0.5
      1.500000
      1.648721

      2
      1.0
      2.250000
      2.718282

      3
      1.5
      3.375000
      4.481689

      4
      2.0
      5.062500
      7.389056

      5
      2.5
      7.593750
      12.182494

      6
      3.0
      11.390625
      20.085537

      7
      3.5
      17.085938
      33.115452

      8
      4.0
      25.628906
      54.598150

      9
      4.5
      38.443359
      90.017131

      10
      5.0
      57.665039
      148.413159

      11
      5.5
      86.497559
      244.691932

      12
      6.0
      129.746338
      403.428793

      13
      6.5
      194.619507
      665.141633

      14
      7.0
      291.929260
      1096.633158

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df2['x'], df2['euler'], label='Euler approximation with step 0.5')
plt.plot(df2['x'], df2['actual'], label='Actual values')
ax.legend(loc='upper left', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Solutions for the ODE", xlabel="Time", ylabel="y");
```

## Example 3

Consider the following ODE

$$y'' + y = 0$$

with initial conditions

$$y(0) = 1$$

and

$$y'(0) = 0$$

The exact solution for this IVP is $y(t) = \mathit{cos}(t)$.

Let's apply some transformations over this ODE to represent it as a system of first-order ODE.
$u_{1} = y$ e $u_{2} = y'$. Note that $u_{2}' = y''$, so
$y'' = f(x, y, y')$ becomes $u_{2}' = g(x, u_{1}, u_{2})$.

The previous ODE can now be written as

$u_{2}' = -u_{1}$ com $u_{1}(0) = 1$ e $u_{2}(0) = 0$.

```python
def example3(x_k, u_k):
    return np.array([u_k[1], -u_k[0]])

results = euler(x_0=0.0, y_0=np.array([1.0, 0.0]), f=example3, step=0.03, k_max=1000)
x, ys = extract(results)
y_euler, dy_euler = extract(ys)

df3 = pd.DataFrame({'x': x, 'euler': y_euler, 'dy_euler': dy_euler, 'actual': np.cos(x), 'dy_actual': -np.sin(x)})
df3 = df3[['x', 'euler', 'actual', 'dy_euler', 'dy_actual']]

df3.head(15)
```

      x
      euler
      actual
      dy_euler
      dy_actual

      0
      0.00
      1.000000
      1.000000
      0.000000
      -0.000000

      1
      0.03
      1.000000
      0.999550
      -0.030000
      -0.029996

      2
      0.06
      0.999100
      0.998201
      -0.060000
      -0.059964

      3
      0.09
      0.997300
      0.995953
      -0.089973
      -0.089879

      4
      0.12
      0.994601
      0.992809
      -0.119892
      -0.119712

      5
      0.15
      0.991004
      0.988771
      -0.149730
      -0.149438

      6
      0.18
      0.986512
      0.983844
      -0.179460
      -0.179030

      7
      0.21
      0.981128
      0.978031
      -0.209056
      -0.208460

      8
      0.24
      0.974857
      0.971338
      -0.238489
      -0.237703

      9
      0.27
      0.967702
      0.963771
      -0.267735
      -0.266731

      10
      0.30
      0.959670
      0.955336
      -0.296766
      -0.295520

      11
      0.33
      0.950767
      0.946042
      -0.325556
      -0.324043

      12
      0.36
      0.941000
      0.935897
      -0.354079
      -0.352274

      13
      0.39
      0.930378
      0.924909
      -0.382309
      -0.380188

      14
      0.42
      0.918909
      0.913089
      -0.410221
      -0.407760

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df3['x'], df3['euler'], label='Euler approximation with step 0.01')
plt.plot(df3['x'], df3['actual'], label='Actual values')
ax.legend(loc='lower right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Solutions for the ODE", xlabel="Time", ylabel="y");
```

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df3['euler'], df3['dy_euler'], label="Euler's solution")
plt.plot(df3['actual'], df3['dy_actual'], label='Exact solution')
ax.legend(loc='upper right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title='Phase space', xlabel='y', ylabel='dy')
ax.axis('equal');
```

## Example 4: simple pendulum

The simple pendulum system can be represented as ODE as follows:

$$\theta'' + \frac{g}{l}\mathit{sin}(\theta) = 0$$

where $\theta$ is the angle in radians between the vertical line passing through the joint and the wire holding the mass, $g$ is the gravity acceleration and $l$ is the wire length.

We can turn it into a IVP by adding the following equations:

$$\theta(0) = \theta_{0}$$
and
$$\theta'(0) = 0$$

The exact solution for this ODE is

$$\theta(t) = \theta_0 \cos\left({\sqrt{\frac{g}{l}}}\,t\right)$$

whose first derivative relative to $t$ is

$$\theta'(t) = -\theta_{0} \sqrt{\frac{g}{l}} \sin\left({\sqrt{\frac{g}{l}}}\,t\right)$$

Since the IVP has a second-order ODE, we can apply the same transformations we did above to turn it into a system of first-order ODE. Let's call $u_{1}(t) = y(t) = \theta(t)$ and $u_{2}(t) = y'(t) = \theta'(t)$.

This way we end up with a system of first-order ODE that can be solved by using Euler's method or any other first-order method. Now the Euler's method equation becomes:

$$\vec{u}^{(k+1)} = \vec{u}^{(k)} + h_{k} f(t, \vec{u}^{k})$$

where $\vec{u} = [u_{1}, u_{2}]^{T}$.

```python
g = 9.8
l = 10

def example4(t_k, u_k):
    return np.array([u_k[1], -(g/l) * np.sin(u_k[0])])

theta_0 = 0.087 # About 5 degrees.
results = euler(x_0=0.0, y_0=np.array([theta_0, 0.0]), f=example4, step=0.02, k_max=1000)
x, ys = extract(results)
y_euler, dy_euler = extract(ys)

import math
df4 = pd.DataFrame({'x': x, 'euler': y_euler, 'dy_euler': dy_euler,
                       'actual': (theta_0 * np.cos(math.sqrt(g/l)*np.array(x))),
                       'dy_actual': (-theta_0 * math.sqrt(g/l) * np.sin(math.sqrt(g/l)*np.array(x)))})

df4 = df4[['x', 'euler', 'actual', 'dy_euler', 'dy_actual']]

df4.head(15)
```

      x
      euler
      actual
      dy_euler
      dy_actual

      0
      0.00
      0.087000
      0.087000
      0.000000
      -0.000000

      1
      0.02
      0.087000
      0.086983
      -0.001703
      -0.001705

      2
      0.04
      0.086966
      0.086932
      -0.003406
      -0.003410

      3
      0.06
      0.086898
      0.086847
      -0.005108
      -0.005113

      4
      0.08
      0.086796
      0.086727
      -0.006810
      -0.006814

      5
      0.10
      0.086659
      0.086574
      -0.008509
      -0.008512

      6
      0.12
      0.086489
      0.086387
      -0.010205
      -0.010207

      7
      0.14
      0.086285
      0.086166
      -0.011898
      -0.011898

      8
      0.16
      0.086047
      0.085911
      -0.013587
      -0.013585

      9
      0.18
      0.085775
      0.085622
      -0.015272
      -0.015266

      10
      0.20
      0.085470
      0.085300
      -0.016951
      -0.016941

      11
      0.22
      0.085131
      0.084945
      -0.018624
      -0.018609

      12
      0.24
      0.084759
      0.084556
      -0.020290
      -0.020270

      13
      0.26
      0.084353
      0.084134
      -0.021950
      -0.021924

      14
      0.28
      0.083914
      0.083679
      -0.023601
      -0.023568

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df4['x'], df4['euler'], label='Euler approximation with step 0.01')
plt.plot(df4['x'], df4['actual'], label='Actual values')
ax.legend(loc='lower right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Solutions for the ODE", xlabel="Time", ylabel=r"$\theta$");
```

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df4['euler'], df4['dy_euler'], label="Euler's solution")
plt.plot(df4['actual'], df4['dy_actual'], label="Exact solution")
ax.legend(loc='upper right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Phase space", xlabel='y', ylabel="dy")
ax.axis('equal');
```

## Errors

There are two types of errors when solving an ODE numerically. The first is the _rounding error_ which is due to the precision of floating-point arithmetic. The second type is _truncation error_ which is due to the method used.

Truncation error can be of two different types: global error and local error.

The global truncation error in step $k$, $e_{k}$, is the difference between the computed solution through the method, $y_{k}$, and the true solution of the ODE, $y(t_{k})$:

$$\vec{e}_{k} = \vec{y}_{k} - \vec{y}(t_{k})$$

The local truncation error in step $k$, $\tau_{k}$, is the difference between the approximate and exact increment per unit step:

$$\vec{\tau}_{k} = \frac{\vec{y}(t_{k}) - \vec{y}(t_{k-1}) - f(t, \vec{y}_{k})}{h}$$

We follow with the pendulum example to demonstrate error computations in practice.

We know that  the true solution for the simple pendulum IVP

$$\theta'' + \frac{g}{l}\mathit{sin}(\theta) = 0$$

with initial values

$$\theta(0) = \theta_{0}$$
and
$$\theta'(0) = 0$$

is

$$\theta(t) = \theta_0 \cos\left({\sqrt{\frac{g}{l}}}\,t\right)$$

whose first derivative relative to $t$ is

$$\theta'(t) = -\theta_{0} \sqrt{\frac{g}{l}} \sin\left({\sqrt{\frac{g}{l}}}\,t\right)$$

Then we know the true solution $\theta$ at each time $t_{k}$ and its first derivative. We can proceed and estimate the global and local errors and see how they behave.

```python
df4['global_err'] = df4['actual'] - df4['euler']
df4.loc[0, 'local_err'] = 0.0
df4.loc[1:, 'local_err'] = (df4['actual'].diff() - df4['euler'].diff())

df4.head(15)
```

      x
      euler
      actual
      dy_euler
      dy_actual
      global_err
      local_err

      0
      0.00
      0.087000
      0.087000
      0.000000
      -0.000000
      0.000000
      0.000000

      1
      0.02
      0.087000
      0.086983
      -0.001703
      -0.001705
      -0.000017
      -0.000017

      2
      0.04
      0.086966
      0.086932
      -0.003406
      -0.003410
      -0.000034
      -0.000017

      3
      0.06
      0.086898
      0.086847
      -0.005108
      -0.005113
      -0.000051
      -0.000017

      4
      0.08
      0.086796
      0.086727
      -0.006810
      -0.006814
      -0.000068
      -0.000017

      5
      0.10
      0.086659
      0.086574
      -0.008509
      -0.008512
      -0.000085
      -0.000017

      6
      0.12
      0.086489
      0.086387
      -0.010205
      -0.010207
      -0.000102
      -0.000017

      7
      0.14
      0.086285
      0.086166
      -0.011898
      -0.011898
      -0.000119
      -0.000017

      8
      0.16
      0.086047
      0.085911
      -0.013587
      -0.013585
      -0.000136
      -0.000017

      9
      0.18
      0.085775
      0.085622
      -0.015272
      -0.015266
      -0.000153
      -0.000017

      10
      0.20
      0.085470
      0.085300
      -0.016951
      -0.016941
      -0.000170
      -0.000017

      11
      0.22
      0.085131
      0.084945
      -0.018624
      -0.018609
      -0.000186
      -0.000016

      12
      0.24
      0.084759
      0.084556
      -0.020290
      -0.020270
      -0.000203
      -0.000016

      13
      0.26
      0.084353
      0.084134
      -0.021950
      -0.021924
      -0.000219
      -0.000016

      14
      0.28
      0.083914
      0.083679
      -0.023601
      -0.023568
      -0.000235
      -0.000016

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df4['x'], df4['global_err'], label="Global truncation error")
plt.plot(df4['x'], df4['local_err'], label="Local truncation error")
ax.legend(loc='upper left', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Truncation errors", xlabel="Time", ylabel="Error");
```

As we can see, the global error grows indefinitely although the local errors remain near zero. This seems to be the case where the global error is the sum of the local errors. However, it is important to note that this is not always the case: The global error may be greater or smaller than the sum of local errors. In general, if the solutions of the ODE are diverging, then the global error is greater than the sum of local errors. On the other hand, if the solutions of the ODE are converging, then the global error is smaller than the sum of local errors.

```python
np.allclose(df4['global_err'], df4['local_err'].cumsum()) # Equality for floats.
```

    True

## References

* Guidi, L., Notas da disciplina Cálculo Numérico. Disponível em [Notas da disciplina Cálculo Numérico](http://www.mat.ufrgs.br/~guidi/grad/MAT01169/calculo_numerico.pdf)
* Heath, M. T., Scientific Computing: An Introductory Survey, 2nd Edition, McGraw Hill, 2002.