# Baseball problem: Solution
Copyright © 2018 Oliver Beckstein.

This notebook is based on `11-ODE-applications.ipynb`. The whole problem was live-coded from scratch during class and this notebook is a cleaned-up and commented solution. The lesson showed how
1. to define the physical problem (obtain trajectory of a baseball with spin and air resistance)
2. to derive the underlying equations (Newton's equations of motions and the forces acting on the ball)
3. to adapt an algorithm to solve the equations (use RK4 for integrating the ODEs and express the problem in ODE standard form so that one can use `ode.rk4()`)
4. to visualize and discuss the results (plot trajectories and assess the influence of spin)

## Packages
We use numpy and our ODE solvers (`ode.py`):

```python
import numpy as np
import ode
```

For 3D graphs you can use
```
%matplotlib notebook
```
but you have to do this as the very first thing (before loading `matplotlib`). You will also have to "close" active figures before you can plot into a new one.

```python
%matplotlib inline
#%matplotlib notebook
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

## Simple Baseball physics

- quadratic air resistance (with velocity-dependent drag coefficient)
- Magnus force due to spin

### Quadratic air resistance
Occurs at high Reynolds numbers, i.e., turbulent flow. Only approximate:

$$
\mathbf{F}_2 = -b_2 v \mathbf{v}
$$

with
$$
b_2 = \frac{1}{2} C_D \rho A
$$
where $C_D$ is the drag coefficient, $\rho$ the density of air, and $A = r^2 \pi$ the cross sectional area of the baseball.

### Magnus effect

**Magnus effect**: airflow is changed around a spinning object. The Magnus force is

$$
\mathbf{F}_M = \alpha \boldsymbol{\omega} \times \mathbf{v}
$$

where $\boldsymbol{\omega}$ is the ball's angular velocity in rad/s (e.g., 200/s for a baseball).

For a sphere the proportionality constant $\alpha$ can be written

$$
\mathbf{F}_M = \frac{1}{2} C_L \rho A \frac{v}{\omega} \boldsymbol{\omega} \times \mathbf{v}
$$

where $C_L$ is the lift coefficient, $\rho$ the air density, $A$ the ball's cross section. (Advantage of defining $C_L$ this way: when spin and velocity are perpendicular, the Magnus force is simply $F_M = \frac{1}{2} C_L \rho A v^2$.)

$C_L$ is mainly a function of the *spin parameter*

$$
S = \frac{r\omega}{v}
$$

with the radius $r$ of the ball. In general we write

$$
\mathbf{F}_M = \frac{1}{2} C_L(S)  \frac{\rho A r}{S} \boldsymbol{\omega} \times \mathbf{v}
$$

For a baseball, experimental data show approximately a power law dependence of $C_L$ on $S$

$$
C_L(S) = 0.62 \times S^{0.7}
$$

All together:

\begin{align}
\mathbf{F}_M &= \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
v &= \sqrt{\mathbf{v}\cdot\mathbf{v}}\\
S &= \frac{r\omega}{v}\\
C_L &= 0.62 \times S^{0.7}\\
\alpha &= \frac{1}{2} C_L  \frac{\rho A r}{S}
\end{align}

### Equations of motion

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= -g \hat{\mathbf{e}}_y -\frac{b_2}{m} v \mathbf{v} + \alpha\ \boldsymbol{\omega} \times \mathbf{v}
\end{align}

(quadratic drag $-\frac{b_2}{m} v \mathbf{v}$ included.)

### Baseball simulation

Implement the full baseball equations of motions:
- gravity $a_\text{gravity}$
- quadratic drag $a_\text{drag}$
- Magnus effect $a_\text{Magnus}$

For the cross product you can look at [numpy.cross()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cross.html).

```python
def C_L(S):
    return 0.62 * S**0.7

def simulate_baseball(v0, omega, r0=None,
                      h=0.01, C_D=0.40, g=9.81, rho=1.225,
                      r=0.07468/2, m=0.14883,
                      R_homeplate=18.4):
    """simulate baseball pitch

    Parameters
    ----------
    v0 : array
         initial velocity (vx, vy) in m/s
    omega : array
         angular velocity vector of the ball ("spin"), in rad/s
    r0 : array, optional
         initial position of the ball (in m) when it leaves the pitcher's hand
         as (x, y); the default is (0, 2)
    h : float, optional
         integration time step in s, default is 0.01 s
    C_D : float, optional
         drag coefficient, default is 0.40
    g : float, optional
         acceleration due to gravity, default 9.81 kg/(m*s^2)
    rho : float, optional
         density of air, default 1.225 kg/m^3
    r : float, optional
         radius of the baseball
    m : float, optional
         mass of the baseball
    R_homeplate : float, optional
         distance of the catcher from the pitcher

    Returns
    -------

    positions : array
         The array contains an entry (time, x, y, z) for each time step.
    """
    # all SI units (kg, m)
    if r0 is None:
        r0 = np.array([0, 2])  # pitching at 2m height

    omega = np.asarray(omega)

    domega = np.linalg.norm(omega)
    A = np.pi*r**2
    rhoArm = rho * A * r / m
    b2 = 0.5 * C_D * rho * A

    a_gravity = np.array([0, -g, 0])

    def f(t, y):
        # y = [x, y, z, vx, vy, vz]
        v = y[3:]
        dv = np.linalg.norm(v)
        S = r*domega/dv
        a_magnus = 0.5 * C_L(S) * rhoArm / S * np.cross(omega, v)
        a_drag = -b2/m * dv * v
        a = a_gravity + a_drag + a_magnus
        return np.array([y[3], y[4], y[5],
                         a[0], a[1], a[2]])

    x0, y0 = r0
    vx, vy = v0
    t = 0
    positions = []
    # initialize 3D!
    y = np.array([x0, y0, 0, vx, vy, 0], dtype=np.float64)

    while y[0] < R_homeplate and y[1] >= 0.2:
        positions.append([t, y[0], y[1], y[2]])  # record t, x and y, z
        y[:] = ode.rk4(y, f, t, h)
        t += h

    return np.array(positions)

```

#### Simulate throws

Simulate baseball throw for initial velocity $\mathbf{v} = (30\,\text{m/s}, 0)$.

Plot x vs y and x vs z (to see curving).

Try out different spins; a good value is $\boldsymbol{\omega} = 200\,\text{rad/s} \times (0, 1, 1)$.

200 rad/s is 32 revolutions per second or almost 2000 rpm.

```python
print(200/(2*np.pi), "revolutions per second")
print(200/(2*np.pi) * 60, "rpm")
```

    31.830988618379067 revolutions per second
    1909.859317102744 rpm

Simulate the baseball throw with
- almost no spin: $\omega = 0.001 \times (0, 0, 1)$  (our code does not handle $\omega = 0$ gracefully...)
- full upward spin: $\omega = 200 \times (0, 0, 1)$
- sideways spin: $\omega = 200 \times (0, 1, 1)$

```python
r = simulate_baseball([30, 0], omega=0.001*np.array([0,0,1]))
rz = simulate_baseball([30, 0], omega=200.*np.array([0,0,1]))
rzy = simulate_baseball([30, 0], omega=200.*np.array([0,1,1]))
```

#### Plotting

Plot the three scenarios in 2D planes: x-y (side view) and x-z (top view).

```python
idx = 2  # y
plt.plot(r[:,1], r[:,idx], label="no spin")
plt.plot(rz[:,1], rz[:,idx], label="upward spin")
plt.plot(rzy[:,1], rzy[:,idx], label="sideways spin")
plt.xlabel("$x$ (m)")
plt.ylabel("$y$ (m)")
plt.legend(loc="best")
plt.title("Side view");
```

Note how the balls with spin fly higher: the Magnus force lifts them up (due to the component of the spin along $z$).

```python
idx = 3  # y
plt.plot(r[:,1], r[:,idx], label="no spin")
plt.plot(rz[:,1], rz[:,idx], label="upward spin")
plt.plot(rzy[:,1], rzy[:,idx], label="sideways spin")
plt.xlabel("$x$ (m)")
plt.ylabel("$z$ (m)")
plt.legend(loc="best")
plt.title("Top-down view");
```

The ball with a component of the spin along the $y$ direction ($\boldsymbol{\omega} = \omega_0 \, \left(\begin{array}{c} 0\\1\\1\end{array}\right)$) has a 0.6 m deflection in the $-z$ direction.

#### 3D plot
Use simple `matplotlib` 3D plot. (BONUS: Make it work with vpython)

If we use the [`notebook` backend for matplotlib](http://ipython.readthedocs.io/en/stable/interactive/plotting.html) then we will be able to interactively rotate our [matplotlib 3D graphics](http://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html). (Note: If this does not seem to work, disable adblockers and allow javascript on the page.)

```python
%matplotlib notebook
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(r[:,1], r[:,3], r[:,2], '.', label="no spin")
ax.plot(rz[:,1], rz[:,3], rz[:,2], '.', label="up spin")
ax.plot(rzy[:,1], rzy[:,3], rzy[:,2], '.', label="sideways spin")

# hand of the catcher, 0.2m above homeplate
ax.plot([18.4, 18.4], [0, 0], [0, 0.2], color="black", lw=6)

# drop line for last point
x,y,z = rzy[-1, 1:]
ax.plot([x,x], [z,z], [y,0], color="red")

ax.set_xlabel("$x$ (m)")
ax.set_ylabel("$z$ (m)")
ax.set_zlabel("$y$ (m)")
ax.legend(loc="upper left", numpoints=1)
ax.figure.tight_layout()
fig.savefig("baseball.png")
```

In the plot above we also added the hand of the catcher as a heavy black bar at 0.2 m above the home plate, and a drop-down bar in red to indicate the height of the ball thrown with *sideways spin* to make cleared that it also arrives at almost the same height as the ball with *up spin*.

```python

```


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *
Image(url= "https://i.imgur.com/NSkkQVE.png")
```

```python
L  = Function("L")
f = Function("f")
x,a = symbols('x a')
L(x)
```

$\displaystyle L{\left(x \right)}$

```python
Eq(L(x) , f(a) + Derivative(f(a),a)*(x-a))
```

$\displaystyle L{\left(x \right)} = \left(- a + x\right) \frac{d}{d a} f{\left(a \right)} + f{\left(a \right)}$

```python
expr = 1/x
x1 = 0.5
Eq(f(x), expr)
```

$\displaystyle f{\left(x \right)} = \frac{1}{x}$

```python
eq1 = Eq(f(x), expr).subs(x,x1)
eq1
```

$\displaystyle f{\left(0.5 \right)} = 2.0$

```python
fa = eq1.rhs
def f(x):
    return expr
df = diff(f(x),x)
eq2 = Eq(Derivative(f(x)),df).subs(x,x1)
eq2
```

$\displaystyle \left. \frac{d}{d x} \frac{1}{x} \right|_{\substack{ x=0.5 }} = -4.0$

```python
dfa= eq2.rhs
x = .503
Eq(L(x) , fa+ dfa*(x-x1)).evalf()
```

$\displaystyle L{\left(0.503 \right)} = 1.988$

```python
print(Eq(L(x) , fa+ dfa*(x-x1)).evalf().rhs)
```

    1.98800000000000

```python
Image(url= "https://i.imgur.com/zWZp8ZD.png")
```


```python
# Chain rule

In calculus, the chain rule is a formula to compute the derivative of a composite function.[Chain rule](https://en.wikipedia.org/wiki/Chain_rule).
```

$f ∘ g$ — the function which maps x to ${\displaystyle f(g(x))}f(g(x))$ — in terms of the derivatives of f and g and the product of functions as follows:

$(f\circ g)'=(f'\circ g)\cdot g'.$

 in Lagrange's notation, as follows:

$F'(x)=f'(g(x))g'(x).$

in Leibniz's notation

${\frac  {dz}{dx}}={\frac  {dz}{dy}}\cdot {\frac  {dy}{dx}}.$

${\displaystyle \left.{\frac {dz}{dx}}\right|_{x}=\left.{\frac {dz}{dy}}\right|_{y(x)}\cdot \left.{\frac {dy}{dx}}\right|_{x}}.$

## One dimension

height above sea level in meters: g(t) = 4000 − 4.9t**2
atmospheric pressure at a height h is: f(h) = 101325 e−0.0001h

g′(t) = −9.8t
f′(h) = −10.1325e−0.0001h
(f ∘ g)(t)
(f ∘ g)′(t)

$(f \circ g)'(t) = f'(g(t))\cdot g'(t).$
$(f \circ g)'(t) = \big(\mathord{-}10.1325e^{-0.0001(4000 - 4.9t^2)}\big)\cdot\big(\mathord{-}9.8t\big).$

```python

import math
from sympy import *
import pint
def cal_units():
    ureg = pint.UnitRegistry()
    return ureg.pascal*(ureg['meter/second**2'] * ureg.second**2)**-1*ureg.meter/ureg.second**2*ureg.second
def get_change_rate( seconds):
    # (f \circ g)'(t) = f'(g(t))\cdot g'(t).
    if cal_units().units=='pascal / second':
        t, e = symbols('t e')
        g_t='1/2*g*t**2'#.5*g*((t+h)**2-t**2)/h, .5*g*2t=g*t
        h='4000-{0}'.format(g_t)
        f_prime_h = '-10.1325*e**(-0.0001*({0}))'.format(h)
        g_t_prime='g*t'
        f_g_prime_t = '{0}*(-{1})'.format(f_prime_h, g_t_prime)
        print(f_g_prime_t)
        change_rate = sympify(f_g_prime_t).evalf(subs={'t': seconds, 'e': math.e,'g':9.8})
        return change_rate
!python3 "tests/calculus/test_units.py"
print("completed unit test inside the notebook")
```

```python

```

```python

import numpy as np
from bqplot import pyplot as plt

plt.figure(1, title='Line Chart')
np.random.seed(0)
n = 200
x = np.linspace(0.0, 10.0, n)
y = np.cumsum(np.random.randn(n))
plt.plot(x, y)
plt.show()
```

```python
def add(a, b):
    return a + b
```

```python
import unittest

class TestNotebook(unittest.TestCase):

    def test_add(self):
        self.assertEqual(add(2, 2), )

unittest.main(argv=[''], verbosity=2, exit=False)
```

```python

```


# Rigid-body transformations in a plane (2D)

> Marcos Duarte, Renato Naville Watanabe
> [Laboratory of Biomechanics and Motor Control](http://pesquisa.ufabc.edu.br/bmclab)
> Federal University of ABC, Brazil

Contents

1  Affine transformations

2  Kinematics of a rigid body

3  Python setup

4  Translation

5  Rotation

5.1  Using trigonometry

5.2  Using direction cosines

5.3  3. Using a basis

5.4  Local-to-Global and Global-to-local coordinate systems' rotations

5.5  The orthogonality of the rotation matrix

5.6  Rotation of a Vector

5.7  Calculation of rotation matrix using  a basis

5.8  Determination of the unknown angle of rotation

5.9  Joint angle as a sequence of rotations of adjacent segments

5.10  Kinematic chain in a plain (2D)

6  Translation and rotation

6.1  Transformation matrix

7  Further reading

8  Video lectures on the Internet

9  Problems

10  References

## Affine transformations

Translation and rotation are two examples of [affine transformations](https://en.wikipedia.org/wiki/Affine_transformation). Affine transformations preserve straight lines, but not necessarily the distance between points. Other examples of affine transformations are scaling, shear, and reflection. The figure below illustrates different affine transformations in a plane. Note that a 3x3 matrix is shown on top of each transformation; these matrices are known as the transformation matrices and are the mathematical representation of the physical transformations. Next, we will study how to use this approach to describe the translation and rotation of a rigid-body.

Figure. Examples of affine transformations in a plane applied to a square (with the letter F in it) and the corresponding transformation matrices (image from Wikipedia).

## Kinematics of a rigid body

The kinematics of a rigid body is completely described by its pose, i.e., its position and orientation in space (and the corresponding changes are translation and rotation). The translation and rotation of a rigid body are also known as rigid-body transformations (or simply, rigid transformations).

Remember that in physics, a [rigid body](https://en.wikipedia.org/wiki/Rigid_body) is a model (an idealization) for a body in which deformation is neglected, i.e., the distance between every pair of points in the body is considered constant. Consequently, the position and orientation of a rigid body can be completely described by a corresponding coordinate system attached to it. For instance, two (or more) coordinate systems can be used to represent the same rigid body at two (or more) instants or two (or more) rigid bodies in space.

Rigid-body transformations are used in motion analysis (e.g., of the human body) to describe the position and orientation of each segment (using a local (anatomical) coordinate system defined for each segment) in relation to a global coordinate system fixed at the laboratory. Furthermore, one can define an additional coordinate system called technical coordinate system also fixed at the rigid body but not based on anatomical landmarks. In this case, the position of the technical markers is first described in the laboratory coordinate system, and then the technical coordinate system is calculated to recreate the anatomical landmarks position in order to finally calculate the original anatomical coordinate system (and obtain its unknown position and orientation through time).

In what follows, we will study rigid-body transformations by looking at the transformations between two coordinate systems. For simplicity, let's first analyze planar (two-dimensional) rigid-body transformations and later we will extend these concepts to three dimensions (where the study of rotations are more complicated).

## Python setup

```python
import numpy as np
np.set_printoptions(precision=4, suppress=True)
```

## Translation

In a two-dimensional space, two coordinates and one angle are sufficient to describe the pose of the rigid body, totalizing three degrees of freedom for a rigid body. Let's see first the transformation for translation, then for rotation, and combine them at last.

A pure two-dimensional translation of a coordinate system in relation to other coordinate system and the representation of a point in these two coordinate systems are illustrated in the figure below (remember that this is equivalent to describing a translation between two rigid bodies).

Figure. A point in two-dimensional space represented in two coordinate systems (Global and local), with one system translated.

The position of point $\mathbf{P}$ originally described in the local coordinate system but now described in the Global coordinate system in vector form is:

\begin{equation}
\mathbf{P_G} = \mathbf{L_G} + \mathbf{P_l}
\end{equation}

Or for each component:

\begin{equation}
\mathbf{P_X} = \mathbf{L_X} + \mathbf{P}_x \\
\mathbf{P_Y} = \mathbf{L_Y} + \mathbf{P}_y
\end{equation}

And in matrix form is:

\begin{equation}
\begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y}
\end{bmatrix} =
\begin{bmatrix}
\mathbf{L_X} \\
\mathbf{L_Y}
\end{bmatrix} +
\begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y
\end{bmatrix}
\end{equation}

Because position and translation can be treated as vectors, the inverse operation, to describe the position at the local coordinate system in terms of the Global coordinate system, is simply:

\begin{equation}
\mathbf{P_l} = \mathbf{P_G} -\mathbf{L_G}
\end{equation}

\begin{equation}
\begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y
\end{bmatrix} =
\begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y}
\end{bmatrix} -
\begin{bmatrix}
\mathbf{L_X} \\
\mathbf{L_Y}
\end{bmatrix}
\end{equation}

From classical mechanics, this transformation is an example of [Galilean transformation](http://en.wikipedia.org/wiki/Galilean_transformation).

For example, if the local coordinate system is translated by $\mathbf{L_G}=[2, 3]$ in relation to the Global coordinate system, a point with coordinates $\mathbf{P_l}=[4, 5]$ at the local coordinate system will have the position $\mathbf{P_G}=[6, 8]$ at the Global coordinate system:

```python
LG = np.array([2, 3])  # (Numpy 1D array with 2 elements)
Pl = np.array([4, 5])
PG = LG + Pl
PG
```

    array([6, 8])

This operation also works if we have more than one data point (NumPy knows how to handle vectors with different dimensions):

```python
Pl = np.array([[4, 5], [6, 7], [8, 9]])  # 2D array with 3 rows and two columns
PG = LG + Pl
PG
```

    array([[ 6,  8],
           [ 8, 10],
           [10, 12]])

## Rotation

A pure two-dimensional rotation of a coordinate system in relation to other coordinate system and the representation of a point in these two coordinate systems are illustrated in the figure below (remember that this is equivalent to describing a rotation between two rigid bodies). The rotation is around an axis orthogonal to this page, not shown in the figure (for a three-dimensional coordinate system the rotation would be around the $\mathbf{Z}$ axis).

Figure. A point in the two-dimensional space represented in two coordinate systems (Global and local), with one system rotated in relation to the other around an axis orthogonal to both coordinate systems.

Consider we want to express the position of point $\mathbf{P}$ in the Global coordinate system in terms of the local coordinate system knowing only the coordinates at the local coordinate system and the angle of rotation between the two coordinate systems.

There are different ways of deducing that, we will see three of these methods next.

### Using trigonometry

From figure below, the coordinates of point $\mathbf{P}$ in the Global coordinate system can be determined finding the sides of the triangles marked in red.

Figure. The coordinates of a point at the Global coordinate system in terms of the coordinates of this point at the local coordinate system.

Then:

\begin{equation}
\mathbf{P_X} = \mathbf{P}_x \cos \alpha - \mathbf{P}_y \sin \alpha \\
\\
\mathbf{P_Y} = \mathbf{P}_x \sin \alpha + \mathbf{P}_y \cos \alpha
\end{equation}

The equations above can be expressed in matrix form:

\begin{equation}
\begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y}
\end{bmatrix} =
\begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix} \begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y
\end{bmatrix}
\end{equation}

Or simply:

\begin{equation}
\mathbf{P_G} = \mathbf{R_{Gl}}\mathbf{P_l}
\end{equation}

Where $\mathbf{R_{Gl}}$ is the rotation matrix that rotates the coordinates from the local to the Global coordinate system:

\begin{equation}
\mathbf{R_{Gl}} = \begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix}
\end{equation}

So, given any position at the local coordinate system, with the rotation matrix above we are able to determine the position at the Global coordinate system. Let's check that before looking at other methods to obtain this matrix.

For instance, consider a local coordinate system rotated by $45^o$ in relation to the Global coordinate system, a point in the local coordinate system with position $\mathbf{P_l}=[1, 1]$ will have the following position at the Global coordinate system:

```python
α = np.pi / 4
RGl = np.array([[np.cos(α), -np.sin(α)],
                [np.sin(α),  np.cos(α)]])
Pl = np.array([[1, 1]]).T  # transpose the array for correct matrix multiplication
PG = RGl @ Pl  # the operator @ is used for matrix multiplication of arrays
PG
```

    array([[0.    ],
           [1.4142]])

And if we have the points [1,1], [0,1], [1,0] at the local coordinate system, their positions at the Global coordinate system are:

```python
Pl = np.array([[1, 1], [0, 1], [1, 0]]).T  # transpose array for matrix multiplication
PG = RGl@Pl
PG
```

    array([[ 0.    , -0.7071,  0.7071],
           [ 1.4142,  0.7071,  0.7071]])

### Using direction cosines

Another way to determine the rotation matrix is to use the concept of direction cosine.

> Direction cosines are the cosines of the angles between any two vectors.

For the present case with two coordinate systems, they are  the cosines of the angles between each axis of one coordinate system and each axis of the other coordinate system. The figure below illustrates the directions angles between the two coordinate systems, expressing the local coordinate system in terms of the Global coordinate system.

Figure. Definition of direction angles at the two-dimensional space.

\begin{equation}
\mathbf{R_{Gl}} = \begin{bmatrix}
\cos\mathbf{X}x & \cos\mathbf{X}y \\
\cos\mathbf{Y}x & \cos\mathbf{Y}y
\end{bmatrix} =
\begin{bmatrix}
\cos(\alpha) & \cos(90^o+\alpha) \\
\cos(90^o-\alpha) & \cos(\alpha)
\end{bmatrix} =
\begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix}
\end{equation}

The same rotation matrix as obtained before.

Note that the order of the direction cosines is because in our convention, the first row is for the $\mathbf{X}$ coordinate and the second row for the $\mathbf{Y}$ coordinate (the outputs). For the inputs, we followed the same order, first column for the $\mathbf{x}$ coordinate, second column for the $\mathbf{y}$ coordinate.

### 3. Using a basis

Yet another way to deduce the rotation matrix is to view the axes of the rotated coordinate system as unit vectors, versors, of a basis as illustrated in the figure below.

> A basis is a set of linearly independent vectors that can represent every vector in a given vector space, i.e., a basis defines a coordinate system.

Figure. Definition of the rotation matrix using a basis at the two-dimensional space.

The coordinates of these two versors at the local coordinate system in terms of the Global coordinate system are:

\begin{equation}
\begin{array}{l l}
\mathbf{e}_x = \hphantom{-}\cos\alpha\:\mathbf{e_X} + \sin\alpha\:\mathbf{e_Y} \\
\mathbf{e}_y = -\sin\alpha\:\mathbf{e_X} + \cos\alpha\:\mathbf{e_Y}
\end{array}
\end{equation}

Note that as unit vectors, each of the versors above should have norm (length) equals to one, which indeed is the case.

If we express each versor above as different columns of a matrix, we obtain the rotation matrix again:

\begin{equation}
\mathbf{R_{Gl}} = \begin{bmatrix}
\cos\alpha & -\sin\alpha \\\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix}
\end{equation}

This means that the rotation matrix can be viewed as the basis of the rotated coordinate system defined by its versors.

This third way to derive the rotation matrix is in fact the method most commonly used in motion analysis because the coordinates of markers (in the Global/laboratory coordinate system) are what we measure with cameras.

Probably you are wondering how to perform the inverse operation, given a point in the Global coordinate system how to calculate its position in the local coordinate system. Let's see this now.

### Local-to-Global and Global-to-local coordinate systems' rotations

If we want the inverse operation, to express the position of point $\mathbf{P}$ in the local coordinate system in terms of the Global coordinate system, the figure below illustrates that using trigonometry.

Figure. The coordinates of a point at the local coordinate system in terms of the coordinates at the Global coordinate system.

Then:

\begin{equation}
\mathbf{P}_x = \;\;\mathbf{P_X} \cos \alpha + \mathbf{P_Y} \sin \alpha
\end{equation}

\begin{equation}
\mathbf{P}_y = -\mathbf{P_X} \sin \alpha + \mathbf{P_Y} \cos \alpha
\end{equation}

And in matrix form:

\begin{equation}
\begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y
\end{bmatrix} =
\begin{bmatrix}
\hphantom{-}\cos\alpha & \sin\alpha \\
-\sin\alpha & \cos\alpha
\end{bmatrix} \begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y}
\end{bmatrix}
\end{equation}

\begin{equation}
\mathbf{P_l} = \mathbf{R_{lG}}\mathbf{P_G}
\end{equation}

Where $\mathbf{R_{lG}}$ is the rotation matrix that rotates the coordinates from the Global to the local coordinate system (note the inverse order of the subscripts):

\begin{equation}
\mathbf{R_{lG}} = \begin{bmatrix}
\hphantom{-}\cos\alpha & \sin\alpha \\
-\sin\alpha & \cos\alpha
\end{bmatrix}
\end{equation}

If we use the direction cosines to calculate the rotation matrix, because the axes didn't change, the cosines are the same, only the order changes, now $\mathbf{x, y}$ are the rows (outputs) and $\mathbf{X, Y}$ are the columns (inputs):

\begin{equation}
\mathbf{R_{lG}} = \begin{bmatrix}
\cos\mathbf{X}x & \cos\mathbf{Y}x \\
\cos\mathbf{X}y & \cos\mathbf{Y}y
\end{bmatrix} =
\begin{bmatrix}
\cos(\alpha) & \cos(90^o-\alpha) \\
\cos(90^o+\alpha) & \cos(\alpha)
\end{bmatrix} =
\begin{bmatrix}
\hphantom{-}\cos\alpha & \sin\alpha \\
-\sin\alpha & \cos\alpha
\end{bmatrix}
\end{equation}

And defining the versors of the axes in the Global coordinate system for a basis in terms of the local coordinate system would also produce this latter rotation matrix.

The two sets of equations and matrices for the rotations from Global-to-local and local-to-Global coordinate systems are very similar, this is no coincidence. Each of the rotation matrices we deduced, $\mathbf{R_{Gl}}$ and $\mathbf{R_{lG}}$, perform the inverse operation in relation to the other. Each matrix is the inverse of the other.

In other words, the relation between the two rotation matrices means it is equivalent to instead of rotating the local coordinate system by $\alpha$ in relation to the Global coordinate system, to rotate the Global coordinate system by $-\alpha$ in relation to the local coordinate system; remember that $\cos(-\alpha)=\cos(\alpha)$ and $\sin(-\alpha)=-\sin(\alpha)$.

### The orthogonality of the rotation matrix

**[See here for a review about matrix and its main properties](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/Matrix.ipynb)**.

A nice property of the rotation matrix is that its inverse is the transpose of the matrix (because the columns/rows are mutually orthogonal and have norm equal to one).
This property can be shown with the rotation matrices we deduced:

\begin{equation}
\begin{array}{l l}
\mathbf{R}\:\mathbf{R^T} & =
\begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix}
\begin{bmatrix}
\hphantom{-}\cos\alpha & \sin\alpha \\
-\sin\alpha & \cos\alpha
\end{bmatrix} \\
& = \begin{bmatrix}
\cos^2\alpha+\sin^2\alpha & \cos\alpha \sin\alpha-\sin\alpha \cos\alpha\;\; \\
\sin\alpha \cos\alpha-\cos\alpha \sin\alpha & \sin^2\alpha+\cos^2\alpha\;\;
\end{bmatrix} \\
& = \begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix} \\
& = \mathbf{I} \\
\mathbf{R^{-1}} = \mathbf{R^T}
\end{array}
\end{equation}

This means that if we have a rotation matrix, we know its inverse.

The transpose and inverse operators in NumPy are methods of the array:

```python
α = np.pi/4
RGl = np.array([[np.cos(α), -np.sin(α)],
                [np.sin(α),  np.cos(α)]])

print('Orthogonal matrix (RGl):\n', RGl)
print('Transpose (RGl.T):\n', RGl.T)
print('Inverse (RGl.I):\n', np.linalg.inv(RGl))
```

    Orthogonal matrix (RGl):
     [[ 0.7071 -0.7071]
     [ 0.7071  0.7071]]
    Transpose (RGl.T):
     [[ 0.7071  0.7071]
     [-0.7071  0.7071]]
    Inverse (RGl.I):
     [[ 0.7071  0.7071]
     [-0.7071  0.7071]]

Using the inverse and the transpose mathematical operations, the coordinates at the local coordinate system given the coordinates at the Global coordinate system and the rotation matrix can be obtained by:

\begin{equation}
\begin{array}{l l}
\mathbf{P_G} = \mathbf{R_{Gl}}\mathbf{P_l} \implies \\
\mathbf{R_{Gl}^{-1}}\mathbf{P_G} = \mathbf{R_{Gl}^{-1}}\mathbf{R_{Gl}}\mathbf{P_l} \implies \\
\mathbf{R_{Gl}^{-1}}\mathbf{P_G} = \mathbf{I}\:\mathbf{P_l} \implies \\
\mathbf{P_l} = \mathbf{R_{Gl}^{-1}}\mathbf{P_G} = \mathbf{R_{Gl}^T}\mathbf{P_G} \quad \text{or}
\quad \mathbf{P_l} = \mathbf{R_{lG}}\mathbf{P_G}
\end{array}
\end{equation}

Where we referred the inverse of $\mathbf{R_{Gl}}\;(\:\mathbf{R_{Gl}^{-1}})$ as $\mathbf{R_{lG}}$ (note the different order of the subscripts).

Let's show this calculation in NumPy:

```python
α = np.pi/4
RGl = np.array([[np.cos(α), -np.sin(α)],
                [np.sin(α),  np.cos(α)]])
print('Rotation matrix (RGl):\n', RGl)

Pl  = np.array([[1, 1]]).T # transpose the array for correct matrix multiplication
print('Position at the local coordinate system (Pl):\n', Pl)

PG = RGl@Pl
print('Position at the Global coordinate system (PG=RGl*Pl):\n', PG)

Pl = RGl.T@PG
print('Position at the local coordinate system using the inverse of RGl (Pl=RlG*PG):\n', Pl)
```

    Rotation matrix (RGl):
     [[ 0.7071 -0.7071]
     [ 0.7071  0.7071]]
    Position at the local coordinate system (Pl):
     [[1]
     [1]]
    Position at the Global coordinate system (PG=RGl*Pl):
     [[0.    ]
     [1.4142]]
    Position at the local coordinate system using the inverse of RGl (Pl=RlG*PG):
     [[1.]
     [1.]]

### Rotation of a Vector

Another use of the rotation matrix is 'to rotate a vector by a given angle around an axis of the coordinate system as shown in the figure below.

Figure. Rotation of a position vector $\mathbf{P}$ by an angle $\alpha$ in the two-dimensional space.

We will not prove that we use the same rotation matrix, but think that in this case the vector position rotates by the same angle instead of the coordinate system. The new coordinates of the vector position $\mathbf{P'}$ rotated by an angle $\alpha$ is simply the rotation matrix (for the angle $\alpha$) multiplied by the coordinates of the vector position $\mathbf{P}$:

\begin{equation}
\mathbf{P'} = \mathbf{R}_\alpha\mathbf{P}
\end{equation}

Consider for example that $\mathbf{P}=[2,1]$ and $\alpha=30^o$; the coordinates of $\mathbf{P'}$ are:

```python
α  = np.pi/6
R  = np.array([[np.cos(α), -np.sin(α)],
               [np.sin(α),  np.cos(α)]])
P  = np.array([[2, 1]]).T
Pl = R@P
print("P':\n", Pl)
```

    P':
     [[1.2321]
     [1.866 ]]

**In summary, some of the properties of the rotation matrix are:**
1. The columns of the rotation matrix form a basis of (independent) unit vectors (versors) and the rows are also independent versors since the transpose of the rotation matrix is another rotation matrix.
2. The rotation matrix is orthogonal. There is no linear combination of one of the lines or columns of the matrix that would lead to the other row or column, i.e., the lines and columns of the rotation matrix are independent, orthogonal, to each other (this is property 1 rewritten). Because each row and column have norm equal to one, this matrix is also sometimes said to be orthonormal.
3. The determinant of the rotation matrix is equal to one (or equal to -1 if a left-hand coordinate system was used, but you should rarely use that). For instance, the determinant of the rotation matrix we deduced is $\cos\alpha \cos\alpha - \sin\alpha(-\sin\alpha)=1$.
4. The inverse of the rotation matrix is equals to its transpose.

**On the different meanings of the rotation matrix:**
- It represents the coordinate transformation between the coordinates of a point expressed in two different coordinate systems.
- It describes the rotation between two coordinate systems. The columns are the direction cosines (versors) of the axes of the rotated coordinate system in relation to the other coordinate system and the rows are also direction cosines (versors) for the inverse rotation.
- It is an operator for the calculation of the rotation of a vector in a coordinate system.
- Rotation matrices provide a means of numerically representing rotations without appealing to angular specification.

**Which matrix to use, from local to Global or Global to local?**
- A typical use of the transformation is in movement analysis, where there are the fixed Global (laboratory) coordinate system and the local (moving, e.g. anatomical) coordinate system attached to each body segment. Because the movement of the body segment is measured in the Global coordinate system, using cameras for example, and we want to reconstruct the coordinates of the markers at the anatomical coordinate system, we want the transformation leading from the Global coordinate system to the local coordinate system.
- Of course, if you have one matrix, it is simple to get the other; you just have to pay attention to use the right one.

### Calculation of rotation matrix using  a basis

A typical scenario in motion analysis is to calculate the rotation matrix using the position of markers placed on the moving rigid body. With the markers' positions, we create a local basis, which by definition is the rotation matrix for the rigid body with respect to the Global (laboratory) coordinate system. To define a coordinate system using a basís, we also will need to define an origin.

Let's see how to calculate a basis given the markers' positions.
Consider the markers at $m1=[1,1]$, $m2=[1,2]$ and $m3=[-1,1]$ measured in the Global coordinate system as illustrated in the figure below:

Figure. Three points in the two-dimensional space, two possible vectors given these points, and the corresponding basis.

A possible local coordinate system with origin at the position of m1 is also illustrated in the figure above. Intentionally, the three markers were chosen to form orthogonal vectors.
The translation vector between the two coordinate system is:

$$\mathbf{L_{Gl}} = m_1 - [0,0] = [1,1]$$

The vectors expressing the axes of the local coordinate system are:

$$ \hat{\mathbf{x}} = m_2 - m_1 = [1,2] - [1,1] = [0,1] $$

$$ \hat{\mathbf{y}} = m_3 - m_1 = [-1,1] - [1,1] = [-2,0] $$
<s/span>

Note that these two vectors do not form a basis yet because they are not unit vectors (in fact, only *y* is not a unit vector). Let's normalize these vectors:

$$ \begin{array}{}
\hat{\mathbf{e_x}} = \frac{x}{||x||} = \frac{[0,1]}{\sqrt{0^2+1^2}} = [0,1] \\
\\
\hat{\mathbf{e_y}} = \frac{y}{||y||} = \frac{[-2,0]}{\sqrt{2^2+0^2}} = [-1,0]
\end{array} $$

Beware that the versors above are not exactly the same as the ones shown in the right plot of the last figure, the versors above if plotted will start at the origin of the coordinate system, not at [1,1] as shown in the figure.

Since the markers $m1$, $m2$ and $m3$ were carefully chosen, the versors $\hat{\mathbf{e_x}}$ and $\hat{\mathbf{e_y}}$ are orthogonal.

We could have done this calculation in NumPy (we will need to do that when dealing with real data later):

```python
m1 = np.array([1.,1.])    # marker 1
m2 = np.array([1.,2.])    # marker 2
m3 = np.array([-1.,1.])   # marker 3

x = m2 - m1               # vector x
y = m3 - m1               # vector y

vx = x/np.linalg.norm(x)  # versor x
vy = y/np.linalg.norm(y)  # verson y

print("x =", x, ", y =", y, "\nex=", vx, ", ey=", vy)
```

    x = [0. 1.] , y = [-2.  0.]
    ex= [0. 1.] , ey= [-1.  0.]

Now, both $\hat{\mathbf{e}_x}$ and $\hat{\mathbf{e}_y}$ are unit vectors (versors) and they are orthogonal, a basis can be formed with these two versors, and we can represent the rotation matrix using this basis (just place the versors of this basis as columns of the rotation matrix):

\begin{equation}
\mathbf{R_{Gl}} = \begin{bmatrix}
0 & -1 \\
1 & \hphantom{-}0
\end{bmatrix}
\end{equation}

This rotation matrix makes sense because from the figure above we see that the local coordinate system we defined is rotated by 90$^o$ in relation to the Global coordinate system and if we use the general form for the rotation matrix:

\begin{equation}
\mathbf{R} = \begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix} =
\begin{bmatrix}
\cos90^o & -\sin90^o \\
\sin90^o & \hphantom{-}\cos90^o
\end{bmatrix} =
\begin{bmatrix}
0 & -1 \\
1 & \hphantom{-}0
\end{bmatrix}
\end{equation}

So, the position of any point in the local coordinate system can be represented in the Global coordinate system by:

\begin{equation}
\begin{array}{l l}
\mathbf{P_G} =& \mathbf{L_{Gl}} + \mathbf{R_{Gl}}\mathbf{P_l} \\
\\
\mathbf{P_G} =& \begin{bmatrix} 1 \\ 1 \end{bmatrix} + \begin{bmatrix} 0 & -1 \\ 1 & \hphantom{-}0 \end{bmatrix} \mathbf{P_l}
\end{array}
\end{equation}

For example, the point $\mathbf{P_l}=[1,1]$ has the following position at the Global coordinate system:

```python
LGl = np.array([[1, 1]]).T
print('Translation vector:\n', LGl)

RGl = np.array([[0, -1],
                [1,  0]])
print('Rotation matrix:\n', RGl)

Pl  = np.array([[1, 1]]).T
print('Position at the local coordinate system:\n', Pl)

PG = LGl + RGl@Pl
print('Position at the Global coordinate system, PG = LGl + RGl*Pl:\n', PG)
```

    Translation vector:
     [[1]
     [1]]
    Rotation matrix:
     [[ 0 -1]
     [ 1  0]]
    Position at the local coordinate system:
     [[1]
     [1]]
    Position at the Global coordinate system, PG = LGl + RGl*Pl:
     [[0]
     [2]]

### Determination of the unknown angle of rotation

If we didn't know the angle of rotation between the two coordinate systems, which is the typical situation in motion analysis, we simply would equate one of the terms of the two-dimensional rotation matrix in its algebraic form to its correspondent value in the numerical rotation matrix we calculated.

\begin{equation}
\mathbf{R} = \begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix}
\end{equation}

So the angle $\alpha$ can be found by:

\begin{equation}
    \alpha = \arctan\left(\frac{R[1,0]}{R[0,0]}\right)
\end{equation}

For instance, taking the first term of the rotation matrices above: $\cos\alpha = 0$ implies that $\alpha$ is 90$^o$ or 270$^o$, but combining with another matrix term, $\sin\alpha = 1$, implies that $\alpha=90^o$. We can solve this problem in one step using the tangent $(\sin\alpha/\cos\alpha)$ function with two terms of the rotation matrix and calculating the angle with the `arctan2(y, x)` function:

```python
RGl = np.array([[0, -1],
                [1,  0]])
ang = np.arctan2(RGl[1, 0], RGl[0, 0])*180/np.pi
print('The angle is:', ang)
```

    The angle is: 90.0

And this procedure would be repeated for each segment and for each instant of the analyzed movement to find the rotation of each segment.

### Joint angle as a sequence of rotations of adjacent segments

In the notebook about [two-dimensional angular kinematics](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/KinematicsAngular2D.ipynb), we calculated segment and joint angles using simple trigonometric relations. We can also calculate these two-dimensional angles using what we learned here about the rotation matrix.

The segment angle will be given by the matrix representing the rotation from the laboratory coordinate system (G) to a coordinate system attached to the segment and the joint angle will be given by the matrix representing the rotation from one segment coordinate system (l1) to the other segment coordinate system (l2).

So, we have to compute two basis now, one for each segment and the joint angle will be given by the product between the two rotation matrices obtained from both basis.

The description of a point P in the global basis given the description of this point in the l1 basis is:

\begin{equation}
    \begin{bmatrix}
        x_p\\
        y_p
    \end{bmatrix} =R_{Gl1}
    \begin{bmatrix}
        x_{p_1}\\
        y_{p_1}
    \end{bmatrix}
\end{equation}

The description of the same point P in the global basis given the description of this point in the l2 basis is:

\begin{equation}
    \begin{bmatrix}
        x_p\\
        y_p
    \end{bmatrix} =R_{Gl_2}
    \begin{bmatrix}
        x_{p_2}\\
        y_{p_2}
    \end{bmatrix}
\end{equation}

So, to find the description of this point P in the l1 basis given the description of this point in the l2 basis is:

\begin{equation}
    R_{Gl_1}
    \begin{bmatrix}
        x_{p_1}\\
        y_{p_1}
    \end{bmatrix} =
    R_{Gl_2}
    \begin{bmatrix}
        x_{p_2}\\
        y_{p_2}
    \end{bmatrix}
\rightarrow \begin{bmatrix}
        x_{p_1}\\
        y_{p_1}
    \end{bmatrix} =
    \underbrace{R_{Gl_1}^{-1}R_{Gl_2}}_{R_{l_1l_2}}
    \begin{bmatrix}
        x_{p_2}\\
        y_{p_2}
    \end{bmatrix}
\end{equation}

The rotation matrix from $l_2$ to $l_1$ is:

\begin{equation}
   R_{l_1l_2} = R_{Gl_1}^{-1}R_{Gl_2} = R_{Gl_1}^{T}R_{Gl_2}
\end{equation}

The rotation matrices of both basis are:

\begin{equation}
   R_{Gl_1} = \begin{bmatrix}
    \cos(\theta_1) &-\sin(\theta_1)\\
    \sin(\theta_1) &\hphantom{-}\cos(\theta_1)
    \end{bmatrix}, \,
    R_{Gl_2} = \begin{bmatrix}
    \cos(\theta_2) &-\sin(\theta_2)\\
    \sin(\theta_2) &\hphantom{-}\cos(\theta_2)
    \end{bmatrix}
\end{equation}

So, the rotation matrix from $l_2$ to $l_1$ is:

\begin{align}
   R_{l_1l_2} =& \begin{bmatrix}
    \cos(\theta_1) &\sin(\theta_1)\\
    -\sin(\theta_1) &\cos(\theta_1)
    \end{bmatrix}.\begin{bmatrix}
    \cos(\theta_2) &-\sin(\theta_2)\\
    \sin(\theta_2) &\hphantom{-}\cos(\theta_2)
    \end{bmatrix} =\\
    =&\begin{bmatrix}
    \cos(\theta_1)\cos(\theta_2)+\sin(\theta_1)\sin(\theta_2) &\cos(\theta_2)\sin(\theta_1)-\cos(\theta_1)\sin(\theta_2)\\
    \cos(\theta_1)\sin(\theta_2)-\cos(\theta_2)\sin(\theta_1)&\cos(\theta_1)\cos(\theta_2)+\sin(\theta_1)\sin(\theta_2)
    \end{bmatrix}=\\
    =&\begin{bmatrix}    \cos(\theta_2-\theta_1) &-\sin(\theta_2-\theta_1)\\
    \sin(\theta_2-\theta_1) &\hphantom{-}\cos(\theta_2-\theta_1)
    \end{bmatrix}
\end{align}

The angle $\theta_{l_1l_2} = \theta_2-\theta_1$ is the angle between the two reference frames. So to find the $\theta_{l_1l_2}$ is:

\begin{align}
   \theta_{l_1l_2} = \arctan\left(\frac{R_{l_1l_2}[1,0]}{R_{l_1l_2}[0,0]}\right)
\end{align}

Below is an example: To define a two-dimensional basis, we need to calculate vectors perpendicular to each of these lines. Here is a way of doing that. First, let's find three non-collinear points for each basis:

```python
x1, y1, x2, y2 = 0, 0, 1, 1      # points at segment 1
x3, y3, x4, y4 = 1.1, 1, 2.1, 0  # points at segment 2

#The slope of the perpendicular line is minus the inverse of the slope of the line
xl1 = x1 - (y2-y1); yl1 = y1 + (x2-x1)  # point at the perpendicular line 1
xl2 = x4 - (y3-y4); yl2 = y4 + (x3-x4)  # point at the perpendicular line 2
```

With these three points, we can create a basis and the corresponding rotation matrix:

```python
b1x = np.array([x2-x1, y2-y1])
b1x = b1x/np.linalg.norm(b1x)    # versor x of basis 1
b1y = np.array([xl1-x1, yl1-y1])
b1y = b1y/np.linalg.norm(b1y)    # versor y of basis 1
b2x = np.array([x3-x4, y3-y4])
b2x = b2x/np.linalg.norm(b2x)    # versor x of basis 2
b2y = np.array([xl2-x4, yl2-y4])
b2y = b2y/np.linalg.norm(b2y)    # versor y of basis 2

RGl1 = np.array([b1x, b1y]).T    # rotation matrix from segment 1 to the laboratory
RGl2 = np.array([b2x, b2y]).T    # rotation matrix from segment 2 to the laboratory

print('rotation matrix from segment 1 to global:\n', RGl1)
print('rotation matrix from segment 2 to global:\n', RGl2)
```

    rotation matrix from segment 1 to global:
     [[ 0.7071 -0.7071]
     [ 0.7071  0.7071]]
    rotation matrix from segment 2 to global:
     [[-0.7071 -0.7071]
     [ 0.7071 -0.7071]]

Now, the segment and joint angles are simply matrix operations:

```python
print('Rotation matrix for segment 1:\n', RGl1)
print('\nRotation angle of segment 1:', np.arctan2(RGl1[1,0], RGl1[0,0])*180/np.pi)
print('\nRotation matrix for segment 2:\n', RGl2)
print('\nRotation angle of segment 2:', np.arctan2(RGl1[1,0], RGl2[0,0])*180/np.pi)

Rl1l2 = RGl1.T@RGl2  # Rl1l2 = Rl1G*RGl2

print('\nJoint rotation matrix (Rl1l2 = Rl1G*RGl2):\n', Rl1l2)
print('\nJoint angle:', np.arctan2(Rl1l2[1,0], Rl1l2[0,0])*180/np.pi)
```

    Rotation matrix for segment 1:
     [[ 0.7071 -0.7071]
     [ 0.7071  0.7071]]

    Rotation angle of segment 1: 45.0

    Rotation matrix for segment 2:
     [[-0.7071 -0.7071]
     [ 0.7071 -0.7071]]

    Rotation angle of segment 2: 135.0

    Joint rotation matrix (Rl1l2 = Rl1G*RGl2):
     [[ 0. -1.]
     [ 1. -0.]]

    Joint angle: 90.0

Same result as obtained in [Angular kinematics in a plane (2D)](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/KinematicsAngular2D.ipynb).

### Kinematic chain in a plain (2D)

The fact that we simply multiplied the rotation matrices to calculate the rotation matrix of one segment in relation to the other is powerful and can be generalized for any number of segments: given a serial kinematic chain with links 1, 2, ..., n and 0 is the base/laboratory, the rotation matrix between the base and last link is: $\mathbf{R_{n,n-1}R_{n-1,n-2} \dots R_{2,1}R_{1,0}}$, where each matrix in this product (calculated from right to left) is the rotation of one link with respect to the next one.

For instance, consider a kinematic chain with two links, the link 1 is rotated by $\alpha_1$ with respect to the base (0) and the link 2 is rotated by $\alpha_2$ with respect to the link 1.
Using Sympy, the rotation matrices for link 2 w.r.t. link 1 $(R_{12})$ and for link 1 w.r.t. base 0 $(R_{01})$ are:

```python
from IPython.display import display, Math
from sympy import sin, cos, Matrix, simplify, latex, symbols
from sympy.interactive import printing
printing.init_printing()
```

```python
a1, a2 = symbols('alpha1 alpha2')

R12 = Matrix([[cos(a2), -sin(a2)], [sin(a2), cos(a2)]])
display(Math(r'\mathbf{R_{12}}=' + latex(R12)))
R01 = Matrix([[cos(a1), -sin(a1)], [sin(a1), cos(a1)]])
display(Math(r'\mathbf{R_{01}}=' + latex(R01)))
```

$\displaystyle \mathbf{R_{12}}=\left[\begin{matrix}\cos{\left(\alpha_{2} \right)} & - \sin{\left(\alpha_{2} \right)}\\\sin{\left(\alpha_{2} \right)} & \cos{\left(\alpha_{2} \right)}\end{matrix}\right]$

$\displaystyle \mathbf{R_{01}}=\left[\begin{matrix}\cos{\left(\alpha_{1} \right)} & - \sin{\left(\alpha_{1} \right)}\\\sin{\left(\alpha_{1} \right)} & \cos{\left(\alpha_{1} \right)}\end{matrix}\right]$

The rotation matrix of link 2 w.r.t. the base $(R_{02})$ is given simply by $R_{01}*R_{12}$:

```python
R02 = R01*R12
display(Math(r'\mathbf{R_{02}}=' + latex(R02)))
```

$\displaystyle \mathbf{R_{02}}=\left[\begin{matrix}- \sin{\left(\alpha_{1} \right)} \sin{\left(\alpha_{2} \right)} + \cos{\left(\alpha_{1} \right)} \cos{\left(\alpha_{2} \right)} & - \sin{\left(\alpha_{1} \right)} \cos{\left(\alpha_{2} \right)} - \sin{\left(\alpha_{2} \right)} \cos{\left(\alpha_{1} \right)}\\\sin{\left(\alpha_{1} \right)} \cos{\left(\alpha_{2} \right)} + \sin{\left(\alpha_{2} \right)} \cos{\left(\alpha_{1} \right)} & - \sin{\left(\alpha_{1} \right)} \sin{\left(\alpha_{2} \right)} + \cos{\left(\alpha_{1} \right)} \cos{\left(\alpha_{2} \right)}\end{matrix}\right]$

Which simplifies to:

```python
display(Math(r'\mathbf{R_{02}}=' + latex(simplify(R02))))
```

$\displaystyle \mathbf{R_{02}}=\left[\begin{matrix}\cos{\left(\alpha_{1} + \alpha_{2} \right)} & - \sin{\left(\alpha_{1} + \alpha_{2} \right)}\\\sin{\left(\alpha_{1} + \alpha_{2} \right)} & \cos{\left(\alpha_{1} + \alpha_{2} \right)}\end{matrix}\right]$

As expected.

The typical use of all these concepts is in the three-dimensional motion analysis where we will have to deal with angles in different planes, which needs a special manipulation as we will see next.

## Translation and rotation

Consider now the case where the local coordinate system is translated and rotated in relation to the Global coordinate system and a point is described in both coordinate systems as illustrated in the figure below (once again, remember that this is equivalent to describing a translation and a rotation between two rigid bodies).

Figure. A point in two-dimensional space represented in two coordinate systems, with one system translated and rotated.

The position of point $\mathbf{P}$ originally described in the local coordinate system, but now described in the Global coordinate system in vector form is:

\begin{equation}
\mathbf{P_G} = \mathbf{L_G} + \mathbf{R_{Gl}}\mathbf{P_l}
\end{equation}

And in matrix form:

\begin{equation}
\begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y}
\end{bmatrix} =
\begin{bmatrix} \mathbf{L_{X}} \\\ \mathbf{L_{Y}} \end{bmatrix} +
\begin{bmatrix}
\cos\alpha & -\sin\alpha \\
\sin\alpha & \hphantom{-}\cos\alpha
\end{bmatrix} \begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y
\end{bmatrix}
\end{equation}

This means that we first *disrotate* the local coordinate system and then correct for the translation between the two coordinate systems. Note that we can't invert this order: the point position is expressed in the local coordinate system and we can't add this vector to another vector expressed in the Global coordinate system, first we have to convert the vectors to the same coordinate system.

If now we want to find the position of a point at the local coordinate system given its position in the Global coordinate system, the rotation matrix and the translation vector, we have to invert the expression above:

\begin{equation}
\begin{array}{l l}
\mathbf{P_G} = \mathbf{L_G} + \mathbf{R_{Gl}}\mathbf{P_l} \implies \\
\mathbf{R_{Gl}^{-1}}(\mathbf{P_G} - \mathbf{L_G}) = \mathbf{R_{Gl}^{-1}}\mathbf{R_{Gl}}\mathbf{P_l} \implies \\
\mathbf{P_l} = \mathbf{R_{Gl}^{-1}}\left(\mathbf{P_G}-\mathbf{L_G}\right) = \mathbf{R_{Gl}^T}\left(\mathbf{P_G}-\mathbf{L_G}\right) \quad \text{or} \quad \mathbf{P_l} = \mathbf{R_{lG}}\left(\mathbf{P_G}-\mathbf{L_G}\right)
\end{array}
\end{equation}

The expression above indicates that to perform the inverse operation, to go from the Global to the local coordinate system, we first translate and then rotate the coordinate system.

### Transformation matrix

It is possible to combine the translation and rotation operations in only one matrix, called the transformation matrix (also referred as homogeneous transformation matrix):

\begin{equation}
\begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y} \\
1
\end{bmatrix} =
\begin{bmatrix}
\cos\alpha & -\sin\alpha & \mathbf{L_{X}} \\
\sin\alpha & \hphantom{-}\cos\alpha  & \mathbf{L_{Y}} \\
0 & 0 & 1
\end{bmatrix} \begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y \\
1
\end{bmatrix}
\end{equation}

Or simply:

\begin{equation}
\mathbf{P_G} = \mathbf{T_{Gl}}\mathbf{P_l}
\end{equation}

The inverse operation, to express the position at the local coordinate system in terms of the Global coordinate system, is:

\begin{equation}
\mathbf{P_l} = \mathbf{T_{Gl}^{-1}}\mathbf{P_G}
\end{equation}

However, because $\mathbf{T_{Gl}}$ is not orthonormal, which means its inverse is not its transpose. Its inverse in matrix form is given by:

\begin{equation}
\begin{bmatrix}
\mathbf{P}_x \\
\mathbf{P}_y \\
1
\end{bmatrix} =
\underbrace{\begin{bmatrix}
\mathbf{R^{-1}_{Gl}} & \cdot & - \mathbf{R^{-1}_{Gl}}\mathbf{L_{G}} \\
\cdot & \cdot  & \cdot \\
0 & 0 & 1
\end{bmatrix}}_{ \mathbf{T_{Gl}^{-1}}} \begin{bmatrix}
\mathbf{P_X} \\
\mathbf{P_Y} \\
1
\end{bmatrix}
\end{equation}

## Further reading

- Read pages 751-758 of the 16th chapter of the [Ruina and Rudra's book](http://ruina.tam.cornell.edu/Book/index.html) about rotation matrix.

## Video lectures on the Internet

- Khan Academy: [Exemplos de transformação linear: rotações em R2](https://pt.khanacademy.org/math/linear-algebra/matrix-transformations/lin-trans-examples/v/linear-transformation-examples-rotations-in-r2)
- [MA08 - Criando a matriz de rotação em R2](https://www.youtube.com/watch?v=OSYt9BDmTTo)
- [GEOMETRIA ANALÍTICA PARA ITA/IME | TRANSLAÇÃO E ROTAÇÃO DE EIXOS: PARTE 1 ](https://www.youtube.com/watch?v=UGoidDgM7PQ)

## Problems

1. A local coordinate system is rotated 30$^o$ clockwise in relation to the Global reference system.
  A. Determine the matrices for rotating one coordinate system to another (two-dimensional).
  B. What are the coordinates of the point [1, 1] (local coordinate system) at the global coordinate system?
  C. And if this point is at the Global coordinate system and we want the coordinates at the local coordinate system?
  D. Consider that the local coordinate system, besides the rotation is also translated by [2, 2]. What are the matrices for rotation, translation, and transformation from one coordinate system to another (two-dimensional)?
  E. Repeat B and C considering this translation.

2. Consider a local coordinate system U rotated 45$^o$ clockwise in relation to the Global reference system and another local coordinate system V rotated 45$^o$ clockwise in relation to the local reference system U.
  A. Determine the rotation matrices of all possible transformations between the coordinate systems.
  B. For the point [1, 1] in the coordinate system U, what are its coordinates in coordinate system V and in the Global coordinate system?

3. Using the rotation matrix, deduce the new coordinates of a square figure with coordinates [0, 0], [1, 0], [1, 1], and [0, 1] when rotated by 0$^o$, 45$^o$, 90$^o$, 135$^o$, and 180$^o$ (always clockwise).

4. Solve the problem 2 of [Angular kinematics in a plane (2D)](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/KinematicsAngular2D.ipynb) but now using the concept of two-dimensional transformations.

## References

- Robertson G, Caldwell G, Hamill J, Kamen G (2013) [Research Methods in Biomechanics](http://books.google.com.br/books?id=gRn8AAAAQBAJ). 2nd Edition. Human Kinetics.
- Ruina A, Rudra P (2013) [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.
- Winter DA (2009) [Biomechanics and motor control of human movement](http://books.google.com.br/books?id=_bFHL08IWfwC). 4 ed. Hoboken, EUA: Wiley.
- Zatsiorsky VM (1997) [Kinematics of Human Motion](http://books.google.com.br/books/about/Kinematics_of_Human_Motion.html?id=Pql_xXdbrMcC&redir_esc=y). Champaign, Human Kinetics.


# Chapter 3: Random Variables and their Distributions

This Jupyter notebook is the Python equivalent of the R code in section 3.11 R, pp. 138 - 140, [Introduction to Probability, Second Edition](https://www.crcpress.com/Introduction-to-Probability-Second-Edition/Blitzstein-Hwang/p/book/9781138369917), Blitzstein & Hwang.

----

```python
import numpy as np
```

## Distributions in SciPy

All of the named distributions that we'll encounter in this book have been implemented in R, but in this Python-based notebook we will use their equivalents in [SciPy](https://docs.scipy.org/doc/scipy/reference/index.html). In this section we'll explain how to work with the Binomial and Hypergeometric distributions in [`scipy.stats`](https://docs.scipy.org/doc/scipy/reference/tutorial/stats.html). We will also explain in general how to generate r.v.s from any discrete distribution with a finite support. The aforementioned Statistical functions page is a handy list of the distributions in `scipy.stats`. Typing `scipy.stats.[distribution].__doc__` will display more information on the named distribution.

In general for the named discrete distributions in `scipy.stats`, the three functions `pmf`, `cdf`, and `rvs` will give the PMF, CDF, and random generation, respectively.

### Binomial distribution

The Binomial distribution [`binom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom.html#scipy.stats.binom) provides the following three functions: `pmf`, `cdf`, and `rvs`. Unlike R, SciPy provides an implementation of the Bernoulli distribution in [`scipy.stats.bernoulli`](https://docs.scipy.org/doc/scipy/reference/tutorial/stats/discrete_bernoulli.html). But as an alterative for `bernoulli`, we can just use the Binomial functions with $k \in \{0,1\}$ and $n = 1$.

```python
from scipy.stats import binom

# to learn more about scipy.stats.binom, un-comment ouf the following line
#print(binom.__doc__)
```

* `binom.pmf` is `scipy.stats`' implementation of the Binomial PMF. It takes three inputs: the first is the value of `k` at which to evaluate the PMF, and the second and third are the parameters `n` and `p`. For example, `binom.pmf(3, 5, 0.2)` returns the probability $P(X = 3)$ where $X \sim Bin(5, 0.2)$. In other words,

\begin{align}
  binom.pmf(3, 5, 0.2) &= \binom{5}{3} (0.2)^{3} (0.8)^{2} \\
  &= 0.0512
\end{align}

```python
k = 3
n = 5
p = 0.2

binom.pmf(k, n, p)
```

    0.051200000000000016

* `binom.cdf` is the Binomial CDF. It takes three inputs: the first is the value of `k` at which to evaluate the CDF, and the second and third are the parameters. $binom.cdf(3, 5, 0.2)$ is the probability $P(X \leq 3)$ where $X \sim Bin(5, 0.2)$. So

\begin{align}
  binom.cdf(3, 5, 0.2) &= \sum_{k=0}^{3} \binom{5}{k} (0.2)^k (0.8)^{5-k} \\
  &= 0.9933
\end{align}

```python
k = 3
n = 5
p = 0.2

binom.cdf(k, n, p)
```

    0.99328

* `binom.rvs` is a function for generating Binomial random variables. For `rvs`, the first and second inputs are still the parameters `n` and `p`, and the `size` parameter is how many r.v.s we want to generate. Thus the command `binom.rvs(5, 0.2, size=7)` produces realizations of seven i.i.d. $Bin(5, 0.2)$ r.v.s. When we ran this command, we got

```python
# seed the random number generator
np.random.seed(144)

n = 5
p = 0.2

binom.rvs(n, p, size=7)
```

    array([0, 2, 1, 0, 1, 1, 1])

Unless you change the `numpy.random.seed` parameter value in code cell [1] above, you'll get the same values above the first time this notebook is run.

We can also evaluate PMFs and CDFs at an entire vector of values. For example, recall that `numpy.arange(0,n+1)` is a quick way to list the integers from $0$ to $n$. The command `binom.pmf(numpy.arange(0, 5+1), 5, 0.2)` returns 6 numbers, $P(X = 0)\text{, } P(X = 1)\text{, } \cdots \text{, } P(X = 5)$, where $X \sim Bin(5, 0.2)$.

```python
n = 5
p = 0.2

binom.pmf(np.arange(0, n+1), n, p)
```

    array([3.2768e-01, 4.0960e-01, 2.0480e-01, 5.1200e-02, 6.4000e-03,
           3.2000e-04])

## Hypergeometric distribution

The Hypergeometric distribution is implemented in [`scipy.stats.hypergeom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.hypergeom.html#scipy.stats.hypergeom), which similarly provides the following three functions: `pmf`, `cdf`, and `rvs`.  As one might expect, `pmf` is the Hypergeometric PMF, `cdf` is the Hypergeometric CDF, and `rvs` generates Hypergeometric r.v.s. Since the Hypergeometric distribution has three parameters, each of these functions takes at least three inputs. For `pmf` and `cdf`, the first input is the value at which we wish to evaluate the PMF or CDF, and the remaining three inputs are the parameters of the distribution.

```python
from scipy.stats import hypergeom

# to learn more about scipy.stats.hypergeom, un-comment ouf the following line
#print(hypergeom.__doc__)
```

The PMF is defined as

\begin{align}
  p(k, M, n, N) &= \frac{\binom{n}{k} \, \binom{M-n}{N-k}}{\binom{M}{N}}
\end{align}

where
* $M$ = the total number of objects (black and white balls, for instance)
* $n$ = the total number of objects of Type I (say we're interested in the white balls)
* $N$ = the total number of objects drawn without replacement from $M$

Consider the case where we have a total of $M = 17$ balls (black and white), with $n = 10$ among them being white (and 7 being black), and we draw $N = 8$ of the balls without replacement.

```python
n = 10       # white balls <- these are the ones we are interested in
b = 7        # black balls
M = n + b    # total number of balls
N = 8        # number of balls to draw w/out replacement
```

* `hypergeom.pmf(k, 17, 10, 8)` returns the probability $P(X = k)$ where $X \sim HGeom(10, 7, 8)$

```python
# k = 1
hypergeom.pmf(1, M, n, N)
```

    0.0004113533525298236

* `hypergeom.cdf(k, 17, 10, 8)` returns $P(X \leq k)$

```python
# k = 2
hypergeom.cdf(2, M, n, N)
```

    0.013368983957219274

* `hypergeom.rvs(17, 10, 8, size=s)` uses the `size` parameter to specify the number of r.v.s. we want to generate. `hypergeom.rvs(17, 10, 8, size=100)` generates 100 i.i.d. $HGeom(10, 7, 8)$ r.v.s.

```python
np.random.seed(233)

hypergeom.rvs(M, n, N, size=100)
```

    array([6, 3, 5, 5, 5, 4, 3, 4, 7, 4, 6, 4, 6, 6, 4, 4, 5, 4, 6, 5, 5, 4,
           6, 5, 5, 6, 5, 4, 4, 3, 5, 4, 5, 5, 6, 3, 5, 6, 6, 4, 5, 4, 5, 5,
           4, 3, 5, 4, 5, 5, 3, 7, 5, 4, 5, 6, 4, 3, 5, 5, 3, 4, 4, 4, 5, 4,
           4, 5, 6, 4, 4, 4, 5, 5, 5, 4, 5, 4, 4, 4, 7, 4, 5, 5, 5, 4, 5, 5,
           3, 4, 4, 5, 4, 4, 5, 4, 6, 5, 6, 4])

## Discrete distributions with finite support

We can generate r.v.s from any discrete distribution with finite support using the [`numpy.random.choice`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html#numpy.random.choice) function. When we first introduced the `numpy.random.choice` function, we said that it can be used in the form `numpy.random.choice(numpy.arange(1,n+1), k, replace=False)` or `numpy.random.choice(np.arange(1,n+1), k)` to sample $k$ times from the integers 1 through $n$, either without or with replacement. For example, to generate 5 independent $DUnif(1, 2, \ldots, 100$) r.v.s, we can use the function invocation `numpy.random.choice(numpy.arange(1,100+1), 5)`.

It turns out that `numpy.random.choice` is far more versatile. If we want to sample from the values $x_1, \ldots, x_n$ with probabilities $p_1, \ldots, p_n$, we simply create an array `x` containing all the $x_i$ and an array `p` containing all the $p_i$, then feed them into `choice`. For example, suppose we are interested in generating realizations of i.i.d. r.v.s $X_1, \ldots, X_{100}$ whose PMF is

\begin{align}
  P(X_j = 0) &= 0.25 \\
  P(X_j = 1) &= 0.5 \\
  P(X_j = 5) &= 0.1 \\
  P(X_j = 10) &= 0.15
\end{align}

and $P(X_j = x) = 0$ for all other values of $x$. First, we use the `numpy.array`  function to create arrays with the support of the distribution and the PMF probabilities.

```python
x = np.array([0, 1, 5, 10])
p = np.array([0.25, 0.5, 0.1, 0.15])
```

Next, we use the `choice` function. Here's how to get 100 samples from the above PMF:

```python
np.random.seed(377)

np.random.choice(x, 100, p=p)
```

    array([ 0,  1,  1,  5,  0,  1,  1,  0,  1,  0,  0,  1,  1,  0,  1,  5,  1,
            0,  5,  1,  1,  1,  0,  1,  1,  0,  1,  0,  5,  1,  1,  1, 10,  0,
            5,  0, 10,  1,  1,  1,  0,  1,  1,  5,  1,  5, 10, 10,  0,  1,  1,
            1,  1, 10,  0,  1, 10,  0,  0,  1,  1,  5,  1, 10,  1,  0,  1,  1,
            5,  1,  1, 10,  1,  1,  0,  0,  0,  1,  1, 10, 10,  0,  5,  1,  1,
            5,  1,  1,  0,  0,  0,  1,  1,  1,  0,  1,  0,  1,  1,  0])

The inputs are the array `x` to sample from, the number of samples to generate (100 in this case), the probabilities `p` of sampling the values in `x` (if this is omitted, the probabilities are assumed equal), and whether to sample with replacement (the default is `replace=True`).

----

Joseph K. Blitzstein and Jessica Hwang, Harvard University and Stanford University, © 2019 by Taylor and Francis Group, LLC


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

Support Vector Machines (SVM) originated from the statistical learning theory
developed by Vapnik-Chervonenkis. As such, it represents a deep application of
statistical theory that incorporates the VC dimension concepts we
discussed in the first section. Let's start by looking at some pictures.
Consider the two-dimensional classification problem shown in [Figure](#fig:svm_001).  [Figure](#fig:svm_001) shows two classes (gray and white
circles) that can be separated by any of the lines shown. Specifically, any
such separating line can be written as the locus of points ($\mathbf{x}$) in
the two-dimensional plane that satisfy the following,

In the two-dimensional plane, the two classes (gray and white circles) are easily separated by any one of the lines shown.

$$
\beta_0 + \boldsymbol{\beta}^T \mathbf{x} = 0
$$

 To classify  an arbitrary $\mathbf{x}$ using this line, we just
compute the sign of $\beta_0+\boldsymbol{\beta}^T \mathbf{x}$ and assign one
class to the positive sign and the other class to the negative sign.  To
uniquely specify such a separating line (or, hyperplane in a higher-dimensional
space) we need additional criteria.

[Figure](#fig:svm_002) shows the data with two bordering parallel lines that
form a margin around the central separating line.  The *maximal margin
algorithm* finds the widest margin and the unique separating line.  As a
consequence, the algorithm uncovers the elements in the data that touch the
margins. These are the *support* elements. The other elements
away from the border are not relevant to the solution. This reduces
model variance because the solution is insensitive to the removal of
elements other than these supporting elements (usually a small minority).

The maximal margin algorithm finds the separating line that maximizes the margin shown. The elements that touch the margins are the support elements. The dotted elements are not relevant to the solution.

To see how this works for linearly separable classes, consider a
training set consisting of $\lbrace (\mathbf{x},y) \rbrace$ where
$y\in \lbrace -1,1 \rbrace$. For any point $\mathbf{x}_i$, we
compute the functional margin as $\hat{ \gamma_i }=y_i (\beta_0 +
\boldsymbol{\beta}^T \mathbf{x}_i)$. Thus, $\hat{\gamma}_i >0$ when
$\mathbf{x}_i$ is correctly classified. The geometrical margin is
$\gamma = \hat{\gamma}/\lVert\boldsymbol{\beta}\rVert$. When
$\mathbf{x}_i$ is correctly classified, the geometrical margin is
equal to the perpendicular distance from $\mathbf{x}_i$ to the line.
Let's look see how the maximal margin algorithm works.

Let $M$ be the width of the margin.  The maximal margin algorithm is can be
formulated as a quadratic programming problem. We want to simultaneously
maximize the margin $M$ while ensuring that all of the data points are
correctly classified.

$$
\begin{aligned}
& \underset{\beta_0,\boldsymbol{\beta},\lVert\boldsymbol{\beta}\rVert=1}{\text{maximize}}
& & M \\\
& \text{subject to:}
& & y_i(\beta_0+\boldsymbol{\beta}^T \mathbf{x}_i) \geq M, \; i = 1, \ldots, N.
\end{aligned}
$$

 The first line says we want to generate a maximum value for $M$ by
adjusting $\beta_0$ and $\boldsymbol{\beta}$ while keeping
$\lVert\boldsymbol{\beta}\rVert=1$. The functional margins for each $i^{th}$
data element are the constraints to the problem and must be satisfied for every
proposed solution. In words, the constraints enforce that the elements have to
be correctly classified and outside of the margin around the separating line.
With some reformulation, it turns out that
$M=1/\lVert\boldsymbol{\beta}\rVert$ and this can be put into the following
standard format,

$$
\begin{aligned}
& \underset{\beta_0,\boldsymbol{\beta}}{\text{minimize}}
& & \lVert\boldsymbol{\beta}\rVert \\\
& \text{subject to:}
& & y_i(\beta_0+\boldsymbol{\beta}^T \mathbf{x}_i) \geq 1, \; i = 1, \ldots, N.
\end{aligned}
$$

 This is a convex optimization problem  and can be solved using powerful
methods in that area.

The situation becomes more complex when the two classes are not separable and
we have to allow some unavoidable mixing between the two classes in the
solution. This means that the contraints have to modified as in the following,

$$
y_i(\beta_0+\boldsymbol{\beta}^T \mathbf{x}_i) \geq M(1-\xi_i)
$$

 where the $\xi_i$ are the slack variables and represent the
proportional amount tha the prediction is on the wrong side of the margin. Thus,
elements are misclassified when $\xi_i>1$. With these additional variables,
we have a more general formulation of the convex optimization problem,

$$
\begin{aligned}
& \underset{\beta_0,\boldsymbol{\beta}}{\text{minimize}}
& & \lVert\boldsymbol{\beta}\rVert \\\
& \text{subject to:}
& & y_i(\beta_0+\boldsymbol{\beta}^T \mathbf{x}_i) \geq 1-\xi_i, \\\
& & & \xi_i \geq 0, \sum \xi_i \leq \texttt{constant}, \; i = 1, \ldots, N.
\end{aligned}
$$

 which can be rewritten in the following equivalent form,

$$
\begin{equation}
\begin{aligned}
& \underset{\beta_0,\boldsymbol{\beta}}{\text{minimize}}
& & \frac{1}{2}\lVert\boldsymbol{\beta}\rVert + C \sum \xi_i \\\
& \text{subject to:}
& & y_i(\beta_0+\boldsymbol{\beta}^T \mathbf{x}_i) \geq 1-\xi_i, \xi_i \geq 0 \; i = 1, \ldots, N.
\end{aligned}
\end{equation}
\label{eq:svm} \tag{1}
$$

 Because the $\xi_i$ terms are all positive, the objective
is to maximize the margin (i.e., minimize $\lVert\boldsymbol{\beta}\rVert$)
while minimizing the proportional drift of the predictions to the wrong side
of the margin (i.e., $C \sum \xi_i$). Thus, large values of $C$ shunt
algorithmic focus towards the correctly classified points near the
decision boundary and small values focus on further data. The value $C$ is
a hyperparameter for the SVM.

The good news is that all of these complicated pieces are handled neatly inside
of Scikit-learn. The following sets up the linear *kernel* for the SVM (more on
kernels soon),

```python
from sklearn.datasets import make_blobs
from sklearn.svm import SVC
sv = SVC(kernel='linear')
```

 We can create some synthetic data using `make_blobs` and then
fit it to the SVM,

```python
X,y=make_blobs(n_samples=200, centers=2, n_features=2,
               random_state=0,cluster_std=.5)
sv.fit(X,y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
        decision_function_shape='ovr', degree=3, gamma='auto_deprecated',
        kernel='linear', max_iter=-1, probability=False, random_state=None,
        shrinking=True, tol=0.001, verbose=False)

 After fitting, the SVM now has the estimated support vectors and the
coefficients of the $\boldsymbol{\beta}$  in the `sv.support_vectors_` and
`sv.coef_` attributes, respectively. [Figure](#fig:svm_003) shows the two
sample classes (white and gray circles) and the line separating them that was
found by the maximal margin algorithm. The two parallel dotted lines show  the
margin. The large circles enclose the support vectors, which are the data
elements that are relevant to the solution. Notice that only these elements
can touch the edges of the margins.

```python
%matplotlib inline

from matplotlib.pylab import subplots
import numpy as np
xi = np.linspace(X[:,0].min(),X[:,0].max(),100)

fig,ax=subplots()
_=ax.scatter(X[:,0],X[:,1],c=y,s=50,cmap='gray',marker='o',alpha=.3)
_=ax.plot(sv.support_vectors_[:,0],sv.support_vectors_[:,1],'ko',markersize=20,alpha=.2)
_=ax.plot(xi,-sv.coef_[0,0]/sv.coef_[0,1]*xi- sv.intercept_/sv.coef_[0,1],'k',lw=3.)
margin = np.linalg.norm(sv.coef_)
_=ax.plot(xi,-sv.coef_[0,0]/sv.coef_[0,1]*xi-(sv.intercept_+margin/2.)/sv.coef_[0,1],'--k',lw=3.)
_=ax.plot(xi,-sv.coef_[0,0]/sv.coef_[0,1]*xi-(sv.intercept_-margin/2.)/sv.coef_[0,1],'--k',lw=3.)
fig.savefig('fig-machine_learning/svm_003.png')
```

The two class shown (white and gray circles) are linearly separable. The maximal margin solution is shown by the dark black line in the middle. The dotted lines show the extent of the margin.  The large circles indicate the support vectors for the maximal margin solution.

```python

def draw_margins(sv,X,y,ax=None):
    sv.fit(X,y)
    xi = np.linspace(X[:,0].min(),X[:,0].max(),100)
    if ax is None: fig,ax=subplots()
    _=ax.scatter(X[:,0],X[:,1],c=y,s=50,cmap='gray',marker='o',alpha=.3)
    _=ax.plot(sv.support_vectors_[:,0],sv.support_vectors_[:,1],'ko',markersize=20,alpha=.2)
    _=ax.plot(xi,-sv.coef_[0,0]/sv.coef_[0,1]*xi- sv.intercept_/sv.coef_[0,1],'k',lw=3.)
    margin = np.linalg.norm(sv.coef_)
    _=ax.plot(xi,-sv.coef_[0,0]/sv.coef_[0,1]*xi- (sv.intercept_+margin/2.)/sv.coef_[0,1],'--k',lw=3.)
    _=ax.plot(xi,-sv.coef_[0,0]/sv.coef_[0,1]*xi- (sv.intercept_-margin/2.)/sv.coef_[0,1],'--k',lw=3.)
```

```python

X, y = make_blobs(n_samples=50, centers=2, n_features=2,
                  cluster_std=1,random_state=0)

fig,axs = subplots(2,2,sharex=True,sharey=True)
#fig.set_size_inches((12,6))
sv = SVC(kernel='linear',C=.0100)
draw_margins(sv,X,y,ax=axs[0,0])
_=axs[0,0].set_title('C=0.01')
sv = SVC(kernel='linear',C=1)
draw_margins(sv,X,y,ax=axs[0,1])
_=axs[0,1].set_title('C=1')
sv = SVC(kernel='linear',C=100)
draw_margins(sv,X,y,ax=axs[1,0])
_=axs[1,0].set_title('C=100')
sv = SVC(kernel='linear',C=10000)
draw_margins(sv,X,y,ax=axs[1,1])
_=axs[1,1].set_title('C=10000')
fig.savefig('fig-machine_learning/svm_004.png')
```

[Figure](#fig:svm_004) shows what happens when the value of $C$ changes.
Increasing this value emphasizes the $\xi$ part of the objective function in
Equation [1](#eq:svm). As shown in the top left panel, a small value for $C$ means that
the algorithm is willing to accept many support vectors at the expense of
maximizing the margin. That is, the proportional amount that predictions are on
the wrong side of the margin is more acceptable with smaller $C$.  As the value
of $C$ increases, there are fewer support vectors because the optimization
process prefers to eliminate support vectors that are far away from the margins
and accept fewer of these that encroach into the margin. Note that as the value
of $C$ progresses through this figure, the separating line tilts slightly.

The maximal margin algorithm finds the separating line that maximizes the margin shown. The elements that touch the margins are the support elements. The dotted elements are not relevant to the solution.

## Kernel Tricks

Support Vector Machines provide a powerful method to deal with linear
separations, but they can also apply to non-linear boundaries by
exploiting the so-called *kernel trick*.  The convex optimization
formulation of the SVM includes a *dual* formulation that leads to a
solution that requires only the inner-products of the features. The
kernel trick is to substitute inner-products by nonlinear kernel
functions.  This can be thought of as mapping the original features
onto a possibly infinite dimensional space of new features.  That is,
if the data are not linearly separable in two-dimensional space (for
example) maybe they are separable in three-dimensional space (or
higher)?

To make this concrete, suppose the original input space is
$\mathbb{R}^n$ and we want to use a non-linear mapping
$\psi:\mathbf{x} \mapsto \mathcal{F}$ where $\mathcal{F}$ is an
inner-product space of higher dimension.  The kernel trick is to
calculate the inner-product in $\mathcal{F}$ using a kernel
function, $K(\mathbf{x}_i,\mathbf{x}_j) = \langle
\psi(\mathbf{x}_i),\psi(\mathbf{x}_j)\rangle$. The long way to
compute this is to first compute $\psi(\mathbf{x})$ and then do the
inner-product. The kernel-trick way to do it is to use the kernel
function and avoid computing $\psi$. In other words, the kernel
function returns what the inner-product in $\mathcal{F}$ would have
returned if $\psi$ had been applied. For example, to achieve an
$n^{th}$ polynomial mapping of the input space, we can use
$\kappa(\mathbf{x}_i,\mathbf{x}_j)=(\mathbf{x}_i^T\mathbf{x}_j+\theta)^n$.
For example, suppose the input space is $\mathbb{R}^2$ and
$\mathcal{F}=\mathbb{R}^4$ and we have the following mapping,

$$
\psi(\mathbf{x}) : (x_0,x_1) \mapsto (x_0^2,x_1^2,x_0 x_1, x_1 x_0)
$$

 The inner product in $\mathcal{F}$ is then,

$$
\langle \psi(\mathbf{x}),\psi(\mathbf{y})  \rangle = \langle \mathbf{x},\mathbf{y}  \rangle^2
$$

 In other words, the kernel is the square of the inner
product in input space. The advantage of using the kernel instead of
simply enlarging the feature space is computational because you only
need to compute the kernel on all distinct pairs of the input space.
The following example should help make this concrete. First we create
some Sympy variables,

```python
import sympy as S
x0,x1=S.symbols('x:2',real=True)
y0,y1=S.symbols('y:2',real=True)
```

 Next, we create the $\psi$ function that maps into $\mathbb{R}^4$
and the corresponding kernel function,

```python
psi = lambda x,y: (x**2,y**2,x*y,x*y)
kern = lambda x,y: S.Matrix(x).dot(y)**2
```

 Notice that the inner product in $\mathbb{R}^4$  is
equal to the kernel function, which only uses wthe $\mathbb{R}^2$
variables.

```python
print(S.Matrix(psi(x0,x1)).dot(psi(y0,y1)))
print(S.expand(kern((x0,x1),(y0,y1)))) # same as above
```

    x0**2*y0**2 + 2*x0*x1*y0*y1 + x1**2*y1**2
    x0**2*y0**2 + 2*x0*x1*y0*y1 + x1**2*y1**2

**Polynomial Regression Using Kernels.**  Recall our favorite
linear regression problem from the regularization chapter,

$$
\min_{\boldsymbol{\beta}}\Vert y -\mathbf{X}\boldsymbol{\beta}\Vert^2
$$

 where $\mathbf{X}$ is a $n\times m$ matrix with $m>n$. As
we discussed, there are multiple solutions to this problem. The
least-squares solution is the following:

$$
\boldsymbol{\beta}_{LS}=\mathbf{X}^T(\mathbf{X}\mathbf{X}^T)^{\text{-1}}\mathbf{y}
$$

 Given a new feature vector $\mathbf{x}$, the corresponding estimator
for $\mathbf{y}$ is the following,

$$
\hat{\mathbf{y}} = \mathbf{x}^T\boldsymbol{\beta}_{LS}=\mathbf{x}^T\mathbf{X}^T(\mathbf{X}\mathbf{X}^T)^{\text{-1}}\mathbf{y}
$$

 Using the kernel trick, the solution can be written more generally as
the following,

$$
\hat{\mathbf{y}}=\mathbf{k}(\mathbf{x})^T\mathbf{K}^{\text{-1}}\mathbf{y}
$$

 where the $n\times n$ kernel matrix $\mathbf{K}$ replaces
$\mathbf{X}\mathbf{X}^T$ and where $\mathbf{k}(\mathbf{x})$ is a $n$-vector of
components $\mathbf{k}(\mathbf{x})=[\kappa(\mathbf{x}_i,\mathbf{x})]$ and where
$\mathbf{K}_{i,j}=\kappa(\mathbf{x}_i,\mathbf{x}_j)$ for the kernel function
$\kappa$.  With this more general setup, we can substitute
$\kappa(\mathbf{x}_i,\mathbf{x}_j)=(\mathbf{x}_i^T\mathbf{x}_j+\theta)^n$ for
$n^{th}$-order polynomial regression [[bauckhagenumpy]](#bauckhagenumpy). Note that ridge
regression can also be incorporated by inverting $(\mathbf{K}+\alpha
\mathbf{I})$, which can help stabilize poorly conditioned $\mathbf{K}$ matrices
with a tunable $\alpha$ hyper-parameter [[bauckhagenumpy]](#bauckhagenumpy).

For some kernels, the enlarged $\mathcal{F}$ space is infinite-dimensional.
Mercer's conditions provide technical restrictions on the kernel functions.
Powerful and well-studied kernels have been implemented in Scikit-learn. The
advantage of kernel functions may evaporate for when $n\rightarrow m$ in which
case using the $\psi$ functions instead can be more practicable.

```python
from matplotlib.pylab import cm
xi = np.linspace(X[:,0].min(),X[:,0].max(),100)
yi = np.linspace(X[:,1].min(),X[:,1].max(),100)

fig,ax=subplots()
_=ax.scatter(X[:,0],X[:,1],c=y,s=50,cmap='gray',marker='o',alpha=.3)
Xi,Yi = np.meshgrid(xi,yi)
Zi=sv.predict(np.c_[Xi.ravel(),Yi.ravel()]).reshape(Xi.shape)

_=ax.contourf(Xi,Yi,Zi,cmap=cm.Paired,alpha=0.2);
fig.savefig('fig-machine_learning/svm_005.png')
```


# K-means  Clustering and Image Sharpening

In this exercise, you are asked to approximate the depth of field (DOF) effect on an
image. The depth of field is the distance between the nearest and farthest objects in a
scene which appear acceptably sharp in an image. When the DOF is small, the object
of interest (i.e., focused object) appears sharp and the rest appears blurred. This effect
is usually employed for artistic reasons and is achieved by having a large aperture while
capturing an image.

In our case, the image has already been captured, and we are trying to simulate a similar
effect. The task is to first segment the image into 3 areas (front/middle/background
areas), and then to sharpen the object of interest while blurring the rest.

In our case in addition to the RGB image, a depth image has also been captured.
The intensity of each pixel in the depth image correspond to the inverse distance
between the camera and the objects of the scene (up to a scale). Instead of finding the
correct segmentation of the image, which is a difficult task, we are going to perform the
segmentation on the depth image. The depth image, however, is usually a bit noisy, so a
simple thresholding does not produce good results in most cases, thus you will need to
implement a k-means segmentation algorithm.

In this exercise, you are asked to:
* implement the k-means algorithm to segment the depth image into three disjoint
regions. This will produce a label image where the three objects (doll, hand,
background) have a different label.
* generate three binary images each focusing on a different object.
* sharpen the focused object and smooth the rest, using these binary images and
appropriate filters.

<figure>

    <figcaption>Figure 1: Pair of images, captured by Microsoft Kinect. The image on the left depicts the RGB image
and the one on the right shows the corresponding depth image. </figcaption>
</figure>

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import cv2
import math
import scipy.signal as conv
```

```python
# Read the image
img = cv2.imread('images/rgb.png',0)
depth_img = cv2.imread('images/depth_noisy.png',0)

# Display the images
plt.figure(1)
plt.imshow(img,cmap='gray')
plt.figure(2)
plt.imshow(depth_img,cmap='gray')

depth_img = np.float64(depth_img)*4; # rescaled for better results
```

## Exercise 1: Depth-based Segmentation

To start with, You are asked to implement the function ```labelImage = clusterImage(depthImage,max_iters)``` by the following the steps below:
* For every pixel $i, j: 0 ≤ i < H, 0 ≤ j < W$ of the depth image, compute the Euclidean
distance between its depth $D[i,j]$ and each cluster mean depth $k^l_d$: $dist[i,j,l] = \sqrt{(D[i,j] - {k^{l}}_{d})^2}$ and $l = 0,1,2$.
* For each pixel find the smallest distance and assign this pixel to the
corresponding cluster.
* Compute the new mean of each cluster.
* Iterate over the above two steps till the cluster centroids (mean of the clusters) do not move any more or a fixed number of iterations is reached.

What is the disadvantage of the result? Is the segmentation correct?

#### Inputs:

- ```depthImage```: an image where each value indicates the depth of the corresponding pixel.
- ```max_iters```: the maximum number of iterations

#### Outputs:

- ```labelImage```:  output label image where each value indicates to which cluster the corresponding pixels belongs.
- There are three clusters: value 0 for the background, value 1 for the hand and value 2 for the doll.

```python
def clusterImage(depthImage, max_iters):

    k = 3  # Number of clusters
    height, width = depthImage.shape  # Size of depth image
    count = 0  # Iteration count
    distance = [None] * 3  # distance container in iteration

    # initiate centroids
    # we create 3 centroids, in the format [Z X Y], where Z in the inverse depth
    mu = np.zeros((k,3))
    mu[0,:] = [0, 0, 0]  # background centroid
    mu[1,:] = [500, 300, 150]  # hand centroid
    mu[2,:] = [1000, 100, 200]  # doll centroid
    prev_mu = mu + 50  # initialize iteration
    ### Note: For Exercise 1, only Z (inverse) depth is used. X,Y is used in Exercise 2.

    # Create X and Y matrices to store coordinate information
    X, Y = np.meshgrid(np.arange(0, width), np.arange(0, height))
    # print(distance)

    # Iteratively calculate distance between depth and update cluster mean
    while ((count < max_iters) and np.sum((prev_mu - mu) > 0)):
        print("Iteration: {}".format(count))

        # calculate depth distance map for depth image of all clusters
        for k in range(3):
            distance[k] = np.sqrt((depthImage - mu[k,0]) ** 2)

        # assign pixels to the cluster that has min distance
        cluster_assignments = np.argmin(np.array(distance), axis=0)

        # store old cluster mean and update new cluster mean
        prev_mu = np.copy(mu)
        for k in range(3):
            mu[k,:] = np.array([np.mean(depthImage[cluster_assignments == k]), np.mean(X[cluster_assignments == k]), np.mean(Y[cluster_assignments == k])])

        count += 1

    return cluster_assignments
```

```python
cluster_assignments = clusterImage(depth_img, 5)
plt.imshow(cluster_assignments)
```

## Exercise 2: Using Location Information
As you saw in the previous exercise, sometimes the depth information is not enough to correctly segment the image. This can be fixed by taking into account also **the spatial proximity of the pixels** - pixels from the same cluster should be close to each other in the image. This can be
accomplished by adding 2 dimensions to the original depth image representing the i and
j coordinates of each pixel.

You are asked to implement a new function ```labelImage = clusterImage(depthImage,max_iters)``` following the steps:
* For every pixel $i, j: 0 ≤ i < H, 0 ≤ j < W$ of the depth image, compute the Euclidean
distance between its combined depth $D[i,j]$ and position $[i,j]$ and each cluster mean $[k^l_d,k^l_i,k^l_j]$: $dist[i,j,l] = \sqrt{(D[i,j] - {k^{l}}_{d})^2 + (i-k^{l}_{i})^2) + (j-k^{l}_{j})^2)}$ for $l = 0,1,2$.

* For each pixel find the smallest distance and assign this pixel to the
corresponding cluster.
* Compute the new mean of each cluster.
* Iterate over the above two steps till the cluster centroids (mean of the clusters)
do not move any more or a fixed number of iterations is reached.

How does addition of spatial information improve the segmentation results?

#### Inputs:

- ```depthImage```: an image where each value indicates the depth of the corresponding pixel.
- ```max_iters```: maximum number of iterations

#### Outputs:

- ```labelImage```:  output label image where each value indicates to which cluster the corresponding pixels belongs. There are three clusters: value 0 for the background, value 1 for the hand and value 2 for the doll.

```python
def clusterImage(depthImage, max_iters):

    k = 3 # Number of clusters
    height, width = depthImage.shape  # Size of depth image
    count = 0  # Iteration count
    distance = [None] * 3  # distance container in iteration

    #initiate centroids
    #we create 3 centroids, in the format [Z X Y], where Z in the inverse depth
    mu = np.zeros((k,3))
    mu[0,:] = [0, 0, 0] #background centroid
    mu[1,:] = [500, 300, 150] #hand centroid
    mu[2,:] = [1000, 100, 200] #doll centroid
    prev_mu = mu + 50  # initialize iteration

    # Create X and Y matrices to store coordinate information
    X, Y = np.meshgrid(np.arange(0, width), np.arange(0, height))
    # print(distance)

    # Iteratively calculate distance between depth and update cluster mean
    while ((count < max_iters) and np.sum((prev_mu - mu) > 0)):
        print("Iteration: {}".format(count))

        # calculate xyz distance map for depth image of all clusters
        for k in range(3):
            distance[k] = np.sqrt((depthImage - mu[k,0]) ** 2 + (X - mu[k,1]) ** 2 + (Y - mu[k,2]) ** 2)

        # assign pixels to the cluster that has min distance
        cluster_assignments = np.argmin(np.array(distance), axis=0)

        # store old cluster mean and update new cluster mean
        prev_mu = np.copy(mu)
        for k in range(3):
            mu[k,:] = np.array([np.mean(depthImage[cluster_assignments == k]), np.mean(X[cluster_assignments == k]), np.mean(Y[cluster_assignments == k])])

        count += 1

    return cluster_assignments

cluster_assignments = clusterImage(depth_img,5)
plt.imshow(cluster_assignments)
```

## Exercise 3: Image Sharpening

We further need to implement a region-based filtering function to sharpen the object of interest and blur the rest of
the image. In the previous parts you produced a label image (with each object assigned a different label) and produced three binary images corresponding to different objects. Now **the task is to sharpen the focused regions by applying a Laplacian filter and smooth the other regions using a Gaussian filter**. It should produce results as shown below.

To this end we introduce a Laplacian filter parametrized by $\alpha$ defined as:

\begin{align}
L = \frac{\partial ^2}{\partial x^2} + \frac{\partial^2}{\partial y^2} = \frac{4}{\alpha+1} \begin{bmatrix} \frac{\alpha}{4} & \frac{1-\alpha}{4} & \frac{\alpha}{4} \\
\frac{1-\alpha}{4} & -1 & \frac{1-\alpha}{4} \\
\frac{\alpha}{4} & \frac{1-\alpha}{4} & \frac{\alpha}{4} \\
\end{bmatrix}
\end{align}

Then, the sharpening filter is defined as $\delta - L$, where $\delta$ is a $3\times3$ Kronecker delta kernel with a values of 1 at the diagonal elements and zero elsewhere. The Gaussian kernel is parametrized by standard deviation $\sigma$.

* Implement functions
``` s = SharpeningKernel(alpha) ``` and ``` g = GaussianKernel(sigma, sz)``` that compute sharpening filter of size $3\times3$ and gaussian kernel of size $sz\times sz$ respectively (you computed gaussian kernel already in Exercise Session 2).

```python
def SharpeningKernel(alpha):
    l = np.ones((3, 3))

    ### YOUR CODE HERE
    l[0,:] = np.array([alpha/4, (1-alpha)/4, alpha/4])
    l[1,:] = np.array([(1-alpha)/4, -1, (1-alpha)/4])
    l[2,:] = np.array([alpha/4, (1-alpha)/4, alpha/4])

    l = l * 4 / (1+alpha)

    delta = np.zeros((3, 3))
    delta[1,1] = 1

    s = delta - l

    return s

### We implemented gaussian already multiple times,
### so you can feel free to use the following snippet:
def GaussianKernel(sigma,sz):
    x, y = np.mgrid[-sz//2 + 1:sz//2 + 1, -sz//2 + 1:sz//2 + 1]
    g = np.exp(-((x**2 + y**2)/(2.0*sigma**2)))
    g = g/g.sum()

    return g
```

* Implement a function ``` filteredImage = FilterRegions(origImage,binaryForeground, alpha, sigma)``` that smooths the background and sharpens the objects in the foreground.

(You can use `conv.convolve2d` function to do convolution)

#### Outputs:
* ```filteredImage``` - original image with sharpened foreground and smoothed background

#### Inputs:
* ```origImage``` - original image
* ```binaryForeground``` - binary mask with foreground pixels value equal to 1
* ```alpha``` - sharpening parameter
* ```sigma``` - smoothing parameter

```python
def FilterRegions(origImage, binaryForeground, alpha, sigma, sz):
    ### YOUR CODE HERE
    s = SharpeningKernel(alpha)
    g = GaussianKernel(sigma, sz)

    filteredImage = conv.convolve2d(origImage, g, mode='same')
    sharpenedImage = conv.convolve2d(origImage, s, mode='same')

    filteredImage[binaryForeground == 1] = sharpenedImage[binaryForeground == 1]

    return filteredImage
```

* Visualize 3 images by focusing on different objects detected in the image. Use sharpnening filter of size $3\times3$ and $\alpha = 0.5$ and Gaussian kernel of size $7\times7$ and $\sigma = 2.0$.

```python
for i in range(np.amax(cluster_assignments) + 1):
    binaryImg = cluster_assignments == i
    filteredImg = FilterRegions(img,binaryImg, 0.5, 2, 7)
    plt.figure()
    plt.imshow(filteredImg,cmap='gray')
```

```python

```

Image Segmentation & Sharpening/kmeans_sharpening.ipynb


```python
from sympy import *
```

求解方程组
\left(\frac{dx}{dy}\right)^{1/3}$$

在编写LaTeX时，有两种模式：行内模式(inline mode) 和 展示模式(display mode).行内模式也称为随文模式，是将公式镶嵌在普通文段内，跟文字排在一起。 展示模式是将公式独立成一行，并且居中显示。
模式不同，数学公式的渲染方式也不同。
为了在行内编写数学公式，你需要使用单$符, 比如：$y=mx+b$，在 $符之间的所有内容，都将被视为数学符号进行渲染。
为了创建显示模式的表达式，使用 $$符。
$$P(A \mid B) = \frac{ P(B \mid A) P(A) }{ P(B) }$$

求解方程组
$$2x-y-3=0$$
$$3x+y=0$$

```python
from sympy import *
x,y = symbols('x y')
# 使用solve()函数求解x，y的值
result = solve((2*x-y-3,3*x+y),(x,y))
print(result)
```

    {x: 3/5, y: -9/5}

例如需要求以下的极限：

$$ \lim_{x \to +\infty} \frac{x^2}{e^x}$$

```python

x = symbols('x')
expr = x**2/exp(x)
# 无穷用两个o表示
result = limit(expr,x,oo)
print('被求函数为：')
print(expr)
print('结果为：')
pprint(result)
```

    被求函数为：
    x**2*exp(-x)
    结果为：
    0

diff()函数也可以用来求偏导，例如求以下的偏导：
$$f(x,y) =x^2sin(2y)$$

```python
"""求偏导"""
# 定义一个函数f，它是x，y的函数
f = Function('f')(x,y)
f = x**2 * sin(2*y) # 对x求偏导
fx = f.diff(x)
 # 对y求偏导
fy = f.diff(y)
print('所求函数为：')
pprint(f)
print('对x偏导为：')
pprint(fx)
print('对y偏导为：')
pprint(fy)
```

    所求函数为：
     2
    x ⋅sin(2⋅y)
    对x偏导为：
    2⋅x⋅sin(2⋅y)
    对y偏导为：
       2
    2⋅x ⋅cos(2⋅y)

不定积分与定积分

在数学上不定积分的表达式如下：
$$ \int f(x) dx $$
抛开数学上求解不定积分的方法，我们无需关心怎么算，只需关系怎么让计算机帮我们算。例如需要计算以下的不定积分：

$$ \int \frac {x^2} {x^2+1} dx $$

```python
 """求不定积分"""
 x = symbols('x')
 expr = (x**2)/(x**2 + 1)
 result = integrate(expr,x)
 print("被积函数为：")
 pprint(expr)
 print('不定积分结果为：')
 pprint(result)
```

    被积函数为：
       2
      x
    ──────
     2
    x  + 1
    不定积分结果为：
    x - atan(x)

接下来看定积分，定积分比不定积分多了个范围限制，比如求下面的定积分：
已知$f(x)=e^{-x^2}$,求解$\int_0^1 {f'(x)f''(x)} \,{\rm d}x$

```python
 f = Function('f')
 x = symbols('x')
 f = exp(-x**2)
 expr_1 = diff(f,x)
 expr_2 = diff(f,x,2)
 expr = expr_1 * expr_2
 result = integrate(expr,(x,0,1))
 print('被积函数为：')
 pprint(expr)
 print('结果为：')
 pprint(result)
```

    被积函数为：
                         2
         ⎛   2    ⎞  -2⋅x
    -4⋅x⋅⎝2⋅x  - 1⎠⋅ℯ
    结果为：
       -2
    2⋅ℯ

常微分方程与偏微分方程

常微分方程是指微分方程中只有一个变量的方程，解常微分方程需要用到dsolve()函数。例如需要求以下的常微分方程：

$$y''+2y'+3y=0$$

```python
"""常微分方程"""
f = Function('f')
x = symbols('x')
expr = diff(f(x),x,2) + 2*diff(f(x),x) + 3*f(x)
result = dsolve(expr,f(x))
pprint(result)

```

                                          -x
    f(x) = (C₁⋅sin(√2⋅x) + C₂⋅cos(√2⋅x))⋅ℯ

偏微分方程是指微分方程中含有多个变量的方程，解偏微分方程用到pdsolve()函数。例如需要求以下偏微分方程：

Z是X,Y的函数，求解偏微分方程
\frac{\partial^{2}y}{\partial x^{2}}
$$\frac{3\frac{\partial z}{\partial y}}{z}+\frac{2\frac{\partial z}{\partial x}}{z}+1=0$$

```python
 """偏微分方程求解"""
 f = Function('f')
 # 表示z为x，y的函数
 z = f(x,y)
 zx = z.diff(x)
 zy = z.diff(y)
 eq = Eq(1 + (2*(zx/z)) + (3*(zy/z)))

 result = pdsolve(eq)
 print('微分方程为：')
 pprint(eq)
 print('求得函数为：')
 pprint(result)
```

    微分方程为：
          ∂               ∂
        2⋅──(f(x, y))   3⋅──(f(x, y))
          ∂x              ∂y
    1 + ───────────── + ───────────── = 0
           f(x, y)         f(x, y)
    求得函数为：
                              2⋅x   3⋅y
                            - ─── - ───
                               13    13
    f(x, y) = F(3⋅x - 2⋅y)⋅ℯ

对数 log()
e的指数次幂 exp()
上面是求一阶导的方法，求解高阶导的方法如下所示：

```python
a=diff(x**3,x)
print(a)
a=diff(x**3,x,1)
print(a)
a=diff(x**3,x,2)
print(a)
a=diff(x**3,x,3)
print(a)
```

    3*x**2
    3*x**2
    6*x
    6

用dsolve函数解微分方程
dsolve函数是用来解决微分方程（differential equation）的函数。

```python
f = Function('f')
x = Symbol('x')
pprint(diff(f(x),x)-2*f(x)*x)
print(dsolve(diff(f(x),x)-2*f(x)*x,f(x)))
```

                d
    -2⋅x⋅f(x) + ──(f(x))
                dx
    Eq(f(x), C1*exp(x**2))

欧拉恒等式$$e^{i\pi}+1=0$$
此公式被称为欧拉恒等式，其中e是自然 常数，i是虚数单位， 是圆周率。此公式被誉 为数学中最奇妙的公式，它将5个基本数学常数 用加法、乘法和幂运算联系起来。 从SymPy库载入的符号中，E表示自然常 数，I表示虚数单位，pi表示圆周率，因此上面 的公式可以直接如下计算：

```python
print(E**(I*pi)+1)
```

    0

SymPy除了可以直接计算公式的值之外， 还可以帮助做数学公式的推导和证明。欧拉恒等 式可以将 代入下面的欧拉公式得到：
$$e^{ix}=cosx+isinx$$
在SymPy中可以使用expand()将表达式展 开e^ix，用它展开看（expand()中x是复数）：

```python
print(expand(exp(I*x), complex=True) )
```

    I*exp(-im(x))*sin(re(x)) + exp(-im(x))*cos(re(x))

为了指定x为实数，需要重新定义x

x = Symbol("x", real=True)

print(expand(exp(I*x), complex=True))

```python
x = Symbol("x", real=True)

print(expand(exp(I*x), complex=True))
```

    I*sin(x) + cos(x)

```python

```


```python
from sympy import *
init_printing()

from vierfeldertafel_v01 import Vierfelder_Tafel
```

# Beispiel:

| Wahrscheinlichkeit           | Variable     |
|----------------------------- |--------------|
|$P(A) = 0.5$                  | `a1=0.5`,    |
|$P_A(B) = 0.25$               | `b1_a1=0.25`,|
|$P_{\bar{A}}(\bar{B}) = 0.40$ | `b2_a2=0.40` |

```python
v = Vierfelder_Tafel(
    a1 = 0.5,
    b1_a1 = 0.25,
    b2_a2 = 0.40
)

v.anzahl_loesungen
```

```python
v.tafel(digits=5)
```

```python
v.tree_a(digits=5)
```

```python
v.tree_b(digits=5)
```

```python
v.get_value(v.b2_a1) # P_A(B̄)
```

**Häufig benötigt:**

Ā B̄ ∩

```python
print('A\u0304, B\u0304, \u2229')
```