# Free-Body Diagram for particles

> Renato Naville Watanabe
> [Laboratory of Biomechanics and Motor Control](http://pesquisa.ufabc.edu.br/bmclab)
> Federal University of ABC, Brazil

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set_context('notebook', font_scale=1.2)
```

## Free-Body Diagram

In the mechanical modeling of an inanimate or living system, composed by one or more bodies (bodies as units that are mechanically isolated according to the question one is trying to answer), it is convenient to isolate each body (be they originally interconnected or not) and identify each force and moment of force (torque) that act on this body in order to apply the laws of mechanics.

**The free body diagram (FBD) of a mechanical system or model is the representation in a diagram of all forces and moments of force acting on each body, isolated from the rest of the system.**

The term free means that each body, which maybe was part of a connected system, is represented as isolated (free) and any existent contact force is represented in the diagram as forces (action and reaction) acting on the formerly connected bodies. Then, the laws of mechanics are applied on each body, and the unknown movement, force or moment of force can be found if the system of equations is determined (the number of unknown variables can not be greater than the number of equations for each body).

How exactly a FBD is drawn for a mechanical model of something is dependent on what one is trying to find. For example, the air resistance might be neglected or not when modeling the movement of an object and the number of parts the system is divided is dependent on what is needed to know about the model.

The use of FBD is very common in biomechanics; a typical use is to use the FBD in order to determine the forces and torques on the ankle, knee, and hip joints of the lower limb (foot, leg, and thigh) during locomotion, and the FBD can be applied to any problem where the laws of mechanics are needed to solve a problem.

For now, let's study how to draw free-body diagrams for systems that can be modeled as particles.

### Steps to draw a free-body diagram (FBD)

1. Draw separately each object considered in the problem. How you separate depends on what questions you want to answer.
2. Identify the forces acting on each object. If you are analyzing more than one object, remember the Newton's third Law (action and reaction), and identify where the reaction of a force is being applied.
3. Draw all the identified forces, representing them as vectors. The vectors should be represented with the origin in the object. In the case of particles, the origin should be in the center of the particle.
4. If necessary, you should represent the reference frame in the free-body diagram.
5. After this, you can solve the problem using the Newton's second Law (see, e.g, [Newton's Laws](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/Notebooks/newtonLawForParticles.ipynb)) to find the motion of the particle.

## Basic element and forces

### Gravity

The gravity force acts on two masses, each one atracting each other:

\begin{equation}
\vec{{\bf{F}}} = - G\frac{m_1m_2}{||\vec{\bf{r}}||^2}\frac{\vec{\bf{r}}}{||\vec{\bf{r}}||}
\end{equation}

where $G = 6.67.10^{−11} Nm^2/kg^2$ and $\vec{\bf{r}}$ is a vector with length equal to the distance between the masses and directing towards the other mass. Note the forces acting on each mass have the same absolute value.

Since the mass of the Earth is $m_1=5.9736×10^{24}kg$ and its radius is 6.371×10$^6$ m, the gravity force near the surface of the Earth is:

\begin{equation}
\vec{{\bf{F}}} = m\vec{\bf{g}}
\end{equation}

with the absolute value of $\vec{\bf{g}}$ approximately equal to 9.81 $m/s^2$, pointing towards the center of Earth.

### Spring

Spring is an element used to represent a force proportional to some length or displacement. It produces a force in the same direction of the vector linking the spring extremities and opposite to its length or displacement from an equilibrium length. Frequently it has a linear relation, but it could be nonlinear as well. The force exerted by the spring in one of the extremities is:

\begin{equation}
\vec{\bf{F}} = - k(||\vec{\bf{r}}||-l_0)\frac{\vec{\bf{r}}}{||\vec{\bf{r}}||} = -k\vec{\bf{r}} +kl_0\frac{\vec{\bf{r}}}{||\vec{\bf{r}}||} = -k\left(1-\frac{l_0}{||\vec{\bf{r}}||}\right)\vec{\bf{r}}
\end{equation}

where $\vec{\bf{r}}$ is the vector linking the extremity applying the force to the other extremity and $l_0$ is the equilibrium length of the spring.

Since the spring element is a massless element, the force in  both extremities have the same absolute value and opposite directions.

### Damping

Damper is an element used to represent a force proportional to the velocity of displacement. It produces a force in the opposite direction of its velocity.

Frequently it has a linear relation, but it could be nonlinear as well. The force exerted by the damper element in one of its extremities is:

\begin{equation}
\vec{\bf{F}} = - b||\vec{\bf{v}}||\frac{\vec{\bf{v}}}{||\vec{\bf{v}}||} = -b\vec{\bf{v}} = -b\frac{d\vec{\bf{r}}}{dt}
\end{equation}

where $\vec{\bf{r}}$ is the vector linking the extremity applying the force to the other extremity.

Since the damper element is a massless element, , the force in  both extremities have the same absolute value and opposite directions.

## Examples of free-body diagram

Let's see some examples on how to draw the free-body diagram and obtain the motion equations to solve the problems.

### 1. No force acting on the particle

The most trivial situation is a particle with no force acting on it.

The free-body diagram is below, with no force vectors acting on the particle.

Figure. Free-body diagram of a ball with no force acting on it.

In this situation, the resultant force is:

\begin{equation}
\vec{\bf{F}} = 0
\end{equation}

And the second Newton law for this particle is:

\begin{equation}
m\frac{d^2\vec{\bf{r}}}{dt^2} = 0 \quad \rightarrow \quad \frac{d^2\vec{\bf{r}}}{dt^2} = 0
\end{equation}

The motion of of the particle can be found by integrating twice both times, getting the following:

\begin{equation}
\vec{\bf{r}} = \vec{\bf{v}}_0t + \vec{\bf{r}}_0
\end{equation}

The particle continues to change its position with the same velocity it was at the beginning of the analysis. This could be predicted by Newton's first law.

### 2. Gravity force acting on the particle

Now, let's consider a ball with the gravity force acting on it. The free-body diagram is depicted below.

Figure. Free-body diagram of a ball under the influence of gravity.

The only force acting on the ball is the gravitational force:

\begin{equation}
\vec{\bf{F}}_g = - mg \; \hat{\bf{j}}
\end{equation}

Applying Newton's second Law:

\begin{equation}
\vec{\bf{F}}_g = m \frac{d^2\vec{\bf{r}}}{dt^2} \rightarrow - mg \; \hat{\bf{j}} = m \frac{d^2\vec{\bf{r}}}{dt^2} \rightarrow - g \; \hat{\bf{j}} = \frac{d^2\vec{\bf{r}}}{dt^2}
\end{equation}

Now, we can separate the equation in two components (x and y):

\begin{equation}
0 = \frac{d^2x}{dt^2}
\end{equation}

and

\begin{equation}
- g = \frac{d^2y}{dt^2}
\end{equation}

These equations were solved in [this Notebook about the Newton's laws](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/Notebooks/newtonLawForParticles.ipynb).

### 3. Ground reaction force

Now, we will analyze the situation of a particle at rest in contact with the ground. To simplify the analysis, only the vertical movement will be considered.

Figure. Free-body diagram of a ball at rest in contact with the ground.

The forces acting on the particle are the ground reaction force (often called as normal force) and the gravity force. The free-body diagram of the particle is below:

Figure. Free-body diagram of a ball under the influence of gravity.

So, the resultant force in the particle is:

\begin{equation}
    \vec{\bf{F}} = \overrightarrow{\bf{GRF}} + m\vec{\bf{g}} = \overrightarrow{\bf{GRF}} - mg \; \hat{\bf{j}}
\end{equation}

Considering only the y direction:

\begin{equation}
    F = GRF - mg
\end{equation}

Applying Newton's second law to the particle:

\begin{equation}
    m \frac{d^2y}{dt^2} = GRF - mg
\end{equation}

Note that since we have no information about how the force GRF varies along time, we cannot solve this equation. To find the position of the particle along time, one would have to measure the ground reaction force. See [the notebook on Vertical jump](http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/VerticalJump.ipynb) for an application of this model.

## 4. Mass-spring system with horizontal movement

The example below represents a mass attached to a spring and the other extremity of the spring is fixed.

Figure. Mass-spring system with horizontal movement.

The only force force acting on the mass is from the spring. Below is the free-body diagram from the mass.

Figure. Free-body diagram of a mass-spring system.

Since the movement is horizontal, we can neglect the gravity force.

\begin{equation}
    \vec{\bf{F}} = -k\left(1-\frac{l_0}{||\vec{\bf{r}}||}\right)\vec{\bf{r}}
\end{equation}

Applying Newton's second law to the mass:

\begin{equation}
    m\frac{d^2\vec{\bf{r}}}{dt^2} = -k\left(1-\frac{l_0}{||\vec{\bf{r}}||}\right)\vec{\bf{r}} \rightarrow \frac{d^2\vec{\bf{r}}}{dt^2} = -\frac{k}{m}\left(1-\frac{l_0}{||\vec{\bf{r}}||}\right)\vec{\bf{r}}
\end{equation}

Since the movement is unidimensional, we can deal with it scalarly:

\begin{equation}
    \frac{d^2x}{dt^2} = -\frac{k}{m}\left(1-\frac{l_0}{x}\right)x = -\frac{k}{m}(x-l_0)
\end{equation}

To solve this equation numerically, we must break the equations into two first-order differential equation:

\begin{equation}
    \frac{dv_x}{dt} =  -\frac{k}{m}(x-l_0)
\end{equation}

\begin{equation}
    \frac{dx}{dt} =  v_x
\end{equation}

In the numerical solution below, we will use $k = 40 N/m$, $m = 2 kg$, $l_0 = 0.5 m$ and the mass starts from the position $x = 0.8m$ and at rest.

```python
k = 40
m = 2
l0 = 0.5
x0 = 0.8
v0 = 0

x = x0
v = v0

dt = 0.001
t = np.arange(0, 3, dt)

r = np.array([x])

for i in t[1:]:
    dxdt = v
    dvxdt = -k/m*(x-l0)
    x = x + dt*dxdt
    v = v + dt*dvxdt
    r = np.vstack((r,np.array([x])))

plt.figure(figsize=(8, 4))
plt.plot(t, r, lw=4)
plt.xlabel('t(s)')
plt.ylabel('x(m)')
plt.title('Spring displacement')
plt.show()
```

### 5. Linear spring in bidimensional movement at horizontal plane

This example below represents a system with two masses attached to a spring.
To solve the motion of both masses, we have to draw a free-body diagram for each one of the masses.

Figure. Linear spring in bidimensional movement at horizontal plane.

The only force acting on each mass is the force due to the spring. Since the movement is happening at the horizontal plane, the gravity force can be neglected.

Figure. FBD of linear spring in bidimensional movement at horizontal plane.

So, the forces acting on mass 1 is:

\begin{equation}
    \vec{\bf{F_1}} = k\left(||\vec{\bf{r_2}}-\vec{\bf{r_1}}||-l_0\right)\frac{(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}
\end{equation}

and the forces acting on mass 2 is:

\begin{equation}
    \vec{\bf{F_2}} =k\left(||\vec{\bf{r_2}}-\vec{\bf{r_1}}||-l_0\right)\frac{(\vec{\bf{r_1}}-\vec{\bf{r_2}})}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}
\end{equation}

Applying Newton's second law for the masses:

\begin{equation}
     m_1\frac{d^2\vec{\bf{r_1}}}{dt^2} = k\left(||\vec{\bf{r_2}}-\vec{\bf{r_1}}||-l_0\right)\frac{(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}
    \\
    \frac{d^2\vec{\bf{r_1}}}{dt^2} = -\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_1}}+\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_2}}
    \\
    \frac{d^2x_1\hat{\bf{i}}}{dt^2}+\frac{d^2y_1\hat{\bf{j}}}{dt^2} = -\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_1\hat{\bf{i}}+y_1\hat{\bf{j}})+\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_2\hat{\bf{i}}+y_2\hat{\bf{j}})
\end{equation}

\begin{equation}
     m_2\frac{d^2\vec{\bf{r_2}}}{dt^2} = k\left(||\vec{\bf{r_2}}-\vec{\bf{r_1}}||-l_0\right)\frac{(\vec{\bf{r_1}}-\vec{\bf{r_2}})}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}
    \\
    \frac{d^2\vec{\bf{r_2}}}{dt^2} = -\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_2}}+\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_1}}
    \\
    \frac{d^2x_2\hat{\bf{i}}}{dt^2}+\frac{d^2y_2\hat{\bf{j}}}{dt^2} = -\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_2\hat{\bf{i}}+y_2\hat{\bf{j}})+\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_1\hat{\bf{i}}+y_1\hat{\bf{j}})
\end{equation}

Now, we can separate the equations for each of the coordinates:

\begin{equation}
    \frac{d^2x_1}{dt^2} = -\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)x_1+\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)x_2=-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_1-x_2)
\end{equation}

\begin{equation}
    \frac{d^2y_1}{dt^2} = -\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)y_1+\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)y_2=-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_1-y_2)
\end{equation}

\begin{equation}
    \frac{d^2x_2}{dt^2} = -\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)x_2+\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)x_1=-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_2-x_1)
\end{equation}

\begin{equation}
    \frac{d^2y_2}{dt^2} = -\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)y_2+\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)y_1=-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_2-y_1)
\end{equation}

To solve these equations numerically, you must break these equations into first-order equations:

\begin{equation}
    \frac{dv_{x_1}}{dt} = -\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_1-x_2)
\end{equation}

\begin{equation}
    \frac{dv_{y_1}}{dt} = -\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_1-y_2)
\end{equation}

\begin{equation}
    \frac{dv_{x_2}}{dt} = -\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_2-x_1)
\end{equation}

\begin{equation}
    \frac{dv_{y_2}}{dt} = -\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_2-y_1)
\end{equation}

\begin{equation}
    \frac{dx_1}{dt} = v_{x_1}
\end{equation}

\begin{equation}
    \frac{dy_1}{dt} = v_{y_1}
\end{equation}

\begin{equation}
    \frac{dx_2}{dt} = v_{x_2}
\end{equation}

\begin{equation}
    \frac{dy_2}{dt} = v_{y_2}
\end{equation}

Note that if you did not want to know the details about the motion of each mass, but only the motion of the center of mass of the masses-spring system, you could have modeled the whole system as a single particle.

To solve the equations numerically, we will use the $m_1=1 kg$, $m_2 = 2 kg$, $l_0 = 0.5 m$, $k = 90 N/m$ and $x_{1_0} = 0 m$, $x_{2_0} = 0 m$, $y_{1_0} = 1 m$, $y_{2_0} = -1 m$, $v_{x1_0} = -2 m/s$, $v_{x2_0} = 0 m/s$, $v_{y1_0} = 0 m/s$, $v_{y2_0} = 0 m/s$.

```python
x01 = 0
y01= 0.5
x02 = 0
y02 = -0.5
vx01 = 0.1
vy01 = 0
vx02 = -0.1
vy02 = 0

x1= x01
y1 = y01
x2= x02
y2 = y02
vx1= vx01
vy1 = vy01
vx2= vx02
vy2 = vy02
r1 = np.array([x1,y1])
r2 = np.array([x2,y2])

k = 30
m1 = 1
m2 = 1
l0 = 0.5

dt = 0.0001
t = np.arange(0,5,dt)

for i in t[1:]:

    dvx1dt = -k/m1*(x1-x2)*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))
    dvx2dt = -k/m2*(x2-x1)*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))
    dvy1dt = -k/m1*(y1-y2)*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))
    dvy2dt = -k/m2*(y2-y1)*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))

    dx1dt = vx1
    dx2dt = vx2
    dy1dt = vy1
    dy2dt = vy2

    x1 = x1 + dt*dx1dt
    x2 = x2 + dt*dx2dt
    y1 = y1 + dt*dy1dt
    y2 = y2 + dt*dy2dt

    vx1 = vx1 + dt*dvx1dt
    vx2 = vx2 + dt*dvx2dt
    vy1 = vy1 + dt*dvy1dt
    vy2 = vy2 + dt*dvy2dt

    r1 = np.vstack((r1,np.array([x1,y1])))
    r2 = np.vstack((r2,np.array([x2,y2])))

springLength = np.sqrt((r1[:,0]-r2[:,0])**2+(r1[:,1]-r2[:,1])**2)
plt.figure(figsize=(8, 4))
plt.plot(t, springLength, lw=4)
plt.xlabel('t(s)')
plt.ylabel('Spring length (m)')
plt.show()
```

### 6. Particle under action of gravity and linear air resistance

Below is the free-body diagram of a particle with the gravity force and a linear drag force due to the air resistance.

Figure. Particle under action of gravity and linear air resistance.

the forces being applied in the ball are:

\begin{equation}
\vec{\bf{F}} = -mg \hat{\bf{j}} - b\vec{\bf{v}} = -mg \hat{\bf{j}} - b\frac{d\vec{\bf{r}}}{dt} = -mg \hat{\bf{j}} - b\left(\frac{dx}{dt}\hat{\bf{i}}+\frac{dy}{dt}\hat{\bf{j}}\right) = - b\frac{dx}{dt}\hat{\bf{i}} - \left(mg + b\frac{dy}{dt}\right)\hat{\bf{j}}
\end{equation}

Writing down Newton's second law:

\begin{equation}
\vec{\bf{F}} = m \frac{d^2\vec{\bf{r}}}{dt^2} \rightarrow - b\frac{dx}{dt}\hat{\bf{i}} - \left(mg + b\frac{dy}{dt}\right)\hat{\bf{j}} = m\left(\frac{d^2x}{dt^2}\hat{\bf{i}}+\frac{d^2y}{dt^2}\hat{\bf{j}}\right)
\end{equation}

Now, we can separate into one equation for each coordinate:

\begin{equation}
- b\frac{dx}{dt} = m\frac{d^2x}{dt^2} -\rightarrow \frac{d^2x}{dt^2} = -\frac{b}{m} \frac{dx}{dt}
\end{equation}

\begin{equation}
-mg - b\frac{dy}{dt} = m\frac{d^2y}{dt^2} \rightarrow \frac{d^2y}{dt^2} = -\frac{b}{m}\frac{dy}{dt} - g
\end{equation}

These equations were solved in [this notebook](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/Notebooks/newtonLawForParticles.ipynb).

### 7. Particle under action of gravity and nonlinear air resistance

Below, is the free-body diagram of a particle with the gravity force and a drag force due to the air resistance proportional to the square of the particle velocity.

Figure. Particle under action of gravity and nonlinear air resistance.

The forces being applied in the ball are:

\begin{equation}
\vec{\bf{F}} = -mg \hat{\bf{j}} - bv^2\hat{\bf{e_t}} = -mg \hat{\bf{j}} - b (v_x^2+v_y^2) \frac{v_x\hat{\bf{i}}+v_y\hat{\bf{j}}}{\sqrt{v_x^2+v_y^2}} =   -mg \hat{\bf{j}} - b \sqrt{v_x^2+v_y^2} \,(v_x\hat{\bf{i}}+v_y\hat{\bf{j}}) =  -mg \hat{\bf{j}} - b \sqrt{\left(\frac{dx}{dt} \right)^2+\left(\frac{dy}{dt} \right)^2} \,\left(\frac{dx}{dt} \hat{\bf{i}}+\frac{dy}{dt}\hat{\bf{j}}\right)
\end{equation}

Writing down Newton's second law:

\begin{equation}
\vec{\bf{F}} = m \frac{d^2\vec{\bf{r}}}{dt^2} \rightarrow -mg \hat{\bf{j}} - b \sqrt{\left(\frac{dx}{dt} \right)^2+\left(\frac{dy}{dt} \right)^2} \,\left(\frac{dx}{dt} \hat{\bf{i}}+\frac{dy}{dt}\hat{\bf{j}}\right) = m\left(\frac{d^2x}{dt^2}\hat{\bf{i}}+\frac{d^2y}{dt^2}\hat{\bf{j}}\right)
\end{equation}

Now, we can separate into one equation for each coordinate:

\begin{equation}
- b \sqrt{\left(\frac{dx}{dt} \right)^2+\left(\frac{dy}{dt} \right)^2} \,\frac{dx}{dt} = m\frac{d^2x}{dt^2} \rightarrow \frac{d^2x}{dt^2} = - \frac{b}{m} \sqrt{\left(\frac{dx}{dt} \right)^2+\left(\frac{dy}{dt} \right)^2} \,\frac{dx}{dt}
\end{equation}

\begin{equation}
-mg - b \sqrt{\left(\frac{dx}{dt} \right)^2+\left(\frac{dy}{dt} \right)^2} \,\frac{dy}{dt} = m\frac{d^2y}{dt^2} \rightarrow \frac{d^2y}{dt^2} = - \frac{b}{m} \sqrt{\left(\frac{dx}{dt} \right)^2+\left(\frac{dy}{dt} \right)^2} \,\frac{dy}{dt} -g
\end{equation}

These equations were solved numerically in [this notebook](https://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/Notebooks/newtonLawForParticles.ipynb).

## 8. Linear spring and damping on bidimensional horizontal movement

This situation is very similar to the example of horizontal movement with one spring and two masses, with a damper added in parallel to the spring.

Figure. Linear spring and damping on bidimensional horizontal movement.

Now, the forces acting on each mass are the force due to the spring and the force due to the damper.

Figure. FBD of linear spring and damping on bidimensional horizontal movement.

So, the forces acting on mass 1 is:

\begin{equation}
    \vec{\bf{F_1}} = b\frac{d(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{dt} +  k\left(||\vec{\bf{r_2}}-\vec{\bf{r_1}}||-l_0\right)\frac{(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||} =  b\frac{d(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{dt} +  k\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(\vec{\bf{r_2}}-\vec{\bf{r_1}})
\end{equation}

and the forces acting on mass 2 is:

\begin{equation}
    \vec{\bf{F_2}} = b\frac{d(\vec{\bf{r_1}}-\vec{\bf{r_2}})}{dt} +  k\left(||\vec{\bf{r_2}}-\vec{\bf{r_1}}||-l_0\right)\frac{(\vec{\bf{r_1}}-\vec{\bf{r_2}})}{||\vec{\bf{r_1}}-\vec{\bf{r_2}}||}= b\frac{d(\vec{\bf{r_1}}-\vec{\bf{r_2}})}{dt} +  k\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(\vec{\bf{r_1}}-\vec{\bf{r_2}})
\end{equation}

Applying the Newton's second law for the masses:

\begin{equation}
     m_1\frac{d^2\vec{\bf{r_1}}}{dt^2} = b\frac{d(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{dt}+k\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(\vec{\bf{r_2}}-\vec{\bf{r_1}})
    \end{equation}

\begin{equation}
     \frac{d^2\vec{\bf{r_1}}}{dt^2} = -\frac{b}{m_1}\frac{d\vec{\bf{r_1}}}{dt} -\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_1}} + \frac{b}{m_1}\frac{d\vec{\bf{r_2}}}{dt}+\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_2}}
\end{equation}

\begin{equation}
     \frac{d^2x_1\hat{\bf{i}}}{dt^2}+\frac{d^2y_1\hat{\bf{j}}}{dt^2} = -\frac{b}{m_1}\left(\frac{dx_1\hat{\bf{i}}}{dt}+\frac{dy_1\hat{\bf{j}}}{dt}\right)-\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_1\hat{\bf{i}}+y_1\hat{\bf{j}})+\frac{b}{m_1}\left(\frac{dx_2\hat{\bf{i}}}{dt}+\frac{dy_2\hat{\bf{j}}}{dt}\right)+\frac{k}{m_1}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_2\hat{\bf{i}}+y_2\hat{\bf{j}}) = -\frac{b}{m_1}\left(\frac{dx_1\hat{\bf{i}}}{dt}+\frac{dy_1\hat{\bf{j}}}{dt}-\frac{dx_2\hat{\bf{i}}}{dt}-\frac{dy_2\hat{\bf{j}}}{dt}\right)-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_1\hat{\bf{i}}+y_1\hat{\bf{j}}-x_2\hat{\bf{i}}-y_2\hat{\bf{j}})
\end{equation}

\begin{equation}
     m_2\frac{d^2\vec{\bf{r_2}}}{dt^2} = b\frac{d(\vec{\bf{r_1}}-\vec{\bf{r_2}})}{dt}+k\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(\vec{\bf{r_1}}-\vec{\bf{r_2}})
\end{equation}

\begin{equation}
     \frac{d^2\vec{\bf{r_2}}}{dt^2} = -\frac{b}{m_2}\frac{d\vec{\bf{r_2}}}{dt} -\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_2}} + \frac{b}{m_2}\frac{d\vec{\bf{r_1}}}{dt}+\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)\vec{\bf{r_1}}
\end{equation}

\begin{equation}
    \frac{d^2x_2\hat{\bf{i}}}{dt^2}+\frac{d^2y_2\hat{\bf{j}}}{dt^2} = -\frac{b}{m_2}\left(\frac{dx_2\hat{\bf{i}}}{dt}+\frac{dy_2\hat{\bf{j}}}{dt}\right)-\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_2\hat{\bf{i}}+y_2\hat{\bf{j}})+\frac{b}{m_2}\left(\frac{dx_1\hat{\bf{i}}}{dt}+\frac{dy_1\hat{\bf{j}}}{dt}\right)+\frac{k}{m_2}\left(1-\frac{l_0}{||\vec{\bf{r_2}}-\vec{\bf{r_1}}||}\right)(x_1\hat{\bf{i}}+y_1\hat{\bf{j}})=-\frac{b}{m_2}\left(\frac{dx_2\hat{\bf{i}}}{dt}+\frac{dy_2\hat{\bf{j}}}{dt}-\frac{dx_1\hat{\bf{i}}}{dt}-\frac{dy_1\hat{\bf{j}}}{dt}\right)-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_2\hat{\bf{i}}+y_2\hat{\bf{j}}-x_1\hat{\bf{i}}-y_1\hat{\bf{j}})
\end{equation}

Now, we can separate the equations for each of the coordinates:

\begin{equation}
    \frac{d^2x_1}{dt^2} = -\frac{b}{m_1}\left(\frac{dx_1}{dt}-\frac{dx_2}{dt}\right)-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_1-x_2)
\end{equation}

\begin{equation}
    \frac{d^2y_1}{dt^2} = -\frac{b}{m_1}\left(\frac{dy_1}{dt}-\frac{dy_2}{dt}\right)-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_1-y_2)
\end{equation}

\begin{equation}
    \frac{d^2x_2}{dt^2} = -\frac{b}{m_2}\left(\frac{dx_2}{dt}-\frac{dx_1}{dt}\right)-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_2-x_1)
\end{equation}

\begin{equation}
    \frac{d^2y_2}{dt^2} = -\frac{b}{m_2}\left(\frac{dy_2}{dt}-\frac{dy_1}{dt}\right)-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_2-y_1)
\end{equation}

If you want to solve these equations numerically, you must break these equations into first-order  equations:

\begin{equation}
    \frac{dv_{x_1}}{dt} = -\frac{b}{m_1}\left(v_{x_1}-v_{x_2}\right)-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_1-x_2)
\end{equation}

\begin{equation}
    \frac{dv_{y_1}}{dt} =  -\frac{b}{m_1}\left(v_{y_1}-v_{y_2}\right)-\frac{k}{m_1}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_1-y_2)
\end{equation}

\begin{equation}
    \frac{dv_{x_2}}{dt} = -\frac{b}{m_2}\left(v_{x_2}-v_{x_1}\right)-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(x_2-x_1)
\end{equation}

\begin{equation}
    \frac{dv_{y_2}}{dt} = -\frac{b}{m_2}\left(v_{y_2}-v_{y_1}\right)-\frac{k}{m_2}\left(1-\frac{l_0}{\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}}\right)(y_2-y_1)
\end{equation}

\begin{equation}
    \frac{dx_1}{dt} = v_{x_1}
\end{equation}

\begin{equation}
    \frac{dy_1}{dt} = v_{y_1}
\end{equation}

\begin{equation}
    \frac{dx_2}{dt} = v_{x_2}
\end{equation}

\begin{equation}
    \frac{dy_2}{dt} = v_{y_2}
\end{equation}

To solve the equations numerically, we will use the $m_1=1 kg$, $m_2 = 2 kg$, $l_0 = 0.5 m$, $k = 10 N/m$, $b = 0.6 Ns/m$ and $x_{1_0} = 0 m$, $x_{2_0} = 0 m$, $y_{1_0} = 1 m$, $y_{2_0} = -1 m$, $v_{x1_0} = -2 m/s$, $v_{x2_0} = 1 m/s$, $v_{y1_0} = 0 m/s$, $v_{y2_0} = 0 m/s$.

```python
x01 = 0
y01= 1
x02 = 0
y02 = -1

vx01 = -2
vy01 = 0
vx02 = 1
vy02 = 0

x1= x01
y1 = y01
x2= x02
y2 = y02
vx1= vx01
vy1 = vy01
vx2= vx02
vy2 = vy02
r1 = np.array([x1,y1])
r2 = np.array([x2,y2])

k = 10
m1 = 1
m2 = 2
b = 0.6
l0 = 0.5

dt = 0.001
t = np.arange(0,5,dt)

for i in t[1:]:
    dvx1dt = -b/m1*(vx1-vx2) -k/m1*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))*(x1-x2)
    dvx2dt = -b/m2*(vx2-vx1) -k/m2*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))*(x2-x1)
    dvy1dt = -b/m1*(vy1-vy2) -k/m1*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))*(y1-y2)
    dvy2dt = -b/m2*(vy2-vy1) -k/m2*(1-l0/np.sqrt((x2-x1)**2+(y2-y1)**2))*(y2-y1)
    dx1dt = vx1
    dx2dt = vx2
    dy1dt = vy1
    dy2dt = vy2
    x1 = x1 + dt*dx1dt
    x2 = x2 + dt*dx2dt
    y1 = y1 + dt*dy1dt
    y2 = y2 + dt*dy2dt
    vx1 = vx1 + dt*dvx1dt
    vx2 = vx2 + dt*dvx2dt
    vy1 = vy1 + dt*dvy1dt
    vy2 = vy2 + dt*dvy2dt
    r1 = np.vstack((r1,np.array([x1,y1])))
    r2 = np.vstack((r2,np.array([x2,y2])))

springDampLength = np.sqrt((r1[:,0]-r2[:,0])**2+(r1[:,1]-r2[:,1])**2)

plt.figure(figsize=(8, 4))
plt.plot(t, springDampLength, lw=4)
plt.xlabel('t(s)')
plt.ylabel('Spring length (m)')
plt.show()

plt.figure(figsize=(8, 4))
plt.plot(r1[:,0], r1[:,1], 'r.', lw=4)
plt.plot(r2[:,0], r2[:,1], 'b.', lw=4)
plt.plot((m1*r1[:,0]+m2*r2[:,0])/(m1+m2), (m1*r1[:,1]+m2*r2[:,1])/(m1+m2),'g.')
plt.xlim(-2,2)
plt.ylim(-2,2)
plt.xlabel('x(m)')
plt.ylabel('y(m)')
plt.title('Masses position')
plt.legend(('Mass1','Mass 2','Masses center of mass'))
plt.show()
```

### 9. Simple muscle model

The diagram below shows a simple muscle model. The spring in the left represents the tendinous tissues and the spring in the right represents the elastic properties of the muscle fibers. The damping is present to model the viscous properties of the muscle fibers, the element CE is the contractile element (force production) and the mass $m$ is the muscle mass.

The length $L_{MT}$ is the length of the muscle plus the tendon. In our model $L_{MT}$ is constant, but it could be a function of the joint angle.

Figure. Simple muscle model.

The length of the tendon will be denoted by $l_T(t)$ and the muscle length, by $l_{m}(t)$. Both lengths are related by each other  by the following expression:

\begin{equation}
l_t(t) + l_m(t) = L_{MT}
\end{equation}

The free-body diagram of the muscle mass is depicted below.

Figure. FBD of simple muscle model.

The resultant force being applied in the muscle mass is:

$$\vec{\bf{F}} = -k_T(||\vec{\bf{r_m}}||-l_{t_0})\frac{\vec{\bf{r_m}}}{||\vec{\bf{r_m}}||} + b\frac{d(L_{MT}\hat{\bf{i}} - \vec{\bf{r_{m}}})}{dt} + k_m (||L_{MT}\hat{\bf{i}} - \vec{\bf{r_{m}}}||-l_{m_0})\frac{L_{MT}\hat{\bf{i}} - \vec{\bf{r_{m}}}}{||L_{MT}\hat{\bf{i}} - \vec{\bf{r_{m}}}||} +\vec{\bf{F}}{\bf{_{CE}}}(t)$$

where $\vec{\bf{r_m}}$ is the muscle mass position.

Since the model is unidimensional, we can assume that the force $\vec{\bf{F}}\bf{_{CE}}(t)$ is in the x direction, so the analysis will be done only in this direction.

$$F = -k_T(l_t-l_{t_0}) + b\frac{d(L_{MT} - l_t)}{dt} + k_m (l_m-l_{m_0}) + F_{CE}(t) \\
F = -k_T(l_t-l_{t_0}) -b\frac{dl_t}{dt} + k_m (L_{MT}-l_t-l_{m_0}) + F_{CE}(t) \\
F = -b\frac{dl_t}{dt}-(k_T+k_m)l_t+F_{CE}(t)+k_Tl_{t_0}+k_m(L_{MT}-l_{m_0})$$

Applying the Newton's second law:

$$m\frac{d^2l_t}{dt^2} = -b\frac{dl_t}{dt}-(k_T+k_m)l_t+F_{CE}(t)+k_Tl_{t_0}+k_m(L_{MT}-l_{m_0})$$

To solve this equation, we must break the equation into two first-order differential equations:

\begin{equation}
\frac{dvt}{dt} = - \frac{b}{m}v_t - \frac{k_T+k_m}{m}l_t +\frac{F_{CE}(t)}{m} + \frac{k_T}{m}l_{t_0}+\frac{k_m}{m}(L_{MT}-l_{m_0})
\end{equation}

\begin{equation}
\frac{d l_t}{dt} = v_t
\end{equation}

Now, we can solve these equations by using some numerical method. To obtain the solution, we will use the damping factor of the muscle as $b = 10\,Ns/m$, the muscle mass is $m = 2 kg$, the stiffness of the tendon as $k_t=1000\,N/m$ and the elastic element of the muscle as $k_m=1500\,N/m$. The tendon-length is $L_{MT} = 0.35\,m$, and the tendon equilibrium length is $l_{t0} = 0.28\,m$ and the muscle fiber equilibrium length is $l_{m0} = 0.07\,m$. Both the tendon and the muscle fiber are at their equilibrium lengths and at rest.

Also, we will model the force of the contractile element as a Heaviside step of $90\,N$ (90 N beginning at $t=0$), but normally it is modeled as a function of $l_m$ and $v_m$ having a neural activation signal as input.

```python
m = 2
b = 10
km = 1500
kt = 1000
lt0 = 0.28
lm0 = 0.07

Lmt = 0.35
vt0 = 0

dt = 0.0001
t = np.arange(0, 10, dt)

Fce = 90

lt = lt0
vt = vt0
ltp = np.array([lt0])
lmp = np.array([lm0])
Ft = np.array([0])
for i in range(1,len(t)):
    dvtdt = -b/m*vt-(kt+km)/m*lt  + Fce/m + kt/m*lt0 +km/m*(Lmt-lm0)
    dltdt = vt
    vt = vt + dt*dvtdt
    lt = lt + dt*dltdt
    Ft = np.vstack((Ft,np.array(kt*(lt-lt0))))
    ltp = np.vstack((ltp,np.array(lt)))
    lmp = np.vstack((lmp,np.array(Lmt - lt)))

plt.figure(figsize=(8, 4))
plt.plot(t, Ft, lw=4)
plt.xlabel('t(s)')
plt.ylabel('Tendon force (N)')
plt.show()
plt.figure(figsize=(8, 4))
plt.plot(t, ltp, lw=4)
plt.plot(t, lmp, lw=4)
plt.xlabel('t(s)')
plt.ylabel('Length (m)')
plt.legend(('Tendon length', 'Muscle fiber length'))
plt.show()
```

## Problems

1. Solve the problems 2.3.9, 2.3.20, 11.1.6, 13.1.6 (a, b, c, d, f), 13.1.7, 13.1.10 (a, b) from Ruina and Pratap's book.

## References

- Ruina A, Rudra P (2019) [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.
- Nigg & Herzog (2006) [Biomechanics of the Musculo-skeletal System](https://books.google.com.br/books?id=hOIeAQAAIAAJ&dq=editions:ISBN0470017678). 3rd Edition. Wiley.


「PyTorch入門  5. 自動微分」
===============================================================
【原題】AUTOMATIC DIFFERENTIATION WITH TORCH.AUTOGRAD

【原著】
[Suraj Subramanian](https://github.com/suraj813)、[Seth Juarez](https://github.com/sethjuarez/) 、[Cassie Breviu](https://github.com/cassieview/) 、[Dmitry Soshnikov](https://soshnikov.com/)、[Ari Bornstein](https://github.com/aribornstein/)

【元URL】https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html

【翻訳】電通国際情報サービスISID AIトランスフォーメーションセンター　小川 雄太郎

【日付】2021年03月18日

【チュトーリアル概要】

本チュートリアルでは、PyTorchで計算履歴を保存し、自動で微分操作を実現するAUTOGRADについて解説を行います。

---

Automatic Differentiation with ``torch.autograd``
=======================================

ニューラルネットワークを訓練する際、その学習アルゴリズムとして、基本的には**バックプロパゲーション（back propagation）**が使用されます。

バックプロパゲーションでは、モデルの重みなどの各パラメータは、損失関数に対するその変数の微分値（勾配）に応じて調整されます。

これらの勾配の値を計算するために、PyTorchには``torch.autograd`` という微分エンジンが組み込まれています。

autogradはPyTorchの計算グラフに対する勾配の自動計算を支援します。

シンプルな1レイヤーのネットワークを想定しましょう。

入力を``x``、パラメータを``w`` と ``b``、そして適切な損失関数を決めます。

PyTorchでは例えば以下のように実装します。

```python
%matplotlib inline
```

```python
import torch

x = torch.ones(5)  # input tensor
y = torch.zeros(3)  # expected output
w = torch.randn(5, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)
z = torch.matmul(x, w)+b
loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)
```

---

テンソル、関数、計算グラフの関係
------------------------------------------

上記のコードは以下の**計算グラフ(computational graph)**を示しています。

上記のネットワークでは、``w``と``b``が最適したいパラメータです。

そのため、これらの変数に対する損失関数の微分値を計算する必要があります。

これらのパラメータで微分を可能にするために、``requires_grad``属性をこれらのテンソルに追記します。

【注意】

``requires_grad``はテンソルを定義する際、もしくはその後に、``x.requires_grad_(True)``を実行するなどして指定できます。

計算グラフを構築する際に、テンソルに適用する関数は実際には ``Function``クラスのオブジェクトです。

これらオブジェクトでは、順伝搬時に入力をどのように処理するのか定義されています。

加えて、バックプロパゲーション時に勾配をどのように計算するのかも把握しています。

そして勾配は、テンソルの ``grad_fn`` プロパティに格納されます。

 ``Function``のさらなる詳細については、[こちら](https://pytorch.org/docs/stable/autograd.html#function)を参照ください。

```python
print('Gradient function for z =',z.grad_fn)
print('Gradient function for loss =', loss.grad_fn)
```

    Gradient function for z = <AddBackward0 object at 0x7f7a86429b10>
    Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward object at 0x7f7a86429b50>

---

勾配の計算
-------------------
ニューラルネットワークの各パラメータを最適化するために、入力``x``と出力``y``が与えられたもとで、損失関数の各変数の偏微分値、

すなわち

$\frac{\partial loss}{\partial w}$ 、$\frac{\partial loss}{\partial b}$

を求める必要があります。

これらの偏微分値を求めるために``loss.backward()``を実行し、``w.grad``と``b.grad``の値を導出します。

```python
loss.backward()
print(w.grad)
print(b.grad)
```

    tensor([[0.1212, 0.1953, 0.3046],
            [0.1212, 0.1953, 0.3046],
            [0.1212, 0.1953, 0.3046],
            [0.1212, 0.1953, 0.3046],
            [0.1212, 0.1953, 0.3046]])
    tensor([0.1212, 0.1953, 0.3046])

【注意】

``grad``は計算グラフのleaf node（かつ、``requires_grad``が``True``の変数）のみで求めることができます。

全ての変数で勾配が計算できるわけではない点にはご注意ください。

また、勾配の計算は各計算グラフに対して、``backward``を実行し、1度だけ計算できます（1度しか計算できないのは、パフォーマンスの観点からの仕様です）。

仮に、同じ計算グラフに対して複数回``backward``を実行したい場合には、``retain_graph=True``を``backward``時に引数として渡す必要があります。

---

勾配計算をしない方法
---------------------------

デフォルトでは全てのテンソルは``requires_grad=True``であり、計算履歴が保持され、勾配計算可能な状態です。

ですが、勾配計算が不要なケースも存在します。

例えば訓練済みモデルで推論するケースなどです。

すなわち、ネットワークの順伝搬関数のみを使用する場合となります。

実装コードで勾配計算を不要にするには、``torch.no_grad()``のブロックにそれらのコードを記載するようにします。

```python
z = torch.matmul(x, w)+b
print(z.requires_grad)

with torch.no_grad():
    z = torch.matmul(x, w)+b
print(z.requires_grad)
```

    True
    False

同様に、``detach()`` をテンソルに使用することでも実現できます。

```python
z = torch.matmul(x, w)+b
z_det = z.detach()
print(z_det.requires_grad)
```

    False

勾配の計算、追跡を不能にしたいケースはいく種類かあります。

- ネットワークの一部のパラメータを固定したい（frozen parameters）ケース。これは[ファインチューニング](https://colab.research.google.com/github/YutaroOgawa/pytorch_tutorials_jp/blob/main/notebook/2_Image_Video/2_1_transfer_learning_tutorial_jp.ipynb)時によくあるケースです。

- 順伝搬の計算スピードを高速化したいケース。

---

計算グラフについて補足
----------------------------

概念的には、autogradはテンソルとそれらに対する演算処理を[`Function`](https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)を構成単位として、DAG（a directed acyclic graph）の形で保存したグラフです。

DAGにおいて、各leafは入力テンソル、そしてrootは出力テンソルです。

このグラフをrootから各leafまでchain rule（微分の連鎖律）で追跡することによって各変数に対する偏微分の値を求めることができます。

順伝搬では autograd は2つの処理を同時に行っています。

- 指定された演算を実行し、計算結果のテンソルを求める
- DAGの各操作の*gradient function* を更新する

逆伝搬では、``.backward()``がDAGのrootのテンソルに対して実行されると、autogradは、

- 各変数の ``.grad_fn``を計算する
- 各変数の``.grad``属性に微分値を代入する
- 微分の連鎖律を使用して、各leafのテンソルの微分値を求める

を行います。

【注意】

PyTorchではDAGは動的です（Functionで計算処理される際に逐次構築されていきます）。

そして、`.backward()`を呼び出すたびに、autogradは再度新しいグラフを作成します。

この特性こそが、モデルの順伝搬時に制御フロー文（if文やfor文）を使える理由であり、必要に応じて反復ごとに形や大きさ、操作を変えることができます。

【日本語訳注】

上記内容は初心者の方には非常に難しい話です。

PyTorchは Define-by-run 形式であり、事前に計算グラフを定義するのではなく、計算を実行する際に、柔軟に計算グラフを作ってくれます。

一方で、Define-and-run形式のディープラーニングフレームワーク（例えば、TensorFlow v1など）は、事前に計算グラフを定義する必要があるため、for文やif文といった制御フローの構文を柔軟に使いづらいです。

このことを上記内容では説明しています。

---

補注：テンソルに対する勾配とヤコビ行列
--------------------------------------

多くの場合、スカラー値を出力する損失関数に対して、とある変数の勾配を計算します。

ですが、関数の出力がスカラー値ではなく、任意のテンソルであるケースもあります。

このような場合、PyTorchでは実際の勾配ではなく、いわゆるヤコビ行列（Jacobian
matrix）を計算することができます。

ベクトル関数 $\vec{y}=f(\vec{x})$,において、
$\vec{x}=\langle x_1,\dots,x_n\rangle$ 、そして　$\vec{y}=\langle y_1,\dots,y_m\rangle$の場合、

その勾配、 $\vec{y}$ with respect to $\vec{x}$ はヤコビ行列 で与えられます。

\begin{split}\begin{align}J=\left(\begin{array}{ccc}
   \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\
   \vdots & \ddots & \vdots\\
   \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}
   \end{array}\right)\end{align}\end{split}

ヤコビ行列そのものを計算する代わりにPyTorchでは、**Jacobian Product**、 $v^T\cdot J$ を、入力ベクトル$v=(v_1 \dots v_m)$ に対して計算します。

これは、$v$を引数として ``backward``メソッドを呼び出すことで計算されます。

なお$v$ のサイズは積を計算したい、元のテンソルの大きさと同じである必要があります。

```python
inp = torch.eye(5, requires_grad=True)
out = (inp+1).pow(2)
out.backward(torch.ones_like(inp), retain_graph=True)
print("First call\n", inp.grad)
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nSecond call\n", inp.grad)
inp.grad.zero_()
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nCall after zeroing gradients\n", inp.grad)
```

    First call
     tensor([[4., 2., 2., 2., 2.],
            [2., 4., 2., 2., 2.],
            [2., 2., 4., 2., 2.],
            [2., 2., 2., 4., 2.],
            [2., 2., 2., 2., 4.]])

    Second call
     tensor([[8., 4., 4., 4., 4.],
            [4., 8., 4., 4., 4.],
            [4., 4., 8., 4., 4.],
            [4., 4., 4., 8., 4.],
            [4., 4., 4., 4., 8.]])

    Call after zeroing gradients
     tensor([[4., 2., 2., 2., 2.],
            [2., 4., 2., 2., 2.],
            [2., 2., 4., 2., 2.],
            [2., 2., 2., 4., 2.],
            [2., 2., 2., 2., 4.]])

上記において、同じ変数`inp`に対して、``backward``を2回目実行した際には、勾配が異なる値になった点に注意してください。

これはPyTorchでは``backward``を実行すると、勾配を蓄積（accumulate）する仕様だからです。

すなわち、計算グラフの全leafの``grad``には、勾配が足し算されます。

そのため適切に勾配を計算するには、``grad``を事前に0にリセットする必要があります。

なお実際にPyTorchでディープラーニングモデルの訓練を行う際には、**オプティマイザー（optimizer）**が、勾配をリセットする役割を担ってくれます。

【注意】

本チュートリアルの最初の方で、``backward()``関数を引数のパラメータなしに実行していました。

これは実質的には、``backward(torch.tensor(1.0))``を実行しているのと同じとなります。

``backward()``はスカラー値の関数（例えば訓練時の損失関数）に対して、各パラメータの勾配を計算する際に便利です。

--------------

さらなる詳細
--------------
以下のページも参考ください。

- [`Autograd Mechanics`](https://pytorch.org/docs/stable/notes/autograd.html)

以上。


## Introduction

Problem With Many Input Variables

- If your data is represented using rows and columns, such as in a spreadsheet, then the input variables are the columns that are fed as input to a model to predict the target variable. Input variables are also called features.
- The **curse of dimensionality** refers to all the problems that arise when working with data in the higher dimensions, that did not exist in the lower dimensions.
  - **Sample size**: As the number of features increase, the number of samples also increases proportionally. The more features we have, the more number of samples we will need to have all combinations of feature values well represented in our sample.
  - **Data fit for purpose**: Having a small dataset with large number of feature may become non-representative sample and may impact the performance of the machine learning model
  - **Model Complexity**: As the number of features increases, the model becomes more complex. The more the number of features, the more the chances of overfitting. A machine learning model that is trained on a large number of features, gets increasingly dependent on the data it was trained on and in turn overfitted, resulting in poor performance on real data, beating the purpose. Some algorithms do not perform well when we have a large dimensions.
  - **Storage and training**: High dimension means large input data matrix and requires lot many parameters to be estimated. Less dimensions mean less computing. Less data means that algorithms train faster and requires less storage space.
  - **Multicolinearity or Redundant Features**: Highly correlated variables or Redundant features does not bring any value to the model and increases the storage and training effort.
  - **Data interpretation and visualization**: The higher the number of features, the harder it gets to visualize the training set and then work on it.

Source: medium.com

# Feature Selection (Elimination)

Only keeping the most relevant variables from the original dataset

Let’s imagine our dataset also has a column on customers’ favorite movies, and we realize that this is not incredibly helpful in helping us predict how much they’ll spend at our store. With feature selection, we’d just exclude this entire feature. This particular example might be obvious, but that isn’t always the case. We can perform feature selection programmatically via three key methods: wrapper, filter, and embedded methods.

Techniques for Dimensionality Reduction
- Feature Selection Methods
- Matrix Factorization
- Manifold Learning
- Autoencoder Methods
Tips for Dimensionality Reduction

## Filter Methods

Filter methods greedily eliminate features based on their statistics. An example of such a statistic could be the correlation to the target. Filter methods are the simplest and fastest approaches.

Filter methods apply a threshold on a statistical measure to determine whether a feature will be useful. Two key methods include:

- Variance thresholds - Remove any features that don’t vary widely. For example, in our Willy Wonka dataset, we can see that all of our customers live in the U.S., so that feature would have a variance of 0, and we’d likely want to remove it.
- Correlation thresholds - Remove highly correlated features, as the data is redundant. For example, let’s imagine that we also had a year of birth column in the above dataset. This would be highly correlated with our age column and would not be helpful in generating any stronger predictions.

These methods are very straightforward and intuitive. However, they require manual setting of a threshold which can be dangerous, as too low of a threshold may mean the removal of important information.

Additionally, these two methods are often not sufficient if we want to seriously reduce the dimensionality of our dataset.

### Variance Threshold

Variance Threshold is a baseline approach to feature selection. As the name suggests, it drops all features where the variance along the column does not exceed a threshold value.

#### Low Variance Filter
Variance thresholds remove features whose values don't change much from observation to observation (i.e. their variance falls below a threshold). These features provide little value.

For example, if you had a public health dataset where 96% of observations were for 35-year-old men, then the 'Age' and 'Gender' features can be eliminated without a major loss in information.

Because variance is dependent on scale, you should always normalize your features first.

- Strengths: Applying variance thresholds is based on solid intuition: features that don't change much also don't add much information. This is an easy and relatively safe way to reduce dimensionality at the start of your modeling process.
- Weaknesses: If your problem does require dimensionality reduction, applying variance thresholds is rarely sufficient. Furthermore, you must manually set or tune a variance threshold, which could be tricky. We recommend starting with a conservative (i.e. lower) threshold.
Implementations: Python / R

Feature selection is the process of identifying and selecting relevant features for your sample.
Feature selection can be done either manually or programmatically.
- Manually: by examining correlation between features
- Variance Threshold
- Univariate selection

Methods listed under Feature Selection/Elimination below are usually employed during the data pre-processing stage or prior to the predictive modelling stage.

Feature Selection / Elimination (Pre-Modelling)
Dropping Features with Missing Values

Dropping Features or observations with missing values should be the option of the last resort and it is always advisable to go back to the data source and try to plug in the missing values or if not possible to impute the same. However dropping columns with percentage or ratio of missing values above a threshold is also an option, though there is no hard and fast threshold cut-off.

```python
>>> X = [[0, 2, 0, 3], [0, 1, 4, 3], [0, 1, 1, 3]]
>>> selector = VarianceThreshold()
>>> selector.fit_transform(X)
array([[2, 0],
       [1, 4],
       [1, 1]])
```

#### **Univariate Selection**

Univariate describes a type of data which consists of observations on only a single characteristic or attribute.

Univariate feature selection examines each feature individually to determine the strength of the relationship of the feature with the response variable. It uses statistical tests to select features.

Some examples of statistical tests that can be used to evaluate feature relevance are
- Pearson Correlation,
- Maximal information coefficient,
- Distance correlation,
- ANOVA and
- Chi-square.

Chi-square is used to find the relationship between categorical variables and Anova is preferred when the variables are continuous.

### Correlation Thresholds

Correlation thresholds remove features that are highly correlated with others (i.e. its values change very similarly to another's). These features provide redundant information.

- High Correlation filter
High correlation between two variables means they have similar trends and are likely to carry similar information. This can bring down the performance of some models drastically (linear and logistic regression models, for instance). We can calculate the correlation between independent numerical variables that are numerical in nature. If the correlation coefficient crosses a certain threshold value, we can drop one of the variables (dropping a variable is highly subjective and should always be done keeping the domain in mind).

As a general guideline, we should keep those variables which show a decent or high correlation with the target variable.

For example, if you had a real-estate dataset with 'Floor Area (Sq. Ft.)' and 'Floor Area (Sq. Meters)' as separate features, you can safely remove one of them.

Which one should you remove? Well, you'd first calculate all pair-wise correlations. Then, if the correlation between a pair of features is above a given threshold, you'd remove the one that has larger mean absolute correlation with other features.

- Strengths: Applying correlation thresholds is also based on solid intuition: similar features provide redundant information. Some algorithms are not robust to correlated features, so removing them can boost performance.
- Weaknesses: Again, you must manually set or tune a correlation threshold, which can be tricky to do. Plus, if you set your threshold too low, you risk dropping useful information. Whenever possible, we prefer algorithms with built-in feature selection over correlation thresholds. Even for algorithms without built-in feature selection, Principal Component Analysis (PCA) is often a better alternative.
Implementations: Python / R

## Wrapper Methods

Wrapper methods use the predictive algorithm on a validation set with subsets of features to identify the most useful ones. Finding the optimal subset has a large computational cost. Wrapper methods make greedy decisions like backward/forward selection, which greedily eliminate/select features one after the other.

Wrapper methods iterate through different combinations of features and perform a model retrain on each. The feature combination which resulted in the best model performance metric (accuracy, AUC, RMSE, etc.) is selected.

Two key wrapper methods include:
- Forward selection
- Backward elimination

NOTE : Both Backward Feature Elimination and Forward Feature Selection are time consuming and computationally expensive.They are practically only used on datasets that have a small number of input variables.

### **Forward selection**

Begin with zero features in your model and iteratively add the next most predictive feature until no additional performance is reached with the addition of another feature.

This is the inverse process to backward feature elimination. We start with one column only, progressively adding one column at a time, i.e., the column that produces the highest increase in performance. Both algorithms, backward feature elimination and forward feature construction, are quite expensive in terms of time and computation. They are practical only when applied to a dataset with an already relatively low number of input columns.

Steps:
- We start with a single feature. Essentially, we train the model n number of times using each feature separately
- The variable giving the best performance is selected as the starting variable
- Then we repeat this process and add one variable at a time. The variable that produces the highest increase in performance is retained
- We repeat this process until no significant improvement is seen in the model’s performance

```python
# Forward Feature Selection
from sklearn.feature_selection import f_regression
ffs = f_regression(df,train.Item_Outlet_Sales )
variable = []
for i in range(0,len(df.columns)-1):
    if ffs[0][i] >=10:
       variable.append(df.columns[i])
```

### **Backward elimination**

Begin with all features in your model and iteratively remove the least significant feature until performance starts to drop.
These methods are typically not preferred as they take a long time to compute and tend to overfit.

In this technique, at a given iteration, the selected classification algorithm is trained on n input columns. Then we remove one input column at a time and train the same model on n-1 columns.  The input column whose removal has produced the smallest increase in the error rate is removed, leaving us with n-1 input columns. The classification is then repeated using n-2 columns, and so on. Each iteration k produces a model trained on n-k columns and an error rate e(k). By selecting the maximum tolerable error rate, we define the smallest number of columns necessary to reach that classification performance with the selected machine learning algorithm.

Steps:
- We first take all the n variables present in our dataset and train the model using them
- We then calculate the performance of the model
- Now, we compute the performance of the model after eliminating each variable (n times), i.e., we drop one variable every time and train the model on the remaining n-1 variables
- We identify the variable whose removal has produced the smallest (or no) change in the performance of the model, and then drop that variable
- Repeat this process until no variable can be dropped

```python
# Backward Feature Elimination
#We need to specify the algorithm and number of features to select, and
#we get back the list of variables obtained from backward feature elimination.
#We can also check the ranking of the variables using the “rfe.ranking_” command.
from sklearn.linear_model import LinearRegression
from sklearn.feature_selection import RFE
from sklearn import datasets
lreg = LinearRegression()
rfe = RFE(lreg, 10)
rfe = rfe.fit_transform(df, train.Item_Outlet_Sales)
```

## Embedded Methods

Embedded feature selection combines key features of wrapper and filter methods and is often a great place to start, as long as you’re using a compatible algorithm.

Certain machine learning algorithms are able to embed feature selection into the model training process. Key embedded methods include:

- **Regularized linear regression** - These methods add a penalty that corresponds to the value of the coefficients. This way, coefficients with lower predictive power are weighted less heavily, and the overall variance of the dataset is reduced. There are two key types of regularized linear regression:
  - **Lasso regression** - In lasso regression, also referred to as L1 regularization, the penalty is equivalent to the absolute value of the coefficient. This means that coefficients of unimportant (redundent) features can be reduced to zero, which effectively eliminates them from the model.
  - **Ridge regression** - In ridge regression, also referred to as L2 regularization, the penalty is equivalent to the square of the coefficient value. Because coefficient values are squared, coefficients cannot be entirely eliminated from the model, but can be weighted less heavily.
- **Regularized trees** - When building tree-based models, we can use metrics like entropy (discussed in our tree-based models post) to measure feature importance and can then elect to only keep the top N most important features.

### Random Forest

Random Forest is one of the most widely used algorithms for feature selection. It comes packaged with in-built feature importance so you don’t need to program that separately. This helps us select a smaller subset of features.

We need to convert the data into numeric form by applying one hot encoding, as Random Forest (Scikit-Learn Implementation) takes only numeric inputs. Let’s also drop the ID variables (Item_Identifier and Outlet_Identifier) as these are just unique numbers and hold no significant importance for us currently.

```python
from sklearn.ensemble import RandomForestRegressor
df=df.drop(['Item_Identifier', 'Outlet_Identifier'], axis=1)
model = RandomForestRegressor(random_state=1, max_depth=10)
df=pd.get_dummies(df)
model.fit(df,train.Item_Outlet_Sales)

# plot the feature importance graph
features = df.columns
importances = model.feature_importances_
indices = np.argsort(importances)[-9:]  # top 10 features
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='b', align='center')
plt.yticks(range(len(indices)), [features[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()

# pick the top-most features
from sklearn.feature_selection import SelectFromModel
feature = SelectFromModel(model)
Fit = feature.fit_transform(df, train.Item_Outlet_Sales)
```

# Feature Extraction (Projection)

By finding a smaller set of new variables, each being a combination of the input variables, containing basically the same information as the input variables. Feature engineering is manually generating new features from existing features, by applying some transformation or performing some operation on them.

Feature extraction allows us to keep parts of features, so that we just keep the most helpful bits of each feature. This way, we don’t lose too much information by removing entire features. Principal Component Analysis (PCA) is far and away the most common method of performing feature extraction and is the method we’ll explore in this post.

PCA orients data along the direction of the component with maximum variance whereas LDA projects the data to signify the class separability

## Linear Dimensionality Reduction Methods

Linear methods linearly combine the original features to compress the original dataset into fewer dimensions.
The most common and well known dimensionality reduction methods are the ones that apply linear transformations, like
- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
- Singular Value Decomposition (SVD)
- Non-Matrix Factorization (NMF)
- Factor Analysis (FA)
 - Confirmatory Factor Analysis (CFA)
 - Exploratory Factor Analysis (EFA)

### Principal Component Analysis (PCA) - Unsupervised

**Objective**: Higher dimensional space is mapped (projected) to data in a lower dimension space, maximizing the explained variance in lower dimension space.

Popularly used for dimensionality reduction in continuous data, PCA rotates and projects data along the direction of increasing variance. The features with the maximum explained variance are the principal components.

The purpose of PCA is to reduce the number of features included while still capturing the key information, or the spread of the data, as measured by the total variance.

- **Feature scaling** is required prior to PCA
- PCA technique generates smaller feature set by linearly combining the features within the original feature(variables) set. The new features are called principal components.
- PCA works on eigen decomposition. It uses **Covariance matrix** as a measure of how each variable is associated with one another and finds orthogonal components (**Principal components**) that explains total variablility of the data. The directions in which data are dispersed are explained by **Eigenvectors**. The relative importance of these different directions are given by **Eigenvalues**.
- Principal components are extracted in such a way that the principal components explains total variance in the dataset in decreasing order of variance
  - The first principal component is the line which captures the majority of the variance within the dataset,
  - the second principal component captures the second most variance
  - Third principal component tries to explain the variance which is not explained by the first two principal components and so on
- Each of these principal components is orthogonal to the next that means they are uncorrelated.
- Dimensionality reduction on dataset is performed by selecting only significant principal components using a threshold (removing the least significant PCs) for model training
- PCA combine original feature to create new orthogonal features which can not be intrepretable in practical terms. If interpretation is important, unfortunately PCA will likely be problematic.

>Note:
- We should only use training data for PCA components. Because, this would violate the entire assumption of generalization since test data would get ‘leaked’ into the training set.
- We should not perform PCA on test and train data sets separately. Because, the resultant vectors from train and test PCAs will have different directions (due to unequal variance)
- For Test set, we should use same transformation as we did to training set, including the center and scaling feature.

It involves the following steps:

- Standardize the input data
- Construct the covariance matrix of the data.
- Compute the eigenvectors of this matrix.
- Eigenvectors corresponding to the largest eigenvalues are used to reconstruct a large fraction of variance of the original data.

Hence, we are left with a lesser number of eigenvectors, and there might have been some data loss in the process. But, the most important variances should be retained by the remaining eigenvectors.

>There are two key methods of computing the principal components
- **Eigen decomposition** and
- **Singular value decomposition (SVD)**

Advantages of Dimensionality Reduction
- It helps in data compression, and hence reduced storage space.
- It also helps remove redundant features, if any.
- PCA is fast and simple to implement, which means you can easily test algorithms with and without PCA to compare performance. In addition, PCA offers several variations and extensions (i.e. kernel PCA, sparse PCA, etc.) to tackle specific roadblocks.

Disadvantages of Dimensionality Reduction
- It may lead to some amount of data loss.
- PCA tends to find linear correlations between variables, which is sometimes undesirable.
- PCA fails in cases where mean and covariance are not enough to define datasets.
- We may not know how many principal components to keep- in practice, some thumb rules are applied.
- The new principal components may not be interpretable to business, which may be a deal-breaker in some settings. In addition, we need to still manually set or tune a threshold for cumulative explained variance.

### How PCA works

Solving PCA: Eigenvectors of Covariance

Solution to PCA is based on an important property of eigenvector decomposition.

Columns are variables, and rows are observations

#### Matrix Dimension
- $X_{n,p}$ = Input data or original features matrix
- $Z_{n,p}$ = Transformed (new features) data or principal components matrix
- $D_{p,p}$ = Diagonal matrix whose diagnoal elements are Eigen values ($\lambda_j$) of $X^T X$, a symmetric and diagonal matrix
- $E_{p,p}$ = Eigen vector matrix, a symmetric matrix, where each column is a eigen vector (orthogonal  matrix) for every eigen value $\lambda_j$ of covariance matrix $X^TX$
- $S_X$ = $\sum$ = $\frac{1}{n-1}X^TX$ = $\frac{1}{n-1}XX^T$ = Covarinance matrix (pxp) of original features, a symmetric matrix
- $X^TX = XX^T$ = Product matrix (pxp) of original features, a symmetric matrix
- $S_Z \equiv \frac{1}{n-1}Z^TZ$ = Covarinance matrix(pxp) of new features (transformed) or principal components, a symmetric matrix
- n = number of training examples
- p = number of input features (variables)

Input or original feature matrix
>\begin{equation} X_{n,p} =
\begin{bmatrix}
X^1_1 & X^1_2 & X^1_3 & \ldots & X^1_p\\
X^2_1 & X^2_2 & X^2_3 & \ldots & X^2_p\\
\ldots & \ldots& \ldots & \ldots & \ldots\\
X^n_1 & X^n_2 & X^n_3 & \ldots & X^n_p\\
\end{bmatrix}
\end{equation}

Transformed or new feature matrix (principal components)
>\begin{equation} Z_{n,p} =
\begin{bmatrix}
\sum_{i=1}^{p}X^1_ie^{\lambda_1}_i & \sum_{i=1}^{p}X^1_ie^{\lambda_2}_i & \ldots & \sum_{i=1}^{p}X^1_ie^{\lambda_p}_i\\
\sum_{i=1}^{p}X^2_ie^{\lambda_1}_i & \sum_{i=1}^{p}X^2_ie^{\lambda_2}_i & \ldots & \sum_{i=1}^{p}X^2_ie^{\lambda_p}_i\\
\ldots & \ldots & \ldots & \ldots\\
\sum_{i=1}^{p}X^n_ie^{\lambda_1}_i & \sum_{i=1}^{p}X^n_ie^{\lambda_2}_i & \ldots & \sum_{i=1}^{p}X^n_ie^{\lambda_p}_i\\
\end{bmatrix}
= \begin{bmatrix}(X^1_1e^{\lambda_1}_1+X^1_2e^{\lambda_1}_2+X^1_3e^{\lambda_1}_3+X^1_pe^{\lambda_1}_p) & (X^1_1e^{\lambda_2}_1+X^1_2e^{\lambda_2}_2+X^1_3e^{\lambda_2}_3+X^1_pe^{\lambda_2}_p) & \ldots & \ldots & (X^1_1e^{\lambda_p}_1+X^1_2e^{\lambda_p}_2+X^1_3e^{\lambda_p}_3+X^1_pe^{\lambda_p}_p)\\
(X^1_1e^{\lambda_1}_1+X^1_2e^{\lambda_1}_2+X^1_3e^{\lambda_1}_3+X^1_pe^{\lambda_1}_p) & (X^2_1e^{\lambda_2}_1+X^2_2e^{\lambda_2}_2+X^2_3e^{\lambda_2}_3+X^2_pe^{\lambda_2}_p) & \ldots & \ldots & (X^2_1e^{\lambda_p}_1+X^2_2e^{\lambda_p}_2+X^2_3e^{\lambda_p}_3+X^2_pe^{\lambda_p}_p)\\
\ldots & \ldots& \ldots & \ldots\\
(X^1_1e^{\lambda_1}_1+X^1_2e^{\lambda_1}_2+X^1_3e^{\lambda_1}_3+X^1_pe^{\lambda_1}_p) & (X^n_1e^{\lambda_2}_1+X^n_2e^{\lambda_2}_2+X^n_3e^{\lambda_2}_3+X^n_pe^{\lambda_2}_p) & \ldots & \ldots & (X^n_1e^{\lambda_p}_1+X^n_2e^{\lambda_p}_2+X^n_3e^{\lambda_p}_3+X^n_pe^{\lambda_p}_p)\\
\end{bmatrix}
\end{equation}

- Each row of $Z$ is the transformed version of input variables of $X$,
- For a training example (row), each column denotes principal components of new feature space
- The value at position (row i, col j) i.e. $Z^i_j$ represents the j-th principal component (new feature) for i-th training example, $Z^i_j = (X^i_1e^{\lambda_j}_1+X^i_2e^{\lambda_j}_2+X^i_3e^{\lambda_j}_3+X^i_pe^{\lambda_j}_p)$

Eigen value matrix D of ($X^TX$), a diagonal matrix:
>\begin{equation} D_{p,p} =
\begin{bmatrix}
\lambda_1 & 0 & 0 & \ldots & 0 \\
0 & \lambda_2 & 0 & \ldots & 0\\
0 & 0 & \lambda_3 & \ldots & 0\\
\ldots & \ldots & \ldots & \ldots & \ldots\\
0 & 0 & 0 & \ldots & \lambda_p \\
\end{bmatrix}
\end{equation}

Eigen vectors matrix for all eigen values
>\begin{equation} E_{p,p} =
\begin{bmatrix}
e^{\lambda_1}_1 & e^{\lambda_2}_1 & e^{\lambda_3}_1 & \ldots & e^{\lambda_p}_1\\
e^{\lambda_1}_2 & e^{\lambda_2}_2 & e^{\lambda_3}_2 & \ldots & e^{\lambda_p}_2\\
\ldots & \ldots& \ldots & \ldots & \ldots\\
e^{\lambda_1}_p & e^{\lambda_2}_p & e^{\lambda_3}_p & \ldots & e^{\lambda_p}_p\\
\end{bmatrix}
\end{equation}

- Each columns of $E$ is loadings of principal component of $X$  i.e. eigen vectors of the covariance matrix $X^TX$.
- In other words, each column vector $[e^{\lambda_j}_1, e^{\lambda_j}_1, ..., e^{\lambda_j}_p]$ in E is a eigen vector corresponding to the eigen value $\lambda_j$ obtained from characterisitic equation.

---

Source: sharetechnote.com, medium.com@kaizhao44

#### Find Eigen Vector

Find some orthonormal matrix $E$ where

>$Z = XE$ such that $S_Z$ is diagonalized.

Covarinance matrix (pxp) of new features,

>$S_Z = \frac{1}{n-1} (XE)^T XE = \frac{1}{n-1}E^T(X^TX) E$

For an orthonormal matrix $E$
> $E^{-1} = E^T$ and $E^{-1}E = EE^{-1} = EE^T = E^TE = I$

Note:
> A symmetric matrix is diagonalized by a matrix of its orthonormal eigenvectors. Therefore, symmetric matrix $X^T X$ is always diagonalized by an orthogonal matrix $E$ of whose columns are the loadings of principal components of $X$ i.e. eigen vectors of the covariance matrix $X^T X$.

$X^T X$ can be decomposed to its eigen value matrix and eigen vector matrix as below:
> $X^T X = E D E^T$

Rewriting $S_Z$, we will get:

>$S_Z = \frac{1}{n-1}E^T (X^T X) E = \frac{1}{n-1}E^T (E D E^T) E = \frac{1}{n-1}D$

It is evident that the choice of E diagonalizes $S_Z$.

In other words, if we choose E as eigen vector of $X^TX$, the covariance matrix of new features $Z = XE^T$ (i.e. $S_Z$) will be a diagonal matrix.

This was the goal for PCA. We can summarize the
results of PCA in the matrices E and $S_Z$.
- The principal components of X are the eigenvectors of $X^TX$ or the rows of E.
- The i-th diagonal value of $S_Z$ is the variance of
X along i-th principal component.

---

The matrix $X^T X$ has r ≤ p orthonormal eigenvectors
where r is the rank of the matrix. The rank of $X^T X$ is less than p when $X^T X$ is degenerate or all data occupy a subspace of dimension r ≤ p. Maintaining the constraint of orthogonality, we can remedy this situation by selecting (p − r) additional orthonormal vectors to "fill up" the eigen vector matrix $E$. These additional vectors do not effect the final solution because the variances associated with these directions are zero

In practice computing PCA considers a scaled data set wherein X is subtracted off with the mean of each original features.

### How to find Principal Components using Eigen Decomposition

New features (latent or principal components) in $Z$ are a linear combination of original features in $X$ and vice versa,

For a training examle, the new feature space corresponding to j-th eigen value,
> $Z_j = {E_{\lambda_j}}^TX$

Therefore, the new features can be written as
> $ Z_j = \sum_{i=1}^{p}e^{\lambda_j}_iX_i = e^{\lambda_j}_1X_1+e^{\lambda_j}_2X_2+\ldots+e^{\lambda_j}_pX_p$ where $j=1, 2, 3, ..., p$

From these p principal components, we select only top k principal components (formed using eigen vectors corresponding to largest k eigen values that explains most of the variance in the data), hence reducing the overall dimension of the dataset.

where
- $Z_j$ is the new feature (principal component) in transformed space
- The column vector $[e^{\lambda_j}_1, e^{\lambda_j}_1, ..., e^{\lambda_j}_p]$ in matrix E is the eigen vector corresponding to the eigen value $\lambda_j$ obtained from characterisitic equation
- We select only few $Z_j$ for model building and prediction which explains most of the explained variance in the data set
- Explained variance by j-th principal component $Z_j$,
> $\frac{\lambda_{j}}{\sum_{j=1}^{p} \lambda_{j}}$

#### Detailed steps:

1. Standardize the data set

    This is done by subtracting the respective feature mean and dividing by the feature standard deviation for each value of each variable.

    If there are n training examples, and if we have p features (variables) $X_{1}, X_{2}, X_{p}$ then standardized features $X'$ will be

    \begin{equation}
    \begin{bmatrix} X_1\\X_2\\ \vdots \\ X_p\end{bmatrix} \equiv
    \begin{bmatrix} X'_1\\ X'_2\\ \vdots \\X'_p\end{bmatrix} \end{equation}

    Where,
    $X'_1 = \frac{X_1-\bar{X_1}}{s_1}$,
    $X'_2 = \frac{X_2-\bar{X_2}}{s_2} \ldots$,
    $X'_p = \frac{X_p-\bar{X_p}}{s_p}$

    **Mean** of feature i,
    >$\bar{X}_i = \frac{1}{n} \sum X_i$

    **Unbiased Variance** of feature i (it is an estimate of population variance),
    >$var_i = s^2_i = cov_{i,i} = \frac{\sum(X_i-\bar{X_i})^2}{n-1}$

    **Correlation** between the original feature vector i and j as

    >$cor_{i,j} = \frac{1}{n-1} \sum_{1}^{n}\frac{(X_i-\bar{X_i})}{s_i} \frac{(X_j-\bar{X_j})}{s_j} =  \sum_{1}^{n}\frac{(X_i-\bar{X_i})}{\sqrt{{\sum(X_j-\bar{X_j})^2}}}\frac{(X_j-\bar{X_j})}{\sqrt{{\sum(X_j-\bar{X_j})^2}}}$

    **Correlation** in standardized form
    > $cor_{i,j} =  \frac{1}{n-1} \sum_{1}^{n} X'_i*X'_j$

    This produces a dataset whose mean is zero. Once the standardization is done, all the variables will be transformed to the same scale.

2. Construct **Covariance matrix $\sum$** of the standardized data (it is simply a correlation matrix of original features). Covariance matrix is a square matrix:

  > \begin{equation} \sum =
    \begin{bmatrix}
        cov_{1,1} & cov_{1,2} & cov_{1,3} & \dots  & cov_{1,p} \\
        cov_{2,1} & cov_{2,2} & cov_{2,3} & \dots  & cov_{2,p} \\
        \vdots & \vdots & \vdots & \ddots & \vdots \\
        cov_{p,1} & cov_{p,2} & cov_{p,3} & \dots  & cov_{p,p}
    \end{bmatrix}
    \end{equation}

    Where, diagonal elements are variance of standardized features which is simply the unbiased variance from original features.

3. Find the Eigen vectors and Eigen values of the Covariance matrix

    If $\lambda_j$ is eigen value (scalar / constant) and $E_j$ is the corresponding eigen vector, **a column vector** (of covariance matrix $\sum$), it can be written in equation form:
    > $\sum E_j = \lambda_j E_j$, this gives
    $(\sum - \lambda_j I) = 0 $

    - Find Eigen Values (the roots) using characteristic equation
    > $det(\sum - \lambda_j I) = 0 $

     In other words,
    > $(\lambda_1-\lambda_j)(\lambda_2-\lambda_j)\ldots(\lambda_p-\lambda_j) = 0$

    Eigen Values (Roots) of the polynomial, $\lambda_j = \lambda_1, \lambda_2, \lambda_3, \ldots \lambda_p$

    Sum of variances of all original features = sum of diagonal elements of covariance matrix (Trace)
    > $tr(\sum) = \lambda_1 + \lambda_2 + \lambda_3 + \ldots + \lambda_p$

    > $det(\sum) = \lambda_1 * \lambda_2 * \lambda_3 * \ldots * \lambda_p$

  - Find eigenvector $E_j$ corresponding to each eigenvalue using below equation:
    > $(\sum - \lambda_j I) E_j = 0$

    There will be p columns of eigen vectors (also called Feature Vector or Loadings of principal component) correspondiing to each eigen value

    For j-th eigen value $\lambda_j$, corresponding eigen vector will be

    >  \begin{equation} E_{\lambda_j} =
    \begin{bmatrix}
    e^{\lambda_j}_1\\e^{\lambda_j}_2\\ \vdots \\ e^{\lambda_j}_p
    \end{bmatrix}
    \end{equation}

    Each set of eigenvectors multiplies with the feature vector $X$ will give a new feature vector

4. Sort the eigenvalues by decreasing order (top to bottom in the matrix) to rank the corresponding eigenvectors

    Eigen value matrix:
    \begin{bmatrix} \lambda_1\\\lambda_2\\ \vdots \\ \lambda_p\end{bmatrix}

    where, $\lambda_1 > \lambda_2 > \lambda_3 > \ldots > \lambda_p$

5. Transform p-dimensional input dataset X ($X_1, X_2, X_3, \ldots, X_p$) to obtain $Z$ ($Z_1, Z_2, Z_3, \ldots, Z_p$), a new p-dimensional feature space (each variable is called principal components)
the new feature subspace

6. Select k dimension ($Z_1, Z_2, Z_3, \ldots, Z_k$) out of all p-dimensions of $Z$ ($Z_1, Z_2, Z_3, \ldots, Z_p$) which corresponds to the k largest eigenvalues, where k is the dimensionality of new data (k < p)

```python
# PCA using Eigen Decomposition
from sklearn.decomposition import PCA
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import scale
#Scaling the values
X = scale(X_train)
from sklearn.decomposition import PCA
pca = PCA(n_components=4)
pca_result = pca.fit_transform(df[feat_cols].values)
plt.plot(range(4), pca.explained_variance_ratio_)
plt.plot(range(4), np.cumsum(pca.explained_variance_ratio_))
plt.title("Component-wise and Cumulative Explained Variance")
#After selecting 2 PCs
pca = PCA(n_components=2)
pca.fit(X)
X1=pca.fit_transform(X)
```

### Singular value decomposition (SVD)

 (1) Types of SVD (2) Eigen Decomposition vs SVD

In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix that generalizes the eigen decomposition of a square (mxm or nxn) normal matrix to any rectangular (mxn) matrix via an extension of the polar decomposition.

Let M be an arbitrary (m×n) matrix of rank r
- $M^TM$ will be square and symmetric (nxn) matrix
- $MM^T$ will be square and symmetric (mxm) matrix
- Calculating the SVD consists of finding the eigenvalues and eigenvectors of  $M^TM$ and $MM^T$
- The eigenvectors of  $MM^T$  make up the columns of U, $MM^T = UD^2U^T$
- The eigenvectors of $M^TM$ make up the columns of V, $M^TM = VD^2V^T$
- Both matrices have the same positive eigenvalues The square roots of these eigenvalues are called singular values. Both have the same rank r as M
- U and V are invertible for any matrix M in SVD
- U and V are orthogonal matrix, $UU^T = U^TU = I$, $VV^T = V^TV = I$
- Also, $\Sigma$ is the diagonal matrix formed with the singular values (square roots of eigenvalues from $M^TM$ or $MM^T$)
> $\Sigma_{m,m} = \sqrt{eig(diag(MM^T))}, \Sigma_{n,n} = \sqrt{eig(diag(M^TM))}$
- The diagonal entries of the $\Sigma$ matrix are arranged in descending order.

> $M = \sum_{i=1}^{r} \sigma_i u_i v_i$ where $\sigma_i = \sqrt{\lambda_i}$

\begin{equation} MM^T = \Sigma_{m,m} =
\begin{bmatrix}
\sqrt{\lambda_1} & 0 & 0 & \ldots & 0\\
0& \sqrt{\lambda_2} & 0 & \ldots  & 0\\
0 & 0 & \sqrt{\lambda_3} & \ldots & 0\\
\ldots & \ldots & \ldots & \ldots & \ldots \\
0 & 0 & 0 & \ldots & \sqrt{\lambda_m}\\
\end{bmatrix}
\end{equation}

where, $\lambda_1 > \lambda_2 > \lambda_3 > \ldots > \lambda_m$

\begin{equation} M^TM = \Sigma_{n,n} =
\begin{bmatrix}
\sqrt{\lambda_1} & 0 & 0 & \ldots & 0\\
0& \sqrt{\lambda_2} & 0 & \ldots  & 0\\
0 & 0 & \sqrt{\lambda_3} & \ldots & 0\\
\ldots & \ldots & \ldots & \ldots & \ldots \\
0 & 0 & 0 & \ldots & \sqrt{\lambda_n}\\
\end{bmatrix}
\end{equation}

where, $\lambda_1 > \lambda_2 > \lambda_3 > \ldots > \lambda_n$

1. Organize a data set as an (m × n) matrix, where
m is the number of variables and n is the number of examples
2. Scale features before performing SVD (Subtract off the mean for each variable in M and divide it by standard deviation)
3. Calculate the SVD or the eigenvectors of the covariance

 (1) original matrix as a linear combination of low-rank matrices (2) Rotation (U & V) , Scaling (S)

1. Full SVD:
> $M_{m,n} = U_{m,m} \Sigma_{m,n} V_{n,n}^T $

2. Thin SVD:
(remove colums of U not corresponding to rows of $V^T$),
> $M_{m,n} = U_{m,n} \Sigma_{n,n} V_{n,n}^T $

If n<<m, Only the n column vectors of U corresponding to the row vectors of V* are calculated. The remaining column vectors of U are not calculated.

3. Compact SVD:
(remove vanishing singular values and corresponding columns/rows in U and $V^T$),
>  $M_{m,n} = U_{m,r} \Sigma_{r,r} V_{r,n}^T $

If r<<n, Only the r column vectors of U and r row vectors of $V^T$ corresponding to the non-zero singular values Σr are calculated. The remaining vectors of U and V* are not calculated.

4. Truncated SVD (keep only largest t singular values and corresponding columns/rows in U and $V^T$)
> $M_{m,n} = U_{m,t} \Sigma_{t,t} V_{t,n}^T $

If t<<r, Only the t column vectors of U and t row vectors of V* corresponding to the t largest singular values Σt are calculated. The rest of the matrix is discarded.

> Note:
The rank of a matrix is the maximum number of linearly independent row (or column) vectors in the matrix. A vector R is said to be linearly independent of vectors P and Q if it cannot be expressed as a linear combination of P and Q. $R \ne aP+bQ$

```python
# Full Singular-value decomposition
from numpy import array
from scipy.linalg import svd
# define a matrix
A = array([[1, 2], [3, 4], [5, 6]])
print(A)
# SVD
U, s, VT = svd(A)
print(U)
print(s)
print(VT)
# create m x n Sigma matrix
Sigma = zeros((A.shape[0], A.shape[1]))
# populate Sigma with n x n diagonal matrix
Sigma[:A.shape[1], :A.shape[1]] = diag(s)
# reconstruct matrix
B = U.dot(Sigma.dot(VT))
print(B)
```

```python
# PCA using SVD
from numpy import array
from sklearn.decomposition import TruncatedSVD
# define array
A = array([
	[1,2,3,4,5,6,7,8,9,10],
	[11,12,13,14,15,16,17,18,19,20],
	[21,22,23,24,25,26,27,28,29,30]])
print(A)
# svd
svd = TruncatedSVD(n_components=2, random_state=42)
svd.fit(A)
result = svd.transform(A)
print(result)
print("Singular values:")
print(svd.singular_values_)
plt.figure(figsize=(12,8))
plt.title('SVD Components')
plt.scatter(result[:,0], result[:,1])
```

### Independent Component Analysis (ICA)

It is based on information-theory and is also one of the most widely used dimensionality reduction techniques. The major difference between PCA and ICA is that PCA looks for uncorrelated factors while ICA looks for independent factors.

If two variables are uncorrelated, it means there is no linear relation between them. If they are independent, it means they are not dependent on other variables. For example, the age of a person is independent of what that person eats, or how much television he/she watches.

This algorithm assumes that the given variables are linear mixtures of some unknown latent variables. It also assumes that these latent variables are mutually independent, i.e., they are not dependent on other variables and hence they are called the independent components of the observed data.

Here, image (a) represents the PCA results while image (b) represents the ICA results on the same dataset.

The equation of PCA is x = Wχ.

Here,

x is the observations
W is the mixing matrix
χ is the source or the independent components
Now we have to find an un-mixing matrix such that the components become as independent as possible. Most common method to measure independence of components is Non-Gaussianity:

As per the central limit theorem, distribution of the sum of independent components tends to be normally distributed (Gaussian).

So we can look for the transformations that maximize the kurtosis of each component of the independent components. Kurtosis is the third order moment of the distribution. To learn more about kurtosis, head over here.
Maximizing the kurtosis will make the distribution non-gaussian and hence we will get independent components

```python
# ICA
from sklearn.decomposition import FastICA
ICA = FastICA(n_components=3, random_state=12)
X=ICA.fit_transform(df[feat_cols].values)
plt.figure(figsize=(12,8))
plt.title('ICA Components')
plt.scatter(X[:,0], X[:,1])
plt.scatter(X[:,1], X[:,2])
plt.scatter(X[:,2], X[:,0])
```

### Linear discriminant analysis (LDA) - Supervised

(not to be confused with latent Dirichlet allocation)

**Objective**: Higher dimensional space is mapped (projected) to data in a lower dimension space, maximizing ratio of between class variation and within class variation in lower dimension space.

- **Feature Scaling** and normalization is required prior to LDA transformation
- LDA is a supervised method that can only be used with labeled data. It is commonly used for the supervised classification problems
- Unlike PCA, LDA doesn't maximize explained variance. Instead, it maximizes the separability between classes.
- LDA maximize the distance between means of the classes, and minimize the variation within each class

Extensions to LDA:
- Quadratic Discriminant Analysis (QDA): Each class uses its own estimate of variance (or covariance when there are multiple input variables).
- Flexible Discriminant Analysis (FDA): Where non-linear combinations of inputs is used such as splines.
- Regularized Discriminant Analysis (RDA): Introduces regularization into the estimate of the variance (actually covariance), moderating the influence of different variables on LDA.

Notes:
- Strengths: LDA is supervised, which can (but doesn't always) improve the predictive performance of the extracted features. Furthermore, LDA offers variations (i.e. quadratic LDA) to tackle specific roadblocks.
- Weaknesses: As with PCA, the new features are not easily interpretable, and you must still manually set or tune the number of components to keep. LDA also requires labeled data, which makes it more situational.
Implementations: Python / R

Linear discriminant analysis
LDA is a classification and dimensionality reduction techniques, which can be interpreted from two perspectives. The first is interpretation is probabilistic and the second, more procedure interpretation, is due to Fisher. The first interpretation is useful for understanding the assumptions of LDA. The second interpretation allows for a better understanding on how LDA performs dimensionality reduction.

The probabilistic interpretation

Each class k∈{1,…,K} is assigned a prior π^k such that that ∑ki=1π^k=1. According to Bayes’ rule, the posterior probability is

$Pr(G=k|X=x)=fk(x)πk∑Kl=\sum_{l=1}^{k}f_l(x)π_l$

where fk(x) is the density of X conditioned on k. The maximum-a-posteriori (MAP) estimator simplifies to

G(x)=argmaxkPr(G=k|X=x)=argmaxkfk(x)πk

because the denominator is identical for all classes.

LDA assumes that the density is Gaussian:

fk(x)=|2πΣk|−1/2exp(−12(x−μk)TΣ−1k(x−μk))

where Σk is the covariance matrix for the samples from class k and |.| is the determinant. LDA assumes that all classes have the same covariance matrix, i.e. Σk=Σ,∀k.

Plugging fk into the classification function, we arrive at the classification function:

G(x)=argmaxkδk(x)

where

δk(x)=xTΣ−1μk−12μTkΣ−1μk+logπk

is the discriminant function for class k. So, now that we have a classifier, how we can compute it?

To find the covariance matrix, we simply compute

Σ^=∑k=1K1N−K∑gi=k(xi−μ^k)(xi−μ^k)T.

Note that the deviation from the means is divided by N−K, the degrees of freedom, to obtain an unbiased estimator.

The means of the classes, which are also called centroids, are defined by

μ^k=1Nk∑gi=kxi.

The priors πk are set to the prevalence ratio of the class-specific observations:

π^k=NkN.

With this, we have defined all parameters required for the classifier.

The dimensionality reduction procedure of LDA involves the within-class variance, W=Σ^, and the between-class variance, B. The between-class variance indicates the deviation of centroids from the overall mean, μ^=∑Kk=1π^kμ^k, and is defined as:

B=∑k=1Kπ^k(μ^k−μ^)(μ^k−μ^)T.

Finding a sequence of optimal substeps involves three steps:

Compute the K×p matrix M containing the centroids, μk, and determine the common covariance matrix W.
Compute M∗=MW−12 using the eigen-decomposition of W.
Compute B∗ (the between-class covariance) and its eigen-decomposition B∗=V∗DBV∗T. The columns v∗l of V∗ define the coordinates of the reduced subspace.
The l-th discriminant variable (one of the K−1 new dimensions) is determined by Zl=vTlX with vl=W−12v∗l.

### Factor Analysis

Factor Analysis a technique that is used to reduce a large number of variables into fewer numbers of factors. The values of observed data are expressed as functions of a number of possible causes in order to find which are the most important. The observations are assumed to be caused by a linear transformation of lower dimensional latent factors and added Gaussian noise.

In the Factor Analysis technique, variables are grouped by their correlations, i.e., all variables in a particular group will have a high correlation among themselves, but a low correlation with variables of other group(s). Here, each group is known as a factor. These factors are small in number as compared to the original dimensions of the data. However, these factors are difficult to observe.

## Non-linear Dimensionality Reduction Methods

Non-Linear methods are more complex but can find useful reductions of the dimensions where linear methods fail. Reducing the dimensionality to only rotation and scale for Figure 1 would not be possible for a linear method. Non-linear Dimensionality Reduction methods include the
- kernel PCA
- t-SNE
- Autoencoders
- Self-Organizing Maps
- IsoMap
- UMap.

Non-linear transformation methods or manifold learning methods are used when the data doesn’t lie on a linear subspace. It is based on the manifold hypothesis which says that in a high dimensional structure, most relevant information is concentrated in small number of low dimensional manifolds. If a linear subspace is a flat sheet of paper, then a rolled up sheet of paper is a simple example of a nonlinear manifold. Informally, this is called a Swiss roll, a canonical problem in the field of non-linear dimensionality reduction. Some popular manifold learning methods are,

Shows the resulting projection from applying different manifold learning methods on a 3D S-Curve

#### **Multi-dimensional scaling (MDS)**
A technique used for analyzing similarity or dissimilarity of data as distances in a geometric spaces. Projects data to a lower dimension such that data points that are close to each other (in terms if Euclidean distance) in the higher dimension are close in the lower dimension as well.

#### **Isometric Feature Mapping (Isomap)**
Projects data to a lower dimension while preserving the geodesic distance (rather than Euclidean distance as in MDS). Geodesic distance is the shortest distance between two points on a curve.

#### **Locally Linear Embedding (LLE)**
Recovers global non-linear structure from linear fits. Each local patch of the manifold can be written as a linear, weighted sum of its neighbours given enough data.

#### **Hessian Eigenmapping (HLLE)**
Projects data to a lower dimension while preserving the local neighbourhood like LLE but uses the Hessian operator to better achieve this result and hence the name.

#### **Spectral Embedding (Laplacian Eigenmaps)**
Uses spectral techniques to perform dimensionality reduction by mapping nearby inputs to nearby outputs. It preserves locality rather than local linearity

#### **t-distributed Stochastic Neighbor Embedding (t-SNE)**
Computes the probability that pairs of data points in the high-dimensional space are related and then chooses a low-dimensional embedding which produce a similar distribution.

### Auto-encoders
Another popular dimensionality reduction method that gives spectacular results are auto-encoders, a type of artificial neural network that aims to copy their inputs to their outputs.

They compress the input into a latent-space representation, and then reconstructs the output from this representation. An autoencoder is composed of two parts :

- Encoder: compresses the input into a latent-space representation.
- Decoder: reconstruct the input from the latent space representation.

Auto Encoder Architecture

### Autoencoders
Autoencoders are neural networks that are trained to reconstruct their original inputs. For example, image autoencoders are trained to reproduce the original images instead of classifying the image as a dog or a cat.

So how is this helpful? Well, the key is to structure the hidden layer to have fewer neurons than the input/output layers. Thus, that hidden layer will learn to produce a smaller representation of the original image.

Autoencoder diagram
Because you use the input image as the target output, autoencoders are considered unsupervised. They can be used directly (e.g. image compression) or stacked in sequence (e.g. deep learning).

An autoencoder is a neural network, with as many n output units as input units, at least one hidden layer with m units where m < n, and trained with the backpropagation algorithm to reproduce the input vector onto the output layer. It reduces the numeric columns in the data by using the output of the hidden layer to represent the input vector.

The first part of the autoencoder — from the input layer to the hidden layer of m units — is called the encoder. It compresses the n dimensions of the input dataset into an m-dimensional space. The second part of the autoencoder — from the hidden layer to the output layer — is known as the decoder. The decoder expands the data vector from an m-dimensional space into the original n-dimensional dataset and brings the data back to their original values.

- Strengths: Autoencoders are neural networks, which means they perform well for certain types of data, such as image and audio data.
- Weaknesses: Autoencoders are neural networks, which means they require more data to train. They are not used as general-purpose dimensionality reduction algorithms.
Implementations: Python / R

### t-distributed Stochastic Neighbor Embedding (t-SNE)
Computes the probability that pairs of data points in the high-dimensional space are related and then chooses a low-dimensional embedding which produce a similar distribution.

This technique reduces the n numeric columns in the dataset to fewer dimensions m (m < n) based on nonlinear local relationships among the data points. Specifically, it models each high-dimensional object by a two- or three-dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points in the new lower dimensional space.

In the first step, the data points are modeled through a multivariate normal distribution of the numeric columns. In the second step, this distribution is replaced by a lower dimensional t-distribution, which follows the original multivariate normal distribution as closely as possible. The t-distribution gives the probability of picking another point in the dataset as a neighbor to the current point in the lower dimensional space. The perplexity parameter controls the density of the data as the “effective number of neighbors for any point.” The greater the value of the perplexity, the more global structure is considered in the data. The t-SNE technique works only on the current dataset. It is not possible to export the model to apply it to new data.

## Feature Importance

Methods listed under Feature Importance involve fitting a predictive model to the dataset. The objective here is to identify the top independent features that contribute to the variations in the target feature. This is done either by starting with an individual feature and adding more to see the impact on model performance or starting with the full model and eliminating one feature at a time from the model to asses the impact on model performance.

## References:

- https://towardsdatascience.com/understanding-singular-value-decomposition-and-its-application-in-data-science-388a54be95d
- https://jonathan-hui.medium.com/machine-learning-singular-value-decomposition-svd-principal-component-analysis-pca-1d45e885e491
- https://towardsdatascience.com/feature-extraction-using-principal-component-analysis-a-simplified-visual-demo-e5592ced100a
- https://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf
- https://cs.adelaide.edu.au/~javen/talk/L6%20DimensionalityReduction.pdf
- https://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch18.pdf
- http://courses.washington.edu/css581/lecture_slides/17_dimensionality_reduction.pdf
- https://www.cs.cmu.edu/~guestrin/Class/10701-S05/slides/dimensionality.pdf
- http://www.cse.psu.edu/~rtc12/CSE586Spring2010/lectures/pcaLectureShort.pdf
- https://www.mygreatlearning.com/blog/dimensionality-reduction/
- https://www.learndatasci.com/tutorials/applied-dimensionality-reduction-techniques-using-python/
- https://www.analyticsvidhya.com/blog/2018/08/dimensionality-reduction-techniques-python/
- https://elitedatascience.com/dimensionality-reduction-algorithms
- https://blog.dataiku.com/dimensionality-reduction-how-it-works-in-plain-english
- https://machinelearningmastery.com/dimensionality-reduction-for-machine-learning
- https://www.geeksforgeeks.org/dimensionality-reduction/
- https://www.geeksforgeeks.org/ml-linear-discriminant-analysis/
- https://thenewstack.io/3-new-techniques-for-data-dimensionality-reduction-in-machine-learning/
- https://towardsdatascience.com/the-basics-of-data-prep-7bb5f3af77ac
- https://towardsdatascience.com/dimensionality-reduction-for-dummies-part-3-f25729f74c0a