###### Content under Creative Commons Attribution license CC-BY 4.0 version, code under MIT license (c) 2014 by Enrique A. López-Guerra.

# Dynamics of a tapping probe that aims at seeing atoms

Purpose of the notebook: show an application of numerical methods to simulate the dynamics of a probe in atomic force microscopy.

Requirements to take the best advantage of this notebook: knowing the fundamentals of  [Harmonic Oscillators](http://en.wikipedia.org/wiki/Harmonic_oscillsator) in clasical mechanics and [Fundamentals of Vibrations](http://en.wikipedia.org/wiki/Vibration).

## Introduction

Since the atomic force microscope (AFM) was invented in 1986 it has become one of the main tools to study matter at the micro and nanoscale. This powerful tool is so versatile that it can be used to study a wide variety of materials, ranging from stiff inorganic surfaces to soft biological samples.

In its early stages the AFM was used in permanent contact with the sample (the probe is dragged over the sample during the whole operation), which brought about important drawbacks, such as rapid probe wear and often sample damage, but these obstacles have been overcome with the development of dynamic techniques.

In this Ipython notebook, we will focus on the operation of the probe in dynamic mode which is the most frequently used nowadays due to its advantages over the permanent contact techniques.

```
from IPython.display import Image;
Image(filename='C:/Users/enrique/Github Repositories/FinalProjectMAE6286_2/FinalProjectMAE6286/Fig1.jpg')
```

####Figure 1.   Schematics of the setup of a atomic force microscope (Adapted from reference 6)

In AFM the interacting probe is in general a rectangular cantilever (please check the image above that shows the AFM setup where you will be able to see the probe!).
Probably the most used dynamic technique in AFM is the Tapping Mode. In this method the probe taps a surface in intermittent contact fashion. The purpose of tapping the probe over the surface instead of dragging it is to reduce frictional forces that may cause damage of soft samples and wear of the tip. Besides with the tapping mode we can get more information about the sample!  HOW???

In Tapping Mode AFM the cantilever is shaken to oscillate up and down at a specific frequency (most of the time shaken at its natural frequency). Then the deflection of the tip is measured at that frequency to get information about the sample. Besides acquiring the topography of the sample, the phase lag between the excitation and the response of the cantilever can be related to compositional material properties!
In other words one can simultaneously get information about how the surface looks and also get compositional mapping of the surface!  THAT SOUNDS POWERFUL!!!

```
from IPython.display import Image;
Image(filename="C:/Users/enrique/Github Repositories/FinalProjectMAE6286_2/FinalProjectMAE6286/Fig2DHO.jpg")
```

####Figure 2. Schematics of a damped harmonic oscillator without tip-sample interactions

## Analytical Solution

The motion of the probe can be derived using [Euler-Bernoulli's](http://en.wikipedia.org/wiki/Euler%E2%80%93Bernoulli_beam_theory) equation. However that equation has partial derivatives (it depends on time and space) because it deals with finding the position of each point of the beam in a certain time, which cant make the problem too expensive computationally for our purposes. In our case, we have the advantage that we are only concerned about the position of the tip (which is the only part of the probe that will interact with the sample). As a consequence many researchers in AFM have successfully made approximations using a simple mass point model approximation [see ref. 2] like the one in figure 2 (with of course the addition of tip sample forces!  We will see more about this later).

First we will study the system of figure 2 AS IS (without addition of tip-sample force term), WHY?  Because we want to get an analytical solution to get a reference of how our integration schemes are working, and the addition of tip sample forces to our equation will prevent the acquisition of straightforward analytical solutions :(

Then, the equation of motion of the damped harmonic oscillator of figure 2, which is DRIVEN COSINUSOIDALLY (remember that we are exciting our probe during the scanning process) is:

$$\begin{equation}
m \frac{d^2z}{dt^2} = - k z - \frac{m\omega_0}{Q}\frac{dz}{dt} + F_0\cos(\omega t)
\end{equation}$$
where k is the stiffness of the cantilever, z is the vertical position of the tip with respect to the cantilever base position, Q is the quality factor (which is related to the damping of the system), $F_0$ is the driving force amplitude, $\omega_0$ is the resonance frequency of the oscillator, and $\omega$ is the frequency of the oscillating force.

The analytical solution of the above ODE is composed by a transient term and a steady state term. We are only interested in the steady state part because during the scanning process it is assumed that the probe has achieved that state.

The steady state solution is given by:
$$\begin{equation}
A\cos (\omega t - \phi)
\end{equation}$$

where A is the steady state amplitude of the oscillation response, which depends on the cantilever parameters and the driving parameters, as can be seen in the following relation:
$$\begin{equation}
A = \frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+(\frac{\omega\omega_0}{Q})^2}}
\end{equation}$$

and $\phi$ is given by:
$$\begin{equation}
\phi = \arctan \big( \frac{\omega\omega_0/Q}{\omega_0^2 - \omega^2} \big)
\end{equation}$$

Let's first name the variables that we are going to use. Because we are dealing with a damped harmonic oscillator model we have to include variables suche as: spring stiffness, resonance frequency, quality factor (related to damping coefficient), target oscillation amplitude, etc.

```
import numpy
import matplotlib.pyplot as plt;

k = 10.
fo = 45000
wo = 2.0*numpy.pi*fo
Q = 25.

period = 1./fo
m = k/(wo**2)
Ao = 60.e-9
Fd = k*Ao/Q

spp = 28. # time steps per period
dt = period/spp #Intentionally chosen to be quite big
#you can decrease dt by increasing the number of steps per period

simultime = 100.*period
N = int(simultime/dt)

#Analytical solution

time_an = numpy.linspace(0,simultime,N)  #time array for the analytical solution
z_an = numpy.zeros(N)  #position array for the analytical solution

#Driving force amplitude this gives us 60nm of amp response (A_target*k/Q)
Fo_an = 24.0e-9

A_an = Fo_an*Q/k  #when driven at resonance A is simply Fo*Q/k
phi = numpy.pi/2  #when driven at resonance the phase is pi/2

z_an[:] = A_an*numpy.cos(wo*time_an[:] - phi) #this gets the analytical solution

#slicing the array to include only steady state (only the last 10 periods)
z_an_steady = z_an[(90.*period/dt):]
time_an_steady = time_an[(90.*period/dt):]

plt.title('Plot 1    Analytical Steady State Solution of Eq 1', fontsize=20);
plt.xlabel('time, ms', fontsize=18);
plt.ylabel('z_Analytical, nm', fontsize=18);
plt.plot(time_an_steady*1e3, z_an_steady*1e9, 'b--');

```

### Approximating through Euler's method

This method was already discussed in the second notebook called "Pughoid Oscillation" located on the second IPython Notebook of the series _"The phugoid model of glider flight"_, the first learning module of the course [**"Practical Numerical Methods with Python."**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about).
We will review it briefly since the intention is to make this notebook self-contained. Let's begin...

If we perform a Taylor series expansion of $z_{n+1}$ around $z_{n}$ we get:

$$z_{n+1} = z_{n} + \Delta t\frac{dz}{dt}\big|_n + {\mathcal O}(\Delta t^2)$$

The Euler formula neglects terms in the order of two or higher, ending up as:

$$\begin{equation}
z_{n+1} = z_{n} + \Delta t\frac{dz}{dt}\big|_n
\end{equation}$$

It can be easily seen that the truncation error of the Euler algorithm is in the order of ${\mathcal O}(\Delta t^2)$.

This is a second order ODE, but we can convert it to a system of two coupled 1st order differential equations. To do it we will define $\frac{dz}{dt} = v$. Then equation (1) will be decomposed as:
$$\begin{equation}
\frac{dz}{dt} = v
\end{equation}$$

$$\begin{equation}
\frac{dv}{dt} = -kz-\frac{m\omega_0}{Q}+F_o\cos(\omega t)
\end{equation}$$

These coupled equations will be used during Euler's aproximation and also during our integration using Runge Kutta 4 method.

```
t= numpy.linspace(0,simultime,N) #time grid for Euler method

#Initializing variables for Euler
vdot_E = numpy.zeros(N)
v_E = numpy.zeros(N)
z_E = numpy.zeros(N)

#Initial conditions
z_E[0]= 0.0
v_E[0]=0.0

for i in range (N-1):
    vdot_E[i] =( ( -k*z_E[i] - (m*wo/Q)*(v_E[i])   +\
                    Fd*numpy.cos(wo*t[i]) ) / m)  #Equation 7
    v_E[i+1] = v_E[i] + dt*vdot_E[i]  #Based on equation 5
    z_E[i+1] = z_E[i] + v_E[i]*dt  #Equation 5

plt.title('Plot 2  Eulers approximation of Equation1', fontsize=20);
plt.plot(t*1e3,z_E*1e9);
plt.xlabel('time, s', fontsize=18);
plt.ylabel('z_Euler, nm', fontsize=18);

```

This looks totally unphysical! We were expecting to have a steady state oscillation of 60 nm and we got a huge oscillation that keeps growing. Can it be due to the scheme? The timestep that we have chosen is quite big with respect to the oscillation period. We have intentionally set it to ONLY 28 time steps per period (That could be the reason why the scheme can't capture the physics of the problem). That's quite discouraging. However the timestep is quite big and it really gets better as you decrease the time step. Try it! Reduce the time step and see how the numerical solution acquires an amplitude of 60 nm as the analytical one. At this point we can't state anything about accuracy before doing an analysis of error (we will make this soon). But first, let's try to analyze if another more efficient scheme can capture the physics of our damped harmonic oscillator even with this large time step.

### Let's try to get more accurate... Verlet Algorithm

This is a very popular algorithm widely used in molecular dynamics simulations. Its popularity has been related to high stability when compared to the simple Euler method, it is also very simple to implement and accurate as we will see soon!  Verlet integration can be seen as using the central difference approximation to the second derivative. Consider the Taylor expansion of $z_{n+1}$ and $z_{n-1}$ around $z_n$:

$$\begin{equation}
z_{n+1} = z_n + \Delta t \frac{dz}{dt}\big|_n + \frac{\Delta t^2}{2} \frac{d^2 z}{d t^2}\big|_n + \frac{\Delta t^3}{6} \frac{d^3 z}{d t^3}\big|_n + {\mathcal O}(\Delta t^4)
\end{equation}$$

$$\begin{equation}
z_{n-1} = z_n - \Delta t \frac{dz}{dt}\big|_n + \frac{\Delta t^2}{2} \frac{d^2 z}{dt^2}\big|_n - \frac{\Delta t^3}{6}
\frac{d^3 z}{d t^3}\big|_n + {\mathcal O}(\Delta t^4)
\end{equation}$$

Adding up these two expansions and solving for $z_{n+1}$ we get:

$$z_{n+1}= 2z_{n} - z_{n-1} + \frac{d^2 z}{d t^2} \Delta t^2\big|_n + {\mathcal O}(\Delta t^4) $$

Verlet algorithm neglects terms on the order of 4 or higher, ending up with:

$$\begin{equation}
z_{n+1}= 2z_{n} - z_{n-1} + \frac{d^2 z}{d t^2} \Delta t^2\big|_n
\end{equation}$$

This looks nice; it seems that the straightforward calculation of the second derivative will give us good results. BUT have you seen that we also need the value of the first derivative (velocity) to put it into the equation of motion that we are integrating (see equation 1). YES, that's a main drawback of this scheme and therefore it's mainly used in applications where the equation to be integrated doesn't have first derivative. But don't panic we will see what can we do...

What about subtracting equations 8 and 9 and then solving for $\frac{dz}{dt}\big|_n$:
$$
\frac{dz}{dt}\big|_n =  \frac{z_{n+1} - z_{n-1}}{2\Delta t} + {\mathcal O}(\Delta t^2)
$$
If we neglect terms on the order of 2 or higher we can calculate velocity:
$$\begin{equation}
\frac{dz}{dt}\big|_n =  \frac{z_{n+1} - z_{n-1}}{2\Delta t}
\end{equation}$$

This way of calculating velocity is pretty common in Verlet integration in applications where velocity is not explicit in the equation of motion. However for our purposes of solving equation 1 (where first derivative is explicitly present) it seems that we will lose accuracy because of the velocity, we will discuss more about this soon after...

Have you noticed that we need a value $z_{n-1}$? Does it sound familiar?  YES!  This is not a self-starting method. As a result we will have to overcome the issue by setting the initial conditions of the first step using Euler approximation. This is a bit annoying, but a couple of extra lines of code won't kill you :)

```
time_V = numpy.linspace(0,simultime,N)

#Initializing variables for Verlet
zdoubledot_V = numpy.zeros(N)
zdot_V = numpy.zeros(N)
z_V = numpy.zeros(N)

#Initial conditions Verlet. Look how we use Euler for the first step approximation!
z_V[0] = 0.0
zdot_V[0] = 0.0
zdoubledot_V[0] = (   ( -k*z_V[0] - (m*wo/Q)*zdot_V[0] +\
                            Fd*numpy.cos(wo*t[0]) )  ) / m
zdot_V[1] = zdot_V[0] + zdoubledot_V[0]*dt
z_V[1] = z_V[0] + zdot_V[0]*dt
zdoubledot_V[1] = (   ( -k*z_V[1] - (m*wo/Q)*zdot_V[1] +\
                            Fd*numpy.cos(wo*t[1]) )  ) / m

#VERLET ALGORITHM

for i in range(2,N):
    z_V[i] = 2*z_V[i-1] - z_V[i-2] + zdoubledot_V[i-1]*dt**2 #Eq 10
    zdot_V[i] = (z_V[i]-z_V[i-2])/(2.0*dt) #Eq 11
    zdoubledot_V[i] = (   ( -k*z_V[i] - (m*wo/Q)*zdot_V[i] +\
                               Fd*numpy.cos(wo*t[i]) )  ) / m #from eq 1

plt.title('Plot 3  Verlet approximation of Equation1', fontsize=20);
plt.xlabel('time, ms', fontsize=18);
plt.ylabel('z_Verlet, nm', fontsize=18);
plt.plot(time_V*1e3, z_V*1e9, 'g-');
plt.ylim(-65,65);

```

It WAS ABLE to capture the physics! Even with the big time step that we use with Euler scheme!

As you can see, and as we previously discussed the harmonic response is composed of a transient and a steady part. We are only concerned about the steady-state, since it is assumed that the probe achieves steady state motion during the imaging process. Therefore, we are going to slice our array in order to show only the last 10 oscillations, and we will see if it resembles the analytical solution.

```
#Slicing the full response vector to get the steady state response
z_steady_V = z_V[(90*period/dt):]
time_steady_V = time_V[(90*period/dt):]

plt.title('Plot 3  Verlet approx. of steady state sol. of Eq 1', fontsize=20);
plt.xlabel('time, ms', fontsize=18);
plt.ylabel('z_Verlet, nm', fontsize=18);
plt.plot(time_steady_V*1e3, z_steady_V*1e9, 'g-');
plt.ylim(-65,65);
plt.show();

```

### Let's use now one of the most popular schemes...     The Runge Kutta 4!

The Runge Kutta 4 (RK4) method is very popular for the solution of ODEs. This method is designed to solve 1st order differential equations. We have converted our 2nd order ODE to a system of two coupled 1st order ODEs when we implemented the Euler scheme (equations 5 and 6). And we will  have to use these equations for the RK4 algorithm.

In order to clearly see the RK4 implementation we are going to put equations 5 and 6 in the following form:
$$\begin{equation}
\frac{dz}{dt}=v \Rightarrow f1(t,z,v)
\end{equation}$$
$$\begin{equation}
\frac{dv}{dt} = -kz-\frac{m\omega_0}{Q}+F_ocos(\omega t) \Rightarrow f2(t,z,v)
\end{equation}$$

It can be clearly seen that we have two coupled equations f1 and f2 and both depend in t, z, and v.

The RK4 equations for our special case where we have two coupled equations, are the following:
$$\begin{equation}
k_1 = f1(t_i, z_i, v_i)
\end{equation}$$
$$\begin{equation}
m_1 = f2(t_i, z_i, v_i)
\end{equation}$$
$$\begin{equation}
k_2 = f1(t_i +1/2\Delta t, z_i + 1/2k_1\Delta t, v_i + 1/2m_1\Delta t)
\end{equation}$$
$$\begin{equation}
m_2 = f2(t_i +1/2\Delta t, z_i + 1/2k_1\Delta t, v_i + 1/2m_1\Delta t)
\end{equation}$$
$$\begin{equation}
k_3 = f1(t_i +1/2\Delta t, z_i + k_2\Delta t, v_i + 1/2m_2\Delta t)
\end{equation}$$
$$\begin{equation}
m_3 = f2(t_i +1/2\Delta t, z_i + 1/2k_2\Delta t, v_i + 1/2m_2\Delta t)
\end{equation}$$
$$\begin{equation}
k_4 = f1(t_i + \Delta t, z_i + k_3\Delta t, v_i + m_3\Delta t)
\end{equation}$$
$$\begin{equation}
k_4 = f2(t_i + \Delta t, z_i + k_3\Delta t, v_i + m_3\Delta t)
\end{equation}$$

$$\begin{equation}
f1_{n+1} = f1_n + \Delta t/6(k_1+2k_2+2k_3+k_4)
\end{equation}$$
$$\begin{equation}
f2_{n+1} = f2_n + \Delta t/6(m_1+2m_2+2m_3+m_4)
\end{equation}$$

Please notice how k values and m values are used sequentially, since it is crucial in the implementation of the method!

```
#Definition of v, z, vectors
vdot_RK4 = numpy.zeros(N)
v_RK4 = numpy.zeros(N)
z_RK4 = numpy.zeros(N)
k1v_RK4 = numpy.zeros(N)
k2v_RK4 = numpy.zeros(N)
k3v_RK4 = numpy.zeros(N)
k4v_RK4 = numpy.zeros(N)

k1z_RK4 = numpy.zeros(N)
k2z_RK4 = numpy.zeros(N)
k3z_RK4 = numpy.zeros(N)
k4z_RK4 = numpy.zeros(N)

#calculation of velocities RK4

#INITIAL CONDITIONS
v_RK4[0] = 0
z_RK4[0] = 0

for i in range (1,N):
       #RK4
        k1z_RK4[i] = v_RK4[i-1]   #k1 Equation 14
        k1v_RK4[i] = ((   ( -k*z_RK4[i-1] - (m*wo/Q)*v_RK4[i-1] + \
                           Fd*numpy.cos(wo*t[i-1]) )  ) / m )   #m1 Equation 15

        k2z_RK4[i] = ((v_RK4[i-1])+k1v_RK4[i]/2.*dt) #k2 Equation 16
        k2v_RK4[i] = ((   ( -k*(z_RK4[i-1]+ k1z_RK4[i]/2.*dt) - (m*wo/Q)*\
                           (v_RK4[i-1] +k1v_RK4[i]/2.*dt) + Fd*\
                           numpy.cos(wo*(t[i-1] + dt/2.)) )  ) / m ) #m2 Eq 17

        k3z_RK4[i] = ((v_RK4[i-1])+k2v_RK4[i]/2.*dt) #k3, Equation 18
        k3v_RK4[i] = ((   ( -k*(z_RK4[i-1]+ k2z_RK4[i]/2.*dt) - (m*wo/Q)*\
                            (v_RK4[i-1] +k2v_RK4[i]/2.*dt) + Fd*\
                            numpy.cos(wo*(t[i-1] + dt/2.)) )  ) / m ) #m3, Eq 19

        k4z_RK4[i] = ((v_RK4[i-1])+k3v_RK4[i]*dt) #k4, Equation 20
        k4v_RK4[i] = ((   ( -k*(z_RK4[i-1] + k3z_RK4[i]*dt) - (m*wo/Q)*\
                           (v_RK4[i-1] + k3v_RK4[i]*dt) + Fd*\
                           numpy.cos(wo*(t[i-1] + dt)) )  ) / m )#m4, Eq 21

        #Calculation of velocity, Equation 23
        v_RK4[i] = v_RK4[i-1] + 1./6*dt*(k1v_RK4[i] + 2.*k2v_RK4[i] +\
                        2.*k3v_RK4[i] + k4v_RK4[i] )
        #calculation of position, Equation 22
        z_RK4 [i] = z_RK4[i-1] + 1./6*dt*(k1z_RK4[i] + 2.*k2z_RK4[i] +\
                        2.*k3z_RK4[i] + k4z_RK4[i] )

#slicing array to get steady state
z_steady_RK4 = z_RK4[(90.*period/dt):]
time_steady_RK4 = t[(90.*period/dt):]

plt.title('Plot 3  RK4 approx. of steady state sol. of Eq 1', fontsize=20);
plt.xlabel('time, ms', fontsize=18);
plt.ylabel('z_RK4, nm', fontsize=18);
plt.plot(time_steady_RK4 *1e3, z_steady_RK4*1e9, 'r-');
plt.ylim(-65,65);
plt.show();

```

### Error Analysis

Let's plot together our solutions using the different schemes along with our analytical reference.

```
plt.title('Plot 4  Schemes comparison with analytical sol.', fontsize=20);
plt.plot(time_an_steady*1e3, z_an_steady*1e9, 'b--' );
plt.plot(time_steady_V*1e3, z_steady_V*1e9, 'g-' );
plt.plot(time_steady_RK4*1e3, z_steady_RK4*1e9, 'r-');
plt.xlim(2.0, 2.06);
plt.legend(['Analytical solution', 'Verlet method', 'Runge Kutta 4']);
plt.xlabel('time, ms', fontsize=18);
plt.ylabel('z_position, nm', fontsize=18);

```

It was pointless to include Euler in the last plot because it was not following the physics at all for this given time step. REMEMBER that Euler can give fair approximations, but you MUST decrease the time step in this particular case if you want to see the sinusoidal trajectory!
It seems our different schemes are giving different quality in approximating the solution. However it's hard to conclude something strong based on this qualitative observations. In order to state something stronger we have to perform an error analysis. We will choose L1 norm for this purpose (You can find more information about this [L1](http://en.wikipedia.org/wiki/Taxicab_geometry) ) and it was also discussed on the second IPython Notebook of the series _"The phugoid model of glider flight"_, the first learning module of the course [**"Practical Numerical Methods with Python."**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about).

```
"""ERROR ANALYSIS EULER, VERLET AND RK4"""

# time-increment array
dt_values = numpy.array([8.0e-7, 2.0e-7, 0.5e-7, 1e-8, 0.1e-8])

# array that will contain solution of each grid
z_values_E = numpy.zeros_like(dt_values, dtype=numpy.ndarray)
z_values_V = numpy.zeros_like(dt_values, dtype=numpy.ndarray)
z_values_RK4 = numpy.zeros_like(dt_values, dtype=numpy.ndarray)
z_values_an = numpy.zeros_like(dt_values, dtype=numpy.ndarray)

for n, dt in enumerate(dt_values):
    simultime = 100*period
    timestep = dt
    N = int(simultime/dt)
    t = numpy.linspace(0.0, simultime, N)

    #Initializing variables for Verlet
    zdoubledot_V = numpy.zeros(N)
    zdot_V = numpy.zeros(N)
    z_V = numpy.zeros(N)

    #Initializing variables for RK4
    vdot_RK4 = numpy.zeros(N)
    v_RK4 = numpy.zeros(N)
    z_RK4 = numpy.zeros(N)
    k1v_RK4 = numpy.zeros(N)
    k2v_RK4 = numpy.zeros(N)
    k3v_RK4 = numpy.zeros(N)
    k4v_RK4 = numpy.zeros(N)

    k1z_RK4 = numpy.zeros(N)
    k2z_RK4 = numpy.zeros(N)
    k3z_RK4 = numpy.zeros(N)
    k4z_RK4 = numpy.zeros(N)

    #Initial conditions Verlet (started with Euler approximation)
    z_V[0] = 0.0
    zdot_V[0] = 0.0
    zdoubledot_V[0] = (   ( -k*z_V[0] - (m*wo/Q)*zdot_V[0] + \
                           Fd*numpy.cos(wo*t[0]) )  ) / m
    zdot_V[1] = zdot_V[0] + zdoubledot_V[0]*timestep**2
    z_V[1] = z_V[0] + zdot_V[0]*dt
    zdoubledot_V[1] = (   ( -k*z_V[1] - (m*wo/Q)*zdot_V[1] + \
                           Fd*numpy.cos(wo*t[1]) )  ) / m

    #Initial conditions Runge Kutta
    v_RK4[1] = 0
    z_RK4[1] = 0

    #Initialization variables for Analytical solution
    z_an = numpy.zeros(N)

    # time loop
    for i in range(2,N):

        #Verlet
        z_V[i] = 2*z_V[i-1] - z_V[i-2] + zdoubledot_V[i-1]*dt**2 #Eq 10
        zdot_V[i] = (z_V[i]-z_V[i-2])/(2.0*dt) #Eq 11
        zdoubledot_V[i] = (   ( -k*z_V[i] - (m*wo/Q)*zdot_V[i] +\
                                   Fd*numpy.cos(wo*t[i]) )  ) / m #from eq 1

        #RK4
        k1z_RK4[i] = v_RK4[i-1]   #k1 Equation 14
        k1v_RK4[i] = ((   ( -k*z_RK4[i-1] - (m*wo/Q)*v_RK4[i-1] + \
                           Fd*numpy.cos(wo*t[i-1]) )  ) / m )   #m1 Equation 15

        k2z_RK4[i] = ((v_RK4[i-1])+k1v_RK4[i]/2.*dt) #k2 Equation 16
        k2v_RK4[i] = ((   ( -k*(z_RK4[i-1]+ k1z_RK4[i]/2.*dt) - (m*wo/Q)*\
                           (v_RK4[i-1] +k1v_RK4[i]/2.*dt) + Fd*\
                           numpy.cos(wo*(t[i-1] + dt/2.)) )  ) / m )  #m2 Eq 17

        k3z_RK4[i] = ((v_RK4[i-1])+k2v_RK4[i]/2.*dt) #k3, Equation 18
        k3v_RK4[i] = ((   ( -k*(z_RK4[i-1]+ k2z_RK4[i]/2.*dt) - (m*wo/Q)*\
                            (v_RK4[i-1] +k2v_RK4[i]/2.*dt) + Fd*\
                            numpy.cos(wo*(t[i-1] + dt/2.)) )  ) / m ) #m3, Eq 19

        k4z_RK4[i] = ((v_RK4[i-1])+k3v_RK4[i]*dt) #k4, Equation 20
        k4v_RK4[i] = ((   ( -k*(z_RK4[i-1] + k3z_RK4[i]*dt) - (m*wo/Q)*\
                           (v_RK4[i-1] + k3v_RK4[i]*dt) + Fd*\
                           numpy.cos(wo*(t[i-1] + dt)) )  ) / m )#m4, Equation 21

        #Calculation of velocity, Equation 23
        v_RK4[i] = v_RK4[i-1] + 1./6*dt*(k1v_RK4[i] + 2.*k2v_RK4[i] +\
                        2.*k3v_RK4[i] + k4v_RK4[i] )
        #calculation of position, Equation 22
        z_RK4 [i] = z_RK4[i-1] + 1./6*dt*(k1z_RK4[i] + 2.*k2z_RK4[i] +\
                         2.*k3z_RK4[i] + k4z_RK4[i] )

        #Analytical solution
        A_an = Fo_an*Q/k  #when driven at resonance A is simply Fo*Q/k
        phi = numpy.pi/2  #when driven at resonance the phase is pi/2
        z_an[i] = A_an*numpy.cos(wo*t[i] - phi)  #Analytical solution eq. 1

    #Slicing the full response vector to get the steady state response
    z_steady_V = z_V[(80*period/timestep):]
    z_an_steady = z_an[(80*period/timestep):]
    z_steady_RK4 = z_RK4[(80*period/timestep):]
    time_steady = t[(80*period/timestep):]

    z_values_V[n] = z_steady_V.copy()  # error for certain value of timestep
    z_values_RK4[n] = z_steady_RK4.copy() #error for certain value of timestep
    z_values_an[n] = z_an_steady.copy()  #error for certain value of timestep

def get_error(z, z_exact, dt):
    #Returns the error with respect to the analytical solution using L1 norm

    return dt * numpy.sum(numpy.abs(z-z_exact))

#NOW CALCULATE THE ERROR FOR EACH RESPECTIVE DELTA T
error_values_V = numpy.zeros_like(dt_values)
error_values_RK4 = numpy.zeros_like(dt_values)

for i, dt in enumerate(dt_values):
    ### call the function get_error() ###
    error_values_V[i] = get_error(z_values_V[i], z_values_an[i], dt)
    error_values_RK4[i] = get_error(z_values_RK4[i], z_values_an[i], dt)

plt.figure(1);
plt.title('Plot 5  Error analysis Verlet based on L1 norm', fontsize=20);
plt.tick_params(axis='both', labelsize=14);
plt.grid(True);                         #turn on grid lines
plt.xlabel('$\Delta t$ Verlet', fontsize=16);  #x label
plt.ylabel('Error Verlet', fontsize=16);       #y label
plt.loglog(dt_values, error_values_V, 'go-');  #log-log plot
plt.axis('equal');                      #make axes scale equally;

plt.figure(2);
plt.title('Plot 6  Error analysis RK4 based on L1 norm', fontsize=20);
plt.tick_params(axis='both', labelsize=14);
plt.grid(True);     #turn on grid lines
plt.xlabel('$\Delta t$ RK4', fontsize=16);  #x label
plt.ylabel('Error RK4', fontsize=16);       #y label
plt.loglog(dt_values, error_values_RK4,  'co-');  #log-log plot
plt.axis('equal');                      #make axes scale equally;

```

As we can see Runge Kutta 4 converges faster than Verlet for the range of time steps studied. And the difference between both is near one order of magnitude. One additional advantage with Runge Kutta 4 is that the method is very stable, even with big time steps (eg. 10 time steps per period) the method is able to catch up the physics of the oscillation, something where Verlet is not so good at.

### Let's add a sample and oscillate our probe over it!

It is very common in the field of probe microscopy to model the tip sample interactions through DMT contact mechanics.
DMT stands for Derjaguin, Muller and Toporov who were the scientists that developed the model (see ref 1). This model uses Hertz contact mechanics (see ref 2) with the addition of long range tip-sample interactions. These long range tip-sample interactions are ascribed to intermolecular interactions between the atoms of the tip and the upper atoms of the surface, and include mainly the contribution of van de Waals forces and Pauli repulsion from electronic clouds when the atoms of the tip meet closely the atoms of the surface. Figure 2 displays a force vs distance curve (FD curve) where it is shown how the forces between the tip and the sample behave with respect to the separation. It can be seen that at positive distances the tip starts "feeling" attraction from the tip (from the contribution of van der Waals forces) where the slope of the curve is positive and at some minimum distance ($a_0$) the tip starts experiencing repulsive interactions arising from electronic cloud repulsion (area where the slope of the curve is negative and the forces are negative). At lower distances, an area known as "contact area" arises and it is characterized by a negative slope and an emerging positive force.

```
from IPython.display import Image;
Image(filename="C:/Users/enrique/Github Repositories/FinalProjectMAE6286_2/FinalProjectMAE6286/Fig3FDcurve.jpg")
```

####Figure 3.   Force vs Distance profile depicting tip-sample interactions in AFM (Adapted from reference 6)

In Hertz contact mechanics, one central aspect is to consider that the contact area increases as the sphere is pressed against an elastic surface, and this increase of the contact area "modulates" the effective stiffness of the sample. This concept is represented in figure 4 where the sample is depicted as comprised by a series of springs that are activated as the tip goes deeper into the sample. In other words, the deeper the sample goes, the larger the contact area and therefore more springs are activated (see more about this on reference 5).

```
from IPython.display import Image;
Image(filename="C:/Users/enrique/Github Repositories/FinalProjectMAE6286_2/FinalProjectMAE6286/Fig4Hertzspring.jpg")
```

####Figure 4.  Conceptual representation of Hertz contact mechanics
This concept is represented mathematically by a non-linear spring whose elastic coefficient is a function of the contact area which at the same time depends on the sample indentation ( k(d)  ).
$$F_{ts} = k(d)d$$
where
$$k(d) = 4/3E*\sqrt{Rd}$$
being $\sqrt{Rd}$ the contact area when a sphere of radius R indents a half-space to depth d.
$E*$ is the effective Young's modulus of the tip-sample interaction.

The long range attractive forces are derived using Hamaker's equation (see reference 4): $if$ $d > a_0$
$$F_{ts} = \frac{-HR}{6d^2}$$

where H is the Hamaker constant, R the tip radius and d the tip sample distance. $a_0$ is defined as the intermolecular distance and normally is chosen to be 0.2 nm.

In summary the equations that we will include in our code to take care of the tip sample interactions are the following:

$$\begin{equation}
Fts_{DMT} = \begin{cases} \frac{-HR}{6d^2} \quad \quad d \leq{a_0}\\ \\
\frac{-HR}{6d^2} + 4/3E*R^{1/2}d^{3/2}   \quad \quad d> a_0 \end{cases}
\end{equation}$$

where the effective Young's modulus E* is defined by:
$$\begin{equation}
1/E* = \frac{1-\nu^2}{E_t}+\frac{1-\nu^2}{E_s}
\end{equation}$$
where $E_t$ and $E_s$ are the tip and sample Young's modulus respectively. $\nu_t$ and $\nu_s$ are tip and sample Poisson ratios, respectively.

##### Enough theory, Let's make our code!

Now we will have to solve equation (1) but with the addition of tip-sample interactions which are described by equation (5). So we have a second order non-linear ODE which is no longer analytically straightforward:

$$\begin{equation}
m \frac{d^2z}{dt^2} = - k z - \frac{m\omega_0}{Q}\frac{dz}{dt} + F_0 cos(\omega t) + Fts_{DMT}
\end{equation}$$

Therefore we have to use numerical methods to solve it. RK4 has shown to be more accurate to solve equation (1) among the methods reviewed in the previous section of the notebook, and therefore it is going to be the chosen method to solve equation (6).

Now we have to declare all the variables related to the tip-sample forces. Since we are modeling our tip-sample forces using Hertz contact mechanics with addition of long range Van der Waals forces we have to define the Young's modulus of the tip and sample, the diameter of the tip of our probe, Poisson ratio, etc.

```
#DMT parameters  (Hertz contact mechanics with long range Van der Waals forces added
a=0.2e-9 #intermolecular parameter
H=6.4e-20 #hamaker constant of sample
R=20e-9 #tip radius of the cantilever
Es=70e6 #elastic modulus of sample
Et=130e9 #elastic modulus of the tip
vt=0.3 #Poisson coefficient for tip
vs=0.3 #Poisson coefficient for sample
E_star= 1/((1-pow(vt,2))/Et+(1-pow(vs,2))/Es)  #Effective Young Modulus
```

Now let's declare the timestep, the simulation time and let's oscillate our probe!

```
#IMPORTANT distance where you place the probe above the sample
z_base = 40.e-9

spp = 280. # time steps per period
dt = period/spp

simultime = 100.*period
N = int(simultime/dt)
t = numpy.linspace(0,simultime,N)

#Initializing variables for RK4
v_RK4 = numpy.zeros(N)
z_RK4 = numpy.zeros(N)
k1v_RK4 = numpy.zeros(N)
k2v_RK4 = numpy.zeros(N)
k3v_RK4 = numpy.zeros(N)
k4v_RK4 = numpy.zeros(N)

k1z_RK4 = numpy.zeros(N)
k2z_RK4 = numpy.zeros(N)
k3z_RK4 = numpy.zeros(N)
k4z_RK4 = numpy.zeros(N)

TipPos = numpy.zeros(N)
Fts = numpy.zeros(N)
Fcos = numpy.zeros(N)

for i in range(1,N):
   #RK4
    k1z_RK4[i] = v_RK4[i-1]   #k1 Equation 14
    k1v_RK4[i] = ((   ( -k*z_RK4[i-1] - (m*wo/Q)*v_RK4[i-1] + \
                       Fd*numpy.cos(wo*t[i-1]) +Fts[i-1])  ) / m )   #m1 Equation 15

    k2z_RK4[i] = ((v_RK4[i-1])+k1v_RK4[i]/2.*dt) #k2 Equation 16
    k2v_RK4[i] = ((   ( -k*(z_RK4[i-1]+ k1z_RK4[i]/2.*dt) - (m*wo/Q)*\
                       (v_RK4[i-1] +k1v_RK4[i]/2.*dt) + Fd*\
                       numpy.cos(wo*(t[i-1] + dt/2.))  +Fts[i-1])  ) / m )  #m2 Eq 17

    k3z_RK4[i] = ((v_RK4[i-1])+k2v_RK4[i]/2.*dt) #k3, Equation 18
    k3v_RK4[i] = ((   ( -k*(z_RK4[i-1]+ k2z_RK4[i]/2.*dt) - (m*wo/Q)*\
                        (v_RK4[i-1] +k2v_RK4[i]/2.*dt) + Fd*\
                        numpy.cos(wo*(t[i-1] + dt/2.))  +Fts[i-1])  ) / m ) #m3, Eq19

    k4z_RK4[i] = ((v_RK4[i-1])+k3v_RK4[i]*dt) #k4, Equation 20
    k4v_RK4[i] = ((   ( -k*(z_RK4[i-1] + k3z_RK4[i]*dt) - (m*wo/Q)*\
                       (v_RK4[i-1] + k3v_RK4[i]*dt) + Fd*\
                       numpy.cos(wo*(t[i-1] + dt))  +Fts[i-1])  ) / m )#m4, Eq 21

    #Calculation of velocity, Equation 23
    v_RK4[i] = v_RK4[i-1] + 1./6*dt*(k1v_RK4[i] + 2.*k2v_RK4[i] +\
                    2.*k3v_RK4[i] + k4v_RK4[i] )
    #calculation of position, Equation 22
    z_RK4 [i] = z_RK4[i-1] + 1./6*dt*(k1z_RK4[i] + 2.*k2z_RK4[i] +\
                     2.*k3z_RK4[i] + k4z_RK4[i] )

    TipPos[i] = z_base + z_RK4[i] #Adding base position to z position

    #calculation of DMT force

    if  TipPos[i] > a: #this defines the attractive regime
        Fts[i] = -H*R/(6*(TipPos[i])**2)
    else:  #this defines the repulsive regime
        Fts[i] = -H*R/(6*a**2)+4./3*E_star*numpy.sqrt(R)*(a-TipPos[i])**1.5

    Fcos[i] = Fd*numpy.cos(wo*t[i])  #Driving force (this will be helpful to plot the driving force)

#Slicing arrays to get steady state
TipPos_steady = TipPos[(95*period/dt):]
t_steady = t[(95*period/dt):]
Fcos_steady = Fcos[(95*period/dt):]
Fts_steady = Fts[(95*period/dt):]

plt.figure(1);
fig, ax1 = plt.subplots();
ax2 = ax1.twinx();
ax1.plot(t_steady*1e3,TipPos_steady*1e9, 'g-');
ax2.plot(t_steady*1e3, Fcos_steady*1e9, 'b-');
ax1.set_xlabel('Time,s');
ax1.set_ylabel('Tip position (nm)', color='g');
ax2.set_ylabel('Drive Force (nm)', color='b');
plt.title('Plot 7   Tip response and driving force', fontsize = 20);
plt.show();

plt.figure(2);
plt.title('Plot 8  Force-Distance curve', fontsize=20);
plt.plot(TipPos*1e9, Fts*1e9, 'b--' );
plt.xlabel('Tip Position, nm', fontsize=18);
plt.ylabel('Force, nN', fontsize=18);
plt.xlim(-20, 30);

```

Check that we have two sinusoidals. The one in green (the output) is the response signal of the tip (the tip trajectory in time) while the blue one (the input) is the cosinusoidal driving force that we are using to excite the tip. When the tip is excited in free air (without tip sample interactions) the phase lag between the output and the input is 90 degrees. You can test that with the previous code by only changing the position of the base to a high-enough position that it does not interact with the sample. However in the above plot the phase lag is less than 90 degrees. Interestingly the phase can give relative information about the material properties of the sample. There is a well-developed theory of this in tapping mode AFM and it's called phase spectroscopy. If you are interested in this topic you can read reference 1.
Also look at the above plot and see that the response amplitude is no longer 60 nm as we initially set (in this case is near 45 nm!). It means that we have experienced a significant amplitude reduction due to the tip sample interactions.
Besides with the data acquired we are able to plot a Force-curve as the one shown in Figure 3! It shows the attractive and repulsive interactions of our probe with the surface.

We have arrived to the end of the notebook. I hope you have found it interesting!

```
from IPython.core.display import HTML
css_file = 'C:/Users/enrique/Github Repositories/FinalProjectMAE6286_2/FinalProjectMAE6286/numericalmoocstyle.css'
HTML(open(css_file, "r").read())

```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

REFERENCES

1. Garcı́a, Ricardo, and Ruben Perez. "Dynamic atomic force microscopy methods." Surface science reports 47.6 (2002): 197-301.
2. B. V. Derjaguin, V. M. Muller, and Y. P. Toporov, J. Colloid
Interface Sci. 53, 314  (1975)
3. Hertz, H. R., 1882, Ueber die Beruehrung elastischer Koerper (On Contact Between Elastic Bodies), in Gesammelte Werke (Collected Works), Vol. 1, Leipzig, Germany, 1895.
4. Van Oss, Carel J., Manoj K. Chaudhury, and Robert J. Good. "Interfacial Lifshitz-van der Waals and polar interactions in macroscopic systems." Chemical Reviews 88.6 (1988): 927-941.
5. Enrique A. López-Guerra, and Santiago D. Solares. "Modeling viscoelasticity through spring–dashpot models in intermittent-contact atomic force microscopy." Beilstein journal of nanotechnology 5, no. 1 (2014): 2149-2163.
6. Enrique A. López-Guerra, and Santiago D. Solares, "El microscopio de Fuerza Atómica: Metodos y Aplicaciones." Revista UVG (2013) No. 28, 14-23..

```

```


#Linear Algebra - Motivation by an application#

Let us consider the $\textit{wine}$ data set from scikit-learn package. Let us load the data set and analyze its structure.

We shall check the following:

1.   Number of instances (or samples)
2.   Number of attributes (or features)
3.   Description of attributes
4.   Type of class labels (which might not be necessary for today's discussion)

```python
import numpy as np #numpy package will be useful for most of the array operations in the code
from sklearn.datasets import load_wine
```

```python
#Let us load the wine data and store the result in X_y variable. The idea behind naming of the variable will be illustrated in the next few tabs.
X_y = load_wine()
```

```python
#Let us print the contents of the X_y variable

print(type(X_y))
print(X_y)
```

    <class 'sklearn.utils.Bunch'>
    {'data': array([[1.423e+01, 1.710e+00, 2.430e+00, ..., 1.040e+00, 3.920e+00,
            1.065e+03],
           [1.320e+01, 1.780e+00, 2.140e+00, ..., 1.050e+00, 3.400e+00,
            1.050e+03],
           [1.316e+01, 2.360e+00, 2.670e+00, ..., 1.030e+00, 3.170e+00,
            1.185e+03],
           ...,
           [1.327e+01, 4.280e+00, 2.260e+00, ..., 5.900e-01, 1.560e+00,
            8.350e+02],
           [1.317e+01, 2.590e+00, 2.370e+00, ..., 6.000e-01, 1.620e+00,
            8.400e+02],
           [1.413e+01, 4.100e+00, 2.740e+00, ..., 6.100e-01, 1.600e+00,
            5.600e+02]]), 'target': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2]), 'target_names': array(['class_0', 'class_1', 'class_2'], dtype='<U7'), 'DESCR': '.. _wine_dataset:\n\nWine recognition dataset\n------------------------\n\n**Data Set Characteristics:**\n\n    :Number of Instances: 178 (50 in each of three classes)\n    :Number of Attributes: 13 numeric, predictive attributes and the class\n    :Attribute Information:\n \t\t- Alcohol\n \t\t- Malic acid\n \t\t- Ash\n\t\t- Alcalinity of ash  \n \t\t- Magnesium\n\t\t- Total phenols\n \t\t- Flavanoids\n \t\t- Nonflavanoid phenols\n \t\t- Proanthocyanins\n\t\t- Color intensity\n \t\t- Hue\n \t\t- OD280/OD315 of diluted wines\n \t\t- Proline\n\n    - class:\n            - class_0\n            - class_1\n            - class_2\n\t\t\n    :Summary Statistics:\n    \n    ============================= ==== ===== ======= =====\n                                   Min   Max   Mean     SD\n    ============================= ==== ===== ======= =====\n    Alcohol:                      11.0  14.8    13.0   0.8\n    Malic Acid:                   0.74  5.80    2.34  1.12\n    Ash:                          1.36  3.23    2.36  0.27\n    Alcalinity of Ash:            10.6  30.0    19.5   3.3\n    Magnesium:                    70.0 162.0    99.7  14.3\n    Total Phenols:                0.98  3.88    2.29  0.63\n    Flavanoids:                   0.34  5.08    2.03  1.00\n    Nonflavanoid Phenols:         0.13  0.66    0.36  0.12\n    Proanthocyanins:              0.41  3.58    1.59  0.57\n    Colour Intensity:              1.3  13.0     5.1   2.3\n    Hue:                          0.48  1.71    0.96  0.23\n    OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71\n    Proline:                       278  1680     746   315\n    ============================= ==== ===== ======= =====\n\n    :Missing Attribute Values: None\n    :Class Distribution: class_0 (59), class_1 (71), class_2 (48)\n    :Creator: R.A. Fisher\n    :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n    :Date: July, 1988\n\nThis is a copy of UCI ML Wine recognition datasets.\nhttps://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data\n\nThe data is the results of a chemical analysis of wines grown in the same\nregion in Italy by three different cultivators. There are thirteen different\nmeasurements taken for different constituents found in the three types of\nwine.\n\nOriginal Owners: \n\nForina, M. et al, PARVUS - \nAn Extendible Package for Data Exploration, Classification and Correlation. \nInstitute of Pharmaceutical and Food Analysis and Technologies,\nVia Brigata Salerno, 16147 Genoa, Italy.\n\nCitation:\n\nLichman, M. (2013). UCI Machine Learning Repository\n[https://archive.ics.uci.edu/ml]. Irvine, CA: University of California,\nSchool of Information and Computer Science. \n\n.. topic:: References\n\n  (1) S. Aeberhard, D. Coomans and O. de Vel, \n  Comparison of Classifiers in High Dimensional Settings, \n  Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of  \n  Mathematics and Statistics, James Cook University of North Queensland. \n  (Also submitted to Technometrics). \n\n  The data was used with many others for comparing various \n  classifiers. The classes are separable, though only RDA \n  has achieved 100% correct classification. \n  (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data)) \n  (All results using the leave-one-out technique) \n\n  (2) S. Aeberhard, D. Coomans and O. de Vel, \n  "THE CLASSIFICATION PERFORMANCE OF RDA" \n  Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of \n  Mathematics and Statistics, James Cook University of North Queensland. \n  (Also submitted to Journal of Chemometrics).\n', 'feature_names': ['alcohol', 'malic_acid', 'ash', 'alcalinity_of_ash', 'magnesium', 'total_phenols', 'flavanoids', 'nonflavanoid_phenols', 'proanthocyanins', 'color_intensity', 'hue', 'od280/od315_of_diluted_wines', 'proline']}

```python
#So we found that the X_y variable is of type sklearn.utils.bunch
#further, we see that the X_y variable consists of dictionary-type structures named "data",  "target" and "target_names"

# Let us now segregate the data attributes and class labels into separate variables

X = X_y.data
y = X_y.target

print('X shape:', X.shape, 'type(X):', type(X) )
print(' y shape:', y.shape, ' type(y):', type(y))

```

    X shape: (178, 13) type(X): <class 'numpy.ndarray'>
     y shape: (178,)  type(y): <class 'numpy.ndarray'>

```python
#Thus we see that X and y are numpy ndarrays.
#We also see that there are 178 samples (or instances) in the data set and each sample has 13 attributes.
#The shape of target variable y indicates that y contains the labels for these 178 samples

#We can print the unique labels available in y
print(np.unique(y))
```

    [0 1 2]

```python
# Let us also segregate the data set description into a different variable and check the contents
X_y_description = X_y.DESCR
print('type(X_y_description):', type(X_y_description))
print(X_y_description)
```

    type(X_y_description): <class 'str'>
    .. _wine_dataset:

    Wine recognition dataset
    ------------------------

    **Data Set Characteristics:**

        :Number of Instances: 178 (50 in each of three classes)
        :Number of Attributes: 13 numeric, predictive attributes and the class
        :Attribute Information:
     		- Alcohol
     		- Malic acid
     		- Ash
    		- Alcalinity of ash
     		- Magnesium
    		- Total phenols
     		- Flavanoids
     		- Nonflavanoid phenols
     		- Proanthocyanins
    		- Color intensity
     		- Hue
     		- OD280/OD315 of diluted wines
     		- Proline

        - class:
                - class_0
                - class_1
                - class_2

        :Summary Statistics:

        ============================= ==== ===== ======= =====
                                       Min   Max   Mean     SD
        ============================= ==== ===== ======= =====
        Alcohol:                      11.0  14.8    13.0   0.8
        Malic Acid:                   0.74  5.80    2.34  1.12
        Ash:                          1.36  3.23    2.36  0.27
        Alcalinity of Ash:            10.6  30.0    19.5   3.3
        Magnesium:                    70.0 162.0    99.7  14.3
        Total Phenols:                0.98  3.88    2.29  0.63
        Flavanoids:                   0.34  5.08    2.03  1.00
        Nonflavanoid Phenols:         0.13  0.66    0.36  0.12
        Proanthocyanins:              0.41  3.58    1.59  0.57
        Colour Intensity:              1.3  13.0     5.1   2.3
        Hue:                          0.48  1.71    0.96  0.23
        OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71
        Proline:                       278  1680     746   315
        ============================= ==== ===== ======= =====

        :Missing Attribute Values: None
        :Class Distribution: class_0 (59), class_1 (71), class_2 (48)
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    This is a copy of UCI ML Wine recognition datasets.
    https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data

    The data is the results of a chemical analysis of wines grown in the same
    region in Italy by three different cultivators. There are thirteen different
    measurements taken for different constituents found in the three types of
    wine.

    Original Owners:

    Forina, M. et al, PARVUS -
    An Extendible Package for Data Exploration, Classification and Correlation.
    Institute of Pharmaceutical and Food Analysis and Technologies,
    Via Brigata Salerno, 16147 Genoa, Italy.

    Citation:

    Lichman, M. (2013). UCI Machine Learning Repository
    [https://archive.ics.uci.edu/ml]. Irvine, CA: University of California,
    School of Information and Computer Science.

    .. topic:: References

      (1) S. Aeberhard, D. Coomans and O. de Vel,
      Comparison of Classifiers in High Dimensional Settings,
      Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of
      Mathematics and Statistics, James Cook University of North Queensland.
      (Also submitted to Technometrics).

      The data was used with many others for comparing various
      classifiers. The classes are separable, though only RDA
      has achieved 100% correct classification.
      (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data))
      (All results using the leave-one-out technique)

      (2) S. Aeberhard, D. Coomans and O. de Vel,
      "THE CLASSIFICATION PERFORMANCE OF RDA"
      Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of
      Mathematics and Statistics, James Cook University of North Queensland.
      (Also submitted to Journal of Chemometrics).

```python
#Let us focus our discussion on the feature matrix X and keep aside the label matrix y for the time being.

#Let us compute two new matrices from X.

XXT = np.matmul(X,np.transpose(X)) #X X^T matrix
XTX = np.matmul(X.T,X) #X^T X matrix

print('XXT shape:', XXT.shape, 'XTX shape:', XTX.shape)

```

    XXT shape: (178, 178) XTX shape: (13, 13)

$XX^\top$ and $X^\top X$ are called covariance matrices.

In particular, $XX^\top$ is of shape $\textit{num samples} \times \textit{num samples}$ and captures the correlation (upto scaling) between distinct samples in the data set. Hence we call $XX^\top$ to be sample covariance matrix.

Similarly, $X^\top X$ is of shape $\textit{num features} \times \textit{num features}$ and captures the correlation (upto scaling) between different attributes in the data set.  Hence we call $X^\top X$ to be feature covariance matrix.

In fact, these covariance matrices satisfy several important properties:

1.   $XX^\top$ and $X^\top X$ are square and symmetric
2.   $XX^\top$ and $X^\top X$ are positive semi-definite (that is, their eigen values are non-negative)
3.   $XX^\top$ and $X^\top X$ have same positive eigen values.
4.   $XX^\top$ and $X^\top X$ have the same rank as $X$

Let us quickly check each of these facts.

```python
#checking if XXT and XTX are square and symmetric

print('XXT matrix:')
if XXT.shape[0] == XXT.shape[1]:
  print('XXT is square')
else:
  print('XXT is not square')

if np.all(XXT == XXT.T):
  print('XXT is symmetric')
else:
  print('XXT is not symmetric')

print('XTX matrix:')
if XTX.shape[0] == XTX.shape[1]:
  print('XTX is square')
else:
  print('XTX is not square')

if np.all(XTX == XTX.T):
  print('XTX is symmetric')
else:
  print('XTX is not symmetric')

```

    XXT matrix:
    XXT is square
    XXT is symmetric
    XTX matrix:
    XTX is square
    XTX is symmetric

```python
#checking if XXT and XTX are positive semi-definite
#First we will use linalg.eig in numpy, which can be used to find eigen values and eigen vectors for any square matrix

print('eigen values of XXT:')
eig_val_XXT, eig_vec_XXT = np.linalg.eig(XXT)
print(eig_val_XXT)

print('eigen values of XTX:')
eig_val_XTX, eig_vec_XTX = np.linalg.eig(XTX)
print(eig_val_XTX)

```

    eigen values of XXT:
    [ 1.18519582e+08+0.00000000e+00j  2.43603495e+05+0.00000000e+00j
      3.26599028e+03+0.00000000e+00j  9.06017549e+02+0.00000000e+00j
      3.43836011e+02+0.00000000e+00j  2.09178961e+02+0.00000000e+00j
      1.21794126e+02+0.00000000e+00j  2.79829387e+01+0.00000000e+00j
      1.98611790e+01+0.00000000e+00j  1.27825659e+01+0.00000000e+00j
      6.76635454e+00+0.00000000e+00j  3.94740676e+00+0.00000000e+00j
      1.47358714e+00+0.00000000e+00j  8.16484538e-09+7.00475770e-09j
      8.16484538e-09-7.00475770e-09j -8.78488523e-09+4.61096884e-09j
     -8.78488523e-09-4.61096884e-09j -3.85299105e-09+7.93378079e-09j
     -3.85299105e-09-7.93378079e-09j -5.50564039e-09+0.00000000e+00j
     -3.31878802e-09+3.97313840e-09j -3.31878802e-09-3.97313840e-09j
      4.76637245e-09+1.44041310e-09j  4.76637245e-09-1.44041310e-09j
      1.78453152e-09+4.05771609e-09j  1.78453152e-09-4.05771609e-09j
      2.95389491e-10+3.96180747e-09j  2.95389491e-10-3.96180747e-09j
     -2.81934940e-09+1.66178482e-09j -2.81934940e-09-1.66178482e-09j
      3.49888290e-09+0.00000000e+00j  2.54716676e-09+2.19873272e-09j
      2.54716676e-09-2.19873272e-09j -2.76272848e-09+4.55868259e-10j
     -2.76272848e-09-4.55868259e-10j -6.75949885e-10+2.56932005e-09j
     -6.75949885e-10-2.56932005e-09j  2.07194237e-09+1.64050189e-09j
      2.07194237e-09-1.64050189e-09j -1.28216729e-09+1.99990757e-09j
     -1.28216729e-09-1.99990757e-09j -1.91148458e-09+1.07500381e-09j
     -1.91148458e-09-1.07500381e-09j  2.12975971e-09+4.97860910e-10j
      2.12975971e-09-4.97860910e-10j  2.15283831e-09+0.00000000e+00j
     -5.91183385e-10+1.95509477e-09j -5.91183385e-10-1.95509477e-09j
     -1.87799368e-09+0.00000000e+00j -1.73746365e-09+5.72397535e-10j
     -1.73746365e-09-5.72397535e-10j -1.40707922e-09+1.10202492e-09j
     -1.40707922e-09-1.10202492e-09j  8.71232397e-10+1.73068474e-09j
      8.71232397e-10-1.73068474e-09j  1.53635325e-09+9.03696715e-10j
      1.53635325e-09-9.03696715e-10j -3.40702148e-10+1.63628912e-09j
     -3.40702148e-10-1.63628912e-09j  1.03321649e-09+1.29325815e-09j
      1.03321649e-09-1.29325815e-09j  5.72467114e-10+1.56215441e-09j
      5.72467114e-10-1.56215441e-09j -1.65900373e-09+0.00000000e+00j
      8.31047752e-10+1.35236312e-09j  8.31047752e-10-1.35236312e-09j
      1.70651695e-10+1.59948923e-09j  1.70651695e-10-1.59948923e-09j
     -1.47306767e-09+4.34337071e-10j -1.47306767e-09-4.34337071e-10j
      1.55861298e-09+0.00000000e+00j  1.23296833e-09+7.13744817e-10j
      1.23296833e-09-7.13744817e-10j  1.30868388e-09+5.40022813e-10j
      1.30868388e-09-5.40022813e-10j -1.36943695e-09+0.00000000e+00j
     -4.93863582e-10+1.23731299e-09j -4.93863582e-10-1.23731299e-09j
      2.95896532e-10+1.33846166e-09j  2.95896532e-10-1.33846166e-09j
      1.35474172e-09+0.00000000e+00j -1.04516331e-09+7.17045324e-10j
     -1.04516331e-09-7.17045324e-10j  8.14890706e-10+9.99760455e-10j
      8.14890706e-10-9.99760455e-10j -7.87537598e-10+9.16952292e-10j
     -7.87537598e-10-9.16952292e-10j  1.11930991e-09+1.99600144e-10j
      1.11930991e-09-1.99600144e-10j -9.98152180e-10+4.23679978e-10j
     -9.98152180e-10-4.23679978e-10j -6.22236774e-10+8.84253201e-10j
     -6.22236774e-10-8.84253201e-10j  1.02725611e-09+0.00000000e+00j
      2.68207579e-10+1.06111836e-09j  2.68207579e-10-1.06111836e-09j
     -7.61296914e-10+6.55564876e-10j -7.61296914e-10-6.55564876e-10j
     -9.76217171e-10+2.66901178e-10j -9.76217171e-10-2.66901178e-10j
     -9.87794761e-10+0.00000000e+00j -1.38517802e-10+9.24559117e-10j
     -1.38517802e-10-9.24559117e-10j  4.58616739e-10+8.34391879e-10j
      4.58616739e-10-8.34391879e-10j  9.05308658e-10+2.99883932e-10j
      9.05308658e-10-2.99883932e-10j  6.92791185e-10+6.24609935e-10j
      6.92791185e-10-6.24609935e-10j -6.15545095e-11+8.94996563e-10j
     -6.15545095e-11-8.94996563e-10j  1.31219248e-10+8.76234521e-10j
      1.31219248e-10-8.76234521e-10j  8.82039153e-10+0.00000000e+00j
      6.27594960e-10+5.79021762e-10j  6.27594960e-10-5.79021762e-10j
     -2.20509350e-10+7.60713489e-10j -2.20509350e-10-7.60713489e-10j
     -4.85955117e-10+6.10691194e-10j -4.85955117e-10-6.10691194e-10j
      6.66996029e-10+3.57091604e-10j  6.66996029e-10-3.57091604e-10j
     -6.70994578e-10+2.51533856e-10j -6.70994578e-10-2.51533856e-10j
     -2.76180031e-10+6.71950279e-10j -2.76180031e-10-6.71950279e-10j
     -5.11335175e-10+4.45637948e-10j -5.11335175e-10-4.45637948e-10j
      3.24906586e-10+6.22671944e-10j  3.24906586e-10-6.22671944e-10j
      6.61005331e-10+9.84489412e-11j  6.61005331e-10-9.84489412e-11j
      4.31400160e-10+4.65951833e-10j  4.31400160e-10-4.65951833e-10j
     -6.35645162e-10+0.00000000e+00j -3.96438453e-10+3.78091908e-10j
     -3.96438453e-10-3.78091908e-10j -1.58633526e-10+5.45518283e-10j
     -1.58633526e-10-5.45518283e-10j  5.43050134e-10+0.00000000e+00j
     -5.30618707e-10+9.70356583e-11j -5.30618707e-10-9.70356583e-11j
      4.90531187e-12+5.44696080e-10j  4.90531187e-12-5.44696080e-10j
      9.08473835e-11+5.33328516e-10j  9.08473835e-11-5.33328516e-10j
      2.14844809e-10+4.80451933e-10j  2.14844809e-10-4.80451933e-10j
     -1.99344635e-10+4.40774782e-10j -1.99344635e-10-4.40774782e-10j
     -4.31064984e-10+9.50364359e-11j -4.31064984e-10-9.50364359e-11j
      4.42409839e-10+8.15724991e-11j  4.42409839e-10-8.15724991e-11j
      1.95740846e-10+3.42476791e-10j  1.95740846e-10-3.42476791e-10j
     -3.35200146e-10+1.75130711e-10j -3.35200146e-10-1.75130711e-10j
      3.35991481e-10+1.47187252e-10j  3.35991481e-10-1.47187252e-10j
      4.72994234e-11+2.81134477e-10j  4.72994234e-11-2.81134477e-10j
     -6.32527236e-11+2.75319297e-10j -6.32527236e-11-2.75319297e-10j
     -1.81113028e-10+2.18870253e-10j -1.81113028e-10-2.18870253e-10j
     -2.50515866e-10+4.22188977e-11j -2.50515866e-10-4.22188977e-11j
      2.30159672e-10+0.00000000e+00j  1.90971257e-10+1.41175747e-10j
      1.90971257e-10-1.41175747e-10j -4.90896629e-11+1.70423347e-10j
     -4.90896629e-11-1.70423347e-10j -1.25763434e-10+0.00000000e+00j
      1.62088143e-10+0.00000000e+00j -1.61020288e-11+0.00000000e+00j
      2.96760719e-11+0.00000000e+00j  6.17418445e-11+0.00000000e+00j]
    eigen values of XTX:
    [1.18519582e+08 2.43603495e+05 3.26599028e+03 9.06017549e+02
     3.43836011e+02 2.09178961e+02 1.21794126e+02 2.79829387e+01
     1.98611790e+01 1.27825659e+01 6.76635454e+00 3.94740676e+00
     1.47358714e+00]

## Important note about the results obtained from $\texttt{np.linalg.eig}$##

Note that the results from $\texttt{np.linalg.eig}$ show complex eigen values for $XX^\top$ matrix.

However from theory if $A$ is a real symmetric matrix, then the eigen values need to be real. We present a proof of the result below.

So, the results from $\texttt{np.linalg.eig}$ are due to numerical errors due to floating point approximations and truncations involved in the algorithms used to compute the eigen values and eigen vectors.

$\textbf{Result:}$ If $A$ is a real symmetric matrix, then the eigen values are real.

A simple proof is as below:

Let $\lambda$ be a complex number of the form $a+bi$ which is an eigen value for a real symmetric matrix $A$ corresponding to the eigen vector $x\neq \mathbf{0}$. Recall that the complex conjugate $\bar{\lambda}$ of $\lambda$ is given by $a-bi$.

Then we have:

$
\begin{align}
Ax &= \lambda x \nonumber \\
\text{Taking complex conjugates } & \text { on both sides, we have} \nonumber \\
\overline{Ax} &= \overline{\lambda x} \nonumber \\
\bar{A} \bar{x} &= \bar{\lambda} \bar{x} \nonumber \\
\end{align}
$

However $\bar{A} = A$ since $A$ is real. Thus we have $A\bar{x} = \bar{\lambda} \bar{x}$. Hence $\bar{x} \neq \mathbf{0}$ is an eigen vector of $A$ with a corresponding eigen value $\bar{\lambda}$.

Now consider $A \bar{x} = \lambda \bar{x}$. Pre-multiplying by $x^\top$ we have:

$
\begin{align}
x^\top A \bar{x} &= \lambda x^\top \bar{x} \nonumber \\
(Ax)^\top \bar{x} &= \lambda x^\top \bar{x} \nonumber \\
\implies (\lambda x)^\top \bar{x} &= \lambda x^\top \bar{x} \nonumber \\
\lambda x^\top \bar{x} &= \lambda x^\top \bar{x} \nonumber \\
\end{align}
$

Since $r=x^\top \bar{x} \neq 0$ is a real quantity $\textbf{(check this!)}$ we have:   $\lambda r=\bar{\lambda} r \implies \lambda =\bar{\lambda}$.

Now $\lambda = \bar{\lambda} \implies a+bi = a-bi$ is possible if and only if $b=0$ or $\lambda$ is a real number. Hence the proof follows.

```python
#checking if XXT and XTX are positive semi-definite
#we will use linalg.eigh in numpy, which can be useful for symmetric matrices

print('eigen values of XXT:')
eig_val_XXT, eig_vec_XXT = np.linalg.eigh(XXT)
print(eig_val_XXT)

print('eigen values of XTX:')
eig_val_XTX, eig_vec_XTX = np.linalg.eigh(XTX)
print(eig_val_XTX)

```

    eigen values of XXT:
    [-3.36866745e-08 -1.50220377e-08 -1.48359545e-08 -1.27703210e-08
     -1.12251019e-08 -8.61716632e-09 -7.68185582e-09 -6.08380757e-09
     -5.20923555e-09 -4.79181820e-09 -4.64584954e-09 -4.45823382e-09
     -4.06583460e-09 -3.76302785e-09 -3.52082671e-09 -3.48366439e-09
     -3.19645170e-09 -3.10620238e-09 -2.93389846e-09 -2.77958523e-09
     -2.56766319e-09 -2.50504266e-09 -2.30319114e-09 -2.20061929e-09
     -2.14217319e-09 -2.08954272e-09 -1.96743185e-09 -1.84108486e-09
     -1.80269071e-09 -1.71241283e-09 -1.51163812e-09 -1.48936809e-09
     -1.48435290e-09 -1.35417436e-09 -1.24807262e-09 -1.22321156e-09
     -1.20566808e-09 -1.13322046e-09 -1.11424634e-09 -1.09010928e-09
     -1.08372074e-09 -1.06103073e-09 -1.03223767e-09 -9.47104518e-10
     -9.09229967e-10 -8.51452866e-10 -8.02550788e-10 -7.64072662e-10
     -7.60879699e-10 -7.10046986e-10 -7.09985259e-10 -6.32793396e-10
     -6.23683235e-10 -6.11804021e-10 -5.96182200e-10 -5.64414286e-10
     -5.50822138e-10 -5.22651034e-10 -5.18204553e-10 -4.73363967e-10
     -4.69052307e-10 -4.18885367e-10 -3.94884877e-10 -3.87769120e-10
     -3.76018153e-10 -3.65119848e-10 -3.20706412e-10 -3.01678535e-10
     -2.99146215e-10 -2.72855423e-10 -2.54613653e-10 -2.53768335e-10
     -2.01730708e-10 -1.76417037e-10 -1.70902605e-10 -1.56512439e-10
     -1.48077539e-10 -1.38869646e-10 -1.19445082e-10 -1.05948808e-10
     -9.11735111e-11 -7.68250932e-11 -4.99434887e-11 -2.33208064e-11
      1.17301840e-12  8.98322914e-12  1.45778566e-11  1.79357480e-11
      2.05773313e-11  2.83472760e-11  6.61276254e-11  1.02905791e-10
      1.07933527e-10  1.37248301e-10  1.47154263e-10  1.50235595e-10
      1.91329168e-10  2.25078341e-10  2.39138661e-10  2.52945440e-10
      2.60298563e-10  2.69253428e-10  2.93221147e-10  3.34018263e-10
      3.74848304e-10  3.90900848e-10  4.46647264e-10  4.56286654e-10
      4.66316438e-10  4.98861197e-10  5.17910756e-10  5.61613402e-10
      5.68228704e-10  6.01208053e-10  6.05700178e-10  6.61568349e-10
      6.77849396e-10  7.10021616e-10  7.65168035e-10  7.76394660e-10
      8.08790898e-10  8.09837184e-10  8.68174581e-10  8.77950339e-10
      9.57750703e-10  1.00136493e-09  1.01581349e-09  1.01781165e-09
      1.14979337e-09  1.19153004e-09  1.19635617e-09  1.26673590e-09
      1.30140230e-09  1.43212625e-09  1.43720073e-09  1.59498582e-09
      1.69910665e-09  1.73746943e-09  1.89262905e-09  1.90296137e-09
      1.99824901e-09  2.08413188e-09  2.13495002e-09  2.18009110e-09
      2.27190931e-09  2.59944712e-09  2.83685589e-09  2.84011309e-09
      3.01278178e-09  3.12271348e-09  3.30368788e-09  3.80372557e-09
      4.05022479e-09  4.18260702e-09  4.53468971e-09  5.09438665e-09
      6.02963870e-09  6.98779974e-09  9.13083480e-09  1.02320175e-08
      1.15651105e-08  1.35849878e-08  1.57209102e-08  2.28770267e-08
      3.49258392e-08  1.47358714e+00  3.94740676e+00  6.76635454e+00
      1.27825659e+01  1.98611790e+01  2.79829387e+01  1.21794126e+02
      2.09178961e+02  3.43836011e+02  9.06017549e+02  3.26599028e+03
      2.43603495e+05  1.18519582e+08]
    eigen values of XTX:
    [1.47358714e+00 3.94740676e+00 6.76635454e+00 1.27825659e+01
     1.98611790e+01 2.79829387e+01 1.21794126e+02 2.09178961e+02
     3.43836011e+02 9.06017549e+02 3.26599028e+03 2.43603495e+05
     1.18519582e+08]

## Important note about the results obtained from $\texttt{np.linalg.eigh}$##

Note that the results from $\texttt{np.linalg.eigh}$ show some negative eigen values for $XX^\top$ matrix.

However from theory, we know that if $A$ is a $n \times n$ symmetric and positive semi-definite matrix, then  the eigen values are non-negative.

A proof of this result is given below.

Hence the results from $\texttt{np.linalg.eigh}$ should be interpreted carefully. The negative values are again due to numerical errors in algorithms.

$\mathbf{Result:}$ If $A$ is a $n \times n$ symmetric and positive semi-definite matrix, then  the eigen values are non-negative.

$\textbf{Recall:}$ Since $A$ is symmetric and positive semi-definite, we know that $x^\top Ax\geq 0$ for any $x \in {\mathbb{R}}^n$.

Now consider $v \neq \mathbf{0}$ as an eigen vector of $A$ with eigen value $\lambda$, we have: $Av = \lambda v$.

Then we have by pre-multiplying by $v^\top$:

$
\begin{align}
v^\top Av &= v^\top (\lambda v) \nonumber \\
\implies \lambda v^\top  v &\geq 0 \nonumber
\end{align}
$

Since we know that for a vector $v=\begin{bmatrix}
v_1 \\
v_2 \\
\vdots \\
v_n
\end{bmatrix} \in {\mathbb{R}}^n$, we have $v^\top v = \|v\|_2^2 = \sum_{i=1}^{n} v_i^2 \geq 0$ .

Since $v \neq \mathbf{0}$, we have $\|v\|_2^2 > 0$.

Thus  from $\lambda v^\top  v \geq 0$ and $v^\top v > 0$, we have $\lambda  \geq 0$.

```python
#checking if XXT and XTX are have same positive eigen values
#we will use linalg.eigh in numpy, which can be useful for symmetric matrices

print('positive eigen values of XXT:')
eig_val_XXT,_ = np.linalg.eigh(XXT)
print( eig_val_XXT[np.where(eig_val_XXT > 0)])

print('positive eigen values of XXT:')
eig_val_XTX,_ = np.linalg.eigh(XTX)
print( eig_val_XTX[np.where(eig_val_XTX > 0)])

```

    positive eigen values of XXT:
    [1.17301840e-12 8.98322914e-12 1.45778566e-11 1.79357480e-11
     2.05773313e-11 2.83472760e-11 6.61276254e-11 1.02905791e-10
     1.07933527e-10 1.37248301e-10 1.47154263e-10 1.50235595e-10
     1.91329168e-10 2.25078341e-10 2.39138661e-10 2.52945440e-10
     2.60298563e-10 2.69253428e-10 2.93221147e-10 3.34018263e-10
     3.74848304e-10 3.90900848e-10 4.46647264e-10 4.56286654e-10
     4.66316438e-10 4.98861197e-10 5.17910756e-10 5.61613402e-10
     5.68228704e-10 6.01208053e-10 6.05700178e-10 6.61568349e-10
     6.77849396e-10 7.10021616e-10 7.65168035e-10 7.76394660e-10
     8.08790898e-10 8.09837184e-10 8.68174581e-10 8.77950339e-10
     9.57750703e-10 1.00136493e-09 1.01581349e-09 1.01781165e-09
     1.14979337e-09 1.19153004e-09 1.19635617e-09 1.26673590e-09
     1.30140230e-09 1.43212625e-09 1.43720073e-09 1.59498582e-09
     1.69910665e-09 1.73746943e-09 1.89262905e-09 1.90296137e-09
     1.99824901e-09 2.08413188e-09 2.13495002e-09 2.18009110e-09
     2.27190931e-09 2.59944712e-09 2.83685589e-09 2.84011309e-09
     3.01278178e-09 3.12271348e-09 3.30368788e-09 3.80372557e-09
     4.05022479e-09 4.18260702e-09 4.53468971e-09 5.09438665e-09
     6.02963870e-09 6.98779974e-09 9.13083480e-09 1.02320175e-08
     1.15651105e-08 1.35849878e-08 1.57209102e-08 2.28770267e-08
     3.49258392e-08 1.47358714e+00 3.94740676e+00 6.76635454e+00
     1.27825659e+01 1.98611790e+01 2.79829387e+01 1.21794126e+02
     2.09178961e+02 3.43836011e+02 9.06017549e+02 3.26599028e+03
     2.43603495e+05 1.18519582e+08]
    positive eigen values of XXT:
    [1.47358714e+00 3.94740676e+00 6.76635454e+00 1.27825659e+01
     1.98611790e+01 2.79829387e+01 1.21794126e+02 2.09178961e+02
     3.43836011e+02 9.06017549e+02 3.26599028e+03 2.43603495e+05
     1.18519582e+08]

##Note about the precision of eigen values:##

Please note that there are a few positive eigen values which are close to zero. However in this case, we can guarantee that these values should be zero.

These issues highlight some numerical issues when computing eigen values and eigen vectors using $\texttt{numpy.linalg.eig}$ and $\texttt{numpy.linalg.eigh}$.

```python
#check rank of matrices XXT and XTX

print('rank(XXT):', np.linalg.matrix_rank(XXT), 'rank(XTX):', np.linalg.matrix_rank(XTX))
```

    rank(XXT): 13 rank(XTX): 13

## Rank of matrix $A$ :##

Note that for a matrix $A$ of size $m \times n$, $rank(A)$ is defined as the number of linearly independent rows or columns in $A$.

Hence $rank(A) \leq \min\{m,n\}$.

If $rank(A) = \min\{m,n\}$ then the matrix is called full rank.

Note that for $\textit{wine}$ data set, we have $X^\top X$ to be of size $13 \times 13$. Also note that $rank(X^\top X)=13$, hence $X^\top X$ is full-rank.

## Checking alignment of sample features using dot products ##

```python
#Take some samples of the same class

X_class0 = X[np.where(y==0),:].squeeze()
print(X_class0.shape)

X_class1 = X[np.where(y==1),:].squeeze()
print(X_class1.shape)

X_class2 = X[np.where(y==2),:].squeeze()
print(X_class2.shape)

```

    (59, 13)
    (71, 13)
    (48, 13)

```python
#Let us find the dot product between some samples of class 0
print(np.dot(X_class0[0]/np.linalg.norm(X_class0[0]),X_class0[1]/np.linalg.norm(X_class0[1])) )
```

    0.9997092287724737

```python
#Let us find the dot product between some samples of class 1
print(np.dot(X_class1[30]/np.linalg.norm(X_class1[30]),X_class1[42]/np.linalg.norm(X_class1[42])) )
```

    0.9990245635883316

```python
#Let us find the dot product between some samples of class 2
print(np.dot(X_class2[20]/np.linalg.norm(X_class2[20]),X_class2[32]/np.linalg.norm(X_class2[32])) )
```

    0.9982866107350962

```python
#Let us find the dot product between some samples of class 0 and class 2
print(np.dot(X_class0[0]/np.linalg.norm(X_class0[0]),X_class1[10]/np.linalg.norm(X_class1[10])) )
```

    0.9960370718188822

## Exercise ##

Try the above code functionalities on $\textit{iris}$ data set from $\texttt{scikit-learn}$ package.

```python

```