# Let's start!
OPEN the jupyter notebook **Tutorial1-Part1** downloaded from the **indico timetable: https://indico.cern.ch/event/1088622/timetable/#20220111** to work locally or from the following link: **https://github.com/fusterma/JUAS2022** to work online.

# Tutorials summary

The goal of these workshops is to do numerical exercices using MAD-X to visualize transverse dynamics concepts from a different point of view.

**Friday 14th of January**
- **Tutorial 1 - Part 1**: Introduction to the tools, small numerical exercises (all together).
- **Tutorial 1 - Part 2**: My first circular accelerator: FODO cell – optics and first matching (groups of 3/4 students).
- **Tutorial 1 - Part 3**: Adding dipoles to the FODO cell – MAD-X matching block (groups of 3/4 students).

$\color{red}{\text{WEEKEND: homework exercice}}$

**Monday 17th of January**
- **Tutorial 2 - Part 1**: Natural chromaticity – MAD-X tracking module (groups of 3/4 students).
- **Tutorial 2 - Part 2**: Chromaticity correction – impact of non-linearities (groups of 3/4 students).
- **Tutorial 2 - Part 3**: Design of a transfer line  - optics and matching (groups of 3/4 students).

$\color{red}{\text{Homework + Tutorial 1 and Tutorial 2 jupyter-notebooks}}$ (to be delivered as late on Wednesday 18th to nuria.fuster@ific.uv.es). This will be considered as a BONUS to pass the accelerator design workshop oral exam.

$\color{red}{\text{VERY IMPORTANT}}$: Save your jupyter-notebooks and download them to your computer after finishing the tutorials!! Otherwise your work will be lost!

- The tutorials solutions will be uploaded on the indico timetable on Wednesday 19th.

$\color{blue}{\text{Notes}}$:

- For most of the tutorials we will split in groups of 3/4 students. These groups will be kept also for the accelerator design workshops on the third and fourth weeks of the JUAS course.

- The timing of the tutorials (except Tutorial 1 - Part 1) will be:
    - 5 minutes for the introduction to the problem.
    - 25 minutes to work within your team and with your tutor on the problem.
    - 15 minutes for going through the solutions and discussion.

- Tutors (Axel, Guido, Tessa, Davide and Nuria) will go around to help you and answer questions!

- You can work as a team in the groups or by yourself but when you need help please ask your team mates or the tutor and use the screen share option.

- The problems are long with many questions, we don't expect you to solve all of them. Some BONUS questions are there for discussion or homework.

- We encourage you to use the Slack MAD-X channel during the workshops for the discussion part, during the weekend and all JUAS to post questions.

# Tutorial 1: Part 1

Objectives:

- [Get familiar with the jupyter-notebooks.](#introjupyter)

- [Get familiar with the basic python commands that we will use during the tutorials.](#intropython)

- [How do we compute the optics of a lattice?](#firstexercice)

- [Get familiar with python commands to send the information to the MAD-X code and review the main MADX blocks for optics calculations.](#intromadx)

# Jupyter notebook

- **OPEN** a jupyter notebook go to **FILE -> OPEN**.

- **EDIT/INSERT/DELATE** a cell.

- **RUN** (press bottom on the top command line or press CAPS+ENTER).

- If working online **SAVE and DOWNLOAD**: after we finish one tutorial you need to SAVE and DOWNLOAD the jupyter-notebook into your PC. Otherwise your progres will be lost!

- **SAVE TO BROWSER STORAGE** using the "cloud" icon (for those working on BINDER).

# Basic python commands

The python universe has a huge number of libraries that extend the capabilities of python.
Nearly all of these are open source. For this workshop we will use the following:

```python
############################
# Import special libraries #
############################
#For plotting
from matplotlib import pyplot as plt
# For numerical calulations (np.max(), np.min(), np.mean()...)
import numpy as np
# For symbolic computation (solving algebra problems)
import sympy as sp
# For structuring the data, visualization of tables and data manipultion
import pandas as pd
# Library that allows us to use the MAD-X models
from cpymad.madx import Madx
# Plot display
%matplotlib notebook
```

If you want to learn more about **python**:

https://www.youtube.com/watch?v=kqtD5dpn9C8

https://www.kaggle.com/learn/python

More about the **cpymad** library: http://hibtc.github.io/cpymad/getting-started

# Scalars, arrays and matrices in Python

```python
# Scalar
a=20
b=30
print(a*b)
c=a+b
print(c)
```

    600
    50

```python
# Arrays and matrices
sp.Matrix([1,2,3,4]) # 1D array
```

$\displaystyle \left[\begin{matrix}1\\2\\3\\4\end{matrix}\right]$

```python
sp.Matrix([[1,2],[3,4]]) # 2x2 matrix
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
A=sp.Matrix([[1,2],[3,4]])
B=sp.Matrix([[1,2],[3,4]])
A+B
```

$\displaystyle \left[\begin{matrix}2 & 4\\6 & 8\end{matrix}\right]$

```python
A*B
```

$\displaystyle \left[\begin{matrix}7 & 10\\15 & 22\end{matrix}\right]$

# Plots in Python

```python
# Plot
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=[0,1,2,3,4,5,6,7,8,9,10]
y=[0,1,2,3,4,5,6,7,8,9,10]
plt.plot(x,y,'.-b',label='test')
plt.legend()
plt.grid()
plt.xlabel('s [m]')
plt.ylabel('x[m]')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'x[m]')

# How do we compute the optics of a lattice?

- I want to motivate the use of optics codes such as MAD-X and at the same time illustrate the basic numerical approach behind some of the methods.

- The TWISS method is based on matrix multiplications where first and second order transport matrices are used to get the optics of the machine.

# FODO cell
- Compute the linear optics functions of a FODO cell, which is the simplier combination of quadrupoles required to focuse the beam in both, vertical and horizontal planes.

# Thin lens approximation (f >> $l_q$)

- To do some first estimations analytically one uses the thin lens approximation.

```python
# Symbolic computation

# Symbols defintion
K = sp.Symbol("K", positive = True)
Lq = sp.Symbol("Lq", positive = True)
Ld = sp.Symbol("Ld", positive = True)
```

```python
# Could you try to program the matrices and compute the FODO transfer matrix?
# HINT: A=sp.Matrix[[1,0],[1,0]]
```

```python
#Matrices definition

Mfoc=sp.Matrix([[1,0],[-K*Lq,1]])

Mdefoc=sp.Matrix([[1,0],[K*Lq,1]])

Mdrift=sp.Matrix([[1,Ld],[0,1]])
```

```python
#################################################
# Transport matrix of a focusing quadrupople    #
#################################################
Mfoc
```

$\displaystyle \left[\begin{matrix}1 & 0\\- K Lq & 1\end{matrix}\right]$

```python
#################################################
# Transport matrix of  a defocusing quadrupople #
#################################################
Mdefoc
```

$\displaystyle \left[\begin{matrix}1 & 0\\K Lq & 1\end{matrix}\right]$

```python
##############################################
# Transport matrix of a drift                #
##############################################
Mdrift
```

$\displaystyle \left[\begin{matrix}1 & Ld\\0 & 1\end{matrix}\right]$

```python
#Matrix multiplication
M=Mdrift*Mdefoc*Mdrift*Mfoc
#Matrix simplification
M=sp.simplify(M)
#Print of the matrix
M
```

$\displaystyle \left[\begin{matrix}- K^{2} Ld^{2} Lq^{2} - K Ld Lq + 1 & Ld \left(K Ld Lq + 2\right)\\- K^{2} Ld Lq^{2} & K Ld Lq + 1\end{matrix}\right]$

```python
# Matrix elements computation
# f=200 m, lq=1 m, ld= 30 m
M_thin = M.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_thin
```

$\displaystyle \left[\begin{matrix}0.8275 & 64.5\\-0.00075 & 1.15\end{matrix}\right]$

```python
#And for 3 FODO cells?
M3=M*M*M
M3=sp.simplify(M3)
M3
```

$\displaystyle \left[\begin{matrix}- K^{6} Ld^{6} Lq^{6} - K^{5} Ld^{5} Lq^{5} + 5 K^{4} Ld^{4} Lq^{4} + 4 K^{3} Ld^{3} Lq^{3} - 6 K^{2} Ld^{2} Lq^{2} - 3 K Ld Lq + 1 & Ld \left(K^{5} Ld^{5} Lq^{5} + 2 K^{4} Ld^{4} Lq^{4} - 4 K^{3} Ld^{3} Lq^{3} - 8 K^{2} Ld^{2} Lq^{2} + 3 K Ld Lq + 6\right)\\K^{2} Ld Lq^{2} \left(- K^{4} Ld^{4} Lq^{4} + 4 K^{2} Ld^{2} Lq^{2} - 3\right) & K^{5} Ld^{5} Lq^{5} + K^{4} Ld^{4} Lq^{4} - 4 K^{3} Ld^{3} Lq^{3} - 3 K^{2} Ld^{2} Lq^{2} + 3 K Ld Lq + 1\end{matrix}\right]$

```python
M_test = M3.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_test
```

$\displaystyle \left[\begin{matrix}0.430943921875 & 187.727653125\\-0.0021828796875 & 1.3695821875\end{matrix}\right]$

# What can we do with the  transfer matrix?
This matrix describes the optical properties of the lattice and defines the beam parameters.

   - We can propagate the phase space coordinates of a particle with a given set of initial coordinates.

```python
# Try with paralel particle going through the center of the first quadrupole or with a certain amplitude
x=sp.Matrix([[1],[0]])
x
```

$\displaystyle \left[\begin{matrix}1\\0\end{matrix}\right]$

```python
x2=M_thin*x
x2
```

$\displaystyle \left[\begin{matrix}0.8275\\-0.00075\end{matrix}\right]$

- We can compute the periodic solution TWISS functions.

```python
# Transfer matrix
R11, R12, R21, R22 = sp.symbols('R11,R12,R21,R22')

Mt=sp.Matrix([[R11,R12],[R21,R22]])

Mt
```

$\displaystyle \left[\begin{matrix}R_{11} & R_{12}\\R_{21} & R_{22}\end{matrix}\right]$

```python
# In case of periodic conditions in the accelerator there is another way to describe the particles trajectories.
# Periodic solution one-turn-transfer matrix in terms of twiss functions:

a, b, g, m = sp.symbols(r'\alpha,\beta, \gamma,\mu')

M=sp.Matrix([[sp.cos(m)+a*sp.sin(m),b*sp.sin(m)],[-g*sp.sin(m),sp.cos(m)-a*sp.sin(m)]])

M
```

$\displaystyle \left[\begin{matrix}\alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)} & \beta \sin{\left(\mu \right)}\\- \gamma \sin{\left(\mu \right)} & - \alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)}\end{matrix}\right]$

```python
#For the phase advance we use the trace of the matrix
sp.Eq(sp.cos(m),(R11+R22)/2)
```

$\displaystyle \cos{\left(\mu \right)} = \frac{R_{11}}{2} + \frac{R_{22}}{2}$

```python
#For the beta function we use the R12 matrix element
sp.Eq(b,R12/sp.sin(m))
```

$\displaystyle \beta = \frac{R_{12}}{\sin{\left(\mu \right)}}$

```python
#For the alfa function we use the trace of the matrix also
sp.Eq(a,(R11-R22)/(2*sp.sin(m)))
```

$\displaystyle \alpha = \frac{R_{11} - R_{22}}{2 \sin{\left(\mu \right)}}$

```python
#For the gamma we use the R21 element
sp.Eq(g,-(R21/sp.sin(m)))
```

$\displaystyle \gamma = - \frac{R_{21}}{\sin{\left(\mu \right)}}$

Once you have computed the periodic TWISS functions you can propagate them to any point in the machine using the transfer matrix of the TWISS functions from Transverse dynamics course.

```python
sp.Matrix([[R11**2, -2*R12*R11, R12**2],[-R11*R21,R12*R21+R22*R11, -R12*R22],[R21**2,-2*R22*R21, R22**2]])
```

$\displaystyle \left[\begin{matrix}R_{11}^{2} & - 2 R_{11} R_{12} & R_{12}^{2}\\- R_{11} R_{21} & R_{11} R_{22} + R_{12} R_{21} & - R_{12} R_{22}\\R_{21}^{2} & - 2 R_{21} R_{22} & R_{22}^{2}\end{matrix}\right]$

**Using the periodic one-turn-matrix and the stability condition of a FODO cell, one can define some interesting relations between the TWISS parameters and the magnetic properties of the lattice.**

# Figure 1: Relation between $\Delta \mu$, K, $L_{cell}$, $L_q$

```python
# Relation between the phase advance of the cell and K, Lcell, Lq (from periodic solution an stbility condition)
a, b, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(d*m/pi,2/pi*sp.asin(K*Lq*Lc/4))
```

$\displaystyle \frac{\Delta \mu}{\pi} = \frac{2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)}}{\pi}$

```python
# Parametric plots
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=np.arange(0,4.01,0.01)
y=2*np.arcsin(x/4)/np.pi
fig, ax1 = plt.subplots()
ax1.plot(x,y,'-')
ax1.set_ylabel("$\Delta \mu / \pi [rad]$", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
ax1.grid()
ax1.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <IPython.core.display.Javascript object>

# Exercice:
- What is the quadrupole strenght to match a FODO cell phase advance of 45$^\circ$ if the $L_{quad}$=5 m and $L_{cell}$=100 m?

- And for a FODO cell phase advance of 90$^\circ$?

- What is the maximum phase advance in a FODO cell?

```python
#for 45 degrees K*Lcell*lq=1.528
#K[m^(-2)]=
1.528/100/5
```

    0.003056

```python
#for 90 degrees K*Lcell*lq=2.842
#K[m^(-2)]=
2.842/100/5

#For a larger phase advance one needs to increase the strength of the quads
```

    0.005684

```python
#The function goes assimptotically to 1 corresponding to a phase advance of 180 degrees.
```

# Figure 2: Relation between $\beta_{max}$ and $\beta_{min}$ with K, $L_{cell}$, $L_q$

```python
# Relation between the beta of the cell and K, Lcell, Lq
a, bmin, bmax, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta_{min}, \beta_{max}, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(bmin/Lc,(1-(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

$\displaystyle \frac{\beta_{min}}{L_{cell}} = \frac{- \frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}}$

```python
sp.Eq(bmax/Lc,(1+(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

$\displaystyle \frac{\beta_{max}}{L_{cell}} = \frac{\frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}}$

```python
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

x=np.arange(0.5,3.90,0.01)
betamax=(1+(x/4))/(np.sin(2*np.arcsin(x/4)))
betamin=(1-(x/4))/(np.sin(2*np.arcsin(x/4)))
fig, ax1 = plt.subplots()
ax1.plot(x,betamax,'-',label=r"$\beta_{max}/L_{cell}$")
ax1.plot(x,betamin,'-',label=r"$\beta_{min}/L_{cell}$")
ax1.set_ylabel("[-]", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
plt.grid()
plt.legend()
plt.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <IPython.core.display.Javascript object>

# Exercice:
- What is K and $L_{cell}$ to match a FODO cell with a phase advance of 90$^\circ$ and a $\beta_{max}$ of 200 m? The  $L_{quad}$=2 m.

HINT: You may need to combine the data from both plots.

```python
# From Figure 1 we get that for 90 degrees K*Lcell*Lq=2.842, for this value in Figure 2 we get that bmax/Lcell=1.960
# Lcell=
200/1.69
```

    118.34319526627219

```python
#And replacing in K*Lcell*Lq=1.528 from Figure 1.
# K=
2.842/118/2
```

    0.012042372881355932

The exact solution of the particle motion has to be calculted in full detail but using some approximations we can make the first steps easier and estimate the order of magnitud of some magnetic properties of our lattice.

# Thick lens computation

```python
K = sp.Symbol("K")
Lq = sp.Symbol("Lq")
Ld= sp.Symbol("Ld")

Mfoc=sp.Matrix([[sp.cos(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq)],[-(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq),sp.cos(sp.sqrt(K)*Lq)]])

Mdefoc=sp.Matrix([[sp.cosh(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq)],[(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq),sp.cosh(sp.sqrt(K)*Lq)]])

Mdrift=sp.Matrix([[1,Ld],[0,1]])
```

```python
#################################################
# Transport matrix of a focusing quadrupople    #
#################################################
Mfoc
```

$\displaystyle \left[\begin{matrix}\cos{\left(\sqrt{K} Lq \right)} & \frac{\sin{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \sin{\left(\sqrt{K} Lq \right)} & \cos{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
################################################
# Transport matrix of a defocusing quadrupople #
################################################
Mdefoc
```

$\displaystyle \left[\begin{matrix}\cosh{\left(\sqrt{K} Lq \right)} & \frac{\sinh{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\\sqrt{K} \sinh{\left(\sqrt{K} Lq \right)} & \cosh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
##############################################
# Transport matrix of a dipole               #
##############################################
Mdrift
```

$\displaystyle \left[\begin{matrix}1 & Ld\\0 & 1\end{matrix}\right]$

```python
M=Mdrift*Mdefoc*Mdrift*Mfoc
M=sp.simplify(M)
M
```

$\displaystyle \left[\begin{matrix}\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)} - \left(2 \sqrt{K} Ld \cosh{\left(\sqrt{K} Lq \right)} + K Ld^{2} \sinh{\left(\sqrt{K} Lq \right)} + \sinh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} & \frac{\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} + \left(2 \sqrt{K} Ld \cosh{\left(\sqrt{K} Lq \right)} + K Ld^{2} \sinh{\left(\sqrt{K} Lq \right)} + \sinh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \left(\left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \sin{\left(\sqrt{K} Lq \right)} - \cos{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}\right) & \left(\sqrt{K} Ld \sinh{\left(\sqrt{K} Lq \right)} + \cosh{\left(\sqrt{K} Lq \right)}\right) \cos{\left(\sqrt{K} Lq \right)} + \sin{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
# Matrix elements computation
# f=200 m, lq=1 m, ld= 30 m
M_thick = M.subs(K, 1/(200*1)).subs(Lq, 1).subs(Ld,30) # units K in m-2, Lq and Ld in m
M_thick
```

$\displaystyle \left[\begin{matrix}0.821745966061752 & 66.6422445425746\\-0.000766666456349212 & 1.15474570697446\end{matrix}\right]$

```python
M_thin
```

$\displaystyle \left[\begin{matrix}0.8275 & 64.5\\-0.00075 & 1.15\end{matrix}\right]$

```python
#And for 3 FODO cells?
M3=M*M*M
```

In real world applications, lattices (including FODO) are not designed by hand but
dedicated software is used to do the design and simulation as for example MAD-X.

The TWISS in MADX it is based in matrix multiplications similar to what has been shown here but accounting also for second order matrices.

# Wha is next?

- Now we are going to do optics calculations using the MAD-X TWISS command (handle thousands of elements).

- We will use the MATCHING MAD-X tool to compute the required magnetic properties for a desired TWISS functions.

- We will use MAD-X to visulize the impact of some properties of the lattice on the TWISS and single particle DYNAMICS.

# An introduction to MAD-X using the python interface

In this first part we are going to get familiar with MAD-X syntax.

For more information please refer to the [MAD-X online manual](http://cern.ch/madx/releases/last-rel/madxuguide.pdf).

**Basic steps:**

   - Load the cpymad library.
   - Instantiate the MAD-X class (we create an object of the class).
   - Access the methods in the class.
       - From the methods available we will be mainly using the method "input" to send to MAD-X the commands.

```python
#Load the cpymad library
from cpymad.madx import Madx
```

```python
#Launching MAD-X
myMad = Madx(stdout=True)
```

```python
#String that will be interpreted by MAD-X
myString='''
stop;
'''
```

```python
#Using the "input" method to send the commandas to the MAD-X class
myMad.input(myString);
```

With the 'stop;' instruction we exit from MAD-X, so, as done in the following cell we need to re-instantiate our MAD-X object with the
**myMad = Madx()** instruction.

---
It is a good practice to make header, please use '!' to comment the single line.

```python
myMad = Madx(stdout=True)
```

```python
# Define and print a value
myString='''

!***************************************
! It is a good practice to make a header
!***************************************

a= 20;
value a;

'''
myMad.input(myString);
```

---
Use the **help** keyword (very rudimental help)

```python
# To get information about the MAD-X methods (twiss, beam,match... ) use the commnd "help"
myString='''
help, twiss;
'''
myMad.input(myString);
```

```python
myString='''
help, drift;
'''
myMad.input(myString);
```

# Let's define the main ingredients to get some results from MAD-X!

-Definition of machine parameters

-Magnets definition

-Sequence definition

-Beam definition

-Activate the sequence

-Actions

```python
myString='''

! *********************************************************************
! Definition of some parameters
! *********************************************************************

l_cell=60;
quadrupoleLenght=1;
myK:=0.005;// m^-2

! *********************************************************************
! Definition of magnets
! *********************************************************************
QF: quadrupole, L=quadrupoleLenght, K1:=myK;
QD: quadrupole, L=quadrupoleLenght, K1:=-myK;

! *********************************************************************
! Definition of sequence
! *********************************************************************
myCell:sequence, refer=entry, L=L_CELL;
quadrupole1: QF, at=0;
marker1: marker, at=15;
quadrupole2: QD, at=30;
endsequence;

! *********************************************************************
! Definition of beam
! *********************************************************************
beam, particle=proton, energy=1;

! *********************************************************************
! Use of the sequence
! *********************************************************************
use, sequence=myCell;

! *********************************************************************
! TWISS
! *********************************************************************

select, flag=TWISS, column=keyword, name, s, betx, bety,alfx, alfy, x, y, dx, dy;

twiss, file=Test_Nuria.madx;

plot, haxis=s, vaxis=betx,bety,dx,colour=100,file=Test_Nuria;

'''
myMad.input(myString);
```

The OUTPUT generated by MADX can be found by accessing the jupyter-notebook files-view.
- For accessing: CLIC on the jupyter-logo on the top of the page using the right buttom of your mouse and use the option "Open Link in a New Tab".

**Output:**

- SUMM table

- TWISS table

- TWISS .txt file

- TWISS .ps plot

# Accessing the data

- Open the files generated by MAD-X.
- Use python and output the required data on the jupyter-notebook.
    - Using MAD-X commands and the **input()** method.
    - Using **cpymad** methods:
        - **myMad.table.twiss.dframe()**
        - **myMad.table.summ.dframe()**

```python
#######################
#Using MAD-X commands #
#######################

myString='''
value, table(SUMM,Q1);
value, table(SUMM,betxmax);
'''
myMad.input(myString);
```

And for the vertical plane?

```python
#######################
#Using MAD-X commands #
#######################

myString='''
value, table(SUMM,Q2);
value, table(SUMM,betymax);
'''
myMad.input(myString);
```

```python
myString='''
value, table(TWISS,MYCELL$END,betx);
value, table(TWISS,MYCELL$END,bety);
'''
myMad.input(myString);
```

# Using python pandas library

Pandas dataframe are very convenient, have a look in https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf.

```python
# Using another method from cpymad "table.twiss.dframe()"
myDF=myMad.table.twiss.dframe()
```

```python
myDF
```

      name
      keyword
      s
      betx
      alfx
      mux
      bety
      alfy
      muy
      x
      ...
      sig54
      sig55
      sig56
      sig61
      sig62
      sig63
      sig64
      sig65
      sig66
      n1

      #s
      mycell$start:1
      marker
      0.0
      434.996645
      -1.086796
      0.000000
      376.179327
      0.941387
      0.000000
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole1
      quadrupole1:1
      quadrupole
      1.0
      434.996645
      1.086796
      0.000366
      376.179327
      -0.941387
      0.000423
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_0[0]
      drift_0:0
      drift
      15.0
      405.549112
      1.016599
      0.005672
      403.520929
      -1.011585
      0.006144
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      marker1
      marker1:1
      marker
      15.0
      405.549112
      1.016599
      0.005672
      403.520929
      -1.011585
      0.006144
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_1[0]
      drift_1:0
      drift
      30.0
      376.179327
      0.941387
      0.011785
      434.996645
      -1.086796
      0.011843
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole2
      quadrupole2:1
      quadrupole
      31.0
      376.179327
      -0.941387
      0.012209
      434.996645
      1.086796
      0.012209
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_2[0]
      drift_2:0
      drift
      60.0
      434.996645
      -1.086796
      0.023628
      376.179327
      0.941387
      0.023628
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      #e
      mycell$end:1
      marker
      60.0
      434.996645
      -1.086796
      0.023628
      376.179327
      0.941387
      0.023628
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

8 rows × 256 columns

```python
myDF[['name','s','betx','bety','alfx','alfy']]
```

      name
      s
      betx
      bety
      alfx
      alfy

      #s
      mycell$start:1
      0.0
      434.996645
      376.179327
      -1.086796
      0.941387

      quadrupole1
      quadrupole1:1
      1.0
      434.996645
      376.179327
      1.086796
      -0.941387

      drift_0[0]
      drift_0:0
      15.0
      405.549112
      403.520929
      1.016599
      -1.011585

      marker1
      marker1:1
      15.0
      405.549112
      403.520929
      1.016599
      -1.011585

      drift_1[0]
      drift_1:0
      30.0
      376.179327
      434.996645
      0.941387
      -1.086796

      quadrupole2
      quadrupole2:1
      31.0
      376.179327
      434.996645
      -0.941387
      1.086796

      drift_2[0]
      drift_2:0
      60.0
      434.996645
      376.179327
      -1.086796
      0.941387

      #e
      mycell$end:1
      60.0
      434.996645
      376.179327
      -1.086796
      0.941387

```python
myDF["s"]
```

    #s              0.0
    quadrupole1     1.0
    drift_0[0]     15.0
    marker1        15.0
    drift_1[0]     30.0
    quadrupole2    31.0
    drift_2[0]     60.0
    #e             60.0
    Name: s, dtype: float64

```python
myDF["betx"]
```

    #s             434.996645
    quadrupole1    434.996645
    drift_0[0]     405.549112
    marker1        405.549112
    drift_1[0]     376.179327
    quadrupole2    376.179327
    drift_2[0]     434.996645
    #e             434.996645
    Name: betx, dtype: float64

```python
myDF2=myMad.table.summ.dframe()
```

```python
myDF2
```

      length
      orbit5
      alfa
      gammatr
      q1
      dq1
      betxmax
      dxmax
      dxrms
      xcomax
      ...
      ycorms
      deltap
      synch_1
      synch_2
      synch_3
      synch_4
      synch_5
      synch_6
      synch_8
      nflips

      #e
      60.0
      -0.0
      1.110223e-16
      9.490627e+07
      0.023628
      -0.068435
      434.996645
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

1 rows × 27 columns

```python
myDF2["q1"]
```

    #e    0.023628
    Name: q1, dtype: float64

# Basic plot

```python
#Plot
%matplotlib notebook
plt.rcParams['savefig.dpi'] = 90
plt.rcParams['figure.dpi'] = 90

plt.plot(myDF['s'],myDF['betx'],'.-b',label='$\\beta_x$')
plt.plot(myDF['s'],myDF['bety'],'.-r',label='$\\beta_y$')
#Labels of the plot
plt.xlabel('s [m]')
plt.ylabel('[m]')
#Legend and grid
plt.legend(loc='best')
plt.grid()
```

    <IPython.core.display.Javascript object>

# For reference...

---
This is an example to get familiar with the use of the physical constants and the formatting of the output. Have a look on the difference.

```python
myString='''
a=pi;
value a;
set, format="22.20e";
value a;
'''
myMad.input(myString);
```

#
This is an example to get familiar with if and deferred expression. Please note the after the block delimited with {...} the ; can be omitted. Pay attention to circular call!

```python
myString='''
if (1==1){
option, echo=false, info=true;
a=pi;
b:=a;
c=a;
value a;
value b;
value c;
a=CLIGHT*cos(a);
value a;
value b;
value c;}
! BEWARE of circular call!
!a:=a+1;
! When evaluating you will get a fatal error
! value a;
option, echo=true, info=true;
'''
myMad.input(myString);
```

# ---
This is an example to get familiar with **while** and **macros** loops.

```python
myString='''
a(myvariable1,myvariable2): macro = {
value, myvariable1;
value, myvariable1*myvariable2;
}

N=1;
while (N<10){
exec, a(N,N);
N=N+1;
}
'''
myMad.input(myString);
```

---
### List of functions
In MAD-X the following functions are available

- SQRT(x) square root,
- LOG(x) natural logarithm,
- LOG10(x) logarithm base 10,
- EXP(x) exponential,
- SIN(x) trigonometric sine,
- COS(x) trigonometric cosine,
- TAN(x) trigonometric tangent,
- ASIN(x) arc sine,
- ACOS(x) arc cosine,
- ATAN(x) arc tangent,
- SINH(x) hyperbolic sine,
- COSH(x) hyperbolic cosine,
- TANH(x) hyperbolic tangent,
- SINC(x) cardinal sine function,
- ABS(x) absolute value,
- ERF(x) Gauss error,
- ERFC(x) complementary error,
- FLOOR(x) floor, largest previous integer,
- CEIL(x) ceiling, smallest next integer,
- ROUND(x) round, closest integer,
- FRAC(x) fractional part of number,
- RANF() random number, uniformly distributed in [0,1],
- GAUSS() random number, gaussian distribution with unit standard deviation,
- TGAUSS(x) random number, gaussian distribution with unit standard deviation, truncated at x standard deviations;

---
### List of physical constant

| MAD-X name  | symbol  |  value |unit|
|:-:|:-:|:-:|:-:|
|PI| π |4 * atan(1)| 1|
|TWOPI|2π| 2 * PI| 1|
|DEGRAD| 180/π |180 / PI| deg/rad|
|RADDEG| π/180 |PI / 180 |rad/deg|
|E| e |exp(1) |1|
|EMASS| me |0.510998928e−3| GeV|
|PMASS| mp |0.938272046| GeV|
|NMASS| u |0.931494061| GeV|
|MUMASS| mµ| 0.1056583715 |GeV|
|CLIGHT| c| 299792458| m/s|
|QELECT| e| 1.602176565e−19| A.s|
|HBAR| ¯h| 6.58211928e−25| MeV.s|
|ERAD| re| 2.8179403267e−15| m|
|PRAD| re(me/mp)| ERAD*EMASS/PMASS| m|


# Data Visualization for Exploration with `taladrod` Dataset

This notebook details data visualization for exploring a dataset. The goal is to understand more about the data as a human, not to make beautiful graphs, communicate, or feature engineering input into models.

```python
#reload scripts when we edit them
%load_ext autoreload
%autoreload 2

import pandas as pd
import numpy as np
import scipy.stats as st

#ggplot equivalent: plotnine
from plotnine import *

#scales package equivalent: mizani
from mizani.breaks import *
from mizani.formatters import *

#widgets
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

#utility
import utils

#stop the annoying warnings
import warnings
warnings.filterwarnings('ignore')
```

```python
'''
Snippet for plotnine with Thai font by @korakot
https://gist.github.com/korakot/01d181229b21411b0a20784e0ca20d3d
'''

from plotnine import *
import matplotlib.font_manager as fm

fm.fontManager.ttflist += fm.createFontList(['thsarabunnew-webfont.ttf'])
theme_set(theme_gray(11, 'TH Sarabun New'))
```

```python
df = pd.read_csv('data/taladrod/taladrod.csv')
df.head()
```

      brand
      series
      gen
      year
      color
      gear
      gas
      sales_price
      original_price
      market_price
      description
      contact_location
      subscribers
      date_diff
      dow

      0
      BMW
      others
      others
      2013.0
      สีขาว
      เกียร์ออโต้
      xxna
      1090000.0
      2697000.0
      2697000.0
      BMW 320i Modern ปี 2013 รถมือเดียวภายในภายนอกด...
      กรุงเทพ
      858.0
      NaN
      2

      1
      TOYOTA
      others
      others
      2009.0
      สีขาว
      เกียร์ออโต้
      LPG
      489000.0
      NaN
      NaN
      ฟรีดาวนventury G ออโต้ปี2011รถสวยมากมือเดียวไม...
      กรุงเทพ
      137.0
      NaN
      2

      2
      BENZ
      others
      others
      2004.0
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      999000.0
      NaN
      NaN
      E55 AMG V8 SuperCharge Tune by Zugus 500hp++ 7...
      นนทบุรี
      960.0
      NaN
      2

      3
      others
      others
      others
      2011.0
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      839000.0
      1978000.0
      1978000.0
      HYUNDAI GRAND STAREX 2.5 VIP ปี 2011\n- auto a...
      กรุงเทพ
      74.0
      NaN
      2

      4
      others
      others
      others
      2001.0
      สีดำ
      เกียร์ออโต้
      xxna
      655000.0
      NaN
      NaN
      ขายLand Rover Discovery 2 ปี 2001 เบนซิน วิ่งด...
      กรุงเทพ
      60.0
      NaN
      2

## Warming Up: Missing Values

We use grammar of graphics implementation `ggplot` (ported to Python as `plotnine`) to explore the `taladrod` dataset. Grammar of graphics is an especially useful tool since we do not know exactly what kind of plots we want to see and want to be able to add them up as we go.

Source: [A Comprehensive Guide to the Grammar of Graphics for Effective Visualization of Multi-dimensional Data](https://towardsdatascience.com/a-comprehensive-guide-to-the-grammar-of-graphics-for-effective-visualization-of-multi-dimensional-1f92b4ed4149)

```python
df_dirty = pd.read_csv('data/taladrod/taladrod_dirty.csv')
missing = utils.check_missing(df_dirty)
missing['over90'] = missing.per_missing.map(lambda x: True if x>0.9 else False)
missing
```

      col
      per_missing
      over90

      0
      original_price
      0.991266
      True

      1
      gas
      0.935968
      True

      2
      removed_date
      0.889504
      False

      3
      market_price
      0.557618
      False

      4
      subscribers
      0.000549
      False

      5
      model
      0.000000
      False

      6
      gen
      0.000000
      False

      7
      year
      0.000000
      False

      8
      color
      0.000000
      False

      9
      gear
      0.000000
      False

      10
      brand
      0.000000
      False

      11
      sales_price
      0.000000
      False

      12
      series
      0.000000
      False

      13
      description
      0.000000
      False

      14
      contact_location
      0.000000
      False

      15
      scraped_date
      0.000000
      False

      16
      id
      0.000000
      False

```python
g = (ggplot(missing,aes(x='col',y='per_missing',fill='over90')) + #base plot
     geom_col() + #type of plot
     geom_text(aes(x='col',y='per_missing+0.1',label='round(100*per_missing,2)')) +#annotate
     scale_y_continuous(labels=percent_format()) + #y-axis tick
     theme_minimal() + coord_flip()#theme and flipping plot
    )
g
```

## Categorical Variables

We want to know primarily two things about our categorical variables:
1. How each variable is distributed
2. How each variable relate to the dependent variable
    * 2.1 when dependent variable is numerical
    * 2.2 when dependent variable is categorical

```python
cat_vars = ['brand', 'series', 'gen', 'color', 'gear', 'gas','contact_location', 'dow']
cat_df = df[cat_vars].copy()
cat_df.head()
```

      brand
      series
      gen
      color
      gear
      gas
      contact_location
      dow

      0
      BMW
      others
      others
      สีขาว
      เกียร์ออโต้
      xxna
      กรุงเทพ
      2

      1
      TOYOTA
      others
      others
      สีขาว
      เกียร์ออโต้
      LPG
      กรุงเทพ
      2

      2
      BENZ
      others
      others
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      นนทบุรี
      2

      3
      others
      others
      others
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      กรุงเทพ
      2

      4
      others
      others
      others
      สีดำ
      เกียร์ออโต้
      xxna
      กรุงเทพ
      2

### Value Distribution

Even without plotting them out, we can see the value distribution in each variable using `ipywidgets`.

```python
interact(utils.value_dist, df =fixed(cat_df),
         col = widgets.Dropdown(options=list(cat_df.columns),value='brand'))
```

    interactive(children=(Dropdown(description='col', options=('brand', 'series', 'gen', 'color', 'gear', 'gas', '…

    <function utils.value_dist(df, col)>

**Coding Assignment** Implement `cat_plot` function that plots value distribution for each categorical variable.

```python
def cat_plot(df,col):
    return utils.cat_plot(df,col)
#input dataframe and column
#output histogram plot of value distribution

interact(cat_plot, df=fixed(cat_df),
         col = widgets.Dropdown(options=list(cat_df.columns),value='brand'))
```

    interactive(children=(Dropdown(description='col', options=('brand', 'series', 'gen', 'color', 'gear', 'gas', '…

    <function __main__.cat_plot(df, col)>

```python
#excluding others
def cat_plot_noothers(df,col):
    x = df.copy()
    x = x[x[col]!='others']
    return utils.cat_plot(x,col)

interact(cat_plot_noothers, df=fixed(cat_df),
         col = widgets.Dropdown(options=list(cat_df.columns),value='gen'))
```

    interactive(children=(Dropdown(description='col', index=2, options=('brand', 'series', 'gen', 'color', 'gear',…

    <function __main__.cat_plot_noothers(df, col)>

### Numerical and Categorical Variables

```python
#relationship between dependent variable and categorical variable
cat_df['sales_price'] = utils.boxcox(df['sales_price'])
cat_df.head()
```

      brand
      series
      gen
      color
      gear
      gas
      contact_location
      dow
      sales_price

      0
      BMW
      others
      others
      สีขาว
      เกียร์ออโต้
      xxna
      กรุงเทพ
      2
      13.883170

      1
      TOYOTA
      others
      others
      สีขาว
      เกียร์ออโต้
      LPG
      กรุงเทพ
      2
      13.058360

      2
      BENZ
      others
      others
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      นนทบุรี
      2
      13.794288

      3
      others
      others
      others
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      กรุงเทพ
      2
      13.615841

      4
      others
      others
      others
      สีดำ
      เกียร์ออโต้
      xxna
      กรุงเทพ
      2
      13.361382

```python
#relationship between sales price and color
cat_df.groupby('color').sales_price.describe()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      color

      others
      1756.0
      12.568359
      1.019319
      0.000000
      12.072547
      12.542548
      13.028055
      16.905644

      สีขาว
      4207.0
      13.085453
      0.940839
      0.000000
      12.672950
      13.005832
      13.498058
      16.438729

      สีดำ
      2848.0
      13.071142
      0.913027
      0.000000
      12.594734
      13.014780
      13.535466
      16.111071

      สีน้ำตาล
      607.0
      12.660002
      0.814474
      8.853808
      12.144203
      12.782689
      13.140205
      15.635210

      สีบรอนซ์เงิน
      1879.0
      12.760100
      0.923268
      0.000000
      12.421188
      12.779876
      13.151924
      15.854130

      สีเทา
      2740.0
      12.786451
      0.957211
      0.000000
      12.421188
      12.818555
      13.188621
      17.822480

**Coding Assignment** Implement `numcat_plot` function that plots the relationship between a dependent numerical variable and an independent categorical as displayed above. Useful geoms are `geom_boxplot`, `geom_violin` and `geom_jitter`. Optionally remove outliers before plotting.

```python
def numcat_plot(df,num,cat, no_outliers=True, geom=geom_boxplot()):
    return utils.numcat_plot(df, num, cat, no_outliers, geom)
#plot the summary above
```

```python
interact(numcat_plot,
         df=fixed(cat_df),
         num=fixed('sales_price'),
         no_outliers = widgets.Checkbox(value=True),
         geom=fixed(geom_boxplot()), #geom_violin, geom_jitter
         cat= widgets.Dropdown(options=list(cat_df.columns)[:-1],value='gen'))
```

    interactive(children=(Dropdown(description='cat', index=2, options=('brand', 'series', 'gen', 'color', 'gear',…

    <function __main__.numcat_plot(df, num, cat, no_outliers=True, geom=<plotnine.geoms.geom_boxplot.geom_boxplot object at 0x7fd455acd438>)>

```python
interact(numcat_plot,
         df=fixed(cat_df),
         num=fixed('sales_price'),
         no_outliers = widgets.Checkbox(value=True),
         geom=fixed(geom_violin()), #geom_violin, geom_jitter
         cat= widgets.Dropdown(options=list(cat_df.columns)[:-1],value='gear'))
```

    interactive(children=(Dropdown(description='cat', index=4, options=('brand', 'series', 'gen', 'color', 'gear',…

    <function __main__.numcat_plot(df, num, cat, no_outliers=True, geom=<plotnine.geoms.geom_boxplot.geom_boxplot object at 0x7fd455acd438>)>

Sometimes we want to see the numerical distribution filled with categories. This is especially useful plotting the results of a binary classification.

```python
def numdist_plot(df, num,cat, geom=geom_density(alpha=0.5), no_outliers=True):
    return utils.numdist_plot(df, num, cat, geom, no_outliers)

#either
#density: geom_density(alpha=0.5)
#histogram: geom_histogram(binwidth=0.5, position='identity',alpha=0.5)
#position: identity or dodge
numdist_plot(cat_df,'sales_price','gear')
```

```python
numdist_plot(cat_df,'sales_price','gear',
             geom=geom_histogram(binwidth=0.5, position='dodge',alpha=0.5))
```

### Categorical and Categorical Variables

**Exercise** We can cross-tabulate categorical variables to see their relationship by using `facet_wrap`; for instance, if our dependent variable is `gear` and indpendent variable of interest is `color`.

```python
def catcat_plot(df, cat_dep, cat_ind):
    return utils.catcat_plot(df,cat_dep,cat_ind)
```

```python
interact(catcat_plot,
         df=fixed(cat_df),
         cat_dep=widgets.Dropdown(options=list(cat_df.columns)[:-1],value='gear'),
         cat_ind= widgets.Dropdown(options=list(cat_df.columns)[:-1],value='color'))
```

    interactive(children=(Dropdown(description='cat_dep', index=4, options=('brand', 'series', 'gen', 'color', 'ge…

    <function __main__.catcat_plot(df, cat_dep, cat_ind)>

### Multiple Ways of Relationships

You can use `facet_grid` to display multiple ways of relationships; but keep in mind that this is probably what your model is doing anyways so it might not be most human-readable plot to explore.

```python
df.head()
```

      brand
      series
      gen
      year
      color
      gear
      gas
      sales_price
      original_price
      market_price
      description
      contact_location
      subscribers
      date_diff
      dow

      0
      BMW
      others
      others
      2013.0
      สีขาว
      เกียร์ออโต้
      xxna
      1070001.0
      2697000.0
      2697000.0
      BMW 320i Modern ปี 2013 รถมือเดียวภายในภายนอกด...
      กรุงเทพ
      858.0
      NaN
      2

      1
      TOYOTA
      others
      others
      2009.0
      สีขาว
      เกียร์ออโต้
      LPG
      469001.0
      NaN
      NaN
      ฟรีดาวนventury G ออโต้ปี2011รถสวยมากมือเดียวไม...
      กรุงเทพ
      137.0
      NaN
      2

      2
      BENZ
      others
      others
      2004.0
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      979001.0
      NaN
      NaN
      E55 AMG V8 SuperCharge Tune by Zugus 500hp++ 7...
      นนทบุรี
      960.0
      NaN
      2

      3
      others
      others
      others
      2011.0
      สีบรอนซ์เงิน
      เกียร์ออโต้
      xxna
      819001.0
      1978000.0
      1978000.0
      HYUNDAI GRAND STAREX 2.5 VIP ปี 2011\n- auto a...
      กรุงเทพ
      74.0
      NaN
      2

      4
      others
      others
      others
      2001.0
      สีดำ
      เกียร์ออโต้
      xxna
      635001.0
      NaN
      NaN
      ขายLand Rover Discovery 2 ปี 2001 เบนซิน วิ่งด...
      กรุงเทพ
      60.0
      NaN
      2

```python
#getting fancy; not necessarily the best idea
new_df = utils.remove_outliers(cat_df,'sales_price')
g = (ggplot(new_df, aes(x='gen',y='sales_price')) +
     geom_boxplot() + theme_minimal() +
     facet_grid('contact_location~color') +
     theme(axis_text_x = element_text(angle = 90, hjust = 1)) +
     theme(text=element_text(size=11,  family='TH Sarabun New'))
    )
g
```

## Numerical Variables

We want to know two things about numerical variables:
1. Their distributions
2. Their relationships with one another; possibly this involves transforming variables to make them less skewed aka more difficult to see variations

```python
import datetime
now = datetime.datetime.now()
df['nb_year'] = now.year - df['year']
num_vars = ['nb_year','sales_price','market_price','subscribers']
num_df = df[num_vars].dropna() #this is why you need to deal with missing values BEFORE exploration
num_df.describe()
```

      nb_year
      sales_price
      market_price
      subscribers

      count
      6150.000000
      6.150000e+03
      6.150000e+03
      6150.000000

      mean
      4.512195
      7.120724e+05
      1.333557e+06
      525.634959

      std
      2.026358
      7.745356e+05
      1.288468e+06
      386.841217

      min
      0.000000
      1.000000e+00
      3.690000e+05
      1.000000

      25%
      3.000000
      3.390010e+05
      6.490000e+05
      191.250000

      50%
      5.000000
      4.790010e+05
      8.770000e+05
      423.000000

      75%
      6.000000
      7.590010e+05
      1.445000e+06
      772.000000

      max
      8.000000
      1.297000e+07
      1.645000e+07
      1469.000000

`seaborn` has an excellent `pairplot` implementation which not only shows the distribution of values but also their relathionships. It seems like we can get what we want easily; however, as we can see `sales_price` and `market_price` are a little skewed, making it more difficult to see their relationships with other more spread out variables.

```python
import seaborn as sns
sns.pairplot(num_df) #non-normal data is a problem!
```

In a lot of cases, a variable with normally distributed values have more variations and easier for us to see their relationships with other variables. We will try to transform our skewed variables to more "normal" ones to see if that helps.

**Q-Q plot** compares two probability distributions by plotting their quantiles against each other. We can use this to determine the normality of a variable by plotting the sample quantiles (from the data we have) against its theoretical quantiles (where the quantiles would be if the variable is normally distributed).

```python
interact(utils.qq_plot, df=fixed(num_df),
         col=widgets.Dropdown(options=list(num_df.columns)))
```

    interactive(children=(Dropdown(description='col', options=('nb_year', 'sales_price', 'market_price', 'subscrib…

    <function utils.qq_plot(df, col)>

**Box-Cox transformation** is a statistical technique used to make data look like more normally distributed.

\begin{align}
g_\lambda(y) = \left\{
\begin{array}{lr}\displaystyle\frac{y^\lambda - 1}{\lambda} &  \lambda \neq 0\\
        & \\
       \log(y) &  \lambda = 0
     \end{array}
   \right.
\end{align}

**Exercise** Implement `boxcox` transformation according to the equation above.

```python
def boxcox(ser,lamb=0):
    pass
#input a column from pandas dataframe
#output transformed column
```

One way of choosing the hyperparameter $\lambda$ is to look at the Q-Q plot and choose transformation which makes the slope closest to 1.

```python
#see transformation results
def what_lamb(df,col,lamb):
    sample_df = df.copy()
    former_g = utils.qq_plot(sample_df,col)
    sample_df[col] = utils.boxcox(sample_df[col],lamb)
    print(utils.qq_plot(sample_df,col),former_g)

interact(what_lamb, df=fixed(num_df),
         col=widgets.Dropdown(options=list(num_df.columns),value='sales_price'),
         lamb=widgets.FloatSlider(min=-3,max=3,step=0.5,value=0)
         )
```

    interactive(children=(Dropdown(description='col', index=1, options=('nb_year', 'sales_price', 'market_price', …

    <function __main__.what_lamb(df, col, lamb)>

This can also be automated by plotting a slope for each arbitary $\lambda$; for instance from -3 to 3.

```python
lamb_df = utils.boxcox_lamb_df(num_df.subscribers)
interact(utils.boxcox_plot, df=fixed(num_df),
         col=widgets.Dropdown(options=list(num_df.columns),value='sales_price'),
         ls=fixed([i/10 for i in range(-30,31,5)])
         )
```

    interactive(children=(Dropdown(description='col', index=1, options=('nb_year', 'sales_price', 'market_price', …

    <function utils.boxcox_plot(df, col, ls=[-3.0, -2.5, -2.0, -1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0])>

```python
#transform sales and market prices
for col in ['sales_price','market_price']:
    num_df['new_'+col] = utils.boxcox(num_df[col], utils.boxcox_lamb(num_df[col]))
```

You can see that post transformation, we can see the (lack of) relationships between variables clearer.

```python
sns.pairplot(num_df[['nb_year','new_sales_price','new_market_price','subscribers']]) #a little better!
```

For our example, we have only four numerical variables; but imagine when you have ten or more. You may want to plot their distributions separately from relationships.

```python
num_m = num_df.melt()
num_m.head()
```

      variable
      value

      0
      nb_year
      6.0

      1
      nb_year
      8.0

      2
      nb_year
      8.0

      3
      nb_year
      2.0

      4
      nb_year
      3.0

**Coding Assignment** Implement `value_dist_plot` to plot value distribution of all variables.

```python
def value_dist_plot(df,bins=30):
    return utils.value_dist_plot(df,bins)
#input dataframe with only numerical variables
#output distribution plot for each variable
value_dist_plot(num_df)
```

Likewise in case there are too many pairs of relationships, you might plot the relationships pair-by-pair with `ipywidget` and `seaborn`'s `jointplot` function.

```python
interact(utils.jointplot, df=fixed(num_df),
         col_x= widgets.Dropdown(options=list(num_df.columns),value='sales_price'),
         col_y=widgets.Dropdown(options=list(num_df.columns),value='market_price'),
         kind=widgets.Dropdown(options=['scatter','resid','reg','hex','kde','point'],value='scatter'))
```

    interactive(children=(Dropdown(description='col_x', index=1, options=('nb_year', 'sales_price', 'market_price'…

    <function utils.jointplot(df, col_x, col_y, no_outliers=True, kind='reg')>

As you might have noticed, we have not used any statistical concept to describe the relationship, and that is by design. We can also see correlation table with a simple `pandas` function:

```python
#correlation plot if you must; but it's just ONE number for the relationship
num_df.corr(method='pearson').style.background_gradient(cmap='coolwarm')
```

        	        nb_year	        sales_price	        market_price	        subscribers	        new_sales_price	        new_market_price

                        nb_year
                        1
                        -0.220633
                        -0.00786656
                        0.163404
                        -0.275434
                        -0.0152819

                        sales_price
                        -0.220633
                        1
                        0.862739
                        -0.07795
                        0.955499
                        0.819534

                        market_price
                        -0.00786656
                        0.862739
                        1
                        -0.063786
                        0.858924
                        0.948289

                        subscribers
                        0.163404
                        -0.07795
                        -0.063786
                        1
                        -0.0165204
                        0.039791

                        new_sales_price
                        -0.275434
                        0.955499
                        0.858924
                        -0.0165204
                        1
                        0.882803

                        new_market_price
                        -0.0152819
                        0.819534
                        0.948289
                        0.039791
                        0.882803
                        1

```python
def pearson_corr(x,y):
    sub_x = x - x.mean()
    sub_y = y - y.mean()
    return (sub_x * sub_y).sum() / np.sqrt((sub_x**2).sum() * (sub_y**2).sum())

#spearman and kendall: pearson with rank variables
pearson_corr(df.nb_year,df.sales_price)
```

    -0.23316757399518453

However, the famous Anscombe plots show us that it is always better to look at distribution rather than a summary number.

Source: [A Comprehensive Guide to the Grammar of Graphics for Effective Visualization of Multi-dimensional Data](https://towardsdatascience.com/a-comprehensive-guide-to-the-grammar-of-graphics-for-effective-visualization-of-multi-dimensional-1f92b4ed4149)

```python

```


# Markdown in Jupyter Notebooks

In order to be able to enter formatted Markdown text, select "Markdown" in the dropdown for the mode of the cell (it usually says "code").

## Quick reference

### Visual look

* `**bold**`: **bold** text
* `_italic_`: _italic_
* `~~strike~~`: ~~strike~~
* Headers: `# Main`, `## Subheader`, `### Subsub header`, ...

Explore more by selecting words and clicking on the formatting buttons of this editor.

### Structure

Lists:
```
* item1
* item2
```

Enumeration:
```
1. item1
1. item2
```

### Special elements

* Links: `[name of link](http://url.it.links/to)`

## Formuas

It is also possible to embed formulas. They are formatted using [MathJax](https://www.mathjax.org/).

Inline $x^2$ formula.

\begin{equation}
   E = mc^2
\end{equation}

## References

* [Original description](https://daringfireball.net/projects/markdown/)

```python

```


```javascript
%%javascript
$('#appmode-leave').hide();
$('#copy-binder-link').hide();
$('#visit-repo-link').hide();
```

# Water anomalies
Despite its simple molecular structure, water is an exceptionally complicated fluid.
Many of its properties do not follow the trends obeyed by other liquids and are often referred to as water anomalies.
Scientists often report more than 50 _anomalous_ properties of water, and some of the most well-known examples are
1. Water has an unusually high melting point for a molecule of such a low molecular weight.
2. Water has an unusually high boiling point for a molecule of such a low molecular weight.
3. A liquid-liquid transition occurs at about 330 K.
4. Pressure reduces ice's melting point.
5. Cold liquid water has a high density that increases on warming (up to 3.984 °C).

Most of these anomalous behaviours have been explained and there is ample scientific (and non-scientific) literature discussing them. As an example, this website [https://water.lsbu.ac.uk/water/water_anomalies.html](https://water.lsbu.ac.uk/water/water_anomalies.html) will provide a good overview, and plenty of references, on the topic.

In this numerical workshop you will use a computational technique called Molecular Dynamics (MD) to study the how the water density changes with temperature.
MD is one of the most widely used type of atomistic simulations, and it is now routinely used in many research groups to complement experimental studies.

## Molecular dynamics
Molecular dynamics is conceptually very simple, an iterative solution of Newton's equations of motions at the atomic level, but subtly complicated to use for direct quantitative comparison with experiments.
Although a detailed description of MD is beyond the scope of this laboratory, it is worth discussing some basic ideas for you to start appreciating the power and limitations of this technique. There are plenty of webpage and tutorials that describe the working principles of MD; Wikipedia has a fairly good an general overview of this topic [https://en.wikipedia.org/wiki/Molecular_dynamics]( https://en.wikipedia.org/wiki/Molecular_dynamics).
Another very good reference, albeit a bit dated, is this collection of notes from the 1997 ICTP Spring Colleges in Computational Physics[Molecular Dynamics primer](https://web.mst.edu/~vojtat/class_5403/ercolessi.pdf).

In MD the atoms are treated as point particles with a mass and a partial charge. Their interactions are described by simple empirical equations, such as the Coulomb and the van der Waals  (dispersion) forces, supplemented with two-, three- or four-body interactions to better capture the covalent nature of the intramolecular bonds.
For example, in classical molecular dynamics the interaction *energy* between two non-bonded atoms separated by a distance, $r$, can be written as

\begin{equation}
U_{ij} = \frac{1}{4\pi\varepsilon_0}\frac{q_i q_j}{r} + \frac{A}{r^{12}} - \frac{B}{r^6} \tag{1}
\end{equation}

Where the first term is the Coulomb interaction and the last two the repulsive and attractive parts of the van der Waals interactions.
On the other hand the bonded two-, three- and four-body interactions between covalently bonded atoms are typically described by *harmonic* potentials

\begin{eqnarray}
U_{ij}^b &=& K_b(b_{ij}-b_0)^2 \tag{2} \\
U_{ijk}^a &=& K_\theta(\theta_{ijk}-\theta_0)^2 \tag{3} \\
U_{ijkl}^t &=& K_\phi[1+\cos(n\phi_{jikl}-\phi_0)]^2 \tag{3} \\
\end{eqnarray}

where $b_{ij}$, $\theta_{ijk}$ and $\phi_{jikl}$ are the bond lengths, angle and torsional angle between the atoms and the other quantities are fitting parameters, which are key to determine the accuracy of the simulations.

Once the interaction energy is known we can then compute the forces on the atoms as the sum of all pair-wise interactions

\begin{equation}
F_i = -\nabla_i U = \sum_{j\neq i} F_{ij} = -\Bigg[
  \sum \frac{\partial U_{ij}}{\partial x_i} +
  \sum \frac{\partial U_{ij}^a}{\partial x_i} +
  \sum \frac{\partial U_{ijk}^b}{\partial x_i} +
  \sum \frac{\partial U_{ijkl}^t}{\partial x_i} \Bigg] \tag{5}
\end{equation}

Then, by knowing the positions, velocities and forces for all the atoms at a certain time $t$, we can use the Newton's equations of motions to *predict* the positions and velocities of the particles after a certain (short) amount of time as passed

\begin{eqnarray}
a_i(t) &=& \frac{F_i(t)}{m_i} \tag{6} \\
v_i(t+\delta t) &=& v_i(t) + a_i(t)\delta t \tag{7} \\
x_i(t+\delta t) &=& x_i(t) + v_i(t)\delta t + \frac{1}{2}a_i(t)\delta t^2 \tag{8} \\
\end{eqnarray}

where $a_i$, $v_i$ and $x_i$ are the acceleration, velocity and position of particle $i$, and $\delta t$ is called the time step.
These three equations (or some variants of them) are usually called **equations of motions**.
Now that we have the new atomic positions we can compute the new forces on the atoms, and use again the Newton's equations of motions to *propagate* the atoms' positions further.
This iterative procedure will generate a **trajectory** for the atoms, and by using energies and velocities collected along the way we will also get information about the temperature, pressure and other thermodynamic quantities of the system.

### Importance of the time step
The time step is one of the most important quantities in MD, and it is key to understand the potentials and limitations of atomistic molecular dynamics simulations.
In fact, for the above equations of motions to be valid, the time step has to be short enough to describe the fastest **atomic** motion in the system, which in the case of water is the O-H stretching mode.
The O-H stretching has a vibrational frequency of approximately $1\times10^{14}$Hz, *i.e* it takes about $1\times10^{-14}$s to complete one oscillation. Therefore, if we want to describe this very fast atomic motion using discrete points in time we would need 10-20 snapshots. Hence the time step has to be of the order of 1~fs ($1\times10^{-15}$s) or less.

Now, let’s imagine running a simulation with a 1 fs time step and that the computer take 10 ms to calculate energies, forces and do one cycle of the equations of motions.
In the table below you can see how long it would you take to simulate a chemical or physical process depending on the time scale it experimentally occurs

| Experimental time scale  | Phenomenon   | Simulation time
| :-----: | :--------: |:---------
| 10 fs   | O-H vibration                     | 0.1 s
| 1 ps    | H-bond persistence                | 10 s
| 1 ns    | Ion permeation through a membrane | 3 hours
| 1$\mu$s | Conformational rearrangement      | 115 days
| 1 ms    | Protein folding (fast)            | 317 days
| 1 s     | Protein folding (typical)         | 317,000 days

Obviously, the time required to do one MD cycle depends on the number of operations the computer has to perform, hence it increases with the system size.
Although computational power has increased exponentially since MD was first introduced in the 1940s and we can now afford to study systems of millions of atoms or of hundreds of nm in size, there are still strong limitations to what can be reliably simulated due to the finite (small) number of atoms is included in the system (compared to Avogadro's number) and the short time scale that the simulation can span.

### Ensemble
MD codes are more complicated that a simple iterative solution of Newton's equations of motions and they include algorithms to control the temperature, pressure and other thermodynamics quantities of the system.
Of particular relevance for this experience is the need to use **thermostats** and **barostats** to fix the boundary conditions of the simulations.
For this laboratory is not important to know the working details of these algorithms, but is key that you are aware that the variables relating to the temperature and pressure of the simulations are input parameters that may need to be changed.

## Scope of the laboratory
The scope of this virtual experiment is to introduce you to atomistic molecular dynamics simulations and to compute the variations of the water density as a function of temperature, and to compare it with experimental values. As briefly mentioned above, the accuracy of the simulation depends on the parameters that are used to compute the intermolecular interactions.
The water models available for this laboratory are
* SPC/E
* TIP3P
* TIP4P/ew
* TIP5P

which is a very small selections of the available models for water.
The models are listed in increasing level of complexity and one would reasonably expect that the more complex model gives more accurate results.

In this laboratory, you will choose one water model and run a series of simulations to compute the water density at various temperatures and determine the volumetric thermal expansion coefficient of water $\alpha$, which is defined as

\begin{equation*}
\alpha(T) = \frac{1}{V}\bigg(\frac{\mathrm{d}V}{\mathrm{d}T}\bigg)_P = \bigg(\frac{\mathrm{d}\ln V}{\mathrm{d}T}\bigg)_P \tag{9}
\end{equation*}

where the subscript _P_ means that the simulations are performed at constant pressure, and we have explicitly indicated the the volumetric thermal expansion coefficient depends on the temperature. In this laboratory we will assume that the temperature dependence of $\alpha(T)$ can be described with a simple polynomial expansion, which means that $\ln V(T)$ can also be described with a polynomial expansion

\begin{eqnarray*}
\ln V(T) &=& a + bT + cT^2 + \dots \\
\frac{\mathrm{d}\ln V}{\mathrm{d}T} = \alpha(T) &=& b + 2cT + \dots
\end{eqnarray*}

From each simulation you will compute the average volume of the simulation cell, discarding the initial portion of the simulation, where the system is out of equilibrium.

You will then fit the logarithm of the average volumes _vs_ temperature with a polynomial function, and compute the value of its derivative, $\alpha$, at a few selected temperatures.

## Your first MD simulation

You will now run your first MD simulation.
In the next Jupyter Notebook we will go under the hood of a molecular dynamics code in written python to see how the ensemble parameters are passed to the code, how to modify them and what outputs the code produces.

This first simulation will also give you an indication of how long a typical run takes. Please note that the timing depends on the load of the machine, so the more users are working on this laboratory to slower the calculations might get. This could be the "worst case scenario" given that all the class would be accessing the VM.

[Run Molecular Dynamics](templateMD.ipynb)

## Section 3 - Molecular dynamics simulations of water

### Section 3.1 - Introduction

**Task 1** - Describe the scope of the laboratory

**Task 2** - Describe the water model that you have chosen

**Task 3** - Describe The set of temperatures that you have chosen

**Task 4** - Detail the length of the simulations you chose.

### Section 3.1 - Results

**Task 5** - How you computed the average, standard error and standard deviation of the volume/density

**Task 6** - A table with the average (molar) volumes and densities

**Task 7** - A plot of $\ln V$ _vs_ T and the polynomial fitting function

**Task 8** - The volumetric thermal expansion coefficient of water at 20$^\circ$C and 50$^\circ$C

### Section 3.1 - Discussion

**Task 9** - A comparison between your values for the density and the thermal expansion coefficient with literature values for "real" water and obtained from other simulations with the same water model.

```python

```


##### Copyright 2020 The TensorFlow Authors.

```python
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Quantum data

    View on TensorFlow.org

    Run in Google Colab

    View source on GitHub

    Download notebook

Building off of the comparisons made in the [MNIST](https://www.tensorflow.org/quantum/tutorials/mnist) tutorial, this tutorial explores the recent work of [Huang et al.](https://arxiv.org/abs/2011.01938) that shows how different datasets affect performance comparisons. In the work, the authors seek to understand how and when classical machine learning models can learn as well as (or better than) quantum models. The work also showcases an empirical performance separation between classical and quantum machine learning model via a carefully crafted dataset. You will:

1.   Prepare a reduced dimension Fashion-MNIST dataset.
2.   Use quantum circuits to re-label the dataset and compute Projected Quantum Kernel features (PQK).
3.   Train a classical neural network on the re-labeled dataset and compare the performance with a model that has access to the PQK features.

## Setup

```python
!pip -q install tensorflow==2.3.1 tensorflow-quantum
```

    WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
    Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
    To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.

```python
import cirq
import sympy
import numpy as np
import tensorflow as tf
import tensorflow_quantum as tfq

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
np.random.seed(1234)
```

## 1. Data preparation

You will begin by preparing the fashion-MNIST dataset for running on a quantum computer.

### 1.1 Download fashion-MNIST

The first step is to get the traditional fashion-mnist dataset. This can be done using the `tf.keras.datasets` module.

```python
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()

# Rescale the images from [0,255] to the [0.0,1.0] range.
x_train, x_test = x_train/255.0, x_test/255.0

print("Number of original training examples:", len(x_train))
print("Number of original test examples:", len(x_test))
```

    Number of original training examples: 60000
    Number of original test examples: 10000

Filter the dataset to keep just the T-shirts/tops and dresses, remove the other classes. At the same time convert the label, `y`, to boolean: True for 0 and False for 3.

```python
def filter_03(x, y):
    keep = (y == 0) | (y == 3)
    x, y = x[keep], y[keep]
    y = y == 0
    return x,y
```

```python
x_train, y_train = filter_03(x_train, y_train)
x_test, y_test = filter_03(x_test, y_test)

print("Number of filtered training examples:", len(x_train))
print("Number of filtered test examples:", len(x_test))
```

    Number of filtered training examples: 12000
    Number of filtered test examples: 2000

```python
print(y_train[0])

plt.imshow(x_train[0, :, :])
plt.colorbar()
```

### 1.2 Downscale the images

Just like the MNIST example, you will need to downscale these images in order to be within the boundaries for current quantum computers. This time however you will use a PCA transformation to reduce the dimensions instead of a `tf.image.resize` operation.

```python
def truncate_x(x_train, x_test, n_components=10):
  """Perform PCA on image dataset keeping the top `n_components` components."""
  n_points_train = tf.gather(tf.shape(x_train), 0)
  n_points_test = tf.gather(tf.shape(x_test), 0)

  # Flatten to 1D
  x_train = tf.reshape(x_train, [n_points_train, -1])
  x_test = tf.reshape(x_test, [n_points_test, -1])

  # Normalize.
  feature_mean = tf.reduce_mean(x_train, axis=0)
  x_train_normalized = x_train - feature_mean
  x_test_normalized = x_test - feature_mean

  # Truncate.
  e_values, e_vectors = tf.linalg.eigh(
      tf.einsum('ji,jk->ik', x_train_normalized, x_train_normalized))
  return tf.einsum('ij,jk->ik', x_train_normalized, e_vectors[:,-n_components:]), \
    tf.einsum('ij,jk->ik', x_test_normalized, e_vectors[:, -n_components:])
```

```python
DATASET_DIM = 10
x_train, x_test = truncate_x(x_train, x_test, n_components=DATASET_DIM)
print(f'New datapoint dimension:', len(x_train[0]))
```

    New datapoint dimension: 10

The last step is to reduce the size of the dataset to just 1000 training datapoints and 200 testing datapoints.

```python
N_TRAIN = 1000
N_TEST = 200
x_train, x_test = x_train[:N_TRAIN], x_test[:N_TEST]
y_train, y_test = y_train[:N_TRAIN], y_test[:N_TEST]
```

```python
print("New number of training examples:", len(x_train))
print("New number of test examples:", len(x_test))
```

    New number of training examples: 1000
    New number of test examples: 200

## 2. Relabeling and computing PQK features

You will now prepare a "stilted" quantum dataset by incorporating quantum components and re-labeling the truncated fashion-MNIST dataset you've created above. In order to get the most seperation between quantum and classical methods, you will first prepare the PQK features and then relabel outputs based on their values.

### 2.1 Quantum encoding and PQK features
You will create a new set of features, based on `x_train`, `y_train`, `x_test` and `y_test` that is defined to be the 1-RDM on all qubits of:

$V(x_{\text{train}} / n_{\text{trotter}}) ^ {n_{\text{trotter}}} U_{\text{1qb}} | 0 \rangle$

Where $U_\text{1qb}$ is a wall of single qubit rotations and $V(\hat{\theta}) = e^{-i\sum_i \hat{\theta_i} (X_i X_{i+1} + Y_i Y_{i+1} + Z_i Z_{i+1})}$

First, you can generate the wall of single qubit rotations:

```python
def single_qubit_wall(qubits, rotations):
  """Prepare a single qubit X,Y,Z rotation wall on `qubits`."""
  wall_circuit = cirq.Circuit()
  for i, qubit in enumerate(qubits):
    for j, gate in enumerate([cirq.X, cirq.Y, cirq.Z]):
      wall_circuit.append(gate(qubit) ** rotations[i][j])

  return wall_circuit
```

You can quickly verify this works by looking at the circuit:

```python
SVGCircuit(single_qubit_wall(
    cirq.GridQubit.rect(1,4), np.random.uniform(size=(4, 3))))
```

Next you can prepare $V(\hat{\theta})$ with the help of `tfq.util.exponential` which can exponentiate any commuting `cirq.PauliSum` objects:

```python
def v_theta(qubits):
  """Prepares a circuit that generates V(\theta)."""
  ref_paulis = [
      cirq.X(q0) * cirq.X(q1) + \
      cirq.Y(q0) * cirq.Y(q1) + \
      cirq.Z(q0) * cirq.Z(q1) for q0, q1 in zip(qubits, qubits[1:])
  ]
  exp_symbols = list(sympy.symbols('ref_0:'+str(len(ref_paulis))))
  return tfq.util.exponential(ref_paulis, exp_symbols), exp_symbols
```

This circuit might be a little bit harder to verify by looking at, but you can still examine a two qubit case to see what is happening:

```python
test_circuit, test_symbols = v_theta(cirq.GridQubit.rect(1, 2))
print(f'Symbols found in circuit:{test_symbols}')
SVGCircuit(test_circuit)
```

    Symbols found in circuit:[ref_0]

Now you have all the building blocks you need to put your full encoding circuits together:

```python
def prepare_pqk_circuits(qubits, classical_source, n_trotter=10):
  """Prepare the pqk feature circuits around a dataset."""
  n_qubits = len(qubits)
  n_points = len(classical_source)

  # Prepare random single qubit rotation wall.
  random_rots = np.random.uniform(-2, 2, size=(n_qubits, 3))
  initial_U = single_qubit_wall(qubits, random_rots)

  # Prepare parametrized V
  V_circuit, symbols = v_theta(qubits)
  exp_circuit = cirq.Circuit(V_circuit for t in range(n_trotter))

  # Convert to `tf.Tensor`
  initial_U_tensor = tfq.convert_to_tensor([initial_U])
  initial_U_splat = tf.tile(initial_U_tensor, [n_points])

  full_circuits = tfq.layers.AddCircuit()(
      initial_U_splat, append=exp_circuit)
  # Replace placeholders in circuits with values from `classical_source`.
  return tfq.resolve_parameters(
      full_circuits, tf.convert_to_tensor([str(x) for x in symbols]),
      tf.convert_to_tensor(classical_source*(n_qubits/3)/n_trotter))
```

Choose some qubits and prepare the data encoding circuits:

```python
qubits = cirq.GridQubit.rect(1, DATASET_DIM + 1)
q_x_train_circuits = prepare_pqk_circuits(qubits, x_train)
q_x_test_circuits = prepare_pqk_circuits(qubits, x_test)
```

Next, compute the PQK features based on the 1-RDM of the dataset circuits above and store the results in `rdm`, a `tf.Tensor` with shape `[n_points, n_qubits, 3]`. The entries in `rdm[i][j][k]` = $\langle \psi_i | OP^k_j | \psi_i \rangle$ where `i` indexes over datapoints, `j` indexes over qubits and `k` indexes over $\lbrace \hat{X}, \hat{Y}, \hat{Z} \rbrace$ .

```python
def get_pqk_features(qubits, data_batch):
  """Get PQK features based on above construction."""
  ops = [[cirq.X(q), cirq.Y(q), cirq.Z(q)] for q in qubits]
  ops_tensor = tf.expand_dims(tf.reshape(tfq.convert_to_tensor(ops), -1), 0)
  batch_dim = tf.gather(tf.shape(data_batch), 0)
  ops_splat = tf.tile(ops_tensor, [batch_dim, 1])
  exp_vals = tfq.layers.Expectation()(data_batch, operators=ops_splat)
  rdm = tf.reshape(exp_vals, [batch_dim, len(qubits), -1])
  return rdm
```

```python
x_train_pqk = get_pqk_features(qubits, q_x_train_circuits)
x_test_pqk = get_pqk_features(qubits, q_x_test_circuits)
print('New PQK training dataset has shape:', x_train_pqk.shape)
print('New PQK testing dataset has shape:', x_test_pqk.shape)
```

    New PQK training dataset has shape: (1000, 11, 3)
    New PQK testing dataset has shape: (200, 11, 3)

### 2.2 Re-labeling based on PQK features
Now that you have these quantum generated features in `x_train_pqk` and `x_test_pqk`, it is time to re-label the dataset. To achieve maximum seperation between quantum and classical performance you can re-label the dataset based on the spectrum information found in `x_train_pqk` and `x_test_pqk`.

Note: This preparation of your dataset to explicitly maximize the seperation in performance between the classical and quantum models might feel like cheating, but it provides a **very** important proof of existance for datasets that are hard for classical computers and easy for quantum computers to model. There would be no point in searching for quantum advantage in QML if you couldn't first create something like this to demonstrate advantage.

```python
def compute_kernel_matrix(vecs, gamma):
  """Computes d[i][j] = e^ -gamma * (vecs[i] - vecs[j]) ** 2 """
  scaled_gamma = gamma / (
      tf.cast(tf.gather(tf.shape(vecs), 1), tf.float32) * tf.math.reduce_std(vecs))
  return scaled_gamma * tf.einsum('ijk->ij',(vecs[:,None,:] - vecs) ** 2)

def get_spectrum(datapoints, gamma=1.0):
  """Compute the eigenvalues and eigenvectors of the kernel of datapoints."""
  KC_qs = compute_kernel_matrix(datapoints, gamma)
  S, V = tf.linalg.eigh(KC_qs)
  S = tf.math.abs(S)
  return S, V
```

```python
S_pqk, V_pqk = get_spectrum(
    tf.reshape(tf.concat([x_train_pqk, x_test_pqk], 0), [-1, len(qubits) * 3]))

S_original, V_original = get_spectrum(
    tf.cast(tf.concat([x_train, x_test], 0), tf.float32), gamma=0.005)

print('Eigenvectors of pqk kernel matrix:', V_pqk)
print('Eigenvectors of original kernel matrix:', V_original)
```

    Eigenvectors of pqk kernel matrix: tf.Tensor(
    [[-2.09569391e-02  1.05973557e-02  2.16634180e-02 ...  2.80352887e-02
       1.55521873e-02  2.82677952e-02]
     [-2.29303762e-02  4.66355234e-02  7.91163836e-03 ... -6.14174758e-04
      -7.07804322e-01  2.85902526e-02]
     [-1.77853629e-02 -3.00758495e-03 -2.55225878e-02 ... -2.40783971e-02
       2.11018627e-03  2.69009806e-02]
     ...
     [ 6.05797209e-02  1.32483775e-02  2.69536003e-02 ... -1.38843581e-02
       3.05043962e-02  3.85345481e-02]
     [ 6.33309558e-02 -3.04112374e-03  9.77444276e-03 ...  7.48321265e-02
       3.42793856e-03  3.67484428e-02]
     [ 5.86028099e-02  5.84433973e-03  2.64811981e-03 ...  2.82612257e-02
      -3.80136147e-02  3.29943895e-02]], shape=(1200, 1200), dtype=float32)
    Eigenvectors of original kernel matrix: tf.Tensor(
    [[ 0.03835681  0.0283473  -0.01169789 ...  0.02343717  0.0211248
       0.03206972]
     [-0.04018159  0.00888097 -0.01388255 ...  0.00582427  0.717551
       0.02881948]
     [-0.0166719   0.01350376 -0.03663862 ...  0.02467175 -0.00415936
       0.02195409]
     ...
     [-0.03015648 -0.01671632 -0.01603392 ...  0.00100583 -0.00261221
       0.02365689]
     [ 0.0039777  -0.04998879 -0.00528336 ...  0.01560401 -0.04330755
       0.02782002]
     [-0.01665728 -0.00818616 -0.0432341  ...  0.00088256  0.00927396
       0.01875088]], shape=(1200, 1200), dtype=float32)

Now you have everything you need to re-label the dataset! Now you can consult with the flowchart to better understand how to maximize performance seperation when re-labeling the dataset:

In order to maximize the seperation between quantum and classical models, you will attempt to maximize the geometric difference between the original dataset and the PQK features kernel matrices $g(K_1 || K_2) = \sqrt{ || \sqrt{K_2} K_1^{-1} \sqrt{K_2} || _\infty}$ using `S_pqk, V_pqk` and `S_original, V_original`. A large value of $g$ ensures that you initially move to the right in the flowchart down towards a prediction advantage in the quantum case.

Note: Computing quantities for $s$ and $d$ are also very useful when looking to better understand performance seperations. In this case ensuring a large $g$ value is enough to see performance seperation.

```python
def get_stilted_dataset(S, V, S_2, V_2, lambdav=1.1):
  """Prepare new labels that maximize geometric distance between kernels."""
  S_diag = tf.linalg.diag(S ** 0.5)
  S_2_diag = tf.linalg.diag(S_2 / (S_2 + lambdav) ** 2)
  scaling = S_diag @ tf.transpose(V) @ \
            V_2 @ S_2_diag @ tf.transpose(V_2) @ \
            V @ S_diag

  # Generate new lables using the largest eigenvector.
  _, vecs = tf.linalg.eig(scaling)
  new_labels = tf.math.real(
      tf.einsum('ij,j->i', tf.cast(V @ S_diag, tf.complex64), vecs[-1])).numpy()
  # Create new labels and add some small amount of noise.
  final_y = new_labels > np.median(new_labels)
  noisy_y = (final_y ^ (np.random.uniform(size=final_y.shape) > 0.95))
  return noisy_y
```

```python
y_relabel = get_stilted_dataset(S_pqk, V_pqk, S_original, V_original)
y_train_new, y_test_new = y_relabel[:N_TRAIN], y_relabel[N_TRAIN:]
```

## 3. Comparing models
Now that you have prepared your dataset it is time to compare model performance. You will create two small feedforward neural networks and compare performance when they are given access to the PQK features found in `x_train_pqk`.

### 3.1 Create PQK enhanced model
Using standard `tf.keras` library features you can now create and a train a model on the `x_train_pqk` and `y_train_new` datapoints:

```python
#docs_infra: no_execute
def create_pqk_model():
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(32, activation='sigmoid', input_shape=[len(qubits) * 3,]))
    model.add(tf.keras.layers.Dense(16, activation='sigmoid'))
    model.add(tf.keras.layers.Dense(1))
    return model

pqk_model = create_pqk_model()
pqk_model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(learning_rate=0.003),
              metrics=['accuracy'])

pqk_model.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense (Dense)                (None, 32)                1088
    _________________________________________________________________
    dense_1 (Dense)              (None, 16)                528
    _________________________________________________________________
    dense_2 (Dense)              (None, 1)                 17
    =================================================================
    Total params: 1,633
    Trainable params: 1,633
    Non-trainable params: 0
    _________________________________________________________________

```python
#docs_infra: no_execute
pqk_history = pqk_model.fit(tf.reshape(x_train_pqk, [N_TRAIN, -1]),
          y_train_new,
          batch_size=32,
          epochs=1000,
          verbose=0,
          validation_data=(tf.reshape(x_test_pqk, [N_TEST, -1]), y_test_new))
```

### 3.2 Create a classical model
Similar to the code above you can now also create a classical model that doesn't have access to the PQK features in your stilted dataset. This model can be trained using `x_train` and `y_label_new`.

```python
#docs_infra: no_execute
def create_fair_classical_model():
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(32, activation='sigmoid', input_shape=[DATASET_DIM,]))
    model.add(tf.keras.layers.Dense(16, activation='sigmoid'))
    model.add(tf.keras.layers.Dense(1))
    return model

model = create_fair_classical_model()
model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(learning_rate=0.03),
              metrics=['accuracy'])

model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_3 (Dense)              (None, 32)                352
    _________________________________________________________________
    dense_4 (Dense)              (None, 16)                528
    _________________________________________________________________
    dense_5 (Dense)              (None, 1)                 17
    =================================================================
    Total params: 897
    Trainable params: 897
    Non-trainable params: 0
    _________________________________________________________________

```python
#docs_infra: no_execute
classical_history = model.fit(x_train,
          y_train_new,
          batch_size=32,
          epochs=1000,
          verbose=0,
          validation_data=(x_test, y_test_new))
```

### 3.3 Compare performance
Now that you have trained the two models you can quickly plot the performance gaps in the validation data between the two. Typically both models will achieve > 0.9 accuaracy on the training data. However on the validation data it becomes clear that only the information found in the PQK features is enough to make the model generalize well to unseen instances.

```python
#docs_infra: no_execute
plt.figure(figsize=(10,5))
plt.plot(classical_history.history['accuracy'], label='accuracy_classical')
plt.plot(classical_history.history['val_accuracy'], label='val_accuracy_classical')
plt.plot(pqk_history.history['accuracy'], label='accuracy_quantum')
plt.plot(pqk_history.history['val_accuracy'], label='val_accuracy_quantum')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
```

Success: You have engineered a stilted quantum dataset that can intentionally defeat classical models in a fair (but contrived) setting. Try comparing results using other types of classical models. The next step is to try and see if you can find new and interesting datasets that can defeat classical models without needing to engineer them yourself!

## 4. Important conclusions

There are several important conclusions you can draw from this and the [MNIST](https://www.tensorflow.org/quantum/tutorials/mnist) experiments:

1. It's very unlikely that the quantum models of today will beat classical model performance on classical data. Especially on today's classical datasets that can have upwards of a million datapoints.

2. Just because the data might come from a hard to classically simulate quantum circuit, doesn't necessarily make the data hard to learn for a classical model.

3. Datasets (ultimately quantum in nature) that are easy for quantum models to learn and hard for classical models to learn do exist, regardless of model architecture or training algorithms used.


# Python and the Jupyter Notebook

*This notebook is based on teaching material written by the Jupyter team, see https://jupyter.org*

We will be teaching via the Jupyter Notebook. The Jupyter Notebook is an **interactive computing environment** that enables users to author notebook documents that include:
- Live code
- Interactive widgets
- Plots
- Narrative text, Equations
- Images, Videos, and anything else a modern browser-based app can display

These documents provide a **complete and self-contained record of a computation** that can be [converted to various formats](https://nbconvert.readthedocs.io/en/latest/) and shared with others.

The Jupyter Notebook combines three components:

* **The notebook web application**: An interactive browser-based web application for writing and running code interactively and authoring notebook documents.
* **Kernels**: Separate processes started by the notebook web application that runs users' code in a given language and returns output back to the notebook web application. The kernel also handles things like computations for interactive widgets, tab completion and introspection.
* **Notebook documents**: Self-contained documents that contain a representation of all content visible in the notebook web application, like this one for instance. Each notebook document, when opened, has its own kernel.

<!--- Taken from the notebook repo,  notebook/docs/source/examples/Notebook --->

## Notebook web application

The notebook web application enables users to:

* **Edit code in the browser**, with automatic syntax highlighting, indentation, and tab completion/introspection.
* **Run code from the browser**, with the results of computations attached to the code which generated them.
* See the results of computations with **rich media representations**, such as HTML, LaTeX, PNG, SVG, PDF, etc.
* Create and use **interactive JavaScript widgets**, which bind interactive user interface controls and visualizations to reactive kernel side computations.
* Author **narrative text** using the [Markdown](https://daringfireball.net/projects/markdown/) markup language.
* Include mathematical equations using **LaTeX syntax in Markdown**, which are rendered in-browser by [MathJax](https://www.mathjax.org/).

We will show you how to do all of these.

## Kernels

The Notebook allows code to be run in a range of different programming languages.  For each notebook document that a user opens, the web application starts a kernel that runs the code for that notebook. Each kernel is capable of running code in a single programming language and there are kernels available in many different languages, including:

* Python(https://github.com/ipython/ipython)
* Julia (https://github.com/JuliaLang/IJulia.jl)
* R (https://github.com/IRkernel/IRkernel)
* Ruby (https://github.com/minrk/iruby)
* Haskell (https://github.com/gibiansky/IHaskell)
* Scala (https://github.com/Bridgewater/scala-notebook)
* node.js (https://gist.github.com/Carreau/4279371)
* Go (https://github.com/takluyver/igo)

The default kernel runs Python code.

## Notebook documents I

Notebook documents contain the **inputs and outputs** of an interactive session as well as **narrative text** that accompanies the code but is not meant for execution. **Rich output** generated by running code, including HTML, images, video, and plots, is embedded in the notebook, which makes it a complete and self-contained record of a computation.

When you run the notebook web application on your computer, notebook documents are just **files on your local filesystem with a `.ipynb` extension**. This allows you to use familiar workflows for organizing your notebooks into folders and sharing them with others.

Notebooks consist of a **linear sequence of cells**. You move to the next by keying SHIFT-enter. There are a few basic cell types, the most important of which are:

* **Code cells:** Input and output of live code that is run in the kernel

```python
# This is a python code cell, note the In[] in the left margin
# Code execution displays the result below the cell

2 + 3
```

* **Markdown cells:** Narrative text in markdown, can contain embedded LaTeX equations:

### Source

```
\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}
```

### Display

$\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}$

Internally, notebook documents are **[JSON](https://en.wikipedia.org/wiki/JSON) data** with **binary values [base64](http://en.wikipedia.org/wiki/Base64)** encoded. This allows them to be **read and manipulated programmatically** by any programming language. Because JSON is a text format, notebook documents are version control friendly.

**Notebooks can be exported** to different static formats including HTML, reStructeredText, LaTeX, PDF, and slide shows ([reveal.js](http://lab.hakim.se/reveal-js/)) using Jupyter's [nbconvert]((https://nbconvert.readthedocs.io/en/latest/) utility.

Furthermore, any notebook document available from a **public URL on or GitHub can be shared** via [nbviewer](http://nbviewer.jupyter.org) (and GitHub renders them natively). The `nbviewer` site loads the notebook document from the URL and renders it as a static web page. The resulting web page may thus be shared with others **without their needing to install the Jupyter Notebook**.

## The Notebook dashboard

When you first start the notebook server `jupyter-notebook`, your browser will open to the notebook dashboard in the current working directory (to change that, use `jupyter-notebook --notebook-dir`). The dashboard serves as a home page for the notebook.

The top of the notebook list displays clickable breadcrumbs of the current directory. By clicking on these breadcrumbs or on sub-directories in the notebook list, you can navigate your file system.

To create a new notebook, click on the "New" button at the top of the list and select a kernel from the dropdown (as seen below).  Which kernels are listed depend on what's installed on the server.  Some of the kernels in the screenshot below may not exist as an option to you.

Notebooks and files can be uploaded to the current directory by dragging a notebook file onto the notebook list or by the "click here" text above the list.

The notebook list shows green "Running" text and a green notebook icon next to running notebooks (as seen below). Notebooks remain running until you explicitly shut them down; closing the notebook's page is not sufficient.

To shutdown, delete, duplicate, or rename a notebook check the checkbox next to it and an array of controls will appear at the top of the notebook list (as seen below).  You can also use the same operations on directories and files when applicable.

---

# Exercise

Let's get started.
- Make sure you have cloned or downloaded the course material, as described in the [index](../index.ipynb).
- Open the jupyter notebook, using the directory containing today's course material.
- Open a notebook, and then close it (i.e., close that browser tab). Is the notebook running still? What does "running" mean?
- Reopen the notebook you closed. Create a markdown cell with a heading, and a code cell that prints 'hello world'
- Give your notebook a name (click on the text next to the Jupyter logo, right at the top) and save it.

---

## Using the notebook

- Editing vs navigation mode.  Key bindings, press "ESC" (to exit editing mode) and "h".
- Restart the kernel by pressing zero (`0`) twice, or by using the menu: kernel -> restart.

### Shell commands

```python
!ls
```

```python
!head -n 5 stockholm_td_adj.dat
```

### Command Pallette

For everything else, there's the command palette.  Press Ctrl-Shift-F or Ctrl-Shift-P.

## IPython

### Benchmarking

In IPython (the Python kernel for Jupyter), you can time operations by using `%time`.

```python
%%timeit
x = range(500000)
y = [t**2 for t in x]
```

### Reading documentation

Type a function name, followed by `?` to get its docstring.  You can also access this information while editing with SHIFT-TAB.

```python
import numpy as np
np.loadtxt?
```

```python
np.loadtxt(
```

```python
np.loadtxt??
```

```python
np.load*?
```

## Widgets

```python
import sys
!{sys.executable} -m pip install ipywidgets
```

For older notebooks, you may also need to call:

```python
!jupyter nbextension enable --py --sys-prefix widgetsnbextension  # can be skipped for notebook version 5.3 and above
```

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

```python
from ipywidgets import interact

@interact(n=(-1, 5))
def plot_poly(n=1):
    t = np.linspace(-2 * np.pi, 2 * np.pi, 500)
    plt.plot(t**n)
```