# Data structures

Author(s): Jukka Aho

**Abstract**: Description of basis data structures. In this notebook the concepts of `Increment`, `TimeStep`, `DiscreteField`, `ContinuousField`, `FieldSet`, `SpatialBasis` and `TemporalBasis` are intoduced. By combining these atomic structures one is able to form finite elements and interpolate it's fields in time and spatial domain.

- `Increment` is the most atomic structure. It's a vector-like object with 1 dimension. Each element in `Increment` can be scalar, vector or tensor (2 or 4 order). It's easy to extend `Increment` to have other data types too.
- `TimeStep` is container for increments in certain time $t$.
- `DefaultDiscreteField` is container for timesteps for a single field.
- `FieldSet` is container for all fields.
- `DefaultContinuousField` can have continuous field variables.

## Revision history

### 2015-06-14
- Initial version.

### 2015-09-25
- Complete rewrite. The main ideas proposed in earlier version didn't work.

### 2015-10-29
- Third iteration.

### 2015-11-03
- Starts to be ready. Introduced symbolic fields.

### Data fields on elements

Typical element structure so far:

    type MyElement <: Element
        connectivity :: Array{Int, 1}  # describes how dofs of this element is connected to another elements in global level
        basis :: Basis  # describes how to interpolate fields
        fields :: ???
    end

- Fields must be interpolable, in space $\mathbb{C}^n \times \mathbb{R}$, i.e. $f(\boldsymbol{\xi}, t) = \sum_i \phi_i(\boldsymbol{\xi}) f_i(t) = \sum_i \phi_i(\boldsymbol{\xi}) \sum_j \varphi_j(t) f_{ij}$, where $f_{ij}$ is scalar, tensor or vector defined in element area $e$ by some basis functions $\phi(\boldsymbol{\xi})$ and $\varphi(t)$. Parameter $t$ is normally considered as "time" and $\xi$ is dimensionless coordinate. Parameter $t$ has not necessarily to be time, it could be for example angle $\alpha \in [-2\pi, 2\pi]$ or similar.
- We store mainly three fields, scalar field, vector field, tensor field. Field may or may not be dependent from parameters $\xi$ or $t$.
- $t$ is discretized to several steps $\{t_0, t_1, \ldots, t_n\}$. Each discrete time $t_i$ may contain several iterations until convergence. We want to save and get access to all of this data if needed.
- So in practice we have a set of fields $f(\boldsymbol{\xi})$ over time domain $t$. Typically some fields, like Geometry, is introduced only in time $t_0$. Some other fields, like boundary load, may be "active" only on some time $\hat{t} \subset t$. Some care must be taken of how to extrapolate field variables.
- In the simplest case (simple nonlinear quasistatic analysis), we have for instance $t \in [0, 1]$ where boundary conditions are set in $t_0$ and load is set in $t_1$. We may use adaptive strategies to shorten time if convergence issues araises.

### Creating discrete fields

Here's the strategy in short: each non-linear iteration is `Increment`, what is a vector-like object containing data. Elements of `Increment` can be scalars, vectors or tensors. `Increment` belongs to `TimeStep`. `TimeStep` contains one or more increments. Then we have `Field` which contains all timesteps. And finally we have `FieldSet` which contains all fields.

    FieldSet -> Field -> TimeStep -> Increment -> data

for example,

    FieldSet -> Field -> TimeStep -> Increment -> data
    FieldSet -> "temperature" -> 0.0 -> 1 -> [1,2,3,4]
                                        2 -> [2,3,4,5]
    FieldSet -> "temperature" -> 1.0 -> 1 -> [3,4,5,6]
                                        2 -> [5,6,7,8]

and so on. Here we create a `FieldSet` containing one field `temperature` which contains two `TimeStep`s, two `Increment`s in both of them.

```julia
using JuliaFEM: Increment, TimeStep, Field, FieldSet, DefaultDiscreteField

fs = FieldSet()
i1 = Increment([1, 2, 3])
i2 = Increment([2, 3, 4])
t1 = TimeStep(1.0, Increment[i1, i2])
i3 = Increment([2, 3, 4])
i4 = Increment([3, 4, 5])
t2 = TimeStep(2.0, Increment[i3, i4])
f1 = DefaultDiscreteField(TimeStep[t1, t2])
fs["temperature"] = f1
fs
```

    Dict{ASCIIString,JuliaFEM.Field} with 1 entry:
      "temperature" => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(1.0,JuliaFE…

Accessing last increment of last timestep of temperature can be done in the following way:

```julia
fs["temperature"][2][2]
```

    3-element JuliaFEM.Increment{Int64}:
     3
     4
     5

Because the model is deep and quite heavy to type some "shortcuts" are provided, but keep in mind that everything is there in place. Here's the shortened version:

```julia
fs = FieldSet()  # create empty fieldset
fs["temperature"] = [1, 2, 3, 4]  # create temperature field with time t=0
first(fs["temperature"]), last(fs["temperature"])  # pick first and last timestep of temperature
```

    ([1,2,3,4],[1,2,3,4])

This way we can easily create discrete scalar, vector and tensor fields.

```julia
fs2 = FieldSet()
fs2["constant scalar field"] = 1
fs2["scalar field"] = [1, 2, 3, 4]
fs2["vector field"] = reshape(collect(1:8), 2, 4)
fs2["second order tensor field"] = reshape(collect(1:3*3*4), 3, 3, 4)
fs2["fourth order tensor field"] = reshape(collect(1:3*3*3*3*4), 3, 3, 3, 3, 4)
fs2
```

    Dict{ASCIIString,JuliaFEM.Field} with 5 entries:
      "fourth order tensor fi… => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(…
      "constant scalar field"  => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(…
      "vector field"           => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(…
      "scalar field"           => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(…
      "second order tensor fi… => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(…

Or even more compactly:

```julia
FieldSet("geometry" => [1, 2, 3, 4], "temperature" => [0, 0, 0, 0], "density" => 7850)
```

    Dict{ASCIIString,JuliaFEM.Field} with 3 entries:
      "density"     => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(0.0,JuliaFE…
      "geometry"    => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(0.0,JuliaFE…
      "temperature" => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(0.0,JuliaFE…

By default using this "fast typing" fields are defined at time $t=0.0$.

```julia
fs2["vector field"][end].time  # pick last timestep of field "vector field"
```

    0.0

Creating new field with several time steps defined can also be done compactly. Each tuple has time and increment data.

```julia
t0 = TimeStep(0.0, Increment([1, 2, 3, 4]))
t1 = TimeStep(0.5, Increment([2, 3, 4, 5]))
t2 = TimeStep(1.0, Increment([1, 1, 1, 1]))
fs2["time series"] = [t0, t1, t2]

info("time on last timestep: $(fs2["time series"][end].time)")
```

    INFO: time on last timestep: 1.0

Or even without explicitly expressing time. In that case time step size is 1 second starting from 0.

```julia
fs2["time series 2"] = [1, 2], [2, 3], [3, 4]
fs2
```

Adding another field, with different time.

```julia
T0 = first(fs["temperature"])  # pick first timestep (or to be spesific, last increment of first timestep)
T1 = Increment(T0 + 1)  # create new increment from old one
timestep = TimeStep(1.0, T1)  # create new timestep at t=1.0
push!(fs["temperature"], timestep)  # push to field
```

    2-element Array{JuliaFEM.TimeStep,1}:
     JuliaFEM.TimeStep(0.0,JuliaFEM.Increment[[1,2,3,4]])
     JuliaFEM.TimeStep(1.0,JuliaFEM.Increment[[2,3,4,5]])

Normal stuff like dot product works:

```julia
S1 = Increment([1, 2, 3])
S2 = Increment([2, 3, 4])
dot(S1, S2), dot([1, 2, 3], S2), dot(S1, [2, 3, 4])
```

    (20,20,20)

```julia
1/2*(S1 + S2)
```

    3-element Array{Float64,1}:
     1.5
     2.5
     3.5

```julia
dot([1, 2], Increment[S1, S2])
```

    3-element Array{Int64,1}:
      5
      8
     11

Creating empty `Increment`:

```julia
f = zeros(Increment, 2, 4)
```

    4-element JuliaFEM.Increment{Array{Float64,1}}:
     [0.0,0.0]
     [0.0,0.0]
     [0.0,0.0]
     [0.0,0.0]

Create similar increment with new data:

```julia
g = similar(f, ones(8))
```

    4-element JuliaFEM.Increment{Array{Float64,1}}:
     [1.0,1.0]
     [1.0,1.0]
     [1.0,1.0]
     [1.0,1.0]

### Creating continuous (and discrete) fields

In previous section the concept of discrete fields was demonstrated. `DefaultDiscreteField` is subtype of `DiscreteField` which is subtype of `Field`:

```julia
using JuliaFEM: DefaultDiscreteField, DiscreteField, Field
DefaultDiscreteField <: DiscreteField <: Field
```

    true

There exists another type of fields too: `ContinuousField`s. Like the name already suggests it stores continuous time and spatial domain and it can be used to write custom fields. It needs to be callable. In this example `ContinuousField` is created which returns 1x4 dimensional array defined in $\boldsymbol\xi \in [-1,1]^2, t\in[0,1]$:

```julia
using JuliaFEM: FieldSet, DefaultContinuousField, ContinuousField
```

```julia
type MyContinuousField <: ContinuousField
end
function Base.call(f::MyContinuousField, xi::Vector, time::Number)
    time/4*[(1-xi[1])*(1-xi[2]) (1+xi[1])*(1-xi[2]) (1+xi[1])*(1+xi[2]) (1-xi[1])*(1+xi[2])]
end
f = MyContinuousField()
```

    MyContinuousField()

```julia
f([0.0, 0.0], 1.0)
```

    1x4 Array{Float64,2}:
     0.25  0.25  0.25  0.25

```julia
fs = FieldSet()
fs["basis"] = MyContinuousField()
fs
```

    Dict{ASCIIString,JuliaFEM.Field} with 1 entry:
      "basis" => MyContinuousField()

```julia
fs["basis"]([0.0, 0.0], 1.0)
```

    1x4 Array{Float64,2}:
     0.25  0.25  0.25  0.25

Again we have a shortcut to quickly define continuous fields:

```julia
fs["continuous field"] = (xi, t) -> xi[1]*xi[2]*t
fs["continuous field"]([1.0, 2.0], 3.0)
```

    6.0

Naturally we can pass another fields or even fieldsets to continuous field to make fields depend from each other. Here's another example, where `ContinuousField` takes another field and operates it with some function.

```julia
type MyContinuousField2 <: ContinuousField
    basis :: Function
    discrete_field :: DiscreteField
end
function Base.call(field::MyContinuousField2, xi::Vector, time::Number=1.0)
    data = last(field.discrete_field) # get the last timestep last increment
    basis = field.basis(xi) # evaluate basis at point ξ.
    sum([basis[i]*data[i] for i=1:length(data)]) # sum results
end
```

    call (generic function with 1264 methods)

Now we create two fields, one is discrete and another is continuous taking discrete field as input argument:

```julia
fs = FieldSet()
fs["discrete field"] = [1, 2, 3, 4]
basis(xi) = 1/4*[
    (1-xi[1])*(1-xi[2]),
    (1+xi[1])*(1-xi[2]),
    (1+xi[1])*(1+xi[2]),
    (1-xi[1])*(1+xi[2])]
fs["continuous field"] = MyContinuousField2(basis, fs["discrete field"])
```

    MyContinuousField2(basis,JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(0.0,JuliaFEM.Increment[[1,2,3,4]])]))

Results:

```julia
fs["continuous field"]([0.0, 0.0], 1.0), last(fs["discrete field"])
```

    (2.5,[1,2,3,4])

Next we add another discrete field and see that continuous field "connecting" to discrete field updates accordingly:

```julia
T0 = last(fs["discrete field"])
push!(fs["discrete field"], TimeStep(1.0, Increment(T0 + 1.0)))  # push to field
fs
```

    Dict{ASCIIString,JuliaFEM.Field} with 2 entries:
      "continuous field" => MyContinuousField2(basis,JuliaFEM.DefaultDiscreteField(…
      "discrete field"   => JuliaFEM.DefaultDiscreteField([JuliaFEM.TimeStep(0.0,Ju…

Updated results, notice how the value of continuous field changes according to the update of discrete field.

```julia
fs["continuous field"]([0.0, 0.0], 1.0), last(fs["discrete field"])
```

    (3.5,[2.0,3.0,4.0,5.0])

What we just did is that we actually interpolated discrete field using continuous functions. We evaluated discrete field using bilinear basis at midpoint of "element":

```julia
1/4*(1+2+3+4), 1/4*(2+3+4+5)
```

    (2.5,3.5)

From continuous field we can also go back to discrete fields, if needed. Here we evaluate continuous field in four discrete points, let's call them to Gauss quadrature points.

```julia
using JuliaFEM: DiscreteField
type MyDiscreteField <: DiscreteField
    discrete_points :: Vector
    continuous_field :: ContinuousField
end
Base.length(field::MyDiscreteField) = length(field.discrete_points)
Base.endof(field::MyDiscreteField) = endof(field.discrete_points)
Base.last(field::MyDiscreteField) = Float64[field[i] for i=1:length(field)]
function Base.getindex(field::MyDiscreteField, idx::Int64)
    field.continuous_field(field.discrete_points[idx])
end
```

    getindex (generic function with 126 methods)

```julia
discrete_points = 1.0/sqrt(3.0)*Vector[[-1, -1], [1, -1], [1, 1], [-1, 1]]
fs["discrete field 2"] = MyDiscreteField(discrete_points, fs["continuous field"])
last(fs["discrete field 2"])
```

    4-element Array{Float64,1}:
     2.75598
     3.08932
     3.91068
     4.24402

Basically summing the above values together we have (almost) done numerical integration over element area.  By using these two simple concepts we are able to construct rest of the results.

## Interpolation

In earlier discussion the concepts of `DiscreteField` and `ContinuousField` were introduced, so that now we can define discrete set of values and continuous functions. It has also been shown how fields can depend from each other such a way that we can interpolate continuous field from discrete field and vice versa.

This motivates us to create continuous fields that are interpolated from discrete fields using some polynomial basis. By thinking this way interpolation is nothing more than just an application of the earlier results already shown.

Some fields has special meaning in JuliaFEM. Typically one needs to define at least discrete field `geometry` and continuous field `basis` so that basic interpolation is working on element.

Interpolation of fields works of course both in time and spatial dimension. Here's a simple example showing the main concept.

```julia
using JuliaFEM: Basis
```

Yep, it's just continuous field:

```julia
Basis <: ContinuousField
```

    true

Basis + DiscreteField = Interpolation:

```julia
basis(xi) = 1/4*[(1-xi[1])*(1-xi[2])   (1+xi[1])*(1-xi[2])   (1+xi[1])*(1+xi[2])   (1-xi[1])*(1+xi[2])]
dbasis(xi) = 1/4*[
    -(1-xi[2])    (1-xi[2])   (1+xi[2])  -(1+xi[2])
    -(1-xi[1])   -(1+xi[1])   (1+xi[1])   (1-xi[1])]

N = Basis(basis, dbasis)
```

    JuliaFEM.Basis(basis,dbasis)

```julia
N([0.0, 0.0])
```

    1x4 Array{Float64,2}:
     0.25  0.25  0.25  0.25

### Interpolation in time domain

To interpolate in time domain, call `DiscreteField` given time. Result is an `Increment` interpolated to that time. Here we interpolate the position of particle moving $y = \frac{1}{2}t^2$ at time $t=1.0$.

```julia
t = linspace(0, 2, 5)
y = 1/2*t.^2
t, y
```

    (linspace(0.0,2.0,5),[0.0,0.125,0.5,1.125,2.0])

```julia
timesteps = TimeStep[]
for (ti, yi) in zip(t, y)
    increment = Increment(yi)
    push!(timesteps, TimeStep(ti, increment))
end

#fs["particle position"] = t, 1/2*t.^2
pos = Field(timesteps)
pos(1.0)
```

    1-element JuliaFEM.Increment{Float64}:
     0.5

It's also possible to take time derivatives using finite difference approximation. To do so, call `Field` with time and additional argument `Val{:diff}`. Again, same example:

```julia
velocity = pos(1.0, Val{:diff})
```

    1-element Array{Float64,1}:
     1.0

### Interpolation in spatial domain

To interpolate in spatial domain, call `Basis` with `Increment` and coordinate $\boldsymbol\xi$. Increments to interpolate are the latest ones in each time step by default. Result depends from the content of the field. If it is scalar field, result will be scalar, if it's vector the result will be vector and so on.

Let's have a $\Omega = \left[0,1\right]\times\left[0,1\right] \in \mathbb{R}^2$ domain with a displacement field
\begin{equation}
\mathbf{u}\left(X_1, X_2\right)=t\begin{bmatrix}X_{1}\left(X_{2}+1\right)\\
X_{1}\left(4X_{2}-1\right)
\end{bmatrix}.
\end{equation}

Gradient is
\begin{equation}
\mbox{grad}\left(\mathbf{u}\right)=t\begin{bmatrix}X_{2}+1 & X_{1}\\
4X_{2}-1 & 4X_{1}
\end{bmatrix}
\end{equation}

We look for a displacement in center point of the domain at time $t=1.0$:

```julia
X = Field(Vector{Float64}[[0.0,0.0], [1.0,0.0], [1.0,1.0], [0.0,1.0]])
u = Field(
    (0.5, Vector{Float64}[[0.0, 0.0], [0.5, -0.5], [1.0, 1.5], [0.0, 0.0]]),
    (1.5, Vector{Float64}[[0.0, 0.0], [1.5, -1.5], [3.0, 4.5], [0.0, 0.0]]));
```

```julia
# evaluate fields in time t=1.0 -> Increments
X_increment = X(1.0)
u_increment = u(1.0)
X_increment, u_increment
```

    ([[0.0,0.0],[1.0,0.0],[1.0,1.0],[0.0,1.0]],Any[[0.0,0.0],[1.0,-1.0],[2.0,3.0],[0.0,0.0]])

```julia
# evaluate geometry + displacement at midpoint, basically x = X + u
N(X_increment, [0.0, 0.0]) + N(u_increment, [0.0, 0.0])
```

    2-element Array{Float64,1}:
     1.25
     1.0

Calculating gradients is also possible, for scalar or vector fields, if passing additional argument `Val{:grad}`. Here's an example of gradient of vector field, i.e, $u_{i,j} = \frac{\partial u_i}{\partial X_j}$:

```julia
N = Basis(basis, dbasis)
gradu = N(X_increment, u_increment, [0.0, 0.0], Val{:grad})
```

    2x2 Array{Float64,2}:
     1.5  0.5
     1.0  2.0

One can of course combine the above results. In this case field must be first interpolated in time domain and after that in spatial domain. So calculating time derivative of small strain tensor $\epsilon$ is basically:

```julia
du = u(1.0, Val{:diff})
grad_du = N(X_increment, du, [0.0, 0.0], Val{:grad})
strain_rate = 1/2*(grad_du + grad_du')
```

    2x2 Array{Float64,2}:
     1.5   0.75
     0.75  2.0

### Symbolic fields

There's still one kind of field not yet introduced, namely *symbolic fields*:

```julia
u = Field("displacement")
```

    JuliaFEM.SymbolicField("displacement")

The idea is to write equation in expression form using tiny symbolic interface and evaluate equations afterwards. Main benefits:

- possibility to simplify equations symbolically
- prettier syntax
- "lazy evaluation" gives possibilities to optimize performance.

Here's a preliminary example how this could work:

```julia
# in future this will be something like this:
# u = Field("displacement")
# eps = 1/2*(grad(u) + grad(u)')
# strain_rate = diff(eps)
```

```julia
# now this is
expr = JuliaFEM.Expression("1/2*(grad(diff(displacement)) + grad(diff(displacement))')")
print(expr.expr)
```

    (1 / 2) * (grad(diff(displacement)) + grad(diff(displacement))')

Evaluation of expression: call `Basis` with `FieldSet`, $\boldsymbol\xi$ and time as arguments:

```julia
X = Field(Vector[[0.0, 0.0], [1.0, 0.0], [1.0, 1.0], [0.0, 1.0]])
u = Field(
    (0.5, Vector[[0.0, 0.0], [0.5, -0.5], [1.0, 1.5], [0.0, 0.0]]),
    (1.5, Vector[[0.0, 0.0], [1.5, -1.5], [3.0, 4.5], [0.0, 0.0]]))
fs = FieldSet()
fs["geometry"] = X
fs["displacement"] = u
xi = [0.0, 0.0]
time = 1.0
result = N(expr, fs, xi, time);
```

    INFO: expression = (1 / 2) * (grad(diff(displacement)) + grad(diff(displacement))')
    INFO: expression = 1 / 2
    INFO: new expression: 1 / 2
    INFO: expression = grad(diff(displacement)) + grad(diff(displacement))'
    INFO: expression = grad(diff(displacement))
    INFO: inside gradient operator
    INFO: gradient args: Any[:grad,:(diff(displacement))]
    INFO: expression inside gradient
    INFO: grad: evaluate field diff(displacement)
    INFO: taking time derivative of displacement
    INFO: expression = diff(displacement)
    INFO: inside diff operator
    INFO: data = Any[[0.0,0.0],[1.0,-1.0],[2.0,3.0],[0.0,0.0]]
    INFO: evaluate geometry
    INFO: evaluate gradient
    INFO: expression = grad(diff(displacement))'
    INFO: transpose
    INFO: expression = transpose(grad(diff(displacement)))
    INFO: expression = grad(diff(displacement))
    INFO: inside gradient operator
    INFO: gradient args: Any[:grad,:(diff(displacement))]
    INFO: expression inside gradient
    INFO: evaluate gradient
    INFO: new expression: transpose([1.5 0.5
     1.0 2.0])
    INFO: new expression: [1.5 0.5
     1.0 2.0] + transpose([1.5 0.5
     1.0 2.0])
    INFO: new expression: (1 / 2) * ([1.5 0.5
     1.0 2.0] + transpose([1.5 0.5
     1.0 2.0]))

```julia
result
```

    :((1 / 2) * (2x2 Array{Float64,2}:
     1.5  0.5
     1.0  2.0 + transpose(2x2 Array{Float64,2}:
     1.5  0.5
     1.0  2.0)))

What happened is that fields were automatically interpolated and substituted to the equation. The result is `Expr` and can be evaluated like usual:

```julia
strain_rate = eval(result)
```

    2x2 Array{Float64,2}:
     1.5   0.75
     0.75  2.0


# Tutorials summary

The goal of these workshops is to do numerical exercices using MAD-X to visualize transverse dynamics concepts from a different point of view.

**Friday 14th of January**
- Tutorial 1 - Part 1: INTRODUCTION TO THE TOOLS, SMALL NUMERICAL EXERCICES (All together)
- Tutorial 1 - Part 2: FODO + PARAMETRIC PLOTS (groups of 3 students 5min+25min+15min)
- Tutorial 1 - Part 3: FODO + DIPOLES + MATCHING (groups of 3 students 5min+25min+15min)

$\color{red}{\text{WEEKEND: homework exercice (to be delivered on Monday 17th!)}}$

**Monday 17th of January**
- Tutorial 2 - Part 1: CHROMATICITY AND SEXTUPOLES (groups of 3 students 5min+25min+15min)
- Tutorial 2 - Part 2: NON-LINEARITIES AND LARGE AMPLITUDES (groups of 3 students 5min+25min+15min)
- Tutorial 2 - Part 3: TRANFER LINES (groups of 3 students 5min+25min+15min)

$\color{red}{\text{VERY IMPORTANT}}$: Save your jupyter-notebooks and download them to your computer after finishing the tutorials!! Otherwise your work will be lost!

$\color{red}{\text{Tutorial 1 and Tutorial 2 jupyter-notebooks (to be delivered as late on Friday 21st!)}}$ + the homework exercice will count for the final mark on the accelertor design workshop.

$\color{blue}{\text{Notes}}$:

- For most of the tutorials we will split in groups of 3 students. These groups will be kept also for the accelerator design workshops on the third and fouth weeks of the JUAS course.

- Tutors (Axel, Guido, Tessa, Davide and Nuria) will go around to help you and answer questions!

- We suggested that you try to solve the exercice by yourself but when you need help please ask your team mates or the tutor and use the screen share option.

# Tutorial 1: Part 1

Objectives:

- [Get familiar with the jupyter-notebooks.](#introjupyter)

- [Get familiar with the basic python commands that we will use during the tutorials.](#intropython)

- [How do we compute the optics of a lattice?](#firstexercice)

- [Get familiar with python commands to send the information to the MAD-X code and review the the main MADX blocks for optics computtions.](#intromadx)

# Jupyter notebook

- OPEN a jupyter notebook: https://github.com/fusterma/JUAS2022 and click on the binder buttom for the Tutorial1_Part1.

- EDIT/INSERT/DELATE a cell.

- RUN (press bottom on the top command line or press CAPS+ENTER).

- SAVE and DOWNLOAD: after we finish one tutorial you need to SAVE and DOWNLOAD the jupyter-notebook into your PC. Otherwise your progres will be lost!

        - SAVE BUTTOM on the upper TAB and FILE -> DOWNLOAD. f you want to load the jupyter-notebook in another moment go to FILE -> OPEN.

```python
print("Hello world")
```

    Hello world

# Basic Pyhton commands

The python universe has a huge number of libraries that extend the capabilities of python.
Nearly all of these are open source. For this workshop we will use the following:

```python
############################
# Import special libraries #
############################
#For plotting
from matplotlib import pyplot as plt
# For numerical calulations (np.max(), np.min(), np.mean()...)
import numpy as np
# For symbolic computation (solving algebra problems)
import sympy as sp
# For structuring the data, visualization of tables and data manipultion
import pandas as pd
# Library that allows us to use the MAD-X models
from cpymad.madx import Madx
# Plot display
%matplotlib notebook
```

If you want to learn more abot python: https://www.youtube.com/watch?v=kqtD5dpn9C8, https://www.kaggle.com/learn/python

More about the cpymad library: http://hibtc.github.io/cpymad/getting-started

# Scalars, arrays and matrices in Python

```python
# Scalar
a=20
b=30
print(a*b)
```

    600

```python
# Arrays and matrices
sp.Matrix([1,2,3,4]) # 1D array
```

$\displaystyle \left[\begin{matrix}1\\2\\3\\4\end{matrix}\right]$

```python
sp.Matrix([[1,2],[3,4]]) # 2x2 matrix
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
A=sp.Matrix([[1,2],[3,4]])
B=sp.Matrix([[1,2],[3,4]])
A+B
```

$\displaystyle \left[\begin{matrix}2 & 4\\6 & 8\end{matrix}\right]$

```python
A*B
```

$\displaystyle \left[\begin{matrix}7 & 10\\15 & 22\end{matrix}\right]$

# Plots in Python

```python
# Plot
%matplotlib notebook
x=[0,1,2,3,4,5,6,7,8,9,10]
y=[0,1,2,3,4,5,6,7,8,9,10]
plt.plot(x,y,'.-b',label='test')
plt.legend()
plt.grid()
plt.xlabel('s [m]')
plt.ylabel('x[m]')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'x[m]')

# Exercice 1: How do we compute the linear optics of a lattice?

- I want to motivate first the use of optics codes such as MAD-X and at the same time illustrate the logic behind them.

- Most of the methods implmented in MAD-X as for example the TWISS method are based on matrix multiplications where first and second order transport matrices are used to get the TWISS functions and the closed orbit of your machine.

# FODO cell
- Compute the linear optics functions of a FODO cell which is the simplier combination of quadrupoles required to focuse the beam in both, vertical and horizontal planes (no dispersion, no coupling, no chromticity...).

# Thin lens approximation (f>>lq)

- To do some first estimations analytically one uses the thin lense approximation.

```python
# How do we compute the one-turn transfer matrix of the FODO?
K = sp.Symbol("K", positive = True)
Lq = sp.Symbol("Lq", positive = True)
Ld=sp.Symbol("Ld", positive = True)

Mfoc=sp.Matrix([[1,0],[-K*Lq,1]])
Mdefoc=sp.Matrix([[1,0],[K*Lq,1]])
Mdrift=sp.Matrix([[1,Ld],[0,1]])
```

```python
#################################################
# Transport matrix of half focusing quadrupople #
#################################################
Mfoc
```

$\displaystyle \left[\begin{matrix}1 & 0\\- K Lq & 1\end{matrix}\right]$

```python
##############################################
# Transport matrix of defocusing quadrupople #
##############################################
Mdefoc
```

$\displaystyle \left[\begin{matrix}1 & 0\\K Lq & 1\end{matrix}\right]$

```python
##############################################
# Transport matrix of defocusing quadrupople #
##############################################
Mdrift
```

$\displaystyle \left[\begin{matrix}1 & Ld\\0 & 1\end{matrix}\right]$

```python
M=Mfoc*Mdrift*Mdefoc*Mdrift
M=sp.simplify(M)
M
```

$\displaystyle \left[\begin{matrix}K Ld Lq + 1 & Ld \left(K Ld Lq + 2\right)\\- K^{2} Ld Lq^{2} & - K^{2} Ld^{2} Lq^{2} - K Ld Lq + 1\end{matrix}\right]$

```python
M_thin = M.subs(K, 1/(200*5)).subs(Lq, 5).subs(Ld, 45) # units K in m-2, Lq and Ld in m
M_thin
```

$\displaystyle \left[\begin{matrix}1.225 & 100.125\\-0.001125 & 0.724375\end{matrix}\right]$

It is clear that the exact solution of the prticle motion has to be calculted in full detail. But using some approximations we can make the first steps easier and estimate the order of magnitud of some magnetic properties of our lattice.

# What can we do with the  transfer matrix?
   - This matrix describes the optical properties of the lattice and defines the beam parameters.
   - We can propagate the phase space coordinates of a particle with a given set of initial coordinates.
   - We can compute the periodic solution TWISS functions.
   - We can make parametric plots.

```python
# Tranfer matrix
R11, R12, R21, R22 = sp.symbols('R11,R12,R21,R22')
Mt=sp.Matrix([[R11,R12],[R21,R22]])
Mt
```

$\displaystyle \left[\begin{matrix}R_{11} & R_{12}\\R_{21} & R_{22}\end{matrix}\right]$

```python
# In case of periodic conditions in the accelerator there is nother way to describe the particles trjectories.
# Periodic solution oner-turn-transfer matrix in terms of twiss functions
a, b, g, m = sp.symbols(r'\alpha,\beta, \gamma,\mu')
M=sp.Matrix([[sp.cos(m)+a*sp.sin(m),b*sp.sin(m)],[-g*sp.sin(m),sp.cos(m)-a*sp.sin(m)]])
M
```

$\displaystyle \left[\begin{matrix}\alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)} & \beta \sin{\left(\mu \right)}\\- \gamma \sin{\left(\mu \right)} & - \alpha \sin{\left(\mu \right)} + \cos{\left(\mu \right)}\end{matrix}\right]$

```python
print("Mt=Mone-turn-matrix")
```

    Mt=Mone-turn-matrix

```python
#For the phase advance we use the stability condition: Trace(M) < 2
sp.Eq(sp.cos(m)-(R11+R22)/2)
```

    /Users/nuria/opt/anaconda3/lib/python3.8/site-packages/sympy/core/relational.py:495: SymPyDeprecationWarning:

    Eq(expr) with rhs default to 0 has been deprecated since SymPy 1.5.
    Use Eq(expr, 0) instead. See
    https://github.com/sympy/sympy/issues/16587 for more info.

      SymPyDeprecationWarning(

$\displaystyle - \frac{R_{11}}{2} - \frac{R_{22}}{2} + \cos{\left(\mu \right)} = 0$

```python
#For the beta function we use the R12 matrix element
sp.Eq(R12/sp.sin(m)-b)
```

$\displaystyle \frac{R_{12}}{\sin{\left(\mu \right)}} - \beta = 0$

```python
#For the alfa function we use the trace of the matrix also
sp.Eq((R11-R22)/(2*sp.sin(m))-a)
```

$\displaystyle - \alpha + \frac{R_{11} - R_{22}}{2 \sin{\left(\mu \right)}} = 0$

```python
#For the gamma we use the R21 element
sp.Eq((R21/sp.sin(m))+g)
```

$\displaystyle \frac{R_{21}}{\sin{\left(\mu \right)}} + \gamma = 0$

Once you have computed the periodic TWISS functions you can propagate them to any point in the machine using the transfer matrix of the TWISS functions from Transverse dynamics course.

```python
sp.Matrix([[R11**2, -2*R12*R11, R12**2],[-R11*R21,R12*R21+R22*R11, -R12*R22],[R21**2,-2*R22*R21, R22**2]])
```

$\displaystyle \left[\begin{matrix}R_{11}^{2} & - 2 R_{11} R_{12} & R_{12}^{2}\\- R_{11} R_{21} & R_{11} R_{22} + R_{12} R_{21} & - R_{12} R_{22}\\R_{21}^{2} & - 2 R_{21} R_{22} & R_{22}^{2}\end{matrix}\right]$

Using the periodic one-turn-matrix one can define some interesting relations between the TWISS parameters and the magnetic properties of the lattice.

```python
# Relation between the phase advance of the cell and K, Lcell, Lq
a, b, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(d*m/pi-2*sp.asin(K*Lq*Lc/4))
```

$\displaystyle \frac{\Delta \mu}{\pi} - 2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} = 0$

```python
#Add the interesting formula for the FODO cell of mu and bmax and bmin and make the parametric plots
x=np.arange(0,6,0.01)
y=2*np.arcsin(x/4)/np.pi
fig, ax1 = plt.subplots()
ax1.plot(x,y,'-')
ax1.set_ylabel("$\Delta \mu / \pi [rad]$", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
ax1.grid()
ax1.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <ipython-input-24-85ac7794e4a0>:3: RuntimeWarning: invalid value encountered in arcsin
      y=2*np.arcsin(x/4)/np.pi

    <IPython.core.display.Javascript object>

# What is the quadrupole strenght needed to match a FODO cell phase advance of 45$^\circ$ if the $L_{quad}$=5 m and $L_{cell}$=100 m?

```python
# Relation between the beta of the cell and K, Lcell, Lq
a, bmin, bmax, g, m, d, Lq, Lc, K, pi = sp.symbols(r'\alpha,\beta_{min}, \beta_{max}, \gamma,\mu, \Delta, L_{q}, L_{cell} K \pi')
sp.Eq(bmin/Lc-(1-(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
sp.Eq(bmax/Lc-(1+(K*Lq*Lc/4))/(sp.sin(2*sp.asin(K*Lq*Lc/4))))
```

    /Users/nuria/opt/anaconda3/lib/python3.8/site-packages/sympy/core/relational.py:495: SymPyDeprecationWarning:

    Eq(expr) with rhs default to 0 has been deprecated since SymPy 1.5.
    Use Eq(expr, 0) instead. See
    https://github.com/sympy/sympy/issues/16587 for more info.

      SymPyDeprecationWarning(

$\displaystyle - \frac{\frac{K L_{cell} L_{q}}{4} + 1}{\sin{\left(2 \operatorname{asin}{\left(\frac{K L_{cell} L_{q}}{4} \right)} \right)}} + \frac{\beta_{max}}{L_{cell}} = 0$

```python
x=np.arange(0.5,3.90,0.01)
betamax=(1+(x/4))/(np.sin(2*np.arcsin(x/4)))
betamin=(1-(x/4))/(np.sin(2*np.arcsin(x/4)))
fig, ax1 = plt.subplots()
ax1.plot(x,betamax,'-',label=r"$\beta_{max}/L_{cell}$")
ax1.plot(x,betamin,'-',label=r"$\beta_{min}/L_{cell}$")
ax1.set_ylabel("[-]", fontsize=16)
ax1.set_xlabel("$K*L_{quad}*L_{cell}$ [-]", fontsize=16)
plt.grid()
plt.legend()
plt.tick_params(axis='both', labelsize=16)
plt.tight_layout()
```

    <IPython.core.display.Javascript object>

#### Thick lense computation

```python
K = sp.Symbol("K")
Lq = sp.Symbol("Lq")
Ld= sp.Symbol("Ld")

Mfoc=sp.Matrix([[sp.cos(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq)],[-(sp.sqrt(K))*sp.sin(sp.sqrt(K)*Lq),sp.cos(sp.sqrt(K)*Lq)]])
Mdefoc=sp.Matrix([[sp.cosh(sp.sqrt(K)*Lq),1/(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq)],[(sp.sqrt(K))*sp.sinh(sp.sqrt(K)*Lq),sp.cosh(sp.sqrt(K)*Lq)]])
Mdrift=sp.Matrix([[1,Ld],[0,1]])
```

```python
#################################################
# Transport matrix of half focusing quadrupople #
#################################################
Mfoc
```

$\displaystyle \left[\begin{matrix}\cos{\left(\sqrt{K} Lq \right)} & \frac{\sin{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \sin{\left(\sqrt{K} Lq \right)} & \cos{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
##############################################
# Transport matrix of defocusing quadrupople #
##############################################
Mdefoc
```

$\displaystyle \left[\begin{matrix}\cosh{\left(\sqrt{K} Lq \right)} & \frac{\sinh{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\\sqrt{K} \sinh{\left(\sqrt{K} Lq \right)} & \cosh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
##############################################
# Transport matrix of defocusing quadrupople #
##############################################
Mdrift
```

$\displaystyle \left[\begin{matrix}1 & Ld\\0 & 1\end{matrix}\right]$

```python
M=Mfoc*Mdrift*Mdefoc*Mdrift
M=sp.simplify(M)
M
```

$\displaystyle \left[\begin{matrix}\left(\sqrt{K} Ld \cos{\left(\sqrt{K} Lq \right)} + \sin{\left(\sqrt{K} Lq \right)}\right) \sinh{\left(\sqrt{K} Lq \right)} + \cos{\left(\sqrt{K} Lq \right)} \cosh{\left(\sqrt{K} Lq \right)} & \sqrt{K} Ld^{2} \cos{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)} + Ld \sin{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)} + 2 Ld \cos{\left(\sqrt{K} Lq \right)} \cosh{\left(\sqrt{K} Lq \right)} + \frac{\sin{\left(\sqrt{K} Lq \right)} \cosh{\left(\sqrt{K} Lq \right)}}{\sqrt{K}} + \frac{\cos{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)}}{\sqrt{K}}\\- \sqrt{K} \left(\left(\sqrt{K} Ld \sin{\left(\sqrt{K} Lq \right)} - \cos{\left(\sqrt{K} Lq \right)}\right) \sinh{\left(\sqrt{K} Lq \right)} + \sin{\left(\sqrt{K} Lq \right)} \cosh{\left(\sqrt{K} Lq \right)}\right) & - 2 \sqrt{K} Ld \sin{\left(\sqrt{K} Lq \right)} \cosh{\left(\sqrt{K} Lq \right)} + \sqrt{K} Ld \cos{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)} - K Ld^{2} \sin{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)} - \sin{\left(\sqrt{K} Lq \right)} \sinh{\left(\sqrt{K} Lq \right)} + \cos{\left(\sqrt{K} Lq \right)} \cosh{\left(\sqrt{K} Lq \right)}\end{matrix}\right]$

```python
M_thick = M.subs(K, 1/(200*5)).subs(Lq, 5).subs(Ld,45) # units K in m-2, Lq and Ld in m
M_thick
```

$\displaystyle \left[\begin{matrix}1.24801597796155 & 111.155823182084\\-0.00120832527282141 & 0.693651062898885\end{matrix}\right]$

```python
M_thin
```

$\displaystyle \left[\begin{matrix}1.225 & 100.125\\-0.001125 & 0.724375\end{matrix}\right]$

In real world applications, lattices (including FODO) are not designed by hand but
dedicated software is used to do the design and simulation as for example MAD-X.

The TWISS in MADX it is based in matrix multiplications similar to what has been shown here however if you want something more complicated it isn't anymore that easy.

# Wha is next?

- Now we are going to do optics calculations using MAD-X TWISS command (handle thousands of elements, thin or thick lense options, and higher order contributions...).

- We will use the MATCHING MAD-X tool to compute the required magnetic properties for a desired TWISS functions.

- We will use MAD-X to visulize the impact of some propertie of the lattice on the TWISS and single prticle DYNAMICS.

# An introduction to MAD-X using the python interface

In this first part we are going to get familiar with MAD-X syntax.

For more information please refer to the [MAD-X online manual](http://cern.ch/madx/releases/last-rel/madxuguide.pdf).

```python
#Load the cpymad library
from cpymad.madx import Madx
```

```python
#Launching MAD-X
myMad = Madx(stdout=True)
```

      ++++++++++++++++++++++++++++++++++++++++++++
      +     MAD-X 5.07.00  (64 bit, Darwin)      +
      + Support: mad@cern.ch, http://cern.ch/mad +
      + Release   date: 2021.05.03               +
      + Execution date: 2021.12.05 12:04:38      +
      ++++++++++++++++++++++++++++++++++++++++++++

```python
#String that will be interpreted by MAD-X
myString='''
stop;
'''
```

```python
#Using the "input" method to send the commandas to the MAD-X class
myMad.input(myString);
```

      Number of warnings: 0

      ++++++++++++++++++++++++++++++++++++++++++++
      +          MAD-X finished normally         +
      ++++++++++++++++++++++++++++++++++++++++++++

With the 'stop;' instruction we exit from MAD-X, so, as done in the following cell we need to re-istantiate our MAD-X object with the
**myMad = Madx()** instruction.

---
It is a good practice to make header, please use '!' to comment the single line.

```python
# Define and print a value
myMad = Madx()
myString='''

!***************************************
! It is a good practice to make a header
!***************************************

a= 20;
value a;

'''
myMad.input(myString);
```

      ++++++++++++++++++++++++++++++++++++++++++++
      +     MAD-X 5.07.00  (64 bit, Darwin)      +
      + Support: mad@cern.ch, http://cern.ch/mad +
      + Release   date: 2021.05.03               +
      + Execution date: 2021.12.05 12:04:39      +
      ++++++++++++++++++++++++++++++++++++++++++++
    a                  =                 20 ;

---
Use the **help** keyword (very rudimental help)

```python
# To get information about the MAD-X methods (twiss, beam,match... ) use the commnd "help"
myString='''
help, twiss;
'''
myMad.input(myString);
```

    command: twiss  module: twiss
    parameter: betx   double value:                  0
    parameter: alfx   double value:                  0
    parameter: mux   double value:                  0
    parameter: bety   double value:                  0
    parameter: alfy   double value:                  0
    parameter: muy   double value:                  0
    parameter: x   double value:                  0
    parameter: px   double value:                  0
    parameter: y   double value:                  0
    parameter: py   double value:                  0
    parameter: t   double value:                  0
    parameter: pt   double value:                  0
    parameter: dx   double value:                  0
    parameter: dpx   double value:                  0
    parameter: dy   double value:                  0
    parameter: dpy   double value:                  0
    parameter: wx   double value:                  0
    parameter: phix   double value:                  0
    parameter: dmux   double value:                  0
    parameter: wy   double value:                  0
    parameter: phiy   double value:                  0
    parameter: dmuy   double value:                  0
    parameter: ddx   double value:                  0
    parameter: ddpx   double value:                  0
    parameter: ddy   double value:                  0
    parameter: ddpy   double value:                  0
    parameter: r11   double value:                  0
    parameter: r12   double value:                  0
    parameter: r21   double value:                  0
    parameter: r22   double value:                  0
    parameter: energy   double value:                  0
    parameter: chrom   logical: false
    parameter: file   string: (null)
    parameter: save   string: (null)
    parameter: table   string: (null)
    parameter: beta0   string: (null)
    parameter: re11   double value:                  1
    parameter: re12   double value:                  0
    parameter: re13   double value:                  0
    parameter: re14   double value:                  0
    parameter: re15   double value:                  0
    parameter: re16   double value:                  0
    parameter: re21   double value:                  0
    parameter: re22   double value:                  1
    parameter: re23   double value:                  0
    parameter: re24   double value:                  0
    parameter: re25   double value:                  0
    parameter: re26   double value:                  0
    parameter: re31   double value:                  0
    parameter: re32   double value:                  0
    parameter: re33   double value:                  1
    parameter: re34   double value:                  0
    parameter: re35   double value:                  0
    parameter: re36   double value:                  0
    parameter: re41   double value:                  0
    parameter: re42   double value:                  0
    parameter: re43   double value:                  0
    parameter: re44   double value:                  1
    parameter: re45   double value:                  0
    parameter: re46   double value:                  0
    parameter: re51   double value:                  0
    parameter: re52   double value:                  0
    parameter: re53   double value:                  0
    parameter: re54   double value:                  0
    parameter: re55   double value:                  1
    parameter: re56   double value:                  0
    parameter: re61   double value:                  0
    parameter: re62   double value:                  0
    parameter: re63   double value:                  0
    parameter: re64   double value:                  0
    parameter: re65   double value:                  0
    parameter: re66   double value:                  1
    parameter: centre   logical: false
    parameter: ripken   logical: false
    parameter: sectormap   logical: false
    parameter: sectoracc   logical: false
    parameter: sectorpure   logical: false
    parameter: sectortable   string: (null)
    parameter: sectorfile   string: sectormap
    parameter: rmatrix   logical: false
    parameter: sequence   string: (null)
    parameter: line   string: (null)
    parameter: range   string: #s/#e
    parameter: useorbit   string: default
    parameter: keeporbit   string: default
    parameter: tolerance   double value:              1e-06
    parameter: deltap   string: (null)
    parameter: eigenvector   logical: false
    parameter: eigenfile   string: eigenvectors
    parameter: tapering   logical: false
    parameter: clorb_tol   double value:             0.0001
    parameter: tap_itter   integer:       1000
    parameter: notable   logical: false

```python
print(list(myMad.elements))
```

    ['drift', 'rbend', 'sbend', 'matrix', 'quadrupole', 'sextupole', 'octupole', 'multipole', 'solenoid', 'rfcavity', 'elseparator', 'srotation', 'yrotation', 'xrotation', 'hkicker', 'kicker', 'vkicker', 'hmonitor', 'monitor', 'vmonitor', 'ecollimator', 'rcollimator', 'beambeam', 'instrument', 'marker', 'twcavity', 'thinwire', 'wire', 'slmonitor', 'blmonitor', 'imonitor', 'dipedge', 'changeref', 'translation', 'crabcavity', 'placeholder', 'tkicker', 'hacdipole', 'vacdipole', 'nllens', 'rfmultipole', 'collimator', 'changerefp0', 'sixmarker']

```python
myString='''
help, sbend;
'''
myMad.input(myString);
```

    command: sbend  module: element
    double value:              1e+20
    parameter: l   double value:                  0
    parameter: kmax   double value:                  0
    parameter: kmin   double value:                  0
    parameter: calib   double value:                  0
    parameter: polarity   double value:                  0
    parameter: angle   double value:                  0
    parameter: tilt   double value:                  0
    parameter: k0   double value:                  0
    parameter: k0s   double value:                  0
    parameter: k1   double value:                  0
    parameter: k1s   double value:                  0
    parameter: e1   double value:                  0
    parameter: e2   double value:                  0
    parameter: k2   double value:                  0
    parameter: k2s   double value:                  0
    parameter: h1   double value:                  0
    parameter: h2   double value:                  0
    parameter: hgap   double value:                  0
    parameter: fint   double value:                  0
    parameter: fintx   double value:                 -1
    parameter: k3   double value:                  0
    parameter: k3s   double value:                  0
    parameter: fcsr   double value:                  0
    parameter: mech_sep   double value:                  0
    parameter: v_pos   double value:                  0
    parameter: aperture   double array: 0.000000e+00
    parameter: aper_offset   double array: 0.000000e+00
    double array: 0.000000e+00 0.000000e+00 0.000000e+00
    parameter: aper_vx   double array: -1.000000e+00
    parameter: aper_vy   double array: -1.000000e+00
    parameter: knl   double array: 0.000000e+00
    parameter: ksl   double array: 0.000000e+00
    parameter: magnet   integer:          1
    parameter: slot_id   expression: none :: value:                  0
    integer:          0
    parameter: assembly_id   expression: none :: value:                  0
    integer:          0
    parameter: slice   integer:          1
    parameter: thick   logical: false
    parameter: model   integer:         -1
    parameter: method   integer:         -1
    parameter: exact   integer:         -1
    parameter: nst   integer:         -1
    parameter: apertype   string: circle
    parameter: from   string: (null)
    parameter: type   string: (null)
    parameter: fringe   integer:          0
    parameter: bend_fringe   logical: true
    parameter: kill_ent_fringe   logical: false
    parameter: kill_exi_fringe   logical: false
    parameter: dx   double value:                  0
    parameter: dy   double value:                  0
    parameter: ds   double value:                  0
    parameter: dtheta   double value:                  0
    parameter: dphi   double value:                  0
    parameter: dpsi   double value:                  0
    parameter: slice_straight   logical: false
    parameter: aper_tilt   double value:                  0
    parameter: comments   string: (null)

# Let's define the main ingredients to design a simple lattice and compute the optics in MAD-X!

# Definition of main parameters

```python
myString='''
! *********************************************************************
! Definition of parameters
! *********************************************************************

l_cell=100;
quadrupoleLenght=5;
f=200;
myK:=1/f/quadrupoleLenght;// m^-2

'''
myMad.input(myString);
```

# Definition of magets

```python
myString='''

! *********************************************************************
! Definition of magnets
! *********************************************************************
QF: quadrupole, L=quadrupoleLenght, K1:=myK;
QD: quadrupole, L=quadrupoleLenght, K1:=-myK;
'''
myMad.input(myString);
```

# Definition of sequence

```python
myString='''

! *********************************************************************
! Definition of sequence
! *********************************************************************
myCell:sequence, refer=entry, L=L_CELL;
quadrupole1: QF, at=0;
marker1: marker, at=25;
quadrupole2: QD, at=50;
endsequence;

'''
myMad.input(myString);
```

# Definition of the beam

```python
myString='''

! *********************************************************************
! Definition of beam
! *********************************************************************
beam, particle=proton, energy=2;

'''
myMad.input(myString);
```

# Activate sequence

```python
myString='''

! *********************************************************************
! Use of the sequence
! *********************************************************************
use, sequence=myCell;

'''
myMad.input(myString);
```

# Execute action

```python
myString='''

! *********************************************************************
! TWISS
! *********************************************************************
title, 'My first twiss';
twiss;

'''
```

# All together
Output:

-SUMM table

-TWISS table

```python
myMad = Madx(stdout=True)
myString='''
! *********************************************************************
! Definition of parameters
! *********************************************************************

l_cell=100;
quadrupoleLenght=5;
f=200;
myK:=1/f/quadrupoleLenght;// m^-2

! *********************************************************************
! Definition of magnets
! *********************************************************************
QF: quadrupole, L=quadrupoleLenght, K1:=myK;
QD: quadrupole, L=quadrupoleLenght, K1:=-myK;

! *********************************************************************
! Definition of sequence
! *********************************************************************
myCell:sequence, refer=entry, L=L_CELL;
quadrupole1: QF, at=0;
marker1: marker, at=25;
quadrupole2: QD, at=50;
endsequence;

! *********************************************************************
! Definition of beam
! *********************************************************************
beam, particle=proton, energy=2;

! *********************************************************************
! Use of the sequence
! *********************************************************************
use, sequence=myCell;

! *********************************************************************
! TWISS
! *********************************************************************
title, 'My first twiss';
select, flag=TWISS, column=keyword, name, s, betx, bety, x, y, dx, dy;
twiss, file=Tutorial1_Part1.madx;
plot, haxis=s, vaxis=betx,bety,dx,colour=100, title="test",file=Tutorial1_Part1;
'''
myMad.input(myString);
```

      ++++++++++++++++++++++++++++++++++++++++++++
      +     MAD-X 5.07.00  (64 bit, Darwin)      +
      + Support: mad@cern.ch, http://cern.ch/mad +
      + Release   date: 2021.05.03               +
      + Execution date: 2021.12.05 12:04:45      +
      ++++++++++++++++++++++++++++++++++++++++++++
    enter Twiss module

    iteration:   1 error:   0.000000E+00 deltap:   0.000000E+00
    orbit:   0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00

    ++++++ table: summ

                length             orbit5               alfa            gammatr
                   100                 -0                  0                  0

                    q1                dq1            betxmax              dxmax
         0.03853349451     -0.04384718845        463.6232883                  0

                 dxrms             xcomax             xcorms                 q2
                     0                  0                  0      0.03853349451

                   dq2            betymax              dymax              dyrms
        -0.04384718845        463.6232883                  0                  0

                ycomax             ycorms             deltap            synch_1
                     0                  0                  0                  0

               synch_2            synch_3            synch_4            synch_5
                     0                  0                  0                  0

               synch_6            synch_8             nflips
                     0                  0                  0
    Plot - default table plotted: twiss

     GXPLOT-X11  1.50 initialized

     plot number =            1

# Accessing the data

```python
#######################
#Using MAD-X commands #
#######################

myString='''
value, table(SUMM,Q1);
value, table(SUMM,betymax);
'''
myMad.input(myString);
```

    table( summ q1 )   =      0.03853349451 ;
    table( summ betymax ) =        463.6232883 ;

```python
myString='''
value, table(TWISS,MYCELL$END,betx);
value, table(TWISS,MYCELL$END,bety);
'''
myMad.input(myString);
```

    table( twiss mycell$end betx ) =        463.6232883 ;
    table( twiss mycell$end bety ) =        369.7791616 ;

# Using python pandas library

For advance users...

Pandas dataframe are very convenient, have a look in https://pandas.pydata.org/Pandas_Cheat_Sheet.pdf.

```python
# Using another method from cpymad "table.twiss.dframe()"
myDF=myMad.table.twiss.dframe()
```

```python
myDF
```

      name
      keyword
      s
      betx
      alfx
      mux
      bety
      alfy
      muy
      x
      ...
      sig54
      sig55
      sig56
      sig61
      sig62
      sig63
      sig64
      sig65
      sig66
      n1

      #s
      mycell$start:1
      marker
      0.0
      463.623288
      -1.156109
      0.000000
      369.779162
      0.929316
      0.000000
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole1
      quadrupole1:1
      quadrupole
      5.0
      463.623288
      1.156109
      0.001709
      369.779162
      -0.929316
      0.002161
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_0[0]
      drift_0:0
      drift
      25.0
      419.394867
      1.055312
      0.008930
      408.967742
      -1.030113
      0.010350
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      marker1
      marker1:1
      marker
      25.0
      419.394867
      1.055312
      0.008930
      408.967742
      -1.030113
      0.010350
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_1[0]
      drift_1:0
      drift
      50.0
      369.779162
      0.929316
      0.019041
      463.623288
      -1.156109
      0.019493
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      quadrupole2
      quadrupole2:1
      quadrupole
      55.0
      369.779162
      -0.929316
      0.021202
      463.623288
      1.156109
      0.021202
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      drift_2[0]
      drift_2:0
      drift
      100.0
      463.623288
      -1.156109
      0.038533
      369.779162
      0.929316
      0.038533
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      #e
      mycell$end:1
      marker
      100.0
      463.623288
      -1.156109
      0.038533
      369.779162
      0.929316
      0.038533
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

8 rows × 256 columns

```python
myDF[['name','s','betx','bety','alfx','alfy']]
```

      name
      s
      betx
      bety
      alfx
      alfy

      #s
      mycell$start:1
      0.0
      463.623288
      369.779162
      -1.156109
      0.929316

      quadrupole1
      quadrupole1:1
      5.0
      463.623288
      369.779162
      1.156109
      -0.929316

      drift_0[0]
      drift_0:0
      25.0
      419.394867
      408.967742
      1.055312
      -1.030113

      marker1
      marker1:1
      25.0
      419.394867
      408.967742
      1.055312
      -1.030113

      drift_1[0]
      drift_1:0
      50.0
      369.779162
      463.623288
      0.929316
      -1.156109

      quadrupole2
      quadrupole2:1
      55.0
      369.779162
      463.623288
      -0.929316
      1.156109

      drift_2[0]
      drift_2:0
      100.0
      463.623288
      369.779162
      -1.156109
      0.929316

      #e
      mycell$end:1
      100.0
      463.623288
      369.779162
      -1.156109
      0.929316

```python
myDF["s"]
```

    #s               0.0
    quadrupole1      5.0
    drift_0[0]      25.0
    marker1         25.0
    drift_1[0]      50.0
    quadrupole2     55.0
    drift_2[0]     100.0
    #e             100.0
    Name: s, dtype: float64

```python
myDF["betx"]
```

    #s             463.623288
    quadrupole1    463.623288
    drift_0[0]     419.394867
    marker1        419.394867
    drift_1[0]     369.779162
    quadrupole2    369.779162
    drift_2[0]     463.623288
    #e             463.623288
    Name: betx, dtype: float64

# Basic plot

```python
#Plot
plt.plot(myDF['s'],myDF['betx'],'.-b',label='$\\beta_x$')
plt.plot(myDF['s'],myDF['bety'],'.-r',label='$\\beta_y$')
#Labels of the plot
plt.xlabel('s [m]')
plt.ylabel('[m]')
#Legend and grid
plt.legend(loc='best')
plt.grid()
```

    [<matplotlib.lines.Line2D at 0x7f87ca9138b0>]

    [<matplotlib.lines.Line2D at 0x7f87ca913af0>]

    Text(0.5, 66.60000000000001, 's [m]')

    Text(56.93750000000002, 0.5, '[m]')

    <matplotlib.legend.Legend at 0x7f87ca8d8c70>

# For reference

---
This is an example to get familiar with the use of the physical constants and the formatting of the output. Have a look on the difference.

```python
myString='''
a=pi;
value a;
set, format="22.20e";
value a;
'''
myMad.input(myString);
```

    ++++++ info: a redefined
    a                  =        3.141592654 ;
    a                  = 3.14159265358979311600e+00 ;

#
This is an example to get familiar with if and deferred expression. Please note the after the block delimited with {...} the ; can be omitted. Pay attention to circular call!

```python
myString='''
if (1==1){
option, echo=false, info=true;
a=pi;
b:=a;
c=a;
value a;
value b;
value c;
a=CLIGHT*cos(a);
value a;
value b;
value c;}
! BEWARE of circular call!
!a:=a+1;
! When evaluating you will get a fatal error
! value a;
option, echo=true, info=true;
'''
myMad.input(myString);
```

    ++++++ info: a redefined
    a                  =        3.141592654 ;
    b                  =        3.141592654 ;
    c                  =        3.141592654 ;
    ++++++ info: a redefined
    a                  =         -299792458 ;
    b                  =         -299792458 ;
    c                  =        3.141592654 ;

# ---
This is an example to get familiar with **while** and **macros** loops.

```python
myString='''
a(myvariable1,myvariable2): macro = {
value, myvariable1;
value, myvariable1*myvariable2;
}

N=1;
while (N<10){
exec, a(N,N);
N=N+1;
}
'''
myMad.input(myString);
```

    n                  =                  1 ;
    n*n                =                  1 ;
    ++++++ info: n redefined
    n                  =                  2 ;
    n*n                =                  4 ;
    ++++++ info: n redefined
    n                  =                  3 ;
    n*n                =                  9 ;
    ++++++ info: n redefined
    n                  =                  4 ;
    n*n                =                 16 ;
    ++++++ info: n redefined
    n                  =                  5 ;
    n*n                =                 25 ;
    ++++++ info: n redefined
    n                  =                  6 ;
    n*n                =                 36 ;
    ++++++ info: n redefined
    n                  =                  7 ;
    n*n                =                 49 ;
    ++++++ info: n redefined
    n                  =                  8 ;
    n*n                =                 64 ;
    ++++++ info: n redefined
    n                  =                  9 ;
    n*n                =                 81 ;
    ++++++ info: n redefined

---
### List of functions
In MAD-X the following functions are available

- SQRT(x) square root,
- LOG(x) natural logarithm,
- LOG10(x) logarithm base 10,
- EXP(x) exponential,
- SIN(x) trigonometric sine,
- COS(x) trigonometric cosine,
- TAN(x) trigonometric tangent,
- ASIN(x) arc sine,
- ACOS(x) arc cosine,
- ATAN(x) arc tangent,
- SINH(x) hyperbolic sine,
- COSH(x) hyperbolic cosine,
- TANH(x) hyperbolic tangent,
- SINC(x) cardinal sine function,
- ABS(x) absolute value,
- ERF(x) Gauss error,
- ERFC(x) complementary error,
- FLOOR(x) floor, largest previous integer,
- CEIL(x) ceiling, smallest next integer,
- ROUND(x) round, closest integer,
- FRAC(x) fractional part of number,
- RANF() random number, uniformly distributed in [0,1],
- GAUSS() random number, gaussian distribution with unit standard deviation,
- TGAUSS(x) random number, gaussian distribution with unit standard deviation, truncated at x standard deviations;

---
### List of physical constant

| MAD-X name  | symbol  |  value |unit|
|:-:|:-:|:-:|:-:|
|PI| π |4 * atan(1)| 1|
|TWOPI|2π| 2 * PI| 1|
|DEGRAD| 180/π |180 / PI| deg/rad|
|RADDEG| π/180 |PI / 180 |rad/deg|
|E| e |exp(1) |1|
|EMASS| me |0.510998928e−3| GeV|
|PMASS| mp |0.938272046| GeV|
|NMASS| u |0.931494061| GeV|
|MUMASS| mµ| 0.1056583715 |GeV|
|CLIGHT| c| 299792458| m/s|
|QELECT| e| 1.602176565e−19| A.s|
|HBAR| ¯h| 6.58211928e−25| MeV.s|
|ERAD| re| 2.8179403267e−15| m|
|PRAD| re(me/mp)| ERAD*EMASS/PMASS| m|

# Inheritance and -> operator


```python
import viz_model_comparison
```

Now, we get to the heart of our analysis. We will compare the gamma distribution model to the two event distribution model in modeling catastrophe times, and we will examine how different concentrations of tubulin affect catastrophe times. In the past two sections, we used the [first dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_time_to_catastrophe_dic_tidy.csv), and now we move on to the [second dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv), which contains catastrophe times for five different tubulin concentrations––$7\mu M$, $9\mu M$, $10\mu M$, $12\mu M$, and $14\mu M$. Let's first do some exploratory data analysis, and we start with an ECDF plot, as seen in Figure 4a. All Figure 4 plots were generated using the [viz_explore_concentration_dataset](https://liuvictoria.github.io/sleepingbeauty/software/viz_explore_concentration_dataset.py) script.

[Figure 4a](https://liuvictoria.github.io/sleepingbeauty/figures/Fig4a.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_explore_concentration_dataset.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

We see that for higher concentrations, the time to catastrophe increases, and this is pretty consistent as we increase the concentration. This trend is especially noticeable for smaller catastrophe times, especially below the $60^{th}$ percentile. This increase might be explained by a shift in equilibrium dynamics as the concentration increases. Thinking about microtubule formation, increased tubulin availability will drive microtubule formation to use up free tubulin, shifting the equilibrium toward the right; thinking about microtubule catastrophe, more free tubulin will create pressure against creating more free tubulin, which may slow down catastrophe times. Interestingly, this trend is harder to see at longer catastrophe times, since it is conceivable that some arrivals will take a very long time, which is within the realm of Poisson arrival processes. These long arrivals can happen at any concentration level, thus obfuscating the trend at longer catastrophe times.

Zooming in on the graphs of individual concentrations, we can also see that there are inflection points for each concentration. This is comforting, since both the gamma model and two-event story have inflection points. Interestingly, the distributions for $9\mu M$ and $10\mu M$ have noticeable bimodal distributions. This may be problematic since neither our two-story nor our gamma distribution are particularly bi-modal. We can further explore this dataset by looking at a strip box-plot, as seen in Figure 4b.

[Figure 4b](https://liuvictoria.github.io/sleepingbeauty/figures/Fig4b.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_explore_concentration_dataset.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

We can clearly see that the medians catastrophe times increase with concentration. We can also see the bimodal distribution for $9\mu M$ and $10\mu M$. Notably, this plot also suggests that the number of data points for each concentration is different––there appear to be way more samples for $12 \mu M$ compared to $14 \mu M$.

Now, we can start doing some model comparison to see whether the gamma or two-event story is better matched to our data. As a reminder, the gamma distribution is parametrized by $\alpha$, the number of arrivals, and $\beta$, the rate of arrival. The standard catastrophe time is given by $\frac{\alpha}{\beta}$, so we would expect to see either increased $\alpha$, decreased $\beta$, or a combination of both, as the concentration increases. The two event story is parametrized by $\beta_1$ and $\beta_2$, where the MLE has $\beta_1 = \beta_2$. Thus, the standard catastrophe time is characterized by $\frac{2}{beta_1}$, and we would expect $\beta_1$ to decrease as the concentration increases. Let's start our model comparison by reminding ourselves of the MLEs our parameters, which we calculated previously. Table 1 can be generated using the [viz_model_comparison.py](https://liuvictoria.github.io/sleepingbeauty/software/viz_model_comparison.py) module.

```python
df_mle = viz_model_comparison.create_df_mle()
df_mle = df_mle.style.set_caption("Table 1: MLEs of two distribuions at different concentrations")
df_mle
```

Table 1: MLEs of two distribuions at different concentrations

        	        7 uM	        9 uM	        10 uM	        12 uM	        14 uM

                        alpha
                        2.443910
                        2.679864
                        3.210835
                        2.915277
                        3.361502

                        beta
                        0.007550
                        0.008779
                        0.009030
                        0.007661
                        0.007175

                        beta1
                        0.006179
                        0.006552
                        0.005625
                        0.005255
                        0.004269

                        beta2
                        0.006179
                        0.006552
                        0.005625
                        0.005256
                        0.004269

Now let's do some visual analysis. Using the [viz_model_comparison.py](https://liuvictoria.github.io/sleepingbeauty/software/viz_model_comparison.py) script, we generate all plots of Figure 5. First, we examine a predictive ECDF (Figure 5a), where the coral line represents the ECDF of the observed data, and the gray line represents the predictive ECDF of the generative model. The $68$% and $95$% confidence intervals of the predictive ECDF are shaded in dark gray and light gray, respectively. The PNGs rendered below will only show the distributions for $12 \mu M$ because the differences are more accentuated for $12 \mu M$. For a graphical comparison of all of them, please click on the HTML link underneath each PNG image.

[Figure 5a](https://liuvictoria.github.io/sleepingbeauty/figures/Fig5a.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_model_comparison.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

We see that the observed data generally lies within the $95$% confidence interval of the predictive ECDF of the gamma model, although there are systematic deviations; for shorter catastrophe times (i.e. between $200$ and $400$ seconds), the model appears to underestimate the time, while for longer catastrophe times (i.e. between $400$ and $800$ seconds), the model appears to overestimate the time. We see the same trend with the two event model, although the mis-estimation is more egregious, with the observed data frequently falling outside of the $95$% confidence interval. For now, it seems like gamma model is winning out.

We can also look at a Q-Q plot of the observed catastrophe times and bootstrap samples from the generative distribution, parametrized by the MLEs. This can be seen in Figure 5b. The catastrophe times from the generative model are plotted on the y-axis, and the observed catastrophe times are on the x-axis. Notably, the coral region shows the $95$% confidence interval for the quantiles, and the gray line represents the ideal plot, where the quantiles are the same between the observed the observed and model-generated data.

[Figure 5b](https://liuvictoria.github.io/sleepingbeauty/figures/Fig5b.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_model_comparison.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

Interestingly, neither model looks particularly great at modeling the entire distribution of catastrophe times. However, we note that the gamma model appears to fit the data well for catastrophe times under $1000$ seconds, while the two event model veers out of the $95$% confidence interval (in both directions) under $1000$ seconds. Earlier in our exploratory data analysis, we saw that there are few catastrophe times over $1000$ seconds, and gamma's bad job at modeling for large catastrophe times may simply be due to less time points for those outliers. We are also primarily concerned with times below $1000$ seconds, because that is where we really see the concentration effects on the catastrophe time. Thus, the Q-Q plots also favor the gamma model at $12 \mu M$.

For our last graphical comparison, we plot the difference between the predictive ECDF and the observed data. The coral line represents the difference between the percentile of the observed data and the predictive ECDF, at a given catastrophe time. The dark gray and light gray regions represent the $68$% and $95$% confidence region of the predictive ECDF, respectively.

[Figure 5c](https://liuvictoria.github.io/sleepingbeauty/figures/Fig5c.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_model_comparison.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

This visual is probably the most striking out of all three. It is quite evident that the difference does not fall within the $95$% envelope of the predictive ECDF for the two-event model for values below $1000$ seconds, and there is a systematic bias for different values of catastrophe time. On the other hand, while there is also some systematic bias for the gamma model, the difference tends to fall within the $95$% confidence interval for times less than $1000$ seconds. At this point, we're fairly convinced that the gamma model is the model of choice, just by looking at these graphical analyses. Nevertheless, let's confirm our belief using the Akaike information criterion. For a set of parameters $\theta$ with MLE $\theta^*$ and a model with log-likelihood $\ell(\theta;\text{data})$, the AIC is given by

\begin{align}
\text{AIC} = -2\ell(\theta^*;\text{data}) + 2p,
\end{align}

where $p$ is the number of free parameters in a model. Recall that the AIC value can be *very* "loosely interpreted as an estimate of a quantity related to the distance between the true generative distribution and the model distribution," so a better model would have a lower AIC value. For both the gamma and two event distribution, we have two parameters, and we've actually already calculated the log likelihoods when we were trying to maximize it to get our MLEs for our parameters. Then, the Akaike weight of model $i$ in a collection of models is

\begin{align}
w_i = \frac{\mathrm{e}^{-(\text{AIC}_i - \text{AIC}_\mathrm{max})/2}}{\sum_j\mathrm{e}^{-(\text{AIC}_j - \text{AIC}_\mathrm{max})/2}}.
\end{align}

Since we are taking the negative exponent of the AIC value, the model with the greater AIC weight is the better model, with $1$ being the sum of all weights. We display the MLEs, log likelihoods, AIC values, and AIC weights in Table 2. Note that we are displaying this across all concentrations, rather than just for $12 \mu M$. This table can be generated using [viz_concentration_effects.py](https://liuvictoria.github.io/sleepingbeauty/software/viz_concentration_effects.py).

```python
df_mle = viz_model_comparison.create_df_mle()
df_aic = viz_model_comparison.create_df_aic(df_mle)
df_aic = df_mle.style.set_caption("Table 2: MLEs, log likelihoods, AIC values and weights")
df_aic
```

Table 2: MLEs, log likelihoods, AIC values and weights

        	        7 uM	        9 uM	        10 uM	        12 uM	        14 uM

                        alpha
                        2.443910
                        2.679864
                        3.210835
                        2.915277
                        3.361502

                        beta
                        0.007550
                        0.008779
                        0.009030
                        0.007661
                        0.007175

                        beta1
                        0.006179
                        0.006552
                        0.005625
                        0.005255
                        0.004269

                        beta2
                        0.006179
                        0.006552
                        0.005625
                        0.005256
                        0.004269

                        log_like_gamma
                        -4013.447639
                        -1660.259101
                        -1477.790621
                        -4637.174646
                        -966.632060

                        log_like_two
                        -4074.742744
                        -1691.190428
                        -1513.546568
                        -4731.428737
                        -990.741370

                        AIC_gamma
                        8030.895278
                        3324.518203
                        2959.581241
                        9278.349291
                        1937.264119

                        AIC_two
                        8153.485487
                        3386.380857
                        3031.093137
                        9466.857475
                        1985.482740

                        w_gamma
                        1.000000
                        1.000000
                        1.000000
                        1.000000
                        1.000000

                        w_two
                        0.000000
                        0.000000
                        0.000000
                        0.000000
                        0.000000

Wow, it looks like the gamma distribution is the unambiguous winner here, with the gamma distribution consistently having the lower AIC value and taking up almost the entire AIC weight, up to $6$ decimal places. Using both graphical and numerical analyses, we've found that the gamma distribution outperforms the two story model in modeling catastrophe times. Moving on, let's see how the gamma parameters $\alpha$ and $\beta$ vary with concentration. Recall that the standard catastrophe time is given by $\frac{\alpha}{\beta}$, so we would expect to see either increased $\alpha$, decreased $\beta$, or a combination of both, as the concentration increases. For this analysis, we take bootstrap replicates of $\alpha$ and $\beta$ and look at the ECDFs and confidence regions of these parameters. The script to generate Figure 6 can be found in the [viz_concentration_effects.py](https://liuvictoria.github.io/sleepingbeauty/software/viz_concentration_effects.py) module. We start with Figure 6a, with an ECDF of $\alpha$.

[Figure 6a](https://liuvictoria.github.io/sleepingbeauty/figures/Fig6a.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_concentration_effects.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

Of note, the confidence intervals are actually plotted as well, but since we drew enough bootstrap samples, they basically blend into a line. Generally, it appears that as the concentration increases, $\alpha$ increases as well. There is a reversal of this trend between $10 \mu M$ and $12 \mu M$, but it goes back with $14 \mu M$. This general trend suggests that microtubule catastrophe requires more "arrivals" as concentration increases. Before delving into the biological ramifications for this finding, let's look at the ECDF for $\beta$, the rate, in Figure 6b.

[Figure 6b](https://liuvictoria.github.io/sleepingbeauty/figures/Fig6b.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_concentration_effects.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

In general, there does not appear to be any noticeable trend between the concentrations for $\beta$. For concentrations $7$ to $10 \mu M$, it appears that $\beta$ increases, which is actually against our original expection. Nevertheless, this can be explained by looking at $\frac{\alpha}{\beta}$, the characteristic catastrophe time. Earlier, we saw that $\alpha$ increases from $7$ to $10 \mu M$, and we can think of it as "increasing too much" when it is being numerically optimized, and that it overshoots the observed catastrophe time. As a result, $\beta$ is numerically optimized to increase as well, to compensate for that overshot. This can be confirmed for concentrations $12 \mu M$ and $14 \mu M$, which both had relatively mild increases in $\alpha$ (or, in $12 \mu M$'s case, an actual decrease in $\alpha$). Thus, their $\beta$ values are lower. There doesn't seem to be much of a correlation that persists throughout the concentrations, though. We can also view the confidence regions of $\alpha$ and $\beta$ at the same time, for different concentrations in Figure 6c. The figure is plotted with datashading, and the segments represent the $95$% confidence intervals of $\beta$ and $\alpha$ on the y-axis and x-axis, respectively.

[Figure 6c](https://liuvictoria.github.io/sleepingbeauty/figures/Fig6c.html) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_concentration_effects.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

Here, we see that with the exception of $12 \mu M$, increased concentration increases the number of arrivals $\alpha$, and there is not a particularly noticeable trend in the $\beta$ arrival rates. Given that microtubules polymerize faster at increased tubulin concentrations, the higher arrival rates may be indicative that more arrivals of biochemical processes are needed to catch up to the fast-growing microtubule, in order to initiate catastrophe. A possible interpretation is that the longer the microtubule is, the more events need to happen before catastrophe can happen, since catastrophe could need to catch up the length of the molecule.

To further explore how different values of $\alpha$ and $\beta$ affect the gamma distribution, and how this looks overlayed on the observed data, we have created a dashboard, which can be explored in the next section. Here is a PNG still image of the dashboard to get you excited. The green yellow line represents the gamma distribution, parametrized by the shown $\alpha$ and $\beta$ values.

[Explore Dashboard](https://liuvictoria.github.io/sleepingbeauty/dashboard) :star:
[script](https://liuvictoria.github.io/sleepingbeauty/software/viz_dashboard.py) :star:
[dataset](https://liuvictoria.github.io/sleepingbeauty/datasets/gardner_mt_catastrophe_only_tubulin.csv)

```python
#!jupyter nbconvert --to html --TemplateExporter.exclude_input=True analysis.ipynb
```

    [NbConvertApp] Converting notebook analysis.ipynb to html
    [NbConvertApp] Writing 601185 bytes to analysis.html

```python

```


```python
%%shell
set -e

#---------------------------------------------------#
JULIA_VERSION="1.7.1" # any version ≥ 0.7.0
JULIA_PACKAGES="IJulia BenchmarkTools PyCall PyPlot"
JULIA_PACKAGES_IF_GPU="CUDA"
JULIA_NUM_THREADS=4
#---------------------------------------------------#

if [ -n "$COLAB_GPU" ] && [ -z `which julia` ]; then
  # Install Julia
  JULIA_VER=`cut -d '.' -f -2 <<< "$JULIA_VERSION"`
  echo "Installing Julia $JULIA_VERSION on the current Colab Runtime..."
  BASE_URL="https://julialang-s3.julialang.org/bin/linux/x64"
  URL="$BASE_URL/$JULIA_VER/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
  wget -nv $URL -O /tmp/julia.tar.gz # -nv means "not verbose"
  tar -x -f /tmp/julia.tar.gz -C /usr/local --strip-components 1
  rm /tmp/julia.tar.gz

  # Install Packages
  if [ "$COLAB_GPU" = "1" ]; then
      JULIA_PACKAGES="$JULIA_PACKAGES $JULIA_PACKAGES_IF_GPU"
  fi
  for PKG in `echo $JULIA_PACKAGES`; do
    echo "Installing Julia package $PKG..."
    julia -e 'using Pkg; pkg"add '$PKG'; precompile;"' &> /dev/null
  done

  # Install kernel and rename it to "julia"
  echo "Installing IJulia kernel..."
  julia -e 'using IJulia; IJulia.installkernel("julia", env=Dict(
      "JULIA_NUM_THREADS"=>"'"$JULIA_NUM_THREADS"'"))'
  KERNEL_DIR=`julia -e "using IJulia; print(IJulia.kerneldir())"`
  KERNEL_NAME=`ls -d "$KERNEL_DIR"/julia*`
  mv -f $KERNEL_NAME "$KERNEL_DIR"/julia

  echo ''
  echo "Successfully installed `julia -v`!"
  echo "Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then"
  echo "jump to the 'Checking the Installation' section."
fi
```

    Installing Julia 1.7.1 on the current Colab Runtime...
    2022-03-04 14:22:20 URL:https://storage.googleapis.com/julialang2/bin/linux/x64/1.7/julia-1.7.1-linux-x86_64.tar.gz [123374573/123374573] -> "/tmp/julia.tar.gz" [1]
    Installing Julia package IJulia...
    Installing Julia package BenchmarkTools...
    Installing Julia package PyCall...
    Installing Julia package PyPlot...
    Installing IJulia kernel...
    [ Info: Installing julia kernelspec in /root/.local/share/jupyter/kernels/julia-1.7

    Successfully installed julia version 1.7.1!
    Please reload this page (press Ctrl+R, ⌘+R, or the F5 key) then
    jump to the 'Checking the Installation' section.

```python
!pip install julia
!pip install pysr
```

    Collecting julia
      Downloading julia-0.5.7-py2.py3-none-any.whl (67 kB)
         |████████████████████████████████| 67 kB 3.1 MB/s
    Installing collected packages: julia
    Successfully installed julia-0.5.7
    Collecting pysr
      Downloading pysr-0.7.9-py3-none-any.whl (37 kB)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from pysr) (1.3.5)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from pysr) (1.0.2)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pysr) (1.21.5)
    Requirement already satisfied: julia>=0.5.7 in /usr/local/lib/python3.7/dist-packages (from pysr) (0.5.7)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from pysr) (1.7.1)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->pysr) (2.8.2)
    Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->pysr) (2018.9)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->pysr) (1.15.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->pysr) (3.1.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->pysr) (1.1.0)
    Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->pysr) (1.4.1)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->pysr) (1.2.1)
    Installing collected packages: pysr
    Successfully installed pysr-0.7.9

```python
import numpy as np
import pysr
pysr.silence_julia_warning()
```

```python
def RG_decimation(config):
    #config=config.reshape(config.shape[0] * config.shape[1])
    half_root= int(np.sqrt(config.shape[0]/2))
    RG_config = np.zeros([int(config.shape[0]),int(config.shape[1]/2)])
    #print(config.shape,RG_config.shape)
    for i in range(config.shape[0]):
        for j in range(config.shape[1]):
            if ((i+j)%2==0): RG_config[int(i),int(j/2)] = config[i,j]
    #print("Initial",config)
    #print("RG:",RG_config.T)
    return RG_config

def RG_decimation_2(config):
    #config=config.reshape(config.shape[0] * config.shape[1])
    RG_config = np.zeros([int(config.shape[0]/2),int(config.shape[1])])
    #print(config.shape,RG_config.shape)
    for i in range(config.shape[0]):
        for j in range(config.shape[1]):
            if ((i)%2==0): RG_config[int(i/2),int(j)] = config[i,j]
    #print("Initial",config)
    #print("RG:",RG_config.T)
    return RG_config

```

```python
from numpy import genfromtxt
mydata = genfromtxt('scope_0_1.csv', delimiter=',')
X = mydata[:16999:10,0]
y = mydata[:16999:10,1]
print(X.shape)
print(X[-1],y[-1])
```

    (1700,)
    0.181 4.23

```python
import matplotlib.pyplot as plt

plt.plot(X,y)
```

```python
#X = 2 * np.random.randn(100, 5)
data_num=100
size=4
#X = 2 * np.random.randn(100, 5)
#X=np.random.choice([-1, 1], size=(data_num, size, size))
#print(X[0])
#y = np.cos(X[:, 3]) #+ X[:, 0] ** 2 - (0.5*X[:, 2])
'''y = np.zeros((data_num, size, int(size/2)))
print(y.shape)
for i in range (data_num):
  lat=RG_decimation(X[i])
  y[i] = lat'''
#model = PySRRegressor(binary_operators=["+", "-", "*", "/"])
```

    'y = np.zeros((data_num, size, int(size/2)))\nprint(y.shape)\nfor i in range (data_num):\n  lat=RG_decimation(X[i])\n  y[i] = lat'

```python
from pysr import PySRRegressor
model = PySRRegressor(
    niterations=5,
    binary_operators=["+", "-", "*", "/"],
    unary_operators=[
        "cos",
        "exp",
        "sin",
        "inv(x) = 1/x",  # Custom operator (julia syntax)
    ],
    model_selection="best",
    loss="loss(x, y) = (x - y)^2",  # Custom loss function (julia syntax)
    batching = True,
    batchSize = 100
)

```

```python
model.fit(X, y)
```

    PySRRegressor.equations = [
    	    pick     score                                           equation  \
    	0         0.000000                                          3.9570515
    	1         0.078606                              (3.8932116 / cos(x0))
    	2         0.100286                 (4.7688107 - cos(x0 * -3.5116084))
    	3         0.005336                (4.765724 - cos(sin(x0) * 3.58376))
    	4         0.456883         (x0 - ((cos(x0) * 17.72494) + -21.482964))
    	5         0.445390  (cos(cos((x0 - -0.5156789) / 0.1408587)) - -3....
    	6         1.235151  (exp(cos(cos((x0 / 0.13244656) + 0.55321264)))...
    	7   >>>>  1.710561  (cos(cos((x0 - 1.1303052) / 0.09556082) + 0.47...
    	8         0.231785  (cos(sin(cos((x0 - -1.3294694) / 0.080151066))...
    	9         0.057814  (cos(sin(cos(cos(x0 - 1.2529254) / 0.07762672)...
    	10        0.000615  (cos(sin(cos(cos(sin(x0) - 1.2528056) / -0.077...
    	11        0.907877  exp(cos(exp(cos(-0.7438716 - (sin(sin(x0)) / 0...

    	        loss  complexity
    	0   0.107596           1
    	1   0.084993           4
    	2   0.069546           6
    	3   0.069176           7
    	4   0.043806           8
    	5   0.028061           9
    	6   0.008160          10
    	7   0.001475          11
    	8   0.001170          12
    	9   0.001104          13
    	10  0.001103          14
    	11  0.000445          15
    ]

```python
print(model.equations.sympy_format[7])
```

    cos(cos(10.4645397559376*x0 - 11.828123701743) + 0.472653) + 3.2933707

```python
from matplotlib import pyplot as plt
plt.scatter(y[:], model(X)[:,0])
plt.xlabel('Truth')
plt.ylabel('Prediction')
plt.show()
```