# A study on spontaneous decay rate of an atom in presence of a nanofiber using BEM approach

In these notes, I calculate the Local Density of States (LDOS), or the imaginary part of the on-site Green's function and hence the modified spontaneous emission rate of an atom in presence of a nanofiber using the Boundary Element Method (BEM). The BEM code is from Prof. Alejandro Manjavacas's group.

This is an [IJulia notebook](https://github.com/JuliaLang/IJulia.jl), which provides a nice
browser-based [Jupyter](http://jupyter.org/) interface to the [Julia language](http://julialang.org/), a high-level dynamic language (similar to Matlab or Python+SciPy) for technical computing.  The notebook allows us to combine code and results in one place.

We are only manipulating the generated data from the simulation results in this notebook. As a brief recap of the simulation process, I have used a compiled BEM code by Alejandro's group written in C++ called `bem2D` on a cluster computing system. A configuration C++ code is defined in the file `scripts_ldos.cpp` which is in the same folder as `bem2D`. I compiled the script and put the generated executable into another folder called `p21`, for example, by
```
g++ scripts_ldos.cpp -lstdc++ -o ../p21/scripts_ldos
```
Then ran the executable and submitted the generated PBS script to the cluster system to run the simulation:
```
cd ../p21/
./scripts_ldos
qsub ldos_N_1_lam_894_eps_4_0_a_225_x_405_y_0_q_0_1.45_146.pbs
```
Notice that the name of the PBS script is automatically generated based on the configuration parameters for this simulation. The name will vary for different simulations. After running the script, I got a set of data files--one has the same name as the PBS script but with a `.dat` extension for the data table storing the calculated LDOS values; there are another two `*.dat` files for the geometry of boundary and dieletric function distribution. We will look into those data files in the following sections.

## 3D dielectric waveguide simulated in 2D

Just a little more detail on the simulation itself: By assuming the waveguide along z-axis or the light propagation direction is uniform, one can completely solve the boundary condition problem of dipole emission by simulating the field in one single layer of the xy cross section. The z-component of the field only adds in a phase factor. The data of the simulated result is stored in the /data/ folder of this repo.

We can read in this data as a matrix of numbers by the `readdlm` function in Julia with the `header=false` option meaning that it reads the first line as the beginning of the data entries without a list of strings describing each column.

Now, let's plot the results.  I'll use the [PyPlot](https://github.com/stevengj/PyPlot.jl) package in Julia, which is an interface to the sophisticated [matplotlib](http://matplotlib.org/) Python plotting library.   We'll plot three things:

* The waveguide structure in terms of $\epsilon$ and the interface boundary between two media.
* Plot the LDOS components for a fixed dipole position.
* Calculate the waveguide-modified spontaneous decay rate when the dipole varies its position outside of the waveguide.

### Plotting the boundary and index of refraction profile of the waveguide in the xy cross section

Our boundary points are meshed in the files `/data/geom_a_225.data` and `/data/geom_regions_a_225.data`, which positions where the equivalent charges and current sources to be computed in the BEM simulation. The waveguide has a circular cross-section of a radius $a=225nm$ ($nm$ is the unit of length) and a index of refraction of $n_1=n_{core}=1.4496$ for the waveguide material and $n_2=n_{clad}=1$ for the vacuum clad.

It is good to plot out the mesh of index of refraction in space and find out how good is the mesh resolution. This can be done by plotting out the output eps file in a simple data table format, which ends in `.dat`.

The following will first print out some of the data in order to figure out the physics meaning of the dimensions. They should contain the coordinate and index of refraction information for the simulation.

```julia
boundary = readdlm("data/geom_regions_a_225.dat", header=false);
boundarypoints = boundary[:,1:3]
```

    10201×3 Array{Float64,2}:
     -269.999  -269.998  1.0
     -269.999  -264.599  1.0
     -269.999  -259.199  1.0
     -269.999  -253.799  1.0
     -269.999  -248.399  1.0
     -269.999  -242.999  1.0
     -269.999  -237.599  1.0
     -269.999  -232.199  1.0
     -269.999  -226.799  1.0
     -269.999  -221.399  1.0
     -269.999  -215.999  1.0
     -269.999  -210.599  1.0
     -269.999  -205.199  1.0
        ⋮
      269.994   210.599  1.0
      269.994   215.999  1.0
      269.994   221.399  1.0
      269.994   226.799  1.0
      269.994   232.199  1.0
      269.994   237.599  1.0
      269.994   242.999  1.0
      269.994   248.399  1.0
      269.994   253.799  1.0
      269.994   259.199  1.0
      269.994   264.599  1.0
      269.994   269.998  1.0

```julia
epsilon3D = readdlm("data/geom_a_225.dat", header=false);
epsilon2Dpoints = epsilon3D[:,[1,2,4]]
```

    471×3 Array{Float64,2}:
     224.995    1.50075  1.0
     224.955    4.50198  1.0
     224.875    7.50241  1.0
     224.755   10.5015   1.0
     224.595   13.4987   1.0
     224.395   16.4936   1.0
     224.155   19.4855   1.0
     223.875   22.4739   1.0
     223.555   25.4583   1.0
     223.196   28.4382   1.0
     222.796   31.413    1.0
     222.358   34.3823   1.0
     221.879   37.3454   1.0
       ⋮
     222.358  -34.3823   1.0
     222.796  -31.413    1.0
     223.196  -28.4382   1.0
     223.555  -25.4583   1.0
     223.875  -22.4739   1.0
     224.155  -19.4855   1.0
     224.395  -16.4936   1.0
     224.595  -13.4987   1.0
     224.755  -10.5015   1.0
     224.875   -7.50241  1.0
     224.955   -4.50198  1.0
     224.995   -1.50075  1.0

Now we plot out the data in a 2D (xy) plane.

```julia
using PyPlot
#println(convert(Int64,floor(lenz/2)))
x = boundarypoints[:,1];
y = boundarypoints[:,2];
v_regions = boundarypoints[:,3];
lenx = length(x)

fig = figure("Boundary points plot",figsize=(10,10))
ax = fig[:add_subplot](1,2,1)
c = get_cmap("PRGn")
rgbs = [c(norm(value/2.)) for value in v_regions]
scatter(x,y,c=rgbs,linewidths=0,marker=".",s=5)

xlabel(L"x/nm")
ylabel(L"y/nm")
axis("image")
xlim(-300,300)
ylim(-300,300)
tight_layout()
title("meshing points (x,y)");
display(maximum(abs.(x)))

subplot(1,2,2)
x_eps = epsilon2Dpoints[:,1]; y_eps = epsilon2Dpoints[:,2]; v_eps = epsilon2Dpoints[:,3];
ax = fig[:add_subplot](1,2,2)
c = get_cmap("Greys")
rgbs = [c(norm(value/2.)) for value in v_eps]
scatter(x_eps,y_eps,c=rgbs, marker = :.,s=1)

xlabel(L"x/ nm")
ylabel(L"y/ nm")
axis("image")
xlim(-280,280)
ylim(-280,280)
tight_layout()
#gcf() # Needed for IJulia to plot inline
title("boundary points (x,y)");
```

The first plot only shows the 101$\times$101 data points, in which the plotted area is the computing region of a $270nm\times 270nm$ square with the nanofiber region colored in green. The figure on the right covers the boundary points pretty densely, although the point overlap at the $(225,0)nm$ location is not visible.

# Plotting the LDOS components with a fixed dipole position

To calculate the modified decay rates, we need to use the LDOS value at the dipole position. The result is calculated at a series of $k$ points. I expect to see a continuous positive curve when $k\in [0,1]\omega/c$ or in the radiative mode regime and a single positive spark in the $[1,1.45]\omega/c$ or guided mode regime--given the waveguide is a single-mode glass fiber.

```julia
ldos = readdlm("data/ldos_N_1_lam_894_eps_2.1013_0_a_225_x_405_y_0_q_0_1.45_146.dat", header=false);
ldosqpoints = ldos[:,[2,5,6,7,8]]
using PyPlot
#println(convert(Int64,floor(lenz/2)))
q = ldosqpoints[:,1]
ldosx = ldosqpoints[:,2]
ldosy = ldosqpoints[:,3]
ldosz = ldosqpoints[:,4]
ldos_av = ldosqpoints[:,5]
lenx = length(q)

fig = figure("LDOS q plot",figsize=(10,10))
ax = fig[:add_subplot](1,2,1)
cp = ax[:plot](q, ldosx, "b-", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS_x")
axis("image")
xlim(-0.0,2)
#ylim(-300,300)
tight_layout()
gcf() # Needed for IJulia to plot inline
display(maximum(abs.(ldosx)))

ax = fig[:add_subplot](1,2,2)
cp = ax[:plot](q, ldos_av, "r-", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS")
axis("image")
xlim(-0.0,2)
#ylim(-300,300)
tight_layout()
gcf() # Needed for IJulia to plot inline
display(maximum(abs.(ldosx)))
```

As you can see, ***there are negative sparks in the guided mode regime***. It could be a numerical error in the code and may be removed using some tricks.

Now we can integrate the $\mathrm{LDOS}_i$ values over $k$ along the whole axis to obtain the total decay rate or from $0$ to $\omega/c$ to obtain the radiative mode contribution for the decay rate. Each integral can be performed as a sum over all discrete points along the $k$-axis using the Trapezoid approximation as below:
$$\begin{align}\int \mathrm{LDOS}_i(k)dk &= \sum_{j=1,\cdots,N-1} \frac{\mathrm{LDOS}_i(k_j)+\mathrm{LDOS}_i(k_j+1)}{2}\Delta k\\
&= \left(\frac{\mathrm{LDOS}_i(k_N)+\mathrm{LDOS}_i(k_N)}{2}+\sum_{j=2,\cdots,N-1}\mathrm{LDOS}_i(k_j)\right)\Delta k,\end{align}$$
where $N$ is the total number of data points along the $k$-axis, and $\Delta k=k_j-k_{j-1}=k_2-k_1$ as the interval of $k$-vector in a uniform distribution manner.

In our case, the integrand is discontinuous and breaks the continuity at $k=k_0=\omega/c$ point. Therefore, we divide our integration limit into the $[0,n_2)k_0$ and $(n_2,n_1]k_0$ two regions corresponding to radiation mode contribution and guided mode contribution parts, where $n_2=1$ is the index of refraction of the vacuum clad and $n_1=1.4496$ is the index of refraction of the waveguide bulk material. Since there are surdden jumps in the guided mode regime, the integration may have some error using current method, but it shouldn't be too large as the jumps are in small intervals.

```julia
length_of_q=length(ldosqpoints[:,1])
breakpoint = Int(floor((length_of_q-1)/maximum(ldosqpoints[:,1])*1.)) # The breaking point of index is chosen under the fact that the radiation contribution part takes a half space for $n_1=2$.
using NumericalIntegration
ldos_x_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,2])
ldos_x_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,2])
ldos_x = ldos_x_rad+ldos_x_guide;
ldos_y_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,3])
ldos_y_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,3])
ldos_y = ldos_y_rad+ldos_y_guide;
ldos_z_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,4])
ldos_z_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,4])
ldos_z = ldos_z_rad+ldos_z_guide;
ldos_rad = integrate(ldosqpoints[1:breakpoint,1],ldosqpoints[1:breakpoint,5])
ldos_guide = integrate(ldosqpoints[breakpoint:end,1],ldosqpoints[breakpoint:end,5])
ldos_total = ldos_rad + ldos_guide;
# Print out the result.
@printf("\Delta k = %4f k_0.\n",ldosqpoints[2,1]-ldosqpoints[1,1])
@printf("LDOS_x_rad=%5f, LDOS_x_guide=%5f, LDOS_x=%5f;\n",ldos_x_rad,ldos_x_guide,ldos_x)
@printf("LDOS_y_rad=%5f, LDOS_y_guide=%5f, LDOS_y=%5f;\n",ldos_y_rad,ldos_y_guide,ldos_y)
@printf("LDOS_z_rad=%5f, LDOS_z_guide=%5f, LDOS_z=%5f;\n",ldos_z_rad,ldos_z_guide,ldos_z)
@printf("LDOS_rad = %5f, LDOS_guide = %5f, LDOS = %5f.",ldos_rad,ldos_guide,ldos_total)
```

    Delta k = 0.010000 k_0.

Notice that the result above was calculated using a $k$-resolution of $\Delta k= 0.01k_0$. We can compare the results above with a coarser/finer gridding cases.

# LDOS with material losses
The above shows the misbehavior of the guided mode contribution to LDOS. The peaks are due to numerical difficulties when the width of peaks are too narrow where the width of peaks are determined by the loss rate of the material according to uncertainty principle.

Below we will study the cases with material losses.

```julia
ldos_1 = readdlm("data/ldos_N_1_lam_894_eps_2.1013_0.1_a_225_x_405_y_0_q_0_1.45_146.dat", header=false);
ldosqpoints_1 = ldos_1[:,[2,5,6,7,8]]
ldos_01 = readdlm("data/ldos_N_1_lam_894_eps_2.1013_0.01_a_225_x_405_y_0_q_0_1.45_146.dat", header=false);
ldosqpoints_01 = ldos_01[:,[2,5,6,7,8]]
ldos_0001 = readdlm("data/ldos_N_1_lam_894_eps_2.1013_0.0001_a_225_x_405_y_0_q_0_1.45_146.dat", header=false);
ldosqpoints_0001 = ldos_0001[:,[2,5,6,7,8]]
using PyPlot
#println(convert(Int64,floor(lenz/2)))
q = ldosqpoints_1[:,1]
lenx = length(q)
ldosx_1 = ldosqpoints_1[:,2]
ldosy_1 = ldosqpoints_1[:,3]
ldosz_1 = ldosqpoints_1[:,4]
ldos_av_1 = ldosqpoints_1[:,5]
ldosx_01 = ldosqpoints_01[:,2]
ldosy_01 = ldosqpoints_01[:,3]
ldosz_01 = ldosqpoints_01[:,4]
ldos_av_01 = ldosqpoints_01[:,5]
ldosx_0001 = ldosqpoints_0001[:,2]
ldosy_0001 = ldosqpoints_0001[:,3]
ldosz_0001 = ldosqpoints_0001[:,4]
ldos_av_0001 = ldosqpoints_0001[:,5]

fig = figure("LDOS q plot",figsize=(10,10))
ax = fig[:add_subplot](1,2,1)
cp = ax[:plot](q, ldosx_1, "r-", linewidth=2.0)
cp = ax[:plot](q, ldosx_01, "b-.", linewidth=2.0)
cp = ax[:plot](q, ldosx_0001, "m:", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS_x")
axis("image")
xlim(-0.0,1.5)
ylim(-0.5,2)
tight_layout()
gcf() # Needed for IJulia to plot inline
legend([L"imag(\varepsilon=0.1)",L"imag(\varepsilon=0.01)",L"imag(\varepsilon=0.0001)"],loc="lower left")
display(maximum(abs.(ldosx)))

ax = fig[:add_subplot](1,2,2)
cp = ax[:plot](q, ldos_av_1, "r-", linewidth=2.0)
cp = ax[:plot](q, ldos_av_01, "b-.", linewidth=2.0)
cp = ax[:plot](q, ldos_av_0001, "m:", linewidth=2.0)
#ax[:clabel](cp, inline=1, fontsize=5)

xlabel(L"k/(\omega/c)")
ylabel(L"LDOS")
axis("image")
xlim(-0.0,1.5)
ylim(-0.5,2.2)
tight_layout()
gcf() # Needed for IJulia to plot inline
legend([L"imag(\varepsilon=0.1)",L"imag(\varepsilon=0.01)",L"imag(\varepsilon=0.0001)"],loc="lower left")
display(maximum(abs.(ldosx)))
```

## LDOS components as a function of dipole position

We can also plot out the LDOS's when the dipole is placed at different locations along the x-axis. Here we are using some rough parameters just for demonstration purpose.

As plotted below, the dipole is changing position from $270$nm to $572.5$nm from the origin (center of the waveguide cross section) along the x-axis.

```julia
# Load the simulated data.
ldos = readdlm("data/ldos_N_1_lam_894_eps_2.1013_0_a_225_x0_270_x1_572.5_nx_14_y_0_q_0_1.45_146.dat", header=false);
```

```julia
ldosqpoints = ldos[:,[2,3,5,6,7,8]]
lenr = 14; rstart=270.; rend=572.5;
lenq = 146;
breakpoint = Int(floor((lenq-1)/maximum(ldosqpoints[:,1])*1.)) # The breaking point of index is chosen under the fact that the radiation contribution part takes a 1.0/max(k) space for $n_1=1.45$.
rprime = linspace(rstart,rend,lenr);
drprime = rprime[2]-rprime[1]; # Gradient of atom position vector.
ldos_rscan = zeros(lenq,5,lenr);
ldos_int = zeros(lenr,12);
ii=1;
using NumericalIntegration
for ri in rprime
    ind=find(abs.(ldosqpoints[:,2].-ri).<drprime/2.);
    ldos_rscan[:,:,ii]=ldosqpoints[ind,[1,3,4,5,6]];

    ldos_x_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,2,ii])
    ldos_x_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,2,ii])
    ldos_x = ldos_x_rad+ldos_x_guide;
    ldos_y_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,3,ii])
    ldos_y_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,3,ii])
    ldos_y = ldos_y_rad+ldos_y_guide;
    ldos_z_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,4,ii])
    ldos_z_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,4,ii])
    ldos_z = ldos_z_rad+ldos_z_guide;
    ldos_rad = integrate(ldos_rscan[1:breakpoint,1,ii],ldos_rscan[1:breakpoint,5,ii])
    ldos_guide = integrate(ldos_rscan[breakpoint:end,1,ii],ldos_rscan[breakpoint:end,5,ii])
    ldos_total = ldos_rad + ldos_guide;

    ldos_int[ii,1]=ldos_x_rad;
    ldos_int[ii,2]=ldos_x_guide;
    ldos_int[ii,3]=ldos_x;
    ldos_int[ii,4]=ldos_y_rad;
    ldos_int[ii,5]=ldos_y_guide;
    ldos_int[ii,6]=ldos_y;
    ldos_int[ii,7]=ldos_z_rad;
    ldos_int[ii,8]=ldos_z_guide;
    ldos_int[ii,9]=ldos_z;
    ldos_int[ii,10]=ldos_rad;
    ldos_int[ii,11]=ldos_guide;
    ldos_int[ii,12]=ldos_total;
    ii+=1;
end
```

```julia
# Plot the LDOS's.
using PyPlot
fig = figure("LDOS(r') plot",figsize=(10,5))
ax = fig[:add_subplot](1,2,1)
cp = ax[:plot](rprime, ldos_int[:,3], "b:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,6], "r:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,9], "m:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,12], "k-", linewidth=2.0)

xlabel(L"r\prime(nm)")
ylabel(L"LDOS")
ylim(-0.0,2.2)
xlim(225,575)
legend(["LDOS_x","LDOS_y","LDOS_z","LDOS_total"],loc="right")

ax = fig[:add_subplot](1,2,2)
cp = ax[:plot](rprime, ldos_int[:,10], "b:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,11], "r:", linewidth=2.0)
cp = ax[:plot](rprime, ldos_int[:,12], "k-", linewidth=2.0)

xlabel(L"r\prime(nm)")
ylabel(L"LDOS")
ylim(-0.0,2.2)
xlim(225,575)
legend(["LDOS_rad","LDOS_guide","LDOS_total"],loc="right")
```

The LDOS values seem larger than what I expected given $n_1=1.4496$..

## Improved method to calculate the relative decay rates

Now, we consider to only use the radiation part of LDOS from the BEM calculation while using the analytical solution of the Green's function tensor for the guided mode contribution part of the decay rate calculation.
To verify this idea, here we compare the radiation contribution part of the averaged decay rate (or when the dipole is orientated along $[1,1,1]$ direction) against our trusted results calculated from the known analytical solution on Matlab.

```julia
# Read data from the matlab calculation for the exact solution of un-guided mode contribution part for the averaged decay rate.
using MAT
matfile = matopen("data/nanofiber_decay_data.mat")
rp0_test=read(matfile,"rp0_test");
gamma_rad = read(matfile,"gamma_rad");
close(matfile)
# Plot both BEM and exact solution of the un-guided mode contribution to the averaged decay rate as a function of dipole position.
using PyPlot
fig = figure(L"\Gamma_{rad}/\Gamma_0(r') plot",figsize=(10,5))
ax = fig[:add_subplot](1,1,1)
cp = ax[:plot](rprime, ldos_int[:,10]/2.0, "b:", linewidth=2.0)
cp = ax[:plot](rp0_test[1,:]/1.e-9, 1+sum(gamma_rad,2), "r-", linewidth=2.0)

xlabel(L"r\prime(nm)")
ylabel(L"\Gamma_{rad}/\Gamma_0")
ylim(0.0,1.2)
xlim(225,575)
legend([L"\Gamma_{rad}^{BEM}/\Gamma_0",L"\Gamma_{rad}^{exact}/\Gamma_0"],loc="right")
```

The result above shows a good agreement between the BEM method and the exact solution for the radiation mode contribution to the decay rate.
The mismatch becomes considerable only when the dipole is very close to the surface, in which case there is a negative LDOS on the edge of the BEM calculation in the $k$-space.
This small error may be corrected by filtering out the negative LDOS values in the $k$-space.

With the known Green's function tensor expression for the guided modes, one can hence obtain a precise enough result for the decay rates as a function of dipole orientation and atom position.

# Calculation of Green's function tensor using BEM

BEM can output full local field components so that computing the radiative mode contribution to the full Green's function tensor is possible. With the Green's function tensor, one can then calculate the modified decay rates with dipoles orientated along arbitrary directions--including the dipole transitions corresponding to $\sigma_\pm$ and $\pi$ transitions.

```julia
# Load data for different dipole positions.
rp_BEM=[230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,405,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610];
lenrp=length(rp_BEM);
lendr=146;
E_dipolex = readdlm("data/dipolex_E_N_1_lam_894_eps_2.1013_1e-05_a_225_x_405_y_0_q_0_1.45_146.dat", header=false);
k_vec=E_dipolex[:,2];
Ex_dx=zeros(Complex{Float64},lendr,lenrp); Ey_dx=zeros(Complex{Float64},lendr,lenrp); Ez_dx=zeros(Complex{Float64},lendr,lenrp);
Ex_dy=zeros(Complex{Float64},lendr,lenrp); Ey_dy=zeros(Complex{Float64},lendr,lenrp); Ez_dy=zeros(Complex{Float64},lendr,lenrp);
Ex_dz=zeros(Complex{Float64},lendr,lenrp); Ey_dz=zeros(Complex{Float64},lendr,lenrp); Ez_dz=zeros(Complex{Float64},lendr,lenrp);
for ii=1:lenrp
    E_dipolex = readdlm("data/dipolex_E_N_1_lam_894_eps_2.1013_0.001_a_225_x_$(rp_BEM[ii])_y_0_q_0_1.45_146.dat", header=false)
    Ex_dx[:,ii]=E_dipolex[:,5]+im*E_dipolex[:,6];
    Ey_dx[:,ii]=E_dipolex[:,7]+im*E_dipolex[:,8];
    Ez_dx[:,ii]=E_dipolex[:,9]+im*E_dipolex[:,10];

    E_dipoley = readdlm("data/dipoley_E_N_1_lam_894_eps_2.1013_0.001_a_225_x_$(rp_BEM[ii])_y_0_q_0_1.45_146.dat", header=false)
    Ex_dy[:,ii]=E_dipoley[:,5]+im*E_dipoley[:,6];
    Ey_dy[:,ii]=E_dipoley[:,7]+im*E_dipoley[:,8];
    Ez_dy[:,ii]=E_dipoley[:,9]+im*E_dipoley[:,10];

    E_dipolez = readdlm("data/dipolez_E_N_1_lam_894_eps_2.1013_0.001_a_225_x_$(rp_BEM[ii])_y_0_q_0_1.45_146.dat", header=false)
    Ex_dz[:,ii]=E_dipolez[:,5]+im*E_dipolez[:,6];
    Ey_dz[:,ii]=E_dipolez[:,7]+im*E_dipolez[:,8];
    Ez_dz[:,ii]=E_dipolez[:,9]+im*E_dipolez[:,10];
end
# Calculate the diagonal elements of the Green's function tensor from the radiation mode contribution.
c=2.99792458e8;
au=1.72e7; # This is the atomic unit in the CGS-units: $q/a_0^2$ statvolts/cm. In SI units, it becomes $e/(4π\varepsilon_0a_0^2)$ = 5.2e11 V/m.
lambda0=0.895e-6;
ω=2.*pi*c/lambda0;
GFT_rad_ind=zeros(Complex{Float64},3,3,lenrp);
Gxx_rad_ind=zeros(Complex{Float64},lenrp);
Gxy_rad_ind=zeros(Complex{Float64},lenrp);
Gxz_rad_ind=zeros(Complex{Float64},lenrp);
Gyx_rad_ind=zeros(Complex{Float64},lenrp);
Gyy_rad_ind=zeros(Complex{Float64},lenrp);
Gyz_rad_ind=zeros(Complex{Float64},lenrp);
Gzx_rad_ind=zeros(Complex{Float64},lenrp);
Gzy_rad_ind=zeros(Complex{Float64},lenrp);
Gzz_rad_ind=zeros(Complex{Float64},lenrp);
G0=Inf + 2.0/3.*(ω/c)^3*im;
gamma_rad_BEM_rp_average=zeros(lenrp);
gamma_rad_BEM_rp_sigmap=zeros(lenrp);
gamma_rad_BEM_rp_sigmam=zeros(lenrp);
gamma_rad_BEM_rp_pi=zeros(lenrp);
# Define the unitary dipole orientation vector.
e_dipole_sigmap=[-1.;-1.0*im;0]/sqrt(2);
e_dipole_sigmam=[1.; -1.0*im;0]/sqrt(2);
e_dipole_pi=[0.; 0.; 1.];
using NumericalIntegration
for ii=1:lenrp
    Gxx_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ex_dx[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;#G0+
    Gyy_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ey_dy[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;#G0+
    Gzz_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ez_dz[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;#G0+
    Gyx_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ey_dx[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gzx_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ez_dx[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gxy_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ex_dy[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gzy_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ez_dy[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gxz_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ex_dz[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    Gyz_rad_ind[ii]=integrate(k_vec[1:breakpoint],Ey_dz[1:breakpoint,ii],Trapezoidal())*(ω/c)^3/pi^2*au*4.0/3.;
    GFT_rad_ind[:,:,ii]=[Gxx_rad_ind[ii] Gxy_rad_ind[ii] Gxz_rad_ind[ii];
        Gyx_rad_ind[ii] Gyy_rad_ind[ii] Gzy_rad_ind[ii];
        Gzx_rad_ind[ii] Gzy_rad_ind[ii] Gzz_rad_ind[ii]];
end

# Calculate the relative averaged decay rate at the given dipole located at x=405nm and y=0.
gamma0=imag(G0);
for ii =1:lenrp
    gamma_rad_BEM_rp_average[ii]=1+trace(imag(GFT_rad_ind[:,:,ii]))/gamma0/3.;
    gamma_rad_BEM_rp_sigmap[ii]=1+real((e_dipole_sigmap'*imag(GFT_rad_ind[:,:,ii])*e_dipole_sigmap)/gamma0)[1];
    gamma_rad_BEM_rp_sigmam[ii]=1+real((e_dipole_sigmam'*imag(GFT_rad_ind[:,:,ii])*e_dipole_sigmam)/gamma0)[1];
    gamma_rad_BEM_rp_pi[ii]=1+real((e_dipole_pi'*imag(GFT_rad_ind[:,:,ii])*e_dipole_pi)/gamma0)[1];
end

# Recalculate the diagonal GFT elements with a dipole placed at r'=405nm from the fiber axis with lower imaginary part of epsilon.
E_dipolex = readdlm(join(["data/dipolex_E_N_1_lam_894_eps_2.1013_1e-05_a_225_x_","405","_y_0_q_0_1.45_146.dat"]), header=false)
Ex_dx_r0=E_dipolex[:,5]+im*E_dipolex[:,6];
Ey_dx_r0=E_dipolex[:,7]+im*E_dipolex[:,8];
Ez_dx_r0=E_dipolex[:,9]+im*E_dipolex[:,10];
E_dipoley = readdlm(join(["data/dipoley_E_N_1_lam_894_eps_2.1013_1e-05_a_225_x_","405","_y_0_q_0_1.45_146.dat"]), header=false)
Ex_dy_r0=E_dipoley[:,5]+im*E_dipoley[:,6];
Ey_dy_r0=E_dipoley[:,7]+im*E_dipoley[:,8];
Ez_dy_r0=E_dipoley[:,9]+im*E_dipoley[:,10];
E_dipolez = readdlm(join(["data/dipolez_E_N_1_lam_894_eps_2.1013_1e-05_a_225_x_","405","_y_0_q_0_1.45_146.dat"]), header=false)
Ex_dz_r0=E_dipolez[:,5]+im*E_dipolez[:,6];
Ey_dz_r0=E_dipolez[:,7]+im*E_dipolez[:,8];
Ez_dz_r0=E_dipolez[:,9]+im*E_dipolez[:,10];
Gxx_rad_r0=integrate(k_vec[1:breakpoint],Ex_dx_r0[1:breakpoint],Trapezoidal())*(ω/c)^3/pi^2*au*4./3.;
Gyy_rad_r0=integrate(k_vec[1:breakpoint],Ey_dy_r0[1:breakpoint],Trapezoidal())*(ω/c)^3/pi^2*au*4./3.;
Gzz_rad_r0=integrate(k_vec[1:breakpoint],Ez_dz_r0[1:breakpoint],Trapezoidal())*(ω/c)^3/pi^2*au*4./3.;
gamma_rad_BEM_r0=imag(Gxx_rad_r0+Gyy_rad_r0+Gzz_rad_r0)/3./gamma0;

# Plot out gamma_rad as a function of dipole position.
figure(figsize=(16,3));
subplot(1,2,1)
plot((1:breakpoint)/100.,real(Ex_dx[1:breakpoint,4]),"r-")
plot((1:breakpoint)/100.,imag(Ex_dx[1:breakpoint,4]),"b-")
ylabel(L"E(\beta)")
xlabel(L"\beta/k_0")
legend(["Real","Imag"],loc="lower left")

subplot(1,2,2)
a=225.;
plot(rp0_test[1,:]/1.e-9-a, 1+sum(gamma_rad,2), "r-", linewidth=2.0)
plot(rp_BEM-a,real(gamma_rad_BEM_rp_average),"b*-")
plot(405-a,real(gamma_rad_BEM_r0)+1.,"mo");
plot(rp_BEM-a,gamma_rad_BEM_rp_sigmap,"r.-");
plot(rp_BEM-a,gamma_rad_BEM_rp_sigmam,"m--");
plot(rp_BEM-a,gamma_rad_BEM_rp_pi,"k.")
xlabel("(r'-a)/nm")
ylabel(L"\Gamma_{rad}^{BEM}/\Gamma_0")
#ylim([0,1.2])
legend([L"average_{mode\,\,decomp}",L"average_{BEM}",L"average_{no\,\,loss}",L"\sigma_+",L"\sigma_-",L"\pi"],loc="upper center",fontsize=12);
#legend([L"average_{BEM}",L"average_{no\,\,loss}",L"\sigma_+",L"\sigma_-",L"\pi"],loc="upper center",fontsize=12);
gamma_rad_BEM_rp_average
```

On the plots above, the left shows the real and imaginary parts of the induced field component $E_x(r')$ when an $x$-polarized dipole was placed at $r'=405$nm away from the fiber axis. Results are calculated from BEM and are not normalized properly. When the wave number $k\approx k_0=\omega/c$, the values are changing dramatically, despite of a small givin loss to the material.

The plot on the right-hand-side is the modified decay rates from the non-guided modes calculated in different methods and scenarios.
The red solid line (using the verified "exact" mode decomposition method) and the blue stars (using the BEM approach) are the averaged decay rates from the non-guided mode contribution part given as
$$\begin{align}
\frac{\Gamma_{rad}}{\Gamma_0} &= 1+ \frac{\sum_{i=x,y,z}\mathrm{Im}\left[\mathbf{e}_i^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_i\right]}{\sum_{i=x,y,z} \mathrm{Im}\left[\mathbf{e}_i^*\cdot \mathbf{G}_0(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_i\right]}\\
&=1+ \frac{\sum_{i=x,y,z}\mathrm{Im}\left[\mathbf{e}_i^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_i\right]}{3\mathrm{Im}\left[G_0(\mathbf{r}',\mathbf{r}')\right]}\\
&= 1+ \frac{\mathrm{Tr}\left\{\mathrm{Im}\left[ \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\right]\right\}}{3\mathrm{Im}\left[G_0(\mathbf{r}',\mathbf{r}')\right]}
\end{align}$$
where $\mathbf{G}_0=G_0\mathbb{1}$ with $G_0(\mathbf{r}',\mathbf{r}';\omega)=\frac{2}{3}k_0^3$ is the Green's function in vacuum with $k_0=\frac{\omega}{c}$; and the radiation mode contribution of the induced Green's function tensor in presence of the nanofiber is
$$ G_{ind,rad}^{ij}(\mathbf{r}',\mathbf{r}')=\frac{2k_0^2*a.u.}{3\pi^2}\int_{-k_0}^{k_0} d\beta E_j^i(\mathbf{r}')=\frac{4k_0^2*a.u.}{3\pi^2}\int_{0}^{k_0} d\beta E_j^i(\mathbf{r}')$$ with $E_j^i(\mathbf{r}')$ representing the $i$ electric field component with an atomic unit dipole polarized along $j$ direction ($i,j=1,y,z$).
Since the the BEM program uses atomic units and CGS-unit system, the prefactor in the formulas above, $a.u.=\frac{q}{a_0^2}=1.72\times 10^7$statvolts/cm, comes from the atomic unit transformation.
Compared to the previous averaged LDOS calculation (red curve), the averaged decay rate (dots) calculated based on the local feild componets with dipoles orientated along three basis directions using the BEM program matches our exact solution pretty well.
The small mismatch when the dipole position becomes far from the fiber surface may be due to the fact that our "exact" decay rate for comparison was actually truncated up to $10$-th order of the non-guided eigen modes of the nanofiber which yields some error for the final result.
From what we have calculated earlier for LDOS, the induced non-guided mode contribution part to the decay rate could be negative at some dipole positions which is correctly reflected at the series of numbers as the last output of the calculation above.

From the second plot, we also show the non-guided mode induced decay rates that can be coupled to the $\sigma_\pm$ and $\pi$ dipole transitions.
As you can see, when the atoms are placed around $200$nm from the nanofiber surface, different dipole transitions don't make a noticeable difference which is no longer true for the square waveguide case as shown in another simulation notebook.
In details, we calculate the polarization-dependent decay rates from the non-guided mode contributions by
$$\begin{align}
\frac{\Gamma_{rad}}{\Gamma_0} &= 1+ \frac{\mathrm{Im}\left[\mathbf{e}_q^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_q\right]}{ \mathrm{Im}\left[\mathbf{e}_q^*\cdot \mathbf{G}_0(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_q\right]}
=1+ \frac{\mathrm{Im}\left[\mathbf{e}_q^*\cdot \mathbf{G}_{ind,rad}(\mathbf{r}',\mathbf{r}')\cdot \mathbf{e}_q\right]}{\mathrm{Im}\left[G_0(\mathbf{r}',\mathbf{r}')\right]},
\end{align}$$
where the three orthogonal dipole transition bases
$$\begin{align}
\mathbf{e}_\pm &=\mp \frac{\mathbf{e}_{\tilde{x}}\pm i\mathbf{e}_{\tilde{y}}}{\sqrt{2}}\\
\mathbf{e}_0 &=\mathbf{e}_{\tilde{z}}
\end{align}$$
correspond to the $\sigma_\pm$ and $\pi$ transitions of the atoms.
These basis vectors are quantization-axis dependent, but in our calculation above, we assume the $z$-axis or the waveguide axis is the quantization axis, and hence $\mathbf{e}_{\tilde{x}}=\mathbf{e}_x$, $\mathbf{e}_{\tilde{y}}=\mathbf{e}_y$ and $\mathbf{e}_{\tilde{z}}=\mathbf{e}_z$.

```julia
# Export some variables to a MAT file.
#using HDF5,JLD
#h5open("data/Julia_nanofiber_GFT_decayrates_rad.h5", "w") do h5file
#    g=g_create(h5file, "nanofiber")
#    write(h5file, "nanofiber/rp_BEM", rp_BEM)#, "GFT_rad_ind",GFT_rad_ind)#,"omega",ω,"gamma0",gamma0,"e_dipole_sigmap",e_dipole_sigmap, "e_dipole_sigmam",e_dipole_sigmam,"e_dipole_pi",e_dipole_pi,"gamma_rad_BEM_rp_average",gamma_rad_BEM_rp_average,"gamma_rad_BEM_rp_sigmap",gamma_rad_BEM_rp_sigmap,"gamma_rad_BEM_rp_sigmam",gamma_rad_BEM_rp_sigmam,"gamma_rad_BEM_rp_pi",gamma_rad_BEM_rp_pi,"a",a)
#    write(h5file,"nanofiber/GFT_rad_ind",GFT_rad_ind)
#end
#save("data/Julia_nanofiber_GFT_decayrates_rad.jld","rp_BEM",rp_BEM,"imGFT_rad_ind",imag(GFT_rad_ind),"omega0",ω)
using MAT
matopen("data/Julia_nanofiber_GFT_decayrates_rad_D1.mat", "w") do matfile
    write(matfile,"omega0",ω)
    write(matfile,"rp_BEM",collect(rp_BEM))
    write(matfile,"gamma0",gamma0)
    write(matfile,"e_dipole_sigmap",e_dipole_sigmap)
    write(matfile,"e_dipole_sigmam",e_dipole_sigmam)
    write(matfile,"e_dipole_pi",e_dipole_pi)
    write(matfile,"gamma_rad_BEM_rp_average",gamma_rad_BEM_rp_average)
    write(matfile,"gamma_rad_BEM_rp_sigmap",gamma_rad_BEM_rp_sigmap)
    write(matfile,"gamma_rad_BEM_rp_sigmam",gamma_rad_BEM_rp_sigmam)
    write(matfile,"gamma_rad_BEM_rp_pi",gamma_rad_BEM_rp_pi)
    write(matfile,"a",a)
    write(matfile,"GFT_rad_rp",GFT_rad_ind)
    write(matfile,"Gxx_rad_rp",Gxx_rad_ind)
    write(matfile,"Gxy_rad_rp",Gxy_rad_ind)
    write(matfile,"Gxz_rad_rp",Gxz_rad_ind)
    write(matfile,"Gyx_rad_rp",Gyx_rad_ind)
    write(matfile,"Gyy_rad_rp",Gyy_rad_ind)
    write(matfile,"Gyz_rad_rp",Gyz_rad_ind)
    write(matfile,"Gzx_rad_rp",Gzx_rad_ind)
    write(matfile,"Gzy_rad_rp",Gzy_rad_ind)
    write(matfile,"Gzz_rad_rp",Gzz_rad_ind)
end
```

```julia

```


Table of Contents

1  Exercise 1: Compute MSE

2  Interactive Demo: MSE Explorer

2.1  Exercise 2: Solve for the Optimal Estimator

3  Least Squares Optimization Derivation

# Neuromatch Academy: Week 1, Day 3, Tutorial 1
# Model Fitting: Linear regression with MSE

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault,  Ella Batty, Michael Waskom

___
#Tutorial Objectives

This is Tutorial 1 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will learn how to fit simple linear models to data.
- Learn how to calculate the mean-squared error (MSE)
- Explore how model parameters (slope) influence the MSE
- Learn how to find the optimal model parameter using least-squares optimization

---

**acknowledgements:**
- we thank Eero Simoncelli, much of today's tutorials are inspired by exercises asigned in his mathtools class.

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper functions

def plot_observed_vs_predicted(x, y, y_hat, theta_hat):
  """ Plot observed vs predicted data

  Args:
      x (ndarray): observed x values
  y (ndarray): observed y values
  y_hat (ndarray): predicted y values

  """
  fig, ax = plt.subplots()
  ax.scatter(x, y, label='Observed')  # our data scatter plot
  ax.plot(x, y_hat, color='r', label='Fit')  # our estimated model
  # plot residuals
  ymin = np.minimum(y, y_hat)
  ymax = np.maximum(y, y_hat)
  ax.vlines(x, ymin, ymax, 'g', alpha=0.5, label='Residuals')
  ax.set(
      title=fr"$\hat{{\theta}}$ = {theta_hat:0.2f}, MSE = {mse(x, y, theta_hat):.2f}",
      xlabel='x',
      ylabel='y'
  )
  ax.legend()

```

---
# Section 1: Mean Squared Error (MSE)

```python
#@title Video 1: Linear Regression & Mean Squared Error
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="HumajfjJ37E", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=HumajfjJ37E

**Linear least squares regression** is an old but gold  optimization procedure that we are going to use for data fitting. Least squares (LS) optimization problems are those in which the objective function is a quadratic function of the
parameter(s) being optimized.

Suppose you have a set of measurements, $y_{n}$ (the "dependent" variable) obtained for different input values, $x_{n}$ (the "independent" or "explanatory" variable). Suppose we believe the measurements are proportional to the input values, but are corrupted by some (random) measurement errors, $\epsilon_{n}$, that is:

$$y_{n}= \theta x_{n}+\epsilon_{n}$$

for some unknown slope parameter $\theta.$ The least squares regression problem uses **mean squared error (MSE)** as its objective function, it aims to find the value of the parameter $\theta$ by minimizing the average of squared errors:

\begin{align}
\min _{\theta} \frac{1}{N}\sum_{n=1}^{N}\left(y_{n}-\theta x_{n}\right)^{2}
\end{align}

We will now explore how MSE is used in fitting a linear regression model to data. For illustrative purposes, we will create a simple synthetic dataset where we know the true underlying model. This will allow us to see how our estimation efforts compare in uncovering the real model (though in practice we rarely have this luxury).

First we will generate some noisy samples $x$ from [0, 10) along the line $y = 1.2x$ as our dataset we wish to fit a model to.

```python
# @title

# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)

# Let's set some parameters
theta = 1.2
n_samples = 30

# Draw x and then calculate y
x = 10 * np.random.rand(n_samples)  # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples)  # sample from a standard normal distribution
y = theta * x + noise

# Plot the results
fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

Now that we have our suitably noisy dataset, we can start trying to estimate the underlying model that produced it. We use MSE to evaluate how successful a particular slope estimate $\hat{\theta}$ is for explaining the data, with the closer to 0 the MSE is, the better our estimate fits the data.

## Exercise 1: Compute MSE

In this exercise you will implement a method to compute the mean squared error for a set of inputs $x$, measurements $y$, and slope estimate $\hat{\theta}$. We will then compute and print the mean squared error for 3 different choices of theta

```python
def mse(x, y, theta_hat):
  """Compute the mean squared error

  Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """
  ####################################################
  ## TODO for students: compute the mean squared error
  # Fill out function and remove
#   raise NotImplementedError("Student exercise: compute the mean squared error")
  ####################################################

  # Compute the estimated y
  y_hat = x * theta_hat

  # Compute mean squared error
  mse = np.mean((y_hat - y)**2)

  return mse

# Uncomment below to test your function
theta_hats = [0.75, 1.0, 1.5]
for theta_hat in theta_hats:
  print(f"theta_hat of {theta_hat} has an MSE of {mse(x, y, theta_hat):.2f}")
```

    theta_hat of 0.75 has an MSE of 9.08
    theta_hat of 1.0 has an MSE of 3.01
    theta_hat of 1.5 has an MSE of 4.52

```python
# to_remove solution
def mse(x, y, theta_hat):
  """Compute the mean squared error

  Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """

  # Compute the estimated y
  y_hat = theta_hat * x

  # Compute mean squared error
  mse = np.mean((y - y_hat)**2)

  return mse

theta_hats = [0.75, 1.0, 1.5]
for theta_hat in theta_hats:
  print(f"theta_hat of {theta_hat} has an MSE of {mse(x, y, theta_hat):.2f}")
```

    theta_hat of 0.75 has an MSE of 9.08
    theta_hat of 1.0 has an MSE of 3.01
    theta_hat of 1.5 has an MSE of 4.52

The result should be:

theta_hat of 0.75 has an MSE of 9.08\
theta_hat of 1.0 has an MSE of 3.0\
theta_hat of 1.5 has an MSE of 4.52

We see that $\hat{\theta} = 1.0$ is our best estimate from the three we tried. Looking just at the raw numbers, however, isn't always satisfying, so let's visualize what our estimated model looks like over the data.

```python
#@title

#@markdown Execute this cell to visualize estimated models

fig, axes = plt.subplots(ncols=3, figsize=(18, 4))
for theta_hat, ax in zip(theta_hats, axes):

  # True data
  ax.scatter(x, y, label='Observed')  # our data scatter plot

  # Compute and plot predictions
  y_hat = theta_hat * x
  ax.plot(x, y_hat, color='r', label='Fit')  # our estimated model

  ax.set(
      title= fr'$\hat{{\theta}}$= {theta_hat}, MSE = {mse(x, y, theta_hat):.2f}',
      xlabel='x',
      ylabel='y'
  );

axes[0].legend()
```

## Interactive Demo: MSE Explorer

Using an interactive widget, we can easily see how changing our slope estimate changes our model fit. We display the **residuals**, the differences between observed and predicted data, as line segments between the data point (observed response) and the corresponding predicted response on the model fit line.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(theta_hat=widgets.FloatSlider(1.0, min=0.0, max=2.0))
def plot_data_estimate(theta_hat):
  y_hat = theta_hat * x
  plot_observed_vs_predicted(x, y, y_hat, theta_hat)

```

    interactive(children=(FloatSlider(value=1.0, description='theta_hat', max=2.0), Output()), _dom_classes=('widg…

While visually exploring several estimates can be instructive, it's not the most efficient for finding the best estimate to fit our data. Another technique we can use is choose a reasonable range of parameter values and compute the MSE at several values in that interval. This allows us to plot the error against the parameter value (this is also called an **error landscape**, especially when we deal with more than one parameter). We can select the final $\hat{\theta}$ ($\hat{\theta}_{MSE}$) as the one which results in the lowest error.

```python
# @title

# @markdown Execute this cell to loop over theta_hats, compute MSE, and plot results

# Loop over different thetas, compute MSE for each
theta_hat_grid = np.linspace(-2.0, 4.0)
errors = np.zeros(len(theta_hat_grid))
for i, theta_hat in enumerate(theta_hat_grid):
  errors[i] = mse(x, y, theta_hat)

# Find theta that results in lowest error
best_error = np.min(errors)
theta_hat = theta_hat_grid[np.argmin(errors)]

# Plot results
fig, ax = plt.subplots()
ax.plot(theta_hat_grid, errors, '-o', label='MSE', c='C1')
ax.axvline(theta, color='g', ls='--', label=r"$\theta_{True}$")
ax.axvline(theta_hat, color='r', ls='-', label=r"$\hat{{\theta}}_{MSE}$")
ax.set(
  title=fr"Best fit: $\hat{{\theta}}$ = {theta_hat:.2f}, MSE = {best_error:.2f}",
  xlabel=r"$\hat{{\theta}}$",
  ylabel='MSE')
ax.legend();
```

We can see that our best fit is $\hat{\theta}=1.18$ with an MSE of 1.45. This is quite close to the original true value $\theta=1.2$!

---
# Section 2: Least-squares optimization

While the approach detailed above (computing MSE at various values of $\hat\theta$) quickly got us to a good estimate, it still relied on evaluating the MSE value across a grid of hand-specified values. If we didn't pick a good range to begin with, or with enough granularity, we might miss the best possible estimator. Let's go one step further, and instead of finding the minimum MSE from a set of candidate estimates, let's solve for it analytically.

We can do this by minimizing the cost function. Mean squared error is a convex objective function, therefore we can compute its minimum using calculus. Please see video or appendix for this derivation! After computing the minimum, we find that:

\begin{align}
\hat\theta = \frac{\vec{x}^\top \vec{y}}{\vec{x}^\top \vec{x}}
\end{align}

This is known as solving the normal equations. For different ways of obtaining the solution, see the notes on [Least Squares Optimization](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf) by Eero Simoncelli.

### Exercise 2: Solve for the Optimal Estimator

In this exercise, you will write a function that finds the optimal $\hat{\theta}$ value using the least squares optimization approach (the equation above) to solve MSE minimization. It shoud take arguments $x$ and $y$ and return the solution $\hat{\theta}$.

We will then use your function to compute $\hat{\theta}$ and plot the resulting prediction on top of the data.

```python
def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the value for theta_hat arrived from minimizing MSE
  """

  ################################################################################
  ## TODO for students: solve for the best parameter using least squares
  # Fill out function and remove
#   raise NotImplementedError("Student exercise: solve for theta_hat using least squares")
  ################################################################################

  # Compute theta_hat analytically
  # theta_hat = np.sum(x * y)/np.sum(x * x)
  theta_hat = (x.T @ y) / (x.T @ x)

  return theta_hat

# Uncomment below to test your function
theta_hat = solve_normal_eqn(x, y)
y_hat = theta_hat * x
plot_observed_vs_predicted(x, y, y_hat, theta_hat)

```

```python
# to_remove solution
def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the value for theta_hat arrived from minimizing MSE
  """

  theta_hat = (x.T @ y) / (x.T @ x)

  return theta_hat

theta_hat = solve_normal_eqn(x, y)
y_hat = theta_hat * x

with plt.xkcd():
  plot_observed_vs_predicted(x, y, y_hat, theta_hat)

```

We see that the analytic solution produces an even better result than our grid search from before, producing $\hat{\theta} = 1.21$ with MSE = 1.43!

---
# Summary

- Linear least squares regression is an optimization procedure that can be used for data fitting:
    - Task: predict a value for $y$ given $x$
    - Performance measure: $\textrm{MSE}$
    - Procedure: minimize $\textrm{MSE}$ by solving the normal equations
- **Key point**: We fit the model by defining an *objective function* and minimizing it.
- **Note**: In this case, there is an *analytical* solution to the minimization problem and in practice, this solution can be computed using *linear algebra*. This is *extremely* powerful and forms the basis for much of numerical computation throughout the sciences.

---
# Appendix

## Least Squares Optimization Derivation

We will outline here the derivation of the least squares solution.

We first set the derivative of the error expression with respect to $\theta$ equal to zero,

\begin{align}
\frac{d}{d\theta}\frac{1}{N}\sum_{i=1}^N(y_i - \theta x_i)^2 = 0 \\
\frac{1}{N}\sum_{i=1}^N-2x_i(y_i - \theta x_i) = 0
\end{align}

where we used the chain rule. Now solving for $\theta$, we obtain an optimal value of:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

Which we can write in vector notation as:

\begin{align}
\hat\theta = \frac{\vec{x}^\top \vec{y}}{\vec{x}^\top \vec{x}}
\end{align}

This is known as solving the *normal equations*. For different ways of obtaining the solution, see the notes on [Least Squares Optimization](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf) by Eero Simoncelli.


# Small application of 'Tobit' class

### **1: Packages**

```python
import pickle
import requests
import json
import numpy as np
from Estimation import Tobit
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
```

### **2: Load data**

*NB: This collects data from Energidataservice.dk. Data is stored as a pickle to skip downloading.*

```python
UPDATE = 'No'
if UPDATE=='YES':
    # Data from api:
    url = "https://api.energidataservice.dk/datastore_search_sql?sql=SELECT * FROM transmissionlines WHERE \"HourDK\" < '2020-01-01' LIMIT 5000000"
    r = requests.get(url)
    #Electricity balance data:
    url = "https://api.energidataservice.dk/datastore_search_sql?sql=SELECT * FROM electricitybalance WHERE \"HourDK\" < '2020-01-01' LIMIT 5000000"
    r_ebd = requests.get(url)
    print(r,r_ebd)
    # Subset:
    years = [str(y) for y in range(2011,2019)]
    variables = ['HourDK', 'PriceArea','ConnectedArea','ImportCapacity','ExportCapacity','ScheduledExchangeDayAhead','PhysicalExchangeSettlement',
             'CongestionIncomeDKK','HomePriceDKK','ConnectedPriceDKK']
    variables_ebd = ['HourDK','PriceArea','GrossCon','OffshoreWindPower','OnshoreWindPower','SolarPowerProd','ExchangeContinent','ExchangeGreatBelt','ExchangeNordicCountries']
    rawdata = r.json()['result']['records']
    # Extract data to pandas dataframes:
    data = {var: [rawdata[i][var] for i in range(0,len(rawdata))] for var in variables}
    data['year'] = [data['HourDK'][x].split('-')[0] for x in range(0,len(data['HourDK']))]
    data = pd.DataFrame(data)
    rawdata_ebd = r_ebd.json()['result']['records']
    data_ebd = {var: [rawdata_ebd[i][var] for i in range(0,len(rawdata_ebd))] for var in variables_ebd}
    data_ebd['year'] = [data_ebd['HourDK'][x].split('-')[0] for x in range(0,len(data_ebd['HourDK']))]
    data_ebd = pd.DataFrame(data_ebd)
    # Write to pickle:
    f = open("TransmissionData.pkl", "wb")
    pickle.dump([data,data_ebd],f)
    f.close()
else:
    with open("TransmissionData.pkl", "rb") as file:
        [data,data_ebd] = pickle.load(file)
```

### **3: Rearrange data**

```python
data_sliced = data[['HourDK','PriceArea','ConnectedArea','ImportCapacity','ExportCapacity','PhysicalExchangeSettlement','HomePriceDKK','ConnectedPriceDKK','year']].copy()
data_sliced.columns = ['HourDK','EA','EAEA','Icap','Ecap','NX','pe_EA','pe_EAEA','year']
data_ebd['ResDemand'] = data_ebd['GrossCon']-data_ebd['OffshoreWindPower']-data_ebd['OnshoreWindPower']-data_ebd['SolarPowerProd']
ebd = data_ebd[['HourDK','PriceArea','ResDemand','year']].copy()
ebd.columns = ['HourDK','EA','ResDemand','year']
data_all = data_sliced.merge(ebd,on=['HourDK','EA','year'])
```

Data for one year, one transmission line:

```python
EA_,EAEA_,year_ = 'DK1','DE','2017'
data_temp = data_all[(data_all['EA']==EA_) & (data_all['EAEA']==EAEA_) & (data_all['year']==year_)].copy()
```

Final dataset:

```python
data_temp
```

      HourDK
      EA
      EAEA
      Icap
      Ecap
      NX
      pe_EA
      pe_EAEA
      year
      ResDemand

      3
      2017-01-17T23:00:00
      DK1
      DE
      310.0
      0.0
      -4.18
      220.89
      280.649736
      2017
      534.712873

      7
      2017-01-18T00:00:00
      DK1
      DE
      320.0
      -60.0
      -57.16
      215.71
      284.653908
      2017
      390.551799

      11
      2017-01-18T01:00:00
      DK1
      DE
      320.0
      0.0
      -18.12
      212.51
      280.638414
      2017
      305.964304

      21
      2017-01-18T02:00:00
      DK1
      DE
      360.0
      0.0
      -25.45
      211.55
      260.932749
      2017
      222.864581

      53
      2017-01-18T03:00:00
      DK1
      DE
      370.0
      0.0
      -15.09
      212.21
      260.784027
      2017
      227.867328

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      249363
      2017-02-22T02:00:00
      DK1
      DE
      1500.0
      0.0
      666.74
      70.61
      70.615400
      2017
      -849.804246

      249367
      2017-02-22T03:00:00
      DK1
      DE
      1500.0
      0.0
      604.80
      56.93
      56.938312
      2017
      -778.143472

      249371
      2017-02-22T04:00:00
      DK1
      DE
      1500.0
      0.0
      756.79
      37.39
      37.388996
      2017
      -683.427698

      249375
      2017-02-22T05:00:00
      DK1
      DE
      1500.0
      0.0
      798.16
      57.08
      57.086976
      2017
      -535.941648

      472604
      2017-12-31T11:00:00
      DK1
      DE
      1500.0
      -400.0
      1517.27
      205.23
      68.493080
      2017
      427.372986

8762 rows × 10 columns

### **4: Set up data on required form**

The Tobit class requires inputs on the form:
* Initial guess of parameter values, $\theta_0$,
* Explanatory variables as np.array, $x$,
* dependent variable as np.array (vector), $y$.

```python
theta0 = np.array([1000,0,0,0,500])
beta0 = theta0[:-1]
sigma0 = theta0[-1]
x = data_temp[['ResDemand','pe_EA','pe_EAEA']].values
y = abs(data_temp[['Ecap']].values.flatten())
```

### **5: Initialize**

The likelihood estimation ultimately relies on *scipy.optimize.minimize*. You can pass options to the solver through a dictionary, e.g.:

```python
min_settings = {'method': 'Nelder-Mead',
                'tol': None}
```

initialize

```python
Tobit_reg = Tobit(x,y)
```

and pass settings through upd_par function:

```python
Tobit_reg.upd_par(min_settings)
```

### **6: Standard features**

The class contains a number of **static methods** that can be called as the user pleases, these includes:
* *tobit_ll(y,x,theta,intercept=True)*: Negative log-likelihood with data $(y,x)$, parameters $\theta$, with intercept (if True).
* *tobit_sim(theta,X=None,N=100)*: Simulate data from
    $$\begin{align}
        y &= \max\left(X\beta+\epsilon,0\right), && \epsilon\sim N(0,\sigma) \\
        \theta &= [\beta, \sigma]
    \end{align}$$
    where $N$ is number of draws, and $X=None$ enables drawing of random values of $X$ as well (alternative is to pass in actual data here).
* *tobit_novariance(theta,x,intercept=True)*: Returns $max(x\beta,0)$.
* *tobit_censor_prob(theta,x,intercept=True)*: Returns probability $1-\Phi(x\beta/\sigma)$.
* *tobit_cond_exp(theta,x,intercept=True)*: Returns $E[y|x]$.
* *tobit_cond_exp_pos(theta,x,intercept=True)*: Returns $E[y|x,y>0]$.

Besides static methods, the class includes methods:
* *max_ll(self,theta0)*: Maximizes likelihood function with initial parameter guess $\theta_0$. If not successfull a message is returned, otherwise $self.theta$ with estimated
parameters is added.
* *predict(self)*: Computes $E[y|x],E[y|x,y>0]$ for all *self.x* data entries. Adds these as *self.predicted,self.predicted_pos*.
* *plot_fit(self)*: Plots scatterplot of $x \beta$ against data. Includes $E[y|x],E[y|x,y>0]$.

### **7: Run example**

```python
Tobit_reg = Tobit(x,y)
Tobit_reg.max_ll(theta0)
Tobit_reg.predict()
Tobit_reg.plot_fit()
```


# DarkCapPy Template

Author: Adam Green

Email : agree019@ucr.edu

```python

```

This notebook serves to provide a template to calculate the Earth capture scenario.

```python
import numpy as np
import pandas as pd

from DarkCapPy import *
import DarkCapPy.DarkPhoton as DP
print ('Complete')
```

    Dark Photon Module Imported
    Complete

## Import Test

```python
cap1 = DP.photonCCap(mx,ma,epsilon,alpha,alphax)
kappa0 = DP.photonKappa_0(mx,alpha)
cap2 = DP.photonCCapQuick(mx,ma,epsilon,alphax,kappa0)

som = DP.thermAvgSommerfeld(mx,ma,alphax)
sigma = DP.photonSigmaVtree(mx,ma,alphax)
ann = DP.CAnnCalc(mx,sigma,som)

tau = DP.tau(cap1,ann)

gamma = DP.gammaAnn(cap1,ann)
L = DP.decayLength(mx,ma,epsilon,1)
Edecay = DP.epsilonDecay(L)

signal = DP.iceCubeSignal(gamma,Edecay,DP.yr2s(10))

print ('Complete')
```

    Complete

```python
print ('Cap1: ',cap1)
print ('Cap2: ',cap2)

print ('Contour: ', tau/tauCross)
print ('Signal : ', signal)
```

    Cap1:  17393015583359.414
    Cap2:  17613272559662.65
    Contour:  0.02737054836006153
    Signal :  26998236.68491888

# Generate the Sommerfeld File

This section serves to create an external file which contains an arbitrary number of $m_A$ and $\langle S_s \rangle$ points to be interpolated later.

The external file has the following column headers:

    Counter, mX[GeV], mA[GeV], ThermAvgSommerfeld, Kappa0[GeV5]

- Counter serves as an index to loop through
- m_X is specified by the user shortly after the file is created
- m_A is a list of numbers ranging from $0.01$ GeV to $10$ GeV with arbitrary spacing and density. Since these will each correspond to a $\langle S_s \rangle(m_A)$ to be interpolated later, it is suggested to use logarithmic spacing with a high density $(N\geq2500)$
- ThermAvgSommerfeld is the corresponding $\langle S_s \rangle(m_A)$ point

## Define Point Resolution

This cell defines the resolution of the Sommerfeld enhancement. This number should be large ~2500 because it will be used to create an interpolation later on.

```python
num = 2500
```

## Initialize Paramter Arrays

We assume the following range for $m_{A'}$:

\begin{equation}
0.01 \ \text{GeV} \leq m_{A'} \leq 10 \ \text{GeV}
\end{equation}

This will define the range for the mediator mass for the rest of the calculations.

```python
mALogRange = np.logspace(-2, 1, num, base = 10)
mAArray = []

counter = 0
for mATemp in mALogRange:
                 #( Counter, m_X[GeV], m_A[Gev], ThermAvgSommerfeld, Kappa0[GeV5])
    mAArray.append([counter, ' '     , mATemp  , 'None'            , ' '])
    counter += 1

print ('Complete')
```

    Complete

## Create and Write the DataFrame to a .csv

Filenames should follow the convention:

    <#><Unit>Sommerfeld.csv

including the ".csv" extension.

For example, if $m_X = 100$ GeV,

    100GeVSommerfeld.csv

You will also define the value for $m_X$ in this cell

```python
masterSommerfeldDataFrame = \
    pd.DataFrame(mAArray,\
                 columns = \
                 ['Counter','mX[GeV]', 'mA[GeV]', 'ThermAvgSommerfeld', 'Kappa0[GeV5]'])

SommerFile = input('Sommerfeld Filename: ')
writeFile = 'Data_Files/Sommerfeld/'+SommerFile
# writeFile = photonSommerfeldPath(SommerFile)
assert (SommerFile[-4:] == '.csv'), 'File must end with ".csv"'

masterSommerfeldDataFrame.to_csv(writeFile,index=False)

dataFrame = pd.read_csv(writeFile , sep = ',')

m_XValue = float(input('m_X [GeV]: '))
kappa0Value = 'None'

dataFrame.at[0, 'mX[GeV]'] = m_XValue
dataFrame.at[0, 'Kappa0[GeV5]'] = kappa0Value

dataFrame.to_csv(writeFile, index=False)

print ('Complete')
```

## Populating Sommerfeld.csv

In this cell, we:

1). Calculate $\texttt{Kappa0[GeV5]}$

2). Calculate the Sommerfeld enhancement $\langle S_s \rangle$ given each $m_A$ by looping down the list of $m_A$ values

After each itteration of calculations, the .csv is updated and overwritten, but there is a manual overwrite in the next cell just for good measure

```python
sommerFile = input('Sommerfeld Filename: ')
sommWriteFile = 'Data_Files/Sommerfeld/'+sommerFile
# sommWriteFile = photonSommerfeldPath(sommerFile)

sommDataIn = pd.read_csv(sommWriteFile,sep = ',')

# Define the loop range
looprange = len(sommDataIn['Counter'])

# Read in m_X value
Filem_X = float(sommDataIn.at[0,'mX[GeV]'])

##################################
# Calculate Kappa0
##################################
testKappa0 = sommDataIn.at[0,'Kappa0[GeV5]']

if (testKappa0 == 'None'):
    print ('Calculating Kappa0...')

    kappa0Value = float(DP.kappa_0(Filem_X, 1./137))

    sommDataIn.at[0, 'Kappa0[GeV5]'] = kappa0Value
    sommDataIn.to_csv(sommWriteFile, index=False)
    print('Kappa0 value calculated and set')

elif (testKappa0 != 'None'):
    print ('Kappa0 value already calculated')

##################################
# Calculate the Sommerfeld enhancements
##################################

finishedCounter = 0
print ('-------------------')
print ('Starting Sommerfeld Calculations...')
for index in range(0,looprange):
    ##################################
    # Initialize Parameters
    ##################################
    m_AValue = sommDataIn.at[index,'mA[GeV]']
    testSomm = sommDataIn.at[index,'ThermAvgSommerfeld']

    if (testSomm == 'None'):
        alpha_X = DP.alphaTherm(Filem_X, m_AValue)
        thermAvgSomm = DP.thermAvgSommerfeld(Filem_X, m_AValue, alpha_X)

        sommDataIn.at[index,'ThermAvgSommerfeld'] = thermAvgSomm
#         sommDataIn.to_csv(sommWriteFile, sep=',', index=False)

        finishedCounter +=1
        if (index%20 == 0):
            print ('Index: {0} recorded'.format(index))

    elif(testSomm != 'None'):
        finishedCounter +=1

    if (finishedCounter == looprange):
        print ('-------------------')
        print ('All Calculations Complete')

sommDataIn.to_csv(sommWriteFile, index=False)
print ('Complete')
```

    Sommerfeld Filename: testSomm.csv
    Kappa0 value already calculated
    -------------------
    Starting Sommerfeld Calculations...
    Index: 0 recorded
    Index: 20 recorded
    Index: 40 recorded
    Index: 60 recorded
    Index: 80 recorded
    Index: 100 recorded
    Index: 120 recorded
    Index: 140 recorded
    Index: 160 recorded
    Index: 180 recorded
    Index: 200 recorded
    Index: 220 recorded
    Index: 240 recorded
    Index: 260 recorded
    Index: 280 recorded
    Index: 300 recorded
    Index: 320 recorded
    Index: 340 recorded
    Index: 360 recorded
    Index: 380 recorded
    Index: 400 recorded
    Index: 420 recorded
    Index: 440 recorded
    Index: 460 recorded
    Index: 480 recorded
    Index: 500 recorded
    Index: 520 recorded
    Index: 540 recorded
    Index: 560 recorded
    Index: 580 recorded
    Index: 600 recorded
    Index: 620 recorded
    Index: 640 recorded
    Index: 660 recorded
    Index: 680 recorded
    Index: 700 recorded
    Index: 720 recorded
    Index: 740 recorded
    Index: 760 recorded
    Index: 780 recorded
    Index: 800 recorded
    Index: 820 recorded
    Index: 840 recorded
    Index: 860 recorded
    Index: 880 recorded
    Index: 900 recorded
    Index: 920 recorded
    Index: 940 recorded
    Index: 960 recorded
    Index: 980 recorded
    Index: 1000 recorded
    Index: 1020 recorded
    Index: 1040 recorded
    Index: 1060 recorded
    Index: 1080 recorded
    Index: 1100 recorded
    Index: 1120 recorded
    Index: 1140 recorded
    Index: 1160 recorded
    Index: 1180 recorded
    Index: 1200 recorded
    Index: 1220 recorded
    Index: 1240 recorded
    Index: 1260 recorded
    Index: 1280 recorded
    Index: 1300 recorded
    Index: 1320 recorded
    Index: 1340 recorded
    Index: 1360 recorded
    Index: 1380 recorded
    Index: 1400 recorded
    Index: 1420 recorded
    Index: 1440 recorded
    Index: 1460 recorded
    Index: 1480 recorded
    Index: 1500 recorded
    Index: 1520 recorded
    Index: 1540 recorded
    Index: 1560 recorded
    Index: 1580 recorded
    Index: 1600 recorded
    Index: 1620 recorded
    Index: 1640 recorded
    Index: 1660 recorded
    Index: 1680 recorded
    Index: 1700 recorded
    Index: 1720 recorded
    Index: 1740 recorded
    Index: 1760 recorded
    Index: 1780 recorded
    Index: 1800 recorded
    Index: 1820 recorded
    Index: 1840 recorded
    Index: 1860 recorded
    Index: 1880 recorded
    Index: 1900 recorded
    Index: 1920 recorded
    Index: 1940 recorded
    Index: 1960 recorded
    Index: 1980 recorded
    Index: 2000 recorded
    Index: 2020 recorded
    Index: 2040 recorded
    Index: 2060 recorded
    Index: 2080 recorded
    Index: 2100 recorded
    Index: 2120 recorded
    Index: 2140 recorded
    Index: 2160 recorded
    Index: 2180 recorded
    Index: 2200 recorded
    Index: 2220 recorded
    Index: 2240 recorded
    Index: 2260 recorded
    Index: 2280 recorded
    Index: 2300 recorded
    Index: 2320 recorded
    Index: 2340 recorded
    Index: 2360 recorded
    Index: 2380 recorded
    Index: 2400 recorded
    Index: 2420 recorded
    Index: 2440 recorded
    Index: 2460 recorded
    Index: 2480 recorded
    -------------------
    All Calculations Complete
    Complete

### Manual Overwrite

If you interrupt the previous cell, you MUST run this cell. Occassionally, if the interruption occurs during the $\texttt{to_csv}$ command, the external .csv will be blank. However, the dataframe is still stored in Jupyter memory. This cell ensures that the dataframe in Jupyter memory gets written to an external .csv file.

```python
print ('Working Sommerfeld file: {0}'.format(sommWriteFile))

overwrite = input('Overwrite? (y/n): ')
if ((overwrite == 'y') or (overwrite == 'Y')):
    sommDataIn.to_csv(sommWriteFile, index=False)
    print ('Overwrite complete')

else:
    print ('Overwrite aborted')

print ('Complete')
```

    Working Sommerfeld file: testSomm.csv
    Overwrite? (y/n): n
    Overwrite aborted
    Complete

# Generating Equilibrium Plots

## Data Input

### Read In Sommerfeld Data and Interpolate

The following cell reads in an external Sommerfeld file and interpolates the discrete set of $\langle S_s (m_A) \rangle$ into a continuous function. This interpolated function is what will be called during the calculation of the equilibrium time instead of calling the Sommerfeld function explicitly. We also determine the correct units of $m_X$ to display once plotted.

```python
from scipy.interpolate import interpolate

sommerfeldFile = input('Sommerfeld Data File: ')
readFile = 'Data_Files/Sommerfeld/'+sommerfeldFile
# readFile = photonSommerfeldPath(sommerfeldFile)

dataIn = pd.read_csv(readFile, sep = ',')

Filem_X = float(dataIn.at[0,'mX[GeV]'])
FileKappa0 = float(dataIn.at[0,'Kappa0[GeV5]'])

###############################
# Set the correct units for m_X
###############################
m_XUnit = 'None'

if (Filem_X < 1000):
    m_XDisplay = Filem_X
    m_XUnit = 'GeV'

if (Filem_X >= 1000):
    m_XDisplay = Filem_X*10**-3
    m_XUnit = 'TeV'

# Double Check to make sure the file name matches the read-in value
print ('Input m_X: {0} GeV'.format(Filem_X))

###############################
# Interpolate Sommerfeld and alpha_X
###############################
maList = dataIn['mA[GeV]']
sommerfeldList = dataIn['ThermAvgSommerfeld']
SommerfeldInterp = interpolate.interp1d(maList, sommerfeldList)

print ('Compete')
```

    Sommerfeld Data File: testSomm.csv
    Input m_X: 100.0 GeV
    Compete

## Initialize Contour Dictionary

Enter 5 unique contour levels. Note that the contour lines you enter "$\mathrm{L}$" correspond to a contour level of $\frac{\tau}{\tau_{\oplus}} = 10^\mathrm{L}$.

These values are stored as the keys to "$\texttt{contourDictionary}$." By default, the values of this dictionary are an array composed of two sub-arrays. The first sub-array will store the contour lines without Sommerfeld and the second sub-array will store the contour lines with Sommerfeld.

```python
print ('Enter 5 countour lines in ascending or descending order: ')

#contourDictionary = {key: value}
# "key" is the actual contour level
# "value" is an array of plot values: [[],[]]
# The first one is with sommerfeld off, the second array with sommerfeld on.

contourDictionary={}
for i in range(1,6):
    key = (input('Contour Level {0}: '.format(i)))
    type(float(key)) != type(float(0)) # This line ensures that the values entered are numeric
    contourDictionary.update({'{0}'.format(key): [[],[]] } )

print ('Complete')
```

    Enter 5 countour lines in ascending or descending order:
    Contour Level 1: -4
    Contour Level 2: -2
    Contour Level 3: 0
    Contour Level 4: 2
    Contour Level 5: 4
    Complete

## Populate the Contour Dictionary

The following cell populates $\texttt{contourDictionary}$ with the contour values.

```python
for key in contourDictionary:
    contourDictionary[key] = [[],[]]

##################
# Initialize the x-axis
# 0.01 GeV < m_A < 10 GeV
##################
maMin = min(maList)
maMax = max(maList)

maRange = np.logspace(np.log10(maMin), np.log10(maMax), 500, base = 10)

##################
# Populate contourDictionary with plot values
##################
maArray = []

for maTemp in maRange:
    maArray.append(maTemp)

for key in contourDictionary:
    contourline = float(key)

    for mATemp in maRange:
        Alpha_X    = DP.alphaTherm(Filem_X, mATemp)
        SigmaVTree = DP.sigmaVtree(Filem_X, mATemp, Alpha_X)
        Sommerfeld = SommerfeldInterp(mATemp)
        CannNoSomm = DP.cAnn(Filem_X, SigmaVTree)

        EpsilonNoSomm = DP.contourFunction(mATemp, Alpha_X, CannNoSomm, 1, FileKappa0, contourline)
        EpsilonSomm   = DP.contourFunction(mATemp, Alpha_X, CannNoSomm, Sommerfeld, FileKappa0, contourline)

        contourDictionary[key][0].append(EpsilonNoSomm)
        contourDictionary[key][1].append(EpsilonSomm)

print ('Complete')
```

    Complete

## Plot Produciton

The following cell produces a single plot of the equilibrium time. You may choose to turn the Sommerfeld enhancement "on" or "off" by following the prompt.

```python
import matplotlib as mpl
import matplotlib.pyplot as plt
colorScheme1 = mpl.cycler(color=['#c7c47e','#94ab72','#428583','#40379f','#3d0b7e'])

fig1 = plt.figure(figsize = (6,6))

sommParam = int(input('Sommerfeld on? No:0 Yes:1 '))
assert ((sommParam == 0) or (sommParam == 1)), 'Invalid Input, must be 0 or 1.'

########################################
# This uses a custom color scheme defined in Masterfunctions_Graphing
########################################
colors = colorScheme1
mpl.rcParams['axes.prop_cycle'] = colors

########################################
# Produce the contour lines
########################################
testFill = False
for key in contourDictionary:
    if (float(key) == 0):
        testFill = True
    assert (len(maRange) == len(contourDictionary[key][sommParam])), 'ContourDictionary is probably not populated'
    plt.plot(np.log10(maRange), (contourDictionary[key][sommParam]), label = r'10^{0}'.format(key))

########################################
# Shade the valid region of parameter space
########################################
if (testFill == True):
    xs = np.log10(maArray)
    y1 = contourDictionary['0'][sommParam]
    plt.fill_between(xs, y1, -5, color = '#E6FDFF')

########################################
# Define plot label stuff
########################################
plt.legend(loc = 'lower right', fontsize = 12)
plt.xlabel(r"$\log(m_{A'})$ $[GeV]$", fontsize = 14)
plt.ylabel(r'$\log(\varepsilon)$', fontsize = 14)

plt.title(r'$m_X = {0}$ {1}'.format(m_XDisplay, m_XUnit),\
          fontsize = 16,\
          loc = 'right')

plt.title(r'$\tau/\tau_{\oplus}$ Contours',\
          fontsize = 16,\
          loc = 'left')
plt.axis([np.log10(min(maRange)),np.log10(max(maRange)),-10,-5])
plt.grid()
plt.show()

print ('Complete')
```

### Save Figure

```python
filename = input('Figure Name: ')
assert (filename != ''), 'No Filename'
fig1.savefig((filename), dpi = 700)

print ('Complete')
```

    Figure Name: contour1.png
    Complete

# Generating IceCube Signal Plots

## Recipie

The recipie for generating the Signal File is:

1). Define the resolution of the signal plot

2). Check this resolution by reading in a Sommerfeld file and Branching Ratio file

3). Initialize a template dataframe to hold the calculated signal rate data

3). Write that dataframe to an external .csv file (signalData.csv)

4). Append values from the Sommerfeld and Branching Ratio file to signalData.csv

5). Loop through signalData.csv, calculate $C_{Cap}$, $C_{Ann}$, $\Gamma_{Ann}$, and $N_{Sig}$, append them to signalData.csv

Step five can be quite lengthy depending on the desired resolution of the data points. Typically, one would begin these calculations and be forced to wait until they are completed in their entirety; perhaps unable to shut down their machine for hours at a time. The cell "Signal File Calculations" however can be interrupted at any time, and resumed later. This allows the user to run the calculation in multiple stages instead of dedicating a large chunk of time and running the calculation all at once.
The calculation can be stopped by interrupting the Jupyter Kernel. Be sure to run the "Manual Overwrite" cell just to be sure that the calculations are saved.

## Define the Resolution of the Signal Plot

These cells create the "grid" of data points for which $N_{\text{sig} }$ will be calculated.

This cell initializes the resolution of the interpolations for the Sommerfeld enhancement and the reach plot. The variable $\texttt{num}$ defines the resolution of the grid, so if $\texttt{num} = 50$, the reach plot will have a $50 \times 50$ resolution for a total of 2500 calculations. The suggested resolution is

    num = 200

A way to understand what this number does: "The cleaner your Sommerfeld interpolation, the cleaner your reach plots will be." Making this number larger will increase the resolution of the interpolation, cleaning up the plots, but it will take longer to calculate.

A good rule of thumb: Your reach plot resolution will be determined by the resolution of the Sommerfeld Interpolation

```python
# num = 200
num = 200

print ('Complete')
```

    Complete

## External File Interpolations

The following cells read in and create interpolations of the:

1). Sommerfeld Enhancement

2). Branching ratio

The file names for both these files will be written to the signalData.csv file. The reasoning behind this is continuity of the calculation. If you run this notebook from top to bottom, you can calculate everything in one go. However, if you interrup the main calculation cell and restart your computer, all of the vairables we have loaded into Jupyter memroy get wiped. Writing these file names into the signalData.csv file means that to resume the calculation, you only need to run the main calculation cell as opposed to loading variables into Jupyter memory before starting the main calculation

### Sommerfeld Data Interpolation

This interpolation only serves to give you an idea of the resolution of the reach plot. We will read this in and interpolate it again later in the actual calulation.

Implement some sort of separate plot of the original sommerfeld data so we can compare the interpolated value using num with the original graph

```python
import pandas as pd

####################################
# Define Sommerfeld and Signal Files
####################################
SommFileName = input('Sommerfeld File: ')
SommFile = pd.read_csv('Data_Files/Sommerfeld/'+SommFileName)
# SommFile = pd.read_csv(photonSommerfeldPath(SommFileName))

print ('Complete')
```

    Sommerfeld File: testSomm.csv
    Complete

### Plot Interpolation vs Original data

```python
from scipy.interpolate import interpolate
import matplotlib.style
import matplotlib as mpl
mpl.rcParams.update(mpl.rcParamsDefault)

%matplotlib inline

####################################
# Interpolate Sommerfeld data
####################################
mAData = SommFile['mA[GeV]']
SommData = SommFile['ThermAvgSommerfeld']
SommInterp = interpolate.interp1d(mAData,SommData)

####################################
# Create Comparison Plots
####################################
m_XSommerfeld = SommFile.loc[0,'mX[GeV]']
kappa0Sommerfeld = SommFile.loc[0,'Kappa0[GeV5]']

vect1 = []
vect2 = []
for ma in np.logspace(np.log10(min(mAData)),np.log10(max(mAData)), num, base = 10):
    vect1.append(ma)
    vect2.append(SommInterp(ma))

fig1 = plt.figure(figsize = (14,6))
plt.suptitle('Comparison of Sommerfeld Resolutions', fontsize = 18)

ax1 = fig1.add_subplot(1,2,1)
ax1.plot(np.log10(vect1),np.log10(vect2))
ax1.set_xlabel(r"$\log(m_{A'})$ [GeV]", fontsize = 14)
ax1.set_ylabel(r"$\log(\langle S_S \rangle)$", fontsize = 14)
ax1.set_title('num = {0} Resolution'.format(num), fontsize = 14, loc = 'right')
plt.grid()

ax2 = fig1.add_subplot(1,2,2)
ax2.plot(np.log10(mAData), np.log10(SommData))
ax2.set_xlabel(r"$\log(m_{A'})$ [GeV]", fontsize = 14)
ax2.set_ylabel(r"$\log(\langle S_S \rangle)$", fontsize = 14)
ax2.set_title('num = 2500 Resolution', fontsize = 14, loc = 'right')
plt.grid()

print ('Complete')
```

### Branching Ratio Interpolation

In this cell, we read in a .csv file containing discrete branching ratio data points obtained from WebPlotDigitizer and interpolate them into a continuous funciton named $\texttt{branchRatioInterp}$.

By decault, this cell reads in the branching ratio for $\gamma \rightarrow e^+e^-$. To change this, un-comment the first line and comment out the second line.

```python
branchFileName = input('Branching File: ')
# branchFileName = 'brtoe.csv'

BranchData = pd.read_csv(('DataFiles/Branching_Ratio/'+branchFileName))

xListBranch = BranchData['mA[GeV]']
yListBranch = BranchData['BR']

branchRatioInterp = interpolate.interp1d(xListBranch,yListBranch)

fig = plt.figure(figsize = (6,6))
plt.plot(xListBranch,yListBranch)
plt.xlabel(r"$m_{A'}$ [GeV]", fontsize = 14)
plt.ylabel(r"Branching Ratio", fontsize = 14)
plt.suptitle('Branching Ratio Interpolation Visualization', fontsize = 16)
plt.title('num = {0}'.format(num), fontsize = 14, loc = 'right')
plt.grid()

print ('Complete')
```

## Initialize Signal Dataframe

This cell:

1) Initializes the signal data frame

2) Appends the $\varepsilon$ and $m_{A'}$ data points based on the resolution dictated by the variable $\texttt{num}$. By default, the range for mediator mass and kinetic mixing are dictated by the range of $m_{A'}$ in the Sommerfld file, and $10^{-11} \leq \varepsilon \leq 10^{-5}$. To change the range of $\varepsilon$, adjust the arguments of the variable ``\texttt{epsilonLogRange}.''

3) Appends the column headers to the data frame

```python
#########################################
# Define parameter space Ranges
#########################################
# Logarithmic Spacing because we plot this in log-log space
# 0.01 GeV < mA < 10 GeV
# 10^-11 < Epsilon < 10^-5

maMin = (min(mAData))
maMax = (max(mAData))

mALogRange = np.logspace(np.log10(maMin), np.log10(maMax), num, base = 10)
epsilonLogRange = np.logspace(-11, -5, num, base = 10)

#########################################
# Initialize Master Arrays
#########################################
masterSignalArray = []

#########################################
# Populate the Master Arrays with the column headers and 'None' Cells
#########################################
counter = 0
for mATemp in mALogRange:
    for epsilonTemp in epsilonLogRange:
        masterSignalArray.append([counter, ' ', mATemp, epsilonTemp, \
                                  'None', 'None', 'None', 'None', \
                                  ' ', ' ',' ', ' '])
        counter +=1
#########################################
# Create a PANDAS DataFrame from the Master Arrays
#########################################
masterSignalDataframe = pd.DataFrame(data = masterSignalArray, \
    columns=['Counter', 'mX[GeV]', 'mA[GeV]', 'Epsilon',\
             'Capture[s-1]', 'Annihilation[s-1]', 'GammaAnn[s-1]', 'IceCubeSignal', \
             'SommerfeldFile', 'BranchingRatioFile','Resolution','Kappa0[GeV5]'])

print ('Complete')
```

    Complete

## Write to External .csv File

This cell writes the signal data frame to an external csv file.

```python
SignalFile = input('Signal Rate filename: ')
writeFile = 'Data_Files/Signal/'+SignalFile
assert (SignalFile[-4:] == '.csv'), 'Must inclue ".csv" in the file name'

#############################
# Double check that these values are correct
#############################
print ('--------------------')
print ('You are about to commit the following to {0}'.format(SignalFile))
print ('mX[GeV]             : {0}'.format(m_XSommerfeld))
print ('Kappa0[GeV5]        : {0}'.format(kappa0Sommerfeld))
print ('Sommerfeld File     : {0}'.format(SommFileName))
print ('Branching Ratio File: {0}'.format(branchFileName))
print ('Resolution          : {0}'.format(num))

print ()

yesNo = input ('Commit to .csv? (y/n)')
assert (yesNo == 'y' or yesNo == 'n'), 'Invalid input, must be "y" or "n."'

if ((yesNo == 'y') or (yesNo == 'Y')):
    kappa0Value = 'None'
    m_XValue = 'None'

    masterSignalDataframe.at[0, 'mX[GeV]']            = m_XSommerfeld
    masterSignalDataframe.at[0, 'Kappa0[GeV5]']       = kappa0Sommerfeld
    masterSignalDataframe.at[0, 'SommerfeldFile']     = SommFileName
    masterSignalDataframe.at[0, 'BranchingRatioFile'] = branchFileName
    masterSignalDataframe.at[0, 'Resolution']         = num

    masterSignalDataframe.to_csv(writeFile, index=False)

    print ('Commit Executed')

else:
    print ('Commit aborted')

print ('Complete')
```

    Signal Rate filename: testSignal.csv
    --------------------
    You are about to commit the following to testSignal.csv
    mX[GeV]             : 100.0
    Kappa0[GeV5]        : 5.851659530104901e+27
    Sommerfeld File     : testSomm.csv
    Branching Ratio File: brtoe.csv
    Resolution          : 200

    Commit to .csv? (y/n)y

    C:\Users\Adam Green\Anaconda3\lib\site-packages\ipykernel_launcher.py:27: FutureWarning: set_value is deprecated and will be removed in a future release. Please use .at[] or .iat[] accessors instead
    C:\Users\Adam Green\Anaconda3\lib\site-packages\ipykernel_launcher.py:28: FutureWarning: set_value is deprecated and will be removed in a future release. Please use .at[] or .iat[] accessors instead
    C:\Users\Adam Green\Anaconda3\lib\site-packages\ipykernel_launcher.py:29: FutureWarning: set_value is deprecated and will be removed in a future release. Please use .at[] or .iat[] accessors instead
    C:\Users\Adam Green\Anaconda3\lib\site-packages\ipykernel_launcher.py:30: FutureWarning: set_value is deprecated and will be removed in a future release. Please use .at[] or .iat[] accessors instead
    C:\Users\Adam Green\Anaconda3\lib\site-packages\ipykernel_launcher.py:31: FutureWarning: set_value is deprecated and will be removed in a future release. Please use .at[] or .iat[] accessors instead

    Commit Executed
    Complete

----------------------

Run these two cells onces you have a working signalData.csv file and you want to either start or resume calculations.

## SignalData.csv Calculations

This cell scans through all the lines of signalData.csv file and calculates:

1). Capture

2). Annihilation

3). GammaAnn

4). Signal

As mentioned previously, this cell may be stopped, by interrupting the kernel, at any time and resumed later.

```python
import pandas as pd
from scipy.interpolate import interpolate
from datetime import datetime
#########################################
# Read in Signal File
#########################################
signalFileName = input('Signal Rate File: ')
signalWriteFile = 'Data_Files/Signal/'+signalFileName
signalDataIn = pd.read_csv(signalWriteFile, sep = ',')

#########################################
# Create backup of Signal File
#########################################
now = datetime.now()
timeStamp = now.strftime('%b-%d_%H.%M')
signalFileNameCopy = signalFileName[:-4] + '_'+timeStamp+'.csv'
signalDataIn.to_csv(('Data_Files/Signal/Backups/'+signalFileNameCopy), sep=',', index=False)
print ('Backup Created')

#########################################
# Constants in (mA, epsilon) space:
#########################################
Filem_X = float(signalDataIn.at[0, 'mX[GeV]'])
FileKappa0 = float(signalDataIn.at[0, 'Kappa0[GeV5]'])
Alpha = 1./137
LiveTime = DarkCapPy.Configure.Conversions.yr2s(10)

#########################################
# Sommerfeld Interpolation
#########################################
sommFileName = signalDataIn.at[0,'SommerfeldFile']
sommDataIn = pd.read_csv('Data_Files/Sommerfeld/'+sommFileName)
# sommDataIn = pd.read_csv(sommFileName)
mAData1 = sommDataIn['mA[GeV]']
SommDataValues = sommDataIn['ThermAvgSommerfeld']
SommInterp = interpolate.interp1d(mAData1,SommDataValues)

#########################################
# Branching Ratio Interpolation
#########################################
branchFileName = signalDataIn.at[0,'BranchingRatioFile']
# branchDataIn = pd.read_csv(photonBranchPath(branchFileName))
branchDataIn = pd.read_csv('Data_Files/Branching_Ratio/'+branchFileName)

mABranch = branchDataIn['mA[GeV]']
branchingDataValues = branchDataIn['BR']

branchRatioInterp = interpolate.interp1d(mABranch,branchingDataValues)

#########################################
# Begin the loop calculations
#########################################
loopRange = range(0,len(signalDataIn['Counter']))
print ()
print ('--------------------')
finishedCounter = 0
index = 0
for index in loopRange:
    testValue = signalDataIn.loc[index, 'IceCubeSignal']

    if (testValue == 'None'):
        #########################################
        # Initialize loop-specific Quantities
        #########################################
        Filem_ATemp = float(signalDataIn.at[index, 'mA[GeV]'])
        FileEpsilonTemp = float(signalDataIn.at[index, 'Epsilon'])
        Alpha_X = DP.alphaTherm(Filem_X, Filem_ATemp)
        Sommerfeld = SommInterp(Filem_ATemp)
        BranchingRatio = branchRatioInterp(Filem_ATemp)

        #########################################
        # Parameter Spce Calculations
        #########################################
        Capture = DP.cCapQuick(Filem_X, Filem_ATemp, FileEpsilonTemp, Alpha_X, FileKappa0)
        SigmaVTree = DP.sigmaVtree(Filem_X, Filem_ATemp, Alpha_X)
        Annihilation = DP.cAnn(Filem_X, SigmaVTree, thermAvgSomm = Sommerfeld)
        GammaAnn = DP.gammaAnn(Capture, Annihilation)
        L = DP.decayLength(Filem_X, Filem_ATemp, FileEpsilonTemp, BranchingRatio)
        EpsilonDecay = DP.epsilonDecay(L)

        IceCubeSignal = DP.iceCubeSignal(GammaAnn, EpsilonDecay, LiveTime)

        #########################################
        # Append Calculated Values to SignalFile.csv
        #########################################
        signalDataIn.at[index, 'Capture[s-1]']      = Capture
        signalDataIn.at[index, 'Annihilation[s-1]'] = Annihilation
        signalDataIn.at[index, 'GammaAnn[s-1]']     = GammaAnn
        signalDataIn.at[index, 'IceCubeSignal']     = IceCubeSignal

        #########################################
        # Overwrite SignalFile.csv
        #########################################
        signalDataIn.to_csv(signalWriteFile, sep=',',index=False)
        finishedCounter += 1

        if (index % 20 == 0):
            print ('Calculations for index: {0} recorded'.format(index))

    elif (testValue != 'None'):
        finishedCounter += 1

    if (finishedCounter == len(signalDataIn['Counter'])):
        print ('--------------------')
        print ('All Calculations Complete')
        break

print ('Complete')
```

## Manual Overwrite

This cell is a manual overwrite of the signalData.csv file.

It is REQUIRED that you run this if you interrupt the calculation. Failure to do so could result in a loss of all your data.

```python
print ('Working Signal File: {0}'.format(signalWriteFile))

overwrite = input('Overwrite? (y/n): ')
if ((overwrite == 'y') or (overwrite == 'Y')):
    signalDataIn.to_csv(signalWriteFile, sep=',',index=False)
    print ('Overwrite complete')

else:
    print ('Overwrite aborted')

print ('Complete')
```

    Working Signal File: testSignal.csv
    Overwrite? (y/n): y
    Overwrite complete
    Complete

## Plot Signal Events

The general recipie is:

1). Read in a Signal Data file

2). Extract the plot information

3). Plot

## Read in External Signal File

```python
signalFile = input('Signal Rate filename: ')
# readFile = photonSignalDataPath_Complete(GammaFile)
readFile = signalFile
testRead = pd.read_csv(readFile) # This is here to ensure that the file exists

print ('Complete')
```

    Signal Rate filename: 100GeVSignal.csv
    Complete

## Extract Plot Data

We plot the Data under the column

    mA[GeV], Epsilon, IceCubeSignal

```python
# rawData = pd.read_csv(photonSignalDataPath_Complete(GammaFile))
rawData = pd.read_csv(signalFile)
Filem_X = float(rawData.loc[0,'mX[GeV]'])

#Pull out min / max values on each axis
xMin = min(rawData['mA[GeV]'])
xMax = max(rawData['mA[GeV]'])
yMin = min(rawData['Epsilon'])
yMax = max(rawData['Epsilon'])

#########################################
# Define the dimension of the grid.
# For an n x n grid, this will pull out the number "n"
#########################################
gridDim = int(rawData.loc[0,'Resolution'])

#########################################
# Create the n x n grid
#########################################
# Logarithmic Spacing
xLogRange = np.logspace(np.log10(xMin), np.log10(xMax) , gridDim, base = 10)
yLogRange = np.logspace(np.log10(yMin), np.log10(yMax), gridDim, base = 10)

#########################################
# Read in and reshape the Contour data
#########################################
# This reads in contour data as a single column,
#     but we reshape it to a 2D, n x n array to fit the grid

z = [rawData['IceCubeSignal']]
SignalData = np.reshape(z, (len(xLogRange), len(yLogRange)))

print ('Complete')
```

    Complete

## Plot the Signal Events

```python
###############################
# Set the correct units for m_X
###############################
m_XUnit = 'None'
if (Filem_X < 1000):
    m_XDisplay = Filem_X
    m_XUnit = 'GeV'

if (Filem_X >= 1000):
    m_XDisplay = Filem_X * 10**-3
    m_XUnit = 'TeV'

###############################
# Plotting Stuff
###############################
fig0 = plt.figure(figsize = (6,6))

CP = plt.contour(np.log10(xLogRange), np.log10(yLogRange), \
                 np.log10(SignalData.transpose()),\
                 levels=[0,1,2,3],\
                )

xLabelNums = [-2,-1,0,1]
xLabelText = [r'$10^{-2}$',r'$10^{-1}$',r'$10^{0}$',r'$10^{1}$']

yLabelNums = [-5,-6,-7,-8,-9,-10,-11]
yLabelText = [r'$10^{-5}$', r'$10^{-6}$', r'$10^{-7}$', r'$10^{-8}$', \
              r'$10^{-9}$', r'$10^{-10}$', r'$10^{-11}$']

plt.xticks(xLabelNums, xLabelText, fontsize = 12)
plt.yticks(yLabelNums, yLabelText, fontsize = 12)

plt.xlabel(r"$m_{A'}$ $[GeV]$", fontsize = 14)
plt.ylabel(r'$\varepsilon$', fontsize = 18)
plt.suptitle('IceCube Events', fontsize = 16)
plt.title(r'$m_X$ = {0} {1}'.format(m_XDisplay, m_XUnit), loc = 'right', fontsize = 13)

labels = [r'1', r'$10$',r'$10^2$',r'$10^3$']
for i in range(len(labels)):
    CP.collections[i].set_label(labels[i])

plt.legend(loc='upper right', fontsize = 13)
plt.grid()
print ('Complete')
```

### Save Figure

```python
figName = input('Filename: ')

fig0.savefig((figName), dpi = 700,
#              bbox_inches = 'tight', \
#              pad_inches= 1\
            )
print ('Complete')
```

```python

```