```python
%matplotlib inline

import numpy as np
import pandas as pd

import pymc3 as pm

import matplotlib.pyplot as plt
import seaborn as sns
import arviz as az

import warnings
warnings.filterwarnings('ignore')
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

## Archimedes procedure for porous material density determination

Experimental procedure is as follow, for further detail refer to standard ASTM D7263.

1. Measure temperature of liquid to determine its ($\rho_{\text{water}}$). Find object theoretical full density ( $\rho_{\text{solid}}$ ) and wax density ($\rho_{wax}$) from datasheet
2. Measure mass of sample(s) ($M_t$)
3. Apply two coats of wax to the sample(s)
4. Measure mass of Coated sample(s) ($M_c$)
5. Measure mass of Coated Sample(s) submerged ($M_{sub}$)

\begin{equation}
    \text{density}, \rho_b = \frac{M_t}{\frac{(M_c - M_{sub})}{\rho_{water}} - \frac{(M_c - M_t)}{\rho_{wax}}}
\end{equation}

\begin{equation}
    \text{true density}, \pi_d = \frac{ \rho_b}{\rho_{\text{solid}}} * 100
\end{equation}

Parts using two powder type are made. Parts labeled bimodal should have a higher density then parts labelled unimodal.

## Display first few row of excel sheet

```python
data = pd.read_excel('./data/green_sample_benchmark.xlsx', sheet_name = 'wax', index_col= [0,1])
oil_density = pd.read_excel('./data/green_sample_benchmark.xlsx', sheet_name = 'wax_param', index_col= 0)
```

```python
data.head()
```

      powder type
      dry weight
      coated weight
      wet coated
      exp
      operator

      batch
      ID

      2
      3
      Bimodal
      16.1233
      16.7750
      12.6154
      1
      1

      4
      Bimodal
      15.6583
      16.3665
      12.1851
      1
      1

      1
      1
      Unimodal
      14.9923
      16.7300
      11.3864
      1
      1

      5
      Unimodal
      15.5877
      16.7838
      12.0242
      1
      1

      4
      2
      Unimodal
      14.8018
      15.4759
      11.1873
      2
      1

## Display first few row of excel sheet

```python
oil_density.head()
```

      wax density
      water density

      exp

      1
      0.961
      0.997300

      2
      0.930
      0.998020

      3
      0.930
      0.998203

## Carrier fluid density

Density of the carrier fluid ($\rho_{\text{water}}$) is necessary for density calculation. Since the density of the fluid fluctuate depending on temperature and temperature varies from experiment to experiment, it is recorded for each experiment. For experiment 2 it was recorded at the beginning and at the end.

```python
def plot_hist(data, b):
    fig, axes = plt.subplots(1,3)
    axes[0].hist(data['dry weight'], bins = b, density = True)
    axes[0].set_title('dry weight')
    axes[0].set_xlabel('weight (g)')
    axes[0].set_ylabel('%')

    axes[1].hist(data['wet coated'], bins = b, density = True)
    axes[1].set_title('wet weight')
    axes[1].set_xlabel('weight (g)')
    axes[1].set_ylabel('%')

    axes[2].hist(data['coated weight'], bins = b, density = True)
    axes[2].set_title('coated weight')
    axes[2].set_xlabel('weight (g)')
    axes[2].set_ylabel('%')

    plt.tight_layout()
    plt.show()

plot_hist(data, 20)
```

```python
data = data.dropna()
```

## Compute density
Each sample was measured 3 times at each stage. Since we are only interested in the uncertainty in the true density ($\pi_d$), a cartesian product of each measured weights (dry, wet and coated) and liquid density is done for each sample in order to compute true density.

```python
from itertools import product
from sklearn import preprocessing

density_df = pd.DataFrame(columns = ['batch', 'id', 'density', 'operator', 'powder'])

# density of stainless steel 316L (g/cc) taken from data sheet
den_ss316 = 8.0

unique_id = 0

# iterate over each sample to do cartesian product
for idx in data.index.unique():
    sample = data.loc[idx]

    op = int(np.unique(sample['operator']).squeeze())
    powder = str(np.unique(sample['powder type']).squeeze())
    batch = idx[0]

    d_weight = sample['dry weight'].to_numpy()
    wc_weight = sample['wet coated'].to_numpy()
    c_weight = sample['coated weight'].to_numpy()

    exp_num = sample['exp']

    # density of liquid and wax taken from datasheet
    water_den = np.unique(oil_density.loc[exp_num]['water density'].to_numpy())
    wax_den = np.unique(oil_density.loc[exp_num]['wax density'].to_numpy())

    # cartesian product
    prod = product(d_weight, wc_weight, c_weight, water_den, wax_den)

    for d_w, wc_w, c_w, water_d, wax_d in prod:
        #compute true density refer to equation on top
        den = (d_w/((c_w - wc_w)/water_d - (c_w - d_w)/wax_d))/den_ss316*100
        new_entry = {
            'batch' : batch,
            'id' : unique_id,
            'density' : float(den.squeeze()),
            'operator' : op,
            'powder' : powder
        }

        density_df = density_df.append(new_entry, ignore_index = True)

    unique_id += 1
```

```python
ax = sns.histplot(density_df['density'])
```

```python
density_df.to_csv('./data/density_wax.csv', index=False)
```

```python
basic_model = pm.Model()

n_part = unique_id

powder_mean = density_df.groupby('powder').mean()['density']

part_id = density_df['id'].to_numpy().astype(np.int32)
obs_den = density_df['density'].to_numpy().astype(np.int32)

# Relabel data to integers
powder_enc = preprocessing.LabelEncoder()
powder_enc.fit(['Unimodal','Bimodal'])
powder_data = powder_enc.transform(density_df['powder'])

#compute starting point for mean inference
test_val_mu = [powder_mean[c] for c in powder_enc.classes_]

A = 10000
with basic_model:

    # setup independent mu_i
    mu_p = pm.Uniform('muP', 20.0, 60.0, shape = 2, testval = test_val_mu)
    mu_real = pm.math.switch(powder_data, mu_p[1], mu_p[0])

    # setup prior for part variance and error variance
    sig_repeat = pm.HalfCauchy('sig_repeat', 25)
    sig_part = pm.HalfCauchy('sig_part', 25)

    # each part will vary independently (nesting relationship)
    P_t = pm.Normal('P_t', mu = 0, sigma = 1, shape = n_part)

    # setup mu
    mu = pm.Deterministic('mu', mu_real + P_t[part_id]*sig_part)

    # add error and data to model
    density = pm.Normal('density', mu = mu, sigma = sig_repeat, observed = obs_den)

```

```python
with basic_model:
    trace = pm.sample(A, chains = 4, tune = 2000, target_accept=0.9)
    prior = pm.sample_prior_predictive()
    posterior_predictive = pm.sample_posterior_predictive(trace)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [P_t, sig_part, sig_repeat, muP]
    Sampling 4 chains, 0 divergences: 100%|█████████████████████████████████████| 48000/48000 [00:37<00:00, 1276.94draws/s]
    The number of effective samples is smaller than 25% for some parameters.
    100%|██████████████████████████████████████████████████████████████████████████| 40000/40000 [00:35<00:00, 1141.96it/s]

Convert trace to arviz data format for diagnostic

```python
trace_data_az = az.from_pymc3(
    trace=trace,
    prior=prior,
    posterior_predictive=posterior_predictive,
    model=basic_model
)
```

```python
trace_data_az.to_netcdf('./data/wax_trace.nc')
```

    './data/wax_trace.nc'

## Trace plot
The sampler is exploring the probability space by sampling different values of the model input parameters ($\mu_i, \sigma_{\text{part}}, \sigma_{\text{gage}}$). The left graph shows the probability density of the sampled values. The right graph shows the sampled with respect to sample number. The left graph is computed by summing up the right graph. When the left graph looks skewed towards zero or the right graph does not resemble a thick caterpillar, this indicates a biased sampling and a lack of convergence.

```python
pm.traceplot(trace, var_names=['muP','sig_part','sig_repeat'])
plt.show()
```

## Predictive check plot

Predictive posterior plot overlays data generated by the model's posterior over the real data. This is useful to check the fitness of the model.

```python
az.plot_ppc(trace_data_az, data_pairs={"density":"density"}, kind = 'scatter')
plt.show()
```

## Forest plot

Forest plot on estimated real mean $\mu_{i}$ to check if the values make sense

```python
pm.plots.forestplot(trace, var_names=['muP'])
plt.show()
```

## Compute split $\hat{R}$

In the Markov-Chain Monte Carlo (MCMC) sampler, the sampling is done simultaneously in multiple processes (chain). The rank normalized $\hat{R}$ checks for convergence by comparing the variance between chain with the variance within chain. If convergence has been achieved then the variance should be the same ($\hat{R}=1$) (Vehtari et al). [Vehtari et al.](https://arxiv.org/pdf/1903.08008v1.pdf "vehtari") recommends a $\hat{R} < 1.01$.

```python
az.rhat(trace_data_az)
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

<xarray.Dataset>
Dimensions:     (P_t_dim_0: 31, muP_dim_0: 2, mu_dim_0: 213)
Coordinates:
  * P_t_dim_0   (P_t_dim_0) int32 0 1 2 3 4 5 6 7 8 ... 23 24 25 26 27 28 29 30
  * muP_dim_0   (muP_dim_0) int32 0 1
  * mu_dim_0    (mu_dim_0) int32 0 1 2 3 4 5 6 7 ... 206 207 208 209 210 211 212
Data variables:
    P_t         (P_t_dim_0) float64 1.0 1.0 1.0 1.0 ... 1.001 1.002 1.001 1.001
    muP         (muP_dim_0) float64 1.001 1.002
    sig_repeat  float64 1.0
    sig_part    float64 1.0
    mu          (mu_dim_0) float64 1.0 1.0 1.0 1.0 ... 0.9999 0.9999 0.9999

xarray.Dataset

<input id='section-bbe07aab-3cf2-4537-b22f-ad7ab1edd3fa' class='xr-section-summary-in' type='checkbox' disabled >Dimensions:

P_t_dim_0: 31

muP_dim_0: 2

mu_dim_0: 213

<input id='section-586066df-3635-46c5-84af-14058355fad6' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (3)

P_t_dim_0

(P_t_dim_0)

int32

0 1 2 3 4 5 6 ... 25 26 27 28 29 30
<input id='attrs-353c3ca5-8434-4c19-b412-c44b87d53a48' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-1a644771-48c7-494a-a519-44d30e92c710' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30])

muP_dim_0

(muP_dim_0)

int32

0 1
<input id='attrs-97843f40-34d2-4faa-88a0-0334c2c93f38' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-a05d095b-0898-4881-a599-6e28b5f422ad' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([0, 1])

mu_dim_0

(mu_dim_0)

int32

0 1 2 3 4 5 ... 208 209 210 211 212
<input id='attrs-5cf862ef-1aca-4dd3-b656-ed7044a07ec7' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-ab2e65ca-9247-4587-9488-3bc814f2500d' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([  0,   1,   2, ..., 210, 211, 212])

<input id='section-3bbc51d6-55c2-4270-a1e0-faf7c8f7d15d' class='xr-section-summary-in' type='checkbox'  checked>Data variables: (5)

P_t

(P_t_dim_0)

float64

1.0 1.0 1.0 ... 1.002 1.001 1.001
<input id='attrs-22d4414a-4aba-44ec-aedf-6236f54bfad4' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-a4c65f0a-b761-470c-8a9c-08f9c3d64a68' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([1.00026733, 1.00003423, 1.00012612, 1.00022654, 1.00096337,
       1.00024172, 1.00122362, 1.00033709, 1.00018687, 1.00006123,
       1.00012166, 1.00013687, 1.00020056, 1.0002717 , 1.00122593,
       1.00123268, 1.00142097, 1.0012231 , 1.00016313, 1.00027703,
       1.00017211, 1.00020504, 1.00016763, 1.00029545, 1.00023676,
       1.00046128, 1.00107954, 1.00118772, 1.00161922, 1.00127543,
       1.00140505])

muP

(muP_dim_0)

float64

1.001 1.002
<input id='attrs-1a4e7853-e8b6-48ee-97ed-22d29d8794ff' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-735efc67-caa9-4a64-8f42-0117206d6f34' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([1.00069546, 1.00206514])

sig_repeat

()

float64

1.0
<input id='attrs-c45b897f-cf10-4c01-b905-83838f59e648' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-40794b3c-fac4-4d56-a14e-5873a3192e55' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array(1.00028024)

sig_part

()

float64

1.0
<input id='attrs-1e8208b9-2230-425b-a898-b3a6a95ae54d' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-fb886bc2-1b92-4ea8-b4b9-c71db60fa411' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array(1.000121)

mu

(mu_dim_0)

float64

1.0 1.0 1.0 ... 0.9999 0.9999
<input id='attrs-9399848b-8911-4977-83c8-bec9008ee152' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-a9245fff-77c8-477e-8dc5-b14a165faaa2' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([1.00030622, 1.00005919, 1.00007762, 1.00014645, 0.99994332,
       0.99994332, 0.99994332, 0.99994332, 0.99994332, 0.99994332,
       0.99994332, 0.99994332, 1.00004909, 1.00014958, 1.00014958,
       1.00014958, 1.00014958, 1.00014958, 1.00014958, 1.00014958,
       1.00014958, 1.00006133, 1.00006133, 1.00006133, 1.00006133,
       1.00006133, 1.00006133, 1.00006133, 1.00006133, 1.00001818,
       1.00001818, 1.00001818, 1.00001818, 1.00001818, 1.00001818,
       1.00001818, 1.00001818, 1.00013602, 1.00013602, 1.00013602,
       1.00013602, 1.00013602, 1.00013602, 1.00013602, 1.00013602,
       1.00017697, 1.00017697, 1.00017697, 1.00017697, 1.00017697,
       1.00017697, 1.00017697, 1.00017697, 1.00010736, 1.00010736,
       1.00010736, 1.00010736, 1.00010736, 1.00010736, 1.00010736,
       1.00010736, 1.00029036, 1.00029036, 1.00029036, 1.00029036,
       1.00029036, 1.00029036, 1.00029036, 1.00029036, 1.00012212,
       1.00012212, 1.00012212, 1.00012212, 1.00012212, 1.00012212,
       1.00012212, 1.00012212, 1.0000573 , 1.0000573 , 1.0000573 ,
       1.0000573 , 1.0000573 , 1.0000573 , 1.0000573 , 1.0000573 ,
       1.00019664, 1.00019664, 1.00019664, 1.00019664, 1.00019664,
       1.00019664, 1.00019664, 1.00019664, 1.00007433, 1.00007433,
       1.00007433, 1.00007433, 1.00007433, 1.00007433, 1.00007433,
...
       1.00005895, 1.00005895, 1.00015687, 1.00015687, 1.00015687,
       1.00015687, 1.00015687, 1.00015687, 1.00015687, 1.00015687,
       1.00008492, 1.00008492, 1.00008492, 1.00008492, 1.00008492,
       1.00008492, 1.00008492, 1.00008492, 1.0000055 , 1.0000055 ,
       1.0000055 , 1.0000055 , 1.0000055 , 1.0000055 , 1.0000055 ,
       1.0000055 , 1.00001072, 1.00001072, 1.00001072, 1.00001072,
       1.00001072, 1.00001072, 1.00001072, 1.00001072, 1.00015094,
       1.00015094, 1.00015094, 1.00015094, 1.00015094, 1.00015094,
       1.00015094, 1.00015094, 1.00003102, 1.00003102, 1.00003102,
       1.00003102, 1.00003102, 1.00003102, 1.00003102, 1.00003102,
       1.00024829, 1.00024829, 1.00024829, 1.00024829, 1.00024829,
       1.00024829, 1.00024829, 1.00024829, 1.00009565, 1.00009565,
       1.00009565, 1.00009565, 1.00009565, 1.00009565, 1.00009565,
       1.00009565, 1.00013317, 1.00013317, 1.00013317, 1.00013317,
       1.00013317, 1.00013317, 1.00013317, 1.00013317, 1.00023497,
       1.00023497, 1.00023497, 1.00023497, 1.00023497, 1.00023497,
       1.00023497, 1.00023497, 1.00027052, 1.00027052, 1.00027052,
       1.00027052, 1.00027052, 1.00027052, 1.00027052, 1.00027052,
       0.9999481 , 0.9999481 , 0.9999481 , 0.9999481 , 0.9999481 ,
       0.9999481 , 0.9999481 , 0.9999481 ])

<input id='section-815f545a-96f8-4a6d-b663-1e8ae26afbad' class='xr-section-summary-in' type='checkbox' disabled >Attributes: (0)

## Calculate highest density interval (HDI)

Taken from [arviz doc](https://arviz-devs.github.io/arviz/api/generated/arviz.hdi.html#arviz.hdi "arviz") : The HDI is the minimum width Bayesian credible interval (BCI).

```python
hdi_data = az.hdi(trace_data_az)
hdi_data.to_netcdf('./data/wax_inference.nc')
hdi_data
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

<xarray.Dataset>
Dimensions:     (P_t_dim_0: 31, hdi: 2, muP_dim_0: 2, mu_dim_0: 213)
Coordinates:
  * P_t_dim_0   (P_t_dim_0) int32 0 1 2 3 4 5 6 7 8 ... 23 24 25 26 27 28 29 30
  * muP_dim_0   (muP_dim_0) int32 0 1
  * mu_dim_0    (mu_dim_0) int32 0 1 2 3 4 5 6 7 ... 206 207 208 209 210 211 212
  * hdi         (hdi) <U6 'lower' 'higher'
Data variables:
    P_t         (P_t_dim_0, hdi) float64 -1.049 1.244 -1.271 ... -0.2923 1.068
    muP         (muP_dim_0, hdi) float64 55.18 57.98 51.3 54.39
    sig_repeat  (hdi) float64 1.805 2.195
    sig_part    (hdi) float64 2.111 3.67
    mu          (mu_dim_0, hdi) float64 53.73 59.94 53.15 ... 55.22 52.65 55.22

xarray.Dataset

<input id='section-a19b60ca-7ce3-4095-8fbe-22a97c38ed64' class='xr-section-summary-in' type='checkbox' disabled >Dimensions:

P_t_dim_0: 31

hdi: 2

muP_dim_0: 2

mu_dim_0: 213

<input id='section-392e928c-89b9-4ccd-b318-4172fe5dbb9e' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (4)

P_t_dim_0

(P_t_dim_0)

int32

0 1 2 3 4 5 6 ... 25 26 27 28 29 30
<input id='attrs-4d3bccd3-49bc-450f-bd59-0b139f4637a0' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-a8f71915-4984-4cca-9047-b79bb40da226' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30])

muP_dim_0

(muP_dim_0)

int32

0 1
<input id='attrs-737cc0ed-0b93-4d83-b501-85cbd8dcb13b' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-6afb6f84-773c-4102-8af6-f212c76b63db' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([0, 1])

mu_dim_0

(mu_dim_0)

int32

0 1 2 3 4 5 ... 208 209 210 211 212
<input id='attrs-7924bcc9-fbae-42a6-910f-261ee90f1fa1' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-ce267cfc-f01f-4ca3-a8f3-ed8be3c42968' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([  0,   1,   2, ..., 210, 211, 212])

hdi

(hdi)

<U6

'lower' 'higher'
<input id='attrs-ab408480-6f12-413f-90c2-39e1c7783850' class='xr-var-attrs-in' type='checkbox' ><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-dc4b3969-7447-4ea3-940d-9d8649da5850' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

hdi_prob :

0.94

array(['lower', 'higher'], dtype='<U6')

<input id='section-c936f4d3-ace5-4333-8445-032da83f8b06' class='xr-section-summary-in' type='checkbox'  checked>Data variables: (5)

P_t

(P_t_dim_0, hdi)

float64

-1.049 1.244 ... -0.2923 1.068
<input id='attrs-8b57299e-0a4f-4fce-8560-e2272c6c1930' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-f054996e-f379-40f7-8170-5768bafb2137' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([[-1.04933488,  1.24447326],
       [-1.27083368,  0.99612808],
       [-1.3313187 ,  0.94554552],
       [-0.66265308,  1.62071047],
       [-1.13094189,  0.24437355],
       [-0.65425075,  1.63503919],
       [-0.6372083 ,  0.72140311],
       [-1.55271277, -0.18872343],
       [-1.90061442, -0.48978517],
       [-1.53722224, -0.18169102],
       [-1.94174435, -0.52084368],
       [-1.54629878, -0.18179545],
       [-1.56249424, -0.20610827],
       [-1.55414352, -0.18295354],
       [-1.31441208,  0.07183448],
       [-0.61534927,  0.73858143],
       [-0.61215057,  0.74435444],
       [-1.31880845,  0.07199762],
       [-0.51014551,  0.77651572],
       [-0.67749052,  0.61429976],
       [-0.85944827,  0.4439236 ],
       [ 2.29951068,  4.30909656],
       [-0.51993004,  0.78336326],
       [ 0.90052238,  2.39343575],
       [ 0.61526244,  2.02895003],
       [-0.16980057,  1.14024051],
       [-0.64271931,  0.73049639],
       [ 0.02468777,  1.42130246],
       [-0.95554599,  0.40410235],
       [-0.90730907,  0.45238628],
       [-0.29234009,  1.06833432]])

muP

(muP_dim_0, hdi)

float64

55.18 57.98 51.3 54.39
<input id='attrs-92eb2020-0b33-4c56-a8f0-1149763c067e' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-f689afbd-11cb-4792-af70-9938b09ddcf1' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([[55.17577891, 57.98406728],
       [51.30485598, 54.39138247]])

sig_repeat

(hdi)

float64

1.805 2.195
<input id='attrs-f003a600-4372-4a0c-9aab-fb9b340e0ee6' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-3b22a93d-dc77-470e-afbe-b1bc585989fc' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([1.80503309, 2.19504696])

sig_part

(hdi)

float64

2.111 3.67
<input id='attrs-7d667680-440c-402d-bf75-5dee8f31acce' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-180fa20c-f549-47f1-a6d0-a00da982f274' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([2.11095315, 3.67000751])

mu

(mu_dim_0, hdi)

float64

53.73 59.94 53.15 ... 52.65 55.22
<input id='attrs-13dcd85f-a296-4c22-b673-efc2fe5e6a85' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-3784f0d4-9423-49f1-81d4-c72095367207' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([[53.72784035, 59.94158032],
       [53.14746865, 59.31066301],
       [49.22500255, 55.36655408],
       [51.14051129, 57.34039171],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [50.27214229, 52.85997724],
       [51.12082434, 57.33892593],
       [51.68013924, 54.27600952],
       [51.68013924, 54.27600952],
       [51.68013924, 54.27600952],
       [51.68013924, 54.27600952],
       [51.68013924, 54.27600952],
       [51.68013924, 54.27600952],
       [51.68013924, 54.27600952],
...
       [50.77705353, 53.35094816],
       [50.77705353, 53.35094816],
       [50.77705353, 53.35094816],
       [50.77705353, 53.35094816],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [50.98396958, 53.58947559],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835],
       [52.64815033, 55.22254835]])

<input id='section-bf63196a-3c4a-40b0-94b2-be08c20ec08d' class='xr-section-summary-in' type='checkbox' disabled >Attributes: (0)


```julia
ENV["GKS_ENCODING"] = "utf-8"
using JLD
using Plots
include("/home/julian/Hamburg/LadderDGA/run.jl");
```

# Improved Sums

## Motivation

Our goal is to find a good surrogate for the sum over infinitely many Matsubara frequencies, i.e. $\lim_{N \rightarrow \infty}\sum_{i\nu_{-N}}^{i\nu_{N}} f(\nu) \longrightarrow \lim_{N \rightarrow \infty} \text{S}_N[f(\nu)]$.
This operator $\text{S}$ can be defNined in different ways.
Here we will test two different approaches.

### Extrapolation approach

In order to quantify the error made by the finite approximation of the infinite sum, we first construct the following function:
$$
    g_\text{N} = \sum_{x = i \nu_{-N}}^{i \nu_{N}} f(x)
$$
We now want to extract the true limit of $N \rightarrow \infty$ from $g_\text{N}$.
For any finite summation we assume, that there error Terms fall of as $\sum_{i=1}^k \frac{c_i}{(i\nu_N)^i}$ with coefficients $c_i \in \mathbb{C}$.
We can therefore assume that the finite approximation of $g_\text{N}$ has a tail of the form
$$
    S(j) = \lim_{N \rightarrow \infty} S_N(j) = \lim_{N \rightarrow \infty} \sum_{i=0}^N \frac{c_i}{(j)^i}
$$
It is clear that for $N \rightarrow \infty$, only $c_0$ survives and all other terms are artifacts of the finite approximation.
By fitting $S_N$ to the numerical data of $\tilde{f}_\text{N}$ we can determine the coefficients and extract the true value of the sum
$$
    \lim_{N \rightarrow \infty} g_\text{N} = c_0
$$

### Analytic tail approach

In cases in which we know the true tail of a function: $\lim_{n \rightarrow \infty} f(i \nu_n) = \lim_{n \rightarrow \infty} \sum_{j = 1}^N \frac{c_j}{i \nu_n}$ we can use this knowledge to improve convergence of a sum.
Note that $c_0 = 0$ in order for the sequence to converge in the first place.
We write
\begin{align}
    \sum_{n} f(i \nu_n) & = \sum_{n} \left(f(i \nu_n) - \sum_{j = 1}^N \frac{c_j}{i \nu_n}\right) + \sum_{n}\sum_{j = 1}^N \frac{c_j}{i \nu_n}
\end{align}
The $\sum_{n}\sum_{j = 1}^N \frac{c_j}{i \nu_n}$ is analytically

#### Fit to kinetic energy

We now use that the leading order of the $\chi$ tail is known to be $\frac{E_\text{kin}}{(i \omega_n)^2}$ (TODO: cite!).
This tail coefficient is subtracted, analytically summend and only the remaining higher order terms have to be fitted to real data.
\begin{align}
    \sum_{n= 1}^\infty \frac{E_\text{kin}}{(i \omega_n)^2}
     & = \sum_{n= 1}^\infty \frac{E_\text{kin}}{\left(i \frac{(2n) \pi}{\beta}\right)^2} \\
     & = \frac{ E_\text{kin} \beta^2}{- 4 \pi^2} \sum_{n= 1}^\infty \frac{1}{\left( n\right)^2} \\
     & = - \frac{E_\text{kin} \beta^2}{4 \pi^2} \left(
       \frac{\pi^2}{6}
     \right) \\
     & = - \frac{E_\text{kin} \beta^2}{24}
\end{align}
This can be used to reduce the number of tail coefficients we need to fit:
\begin{align}
    \frac{1}{\beta} \sum_{n= -\infty}^\infty \chi (i \omega_n)
           & =  \frac{\chi (i \omega_0)}{\beta} + \frac{1}{\beta}  \sum_{n= \mathbb{N} \setminus \{ 0\}} \left( \chi (i \omega_n) - \frac{E_\text{kin}}{(i \omega_n)^2}  \right) +  \frac{1}{\beta} \sum_{n=\mathbb{N} \setminus \{ 0\} } \frac{E_\text{kin}}{(i \omega_n)^2} \\
        & \stackrel{\text{h.f.}}{=} \frac{\chi (i \omega_0)}{\beta} + \frac{2}{\beta}  \sum_{n= 1 }^\infty \left( \chi (i \omega_n) - \frac{E_\text{kin}}{(i \omega_n)^2}  \right) -  \frac{E_\text{kin} \beta}{12}
\end{align}

#### Kinetic energy from ED

We compute $E_\text{kin}$ from ED using
\begin{align}
    E_\text{kin} & = \frac{1}{\beta} \sum_{k, n} \epsilon_k G(i  \nu_n, k)
    = \frac{1}{\beta} \sum_{n} \Delta (i \nu_n) G_\text{IMP}(i \nu_n)\\
    \Delta(i \nu_n) & = \sum_l \frac{V_l^2}{i\nu_n - \epsilon_l}
\end{align}
For the $\nu_n$ sum we use the knowledge of the full tail:
\begin{align}
    \sum_{n=0}^\infty \Delta (i \nu_n) G_\text{IMP}(i \nu_n) & =
    \sum_{n} \left( \Delta (i \nu_n) G_\text{IMP}(i \nu_n) - \sum_k \frac{V_k^2}{(i\nu_n)^2} \right) + \sum_{n,k} \frac{V_k^2}{(i\nu_n)^2}  \\
    & = \sum_{n} \left( \Delta (i \nu_n) G_\text{IMP}(i \nu_n) - \sum_k \frac{V_k^2}{(i\nu_n)^2} \right) - \sum_{k}V_k^2 \frac{\beta}{2}
\end{align}

Bath parameters are read from ``hubb.andpar`` and the impurity Greens function from ``gm_wim``

## Method for the analytical lsq fit of frequency sums

It turns out, that the fitting procedure for the numerical data above is quite unstable. However, it is possible to compute the coefficients for the least squares fit (lsq fit) analytically i.e. with arbitrary precision. This approach will be discussed here.
Our $n$ datapoints for the fit are given by $g_n$.

In order to minimize $\text{LSQ}(g_n) = \sum_{j=\nu\text{max}_\text{min}}^{\nu\text{max}_\text{max}} \left(S_n(j) - \left(g_{n}\right)_j\right)^2$
with the tail function for data points $g_j$
\begin{equation}
    S_n(j) = \sum_{k = 0}^n \frac{c_{k}}{j^k}
\end{equation}
$k, l$ are a running index for all elements of the set of coefficients $C$ with elements denoted by $c$.

We first construct
$M_{kl} = \sum_{j=\nu\text{max}_\text{min}}^{\nu\text{max}_\text{max}} \frac{1}{j^k j^{l}}$
(derivation: minimize least squares sum analytically) and then for the $n$-th set of $j$ datapoints $\left(g_{n}\right)_j$:
\begin{equation}
\left(b_{n}\right)_{l} = \sum_{j=\nu\text{max}_\text{min}}^{\nu\text{max}_\text{max}} \frac{\left(g_{n}\right)_j}{j^d}
\end{equation}

We can then obtain the fit-coefficients $c_n$ of the $n$-th data point by solving $M c_n = b_n$

# Comparison $\chi_\text{Impurity}$ with  $\chi_\text{numerical}$

## general $\chi_{\sigma \sigma'}$ sums
 - Compute the naive sum $\chi^{\omega}_{\sigma \sigma', \text{naive}}$ as $\frac{1}{\beta^2} \sum_{\nu \nu'}\chi^{\omega \nu \nu'}_{\sigma \sigma'}$
 - Compute the fitted sum $\chi^{\omega}_{\sigma \sigma', \text{fit}}$ as $\frac{1}{\beta^2}  \text{FitSum}_{\nu \nu'}(\chi^{\omega \nu \nu'}_{\sigma \sigma'})$

```julia
macro slice_middle(arr, n)
    :($arr[($n:(length($arr))-$n)])
end

macro slice_usable(arr, usable)
    n = :(ceil(Int64,(length($arr)-length($usable)+2)/2))
    :($arr[($n:(length($arr))-$n+1)])
end

mP = LadderDGA.modelParams
# load asymptotic data
asympt_vars = load(LadderDGA.env.asymptVars);
χchAsympt_in = asympt_vars["chi_ch_asympt"]*2;
χspAsympt_in = asympt_vars["chi_sp_asympt"]*2;
χchDMFT = LadderDGA.impQ_ch.χ
χspDMFT = LadderDGA.impQ_sp.χ
χchED_in = copy(χchDMFT)#asympt_vars["chi_ch_ED"];
χspED_in = copy(χspDMFT)#asympt_vars["chi_sp_ED"];

# naive:
#χchED_ω_naive = vcat(reverse(χDMFTch[2:end]), χDMFTch)
#χspED_ω_naive = vcat(reverse(χDMFTsp[2:end]), χDMFTsp)
χchED_ω_naive = sum(χchED_in,dims=[2,3])[:,1,1]/(mP.β^2);
χspED_ω_naive = sum(χspED_in,dims=[2,3])[:,1,1]/(mP.β^2);

# fit preparation:
nc = [0,1,2,3]
νmax_end   = floor(Int64,size(LadderDGA.impQ_ch.χ,2)/2)
start = ceil(Int64, (1/4)*νmax_end)
W = LadderDGA.build_weights(start, νmax_end, nc)
println("Building fit matrix for ν frequencies from ", start, " to ", νmax_end, " with ", length(nc), " coeffcients")

# fit
reduction_prct = -0.05

χchED_ω_fit = [LadderDGA.sum_freq(χchDMFT[i,:,:], [1,2], true, mP.β, weights=W)[1,1] for i in 1:size(χchDMFT,1)];
χspED_ω_fit = [LadderDGA.sum_freq(χspDMFT[i,:,:], [1,2], true, mP.β, weights=W)[1,1] for i in 1:size(χspDMFT,1)];
usable_ch = LadderDGA.find_usable_interval(real(χchED_ω_fit), reduce_range_prct=reduction_prct)
usable_sp = LadderDGA.find_usable_interval(real(χspED_ω_fit), reduce_range_prct=reduction_prct)
usable_fit_DMFT = intersect(usable_ch, usable_sp)

# ed
χchAsympt = χchAsympt_in[1:ceil(Int64,length(χchED_ω_naive)/2)]
χchAsympt = vcat(reverse(χchAsympt[2:end]), χchAsympt)
χspAsympt = χspAsympt_in[1:ceil(Int64,length(χspED_ω_naive)/2)]
χspAsympt = vcat(reverse(χspAsympt[2:end]), χspAsympt);

```

    Building fit matrix for ν frequencies from 11 to 41 with 4 coeffcients

### Comparison plots

Range is >>extended<< instead of reduced in order to show boundary effects.

```julia
include("src/GFFit.jl")
reduction_prct_pl = -0.1
usable_ch_pl = LadderDGA.find_usable_interval(real(χchED_ω_fit), reduce_range_prct=reduction_prct_pl)

fit_y = real(χchED_ω_fit[usable_ch_pl])
lusable_pl = length(usable_ch_pl)
naive_y = real(χchED_ω_naive[ceil(Int64,length(χchED_ω_naive)/2-lusable_pl/2):floor(Int64,length(χchED_ω_naive)/2+lusable_pl/2)-1])
asym_y = real(χchAsympt[ceil(Int64,length(χchAsympt)/2-lusable_pl/2):floor(Int64,length(χchAsympt)/2+lusable_pl/2)-1])
println("Usable range: ", usable_ch, ". Reduction: $(100*reduction_prct)%")
plot(usable_ch_pl .- 1, asym_y, label="Re[χ_ch(ω)] exact", c=:red, size=(1000,600))
plot!(usable_ch_pl .- 1, naive_y, label="Re[χDMFT_ch(ω)] naive", c=:green)
plot!(usable_ch_pl, fit_y, label="Re[χDMFT_ch(ω)] fit", c=:orange)
plot!(nlQ_ch.usable_ω, real.(nlQ_ch.χ_ω[nlQ_ch.usable_ω]), label="Re[χ_ch(ω)] fit", c=:blue)
plot!(nlQ_ch.usable_ω, repeat([0], length(nlQ_ch.usable_ω)), fillrange=real(nlQ_ch.χ_ω[nlQ_ch.usable_ω]), fillalpha=0.3, c=:orange, label="usable")
```

    Usable range: 31:51. Reduction: -5.0%

```julia
reduction_prct = -0.3
usable_sp_pl = LadderDGA.find_usable_interval(real(χspED_ω_fit), reduce_range_prct=reduction_prct)
fit_y = real(χspED_ω_fit[usable_sp_pl])
lusable_pl = length(usable_sp_pl)
naive_y = real(χspED_ω_naive[ceil(Int64,length(χspED_ω_naive)/2-lusable_pl/2)+1:floor(Int64,length(χchED_ω_naive)/2+lusable_pl/2)])
asym_y = real(χspAsympt[ceil(Int64,length(χspAsympt)/2-lusable_pl/2)+1:floor(Int64,length(χspAsympt)/2+lusable_pl/2)])
println("Usable range: ", usable_ch_pl, ". Reduction: $(100*reduction_prct)%")
plot(usable_sp_pl, asym_y, label="Re[χDMFT_sp(ω)] exact", c=:red, size=(1000,600))
plot!(usable_sp_pl, naive_y, label="Re[χDMFT_sp(ω)] naive", c=:green)
plot!(nlQ_sp.usable_ω, real.(nlQ_sp.χ_ω[nlQ_sp.usable_ω]), label="Re[χ_sp(ω)] fit", c=:blue)
#plot!(nlQ_sp_λ.usable_ω, real.(nlQ_sp_λ.χ_ω[nlQ_sp_λ.usable_ω]), label="Re[χ_sp_λ(ω)] fit", c=:purple)
plot!(nlQ_sp.usable_ω, repeat([0], length(nlQ_sp.usable_ω)), fillrange=real(nlQ_sp.χ_ω[nlQ_sp.usable_ω]), fillalpha=0.3, c=:orange, label="usable")
```

    Usable range: 31:51. Reduction: -30.0%

 ## $\chi_{\uparrow \uparrow}$ (? and $\chi_{\uparrow \downarrow}$ ?) sums

We know that $\sum_\omega \chi_{\uparrow \uparrow}^\omega = \frac{n}{2} (1 - \frac{n}{2})$ and $\sum_\omega \chi_{\uparrow \downarrow}^\omega = \langle n_\uparrow n_\downarrow \rangle - \frac{n^2}{4}$

For $\chi_{\uparrow \uparrow}$ we have knowledge of the tail and can therefore use another approximation scheme. We will test four different methods: naiive sum (appr0), fitted sum (appr1), tail corrected sum (appr2) and taill corrected + fitted sum (appr4).

 - Compute all four variants of the $\omega$ sums for the exact ED results.
 - Compute two approximations of the $\omega$ sum over the full $\chi$: $\chi^{\omega}_{\sigma \sigma', \text{ naiive}} = \frac{1}{\beta^2}  \sum_{\nu \nu'}  \chi^{\omega \nu \nu'}_{\sigma \sigma'}$ and $\chi^{\omega}_{\sigma \sigma', \text{ fit}} = \frac{1}{\beta^2}  \text{FitSum}_{\nu \nu'}  \chi^{\omega \nu \nu'}_{\sigma \sigma'}$
-----
We now want to compute all four approximations for both inputs, we expect $\text{res} \approx \frac{n}{2}(1-\frac{n}{2})$:
 - $\text{res}_{\text{ appr0, naiive/fit}} = \frac{1}{\beta} \sum_{\omega} \chi^{\omega}_{\uparrow\uparrow, \text{ naiive/fit}}$
 - $\text{res}_{\text{ appr1, naiive/fit}} = \text{FitSum}_{\omega}\chi^{\omega}_{\uparrow\uparrow, \text{ naiive/fit}}$
 - $\text{res}_{\text{ appr2, naiive/fit}} = \frac{1}{\beta}  \sum_{\omega}  \left( \chi^{\omega}_{\uparrow\uparrow, \text{ naiive/fit}} - \frac{E_\text{kin}}{(i \omega_n)^2}  \right)  -  \frac{E_\text{kin} \beta}{12}  $
 - $\text{res}_{\text{ appr3, naiive/fit}} =  \text{FitSum}_{\omega}\left( \chi^{\omega}_{\uparrow\uparrow, \text{ naiive/fit}} - \frac{E_\text{kin}}{(i \omega_n)^2}  \right)  -  \frac{E_\text{kin} \beta}{12}  $

## Plot $\chi^\omega_{\uparrow \uparrow} \cdot (i \omega)^2 \rightarrow E_\text{kin}$

```julia
function multiply_tail2(inp, mP)
    setprecision(100) do
        res = zeros(BigFloat, length(inp))
        for n in 1:length(inp)
            res[n] = BigFloat(-inp[n]) * (BigFloat(n - floor(Int64, length(inp)/2) - 1)*BigFloat(2*π/mP.β))^BigFloat(2)
        end
        Float64.(res, RoundDown)
    end
end

function tail2(inp, mP)
    setprecision(1000) do
        res = zeros(BigFloat, length(inp))
        for n in 1:length(inp)
            res[n] = BigFloat(-inp[n]) * (BigFloat(n - floor(Int64, length(inp)/2) - 1)*BigFloat(2*π/mP.β))^BigFloat(2)
        end
        res#Float64.(res, RoundDown)
    end
end

χupupAsympt_tail2 = multiply_tail2(real.(χupupAsympt), mP)
χupup_DGA_tail2 = multiply_tail2(real.(χDGAupup), mP)
#plot(real(χupupAsympt .- χupupAsympt_tail2)[end-200:end], label="v1")
plot(usable_DGA .- length(χDGAupup)/2 .- 0.5, real(χDGAupup[usable_DGA] .- χupup_DGA_tail2[usable_DGA]), label="χ_asympt, ω² sub")
plot!(-20:20,real(χupupAsympt .- χupupAsympt_tail2)[end-10020:end-9980], label="χDGA, ω² sub")
#plot(abs.(tail4[end-200:end]))
#plot(abs.(χupupAsympt[end-200:end]))
```

```julia
# helper functions
function multiply_tail(inp, mP)
    res = copy(inp)
    setprecision(1000) do
        for n in 1:length(inp)
            res[n] *= (LadderDGA.iω(n - floor(Int64, length(inp)/2) - 1))^2
        end
    end
    return res
end

# load and prepare data
iνₙ, GImp_pos    = LadderDGA.readGImp(LadderDGA.env.inputDir * "/gm_wim", only_positive=true)
ϵₖ, Vₖ, μ    = LadderDGA.read_anderson_parameters(LadderDGA.env.inputDir * "/hubb.andpar");
Ekin = real(LadderDGA.E_Kin_ED(iνₙ, ϵₖ, Vₖ, GImp_pos, mP.β))
println("Ekin = $(Ekin)")
χupupAsympt_in = (χchAsympt_in+χspAsympt_in)/2
χupdoAsympt_in = (χchAsympt_in-χspAsympt_in)/2
χupupAsympt = vcat(reverse(χupupAsympt_in[2:end]), χupupAsympt_in)
χupdoAsympt = vcat(reverse(χupdoAsympt_in[2:end]), χupdoAsympt_in)
χDMFTupup_naive = (χchED_ω_naive + χspED_ω_naive)/2
χDMFTupup_fit = (χchED_ω_fit + χspED_ω_fit)/2
χDGAupup = (nlQ_sp.χ_ω + nlQ_ch.χ_ω)/2
println("manual cutoff for lowest and highest 9 frequencies!!")
usable_DGA = intersect(nlQ_sp.usable_ω,nlQ_ch.usable_ω)

# plot
χupupAsympt_tail = multiply_tail(χupupAsympt, mP)
χDMFTupup_naive_tail = multiply_tail(χDMFTupup_naive, mP)
χDMFTupup_fit_tail = multiply_tail(χDMFTupup_fit, mP)
χDGAupup_tail = multiply_tail(χDGAupup, mP)

xasympt = @slice_middle (1:length(χupupAsympt)) 9980
χupupAsympt_slice = @slice_middle real(χupupAsympt) 9980
χupup_Asympt_tail_slice = @slice_middle real(χupupAsympt_tail) 9980
xfit = usable_fit_DMFT .- ceil(Int64,length(χDMFTupup_fit)/2)#[usable_fit_DMFT] .-
plot(xasympt .- 9920, repeat([Ekin], length(χupupAsympt_slice)), label="Kinetic Energy")#, ylims = (-0.1692,-0.168))
#plot!(xasympt, real(χupupAsympt_slice), label="χ_asympt")
plot!(xasympt .- 9920, χupup_Asympt_tail_slice, label="χ_asympt ⋅ ω²")
plot!(usable_DGA, real.(χDGAupup_tail[usable_DGA]), label="χ_DGA ⋅ ω²")
```

    Ekin = -0.1691548157566971
    manual cutoff for lowest and highest 9 frequencies!!

```julia
plot(repeat([Ekin], length(real(χupupAsympt_tail)[end-9000:end])), label="Kinetic Energy")#, ylims = (-0.1692,-0.168))
#plot!(xasympt, real(χupupAsympt_slice), label="χ_asympt")
plot!(real(χupupAsympt_tail)[end-9000:end], label="χ_asympt ⋅ ω²")
```

```julia
plot(usable_fit_DMFT, repeat([Ekin], length(usable_fit_DMFT)), label="Kinetic Energy")
#plot!(xfit, real(χDMFTupup_naive), label="χ_fit")
plot!(usable_fit_DMFT, real(χDMFTupup_naive_tail)[usable_fit_DMFT], label="χ_fit ⋅ ω²")
```

## Compute and compare all corrections

```julia
# helper functions
function subtract_tail!(inp, Ekin, mP)
    iωₙ2_full  = zeros(length(inp))
    for i in 1:length(inp)
        iωₙ2_full[i] = (LadderDGA.iω(i - ceil(Int64, length(inp)/2) - 0))^2
    end
    ind_0 = argmax(real(inp))
    println("zero index: ", ind_0, " - ", argmax(real(iωₙ2_full)))
    for n in 1:length(inp)
        if iωₙ2_full[n] != 0
            inp[n] -= (Ekin/iωₙ2_full[n])
        end
    end
end

# fit preparation:
χupupAsympt_appr0 = sum(copy(χupupAsympt))/mP.β
χupupAsympt_appr1 = copy(χupupAsympt)
χupupAsympt_appr2 = copy(χupupAsympt)
χupupAsympt_appr3 = copy(χupupAsympt)

χupupAsympt_cut = @slice_usable χupupAsympt usable_fit_DMFT
χupupAsympt_cut_appr0 = sum(copy(real.(χupupAsympt_cut)))/mP.β
χupupAsympt_cut_appr1 = copy(real.(χupupAsympt_cut))
χupupAsympt_cut_appr2 = copy(real.(χupupAsympt_cut))
χupupAsympt_cut_appr3 = copy(real.(χupupAsympt_cut))

χDMFTupup_naive_appr0 = sum(copy(real.(χDMFTupup_naive)[usable_fit_DMFT]))/(mP.β)
χDMFTupup_naive_appr1 = copy(real.(χDMFTupup_naive[usable_fit_DMFT]))
χDMFTupup_naive_appr2 = copy(real.(χDMFTupup_naive[usable_fit_DMFT]))
χDMFTupup_naive_appr3 = copy(real.(χDMFTupup_naive[usable_fit_DMFT]))

χDMFTupup_fit_appr0 = sum(copy(real.(χDMFTupup_fit[usable_fit_DMFT])))/(mP.β)
χDMFTupup_fit_appr1 = copy(real.(χDMFTupup_fit[usable_fit_DMFT]))
χDMFTupup_fit_appr2 = copy(real.(χDMFTupup_fit[usable_fit_DMFT]))
χDMFTupup_fit_appr3 = copy(real.(χDMFTupup_fit[usable_fit_DMFT]))

χDGAupup_appr0 = sum(copy(real.(χDGAupup[usable_DGA])))/(mP.β)
χDGAupup_appr1 = copy(real.(χDGAupup[usable_DGA]))
χDGAupup_appr2 = copy(real.(χDGAupup[usable_DGA]))
χDGAupup_appr3 = copy(real.(χDGAupup[usable_DGA]))

nc = [0,1,2,3,4]
νmax_end   = floor(Int64,size(χupupAsympt,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W1 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr1,asympt from ", start, " to ", νmax_end, " with ", length(nc), " coeffcients")
χupupAsympt_appr1 = LadderDGA.sum_freq(χupupAsympt_appr1, [1], true, mP.β, weights=W1)[1]

νmax_end   = floor(Int64,size(χDMFTupup_naive_appr1,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W2 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr1,naive from ", start, " to ", νmax_end, " with ", length(nc), " coeffcients")
χDMFTupup_naive_appr1 = LadderDGA.sum_freq(χDMFTupup_naive_appr1, [1], true, mP.β, weights=W2)[1]

νmax_end   = floor(Int64,size(χDMFTupup_fit_appr1,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W3 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr1,fit from ", start, " to ", νmax_end, " with ", length(nc), " coeffcients")
χDMFTupup_fit_appr1 = LadderDGA.sum_freq(χDMFTupup_fit_appr1, [1], true, mP.β, weights=W3)[1]

νmax_end   = floor(Int64,size(χupupAsympt_cut,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W4 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr1,asympt_cut from ", start, " to ", νmax_end, " with ", length(nc), " coeffcients")
χupupAsympt_cut_appr1 = LadderDGA.sum_freq(χupupAsympt_cut_appr1, [1], true, mP.β, weights=W4)[1]

νmax_end   = floor(Int64,size(χDGAupup_appr1,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W5 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr1,DGA from ", start, " to ", νmax_end, " with ", length(nc), " coeffcients")
χDGAupup_appr1 = LadderDGA.sum_freq(χDGAupup_appr1, [1], true, mP.β, weights=W5)[1]

# tail preparation
iνₙ, GImp    = LadderDGA.readGImp(LadderDGA.env.inputDir * "/gm_wim", only_positive=true)
ϵₖ, Vₖ, μ    = LadderDGA.read_anderson_parameters(LadderDGA.env.inputDir * "/hubb.andpar");
Ekin = real(LadderDGA.E_Kin_ED(iνₙ, ϵₖ, Vₖ, GImp, mP.β))
rem_full = Ekin*(mP.β*mP.β)/12
println("Ekin = $(Ekin), Subterm=$(rem_full)")
subtract_tail!(χupupAsympt_appr2, Ekin, mP)
subtract_tail!(χDMFTupup_naive_appr2, Ekin, mP)
subtract_tail!(χDMFTupup_fit_appr2, Ekin, mP)
subtract_tail!(χupupAsympt_cut_appr2, Ekin, mP)
subtract_tail!(χDGAupup_appr2, Ekin, mP)
χupupAsympt_appr2_res = (sum(χupupAsympt_appr2)-rem_full)/mP.β
χDMFTupup_naive_appr2_res = (sum(χDMFTupup_naive_appr2)-rem_full)/mP.β
χDMFTupup_fit_appr2_res = (sum(χDMFTupup_fit_appr2)-rem_full)/mP.β
χupupAsympt_cut_appr2_res = (sum(χupupAsympt_cut_appr2)-rem_full)/mP.β
χDGAupup_appr2_res = (sum(χDGAupup_appr2)-rem_full)/mP.β

# fit preparation 2:
nc = [0,1,2,3,4]
νmax_end   = floor(Int64,size(χupupAsympt_appr2,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W1 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr3,asympt from ", start, " to ", νmax_end, " with ", length(nc), " coefficients")
χupupAsympt_appr3 = LadderDGA.sum_freq(χupupAsympt_appr2, [1], true, mP.β, correction=-rem_full, weights=W1)[1]

νmax_end   = floor(Int64,size(χDMFTupup_naive_appr2,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W2 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr3,naive from ", start, " to ", νmax_end, " with ", length(nc), " coefficients")
χDMFTupup_naive_appr3 = LadderDGA.sum_freq(χDMFTupup_naive_appr2, [1], true, mP.β, correction=-rem_full, weights=W2)[1]

νmax_end   = floor(Int64,size(χDMFTupup_fit_appr2,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W3 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr3,fit from ", start, " to ", νmax_end, " with ", length(nc), " coefficients")
χDMFTupup_fit_appr3 = LadderDGA.sum_freq(χDMFTupup_fit_appr2, [1], true, mP.β, correction=-rem_full, weights=W3)[1]

νmax_end   = floor(Int64,size(χupupAsympt_cut_appr2,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W4 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr3,asympt_cut from ", start, " to ", νmax_end, " with ", length(nc), " coefficients")
χupupAsympt_cut_appr3 = LadderDGA.sum_freq(χupupAsympt_cut_appr2, [1], true, mP.β, correction=-rem_full/mP.β, weights=W4)[1]

νmax_end   = floor(Int64,size(χDGAupup_appr2,1)/2)
start = ceil(Int64, (1/4)*νmax_end)
W5 = build_weights(start, νmax_end, nc)
println("Building fit matrix for appr3,DGA from ", start, " to ", νmax_end, " with ", length(nc), " coefficients")
χDGAupup_appr3 = LadderDGA.sum_freq(χDGAupup_appr2, [1], true, mP.β, correction=-rem_full, weights=W5)[1]

# naive sums

println("\n----------- Naive Sums -----------\n")
println("Naive sum for χ_asympt_upup_naive: ", χupupAsympt_appr0)
println("Naive sum for χ_upup_DMFT_naive: ", χDMFTupup_naive_appr0)
println("Naive sum for χ_upup_DMFT_fit: ", χDMFTupup_fit_appr0)
println("Naive sum for χ_asympt_upup_cut_naive: ", χupupAsympt_cut_appr0)
println("Naive sum for χ_upup_DGA: ", χDGAupup_appr0)

# Fitted sums
println("\n----------- Fitted Sums -----------\n")
println("Fitted sum for χ_asympt_upup_naive: ", χupupAsympt_appr1)
println("Fitted sum for χ_upup_DMFT_naive: ", χDMFTupup_naive_appr1)
println("Fitted sum for χ_upup_DMFT_fit: ", χDMFTupup_fit_appr1)
println("Fitted sum for χ_asympt_upup_cut_naive: ", χupupAsympt_cut_appr1)
println("Fitted sum for χ_upup_DGA: ", χDGAupup_appr1)

# Tail subtracted sums
println("\n------ Tail Subtracted Sums ------\n")
println("Tail subtracted sum for χ_asympt upup: ", χupupAsympt_appr2_res)
println("Tail subtracted sum for χ_upup_DMFT_naive: ", χDMFTupup_naive_appr2_res )
println("Tail subtracted sum for χ_upup_DMFT_fit: ", χDMFTupup_fit_appr2_res )
println("Tail subtracted sum for χ_asympt_cut upup: ", χupupAsympt_cut_appr2_res)
println("Tail subtracted sum for χ_upup_DGA: ", χDGAupup_appr2_res)

# Tail subtracted sums
println("\n---------- Tail+Fit Sums ----------\n")
println("Tail+Fit sum for χ_asympt upup: ", χupupAsympt_appr3)
println("Tail+Fit sum for χ_upup_DMFT_naive: ", χDMFTupup_naive_appr3)
println("Tail+Fit sum for χ_upup_DMFT_fit: ", χDMFTupup_fit_appr3)
println("Tail+Fit sum for χ_asympt_cut upup: ", χupupAsympt_cut_appr3)
println("Tail+Fit sum for χ_upup_DGA: ", χDGAupup_appr3)

```

    Building fit matrix for appr1,asympt from 2500 to 10000 with 5 coeffcients
    Building fit matrix for appr1,naive from 3 to 10 with 5 coeffcients
    Building fit matrix for appr1,fit from 3 to 10 with 5 coeffcients
    Building fit matrix for appr1,asympt_cut from 3 to 10 with 5 coeffcients
    Building fit matrix for appr1,DGA from 3 to 11 with 5 coeffcients
    Ekin = -0.1691548157566971, Subterm=-3.171652795438071
    zero index: 10001 - 10001
    zero index: 11 - 11
    zero index: 11 - 11
    zero index: 11 - 11
    zero index: 12 - 12
    Building fit matrix for appr3,asympt from 2500 to 10000 with 5 coefficients
    Building fit matrix for appr3,naive from 3 to 10 with 5 coefficients
    Building fit matrix for appr3,fit from 3 to 10 with 5 coefficients
    Building fit matrix for appr3,asympt_cut from 3 to 10 with 5 coefficients
    Building fit matrix for appr3,DGA from 3 to 11 with 5 coefficients

    ----------- Naive Sums -----------

    Naive sum for χ_asympt_upup_naive: 0.24998714641543707
    Naive sum for χ_upup_DMFT_naive: 0.19796034302820809
    Naive sum for χ_upup_DMFT_fit: 0.237838546778797
    Naive sum for χ_asympt_upup_cut_naive: 0.23833485985709635
    Naive sum for χ_upup_DGA: 0.41197378591893485

    ----------- Fitted Sums -----------

    Fitted sum for χ_asympt_upup_naive: 0.2500000000103438
    Fitted sum for χ_upup_DMFT_naive: 0.16018022374958035
    Fitted sum for χ_upup_DMFT_fit: 0.24441564312961173
    Fitted sum for χ_asympt_upup_cut_naive: 0.25040061773489036
    Fitted sum for χ_upup_DGA: 0.4239288421482532

    ------ Tail Subtracted Sums ------

    Tail subtracted sum for χ_asympt upup: 0.2499999999973607
    Tail subtracted sum for χ_upup_DMFT_naive: 0.21019323757312422
    Tail subtracted sum for χ_upup_DMFT_fit: 0.2500714413237132
    Tail subtracted sum for χ_asympt_cut upup: 0.25056775440201245
    Tail subtracted sum for χ_upup_DGA: 0.42314434785116894

    ---------- Tail+Fit Sums ----------

    Tail+Fit sum for χ_asympt upup: 0.2500000000103529
    Tail+Fit sum for χ_upup_DMFT_naive: 0.16017962039660935
    Tail+Fit sum for χ_upup_DMFT_fit: 0.2444150397766407
    Tail+Fit sum for χ_asympt_cut upup: 0.05305272933796227
    Tail+Fit sum for χ_upup_DGA: 0.4239283208618455

```julia
function multiply_tail2(inp, mP)
    setprecision(100) do
        res = zeros(BigFloat, length(inp))
        for n in 1:length(inp)
            res[n] = BigFloat(-inp[n]) * (BigFloat(n - floor(Int64, length(inp)/2) - 1)*BigFloat(2*π/mP.β))^BigFloat(2)
        end
        Float64.(res, RoundDown)
    end
end

function tail2(inp, mP)
    setprecision(1000) do
        res = zeros(BigFloat, length(inp))
        for n in 1:length(inp)
            res[n] = BigFloat(-inp[n]) * (BigFloat(n - floor(Int64, length(inp)/2) - 1)*BigFloat(2*π/mP.β))^BigFloat(2)
        end
        res#Float64.(res, RoundDown)
    end
end

χupupAsympt_tail2 = multiply_tail2(real.(χupupAsympt), mP)
χupup_DGA_tail2 = multiply_tail2(real.(χDGAupup), mP)
χupup_DMFT_tail2 = multiply_tail2(real.(χDMFTupup_fit), mP)
#plot(real(χupupAsympt .- χupupAsympt_tail2)[end-200:end], label="v1")
plot(usable_DGA .- length(χDGAupup)/2 .- 0.5, real(χDGAupup[usable_DGA] .+ χupup_DGA_tail2[usable_DGA]), ylabel="χ - χ ⋅ ω²", label="χDGA, ω² sub",ylim=(Ekin-0.02,Ekin+0.02))
plot!(usable_fit_DMFT .- length(χDMFTupup_fit)/2 .- 0.5, real(χDMFTupup_fit[usable_fit_DMFT] .+ χupup_DMFT_tail2[usable_fit_DMFT]), label="χDMFT")
plot!(-100:100,real(χupupAsympt .+ χupupAsympt_tail2)[end-10100:end-9900], label="χ_asympt, ω² sub")
plot!(-100:100, repeat([Ekin], length(-100:100)))
#plot(abs.(tail4[end-200:end]))
#plot(abs.(χupupAsympt[end-200:end]))
```

```julia
χupupAsympt_middle = @slice_usable χupupAsympt usable_DGA
plot(usable_fit_DMFT,multiply_tail2(real.(χDMFTupup_naive[usable_fit_DMFT]),mP),markershape = :hexagon,markersize = 3,ylabel="χ ⋅ ω²", xlabel="ωₙ", label="naive")
plot!(usable_fit_DMFT,multiply_tail2(real.(χDMFTupup_fit[usable_fit_DMFT]),mP),markershape = :hexagon,markersize = 3, label="DMFT fit")
plot!(usable_DGA,multiply_tail2(real.(χupupAsympt_middle),mP), label="asympt",markershape = :hexagon,markersize = 3)
#plot!(usable_DGA,multiply_tail2(real.(χDGAupup[usable_DGA]),mP), label="DGA")
plot!(usable_DGA,repeat([Ekin], length(usable_DGA)), label="EKin",markershape = :hexagon,markersize = 3)
savefig("plot.png")
```

# Number of coefficients and tail subtraction

In this section, we will ivestigate the impact of different numbers of exponents in die Richardson fit and subtraction of known tails on the overall sum

```julia
usable_ω = intersect(impQ_sp.usable_ω, impQ_ch.usable_ω)
χupup_DMFT_ω = real(impQ_sp.χ_ω .+ impQ_ch.χ_ω)[usable_ω] ./ 2

usable_ω2 = intersect(impQ_sp2.usable_ω, impQ_ch2.usable_ω)
χupup_DMFT_ω2 = real(impQ_sp2.χ_ω .+ impQ_ch2.χ_ω)[usable_ω2] ./ 2

χupup_ω_csum = LadderDGA.build_fνmax_fast(χupup_DMFT_ω, 50);
χupup_ω_csum2 = LadderDGA.build_fνmax_fast(χupup_DMFT_ω2, 50);

iωn = 1im .* 2 .* (-sP.n_iω:sP.n_iω)[usable_ω] .* π ./ mP.β
iωn2 = 1im .* 2 .* (-sP.n_iω:sP.n_iω)[usable_ω2] .* π ./ mP.β
EKin = impQ_sp.tailCoeffs[3]
χupup_DMFT_ω_sub = LadderDGA.subtract_tail(χupup_DMFT_ω,EKin, iωn)
χupup_DMFT_ω_sub2 = LadderDGA.subtract_tail(χupup_DMFT_ω2,EKin, iωn2)

χupup_ω_sub_csum = LadderDGA.build_fνmax_fast(χupup_DMFT_ω_sub, 50);
χupup_ω_sub_csum2 = LadderDGA.build_fνmax_fast(χupup_DMFT_ω_sub2, 50);
```

```julia
exps_l = [0:i for i in 2:6]

res = zeros(length(exps_l),  50 - 1,4)  # no nu fit, nu fit, no nu fit + sub, nu fit + sub
for (j,exps) in enumerate(exps_l)
for i in 1:50 - maximum(exps)
    ind = exps .+ i
    sh = LadderDGA.Richardson(ind,exps,method=:bender)
    sh2 = LadderDGA.Richardson(exps .+ 1,exps,method=:bender)
    res[j,i,1] = LadderDGA.esum_c(real.(χupup_ω_csum)[ind], sh) ./ mP2.β
    res[j,i,2] = LadderDGA.esum_c(real.(χupup_ω_csum2)[ind], sh2) ./ mP2.β
    res[j,i,3] = LadderDGA.esum_c(real.(χupup_ω_sub_csum)[ind], sh) ./ mP2.β .- EKin*mP.β/12
    res[j,i,4] = LadderDGA.esum_c(real.(χupup_ω_sub_csum2)[ind], sh2) ./ mP2.β .- EKin*mP.β/12
end
end
```

```julia
using Plots
using LaTeXStrings
l = @layout([[A B; C D] E{.15w}])
# method $method
colors = palette(:tab10)
p = Array{Plots.Plot,1}(undef, 5)
v = χupup_ω_sub_csum2[end]./ mP2.β .- EKin*mP.β/12
for method in 1:4
    ind = (method == 1 || method == 3) ? (1:15) : (1:size(res,2))
    if method == 4
        p[method] = plot(res[1,ind,method], c=colors[1], xlabel="# csum terms", legend=false, ylims=(0.249,0.2505))
    elseif method == 1
        p[method] = plot(res[1,ind,method], c=colors[1], ylabel=L"$\sum_{q,\omega}  \chi_{\uparrow\uparrow}(\omega, q)$", legend=false)
    elseif method == 2
        p[method] = plot(res[1,ind,method], c=colors[1], legend=false, ylims=(0.245,0.2505))
    elseif method == 3
        p[method] = plot(res[1,ind,method], c=colors[1], xlabel="# csum terms", ylabel=L"$\sum_{q,\omega}  \chi_{\uparrow\uparrow}(\omega, q)$", legend=false)
    end
    for i in 2:size(res,1)
        plot!(p[method], res[i,ind,method], c=colors[i], label="$(i+1) exps", legend=false)
    end
    hline!(p[method], [0.25 0.25 0.25 0.25], c=colors[size(res,1)+1], label="true value")

    hline!(p[method], [v v v v], c=colors[(size(res,1)+2)], label="pure sub")
end

p[5] = plot([0 for y in 1:1, i in 1:(size(res,1)+2)], title=["method 1" "mthod 2" "method 3" "method 4"], palette = :tab10, legendfontsize=12 , foreground_color_legend = nothing, showaxis = false, ticks=false, grid = false, label = hcat(["exps = $(exps_l[i])" for y in 1:1, i in 1:size(res,1)],["true value"  "no bosonic fit"])) #
mval, mind = findmin(abs.(res[3,:,4] .- 0.25))
println("best candidate: tail subtraction + 0:4 exp fit at index $mind range. result = $(round(res[3,mind,4],digits=7)).\nPure tail subtraction for bosonic frequencies gives $(round(v,digits=7))")
plot(p..., layout = l, size = (900, 600))
```

# Test simple $\lambda$ correction

```julia
include("src/lambdaCorrection.jl");
λ, _ = calc_λ_correction(χsp, χLocsp, qMultiplicity, modelParams);
println(λ)
```

# Test $\chi_\text{lDGA}$

## Physical properties
We expect $\frac{1}{2 \beta} \sum_{q,\omega} \left( \chi^\text{ch}(q, \omega)_\text{lDGA} + \chi^{\text{sp}, \lambda}_\text{lDGA}(q, \omega) \right) = \frac{n}{2} (1-\frac{n}{2}) = \frac{1}{4}$
From the first section we assume, that the number of coefficients should be set to $4$ and $\nu_\text{maxstart}$ should be in the last $50 \%$ of the tail.

We start by computing the $\nu, \nu'$ sum over $\chi^\text{sp}$ and $\chi^\text{ch}$.
The correct tail behaviour is checked by plotting both functions as a function of $\omega$.

  - Test $\sum \chi_q^\omega$
  - Test $\sum \chi_{q,\lambda}^\omega$

```julia
maximum(real(χsp[41,:]))
size(Σ_ladder)
```

# Tests for $χ_{sp}^\lambda$
- $\frac{1}{2 \beta} \sum_{\omega, q} (χ^{ch}_\omega (q) + χ^{sp,\lambda}_\omega (q)) \stackrel{!}{=} \chi_\text{Loc} \rightarrow \frac{n}{2}(1-\frac{n}{2})$
- wenige q punkte, $\omega = 0$: $χ_\text{sp}^{-1}(\omega=0,q) \approx \sum_i \frac{1}{\lambda + a_i}$

```julia
plot(real([sum(χsp[i,:] .* qMultiplicity for i = 1:size(χsp,1))/qNorm]))
```

```julia
qNorm      = sum(qMultiplicity)*modelParams.β
xrange = -1:0.1:3
t = [real(sum([sum(((1 ./ χsp[i,:] .+ λ)).^(-1) .* qMultiplicity)  for i in 1:size(χsp,1)])/qNorm - χLocsp) for λ in xrange];
dt = [-real(sum([sum(((1 ./ χsp[i,:] .+ λ)).^(-2)) for i in 1:size(χsp,1)])/qNorm) for λ in xrange];
plot(xrange,t, label="f(λ) = Σ_{q,ω} χ^λ - χLocsp", size=(1200,800))
#plot!(twinx(), xrange,dt, label="f'(λ)", color="green",legend=:bottomright)
vline!([maximum(-real(χsp[41,:]) .^ (-1))], label="λ_max-pole")
plot!(xrange,repeat([0],size(xrange,1)))
```

# Vergleich $\chi^\lambda$ gegen $\chi$

```julia
size(sum(χsp,dims=[2])[:,1])
```

```julia
size(real(sum(χsp,dims=[2])[:,1]))
```

```julia
plot(real(sum(χsp,dims=[2])[1]))
plot!(real(sum(χsp_impr,dims=[2])[1]))
#plot!(repeat([real(χLocsp)], size(χsp_impr,1)))
```

```julia

```

```julia

```

# Test $\Sigma(k,\nu)$

### Investigation of noisy data for $\Sigma$
We inject Gaussian noise into each of the inputs of $\Sigma ( \gamma_{ch}, \gamma_{sp},\chi_{ch}, \chi_{sp}, \chi_0, F_{\uparrow \downarrow}, G(\nu_n + \omega_n))$ in order to investigate the sensitivity of $\Sigma$ to faulty input data.

### Investigation of sum fits for $\Sigma$

The internal sums to compute $\Sigma$ are formally defined over infinitely many Matsubara frequencies.
We want to investigate the usefulness of the previously discussed methods in this context.

```julia
include("src/GFFit.jl")
include("src/ladderDGATools.jl")
@time Σ_ladder = calc_DΓA_Σ(χch, χsp, trilexch, trilexsp, bubble, Σ_loc, FUpDo,
                    qMultiplicity, qGrid, kGrid, modelParams, simParams);
@time Σ_ladder_impr1,Σ_ladder_impr2,tmp = calc_DΓA_Σ_impr(χch, χsp, trilexch, trilexsp, bubble, Σ_loc, FUpDo,
                    qMultiplicity, qGrid, kGrid, modelParams, simParams);
```

```julia
plot(real(tmp[:]))
```

```julia
cut = 0
numin = Int(floor((length(tmp)-cut)*3/4))
numax = Int(floor(length(tmp))-cut)
W2 = build_weights(numin, numax, [0,1,2,3])
plot(real(build_fνmax(tmp, W2, [1])))
```

```julia

```

### Compare naive vs. fit

```julia

```

### Compare to $\Sigma_\text{loc}$

```julia

```

```julia

```


# Introduction to Deep Learning

### Introduction
IEEE (Institute of Electrical and Electronics Engineering) is the world's largest organisation which is dedicated for the benifit of technological development. IEEE has highly cited publications, conferences, technology standards, and professional and educational activities.
Technolites is a club of CVRCoE which is devoted for various technical events. Here students get highly inspired and share their ideas to derive some exceptionally innovative projects and technical event.
For the first time in collaboration with IEEE CVECoE and Technolites CVRCoE we are presenting to you a 6-days workshop on Deep Learning.

### About the Notebook

* Author : Srimanth Tenneti
* Date   : 22/09/2021
* Version : 1.0
* Description : Day-2 Hands on Lab Notebook
* Topic : Perceptron Introduction
* Prerequisites
                  1. Python
                  2. Basic Calculus - Differentiation, Chain Rule, ...

### Instructions

* The notebook contains some Fixme segments. The code will only work if they are filled with the appropriate code.

* Follow the instructions during the Live Session to complete the code.

* In case you face any issues kindly contact the Training Assistants in the live session or contact the IEEE CVRCE members via the Discord Community.

### Excerise 1

```python
def perceptron(i1 , i2 , w1 , w2 , b):
  result = i1 * w1 + i2 * w2 + b
  if result > 0 :
    return True
  else :
    return False

## AND Truth Table ##
#####################
## I1  #  I2  #  O/P#
#####################
## 0   #  0   #  0  #
## 0   #  1   #  0  #
## 1   #  0   #  0  #
## 1   #  1   #  1  #
#####################

### Todo : Estimate the parameters for a AND perceptron
w1 = Fixme
w2 = Fixme
b =  Fixme

#output = perceptron(i1 , i2 , w1 , w2 , b)
# ReLu f(x) = x for all x > 0 and f(x) = 0 for all x < 0

def solution_check(i1 , i2 , output):
  if (i1 == 0 and i2 == 0):
    assert(output == False)
  elif (i1 == 0 and i2 == 1):
    assert(output == False)
  elif (i1 == 1 and i2 == 0):
    assert(output == False)
  elif (i1 == 1 and i2 == 1):
    assert(output == True)
```

```python
output = perceptron(1 , 1 , w1 , w2 , b)
print(output)
solution_check(1 , 1 , output)
```

# Perceptron Implementation

## Forward Pass

\begin{equation}
    O = W1 * X1 + W2 * X2 + B
\end{equation}

#### Sigmoid Activation

\begin{equation}
    \sigma(x) = \frac{1}{1+e^{-x}}
\end{equation}

#### Perceptron Output After Activation

\begin{equation}
    Y' = \sigma(O)
\end{equation}

### Gradient Descent

#### Cost Function - {Mean Square Error}

\begin{equation}
    E(Y',Y) = (Y' - Y)^{2}
\end{equation}

#### Gradient Calculation

\begin{equation}
    E = (\sigma(O) - Y)^{2}
\end{equation}

\begin{equation}
    \frac{\partial E}{\partial W} = \frac{\partial E}{\partial Y'} * \frac{\partial Y'}{\partial O} * \frac{\partial O}{\partial W}
\end{equation}

\begin{equation}
    \frac{\partial E}{\partial W} = 2 * (Y' - Y) * \sigma'(O) * X
\end{equation}

Note
----

\begin{equation}
    \sigma'(x) = \frac{1}{1+e^{-x}} - \frac{1}{(1+e^{-x})^{2}}
\end{equation}

This can be simplified as ...

\begin{equation}
    \sigma'(x) = \sigma(x) * (1 - \sigma(x))
\end{equation}

 #### Update Step for W1
 \begin{equation}
    W1' --> W1 - \alpha  *  \frac{\partial E}{\partial W1}
\end{equation}

 #### Update Step for W2
 \begin{equation}
    W2' --> W2 - \alpha  *  \frac{\partial E}{\partial W2}
\end{equation}

 #### Update Step for B
 \begin{equation}
    B' --> B - \alpha  *  \frac{\partial E}{\partial B}
\end{equation}

### Sigmoid and Sigmoid' implementation

```python
### Todo : Implement sigmoid and sigmoid'
def sigmoid(x):
  return Fixme
def sigmoid_diff(x):
  return Fixme
```

```python
### Analysing the sigmoid behaviour
t = np.linspace(-5 , 5 , 100) #Random data points
y = sigmoid_diff(t)
plt.plot(t,y,c='r')
plt.plot(t,sigmoid(t),c='b')
```

### Dataset

```python
### Dataset -> [Petal Width ,Petal Length , Flower Type]

data = [[3 , 1.5 , 1],
        [2 , 1 , 0],
        [4 , 1.5 , 1],
        [3 , 1  , 0],
        [3.5 , 0.5 , 1],
        [2 , 0.5 , 0], #1 -> Red , #0 -> Blue
        [5.5 , 1 , 1],
        [1 , 1 , 0]]

unknown_flower = [4.5 , 1]
```

### Importing required libraries

```python
import matplotlib.pyplot as plt
import numpy as np
```

```python
# Network

#      o --> Flower Type
# w1 /   \w2
#   o    o  --> lenght & width

# Todo : Random weight initialization

w1 = Fixme
w2 = Fixme
b  = Fixme
```

### Visualizing the data

```python
for i in range(len(data)):
    point = data[i]
    color = 'r'
    if (point[2] == 0):
        color = "b"
    plt.scatter(point[0] , point[1] , c= color)
    plt.scatter(unknown_flower[0] , unknown_flower[1] , c = 'g')
    plt.legend(['Red Flower' , 'Unknown Flower' ,'Blue Flower'])
```

### Training the model

```python
### Training Loop

### Todo : Define the hyperparameters

n_epochs = Fixme # Number of epochs
lr = Fixme # Learning Rate

### Todo : Train the model

```

### Visualizing the costs

```python
def plot(costs):
  plt.plot(costs)
  plt.xlabel("Epochs...")
  plt.ylabel("Costs ...")
  plt.title("Costs Plot")
```

### Inference

```python
idx = np.random.randint(0 , 7)

print(idx)

point[0] = data[idx][0]
point[1] = data[idx][1]

y = data[idx][2]

pred = sigmoid(point[0] * w1 + point[1] * w2 + b) # Forward pass
error = (pred - y) ** 2

print(y)
print(pred)
print(error)
```

```python
pred = perceptron(5.5,1,w1,w2,b)

if (pred > 0.5): # Todo : Define Threshold
    print("Its a red flower")
else:
    print("Blue flower")
```

### Trained Weights

```python
### Trained Weights -> (Only for reference)

w1 = 0.0769270733846528

w2 = 0.060348710894484126

b = 0.060348710894484126
```


# Recursive Least Squares Estimation

## Reference
[1] L. Ljung, System Identification Theory for the User

## 1 Linear Regressions and Least Squares
### 1.1 Autoregressive-moving-average Model
考虑自回归滑动平均模型（ARMA），

### 1.2 Least-squares Criterion

为了得到$\theta$，我们首先需要一个判断准则去评定$\theta$的好坏。自然的，我们会关心模型输出与实际输出的差。所以定义误差为：

\begin{equation}
\epsilon(t,\theta)=y(t)-\phi^T(t)\theta
\end{equation}

为后续计算的方便，取

$$V_N(\theta,Z^N)=\frac{1}{2N}\sum_{t=1}^N[\epsilon]^2$$

为指标函数。上式$V_N$对$\theta$求偏导数，并使用$\phi\phi^T\theta=\phi y$，可以得出对$\theta$的最小二乘估计$\hat{\theta}$。

$$\hat{\theta}=\arg_\theta\min V_N(\theta,Z^N)=[\frac{1}{N}\sum_{t=1}^N\phi(t)\phi^T(t)]^{-1}\frac{1}{N}\sum_{t=1}^N\phi(t)y(t)$$

*注意：这里处理的是SISO情形，即$y(t),u(t)$均是标量。*

定义：

$$R(N)=\frac{1}{N}\sum_{t=1}^N\phi(t)\phi^T(t)\in R^{d\times d}$$

$$f(N)=\frac{1}{N}\sum_{t=1}^N\phi(t)y(t)\in R^d$$

### 1.3 Weighted Least-Squares Estimation(WLS)

如果系统参数时变较快，对过去的输入-输出的拟合意义不大。所以在指标函数中引入权重，衰减过去数据的影响。

修改指标函数为：

$$V_N(\theta,Z^N)=\frac{1}{2N}\sum_{t=1}^N[\alpha_i\epsilon]^2$$(权重固定)

或

$$V_N(\theta,Z^N)=\frac{1}{2N}\sum_{t=1}^N[\beta(N,t)\epsilon]^2$$（权重时变）

## 2 Recursive Least-Squares Algorithm

### 2.1 Prototype of RLS

采用带有权重的最小二乘准则：

$$\hat{\theta_t}=\arg_\theta\min\sum_{k=1}^{t}\beta(t,k)[y(k)-\phi^T(k)\theta]^2$$

可以得到最优估计为：

$$\hat{\theta_t}=\bar{R}^{-1}(t)f(t)$$

其中：

$$\bar{R}(t)=\sum_{k=1}^t\beta(t,k)\phi(k)\phi^T(k)$$

$$f(t)=\sum_{k=1}^t\beta(t,k)\phi(k)y(k)$$

设权重序列满足：

$$\beta(t,k)=\lambda(t)\beta(t-1,k)$$

$$\beta(t,t)=1$$

即：

$$\beta(t,k)=\prod_{j=k+1}^t\lambda(t)$$

*注意：这里处理一种特殊的权重序列。*

在上述权重序列的影响下，回顾$\bar{R}(t),f(t)$的定义式，$\bar{R}(t),f(t)$满足关系：

$$\bar{R}(t)=\lambda(t)\bar{R}(t-1)+\phi(t)\phi^T(t)$$

$$f(t)=\lambda(t)f(t-1)+\phi(t)y(t)$$

将上述关系带入$\hat{\theta}_t$的估计式中：

\begin{align}
\hat{\theta}_t & = \bar{R}(t)^{-1}f(t) \\
& = \bar{R}(t)^{-1}[\lambda(t)f(t-1)+\phi(t)y(t)] \\
& = \bar{R}(t)^{-1}[\lambda(t)\bar{R}(t-1)\hat{\theta}_{t-1}+\phi(t)y(t)] \\
& = \bar{R}(t)^{-1}[\lambda(t)(\bar{R}(t)-\phi(t)\phi^T(t))\hat{\theta}_{t-1}+\phi(t)y(t)] \\
& = \hat{\theta}_{t-1} + \bar{R}^{-1}(t)\phi(t)[y(t)-\phi^T(t)\hat{\theta}_{t-1}]
\end{align}

这样我们就得到了**RLS的两个递推式**：

\begin{eqnarray}
& \hat{\theta}_t & = \hat{\theta}_{t-1} + \bar{R}^{-1}(t)\phi(t)[y(t)-\phi^T(t)\hat{\theta}_{t-1}] \\
& \bar{R}(t) & = \lambda(t)\bar{R}(t-1)+\phi(t)\phi^T(t)
\end{eqnarray}

上式要求计算矩阵的逆，在计算机中计算复杂度较高。所以需要尝试将求逆的操作转化为一个计算较为简便的操作。

引入定理：

$$\exists A,B,C,D:P=A+BCD \rightarrow P^{-1}=[A+BCD]^{-1}=A^{-1}-A^{-1}B[DA^{-1}B+C^{-1}]^{-1}DA^{-1}$$

其中$A,B,C,D,P$均为适当维矩阵。

令$P(t)=\bar{R}^{-1}(t)$，使用关系式凑上述定理的形式，则有$A=\lambda(t)\bar{R}(t-1), B=\phi(t), C=I, D=\phi^T(t)$，
则

\begin{align}
P(t) &= A^{-1}-A^{-1}B[DA^{-1}B+C^{-1}]^{-1}DA^{-1} \\
    &= \frac{1}{\lambda(t)}[P(t-1)-\frac{P(t-1)\phi(t)\phi^T(t)P(t-1)}{\lambda(t)+\phi^T(t)P(t-1)\phi(t)}]
\end{align}

上面的迭代式还可以再进行简化，考虑$P(t)\phi(t)$：

\begin{align}
P(t)\phi(t) &= \frac{1}{\lambda(t)}P(t-1)\phi(t)-\frac{1}{\lambda(t)}\frac{P(t-1)\phi(t)\phi^T(t)P(t-1)\phi(t)}{\lambda(t)+\phi^T(t)P(t-1)\phi(t)} \\
    &=\frac{P(t-1)\phi(t)}{\lambda(t)+\phi^T(t)P(t-1)\phi(t)}
\end{align}

利用上述结果，令$L(t)=P(t)\phi(t)$，我们就获得了带权重的RLS估计算法（也可以称带遗忘因子的RLS估计算法）：

>带权重的RLS估计算法：
\begin{eqnarray}
& L(t) &= \frac{P(t-1)\phi(t)}{\lambda(t)+\phi^T(t)P(t-1)\phi(t)} \\
& \hat{\theta}_t & = \hat{\theta}_{t-1} + L(t)[y(t)-\phi^T(t)\hat{\theta}_{t-1}] \\
& P(t) &= \frac{1}{\lambda(t)}[P(t-1)-L(t)\phi^T(t)P(t-1)] \\
\end{eqnarray}

```python

```

machine-learning/Recursive Least Squares Estimation.ipynb


# Lecture 7: Bias-Variance Tradeoff, Regularization
## 04/10/2021

### Table Of Contents
* [Recap](#recap)
* [Bias-Variance Tradeoff](#bv-tradeoff)
    * [Bias](#bias)
    * [Variance](#variance)
    * [The Tradeoff](#the-tradeoff)
    * [Polynomial Regression](#polynomial-regression)
* [Regularization](#regularization)
    * [Ridge](#ridge)
    * [LASSO](#lasso)
    * [Visualizing Ridge and Lasso](#visualizing-ridge-and-lasso)
    * [Regularization and Bias Variance](#regularization-and-bias-variance)
    * [Lambda](#lambda)
    * [Validation on Lambda](#validation-on-lambda)
* [Exercises](#exercises)

> Bias/Variance is one way to understand Overfitting and Underfitting

If we look at the first model, the points seem to follow some sort of curve, but our predictor is linear (not very complicated) and *unable to fit all the points* no matter how we wiggle it. We have chosen a model which is not complex enough to accurately capture all the information from our data set.

In the last model, the predictor is super complex and adjusts based on every point. In this case, the model changes too much based on the data and isn't considering the 'big picture'. This model is fitting too much to the minor jitters.

**High bias/Low Variance ~ underfitting**

**Low bias/High Variance ~ overfitting**

**Goldilocks - porridge**

Summary: We try to fiddle with bias/variance to get better models. Variance is important! because otherwise we only are building a model for our train not our normal dataset.

 Bias-Variance error decomposition only works for convex loss functions (especially L2): https://www.textbook.ds100.org/ch/11/gradient_convexity.html

# Thinking Experiment

Lets say we were building a model of a dog. On the left we would have a model that would classify everything as a dog (including a hippo) versus on the far right

    Underfitting
    Just Right
    Overfitting

Related kaggle project on dog breed classification.

# Why do we care?

Bias-Variance is *directly related* to what a model predicts and algorithmic bias. (Articles [here](https://www.zdnet.com/article/what-is-bias-in-ai-really-and-why-cant-ai-neutralize-it/), [here](https://academicworks.cuny.edu/cgi/viewcontent.cgi?article=1042&context=jj_etds) and [here](https://www.cio.com/article/3385123/bias-in-ai-creating-artificial-intelligence-that-is-less-like-humans.html))

Bias-Variance is the main idea behind the other methods we will cover here. i.e. **"What are some ways to try and make appropriate bias/variance tradeoffs"**

--

Frequently people feel they should minimize bias even at the expense of variance. They think that presence of bias indicates fundamental issues with their model and algorithm. Yes, variance is also bad but a model with high variance could at least predict well on average, at least it is not fundamentally wrong.

This is mistaken. A high variance and low bias model can preform well in some sort of long-run average sense. However, in practice modelers are always dealing with a concrete actual data set in the context of some business problem. In these cases, long run averages are irrelevant, what is important is the performance of the model on the data you actually have and in this case bias and variance are equally important and one should not be improved at an excessive expense to the other. [Source](http://scott.fortmann-roe.com/docs/BiasVariance.html)

# Bias-Variance Tradeoff

When making this tradeoff we do it by proxy -- using various metrics:

Today we'll perform **model evaluation**, where we'll judge how our linear regression models actually perform. Last week, we talked about **loss functions**, a function that describes a numerical value for how far your model is from the true values.

Today we will use:
$$\text{Mean Squared Error: } \frac{1}{n}\sum_{i=1}^n \left(y_i - \hat{f}(x_i)\right)^2$$

In this loss function, $y_i$ is a scalar, and $x_i$ is a $p$ dimensional vector, because there are $p$ features. This loss is called **mean squared error**, or **MSE**.

(i.e. if we have x=1, y=2, but our model says $\hat{f}(x_{1})=3$. Then the MSE is $(2-3)^2$.

## Model Evaluation

## Bias

When evaluating a model, the most intuitive first step is to look at how well the model performs. For classification, this may be the percentage of data points correctly classified, or for regression it may be how close the predicted values are to actual. The **bias** of a model is *a measure of how close our prediction is to the actual value on average from an average model*.

Note that bias is **not** a measure of a single model, it encapsulates the scenario in which we collect many datasets, create models for each dataset, and average the error over all of models. Bias is not a measure of error for a single model, but a more abstract concept describing the average error over all errors. A low value for the bias of a model describes that on average, our predictions are similar to the actual values.

 Super Mathy: Squared difference between best possible prediction for x, f(x), and our “long-term” expectation for what the learner will do if we averaged over many datasets D: $E_D[h_D(x)]$

## Variance
The **variance** of  a model relates to the variance of the distribution of all models. In the previous section about bias, we envisoned the scenario of collecting many datasets, creating models for each dataset, and averaging the error overall the datasets. Instead, the variance of a model describes the variance in prediction. While we might be able to predict a value very well on average, if the variance of predictions is very high this may not be very helpful, as when we train a model we only have one such instance, and a high model variance tells us little about the true nature of the predictions. A low variance describes that our model will not predict very different values for different datasets.

 Super Mathy: Squared difference btwn our longterm expectation for the learners performance, $E_D[h_D(x)]$, and what we expect in a representative run on a dataset D ($\hat{y}$)

**We can take a look at how bias and variance differ and can be explained in a dataset with the following diagram:**

source:http://scott.fortmann-roe.com/docs/BiasVariance.html

The image describes what bias and variance are in a more simplified example. Consider that we would like to create a model that selects a point close to the center. The models on the top row have low bias, meaning the center of the cluster is close to the red dot on the target. The models on the left column have low variance, the clusters are quite tight, meaning our predictions are close together.

**Question: What do the blue dots represent? What about the bullseye?**

**Question: Which is the of best scenario? Which is the worst?**

## The Tradeoff

We are trying to minimize **expected error**, or the average **MSE** over all datasets. It turns out (with some advanced probability gymnastics), that:

$$\text{Mean Squared Error} = \text{Noise Variance} + \text{Bias}^2 + \text{Variance}$$

Note that $\text{Noise Variance}$ is constant: we assume there is some noise, and $\text{moise variance}$ is simply a value that describes how noisy your dataset will be on average. This is often also called "irreducible noise", as it is literally irreducible, we cannot avoid this.

Furthermore, notice that the equation above is the sum of (squared) bias and variance. Thus there is a literal tradeoff between these two, since decreasing one increases the other. This defines what is known as the **bias variance tradeoff**.

source:http://scott.fortmann-roe.com/docs/BiasVariance.html

**Why does this happen?**

At some point, as we decrease **bias**, instead of getting closer to the **true model** $h$, we go past and try to fit to the $\epsilon$ (noise) that is part of our current dataset. This is equivalent to making our model more noisy, or **overfit** on our dataset, which means that over all datasets, it has more **variance**.

**Questions for understanding**:
> 1. Where does underfitting and overfitting lie in the graph above? How do they relate to bias and variance?
> 2. Why can't we usually just make a bunch of models with low bias and high variance and average them?
> 3. Why is low variance important in models?

## Polynomial Regression

Let's revisit the polynomial problem that we have discussed.

In this case, if our model has degree $d$, we have $d + 1$ features: $x = [x^0, x^1, ..., x^d]$. Now, we have a linear model with $d + 1$ features:

$$\hat{f}(x) = \sum_{i=0}^{d} a_i x^i$$

Model complexity in this case is the degree of the polynomial. As we saw last week, as $d$ increases, model complexity increases. The model gets better, but then gets erratic. This directly corresponds to the bias-variance graph above.

```python
import matplotlib.pyplot as plt
import random
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from plotting import overfittingDemo, plot_multiple_linear_regression, ridgeRegularizationDemo, lassoRegularizationDemo
from scipy.optimize import curve_fit
from sklearn.metrics import mean_squared_error
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline
```

```python
overfittingDemo()
```

As we saw from last time, the best model was a degree 3 model.

# Regularization

We talked about **validation** as a means of combating overfitting. However, this is not the only method to combat overfitting. Another method to do so is to add *regularization* terms to our loss function. **Regularization** basically penalizes complexity in our models. This allows us to add explanatory variables to our model without worrying as much about overfitting. Here's what our ordinary least squares model looks like with a regularization term:

$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda R(\boldsymbol{\theta})$$

We've written the model a little differently here, but the first term is the same as the ordinary least squares regression model you learned last week. This time it's just generalized to any function of $x$ where $\theta$ is a list of parameters, or weights on our explanatory variables, such as coefficients to a polynomial. We're minimizing a loss function to find the best coefficients for our model.

The second term is the **regularization** term. The $\lambda$ parameter in front of it dictates how much we care about our regularization term – the higher $\lambda$ is, the more we penalize large weights, and the more the regularization makes our weights deviate from OLS.

**Question**: What happens when $\lambda = 0$?

So, what is $R(\theta)$ in the equation? There are a variety of different regularization functions that could take its place, but today, we'll just talk about the two most common types of functions: **ridge regression** and **LASSO regression**.

$$\begin{align}\text{ Ridge (L2 Norm)}: &\  R(\boldsymbol{\theta}) = \|\theta\|_2^2 = \sum_{i=1}^p \theta_i^2\\
\text{ LASSO (L1 Norm)}: &\ R(\boldsymbol{\theta}) = \|\theta\|_1=\sum_{i=1}^p \lvert \theta_i\rvert\end{align}$$

## Ridge
$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - \hat{f_\theta}(x_i))^2 + \lambda \|\theta\|_2^2$$
In **ridge** regression, the regularization function is the sum of squared weights. One nice thing about ridge regression is that there is always a unique, mathematical solution to minimizing the loss of that term. The solution involves some linear algebra, which we won't get into in this notebook, but the existence of this formula makes this minimization computationally easy to solve!

$$\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T \boldsymbol{X} + \lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{Y}$$

If you recall, the solution to linear regression was of the form:

$$\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T \boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{Y}$$

And we said that the $\boldsymbol{X}^T\boldsymbol{X}$ isn't always invertible. **What about $\boldsymbol{X}^T \boldsymbol{X} + \lambda\boldsymbol{I}$?**

Turns out, this is always invertible! If you are familiar with linear algebra, this is equivalent to adding $\lambda$ to all the eigenvalues of $X^TX$.

---

L2 regularization. Regularization trades a bit of model accuracy for improved generalization and works by constraining the size of model parameters to "reasonable" values. L2 regularization constrains coefficients to a spherical hyper volume by adding an L2 norm penalty term to the linear model loss function. The term Ridge from the original Hoerl and Kennard Ridge paper was taken from the "ridge traces" on their coefficient plots. Unfortunately, the ridge constraint region looks like a lasso and the lasso constraint region looks like a ridge. [Source](https://explained.ai/statspeak/index.html)

**To see this in practice**, we'll first create a regular linear regression model, and compare how it does against models using regularization on the `mpg` dataset we used from last week! We'll be constructing models of `displacement` vs. `mpg`, and seeing the difference from there!

First, let's construct the `mpg_train` dataset!

```python
mpg = pd.read_csv("mpg.csv", index_col='name')# load mpg dataset
mpg = mpg.loc[mpg["horsepower"] != '?'].astype(float) # remove columns with missing horsepower values
mpg_train, mpg_test = train_test_split(mpg, test_size = .2, random_state = 0) # split into training set and test set
mpg_train, mpg_validation = train_test_split(mpg_train, test_size = .5, random_state = 0)
mpg_train.head()
```

      mpg
      cylinders
      displacement
      horsepower
      weight
      acceleration
      model_year
      origin

      name

      toyota corolla
      34.0
      4.0
      108.0
      70.0
      2245.0
      16.9
      82.0
      3.0

      buick century
      17.0
      6.0
      231.0
      110.0
      3907.0
      21.0
      75.0
      1.0

      cadillac eldorado
      23.0
      8.0
      350.0
      125.0
      3900.0
      17.4
      79.0
      1.0

      bmw 320i
      21.5
      4.0
      121.0
      110.0
      2600.0
      12.8
      77.0
      2.0

      ford fairmont futura
      24.0
      4.0
      140.0
      92.0
      2865.0
      16.4
      82.0
      1.0

**Exercise:** Now, let's create a regular linear regression model using the same process we've learned before (fitting, predicting, finding the loss)!

```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

x_train = np.vander(mpg_train["displacement"], 13)
y_train = mpg_train[["mpg"]]

x_validation = np.vander(mpg_validation["displacement"], 13)
y_validation = mpg_validation[["mpg"]]

# instantiate your model
linear_model = LinearRegression()

# fit the model
linear_model.fit(x_train, y_train)
# make predictions on validation set
linear_prediction = linear_model.predict(x_validation)
# find mean squared error
linear_loss = mean_squared_error(y_validation, linear_prediction)
lin_r2 = r2_score(y_validation, linear_prediction)

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("R2 of Linear Model: {:.2f}".format(lin_r2))
```

    Root Mean Squared Error of Linear Model: 25.09
    R2 of Linear Model: 0.60

**Exercise:** Using what you did above as reference, do the same using a Ridge regression model!

```python
from sklearn.linear_model import Ridge

linear_model = Ridge()
linear_model.fit(x_train, y_train)
linear_prediction = linear_model.predict(x_validation)
ridge_loss = mean_squared_error(y_validation, linear_prediction)
ridge_r2 = r2_score(y_validation, linear_prediction)

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("Root Mean Squared Error of Ridge Model: {:.2f}".format(ridge_loss))
print("R2 of Linear Model: {:.2f}".format(lin_r2))
print("R2 of Ridge Model: {:.2f}".format(ridge_r2))
```

    Root Mean Squared Error of Linear Model: 25.09
    Root Mean Squared Error of Ridge Model: 20.11
    R2 of Linear Model: 0.60
    R2 of Ridge Model: 0.68

A final note on ridge is that ridge regression helps with *robustness*. (Don't worry if you don't understand this part, it's not necessary to understand the rest of the lecture.

Consider the above diagram.

It turns out we can write the weights of least squares regression as a linear combination of the inverses of the singular values. When a matrix is non-invertible, these singular values are zero. However, when a matrix is almost non-invertible, these singular values are very small, leading their inverses to be very large. This leas them to dominate the least squares regression.
Ridge regression fixes this problem by downweighting very small singular values into something more reasonable. In the least squares case, tiny changes to the input data could have large effects on the model if it's along small singular values. Ridge regression is more *robust* to these changes.

## LASSO
$$\hat{\boldsymbol{\theta}} = \arg\!\min_\theta \sum_{i=1}^n (y_i - f_\boldsymbol{\theta}(x_i))^2 + \lambda \|\theta\|_1$$
In **LASSO** regression, the regularization function is **the sum of absolute values of the weights**. One key thing to note about **LASSO** is that it is **sparsity inducing**, meaning it forces weights to be zero values rather than really small values (which can happen in **Ridge Regression**), leaving you with fewer explanatory variables in the resulting model! Unlike Ridge Regression, LASSO doesn't necessarily have a unique solution that can be solved for with linear algebra, so there's no formula that determines what the optimal weights should be.

---

L1 regularization. Regularization trades a bit of model accuracy for improved generalization and works by constraining the size of model parameters to "reasonable" values. L1 regularization constrains coefficients to a diamond shaped hyper volume by adding an L1 norm penalty term to the linear model loss function. The term LASSO means "Least absolute Shrinkage and Selection Operator" from the original Tibshirani paper. The LASSO name is perfectly fine except for the fact that the constraint region has lots of pointy discontinuities and looks like it should be called "ridge regularization" from the shape.
[Source](https://explained.ai/statspeak/index.html)

```python
from sklearn.linear_model import Lasso

linear_model = Lasso()
linear_model.fit(x_train, y_train)
y_val_pred = linear_model.predict(x_validation)

lasso_loss = mean_squared_error(y_validation, linear_prediction)
lasso_r2 = r2_score(y_validation, y_val_pred)

print("Root Mean Squared Error of Linear Model: {:.2f}".format(linear_loss))
print("Root Mean Squared Error of LASSO Model: {:.2f}".format(lasso_loss))
print("Root Mean Squared Error of Ridge Model: {:.2f}".format(ridge_loss))
print("R2 of Linear Model: {:.2f}".format(lin_r2))
print("R2 of Ridge Model: {:.2f}".format(ridge_r2))
print("R2 of Lasso Model: {:.2f}".format(lasso_r2))
```

    Root Mean Squared Error of Linear Model: 25.09
    Root Mean Squared Error of LASSO Model: 20.11
    Root Mean Squared Error of Ridge Model: 20.11
    R2 of Linear Model: 0.60
    R2 of Ridge Model: 0.68
    R2 of Lasso Model: 0.66

As we can see, both **Ridge Regression and LASSO Regression** minimized the loss of our linear regression models, so maybe penalizing some features allowed us to prevent overfitting on our dataset!

#####
### Derivatives and Sparsity

Let's take at another visualization that may help build some intuition behind how both of these regularization methods work!

The above plot is of the ridge regression penalty in blue and the LASSO in red. Note that when the values are small, the LASSO penalty is actually higher than the ridge penalty.

**Question**: Based on these visualizations, could you explain why LASSO is sparsity-inducing?

For a more mathematical motivation, consider the derivative of the LASSO and ridge penalties.

$$\frac{\partial}{\partial w} w^2 = 2w$$
$$\frac{\partial}{\partial w} |w| = sign(w)$$

Recall that gradient descent algorithms go along the derivative of the loss. When we train a neural network to minimize these penalties, a network trained on LASSO will reduce the penalty to 0 (when possible) as the derivative is always 1. However, with the ridge penalty, the loss will essentailly "flatten out" when w approaches zero, and the network will pick a value for w that is close to zero, but not quite zero.

Results from an emperical statement.

## Regularization and Bias Variance
As we mentioned earlier, **bias** is the average linear squares loss term across multiple models of the same family (e.g. same degree polynomial) trained on separate datasets. **Variance** is the average variance of the weight vectors (coefficients) on your features.

Without the regularization term, we’re just minimizing bias; the regularization term means we won’t get the lowest possible bias, but we’re exchanging that for some lower variance so that our model does better at generalizing to data points outside of our training data.

## Lambda

We said that $\lambda$ is how much we care about the regularization term, but what does that look like? Let's return to the polynomial example from last week, and see what the resulting models look like with different values of $\lambda$ given a degree 8 polynomial.

```python
ridgeRegularizationDemo([0, 0.5, 1.0, 5.0, 10.0], 8)
```

From the diagram above, it's difficult to determine which lambda value help fit our model the closest to the true data distribution. So, **how do we know what to use for $\lambda$ (or `alpha` in the `sklearn.linear_model` constructors)?**

That's right, let's use the process of **validation** here! In this case, we'd be finding the value for lambda that **minimizes the loss for ridge regression, and then the one that minimizes the loss for LASSO regression**!

## Validation on Lambda
Let's try to find the best $\lambda$ for the degree 20 polynomial on `displacement` from above.

```python
lambdas = np.arange(0, 200) # create a list of potential lambda values

# create a list containing the corresponding mean_squared_error for each lambda usinb both ridge and lasso regression
ridge_errors = []
lasso_errors = []

for l in lambdas:

    ridge_errors.append(...)

    lasso_errors.append(...)

# finds the index of the minimum value in each list
answer = ridge_errors.index(min(ridge_errors)), lasso_errors.index(min(lasso_errors))
answer
```

As we can see from above, we've been able to determine which lambdas minimizes our ridge regression model and our LASSO regression model through validation by iterating through potential lambda values and finding the ones that minimize our loss for each model!

# Validation

When we train machine learning models, we use three types of data, used for different purposes.

### Training data
The data we use to train the model (typically around 80% of the data)

### Validation data
The data we use to fine tune the model and adjust hyperparameters (typically around 15% of the data)

### Test data
The data we use to evaluate the model's performance (only used once, typically around 5% of the data)

*tl;dr you train your dog classifier on 3 of your dogs and then test it out on your friend's 4 dogs. Don't ask why we have so many dogs*

Cross-validation is a technique for evaluating ML models by training ML models on different subsets of the available input data and evaluating them on the complementary subset of the data. Use cross-validation to detect overfitting, ie, failing to generalize a pattern.  [Source](https://docs.aws.amazon.com/machine-learning/latest/dg/cross-validation.html)

---

Cross-validation is primarily a way of measuring the predictive performance of a statistical model. Every statistician knows that the model fit statistics are not a good guide to how well a model will predict: high
$R^2$ does not necessarily mean a good model. It is easy to over-fit the data by including too many degrees of freedom and so inflate $R^2$ and other fit statistics. For example, in a simple polynomial regression I can just keep adding higher order terms and so get better and better fits to the data. But the predictions from the model on new data will usually get worse as higher order terms are added. [Source](https://robjhyndman.com/hyndsight/crossvalidation/)

When we run cross validation we tend to see stagnant test error but consistently better train error, telling us that our model is really good at fitting our training dataset but not good at modeling the overall dataset.

**Question**
What could you say is happening in each of these scenarios? {overfitting, underfitting, good model}
#### high training error, high validation error
#### low training error, high validation error
#### low training error, low validation error

# Conclusion

Through the course of the notebook, we introduced two main concepts, **bias** and **variance**, and how the two relate with one another when it comes to finding the best model for our dataset! We also went into different methods we can use to minimize overfitting our model, and in turn lower variance, by taking look at a process called **regularization**. We saw that the two main regression models, **ridge regression** and **LASSO regression**, and saw the difference between the two (ridge -> penalize large weights, LASSO -> make weights sparse). We also took a look at different visualizations between the two to build up some more intuition behind how they work, through **graphs** and **derivatives**. Finally, we went through a familiar process (**validation**) to determine what the best values of lambda were for our models, officially ending our journey of **bias** and **variance**, and how we can minimze both in our models!

# Congratulations! You are now a Bias + Variance Ninja!

## Exercises

1. What happens as $\lambda$ increases?
    1. bias increases, variance increases
    2. bias increases, variance decreases
    3. bias decreases, variance increases
    4. bias decreases, variance decreases

**Insert answer here**:

2. **True** or **False**? Bias is how much error your model makes.

**Insert answer here:**

3. What is **sparsity**?

**Insert answer here:**

4. For each of the following, choose **ridge**, **lasso**, **both**, or **neither**:
    1. L1-norm
    2. L2-norm
    3. Induces sparsity
    4. Has analytic (mathematical) solution
    5. Increases bias
    6. Increases variance

**Insert answer here:**

5. Which one is better to use: Ridge Regression or LASSO Regression?

**Insert answer here:**

### Congrats! You've finished our few conceptual questions and learned about bias, variance, and regularization!