# Ikeda $B_e$ assumtion.
Using analytical linear decay solution to calculate the $B_e$

```python
from rolldecayestimators import equations
```

    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 461 ('figure.figsize   : 5, 3   ## figure size in inches')
    Duplicate key in file WindowsPath('C:/Users/maa/.matplotlib/stylelib/paper.mplstyle'), line 462 ('figure.dpi       : 100        ## figure dots per inch')

# Purpose
The quadratic or cubic model can be expressed using the linearized equivalent damping ($B_e$) according to . But I have some doubt about the validity of this, which will be investigated in this notebook.

```python
equations.B_e_equation
```

$\displaystyle B_{e} = B_{1} + \frac{8 B_{2} \omega_{0} \phi_{a}}{3 \pi}$

# Methodology
The linear equivalent damping is basically about replacing a higher order model with a linear model. In this notebook a quadratic model is representing the higher order model.
The quadratic model has two damping coefficients $B_1$ and $B_2$ these to coefficients should be replaced by only one linear damping term $B_e$.  suggests that the equation above can be used to combine $B_1$ and $B_2$ into one coefficient $B_e$.

I will use the least square fitting to find the linear equivalent damping that minimized the error when representing a quadratic model as a linear model. The analytical solution to the linear roll decay model will be used to get a fast and accurate prediction of the least square fit.

# Setup

```python
# %load imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

from jupyterthemes import jtplot
jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
from collections import OrderedDict
#plt.style.use('paper')

#import data
import copy
from mdldb.run import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import rolldecayestimators.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex

from sklearn.metrics import r2_score
from src.data import database
from mdldb import tables

```

```python
from rolldecayestimators.simplified_ikeda_class import SimplifiedIkeda
import rolldecayestimators
from scipy.integrate import solve_ivp
from scipy.optimize import least_squares
from rolldecayestimators import fit_on_amplitudes
```

## Quadratic

```python
Math(vlatex(equations.roll_decay_equation_himeno_quadratic_b))
```

$\displaystyle A_{44} \ddot{\phi} + C_{1} \phi + \left(B_{1} + B_{2} \left|{\dot{\phi}}\right|\right) \dot{\phi} = 0$

```python
eq_acceleration_quadratic = sp.Eq(symbols.phi_dot_dot,
                        sp.solve(equations.roll_decay_equation_himeno_quadratic_b,symbols.phi_dot_dot)[0])

accelaration_quadratic_lambda = lambdify(sp.solve(equations.roll_decay_equation_himeno_quadratic_b,symbols.phi_dot_dot)[0])

Math(vlatex(eq_acceleration_quadratic))
```

$\displaystyle \ddot{\phi} = - \frac{B_{1} \dot{\phi} + B_{2} \left|{\dot{\phi}}\right| \dot{\phi} + C_{1} \phi}{A_{44}}$

```python
class RollDecayQuadratic():

    def __init__(self,A_44, B_1, B_2, C_1):
        self.parameters = {
            'A_44':A_44,
            'B_1':B_1,
            'B_2':B_2,
            'C_1':C_1,
        }

    def time_step(self,t,states):

        phi = states[0]
        phi1d = states[1]
        phi2d = accelaration_quadratic_lambda(**self.parameters, phi=phi, phi1d=phi1d)

        d_states_dt = np.array([phi1d, phi2d])
        return d_states_dt

    def simulate(self,t,phi0=np.deg2rad(10),phi1d0=0):

        initial_state = [phi0,phi1d0]

        t_span = [t[0], t[-1]]

        result = solve_ivp(fun=simulation.time_step, t_span=t_span,  y0=initial_state, t_eval=t)
        assert result.success
        df_result = pd.DataFrame(index=result.t, data=result.y.T, columns = ['phi','phi1d'])
        return df_result

```

## Select suitable parameter range

```python
df_rolldecay = database.load(rolldecay_table_name='rolldecay_quadratic_b', limit_score=0.95,
                             exclude_table_name='rolldecay_exclude')
```

```python
interesting = ['B_1A','B_2A','C_1A']
df_rolldecay[interesting].describe()
```

      B_1A
      B_2A
      C_1A

      count
      2.220000e+02
      222.000000
      222.000000

      mean
      1.425475e-01
      0.281687
      9.042803

      std
      1.071491e-01
      0.258985
      4.378310

      min
      6.451552e-18
      -0.662927
      1.539731

      25%
      5.792787e-02
      0.144445
      5.915274

      50%
      1.167264e-01
      0.241128
      8.312634

      75%
      2.092812e-01
      0.371055
      11.348254

      max
      5.570938e-01
      1.956472
      25.363015

```python
B_1A = df_rolldecay['B_1A'].median()
B_1A
```

    0.11672637102335451

```python
index = (df_rolldecay['B_1A']-B_1A).abs().argmin()
row = df_rolldecay.iloc[index]
```

```python
row['B_1A']
```

    0.118067357553941

```python
B_max_ratio = (df_rolldecay['B_2A']/df_rolldecay['B_1A']).abs().quantile(0.95)
```

## Create simulation objects with varying $B_2$

```python
N=100000
A_44 = 2.2
B_1 = row['B_1A']*A_44
C_1 = row['C_1A']*A_44

simulations = {
}

N= 15
B_2_max = B_max_ratio*B_1
B_2s = np.linspace(0,B_2_max,N)
for B_2 in B_2s:
    simulations[B_2]=RollDecayQuadratic(A_44=A_44, B_1=B_1, B_2=B_2, C_1=C_1)
```

```python
B_1
```

    0.2597481866186702

```python
C_1
```

    8.317077640826263

```python
equations.C_equation_linear
```

$\displaystyle C = GM g m$

```python
A_44_eq = sp.Eq(symbols.A_44, equations.A44)
A_44_eq
```

$\displaystyle A_{44} = \frac{GM g m}{\omega_{0}^{2}}$

```python
eqs = [
    A_44_eq,
    equations.C_equation_linear,

]
omega0_eq = sp.Eq(symbols.omega0,sp.solve(eqs, symbols.omega0, symbols.GM)[1][0])
omega0_eq
```

$\displaystyle \omega_{0} = \sqrt{\frac{C}{A_{44}}}$

## Simulate with all simulation objects

```python
omega0 = np.sqrt(C_1/A_44)
N_oscillations=10
t = np.arange(0,2*np.pi/omega0*N_oscillations,0.01)
phi0=np.deg2rad(10)
phi1d0 = 0
initial_state = [phi0,phi1d0]

results = {}
X_amplitudes = {}
for name,simulation in simulations.items():

    df_result = simulation.simulate(t=t, phi0=phi0, phi1d0=phi1d0)

    results[name]=df_result
    X_amplitudes[name]=rolldecayestimators.measure.calculate_amplitudes_and_damping(X=df_result)
```

```python
omega0
```

    1.9443481778578677

## $B_e$ from anayltica solution

```python
Math(vlatex(equations.diff_eq))
```

$\displaystyle \omega_{0}^{2} y + 2 \omega_{0} \zeta \dot{y} + \ddot{y} = 0$

```python
Math(vlatex(equations.analytical_solution))
```

$\displaystyle \phi = \left(\left(\frac{\zeta \phi_{0}}{\sqrt{1 - \zeta^{2}}} + \frac{\dot{\phi}_{0}}{\omega_{0} \sqrt{1 - \zeta^{2}}}\right) \operatorname{sin}\left(\omega_{0} t \sqrt{1 - \zeta^{2}}\right) + \phi_{0} \operatorname{cos}\left(\omega_{0} t \sqrt{1 - \zeta^{2}}\right)\right) e^{- \omega_{0} t \zeta}$

```python
no_initial_speed = sp.simplify(equations.analytical_solution.subs(symbols.phi_0_dot,0))
Math(vlatex(no_initial_speed))
```

$\displaystyle \phi = \frac{\left(\zeta \operatorname{sin}\left(\omega_{0} t \sqrt{1 - \zeta^{2}}\right) + \sqrt{1 - \zeta^{2}} \operatorname{cos}\left(\omega_{0} t \sqrt{1 - \zeta^{2}}\right)\right) \phi_{0} e^{- \omega_{0} t \zeta}}{\sqrt{1 - \zeta^{2}}}$

```python
no_initial_speed_zeta_small = no_initial_speed.subs(
    [(sp.sqrt(1-symbols.zeta**2),1),
     (symbols.zeta*sp.sin(symbols.omega0*symbols.t),0),
    ])
Math(vlatex(no_initial_speed_zeta_small))
```

$\displaystyle \phi = \phi_{0} e^{- \omega_{0} t \zeta} \operatorname{cos}\left(\omega_{0} t\right)$

```python
B_1_zeta_eq = sp.Eq(symbols.B_1, 2*symbols.zeta*symbols.omega0*symbols.A_44)
B_1_zeta_eq
```

$\displaystyle B_{1} = 2 A_{44} \omega_{0} \zeta$

```python
eqs = [
    B_1_zeta_eq,
    equations.analytical_solution,
      ]

analytical_solution_B_1 = sp.Eq(symbols.phi,
                                sp.simplify(sp.solve(eqs,symbols.zeta,symbols.phi)[0][1]))
analytical_solution_B_1
```

$\displaystyle \phi{\left(t \right)} = \frac{\left(4 A_{44}^{2} \omega_{0}^{2} \phi_{0}{\left(t \right)} \cos{\left(\frac{\omega_{0} t \sqrt{4 - \frac{B_{1}^{2}}{A_{44}^{2} \omega_{0}^{2}}}}{2} \right)} + 2 A_{44}^{2} \omega_{0} \sqrt{4 - \frac{B_{1}^{2}}{A_{44}^{2} \omega_{0}^{2}}} \sin{\left(\frac{\omega_{0} t \sqrt{4 - \frac{B_{1}^{2}}{A_{44}^{2} \omega_{0}^{2}}}}{2} \right)} \frac{d}{d t} \phi_{0}{\left(t \right)} + A_{44} B_{1} \omega_{0} \sqrt{4 - \frac{B_{1}^{2}}{A_{44}^{2} \omega_{0}^{2}}} \phi_{0}{\left(t \right)} \sin{\left(\frac{\omega_{0} t \sqrt{4 - \frac{B_{1}^{2}}{A_{44}^{2} \omega_{0}^{2}}}}{2} \right)} - B_{1}^{2} \phi_{0}{\left(t \right)} \cos{\left(\frac{\omega_{0} t \sqrt{4 - \frac{B_{1}^{2}}{A_{44}^{2} \omega_{0}^{2}}}}{2} \right)}\right) e^{- \frac{B_{1} t}{2 A_{44}}}}{4 A_{44}^{2} \omega_{0}^{2} - B_{1}^{2}}$

```python
analytical_lambda = lambdify(sp.solve(analytical_solution_B_1,symbols.phi)[0])
```

```python
equations.extinction_equation
```

$\displaystyle \phi_{a} = \phi_{0}{\left(t \right)} e^{- \omega_{0} t \zeta}$

```python
sp.Eq(symbols.zeta,sp.solve(equations.extinction_equation,symbols.zeta)[0])
```

$\displaystyle \zeta = \frac{\log{\left(\frac{\phi_{0}{\left(t \right)}}{\phi_{a}} \right)}}{\omega_{0} t}$

```python
def simulate(B_e,X,omega0,A_44):
    t = X.index
    initial_states = X.iloc[0]
    phi_0 = initial_states['phi']
    phi_01d= initial_states['phi1d']
    phi_pred = analytical_lambda(omega0=omega0, A_44=A_44, phi_0=phi_0, phi_01d=phi1d0, t=t, B_1=B_e)
    return phi_pred

def residuals(B_e,X,omega0,A_44):

    phi_pred = simulate(B_e=B_e, X=X, omega0=omega0, A_44=A_44)
    residual = phi_pred - X['phi']
    return residual
```

```python
def linear_equivalent(X,omega0,A_44):
    kwargs={
        'X':X,
        'omega0':omega0,
        'A_44':A_44,
    }
    initial_guess = [B_1]
    result = least_squares(fun=residuals, x0=initial_guess, kwargs=kwargs, method='lm')
    assert result.success is True
    B_e = result.x[0]
    return B_e
```

```python
B_2 = 0
X = results[B_2]
omega0 = np.sqrt(C_1/A_44)
simulation = simulations[B_2]

B_e = linear_equivalent(X=X, omega0=omega0, A_44=A_44)
phi_pred = simulate(B_e=B_e, X=X, omega0=omega0, A_44=A_44)
phi_error = X['phi'] - phi_pred

fig,ax=plt.subplots()

label = r'Numerical: $B_1$:%0.1f, $B_2$:%0.1f' % (simulation.parameters['B_1'],simulation.parameters['B_2'])
X.plot(y='phi',ax=ax, label=label)
label = r'Analytical: $B_e$:%0.1f' % B_e
ax.plot(t,phi_pred,'--', label=label)

## Helpers:
ax.annotate(r'$\phi_0$',xy=(0,phi0),
            xytext=(3,phi0), arrowprops=dict(facecolor='black', shrink=0.05))

t0 = 2*np.pi/omega0
t0s = np.arange(t0,np.max(t),t0)
ax.plot(t0s,np.zeros(len(t0s)), 'ro', label='periods')

ax.legend()

r2_score(y_true=X['phi'], y_pred=phi_pred)
```

```python
df_B_e = pd.DataFrame()
df_B_e.index.name='B_2'

for B_2, X in results.items():
    B_e = linear_equivalent(X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'B_e'] = B_e
    phi_pred = simulate(B_e=B_e, X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'R2'] = r2_score(y_true=X['phi'], y_pred=phi_pred)

    B_e_himeno = lambdas.B_e_lambda(B_1=B_1, B_2=B_2, omega0=omega0, phi_a=phi0)
    df_B_e.loc[B_2,'B_e_himeno'] = B_e_himeno
    phi_pred_himeno = simulate(B_e=B_e_himeno, X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'R2_himeno'] = r2_score(y_true=X['phi'], y_pred=phi_pred_himeno)

df_B_e['B_1'] = B_1
```

```python
fig,ax=plt.subplots()
df_B_e.plot(y=['B_1','B_e','B_e_himeno'], ax=ax)
ax.set_ylabel('$B_e$ [Nm/s] linear equivalent damping')

fig,ax=plt.subplots()
df_B_e.plot(y=['R2','R2_himeno'],ax=ax)
ax.set_ylabel('$R^2$ [-] coefficient of determination')
ylim = ax.get_ylim()
```

## $B_2/2$

```python
df_B_e = pd.DataFrame()
df_B_e.index.name='B_2'

for B_2, X in results.items():
    B_e = linear_equivalent(X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'B_e'] = B_e
    phi_pred = simulate(B_e=B_e, X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'R2'] = r2_score(y_true=X['phi'], y_pred=phi_pred)

    B_e_himeno = lambdas.B_e_lambda(B_1=B_1, B_2=B_2/2, omega0=omega0, phi_a=phi0)
    df_B_e.loc[B_2,'B_e_himeno'] = B_e_himeno
    phi_pred_himeno = simulate(B_e=B_e_himeno, X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'R2_himeno'] = r2_score(y_true=X['phi'], y_pred=phi_pred_himeno)

df_B_e['B_1'] = B_1
```

```python
fig,ax=plt.subplots()
df_B_e.plot(y=['B_1','B_e','B_e_himeno'], ax=ax)
ax.set_ylabel('$B_e$ [Nm/s] linear equivalent damping')

fig,ax=plt.subplots()
df_B_e.plot(y=['R2','R2_himeno'],ax=ax)
ax.set_ylabel('$R^2$ [-] coefficient of determination')
ax.set_ylim(ylim)
```

## $\frac{\phi_a}{2}$

```python
df_B_e = pd.DataFrame()
df_B_e.index.name='B_2'

for B_2, X in results.items():
    B_e = linear_equivalent(X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'B_e'] = B_e
    phi_pred = simulate(B_e=B_e, X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'R2'] = r2_score(y_true=X['phi'], y_pred=phi_pred)

    B_e_himeno = lambdas.B_e_lambda(B_1=B_1, B_2=B_2, omega0=omega0, phi_a=phi0/2)
    df_B_e.loc[B_2,'B_e_himeno'] = B_e_himeno
    phi_pred_himeno = simulate(B_e=B_e_himeno, X=X, omega0=omega0, A_44=A_44)
    df_B_e.loc[B_2,'R2_himeno'] = r2_score(y_true=X['phi'], y_pred=phi_pred_himeno)

df_B_e['B_1'] = B_1
```

```python
fig,ax=plt.subplots()
df_B_e.plot(y=['B_1','B_e','B_e_himeno'], ax=ax)
ax.set_ylabel('$B_e$ [Nm/s] linear equivalent damping')

fig,ax=plt.subplots()
df_B_e.plot(y=['R2','R2_himeno'],ax=ax)
ax.set_ylabel('$R^2$ [-] coefficient of determination')
ax.set_ylim(ylim)
```

```python
X = results[B_2]
simulation = simulations[B_2]
df_error = pd.DataFrame(index=t)

B_e = linear_equivalent(X=X, omega0=omega0, A_44=A_44)
phi_pred = simulate(B_e=B_e, X=X, omega0=omega0, A_44=A_44)
df_error['least squares'] = (X['phi'] - phi_pred).abs()

B_e_himeno = lambdas.B_e_lambda(B_1=B_1, B_2=B_2, omega0=omega0, phi_a=phi0)
phi_pred_himeno = simulate(B_e=B_e_himeno, X=X, omega0=omega0, A_44=A_44)
df_error['himeno'] = (X['phi'] - phi_pred_himeno).abs()

B_e_himeno2 = lambdas.B_e_lambda(B_1=B_1, B_2=B_2/2, omega0=omega0, phi_a=phi0)
phi_pred_himeno2 = simulate(B_e=B_e_himeno2, X=X, omega0=omega0, A_44=A_44)
df_error['himeno B2/2'] = (X['phi'] - phi_pred_himeno2).abs()

fig,ax=plt.subplots()

label = r'Numerical: $B_1$:%0.1f, $B_2$:%0.1f' % (simulation.parameters['B_1'],simulation.parameters['B_2'])
X.plot(y='phi',ax=ax, label=label)

label = r'Least squares: $B_e$:%0.1f' % B_e
ax.plot(t, phi_pred, label=label)

label = r'Himeno: $B_e$:%0.1f' % B_e_himeno
ax.plot(t, phi_pred_himeno, label=label)
ax.legend()

fig,ax=plt.subplots()
df_error.plot(ax=ax)
```

## And time domain

```python
simulation = simulations[B_2]
X = simulation.simulate(t=t)

df_B = pd.DataFrame(columns=simulation.parameters.keys())
df_B.loc['simulation'] = simulation.parameters

name = 'least square'
s = pd.Series(name=name)
s['B_1'] = linear_equivalent(X=X, omega0=omega0, A_44=A_44)
s['B_2'] = 0
df_B.loc[name] = s

name = 'himeno'
s = pd.Series(name=name)
s['B_1'] = lambdas.B_e_lambda(B_1=B_1, B_2=B_2, omega0=omega0, phi_a=phi0)
s['B_2'] = 0
df_B.loc[name] = s

df_B.fillna(method='pad', inplace=True)
df_B
```

    c:\dev\prediction-of-roll-damping-using-fully-nonlinear-potential-flow-and-ikedas-method\venv\lib\site-packages\ipykernel_launcher.py:8: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

    c:\dev\prediction-of-roll-damping-using-fully-nonlinear-potential-flow-and-ikedas-method\venv\lib\site-packages\ipykernel_launcher.py:14: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

      A_44
      B_1
      B_2
      C_1

      simulation
      2.2
      0.259748
      4.464729
      8.317078

      least square
      2.2
      0.883862
      0.000000
      8.317078

      himeno
      2.2
      1.545820
      0.000000
      8.317078

```python
results_ = pd.DataFrame()

for index, row in df_B.iterrows():
    simulation = RollDecayQuadratic(**row)
    X = simulation.simulate(t=t)
    X['model'] = index
    results_=results_.append(X)
```

```python
fig,ax = plt.subplots()
groups = results_.groupby(by='model')
for model,df_ in groups:
    df_.plot(y='phi', ax=ax, label=model)
```

## Looking at amplitudes

```python
B_1_zeta_eq
```

$\displaystyle B_{1} = 2 A_{44} \omega_{0} \zeta$

```python
X = groups.get_group('simulation')
X.drop(columns='model', inplace=True)
amplitudes = rolldecayestimators.measure.calculate_amplitudes_and_damping(X=X)
amplitudes.dropna(inplace=True)
zeta = amplitudes['zeta_n']
amplitudes['B']=2*A_44*omega0*zeta

fig,ax=plt.subplots()
X.plot(y='phi', ax=ax)
amplitudes.plot(y='phi_a', ax=ax)

fig,ax=plt.subplots()
amplitudes.plot(x='phi_a',y='B', ax=ax)
```

```python
parameters = fit_on_amplitudes.fit_quadratic(y=amplitudes['B'],
                            phi_a=amplitudes['phi_a'],
                            omega0=amplitudes['omega0'],
                           )

name = 'amplitude'
s = pd.Series(name=name)
s['B_1'] = parameters['B_1']
s['B_2'] = parameters['B_2']
df_B.loc[name] = s

parameters = fit_on_amplitudes.fit_quadratic(y=amplitudes['B'],
                            phi_a=amplitudes['phi_a']/2,
                            omega0=amplitudes['omega0'],
                           )
name = 'amplitude2'
s = pd.Series(name=name)
s['B_1'] = parameters['B_1']
s['B_2'] = parameters['B_2']
df_B.loc[name] = s

df_B.fillna(method='pad', inplace=True)
df_B
```

    c:\dev\prediction-of-roll-damping-using-fully-nonlinear-potential-flow-and-ikedas-method\venv\lib\site-packages\ipykernel_launcher.py:7: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.
      import sys
    c:\dev\prediction-of-roll-damping-using-fully-nonlinear-potential-flow-and-ikedas-method\venv\lib\site-packages\ipykernel_launcher.py:18: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

      A_44
      B_1
      B_2
      C_1

      simulation
      2.2
      0.259748
      4.464729
      8.317078

      least square
      2.2
      0.883862
      0.000000
      8.317078

      himeno
      2.2
      1.545820
      0.000000
      8.317078

      amplitude
      2.2
      0.273079
      3.448405
      8.317078

      amplitude2
      2.2
      0.273079
      6.896809
      8.317078

```python
results_ = pd.DataFrame()

for index, row in df_B.iterrows():
    simulation = RollDecayQuadratic(**row)
    X = simulation.simulate(t=t)
    X['model'] = index
    results_=results_.append(X)
```

```python
fig,ax = plt.subplots()
groups = results_.groupby(by='model')
df_error = pd.DataFrame(index=t)

for model,df_ in groups:
    df_.plot(y='phi', ax=ax, label=model)

    df_error[model] = (df_['phi'] - groups.get_group('simulation')['phi']).abs()
```

```python
df_error.plot()
```


# メモ

初等代数学 硲文夫

はざまふみお

を読む

方針

1. 数式はうつす。
2. 問題はやる。少なくとも問題は書く。

定義

$a, b$ が $n$ を法として合同であるを、下記のように書く。
$$ a \equiv b  \pmod n $$
またこのような式を合同式とよぶ。

$14\equiv 8 \pmod 3$

$21 \equiv 35 \pmod 7$

$-29 \equiv 15 \pmod{11}$

```python
# 3 本線の合同 \equiv, modular congruence を sympy でどう書くかわからないのでとりあえず Eqで代用する
from sympy.abc import *
from sympy import *
Eq(a % n , b % n)
```

$\displaystyle a\bmod{n} = b\bmod{n}$

```python
from sympy.abc import *
from sympy import *
display(Eq(Mod(14,3),Mod(8,3)))
display(Eq(Mod(21,7),Mod(35,7)))
display(Eq(Mod(-29,11),Mod(15,11)))
```

$\displaystyle \text{True}$

$\displaystyle \text{True}$

$\displaystyle \text{True}$

```python
15 % 11
```

    4

$100 \equiv -32 \pmod{12}$

```python
100 % 12
```

    4

```python
-32 % 12
```

    4

```python
from sympy.abc import *
from sympy import *
display(Eq(a % n , b % n))
a= 100
b= -32
n= 12
display(Eq(a % n , b % n))

```

$\displaystyle a\bmod{n} = b\bmod{n}$

$\displaystyle \text{True}$

$\displaystyle \text{True}$

p.2

定義 1.2

整数 $a$ と正の整数 $b$ が与えられたとき、
$$
a = qb + r \quad (ただし、\,\,0\leqq r \lt b)
$$
    をみたす整数 $q$, $r$ をそれぞれ $a$ を $b$ で割った商、余りという。

命題 1.3

$a \equiv b \,\,\pmod n$ が成り立つための必要十分条件は、$a-b$ が $n$ で割り切れることである。

証明

$a, b$ をそれぞれ $n$ で割って、

\begin{eqnarray}
a &=& qn+r\quad (0 \leqq r\lt n) \tag{1.1}\\
b &=& sn + t \quad (0 \leqq t \lt n) \tag{1.2}
\end{eqnarray}
となったとする。もし $a \equiv b\,\,\pmod n$ ならば、定義により $r=t$ にならなければならない。
したがって、このとき
$$
a-b = (qn + r) - (sn +t) = qn - sn = (q - s) n
$$
となり、$a-b$ は $n$ で割り切れる。逆に $a-b$ が $n$ で割り切れるとしよう。
こととき、(1.1)、(1.2)を代入して
\begin{eqnarray}
a - b &=& (qn + r) - (sn + t)  \\
&=& (q-s)n + (r-t)
\end{eqnarray}
となるが、これが $n$ で割り切れるためんは、$r-t$ が $n$ でわりきれなければならない。
しかし、$r, t$ の範囲より、$-n \lt r - t \lt n$ であり、この中に $n$ で割り切れる数は0しかないから、結局 $r-t = 0$、すなわち $r = t$ となる。
したがって、$a$、$b$ を $n$ で割った余りが等しく、定義によって、$a \equiv b \,\,\pmod n$ となる。
$\unicode{x25af}$

p.3

命題 1.4
(a) $a \equiv b \pmod n \quadならば\quad a+c\equiv b +c \pmod n$
(b) $a \equiv b \pmod n \quadならば\quad a \times c\equiv b \times c \pmod n$
つまり、合同式の両辺に同じ数を足したり掛けたりしてよい。

証明 (a)
命題 1.3 により $a - b = qn$ となるような整数 $q$ がある。
したがって、
$$ (a+c)-(b+c) = an$$
となり、これはやはり命題 1.3 によれば、
$$ a+c \equiv b+c \pmod n $$
である。

証明 (b)
命題 1.3 により $a-b = qn$ となるような整数 $q$ がある。したがって、
\begin{eqnarray}
ac - bc &=& (a-b)c \\
&=& (qn) c \\
&=& (qc)n
\end{eqnarray}
となるから、$ac\equiv bc \pmod n$ である。$\unicode{x25af}$

定義 1.5
$0 \leqq a \leqq n-1, \,\, 0 \leqq b \leqq n - 1$ をみたす $a, b$ に対し
(a) $a+b$ を $n$ で割った余りを、$a +_n b$,
(b) $a-b$ を $n$ で割った余りを、$a -_n b$,
(c) $a \times b$ を $n$ で割った余りを、$a \times_n b$,
と書くことにする。

集合 $\{0,1,\cdots,n-1\}$ のことを、$\boldsymbol Z_n$ と書くことにする。

定義により、$a \in \boldsymbol Z_n,\,\, b \in \boldsymbol Z_n$ ならば $a +_nb,\,\, a -_n b,\,\, a \times_n b$ はどれもまた $\boldsymbol Z_n$ に属することになる。

p.4  例 1.1

3 と 9 を $\boldsymbol Z_{10}$ の元とみると
* $3 +_{10} 9 = 2 \quad$ (12 を 10 で割った余りが 2 だから)
* $3 -_{10} 9 = 4 \quad$ (-6 を 10 で割った余りが 4 だから)
* $3 \times_{10} 9 = 7 \quad (27 を 10 で割った余りが 7 だから)
$

$\boldsymbol Z_2$ においては
\begin{eqnarray}
1 +_2 1 &=& 0 \\
1 -_2 1 &=& 0 \\
1 \times_2 1 &=& 1
\end{eqnarray}

Z_{11}$ では
\begin{eqnarray}
 3 +_{11} 9 &=& 1 \\
 3 -_{11} 9 &=& 5 \\
 3 \times_{11} 9 &=& 5
 end{eqnarray}

命題 1.6
2 以上の整数 $n$ を 1 つ決めておく。このとき以下の諸性質が成り立つ。
足し算に関するもの:
(a) [結合法則]   $(a +_n b) +_n c = a +_n (b +_n c)$
(b) [交換法則]   $a +_n b = b +_n a$
(c) [零元]   $a +_n 0 = a,\quad 0 +_n a = a$
(d) [逆元]   $a +_n (n - a) = 0, \quad (n - a) +_n a = 0$

掛け算に関するもの:
(e) [結合法則]   $(a \times_n b) \times_n c = a \times_n (b \times_n c)$
(f) [交換法則]   $a \times_n b = b \times_n a$
(g) [単位元]   $a \times_n 1 = a,\quad 1 \times_n a = a$
足し算と掛け算に関するもの:
(h) [分配法則]   $(a +_n b) \times_n c = (a \times_n c) +_n (b \times_c c)$

証明 (a)
整数 $a$ を $n$ で割った余りを $r$ とすると、定義により、$a \equiv r$ が成り立つ。なぜなら、$a-r=qn$だから。
したがって、
$$ a+b\equiv a +_n b \pmod n \tag {1.3}$$
つまり、法 $n$ で考えれば、$+$ を $+_n$ でおきかえてもよい。
さて、命題 1.4 の (a) により、合同式の両辺に同じ数を加えてもよいから
$$(a+b)+c \equiv (a+_n b) + c \pmod n \tag{1.4}$$
が成り立つ。この右辺に (1.3) の規則を適用すれば、
$$(a +_n b) + c \equiv (a +_n b) +_n c \pmod n \tag{1.5}$$
となり、したがって (1.4) と (1.5) より
$$(a + b) + c \equiv (a +_n b) +_n c \pmod n \tag {1.6}$$
が成り立つ。全く同様にして、
$$a + (b + c) \equiv a +_n (b +_n c) \pmod n \tag {1.7}$$
がなりたつ。整数の世界ではもちろん
$$(a + b) + c = a + (b + c) \tag{1.8}$$
であるから、 (1.6), (1.7), (1.8) より、
$$(a +_n b) +_n c \equiv a +_n (b +_n c) \pmod n$$
がいえる。ところがこの両辺とも $\boldsymbol Z_n$$ の元だから、法 $n$ で合同ということは等しいということである。よって、

$$(a +_n b) +_n c = a +_n (b +_n c)$$

証明 (b)
\begin{eqnarray}
a + b &\equiv& a+_n b \pmod n \tag{1.9}
\end{eqnarray}

```python
from sympy.abc import x
from sympy import *
(((x + 1) % 2) + 1) % 2
```

$\displaystyle x\bmod{2}$

```python
from sympy.abc import *
from sympy import *
Eq(Mod(x,2), Mod(y,2))
```

$\displaystyle x\bmod{2} = y\bmod{2}$

```python
a ~ b
```


```python
from pycalphad import Database, Model, calculate
import pycalphad.variables as v
import numpy as np
import sympy
```

```python
dbf = Database('NI_AL_DUPIN_2001.TDB')

class NewModel(Model):
    def build_phase(self, dbe, phase_name, symbols, param_search):
        self.models['test'] = -13000*v.T*v.R*v.Y(phase_name, 0, 'AL')*v.Y(phase_name, 0, 'NI')
    bubbles = property(lambda self: self.models['test'] / 10)

# pycalphad needs custom properties to be defined for all phases, at the moment
# Maybe 'bubbles' has no meaning for FCC, so we return NaN
class FCCModel(Model):
    # Constant values need to be wrapped in sympy.S() so they are made symbolic
    bubbles = property(lambda self: sympy.S(np.nan))
```

```python
result = calculate(dbf, ['AL', 'NI', 'VA'], ['LIQUID', 'FCC_L12'], model={'LIQUID': NewModel, 'FCC_L12': FCCModel},
                   output='bubbles', T=1500, P=101325, points={'LIQUID': [[0.9, 0.1], [0.2, 0.8]],
                                                               'FCC_L12': [[0.2, 0.8, 0.2, 0.8, 1], [0.4, 0.6, 0.6, 0.4, 1]]})
```

```python
result['bubbles'].values[np.nonzero(result.Phase.values == 'LIQUID')]
```

    array([-1459194.75, -2594124.  ])

```python
result['bubbles'].values[np.nonzero(result.Phase.values == 'LIQUID')]
```

    array([ nan,  nan])

```python

```


# Feature Selection
Artur Back de Luca

GEIA - Grupo de estudos em Inteligência Aritificial

## Content:

1. The context;
2. The problem;
3. Possible solutions;
4. Feature selection approaches: Filter, Wrapper and Embedded methods;

Occam's razor (law of parsimony)

_"Entities should not be multiplied without necessity."_

When presented with competing hypotheses that make the same predictions, one should select the solution with the fewest assumptions

# 1. Context

What is a feature

In machine learning and pattern recognition, a **feature** (or an attribute) is an individual measurable property or characteristic of a phenomenon being observed. [1]

1. The context

    1997: very few domains operated with more than 40 features [1]

Soybean - 1988 (35 features)

Letter Recognition - 1991 (16 features)

Lung Cancer - 1992 (32 features)

2009: ImageNet - 14 million images with 256x256 pixels (+196k features) each, and more than 20,000 categories [2]

2010: The Wikipedia Corpus - almost 1.9 billion words from more than 4 million articles [3]

2011: Cancer detection based on gene expression (e.g.: Colon dataset - 2,000 features) [4]

Source: [5]

ImageNet: 256x256x3 = 196608

Wikipedia: 1.9 billion words, not distinct, of course - the Second Edition of the 20-volume Oxford English Dictionary, published in 1989, contains full entries for 171,476 words in current use, and 47,156 obsolete words.

# 2. The problem

2. The problem

Too many features, too many problems

What problems arise with too many features?

---

1. require longer training times*

2. jeopardize human interpretability*

3. worsen prediction quality due to sample size effects

4. potentially increase overfitting

* Considered self-explainatory, will not be explained below

2. The problem

3. Worsen prediction quality due to sample size effects

Couldn't a predictor simply disregard irrelevant features?

---

To answer this, we will have to resort to some statistical learning theory, exploring the ways of estimating functional dependency from a given collection of data.

2.3. The problem: Worsen accuracy due to sample size effects

Statistical Learning Theory

Let $X \in \mathbb R^p$ be a input random vector and $Y\in \mathbb R$ be an output random variable, with joint distribution $P(X,Y)$.

The task of learning aims at finding a function $f(X)$ for predicting $Y$ given values of input $X$. This structure requires a *loss function* $L(Y, f(X))$ for identifying and penalizing errors in prediction. With this structure, we can define a criterion for choosing a suitable $f$ known as the *Statistical Risk ($R$)*.

$$
\begin{equation}
\begin{split}
\text{R}(f) & =  \mathbb{E} \thinspace L(Y, f(X)) \\
 & = \int L(y, f(x))\thinspace P(dx,dy) \\
 & = \mathbb{E}_{X}\mathbb{E}_{Y|X} [L(Y, f(X))|X]
\end{split}
\end{equation}
$$

Source: [5]

This criterion tells us how well, on average, the predictor $f$ performs with respect to the chosen loss function

Instead of working with the joint distribution, we can condition the Statistical Risk on $X$
$$
\begin{equation}
\begin{split}
\text{R}(f) & = \int\int [L(y, f(x))|x]\thinspace P(dy)\thinspace P(dx) \\
 & = \mathbb{E}_{X}\mathbb{E}_{Y|X} [L(Y, f(X))|X]
\end{split}
\end{equation}
$$

Finally, what we seek is a $f(X)$ which minimizes the the Risk:

$$f(X)_{opt}= argmin_c \mathbb{E}_{X}\mathbb{E}_{Y|X}[L(Y,c)|X]$$

2.3. The problem: Worsen accuracy due to sample size effects

The optimal solution will be different depending on the metric used

Regression

Mean Squared Error (MSE)

$$
\begin{equation}
\begin{split}
f(X)_{opt} & = argmin_c \mathbb{E}_{X}\mathbb{E}_{Y|X}[(Y-c)^2|X] \\
 & = \mathbb{E}(Y|X=x)
\end{split}
\end{equation}
$$
known as the conditional mean or expectation - the "average" value over an arbitrarily large number of occurrences

Mean Absolute Error (MAP)

$$
\begin{equation}
\begin{split}
f(X)_{opt} & = argmin_c \mathbb{E}_{X}\mathbb{E}_{Y|X}[|Y-c|\thinspace|X] \\
 & = median(Y|X=x)
\end{split}
\end{equation}
$$
or the conditional median of the distribution

Regression loss functions for SLT

**MSE**: the most preferable option, due to its ease of computation of minimum, since it is differentiable. However it is more sensitive to outliers as a big difference becomes even larger by squaring them.

**MAP**: its optimal estimates are more robust than those for the conditional mean. However, MAP has discontinuities in their derivatives, which have hindered their widespread use.

2.3. The problem: Worsen accuracy due to sample size effects

Classification

0-1 loss

$$
\begin{equation}
\begin{split}
f(X)_{opt} & = argmin_c \mathbb{E}_{X}\mathbb{E}_{Y|X}[I(f(X),Y)] \\
 & = \underset{y \in Y}{\max}P(y|X)
\end{split}
\end{equation}
$$
where ${\displaystyle I}$ is the indicator function:

$$I :=\begin{cases}
0\text{, if } f(X) = Y\\
1\text{, otherwise}\\
\end{cases}$$

The decider is also known as the *Optimal Bayes classifier*.

If the joint probability $P(X,Y)$ is known and the classification decision is optimal.

This doesn't mean that there are no errors, rather than the lowest error achievable, resulted by noise among distributions.

2.3. The problem: Worsen accuracy due to sample size effects

If the probability distribution for the problem was known, the classifier wouldn't be affected by adding more of features.

If such features carried the slightest contributions, it is shown that the Bayes optimal classifier tends to zero as the number of features approach infinity.

However, the probability distribution used is an estimation, based on the finite set of samples, causing a **peaking phenomenon**: the prediction accuracy increases with the number of features, but soon reaches a peak, in which the noise becomes larger than the separability increase caused by the new feature.

Source: [6]

This is often called the peaking phenomenon. Some ideas evolved around it are discussed in the great pattern recognition blog by Ela Pekalska and Bob Duin called [37steps](http://37steps.com/about/):
 - [Peaking summarized](http://37steps.com/2545/peaking-summarized/)
 - [Peaking paradox](http://37steps.com/2279/the-peaking-paradox/)
 - [Trunk’s example of the peaking phenomenond](http://37steps.com/2448/trunks-example/)
 - [The curse of dimensionality](http://37steps.com/2349/curse-of-dimensionality/)

2. The problem

 4. potentially increase overfitting

As the number of features increase, the observations become more sparse within the feature space.

Having the observations further apart makes it difficult for the estimator to generalize, increasing its variance, i.e. relying on the specific observations to produce new predictions, causing overfitting.

Consequences for classical:

**Non-parametric (Local methods)**: methods such as the $k$ *nearest neighboors*, as the examples become increasingly sparse, the approximation of the conditional expectation by taking the average of its nearest neighboors becomes correspondingly worse as the query point distance itself from the known examples. Additionally, in high dimensions, as the datapoints become more spread apart, their distances become more uniform, making it dificult for the algorithm to decide which data points are more relevant to region of interest [Source](https://www.youtube.com/watch?v=dZrGXYty3qc&list=PLXP3p2MXX_ahmhsMiP5YWtc7IcABeFOGH&index=2)

**Parametric methods**:  In very high dimensional spaces, there is more than one plane that can be fitted to your data, and without proper type of regularization can cause the model to behave very poorly. . Collinearity often results in overfitting, i.e. in a too efficient modelling of learning samples without model generalization ability. [[7]]()

More examples on  check the answer on [stackexchange](https://stats.stackexchange.com/questions/186184/does-dimensionality-curse-effect-some-models-more-than-others)

What causes these problems?

The curse of dimensionality

2. The problem

The curse of dimensionality

Increasing the number of factors to be taken into consideration requires an exponential growth of observations [5]

           However, having a small number of samples means that many regions of the feature space are never observed

# 3. How to overcome this problem?

Ans.: Dimensionality reduction

Dimensionality reduction

The process of reducing the number attributes under consideration by obtaining a set of principal variables.

It can be carried out through two distinct techniques: feature *extraction* and *selection*

Feature extraction

Feature extraction projects the original high-dimensional features to a new feature space with low dimensionality through a combination of the original features.

 - **Advantages**: It can generate meaning with incomprehensible features
 - **Disavantages**: It may lose physical meanings of comprehensible features

**Types:**
 1. Ad hoc techniques;
 2. Data transformation techniques;
 3. Deep learning techniques.

3.1 Feature extraction

1. Ad hoc techniques

Consists on relying on user expertise to derive meaningful relationship between features
 - **Advantages**: Interpretability
 - **Disavantages**: Time-consuption and prior knowledge

A fundamental step in this process is to perform an exploratory analysis - [*Facets*](https://ai.googleblog.com/2017/07/facets-open-source-visualization-tool.html) may be a helpful tool.

3.1 Feature extraction

2. Data transformation and clustering techniques

Instead of relying solely on the expert's knownledge on the data, several techniques have been already developed to for the purpose of reducing the dimensions of a problem

                    t-SNE

                    MDS

                    PCA

                    scikit.learn

                    scikit.learn

                    scikit.learn

3.1 Feature extraction

3. Deep learning techniques

Deep learning architectures successively combine feature transformations in order to generate higher hierarchical representations that are more associated with the given task in hand.

Examples: Stacked autoenconders, Deep Belief Nets, Sparse filtering and Convolutional layers

```python
from IPython.core.display import HTML

zoom = 0.9
width = 900
height = 800
url = "https://distill.pub/2018/building-blocks/#AttributionGroups"

html = (
    '''
        <style type="text/css">
        #frame {
            zoom: {zoom};
            -moz-transform:scale({zoom});
            -moz-transform-origin: 0 0;
            -o-transform: scale({zoom});
            -o-transform-origin: 0 0;
            -webkit-transform: scale({zoom});
            -webkit-transform-origin: 0 0;
        }
        </style>

        <a style="font-size:0.6em" href="{url}">Source</a>
    '''
    .replace("{zoom}", str(zoom))
    .replace("{width}", str(width))
    .replace("{height}", str(height))
    .replace("{url}", url)
)

```

```python
HTML(html)
```

Source

Feature selection

Feature selection addresses the task of finding a subset of $M$ features from a set of $N$ features, $M < N$, such that the value of a criterion function is optimized over all subsets of size $M$ either:
1. improving prediction quality;
2. decreasing the size of the structure without significantly decreasing performance; or
3. maintaining the original distribution.

Source: [11]

---

 - **Advantages**: It retains the physical meaning of features
 - **Disavantages**: It cannot generate new meaning using the original features

3.2 Feature selection

 What to select?

A good subset of features should include a subset of the relevant features that optimizes some performance function. The relevance of features can be classified as:

 - Strongly relevant: those that if removed handicap the prediction quality of the feature set;
 - Weakly relevant: those that if removed handicap the prediction quality for a particular feature subset;
 - Irrelavant features: those that are not neither strongly or weakly relevant.

Source: [12]

4. Feature selection methods

1. Filter methods;
2. Wrapper methods;
3. Embedded methods.

Disclaimer: This is just the tip of the iceberg

 - There are many more techniques than those here presented;
 - Feature selection methods shown are designed for the assumption that the features are independent (*Flat features*). For some cases (e.g. Image Recognition), this assumption does not hold. Check [[13]](#Reference) for a survey on *Strucutred Features* as well.

4. Feature selection methods

 Filter methods

Filter methods are independent of any algorithms and are applied before  learning. They rely on characteristics of data to assess feature importance thus discarding less relevant features.

 - **Advantages**: Filter methods are typically computationally efficient

 - **Disavantages**: Due to the lack of a specific learning algorithm guiding the feature selection phase, the selected features may not be optimal; Univariate methods do not capture redundancy.

A typical filter method consists of two steps:
1. Feature importance is ranked according to some feature evaluation criteria
2. Lowly ranked features are filtered out either by a threshold of acceptance or by a limiting number of features to be selected.

Some of the evaluation criteria are:
1. Correlation-based Criteria
2. Fisher Score
3. Information Gain

3.2.1 Filter methods

1. Correlation Criteria

One of the simplest criteria used is the Pearson correlation coefficient:

$$R(i) = \frac{cov(X_i, Y)}{\sqrt{var(X_i)var(Y)}}$$

where $X_i$ is the $i_{th}$ feature.

Although useful, the  coefficient fails to detect important non-linear dependecies. To overcome this is the user may perform non-linear transformations onto the input, or use different correlation criteria, such as the [Spearman's rank correlation](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) or [Kendall's Tau](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient).

---

Python implementation: [`scikit-learn.feature_selection`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression) (Pearson) and [`microsoft azure`](https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/feature-selection-modules) (Pearson, Kendall and Spearman)

Additionally, the relationship described by such coefficients can be misleading, as demonstrated by [Anscombe's quartet](https://en.wikipedia.org/wiki/Anscombe's_quartet)

Here all of the relationships shown possess the same correlation coefficient, however, it is evident that they do not share the same relationship between variables. This is an important concept to exemplify the limitations of a coefficient and how far can we trust it

3.2.1 Filter methods

2. Fisher Score

Features with high quality should assign similar values to instances in the same class and different values to instances from different classes. With this intuition, the score for the $i_{th}$ feature $S_i$ will be calculated by Fisher Score as,
$$S_i = \frac{\sum^K_{j=0}n_j(\mu_{ij}-\mu_{i})^2}{\sum^K_{j=0}n_j\thinspace var(X_{ij})^2}$$
where $\mu_{ij}$ and $\mu_{j}$ are the mean of the $i_{th}$ feature and the mean of the $j_{th}$ class of the $i_{th}$ feature, respectively, while $K$ is the total number of features and $n_j$ is the number of instances in the $j_{th}$ class on $X_i$.

---

Python implementation: [`scikit-feature`](https://github.com/jundongl/scikit-feature)

Source: [13]

3.2.1 Filter methods

3. Information Gain (Mutual information)

Information Gain expresses the dependence of one variable to another, quantifying how much knowing a feature reduces uncertainty about the outcome.

$$I_i = \sum_{x_i}\sum_yP(X=x_i, Y=y)\thinspace log\frac{P(X=x_i,Y=y)}{P(X=x_i)P(Y=y)}$$

where the probabilities used are an estimation derived from the training set.

If X and Y were independent then MI would be zero and greater than zero if they were dependent.

We can select our features from feature space by ranking their mutual information with the target variable.

---

Python implementation: [`scikit-learn.feature_selection`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#sklearn.feature_selection.mutual_info_classif)

3.2.1 Filter methods

Addendum: other relevant techniques

When categorical data is used, other statistical tests are used in order to determine whether the a feature is independent or not from a desired variable. [Chi-Square](https://www.quora.com/How-is-chi-test-used-for-feature-selection-in-machine-learning) test, implemented under [`scikit-learn.feature_selection`](https://scikit-learn.org/stable/modules/feature_selection.html) module

Alternatively, the [RELIEF](https://en.wikipedia.org/wiki/Relief_(feature_selection)) is used to determine consistent dependency between features and classes. Python implementation: [`scikit-rebate`](https://github.com/EpistasisLab/scikit-rebate)

3.2.1.1 Filter Methods

*Important:* Even though filter methods try to select useful features individually, [[2, pg. 1165]](#References) shows that a variable that is completely useless by itself can provide a significant performance improvement when taken with others.

3.2 Feature selection

 Wrapper methods

Wrapper methods rely on the predictive performance of a predefined learning algorithm to evaluate the quality of selected features.

 - **Advantages**: Simplicity; Can in principle find the optimal set of features

 - **Disavantages**: Are prone to overfitting; Computationally expensive

A typical wrapper method consists of two steps:
1. search for a subset of features
2. evaluate the selected features.
3. Repeat *1.* and *2.* until some stopping criteria are satisfied.

Finding the optimal set of features has been shown to be unable to solve in polynomial-time*. [[14]](#References),  therefore using a exaustive search approach is intractable. Moreover, oftentimes the search space in too large ($2^n$) Therefore, different search strategies have been employed:
1. Sequential selection algorithms
2. Branch-and-bound
3. Metaheuristics

*Under the assumption that $P ≠ NP$. For more information on computational complexity theory check this article

3.2.2 Wrapper methods

1. Sequential Search

1.  **Sequential Feature Selection (SFS)**

 It Starts with an empty set and gradually adds the feature that most improved the subset predictive performance until it reaches a stopping criteria (predetermined number of features, or the *probing method*).

2. **Sequential Backward Selection (SBS)**

Similar to SFS, but it starts with the complete set of features and gradually removes the feature that gives the lowest decrease in prediction performance.

3. **Sequential Floating Forward Selection (SFFS)**

It starts with an empty number and iteratively adds a SFS step followed by a SBS step.

4. **Sequential Backward Floating Selection (SBFS)**

It starts with the complete set and iteratively adds SBS step followed by a SFS step

The forward methods more efficient than their backward counterpart, as they start with smaller subsets.

---

[Further reading](https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/) and Python implementation: [`mlxtend.feature_selection`](https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/#api)

Source: [15]

3.2.2 Wrapper methods

2. Branch-and-bound

Developed in the 1960's the branch and bound is one of the most commonly known algorithms for solving NP-hard problems.

The subset combinations are arranged in a tree-like structure and bounds on the prediction quality are established throughout the alogrithmic search in order to prevent the exploration of non-optimal candidates

Narendra and Fukunaga (1977) [[16]](#Reference) proposed using branch and bound for feature selection problems. However, an optimal solution can only be guaranteed if monotonicity is satisfied (the addition of a feature to a subset can never decline the prediction quality). This assumption rarely holds in the real-world.

---

Python implementation: [`enusearch`](https://github.com/artur-deluca/enusearch)

Source: [15]

3.2.2 Wrapper methods

3. Metaheuristics

Metaheuristics are iterative techniques that use strategies for exploring and exploiting the search space in order to efficiently find near-optimal solutions. [[17]](#References)

There is a plethora of metaheuristics, some of which have been successfully employed in feature selection:

1. Genetic algortihms (GA) [Kudo and Sklansky (2000)](http://www.sciencedirect.com/science/article/abs/pii/S0031320399000412)
2. Particle swarm optimization (PSO) ([Unler and Murat (2010)](http://www.sciencedirect.com/science/article/pii/S0377221710001633))
3. Tabu search (TS) ([Pacheco, Casado and Núñez (2009)](https://www.sciencedirect.com/science/article/abs/pii/S037722170800828X), [Zhang and Sun (2002)](https://www.sciencedirect.com/science/article/abs/pii/S0031320301000462))

As experimentally demonstrated in Unler and Kudo, metaheuristics find more satisfying results as Sequential search techniques oftentimes in less time. However, due to their stochastic nature, oftentimes the results are not consistent as the ones reproduce by the aforementioned techniques

---
Python implementations: GA: [`gaft`](https://github.com/PytLab/gaft) and PSO:[`PSOpt`](https://github.com/artur-deluca/psopt/)

Source: [15]

Note: Performance assessments are usually done using a validation set or by cross-validation

3.2 Feature selection

 Embedded methods

Embedded methods, are quite similar to wrapper methods since they are also used to optimize the objective function or performance of a learning algorithm or model. The difference to wrapper methods is that an intrinsic model building metric is used during learning

Embedded methods perform variable selection in the process of training and are usually specific to given learning machines.

 - **Advantages**: Make better usage of data (doesn't split it for evaluation). It is less computationally expensive and less prone to overfitting than wrapper methods

 - **Disavantages**: More computationally expensive than filter methods.

Some of the embedded techniques are:

1. Built-in selectors
2. Prunning methods
3. Regularization methods

3.2.3 Embedded methods

1. Built-in selectors

Decision tree (CART) algorithms such as the ID3 and C4.5 use feature selection methods in its operations.

The algorithm iterates through every unused attribute of the dataset and calculates the information gain of that attribute. The set is then split by the attribute with highest information gain to produce subsets of data. The algorithm continues to recur on each subset, considering only attributes never selected before.

The algorithm may stop if each example within a certain subset belongs to a class, or any other limitation (given by the number of splits allowed or available attributes)

In these circumstances, these algorithms can sort through features and decide which are the most relevant among them as the model is being built.

---
Python implementation: [`scikit-learn.tree`](https://scikit-learn.org/stable/modules/tree.html#tree-algorithms-id3-c4-5-c5-0-and-cart)

Further reading

C4.5 made a number of improvements to ID3. Some of these are:

- Handling both continuous and discrete attributes
- Handling training data with missing attribute values
- Handling attributes with differing costs.
- Pruning trees after creation - C4.5 goes back through the tree once it's been created and attempts to remove branches that do not help by replacing them with leaf nodes.

3.2.3 Embedded methods

2. Prunning methods

Recursive feature elimination (RFE)

Recursively eliminates one feature at iteration until predetermined number of features is reached.

It differs from traditional wrapper methods because it uses internal information from the learning algorithm to endorse the elimination.

For linear predictors, RFE eliminates the feature that carries the smallest weight as for decision trees and random forest classifiers, the concept feature importance is used, which can be defined by:
   - *Gini Importance or Mean Decrease in Impurity (MDI)*: the sum over the number of splits (accross all tress) that include the feature, proportionaly to the number of samples it splits.
   - *Permutation Importance or Mean Decrease in Accuracy (MDA)* is assessed for each feature by removing the association between that feature and the target. This is achieved by randomly permuting the values of the feature and measuring the resulting increase in error.

---

Python implementation: [`scikit-learn.feature_selection`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html)

Under scikit-learn, only the Gini importance can used for RFE in tree-based classifiers.

Source: article and [13]

3.2.3 Embedded methods

3. Regularization methods

Regularization methods introduce penalization into the loss function to balance the decision between accuracy and model simplicity. Models with too many parameters have higher penalties, and if a weight associated with a feature sinks below a certain threshold, its value goes to zero.

- Lasso ($L_1$ Regularization)
$$penalty\thinspace(w) = \sum^n_{i=1}\vert w_i\vert$$

- Elastic-net regularization
$$penalty\thinspace(w) = \lambda_1\sum^n_{i=1}\vert w_i\vert + \lambda_2(\sum^n_{i=1}w^2_i), \thinspace\lambda\ge0$$

---

- Adpative Lasso regularization
- Bridge regularization:

---

Python implementation: [`sklearn.linear_model`](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model)

Source: [13]

- L2 regularization (Ridge regression) is not suitable for feature selection since its penalization doesn't reduce coefficients to zero

L1:
- L1 requires a higher coefficient for feature selection than for simple regularization
- When p > n, L1 tends to select n variables before it saturates
- Small nonzero parameters cannot be detected consistently
- High correlations between predictors leads to poor selection performance

Elastic-net:

The quadratic penalty term makes the loss function strongly convex, and it therefore has a unique minimum.

## Wrap up

**1**  *Do you have domain knowledge?*
If yes, construct a better set of “ad hoc” features.

**2** *Are your features commensurate?* If no, consider normalizing them.

**3** *Do you suspect interdependence of features?* If yes, expand your feature set by constructing
conjunctive features or products of features, as much as your computer resources allow you.

**4** *Do you need to prune the input variables (e.g. for cost, speed or data understanding reasons)?* If no, construct disjunctive features or weighted sums of features (e.g. clustering or matrix factorization)

**5** Do you need to assess features individually (e.g. to understand their influence on the system
or because their number is so large that you need to do a first filtering)? If yes, use a variable
ranking method; else, do it anyway to get baseline results.

## Wrap up

**6** Do you need a predictor? If no, stop.

**7** Do you suspect your data is “dirty” (has a few meaningless input patterns and/or noisy
outputs or wrong class labels)? If yes, detect the outlier examples using the top ranking
variables obtained in step 5 as representation; check and/or discard them.

**8** Do you know what to try first? If no:
- Take a linear predictor and use a forward selection method with the “probe” method as a stopping criterion or use the 0-norm embedded method.
- For comparison, following the ranking of step 5, construct a sequence of predictors of same nature using increasing subsets of features. Can you match
or improve performance with a smaller subset? If yes, try a non-linear predictor with that subset.

**9** Do you have new ideas, time, computational resources, and enough examples? If yes,
compare several feature selection methods, including your new idea, correlation coefficients,
backward selection and embedded methods. Use linear and non-linear predictors.
Select the best approach with model selection.

**10** Do you want a stable solution (to improve performance and/or understanding)? If yes, subsample your data and redo your analysis for several subsets.

For a more comprehensive survey on feature selection methods and their recommended applications see: [[18]](#References) and [[this article](http://featureselection.asu.edu/algorithms.php)]

# References

[1]    : Bishop, C. (2006). Pattern recognition and machine learning

[2]    : Guyon, I. and Elissee, A. (2003). An Introduction to Variable and Feature Selection

[3]: Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., C. Berg, A. and Fei-Fei, L. (2015) ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015

[4]:Kossinets, G. (2010). Processed Wikipedia Edit History. Stanford large network dataset collection

[5]: Liu, Q., Sung, A. H., Chen, Z., Liu, J., Chen, L., Qiao, M., Deng, Y. (2011). Gene selection and classification for cancer microarray data based on machine learning and similarity measures.

[6]: Hastie, T., Tibshirani, R., and Friedman, J. H. (2009). The elements of Statistical learning: data mining, inference, and prediction.

[7]: Trunk, G. V. (1979). A Problem of Dimensionality: A Simple Example

    [8]: Verleysen, M. and François, D. (2005). The Curse of Dimensionality in Data Mining and Time Series Prediction

     [9]: Ng, A. Y. and Jordan, M. I. (2001). On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes

     [10]: Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R. P., Tang, J., and Liu, H. (2017). Feature Selection: A Data Perspective

    [11]: Dash, M. and Liu, H. (1997). Feature selection for classification

    [12]: Kohavi, R. (1994). Feature Subset Selection as Search with Probabilistic Estimates

    [13]: Tang, J., Alelyani, S., Liu, H. (2014). Feature selection for classification: A review

    [14]: Amaldi,E., Kann, V. (1998). On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems

    [15]:Chandrashekar, G., Sahin, F. (2014). A survey on feature selection methods

      [16]: Narendra, P. M., Fukunaga, K. (1977). A Branch and Bound Algorithm for Feature Subset Selection

          [17]: Rayward-Smith, V., Osman, I., Reeves, C. D., Smith, G. (1996). Modern Heuristic Search Methods

      [18]: Jović, A., Brkić, K., and Bogunović, N. (2015). A review of feature selection methods with applications


### Notebook configration

**Important**
Due to an error in the while loop of the AQGD optimizer in Qiskit-Aqua version 0.6.4, I have a separate repo for solving it and also I have updated the returned values to include the history of the loss function so that we can plot it.
The following two cells must be run sequentially.

```python
!pip install git+https://github.com/kareem1925/qiskit-aqua.git@kareem1925-mentor
```

    Collecting git+https://github.com/kareem1925/qiskit-aqua.git@kareem1925-mentor
      Cloning https://github.com/kareem1925/qiskit-aqua.git (to revision kareem1925-mentor) to /tmp/pip-req-build-vmp3xf8c
      Running command git clone -q https://github.com/kareem1925/qiskit-aqua.git /tmp/pip-req-build-vmp3xf8c
      Running command git checkout -b kareem1925-mentor --track origin/kareem1925-mentor
      Switched to a new branch 'kareem1925-mentor'
      Branch 'kareem1925-mentor' set up to track remote branch 'kareem1925-mentor' from 'origin'.
    Collecting qiskit-terra>=0.11.0
      Downloading https://files.pythonhosted.org/packages/30/6e/3aada46aa05013279b918002a6123d9cf5d4014d65cef90f7f5306969ccc/qiskit_terra-0.12.0-cp36-cp36m-manylinux2010_x86_64.whl (2.9MB)
         |████████████████████████████████| 2.9MB 2.8MB/s
    Collecting qiskit-ignis>=0.2.0
      Downloading https://files.pythonhosted.org/packages/78/f2/f4f89c65d2c13f473092f7e800519836611ddeeaab8d4591a70555b26a37/qiskit_ignis-0.2.0-py3-none-any.whl (100kB)
         |████████████████████████████████| 102kB 12.4MB/s
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (1.4.1)
    Collecting sympy>=1.3
      Downloading https://files.pythonhosted.org/packages/ce/5b/acc12e3c0d0be685601fc2b2d20ed18dc0bf461380e763afc9d0a548deb0/sympy-1.5.1-py2.py3-none-any.whl (5.6MB)
         |████████████████████████████████| 5.6MB 21.2MB/s
    Requirement already satisfied: numpy>=1.13 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (1.17.5)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (5.4.8)
    Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (0.22.1)
    Collecting dlx
      Downloading https://files.pythonhosted.org/packages/54/c0/b8fb5bb727e983b6f5251433ef941b48f38c65bb0bd6ec509e9185bcd406/dlx-1.0.4.tar.gz
    Collecting docplex
      Downloading https://files.pythonhosted.org/packages/1c/06/de7833f2e70d51286611a2676f130f0978d2c16f65047cb965d0af579d08/docplex-2.12.182.tar.gz (565kB)
         |████████████████████████████████| 573kB 39.3MB/s
    Requirement already satisfied: fastdtw in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (0.3.4)
    Collecting quandl
      Downloading https://files.pythonhosted.org/packages/07/ab/8cd479fba8a9b197a43a0d55dd534b066fb8e5a0a04b5c0384cbc5d663aa/Quandl-3.5.0-py2.py3-none-any.whl
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (45.1.0)
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (2.8.0)
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (2.4)
    Requirement already satisfied: cvxopt in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.7.0) (1.2.4)
    Collecting pyscf
      Downloading https://files.pythonhosted.org/packages/22/37/c9ed47f2b3f92a4f6955ab5454bf364cee6c05edb40e35b31b21b56d39f4/pyscf-1.7.0-cp36-cp36m-manylinux1_x86_64.whl (24.3MB)
         |████████████████████████████████| 24.3MB 28.9MB/s
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra>=0.11.0->qiskit-aqua==0.7.0) (0.3.1.1)
    Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra>=0.11.0->qiskit-aqua==0.7.0) (2.6.0)
    Collecting ply>=3.10
      Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
         |████████████████████████████████| 51kB 8.7MB/s
    Collecting marshmallow-polyfield<6,>=5.7
      Downloading https://files.pythonhosted.org/packages/b5/9a/8c303051f9d0f844c06b8d607f7578a2ac9aec2ffe02947fea5d5ba589d8/marshmallow_polyfield-5.8-py3-none-any.whl
    Collecting marshmallow<4,>=3
      Downloading https://files.pythonhosted.org/packages/54/b5/d10256349371486f195edce0bb5cf53aeb757e887a251cf98286eda8b4ec/marshmallow-3.5.0-py2.py3-none-any.whl (45kB)
         |████████████████████████████████| 51kB 10.1MB/s
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.3->qiskit-aqua==0.7.0) (1.1.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.7.0) (0.14.1)
    Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from docplex->qiskit-aqua==0.7.0) (2.21.0)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from docplex->qiskit-aqua==0.7.0) (1.12.0)
    Collecting docloud>=1.0.375
      Downloading https://files.pythonhosted.org/packages/89/1c/fcb7ea86ae5cb7137a22f00bcaa785de2d102bd4a2b16383ec4a877a835b/docloud-1.0.375.tar.gz
    Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.7.0) (2.6.1)
    Requirement already satisfied: pandas>=0.14 in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.7.0) (0.25.3)
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.7.0) (8.2.0)
    Collecting inflection>=0.3.1
      Downloading https://files.pythonhosted.org/packages/d5/35/a6eb45b4e2356fe688b21570864d4aa0d0a880ce387defe9c589112077f8/inflection-0.3.1.tar.gz
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.2->qiskit-aqua==0.7.0) (4.4.1)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->docplex->qiskit-aqua==0.7.0) (1.24.3)
    Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->docplex->qiskit-aqua==0.7.0) (2.8)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->docplex->qiskit-aqua==0.7.0) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->docplex->qiskit-aqua==0.7.0) (2019.11.28)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.14->quandl->qiskit-aqua==0.7.0) (2018.9)
    Building wheels for collected packages: qiskit-aqua, dlx, docplex, docloud, inflection
      Building wheel for qiskit-aqua (setup.py) ... done
      Created wheel for qiskit-aqua: filename=qiskit_aqua-0.7.0-cp36-none-any.whl size=1740668 sha256=c0819967289d72b641d9c532b1f0cf57bb3b4b658d2281ffc53ed9a626ec6b32
      Stored in directory: /tmp/pip-ephem-wheel-cache-i66m_6uo/wheels/2f/b7/f1/c656b56b5969883339b3d045afd5cc6c163d6a71ce8ab17d41
      Building wheel for dlx (setup.py) ... done
      Created wheel for dlx: filename=dlx-1.0.4-cp36-none-any.whl size=5712 sha256=0bcb67e79c77aa3a3bcbd9ae582689284a8f1a6bab8f9b3d6ecb88750b8ad7e0
      Stored in directory: /root/.cache/pip/wheels/bb/ba/15/fdd0deb104df3254912998150ba9245668db06b00af5912d1a
      Building wheel for docplex (setup.py) ... done
      Created wheel for docplex: filename=docplex-2.12.182-cp36-none-any.whl size=627978 sha256=f87dc56e25bb4f3c7684d5bb1aba4ca219a4786ab3f250e7e08f442fdb690d62
      Stored in directory: /root/.cache/pip/wheels/9b/ef/71/204f676be3eae44c121c5b779077bd06f573d86cc6524aecc8
      Building wheel for docloud (setup.py) ... done
      Created wheel for docloud: filename=docloud-1.0.375-cp36-none-any.whl size=19481 sha256=47585283a927b07182abdf3cb202a143a91bd8e1184f6773466b5a360122a20f
      Stored in directory: /root/.cache/pip/wheels/4e/24/19/9b0e57545d7543ecec35c5faff78c058bf0c2506369c8b1d31
      Building wheel for inflection (setup.py) ... done
      Created wheel for inflection: filename=inflection-0.3.1-cp36-none-any.whl size=6077 sha256=1f2ac5c65c48e37395c581162272bcc62cfa4b69370104d102b5da05a1b06f3c
      Stored in directory: /root/.cache/pip/wheels/9f/5a/d3/6fc3bf6516d2a3eb7e18f9f28b472110b59325f3f258fe9211
    Successfully built qiskit-aqua dlx docplex docloud inflection
    Installing collected packages: sympy, ply, marshmallow, marshmallow-polyfield, qiskit-terra, qiskit-ignis, dlx, docloud, docplex, inflection, quandl, pyscf, qiskit-aqua
      Found existing installation: sympy 1.1.1
        Uninstalling sympy-1.1.1:
          Successfully uninstalled sympy-1.1.1
    Successfully installed dlx-1.0.4 docloud-1.0.375 docplex-2.12.182 inflection-0.3.1 marshmallow-3.5.0 marshmallow-polyfield-5.8 ply-3.11 pyscf-1.7.0 qiskit-aqua-0.7.0 qiskit-ignis-0.2.0 qiskit-terra-0.12.0 quandl-3.5.0 sympy-1.5.1

```python
!pip install qiskit-aer qiskit-ignis qiskit-terra qiskit-ibmq-provider -q
```

         |████████████████████████████████| 23.1MB 132kB/s
         |████████████████████████████████| 143kB 58.9MB/s
         |████████████████████████████████| 153kB 59.8MB/s
         |████████████████████████████████| 71kB 12.0MB/s
         |████████████████████████████████| 51kB 9.2MB/s
         |████████████████████████████████| 2.3MB 73.5MB/s

```python
import numpy as np
from qiskit import *
from qiskit import Aer
from qiskit.aqua.components.optimizers import *
from qiskit.quantum_info import state_fidelity
from qiskit.quantum_info.analysis import average_data,hellinger_fidelity
from qiskit.providers.aer.noise import NoiseModel
from qiskit.aqua.utils import get_subsystem_density_matrix
from qiskit.tools.visualization import plot_histogram
import matplotlib.pyplot as plt

np.set_printoptions(precision=15)
%matplotlib inline
```

```python
plt.style.use("ggplot")

import qiskit.tools.jupyter
%qiskit_version_table
%qiskit_copyright

```

Version Information

Qiskit Software	Version

Qiskit	None

Terra	0.12.0

Aer	0.4.0

Ignis	0.2.0

Aqua	0.7.0

IBM Q Provider	0.5.0

System information

Python	3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0]

OS	Linux

CPUs	1

Memory (Gb)	12.717426300048828

Sat Feb 29 02:04:20 2020 UTC

This code is a part of Qiskit

© Copyright IBM 2017, 2020.

This code is licensed under the Apache License, Version 2.0. You may
obtain a copy of this license in the LICENSE.txt file in the root directory
 of this source tree or at http://www.apache.org/licenses/LICENSE-2.0.
Any modifications or derivative works of this code must retain this
copyright notice, and modified files need to carry a notice indicating
that they have been altered from the originals.

### Quantum Device selection
I've chosen IBM Vigo quantum computer to be the main backend. To run the following cell please get your tokken from IBMQ dashboard.

```python
IBMQ.save_account('', overwrite=True)
provider = IBMQ.load_account()
#provider.backends()
device_backend = provider.get_backend('ibmq_vigo')
noise_model = NoiseModel.from_backend(device_backend)
coupling_map = device_backend.configuration().coupling_map
basis_gates = noise_model.basis_gates
print(noise_model)
# Select the QasmSimulator from the Aer provider
simulator = Aer.get_backend('qasm_simulator')
```

    NoiseModel:
      Basis gates: ['cx', 'id', 'u2', 'u3']
      Instructions with noise: ['id', 'u2', 'cx', 'u3', 'measure']
      Qubits with noise: [0, 1, 2, 3, 4]
      Specific qubit errors: [('id', [0]), ('id', [1]), ('id', [2]), ('id', [3]), ('id', [4]), ('u2', [0]), ('u2', [1]), ('u2', [2]), ('u2', [3]), ('u2', [4]), ('u3', [0]), ('u3', [1]), ('u3', [2]), ('u3', [3]), ('u3', [4]), ('cx', [0, 1]), ('cx', [1, 0]), ('cx', [1, 2]), ('cx', [1, 3]), ('cx', [2, 1]), ('cx', [3, 1]), ('cx', [3, 4]), ('cx', [4, 3]), ('measure', [0]), ('measure', [1]), ('measure', [2]), ('measure', [3]), ('measure', [4])]

### Circuit Preparation

As I've shown in the [Pennylane notebook](https://github.com/kareem1925/Ismailia-school-of-AI/blob/master/machine%20learning%20for%20classifying%20quantum%20states/pennylane-quantum-state-learning.ipynb), I've initalized the same circuit in Qiskit. I only included a noisy backend with its properties and a snapshot to get the statevector. This means that the number of shots won't affect the the statevector unlike the counts measurements which are affected by it. We use one the utils in qiskit to get the density matrix of the first and second qubits only.

```python

def circuit(params,shots):

    q = QuantumRegister(2)
    c = ClassicalRegister(2)
    qc = QuantumCircuit(q,c)
    #qc.h(q[0])

    qc.ry(params[0],q[0])
#    qc.rx(params[1],q[0])
#    qc.ry(params[2],q[0])
#    qc.rx(params[1],q[0])

    qc.cx(q[0],q[1])
    qc.snapshot('0')
    qc.measure(q[0],c[0])
    qc.measure(q[1],c[1])
    job = execute(qc, simulator,
                       noise_model=noise_model,
                       coupling_map=coupling_map,
                       basis_gates=basis_gates,shots=shots,backend_options={"max_parallel_threads":10},optimization_level=3).result()

    statevector = np.array(job.data()['snapshots']['statevector']['0'][0])
    statevector = get_subsystem_density_matrix(statevector,[2,3,4])
    counts = job.get_counts()
    return counts, statevector

```

### Define cost function
we can use the cost function through $3$ ways:

1.   Fiddelity measurement.
2.   Count measurement difference.
3.   average_data function which mimics the Hermitian measurement in Pennylane.

```python
# for count difference in the cost function
labels = np.array([0.5,0,0,0.5])

labels_vector = np.array([1/np.sqrt(2),0,0,1/np.sqrt(2)])

#metric = 'vectorstate'
metric = 'average_data'
def cost_vec(var):
    st_vec = circuit(var,1000)[1]
    return abs(1 - state_fidelity(st_vec,labels_vector))

def cost_count100(var):
    st_vec = circuit(var,100)[0]
    return  -average_data(st_vec,{"00":1, "01":0, "10":0, "11":1})

def cost_count10(var):
    st_vec = circuit(var,10)[0]
    return  -average_data(st_vec,{"00":1, "01":0, "10":0, "11":1})

def cost_count1000(var):
    st_vec = circuit(var,1000)[0]
    return  -average_data(st_vec,{"00":1, "01":0, "10":0, "11":1})

def cost_count1(var):
    st_vec = circuit(var,1)[0]
    return  -average_data(st_vec,{"00":1, "01":0, "10":0, "11":1})

 #   else:
  #    co = np.zeros(4)

   #   if '00' in st_vec[0].keys():
     #   co[0] = st_vec[0]['00']
    #  if '11' in st_vec[0].keys():
       # co[3] = st_vec[0]['11']
      #if '10' in st_vec[0].keys():
        #co[2] = st_vec[0]['10']
      #if '01' in st_vec[0].keys():
       # co[1] = st_vec[0]['01']

      #return (abs(labels - co/shots)).mean()
```

```python
# weight initialization
np.random.seed(20)
init_params = np.random.rand(1)
#init_params = [1.57079633]
print(init_params)
print(cost_vec(init_params))
```

    [0.588130801077274]
    0.2225965707704386

### Training
the data parameter is a list which contains:

1.   The final best weights.
2.   The value of the cost function.
3.   The number of last iteration.
4.   The history of the cost funciton to be plotted. "*I added this in the main code*"

Testing state vector logic

```python
opt = AQGD(disp=True,maxiter=500,tol=1e-17)
data = opt.optimize(objective_function=cost_vec,num_vars=1,initial_point=init_params)
```

    Iteration: 0 	| Energy: 0.2225965707704386
    Iteration: 1 	| Energy: 0.0005448655801651991
    Iteration: 2 	| Energy: 0.12010279019547865
    Iteration: 3 	| Energy: 0.07172049371268607
    Iteration: 4 	| Energy: 0.0065869580967996555
    Iteration: 5 	| Energy: 0.06339344262534208
    Iteration: 6 	| Energy: 0.0121265368623249
    Iteration: 7 	| Energy: 0.014550354842057134
    Iteration: 8 	| Energy: 0.02527386767148765
    Iteration: 9 	| Energy: 0.00014716657275293255
    Iteration: 10 	| Energy: 0.012526687982597151
    Iteration: 11 	| Energy: 0.006426679017094816
    Iteration: 12 	| Energy: 0.0011267351762527245
    Iteration: 13 	| Energy: 0.006579144294567962
    Iteration: 14 	| Energy: 0.000674385191892668
    Iteration: 15 	| Energy: 0.00199277119621466
    Iteration: 16 	| Energy: 0.002250306963876514
    Iteration: 17 	| Energy: 1.4076367247373156e-05
    Iteration: 18 	| Energy: 0.0014384769828521549
    Iteration: 19 	| Energy: 0.00043830507620201686
    Iteration: 20 	| Energy: 0.0002358985695321003
    Iteration: 21 	| Energy: 0.0006402272780533513
    Iteration: 22 	| Energy: 1.8561679406170484e-05
    Iteration: 23 	| Energy: 0.0001308556518718751
    Iteration: 24 	| Energy: 0.00010778295272262373
    Iteration: 25 	| Energy: 1.0
    Iteration: 26 	| Energy: 8.267634017844472e-05
    Iteration: 27 	| Energy: 1.694188336776037e-05
    Iteration: 28 	| Energy: 1.8036830706336104e-05
    Iteration: 29 	| Energy: 3.296417255782558e-05
    Iteration: 30 	| Energy: 1.6266698865674556e-07
    Iteration: 31 	| Energy: 1.6435078886045673e-05
    Iteration: 32 	| Energy: 8.044629692172123e-06
    Iteration: 33 	| Energy: 1.607348638188455e-06
    Iteration: 34 	| Energy: 8.524137231069595e-06
    Iteration: 35 	| Energy: 7.637349563882623e-07
    Iteration: 36 	| Energy: 2.701130442095412e-06
    Iteration: 37 	| Energy: 2.8312683642006675e-06
    Iteration: 38 	| Energy: 3.275443805073053e-08
    Iteration: 39 	| Energy: 1.8908775406556444e-06
    Iteration: 40 	| Energy: 5.237390083845384e-07
    Iteration: 41 	| Energy: 5.529677771232144e-08
    Iteration: 42 	| Energy: 1.566601879332552e-07
    Iteration: 43 	| Energy: 0.9999998204430948
    Iteration: 44 	| Energy: 1.0248889514841153e-09
    Iteration: 45 	| Energy: 1.1411887124168629e-07
    Iteration: 46 	| Energy: 3.501535328354066e-08
    Iteration: 47 	| Energy: 1.8607333007736315e-08
    Iteration: 48 	| Energy: 5.089462051799387e-08
    Iteration: 49 	| Energy: 1.4971066608637784e-09
    Iteration: 50 	| Energy: 2.102963558847648e-08
    Iteration: 51 	| Energy: 1.4317160568211307e-08
    Iteration: 52 	| Energy: 1.1545062683637752e-09
    Iteration: 53 	| Energy: 1.2315898967329986e-08
    Iteration: 54 	| Energy: 1.925201553731881e-09
    Iteration: 55 	| Energy: 3.1147078249915694e-09
    Iteration: 56 	| Energy: 4.5953238991813805e-09
    Iteration: 57 	| Energy: 2.611244553918368e-13
    Iteration: 58 	| Energy: 2.5525070945775497e-09
    Iteration: 59 	| Energy: 1.021415729773878e-09
    Iteration: 60 	| Energy: 3.2101943325812954e-10
    Iteration: 61 	| Energy: 1.236776125956851e-09
    Iteration: 62 	| Energy: 7.296829807046379e-11
    Iteration: 63 	| Energy: 4.425571020760799e-10
    Iteration: 64 	| Energy: 0.9999999996176119
    Iteration: 65 	| Energy: 1.2645662295085458e-11
    Iteration: 66 	| Energy: 2.8522473272118987e-10
    Iteration: 67 	| Energy: 6.222578008419077e-11
    Iteration: 68 	| Energy: 0.0770395097902341
    Iteration: 69 	| Energy: 0.03649318559797987
    Iteration: 70 	| Energy: 0.007295816063807914
    Iteration: 71 	| Energy: 0.0387245331461159
    Iteration: 72 	| Energy: 0.004276572075627039
    Iteration: 73 	| Energy: 0.011500555945579749
    Iteration: 74 	| Energy: 0.01366479802467735
    Iteration: 75 	| Energy: 3.7880180353777604e-05
    Iteration: 76 	| Energy: 0.008389027848085151
    Iteration: 77 	| Energy: 0.002836796510318873
    Iteration: 78 	| Energy: 0.0012506899832827045
    Iteration: 79 	| Energy: 0.0038522085031151843
    Iteration: 80 	| Energy: 0.0001554048980267897
    Iteration: 81 	| Energy: 0.0015036430977772852
    Iteration: 82 	| Energy: 0.0011309004660434718
    Iteration: 83 	| Energy: 6.463031768566552e-05
    Iteration: 84 	| Energy: 0.0009148686304869535
    Iteration: 85 	| Energy: 0.00016635824979549518
    Iteration: 86 	| Energy: 0.00021388934962063288
    Iteration: 87 	| Energy: 0.00035403896500874055
    Iteration: 88 	| Energy: 6.344668641578011e-07
    Iteration: 89 	| Energy: 0.0001853551611270543
    Iteration: 90 	| Energy: 8.296487431591615e-05
    Iteration: 91 	| Energy: 2.040987325313548e-05
    Iteration: 92 	| Energy: 9.321869476353051e-05
    Iteration: 93 	| Energy: 7.0056662258011215e-06
    Iteration: 94 	| Energy: 6.808983852746309e-06
    Iteration: 95 	| Energy: 1.3075476988810841e-05
    Iteration: 96 	| Energy: 9.180157545252143e-08
    Iteration: 97 	| Energy: 5.532882245917747e-06
    Iteration: 98 	| Energy: 2.8811025492014863e-06
    Iteration: 99 	| Energy: 4.946170729658306e-07
    Iteration: 100 	| Energy: 2.9329722003312853e-06
    Iteration: 101 	| Energy: 2.9474773000082166e-07
    Iteration: 102 	| Energy: 8.932665991334687e-07
    Iteration: 103 	| Energy: 9.957755110479027e-07
    Iteration: 104 	| Energy: 7.253362466208557e-09
    Iteration: 105 	| Energy: 6.42632059677517e-07
    Iteration: 106 	| Energy: 1.9110232130081783e-07
    Iteration: 107 	| Energy: 1.0759198876275633e-07
    Iteration: 108 	| Energy: 2.839525818920663e-07
    Iteration: 109 	| Energy: 7.575265525971986e-09
    Iteration: 110 	| Energy: 1.1920206566262692e-07
    Iteration: 111 	| Energy: 7.899604859140652e-08
    Iteration: 112 	| Energy: 6.935292784504554e-09
    Iteration: 113 	| Energy: 1.2711647190410957e-07
    Iteration: 114 	| Energy: 2.572013002577478e-08
    Iteration: 115 	| Energy: 2.7944513369959623e-08
    Iteration: 116 	| Energy: 5.0517082739176544e-08
    Iteration: 117 	| Energy: 2.2802382204645255e-10
    Iteration: 118 	| Energy: 2.532307674663059e-08
    Iteration: 119 	| Energy: 1.2272874272412082e-08
    Iteration: 120 	| Energy: 2.5109940793299756e-09
    Iteration: 121 	| Energy: 1.3088707673247768e-08
    Iteration: 122 	| Energy: 1.1506561259366777e-09
    Iteration: 123 	| Energy: 4.1736243350243285e-09
    Iteration: 124 	| Energy: 4.332040060006648e-09
    Iteration: 125 	| Energy: 5.352851495388222e-11
    Iteration: 126 	| Energy: 2.9091312692131055e-09
    Iteration: 127 	| Energy: 7.965379467123057e-10
    Iteration: 128 	| Energy: 5.204281450232884e-10
    Iteration: 129 	| Energy: 1.2549530303829215e-09
    Iteration: 130 	| Energy: 2.531264087224372e-11
    Iteration: 131 	| Energy: 5.487080789734478e-10
    Iteration: 132 	| Energy: 3.390638880773622e-10
    Iteration: 133 	| Energy: 3.672173676250168e-11
    Iteration: 134 	| Energy: 3.0967761688316386e-10
    Iteration: 135 	| Energy: 4.165012779111521e-11
    Iteration: 136 	| Energy: 8.399192452657189e-11
    Iteration: 137 	| Energy: 1.1168677094275381e-10
    Iteration: 138 	| Energy: 7.227551890309769e-14
    Iteration: 139 	| Energy: 6.551137410326646e-11
    Iteration: 140 	| Energy: 2.3594237674728902e-11
    Iteration: 141 	| Energy: 9.208855900055823e-12
    Iteration: 142 	| Energy: 3.068767462366395e-11
    Iteration: 143 	| Energy: 1.4075407506197735e-12
    Iteration: 144 	| Energy: 1.1650458375811468e-11
    Iteration: 145 	| Energy: 9.139133894109364e-12
    Iteration: 146 	| Energy: 4.496403249731884e-13
    Iteration: 147 	| Energy: 7.2168937492733676e-12
    Iteration: 148 	| Energy: 1.383337888682945e-12
    Iteration: 149 	| Energy: 1.638023050531956e-12
    Iteration: 150 	| Energy: 2.829070311349824e-12
    Iteration: 151 	| Energy: 8.43769498715119e-15
    Iteration: 152 	| Energy: 1.4506174039752295e-12
    Iteration: 153 	| Energy: 6.746825320647076e-13
    Iteration: 154 	| Energy: 0.07703778877181544
    Iteration: 155 	| Energy: 0.03648976974975704
    Iteration: 156 	| Energy: 0.007296500368594305
    Iteration: 157 	| Energy: 0.03872289217548852
    Iteration: 158 	| Energy: 0.004275783345336404
    Iteration: 159 	| Energy: 0.011500693695351893
    Iteration: 160 	| Energy: 0.01366384320283709
    Iteration: 161 	| Energy: 3.791934712582545e-05
    Iteration: 162 	| Energy: 0.008388827965813084
    Iteration: 163 	| Energy: 0.002836467684904198
    Iteration: 164 	| Energy: 0.0012507698601100792
    Iteration: 165 	| Energy: 0.0038520091504193887
    Iteration: 166 	| Energy: 0.0001553583531930247
    Iteration: 167 	| Energy: 0.00150364005935133
    Iteration: 168 	| Energy: 0.0011308046808378158
    Iteration: 169 	| Energy: 6.464422335317188e-05
    Iteration: 170 	| Energy: 0.0009148366992127732
    Iteration: 171 	| Energy: 0.0001663329786621981
    Iteration: 172 	| Energy: 0.0002138956861461594
    Iteration: 173 	| Energy: 0.0003540164160206194
    Iteration: 174 	| Energy: 6.336108134830098e-07
    Iteration: 175 	| Energy: 0.00018535207031833423
    Iteration: 176 	| Energy: 8.295623721721324e-05
    Iteration: 177 	| Energy: 2.041178241030117e-05
    Iteration: 178 	| Energy: 9.321437831055768e-05
    Iteration: 179 	| Energy: 7.004087573481144e-06
    Iteration: 180 	| Energy: 3.1215237451087674e-05
    Iteration: 181 	| Energy: 2.9996751826799617e-05
    Iteration: 182 	| Energy: 5.884356633067966e-07
    Iteration: 183 	| Energy: 2.1041820118528065e-05
    Iteration: 184 	| Energy: 5.251975590203628e-06
    Iteration: 185 	| Energy: 4.032178029023115e-06
    Iteration: 186 	| Energy: 8.843551686732454e-06
    Iteration: 187 	| Energy: 1.2434933160943018e-07
    Iteration: 188 	| Energy: 4.0399597863727266e-06
    Iteration: 189 	| Energy: 1.430281544279044e-05
    Iteration: 190 	| Energy: 7.3318580473547e-07
    Iteration: 191 	| Energy: 5.295842471664258e-06
    Iteration: 192 	| Energy: 4.328478604320729e-06
    Iteration: 193 	| Energy: 1.8116419975289233e-07
    Iteration: 194 	| Energy: 3.335722547181952e-06
    Iteration: 195 	| Energy: 0.07605423753232754
    Iteration: 196 	| Energy: 0.036450698378246216
    Iteration: 197 	| Energy: 0.007074609290864897
    Iteration: 198 	| Energy: 0.03836852589288764
    Iteration: 199 	| Energy: 0.004318972223194217
    Iteration: 200 	| Energy: 0.011309604118400651
    Iteration: 201 	| Energy: 0.05457142130237114
    Iteration: 202 	| Energy: 0.0059443592560117775
    Iteration: 203 	| Energy: 0.016360215691040292
    Iteration: 204 	| Energy: 0.019367657260783933
    Iteration: 205 	| Energy: 4.757275558664453e-05
    Iteration: 206 	| Energy: 0.01183981457412464
    Iteration: 207 	| Energy: 0.0040825512937257
    Iteration: 208 	| Energy: 0.0017306178885501122
    Iteration: 209 	| Energy: 0.9945316632126143
    Iteration: 210 	| Energy: 0.00023572999758791902
    Iteration: 211 	| Energy: 0.0021065038818179893
    Iteration: 212 	| Energy: 0.00162189337432328
    Iteration: 213 	| Energy: 8.479063896116301e-05
    Iteration: 214 	| Energy: 0.0012931755476346396
    Iteration: 215 	| Energy: 0.0002436202182505287
    Iteration: 216 	| Energy: 0.0002964274528969302
    Iteration: 217 	| Energy: 0.0005048948485630866
    Iteration: 218 	| Energy: 1.299413225908097e-06
    Iteration: 219 	| Energy: 0.0002605211140245256
    Iteration: 220 	| Energy: 0.00011981992349596737
    Iteration: 221 	| Energy: 2.7704305141873498e-05
    Iteration: 222 	| Energy: 0.00013223589132205227
    Iteration: 223 	| Energy: 1.05026270373898e-05
    Iteration: 224 	| Energy: 4.354902142933437e-05
    Iteration: 225 	| Energy: 4.297149525356314e-05
    Iteration: 226 	| Energy: 7.261271566250116e-07
    Iteration: 227 	| Energy: 2.9692027566086132e-05
    Iteration: 228 	| Energy: 7.654550555624695e-06
    Iteration: 229 	| Energy: 5.558280849138342e-06
    Iteration: 230 	| Energy: 1.259200260117499e-05
    Iteration: 231 	| Energy: 2.021957014042286e-07
    Iteration: 232 	| Energy: 5.666092529388322e-06
    Iteration: 233 	| Energy: 3.3305485613865926e-06
    Iteration: 234 	| Energy: 4.1557516772261494e-07
    Iteration: 235 	| Energy: 3.138713087880163e-06
    Iteration: 236 	| Energy: 3.8911661337515113e-07
    Iteration: 237 	| Energy: 8.814605187978941e-07
    Iteration: 238 	| Energy: 1.1122506371474827e-06
    Iteration: 239 	| Energy: 2.0249534271599146e-09
    Iteration: 240 	| Energy: 6.709238848934262e-07
    Iteration: 241 	| Energy: 1.9590875981467804e-06
    Iteration: 242 	| Energy: 6.784670980586327e-08
    Iteration: 243 	| Energy: 7.866972144832118e-07
    Iteration: 244 	| Energy: 5.620587022470147e-07
    Iteration: 245 	| Energy: 3.8672515190185663e-08
    Iteration: 246 	| Energy: 4.6948207121921826e-07
    Iteration: 247 	| Energy: 7.882200336872813e-08
    Iteration: 248 	| Energy: 5.095746047345528e-10
    Iteration: 249 	| Energy: 5.047796747259525e-08
    Iteration: 250 	| Energy: 1.524986203982337e-08
    Iteration: 251 	| Energy: 8.339947177304907e-09
    Iteration: 252 	| Energy: 2.2408538580265258e-08
    Iteration: 253 	| Energy: 6.28343599373693e-10
    Iteration: 254 	| Energy: 9.332405404016697e-09
    Iteration: 255 	| Energy: 6.269127661440166e-09
    Iteration: 256 	| Energy: 5.274898295937192e-10
    Iteration: 257 	| Energy: 5.437267747154806e-09
    Iteration: 258 	| Energy: 8.329452683142335e-10
    Iteration: 259 	| Energy: 1.3887087035868717e-09
    Iteration: 260 	| Energy: 2.019285183507691e-09
    Iteration: 261 	| Energy: 1.887379141862766e-14
    Iteration: 262 	| Energy: 1.1301155566911802e-09
    Iteration: 263 	| Energy: 4.457558766546299e-10
    Iteration: 264 	| Energy: 1.4441603468640096e-10
    Iteration: 265 	| Energy: 5.450130347028903e-10
    Iteration: 266 	| Energy: 3.111400026512001e-11
    Iteration: 267 	| Energy: 1.9664203598779295e-10
    Iteration: 268 	| Energy: 1.6764534205293558e-10
    Iteration: 269 	| Energy: 5.879963183019754e-12
    Iteration: 270 	| Energy: 1.2603118548781822e-10
    Iteration: 271 	| Energy: 2.701794343806796e-11
    Iteration: 272 	| Energy: 2.6740498704214133e-11
    Iteration: 273 	| Energy: 5.0841775234289344e-11
    Iteration: 274 	| Energy: 3.3439917501709715e-13
    Iteration: 275 	| Energy: 2.4864554859505006e-11
    Iteration: 276 	| Energy: 1.2603473820149702e-11
    Iteration: 277 	| Energy: 2.3135937610163637e-12
    Iteration: 278 	| Energy: 1.3055445613474603e-11
    Iteration: 279 	| Energy: 1.2490009027033011e-12
    Iteration: 280 	| Energy: 4.0460967909439205e-12
    Iteration: 281 	| Energy: 4.390487973182644e-12
    Iteration: 282 	| Energy: 3.985700658404312e-14
    Iteration: 283 	| Energy: 2.8761437675939305e-12
    Iteration: 284 	| Energy: 8.294476216974545e-13
    Iteration: 285 	| Energy: 4.940492459581947e-13
    Iteration: 286 	| Energy: 2.3028245976774997e-12
    Iteration: 287 	| Energy: 1.7763568394002505e-13
    Iteration: 288 	| Energy: 7.653877531765829e-13
    Iteration: 289 	| Energy: 7.4451556031363e-13
    Iteration: 290 	| Energy: 1.3766765505351941e-14
    Iteration: 291 	| Energy: 5.186961971048731e-13
    Iteration: 292 	| Energy: 1.3145040611561853e-13
    Iteration: 293 	| Energy: 9.85878045867139e-14
    Iteration: 294 	| Energy: 2.191580250610059e-13
    Iteration: 295 	| Energy: 3.4416913763379853e-15
    Iteration: 296 	| Energy: 9.9253938401489e-14
    Iteration: 297 	| Energy: 5.773159728050814e-14
    Iteration: 298 	| Energy: 7.549516567451064e-15
    Iteration: 299 	| Energy: 5.484501741648273e-14
    Iteration: 300 	| Energy: 0.9999999999999934
    Iteration: 301 	| Energy: 1.554312234475219e-14
    Iteration: 302 	| Energy: 1.9539925233402755e-14
    Iteration: 303 	| Energy: 2.220446049250313e-16
    Iteration: 304 	| Energy: 1.199040866595169e-14
    Iteration: 305 	| Energy: 4.107825191113079e-15
    Iteration: 306 	| Energy: 1.887379141862766e-15
    Iteration: 307 	| Energy: 5.551115123125783e-15
    Iteration: 308 	| Energy: 3.3306690738754696e-16
    Iteration: 309 	| Energy: 2.3314683517128287e-15
    Iteration: 310 	| Energy: 1.7763568394002505e-15
    Iteration: 311 	| Energy: 2.220446049250313e-16
    Iteration: 312 	| Energy: 1.5543122344752192e-15
    Iteration: 313 	| Energy: 3.3306690738754696e-16
    Iteration: 314 	| Energy: 4.440892098500626e-16
    Iteration: 315 	| Energy: 6.661338147750939e-16
    Iteration: 316 	| Energy: 2.220446049250313e-16
    Iteration: 317 	| Energy: 4.440892098500626e-16
    Iteration: 318 	| Energy: 2.220446049250313e-16
    Iteration: 319 	| Energy: 2.220446049250313e-16
    Iteration: 320 	| Energy: 3.3306690738754696e-16
    Iteration: 321 	| Energy: 2.220446049250313e-16
    Iteration: 322 	| Energy: 2.220446049250313e-16
    Iteration: 323 	| Energy: 2.220446049250313e-16

```python
data[:3]
```

    (array([1.570796314249988]), 2.220446049250313e-16, 323)

```python
counts,density_matrix = circuit(data[0],1000)
print('counts', counts)
plot_histogram(counts)
```

```python
print("state fidelity: ", state_fidelity(density_matrix,labels_vector))
```

    state fidelity:  0.9999999999999998

### Testing Count measurement

**Quantum Gradients**

We can also compute the gradients ourselves and implement any gradient optimization technique.
[Quantum gradients](https://arxiv.org/abs/1811.11184) can be computed by the following equation:

\begin{align}
  \frac{df}{d\theta }\,=\,\frac{1}{2}\,\left( f\left( \theta +\frac{\pi }{2} \right)-f\left( \theta -\frac{\pi }{2} \right) \right)\\ \\
\end{align}

where $\theta$ is the weights and $f$ is quantum unitary gate.

```python
def gradient(params,cost):
    g = np.zeros(len(params))

    for id,i in enumerate(params):
      params[id] = params[id] + np.pi/2
      f1 = cost(params)
      params[id] = params[id] - np.pi
      f2 = cost(params)
      params[id] = params[id] + np.pi/2
      g[id] = (f1-f2)/2

    return g
```

```python
def optimize(params,lr,cost):
    """

    this fucntion controls SGD by computing the gradients and updating the weights

    """
    g = gradient(params,cost)

    new_weights = params - (lr*g)

    return new_weights
```

**10 shots**

```python

epochs = 400
params = init_params
loss = np.inf
hist = []

for it in range(epochs):
  params = optimize(params,lr=0.1,cost=cost_count10)
  temp = cost_count10(params)
  if temp < loss:
    best = params
    loss = temp
  hist.append(temp)
  if it % 10 == 0:
    print(temp)
```

    -1.0
    -1.0
    -0.8999999999999999
    -1.0
    -0.8
    -1.0
    -0.8999999999999999
    -1.0
    -1.0
    -1.0
    -1.0
    -0.8999999999999999
    -1.0
    -1.0
    -1.0
    -0.8999999999999999
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -0.8999999999999999
    -1.0
    -0.9
    -0.8999999999999999
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -0.9
    -0.9

```python
plt.plot(hist,"*", label="10 shots loss")
plt.tight_layout()
plt.legend()
```

```python
counts,density_matrix = circuit(best,10)
print('counts', counts)
plot_histogram(counts)
```

```python
print("state fidelity: ", state_fidelity(density_matrix,labels_vector))
```

    state fidelity:  0.9145973769270366

**100 shots**

```python

epochs = 400
params = init_params
loss = np.inf
hist = []

for it in range(epochs):
  params = optimize(params,lr=0.1,cost=cost_count100)
  temp = cost_count100(params)
  if temp < loss:
    best = params
    loss = temp
  hist.append(temp)
  if it % 10 == 0:
    print(temp)
```

    -0.95
    -0.97
    -0.96
    -0.97
    -0.97
    -0.94
    -0.94
    -0.95
    -0.9500000000000001
    -0.9199999999999999
    -0.94
    -0.97
    -0.96
    -0.97
    -0.91
    -0.97
    -0.94
    -0.94
    -0.96
    -0.99
    -0.95
    -0.98
    -0.95
    -0.92
    -0.99
    -0.9600000000000001
    -0.98
    -0.95
    -0.96
    -0.96
    -0.95
    -0.94
    -0.96
    -0.94
    -0.96
    -0.97
    -0.97
    -0.96
    -0.96
    -0.9299999999999999

```python
plt.plot(hist,"*", label="100 shots loss")
plt.tight_layout()
plt.legend()
```

```python
counts,density_matrix = circuit(best,100)
print('counts', counts)
plot_histogram(counts)
```

```python
print("state fidelity: ", state_fidelity(density_matrix,labels_vector))
```

    state fidelity:  0.9999946389299257

**1000 shots**

```python

epochs = 400
params = init_params
loss = np.inf
hist = []

for it in range(epochs):
  params = optimize(params,lr=0.1,cost=cost_count1000)
  temp = cost_count1000(params)
  if temp < loss:
    best = params
    loss = temp
  hist.append(temp)
  if it % 10 == 0:
    print(temp)
```

    -0.942
    -0.955
    -0.9359999999999999
    -0.945
    -0.949
    -0.944
    -0.9500000000000001
    -0.9420000000000001
    -0.9580000000000001
    -0.958
    -0.9570000000000001
    -0.96
    -0.9630000000000001
    -0.9450000000000001
    -0.95
    -0.955
    -0.952
    -0.95
    -0.958
    -0.959
    -0.951
    -0.953
    -0.9530000000000001
    -0.958
    -0.946
    -0.956
    -0.96
    -0.956
    -0.958
    -0.9550000000000001
    -0.9550000000000001
    -0.952
    -0.952
    -0.965
    -0.96
    -0.9530000000000001
    -0.9570000000000001
    -0.968
    -0.9690000000000001
    -0.9570000000000001

```python
plt.plot(hist,"*", label="1000 shots loss")
plt.tight_layout()
plt.legend()
```

```python
counts,density_matrix = circuit(best,1000)
print('counts', counts)
plot_histogram(counts)
```

```python
print("state fidelity: ", state_fidelity(density_matrix,labels_vector))
```

    state fidelity:  0.9837140431186004

```python
circuit([-np.pi/2],10000)
```

    ({'00': 4896, '01': 173, '10': 214, '11': 4717},
     array([[ 0.5+0.000000000000000e+00j,  0. +0.000000000000000e+00j,
              0. +0.000000000000000e+00j, -0.5-6.123233995736766e-17j],
            [ 0. +0.000000000000000e+00j,  0. +0.000000000000000e+00j,
              0. +0.000000000000000e+00j,  0. +0.000000000000000e+00j],
            [ 0. +0.000000000000000e+00j,  0. +0.000000000000000e+00j,
              0. +0.000000000000000e+00j,  0. +0.000000000000000e+00j],
            [-0.5+6.123233995736766e-17j,  0. +0.000000000000000e+00j,
              0. +0.000000000000000e+00j,  0.5+0.000000000000000e+00j]]))

**1 shot**

```python

epochs = 400
params = init_params
loss = np.inf
hist = []

for it in range(epochs):
  params = optimize(params,lr=0.1,cost=cost_count1)
  temp = cost_count1(params)
  if temp < loss:
    best = params
    loss = temp
  hist.append(temp)
  if it % 10 == 0:
    print(temp)
```

    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -0.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0
    -1.0

```python
plt.plot(hist,"*", label="1 shot loss")
plt.tight_layout()
plt.legend()
```

```python
counts,density_matrix = circuit(best,1)
print('counts', counts)
plot_histogram(counts)
```

```python
print("state fidelity: ", state_fidelity(density_matrix,labels_vector))
```

    state fidelity:  0.7774034292295615

```python

```

machine learning for classifying quantum states/Qiskit code for learning Q states.ipynb