# Introduction

Given a closed loop system and a simpler identification model of the system we want to estimate the parameters in the controller block of the closed loop system.

Here I try to show that a systematic error in the identified parameters exists and that is a function of the ratio of the external pertubation and the process noise. This bias is present in indirect identification and direct identification methods.

Assumptions:
- Both the true system and the identification model have the same states.
- We can model the plant perfectly in the identification model.

This import makes some of the helper functions in the `src` directory available.

```
import sys
sys.path.append('../src')
```

# Derivation of the Models

The following cell derives the closed-loop non-linear equations of motion for a point mass pendulum on a laterally moving base, i.e. the classic inverted pendulum on a moving cart, which controlled by a simple proportional torque controller at the pin joint and subjected to reference noise, $w$.

```
  iy
  ^       E
  |    |   o m
  |    |0 /      |
  |    | / l     v g
  |   J|/
O -----o------> ix
  |-x->|
```

Symbols:
- $g$: acceleration due to gravity
- $m$: bob mass
- $l$: pendulum length
- $k$: proportional controller gain
- $\hat{\mathbf{i}}_{xyz}$: inertial frame unit vectors
- $\hat{\mathbf{p}}_{xyz}$: pendulum frame unit vectors
- $O$: point that locates the pin joint
- $B$: point that locates the bob
- $x(t), v(t), a(t)$: base position, velocity, and acceleration
- $\theta(t), \omega(t)$: pendulum angle and angular rate
- $w(t)$: white noise on the angle reference tracking

```
import sympy as sm
import sympy.physics.mechanics as me
```

```
sm.physics.vector.init_vprinting()
```

```
m, l, g, k = sm.symbols('m, l, g, k')
theta, omega, x, v, a, w = me.dynamicsymbols('theta, omega, x, v, a, w')

inertial_frame = me.ReferenceFrame('I')
pendulum_frame = inertial_frame.orientnew('P', 'Axis', (theta,
                                                        inertial_frame.z))

pendulum_frame.set_ang_vel(inertial_frame, omega * inertial_frame.z)

origin = me.Point('O')

joint = origin.locatenew('J', x * inertial_frame.x)
joint.set_vel(inertial_frame, v * inertial_frame.x)
joint.set_acc(inertial_frame, a * inertial_frame.x)

end = joint.locatenew('E', l * pendulum_frame.y)
end.v2pt_theory(joint, inertial_frame, pendulum_frame)
end.a2pt_theory(joint, inertial_frame, pendulum_frame)

bob = me.Particle('B', end, m)

kane = me.KanesMethod(inertial_frame, (theta,), (omega,),
                      (theta.diff() - omega,))

gravitational_force = -m * g * inertial_frame.y

control_torque = k * (-theta + w) * inertial_frame.z

loads = [(end, gravitational_force), (pendulum_frame, control_torque)]

fr, frstar = kane.kanes_equations(loads, [bob])
```

```
x = kane.q.col_join(kane.u)
xdot = x.diff()
```

## Non-linear Model

The explicit first order non-linear ordinary differential equations take the following form:

```
non_linear_sym_rhs = sm.simplify(kane.rhs())
sm.Eq(xdot, non_linear_sym_rhs.expand())
```

The model above is what I'll call the true system. We will use a slightly different model to identify parameters of this true system. The identification model will simply be the above model with $w=0$ and I'll introduce new variable names for the parameters that we will identify, in this case simply the gain.

```
k_id = sm.symbols('k_id')
non_linear_sym_id_rhs = non_linear_sym_rhs.subs({w: 0, k: k_id}).expand()
sm.Eq(xdot, non_linear_sym_id_rhs)
```

Now, if $k_{id}$ is the only free parameter in the identifaction process and we use data, with no measurement noise, generated from integrating the true system, then $k_{id}$ will be adjusted such that it accounts for the motion due to the noise term in the model: $\frac{kw}{l^2m}$. By equating the two models and solving for $k_{id}$ we can how the true gain relates to the gain we identify.

```
sol = sm.solve(non_linear_sym_rhs - non_linear_sym_id_rhs, k_id)
sm.Eq(k_id, sm.factor_terms(sol[k_id]))
```

## Linear Model

The system with a linear plant gives the same result:

```
A, B, u = kane.linearize(op_point={theta: 0, omega: 0}, new_method=True, A_and_B=True)
```

```
linear_sym_rhs = sm.simplify(A * x + B * u)
sm.Eq(xdot, linear_sym_rhs.expand())
```

```
linear_sym_id_rhs = linear_sym_rhs.subs({w: 0, k: k_id}).expand()
sm.Eq(xdot, linear_sym_id_rhs)
```

```
sol = sm.solve(linear_sym_rhs - linear_sym_id_rhs, k_id)
sm.Eq(k_id, sm.factor_terms(sol[k_id]))
```

So this tells us that for $k_{id}$ to equal $k$, $\frac{w}{\theta} \rightarrow 0$. So, $w << \theta$ and that can happen if $w$ is very small relative to $\theta$. If $w$ is small it is equivalent to saying that our identification model is a good model of the process. But if it is unknown how bad our model is, $w$ has some magnitude, i.e. the modeling error. But this equation says that if we can cause $\theta$ to be significantly larger than $w$, then $k_{id} \rightarrow k$. $w$ and $\theta$ are both time varying parameters, so it is difficult to say what $\frac{w}{\theta}$ is and they could certainly be correlated. For example, if we choose a linearized model as our identification model, then surely the process noise, $w$, is correlated to $\theta$, i.e. if you exercise the linear model in the non-linear regime, i.e. large $\theta$, then the process noise term must account for this $\theta$ dependent error. But, if they aren't necessarily correlated, one way to increase $\theta$ without increasing $w$ in this model is to increase $a$. By perturbing the model with a term that isn't subjected to the feedback loop, we can also potentially have $k_id \rightarrow k$.

So, the conclusion here is that the ratio of $\frac{a}{w}$ must be large for $k_id \rightarrow k$. This is an example of systematic error due to our identification model being a poor estimate of a true process. I'm not sure how to generalize this yet, but I'm pretty sure it is true for other systems.

Ljung 1999 Chapter 13 sections 4-5 (not so clearly) state that there are bias issues when doing indirect identification, just like there are with direct indetification. The bias is due to a number of things, but one is that you don't have a good noise model, i.e. that your identification model doesn't represent the true system well. There are ways to reduce the bias, of course, but for complex systems where little knowledge is known about any of the processes (i.e. the human) it is difficult to ensure that you are reducing this bias.

Some notes from Ljung 1999 that are relevant:

- pg 432: "The closed loop experiment is informative if and only if $r(t)$ is persistently excited." $r(t)$ is the reference signal in a standard closed loop system when you want to identify the plant. So I think, for our case, this means that the platform acceleration must be persistently excited.
- The previous statement is for linear systems, so he clarifies that "nonlinear of time varying or complex (high order) regulators should in general yield experiments that are informative enough." The regulator is the plant in our case, so we have a complex plant (the human open loop system). So if you can't excite the platform enough, you may still have informative data due to the non-linearity of the plant.
- pg 433: "the bias inclination will be small in freqeuncy ranges where either (or all) of the following holds: 1) the noise model is good, 2) the feedback contribution to the input spectrum is small, 3) the signal to noise ratio is good." In terms of our problem 1) refers to the process noise, i.e. if you don't include a good noise model in your identification model, you get biased results. This is what I show above. I think 2 and 3 essentially mean, in our case, that the reference noise is small relative to the input to the controlle.
- pg 437 shows that you get the same bias as the direct approach (13.62)

# Simulate the True System

```
from collections import OrderedDict

import numpy as np
from scipy.interpolate import interp1d
from scipy.integrate import odeint
import matplotlib.pyplot as plt
from pydy.codegen.ode_function_generators import generate_ode_function

from fast_interpolate import Interpolator
```

First I'll pick some numerical values for the constant parameters.

```
par_map = OrderedDict()
par_map[m] = 1.0
par_map[l] = 1.0
par_map[g] = 9.81
par_map[k] = 15.0
par_map
```

Here I generate numerical functions that can evaluate the right hand side of the closed loop ordinary differential equations for both the non-linear and linear models.

```
non_linear_rhs = generate_ode_function(non_linear_sym_rhs,
                                       kane.q, kane.u, par_map.keys(),
                                       specifieds=u,
                                       constants_arg_type='array',
                                       specifieds_arg_type='function',
                                       generator='cython')
```

```
linear_rhs = generate_ode_function(linear_sym_rhs,
                                   kane.q, kane.u, par_map.keys(),
                                   specifieds=u,
                                   constants_arg_type='array',
                                   specifieds_arg_type='function',
                                   generator='cython')
```

Now I pick a simulation duration and the number of time points.

```
num_nodes = 1001
duration = 10.0

time = np.linspace(0.0, duration, num_nodes)

interval = duration / (num_nodes - 1)
```

Next, I specify the time histories of the inputs so that we can simulate the system. I'll use a sum of sines for both the acceleration $a(t)$ and the reference noise $w(t)$.

```
def sum_of_sines(time, amplitudes, frequencies):

    phases = 2.0 * np.pi * np.random.ranf(len(frequencies))

    sines = np.zeros_like(time)

    for a, w, p in zip(amplitudes, frequencies, phases):
        sines += a * np.sin(w * time + p)

    return sines
```

```
u
```

```
frequencies = np.logspace(np.log10(0.15), np.log10(15.0), num=12)

specifieds = np.empty((len(time), 2), dtype=float)

acceleration = sum_of_sines(time, 0.2 * np.ones_like(frequencies), frequencies)
process_noise = sum_of_sines(time, 0.005 * np.ones_like(frequencies), frequencies)

specifieds[:, 0] = acceleration
specifieds[:, 1] = process_noise
```

Because I will use a variable step integrator, I create a function that interpolates the specified inputs at any time.

```
interpolator = Interpolator(time, specifieds)

result = np.zeros(2)

def perturb(x, t):
    if t > time[-1]:
        return interpolator.interpolate(time[-1], result)
    else:
        return interpolator.interpolate(t, result)
```

Initial state is zero. I make it a small number here to avoid a divide by zero later on.

```
x0 = 0.0001 * np.ones(2)
```

Now I simulate the non-linear and linear systems under the influence of the two specified inputs.

```
x_non_lin_traj = odeint(non_linear_rhs, x0, time,
                        args=(perturb, np.array(par_map.values())))
```

```
x_lin_traj = odeint(linear_rhs, x0, time,
                    args=(perturb, np.array(par_map.values())))
```

And plot the resulting state trajectories.

```
%matplotlib inline
```

```
fig, axes = plt.subplots(4, 1, sharex=True, figsize=(8, 8))

axes[0].plot(time, acceleration)
axes[0].set_ylabel(r'a(t) [$ms^{-2}$]')

axes[1].plot(time, np.rad2deg(process_noise))
axes[1].set_ylabel(r'w(t) [$deg$]')

axes[2].plot(time, np.rad2deg(x_non_lin_traj[:, 0]), 'red', label='non-linear')
axes[2].plot(time, np.rad2deg(x_lin_traj[:, 0]), 'blue', label='linear')
axes[2].set_ylabel(r'$\theta(t)$ [deg]')

axes[3].plot(time, np.rad2deg(x_non_lin_traj[:, 1]), 'red', label='non-linear')
axes[3].plot(time, np.rad2deg(x_lin_traj[:, 1]), 'blue', label='linear')
axes[3].set_ylabel(r'$\omega(t)$ [deg/s]')
axes[3].set_xlabel('Time [s]')

leg = plt.legend()
```

## The Ratio

I'm curious what $k (1 - \frac{w}{\theta})$ looks like. It should hover around $k$ if we expect the identified gain to be anywhere close.

```
plt.plot(time, par_map[k] * (1 - process_noise / x_non_lin_traj[:, 0]))
plt.ylim([par_map[k] - 5.0, par_map[k] + 5.0])
```

The graph doesn't give a good impression but if you divided the standard devations of the process noise and $\theta$ I get a number that may tell something useful:

```
par_map[k] * (1 - process_noise.std() / x_non_lin_traj[:, 0].std())
```

# Indirect Parameter Identification with Direct Collocation

```
from opty.direct_collocation import Problem
```

Now I setup a direct collocation problem that simply minimizes the difference in the identification model's trajectory and the simulated trajectories from above. I'll start by using the non-linear identification model.

```
def obj(free):
    return interval * np.sum((x_non_lin_traj.T.flatten() - free[:-1])**2)

def obj_grad(free):
    grad = np.zeros_like(free)
    grad[:-1] = 2.0 * interval * (free[:-1] - x_non_lin_traj.T.flatten())
    return grad
```

```
known_par_map = par_map.copy()
known_par_map.pop(k)
```

## Non-linear Model

```
prob = Problem(obj,
               obj_grad,
               xdot - non_linear_sym_id_rhs,
               x,
               num_nodes,
               interval,
               known_parameter_map=known_par_map,
               known_trajectory_map={a: acceleration},
               integration_method='midpoint')
```

```
prob.addOption('linear_solver', 'ma57')
```

I'll give the best initial guess I can, just for quick and easy convergence. The "measured" states and the known gain are given.

```
initial_guess = np.hstack((x_non_lin_traj.T.flatten(), par_map[k]))
```

And the solution:

```
sol, info = prob.solve(initial_guess)
sm.Eq(k_id, sol[-1])
```

This compares the actual trajectory to the direct collocation trajectory solution.

```
fig, axes = plt.subplots(3, 1, sharex=True, figsize=(8, 8))

axes[0].plot(time, acceleration)
axes[0].set_ylabel(r'a(t) [$ms^{-2}$]')

axes[1].plot(time, np.rad2deg(x_non_lin_traj[:, 0]), '.k')
axes[1].plot(time, np.rad2deg(sol[:-1].reshape(2, num_nodes).T)[:, 0], 'blue')
axes[1].set_ylabel(r'$\theta(t)$ [deg]')

axes[2].plot(time, np.rad2deg(x_non_lin_traj[:, 1]), '.k', label='Measured')
axes[2].plot(time, np.rad2deg(sol[:-1].reshape(2, num_nodes).T)[:, 1], 'blue', label='Simulated')
axes[2].set_ylabel(r'$\omega(t)$ [deg/s]')
axes[2].set_xlabel('Time [s]')

leg = plt.legend()
```

Here I check if the constraints are satisfied:

```
constraint_violations = prob.con(sol).reshape(2, prob.num_constraints / 2).T
plt.plot(constraint_violations)
leg = plt.legend([sm.latex(s, mode='inline') for s in x])
```

## Linear Model

Now I use the linear identification model and the data from the non-linear simulation to see how well it does.

```
prob = Problem(obj,
               obj_grad,
               xdot - linear_sym_id_rhs,
               x,
               num_nodes,
               interval,
               known_parameter_map=known_par_map,
               known_trajectory_map={a: acceleration},
               integration_method='midpoint')
```

```
prob.addOption('linear_solver', 'ma57')
```

```
initial_guess = np.hstack((x_non_lin_traj.T.flatten(), par_map[k]))
```

```
sol, info = prob.solve(initial_guess)
sm.Eq(k_id, sol[-1])
```

```
fig, axes = plt.subplots(3, 1, sharex=True, figsize=(8, 8))

axes[0].plot(time, acceleration)
axes[0].set_ylabel(r'a(t) [$ms^{-2}$]')

axes[1].plot(time, np.rad2deg(x_non_lin_traj[:, 0]), '.k')
axes[1].plot(time, np.rad2deg(sol[:-1].reshape(2, num_nodes).T)[:, 0], 'blue')
axes[1].set_ylabel(r'$\theta(t)$ [deg]')

axes[2].plot(time, np.rad2deg(x_non_lin_traj[:, 1]), '.k', label='Measured')
axes[2].plot(time, np.rad2deg(sol[:-1].reshape(2, num_nodes).T)[:, 1], 'blue', label='Simulated')
axes[2].set_ylabel(r'$\omega(t)$ [deg/s]')
axes[2].set_xlabel('Time [s]')

leg = plt.legend()
```

```
constraint_violations = prob.con(sol).reshape(2, prob.num_constraints / 2).T
plt.plot(constraint_violations)
```

# Conclusion

It seems that it is impossible to identify the exact gain value if any process noise is present, i.e. the identification model differs from the true system. Cranking up the number of nodes does not reduce the error in the identified parameter. The only way to get the exact gain is to set the process noise to zero, or at least make the ratio of $\frac{a}{w}$ large. It is also impossible for the simpler identification models to track the trajectory of the true system due to the unmodeled process noise. There is a systematic error and this bias potentially exists in both direct and indirect indentification methods of closed loop systems. In the case of identifying a human controller in a motor control task, we have little to no idea of what the process noise could be, so it is difficult to design a proper perturbation level.


```python
from sympy import *
```

```python
init_printing()
```

```python
x = Symbol('x')
```

```python
a = Symbol('a')
```

```python
f0 = x ** 2 + (a ** 2 + a) * x + a -1
```

```python
f1 = x + 3
```

```python
solve([f0, f1], [x, a])
```

\begin{eqnarray*}
x ^ 2 +( a ^2 + a)x + a -1 = 0 \\
x + 3 = 0
\end{eqnarray*}

$
\begin{align*}
(-3)^ 2 +( a ^2 + a)(-3) + a -1 = 0 \\
9 - 3(a ^2 + a ) + a -1 = 0 \\
- 3 a ^2 -3a + a  + 8 = 0 \\
- 3 a ^2 -2a + 8 = 0 \\
3a ^ 2 + 2a  - 8 = 0
\end{align*}
$

\begin{equation*}
x = \frac{-b\pm{\sqrt{b^2 - 4ac}}}{2a}
\end{equation*}

$
\begin{align}
x & = \frac{-2\pm{\sqrt{(2)^2 - 4(3\times{ -8})}}}{2\times{3}}　 \\
  & = \frac{-2\pm{\sqrt{4 + 4 \times{ 24}}}}{6} \\
  & = \frac{-2\pm{2\sqrt{1+24}}}{6} \\
  & = \frac{-2\pm{10}}{6} \\
  & = \frac{-12, 8}{6} \\
  & = -2, \frac{4}{3}
\end{align}
$

```python

```


# Hands-on: movie recommender system
## Collaborative filtering (matrix factorization)

You are an online retailer/travel agent/movie review website, and you would like to help the visitors of your website to explore more of your products/destinations/movies. You got data which either describe the different products/destinations/films, or past transactions/trips/views (or preferences) of your visitors (or both!). You decide to leverage that data to provide relevant and meaningful recommendations.

This notebook implements a simple collaborative system using  factorization of the user-item matrix.

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
```

```
ratings="https://github.com/couturierc/tutorials/raw/master/recommender_system/data/ratings.csv"
movies="https://github.com/couturierc/tutorials/raw/master/recommender_system/data/movies.csv"

# If data stored locally
# ratings="./data/ratings.csv"
# movies="./data/movies.csv"

df_ratings = pd.read_csv(ratings, sep=',')
df_ratings.columns = ['userId', 'itemId', 'rating', 'timestamp']
df_movies = pd.read_csv(movies, sep=',')
df_movies.columns = ['itemId', 'title', 'genres']
```

```
df_movies.head()
```

```
df_ratings.head()
```

## Quick exploration

Hints: use df.describe(), df.column_name.hist(), scatterplot matrix (sns.pairplot(df[column_range])), correlation matrix (sns.heatmap(df.corr()) ), check duplicates, ...

```
# Start your exploration -- use as many cells as you need !

```

## Obtain the user-item matrice by pivoting df_ratings

```
##### FILL HERE (1 line) ######
df_user_item = NULL # Use df.pivot, rows ~ userId's, columns ~ itemId's
################################

# Sort index/rows (userId's) and columns (itemId's)
df_user_item.sort_index(axis=0, inplace=True)
df_user_item.sort_index(axis=1, inplace=True)
```

This matrix has **many** missing values:

```
df_user_item.head()
```

```
df_user_item.describe()
```

For instance, rating for userId=1 for movies with itemId 1 to 10:

```
df_user_item.loc[1][:10]
```

```
# df_user_item.loc[1].dropna().sort_values(ascending=False)
```

Save the movie ids for user 1 for later:

```
item_rated_user_1 = df_user_item.loc[1].dropna().index
item_rated_user_1
```

We want to find the matrix of rank $k$ which is closest to the original matrix.

## What not to do: Fill with 0's or mean values, then Singular Value Decomposition (SVD)

(Adapted from https://github.com/beckernick/matrix_factorization_recommenders/blob/master/matrix_factorization_recommender.ipynb)

Singular Value Decomposition decomposes a matrix $R$ into the best lower rank (i.e. smaller/simpler) approximation of the original matrix $R$. Mathematically, it decomposes R into a two unitary matrices and a diagonal matrix:

$$\begin{equation}
R = U\Sigma V^{T}
\end{equation}$$

where:
- R is users's ratings matrix,
- $U$ is the user "features" matrix, it represents how much users "like" each feature,
- $\Sigma$ is the diagonal matrix of singular values (essentially weights),
- $V^{T}$ is the movie "features" matrix, it represents how relevant each feature is to each movie,

with $U$ and $V^{T}$ orthogonal.

```
df_user_item = df_user_item.fillna(0)
df_user_item.head()
```

```
R = df_user_item.values
```

```
R
```

Apply SVD to R (e.g. using NumPy or SciPy)

```
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R, k = 50)
```

What do $U$, $\Sigma$, $V^T$ look like?

```
U
```

```
sigma
```

```
Vt
```

Get recommendations:

```
# First make sigma a diagonal matrix:
sigma = np.diag(sigma)
```

```
R_after_svd = np.dot(np.dot(U, sigma), Vt)
R_after_svd
```

```

```

Drawbacks of this approach:
- the missing values (here filled with 0's) is feedback that the user did not give, we should not cannot consider it negative/null rating.
- the dense matrix is huge, applying SVD is not scalable.

## Approximate SVD with stochastic gradient descend (SGD)

This time, we do **not** fill missing values.

We inject $\Sigma$ into U and V, and try to find P and q such that $\widehat{R} = P Q^{T}$ is close to  $R$ **for the item-user pairs already rated**.

A first function to simplify the entries (userId/itemId) : we map the set of

```
def encode_ids(data):
    '''Takes a rating dataframe and return:
    - a simplified rating dataframe with ids in range(nb unique id) for users and movies
    - 2 mapping disctionaries

    '''

    data_encoded = data.copy()

    users = pd.DataFrame(data_encoded.userId.unique(),columns=['userId'])  # df of all unique users
    dict_users = users.to_dict()
    inv_dict_users = {v: k for k, v in dict_users['userId'].items()}

    items = pd.DataFrame(data_encoded.itemId.unique(),columns=['itemId']) # df of all unique items
    dict_items = items.to_dict()
    inv_dict_items = {v: k for k, v in dict_items['itemId'].items()}

    data_encoded.userId = data_encoded.userId.map(inv_dict_users)
    data_encoded.itemId = data_encoded.itemId.map(inv_dict_items)

    return data_encoded, dict_users, dict_items

```

Here is the procedure we would like to implement in the function SGD():

1.   itinialize P and Q to random values

2.   for $n_{epochs}$ passes on the data:

    *   for all known ratings $r_{ui}$
        *   compute the error between the predicted rating $p_u \cdot q_i$ and the known ratings $r_{ui}$:
        $$ err = r_{ui} - p_u \cdot q_i $$
        *   update $p_u$ and $q_i$ with the following rule:
        $$ p_u \leftarrow p_u + \alpha \cdot err \cdot q_i  $$
        $$ q_i \leftarrow q_i + \alpha \cdot err \cdot p_u$$

```
# Adapted from http://nicolas-hug.com/blog/matrix_facto_4
def SGD(data,           # dataframe containing 1 user|item|rating per row
        n_factors = 10, # number of factors
        alpha = .01,    # number of factors
        n_epochs = 3,   # number of iteration of the SGD procedure
       ):
    '''Learn the vectors P and Q (ie all the weights p_u and q_i) with SGD.
    '''

    # Encoding userId's and itemId's in data
    data, dict_users, dict_items = encode_ids(data)

    ##### FILL HERE (2 lines) ######
    n_users = NULL  # number of unique users
    n_items = NULL  # number of unique items
    ################################

    # Randomly initialize the user and item factors.
    p = np.random.normal(0, .1, (n_users, n_factors))
    q = np.random.normal(0, .1, (n_items, n_factors))

    # Optimization procedure
    for epoch in range(n_epochs):
        print ('epoch: ', epoch)
        # Loop over the rows in data
        for index in range(data.shape[0]):
            row = data.iloc[[index]]
            u = int(row.userId)      # current userId = position in the p vector (thanks to the encoding)
            i = int(row.itemId)      # current itemId = position in the q vector
            r_ui = float(row.rating) # rating associated to the couple (user u , item i)

            ##### FILL HERE (1 line) ######
            err = NULL    # difference between the predicted rating (p_u . q_i) and the known ratings r_ui
            ################################

            # Update vectors p_u and q_i
            ##### FILL HERE (2 lines) ######
            p[u] = NULL  # cf. update rule above
            q[i] = NULL
            ################################

    return p, q

def estimate(u, i, p, q):
    '''Estimate rating of user u for item i.'''
    ##### FILL HERE (1 line) ######
    return NULL             #scalar product of p[u] and q[i] /!\ dimensions
    ################################
```

```
p, q = SGD(df_ratings)
```

## Get the estimate for all user-item pairs:

Get the user-item matrix filled with predicted ratings:

```
df_user_item_filled = pd.DataFrame(np.dot(p, q.transpose()))
df_user_item_filled.head()
```

However, it is using the encode ids ; we need to retrieve the association of encoded ids to original ids, and apply it:

```
df_ratings_encoded, dict_users, dict_items = encode_ids(df_ratings)
```

```
df_user_item_filled.rename(columns=(dict_items['itemId']), inplace=True)
df_user_item_filled.rename(index=(dict_users['userId']), inplace=True)

# Sort index/rows (userId's) and columns (itemId's)
df_user_item_filled.sort_index(axis=0, inplace=True)
df_user_item_filled.sort_index(axis=1, inplace=True)

df_user_item_filled.head()
```

Originally available ratings for user 1:

```
df_user_item.loc[1][:10]
```

Estimated ratings after the approximate SVD:

```
df_user_item_filled.loc[1][:10]
```

## Give recommendation to a user

For instance 10 recommended movies for user 1

```
recommendations = list((df_user_item_filled.loc[10]).sort_values(ascending=False)[:10].index)
recommendations
```

```
df_movies[df_movies.itemId.isin(recommendations)]
```

vs the ones that were rated initially:

```
already_rated = list((df_user_item.loc[10]).sort_values(ascending=False)[:10].index)
already_rated
```

```
df_movies[df_movies.itemId.isin(already_rated)]
```

This is all the movies in descending order of predicted rating. Let's remove the ones that where alread rated.

---

To put this into production, you'd first separate data into a training and validation set and optimize the number of latent factors (n_factors) by minimizing the Root Mean Square Error.
It is easier to use a framework that allows to do this, do cross-validation, grid search, etc.

# Gradient Descent SVD using Surprise

```
!pip install surprise
#!pip install scikit-surprise # if the first line does not work
```

```
# from surprise import Reader, Dataset, SVD, evaluate

# Following Surprise documentation examples
# https://surprise.readthedocs.io/en/stable/getting_started.html

from surprise import Reader, Dataset, SVD, evaluate, NormalPredictor
from surprise.model_selection import cross_validate
from collections import defaultdict

# As we're loading a custom dataset, we need to define a reader.
reader = Reader(rating_scale=(0.5, 5))

# The columns must correspond to user id, item id and ratings (in that order).
data = Dataset.load_from_df(df_ratings[['userId', 'itemId', 'rating']], reader)

# We'll use the famous SVD algorithm.
algo = SVD()

# Run 5-fold cross-validation and print results
cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=5, verbose=True)
```

#### Tune algorithm parameters with GridSearchCV

```
from surprise.model_selection import GridSearchCV

param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005],
              'reg_all': [0.4, 0.6]}
gs = GridSearchCV(SVD, param_grid, measures=['rmse', 'mae'], cv=3)

gs.fit(data)

# best RMSE score
print(gs.best_score['rmse'])

# combination of parameters that gave the best RMSE score
print(gs.best_params['rmse'])
```

```
# We can now use the algorithm that yields the best rmse:
algo = gs.best_estimator['rmse']
trainset = data.build_full_trainset()
algo.fit(trainset)
```

```
algo.predict(621,1)
```

```
df_data = data.df
df_data = df_data.join(df_movies,how="left", on='itemId',rsuffix='_', lsuffix='')
df_data[df_data['userId']==1].sort_values(by = 'rating',ascending=False)[:10]
```

```
# From Surprise documentation: https://surprise.readthedocs.io/en/stable/FAQ.html
def get_top_n(predictions, n=10):
    '''Return the top-N recommendation for each user from a set of predictions.

    Args:
        predictions(list of Prediction objects): The list of predictions, as
            returned by the test method of an algorithm.
        n(int): The number of recommendation to output for each user. Default
            is 10.

    Returns:
    A dict where keys are user (raw) ids and values are lists of tuples:
        [(raw item id, rating estimation), ...] of size n.
    '''

    # First map the predictions to each user.
    top_n = defaultdict(list)
    for uid, iid, true_r, est, _ in predictions:
        top_n[uid].append((iid, est))

    # Then sort the predictions for each user and retrieve the k highest ones.
    for uid, user_ratings in top_n.items():
        user_ratings.sort(key=lambda x: x[1], reverse=True)
        top_n[uid] = user_ratings[:n]

    return top_n
```

```
# Predict ratings for all pairs (u, i) that are NOT in the training set.
testset = trainset.build_anti_testset()
predictions = algo.test(testset)
```

```
top_n = get_top_n(predictions, n=10)
```

```
top_n.items()
```

```
# Print the recommended items for all user 1
for uid, user_ratings in top_n.items():
    print(uid, [iid for (iid, _) in user_ratings])
    if uid == 1:
        break
```

```
df_movies[df_movies.itemId.isin([318, 750, 1204, 858, 904, 48516, 1221, 912, 1276, 4973])]
```

```

```


# Quantum Angular Momentum Module

This file will show how to use the various objects and methods in the `sympy.physics.quantum.spin` module, with some examples. Much of the work in this module is based off Varschalovich "Quantum Theory of Angular Momentum".

```
from sympy.physics.quantum.spin import Jminus, Jx, Jz, J2, J2Op, JzKet, JzKetCoupled, Rotation, WignerD, couple, uncouple
from sympy.physics.quantum import Dagger, hbar, qapply, represent, TensorProduct
from sympy import factor, pi, S, Sum, symbols
```

## Basic spin states and operators

We can define simple spin states and operators and manipulate them with standard quantum machinery.

Define a spin ket:

```
jz = JzKet(1,1); jz
```

Find the vector representation of the state:

```
represent(jz)
```

$$\left[\begin{smallmatrix}1\\0\\0\end{smallmatrix}\right]$$

Create and evaluate an innerproduct of a bra and a ket:

```
ip = Dagger(jz) * jz; ip
```

```
ip.doit()
```

Apply an angular momentum operator to the state:

```
Jz * jz
```

```
qapply(Jz * jz)
```

```
Jminus * jz
```

```
qapply(Jminus * jz)
```

We can also do this for symbolic angular momentum states:

```
j, m = symbols('j m')
jz = JzKet(j, m); jz
```

```
J2 * jz
```

```
qapply(J2 * jz)
```

Find the matrix representation of a angular momentum operator:

```
represent(Jz, j=1)
```

$$\left[\begin{smallmatrix}\hbar & 0 & 0\\0 & 0 & 0\\0 & 0 & - \hbar\end{smallmatrix}\right]$$

## Utilizing different bases

Angular momentum states and operators are able to go between different spin bases

We can rewrite states as states in another basis:

```
jz = JzKet(1, 1)
jz.rewrite("Jx")
```

Vector representation can also be done into different bases:

```
represent(jz, basis=Jx)
```

$$\left[\begin{smallmatrix}\frac{1}{2}\\- \frac{1}{2} \sqrt{2}\\\frac{1}{2}\end{smallmatrix}\right]$$

When applying operators in another spin basis, any conversion necessary to apply the state is done, then the states are given back in the original basis. So in the following example, the state returned by `qapply` are in the $J_z$ basis:

```
Jx * jz
```

```
qapply(Jx * jz)
```

Rewriting states and applying operators between bases can also be done symbolically. In this case, the result is given in terms of Wigner-D matrix elements (see the next section for more information on the `Rotation` operator).

```
jz = JzKet(j, m)
jz.rewrite("Jx")
```

## Rotation Operator

Arbitrary rotations of spin states, written in terms of Euler angles, can be modeled using the rotation operator. These methods are utilized to go between spin bases, as seen in the section above.

Define an arbitrary rotation operator. The given angles are Euler angles in the `z-y-z` convention.

```
a, b, g = symbols('alpha beta gamma')
Rotation(a, b, g)
```

Find the Wigner-D matrix elements of the rotation operator as given by $\langle j, m'|\mathcal{R}(\alpha, \beta, \gamma)|j,m\rangle$:

```
mp = symbols('mp')
r = Rotation.D(j, m, mp, a, b, g); r
```

Numerical matrix elements can be evaluated using the `.doit()` method:

```
r = Rotation.D(1, 1, 0, pi, pi/2, 0); r
```

```
r.doit()
```

The Wigner small-d matrix elements give rotations when $\alpha=\gamma=0$. These matrix elements can be found in the same manner as above:

```
r = Rotation.d(j, m, mp, b); r
```

```
r = Rotation.d(1, 1, 0, pi/2); r
```

```
r.doit()
```

You can also directly create a Wigner-D matrix element:

```
WignerD(j, m, mp, a, b, g)
```

## Coupled and Uncoupled States and Operators

States and operators can also written in terms of coupled or uncoupled angular momentum spaces.

### Coupled States and Operators

Define a simple coupled state of two $j=1$ spin states:

```
jzc = JzKetCoupled(1, 0, (1, 1)); jzc
```

Note that the Hilbert space of coupled states is the direct sum of the coupled spin spaces. This can be seen in the matrix representation of coupled states:

```
jzc.hilbert_space
```

```
represent(jzc)
```

$$\left[\begin{smallmatrix}0\\0\\1\\0\\0\\0\\0\\0\\0\end{smallmatrix}\right]$$

We can also couple more than two spaces together. See the `JzKetCoupled` documentation for more complex coupling schemes involving more than 2 spaces.

```
jzc = JzKetCoupled(1, 1, (S(1)/2, S(1)/2, 1)); jzc
```

The normal operators are assumed to be diagonal in the corresponding coupled basis:

```
qapply(Jz * jzc)
```

### Uncoupled States and Operators

Uncoupled states are defined as tensor products of states:

```
jzu = TensorProduct(JzKet(1, 1), JzKet(S(1)/2, -S(1)/2)); jzu
```

Vector representation of tensor product states gives the vector in the direct product space:

```
represent(jzu)
```

$$\left[\begin{smallmatrix}0\\1\\0\\0\\0\\0\end{smallmatrix}\right]$$

Uncoupled operators are also defined as tensor products:

```
jzopu = TensorProduct(Jz, 1); jzopu
```

```
qapply(jzopu * jzu)
```

Coupled operators which are diagonalized by uncoupled states (e.g. $J_z$ and uncoupled $J_z$ eigenstates) can also be applied:

```
qapply(Jz * jzu)
```

Rewriting states works as before:

```
jzu.rewrite("Jx")
```

### Coulping and Uncoupling States

The `couple` method will couple an uncoupled state:

```
jzu = TensorProduct(JzKet(1, 1), JzKet(S(1)/2, -S(1)/2))
couple(jzu)
```

Similarly, the uncouple method will uncouple a coupled state

```
jzc = JzKetCoupled(2, 1, (1, S(1)/2, S(1)/2))
uncouple(jzc)
```

Uncoupling can also be done with the `.rewrite` method:

```
jzc.rewrite("Jz", coupled=False)
```

The `uncouple` method can also uncouple normal states if given a set of spin bases to consider:

```
jz = JzKet(2, 1)
uncouple(jz, (1, S(1)/2, S(1)/2))
```

## Example: Spin-orbit Coupling

If we start with a hydrogen atom, i.e. a nucleus of charge $Ze$ orbited by a single electron of charge $e$ with reduced mass $\mu$, ignoring energy from center-of-mass motion, we can write the Hamiltonian in terms of the relative momentum, $p$, and position, $r$, as:

$$H=\frac{p^2}{2\mu} - \frac{Ze^2}{r}$$

The resulting eigenfunctions have a separate radial and angular compents, $\psi=R_{n,l}(r)Y_{l,m}(\phi,\theta)$. While the radial component is a complicated function involving Laguere polynomials, the radial part is the familiar spherical harmonics with orbital angular momentum $\vec{L}$, where $l$ and $m$ give the orbital angular momentum quantum numbers. We represent this as a angular momentum state:

```
l, ml = symbols('l m_l')
orbit = JzKet(l, ml); orbit
```

Now, the spin orbit interaction arises from the electron experiencing a magnetic field as it orbits the electrically charged nucleus. This magnetic field is:

$$\vec{B} = \frac{1}{c}\frac{Ze\vec{v}\times\vec{r}}{r^3} = \frac{Ze\vec{p}\times\vec{r}}{mcr^3}=\frac{Ze\vec{L}}{mc\hbar r^3}$$

Then the spin-orbit Hamiltonian can be written, using the electron's magnetic dipole moment $\mu$, as:

$$H_{SO} = -\vec{\mu}\cdot\vec{B} = -\left(-\frac{g\mu_B \vec{S}}{\hbar}\right)\cdot\left(\frac{Ze\vec{L}}{mc\hbar r^3}\right)$$

Ignoring the radial term:

$$\propto \vec{L}\cdot\vec{S} = J^2 - L^2 - S^2$$

for $\vec{J}$, the coupled angular momentum.

The electron spin angular momentum is given as $\vec{S}$, where the spin wavefunction is:

```
ms = symbols('m_s')
spin = JzKet(S(1)/2, ms); spin
```

From this we build our uncoupled state:

```
state = TensorProduct(orbit, spin); state
```

For clarity we will define $L^2$ and $S^2$ operators. These behave the same as `J2`, they only display differently.

```
L2 = J2Op('L')
S2 = J2Op('S')
```

We also have the spin-orbit Hamiltonian:

```
hso = J2 - TensorProduct(L2, 1) - TensorProduct(1, S2); hso
```

Now we apply this to our state:

```
apply1 = qapply(hso * state); apply1
```

Note this has not applied the coupled $J^2$ operator to the states, so we couple the states and apply again:

```
apply2 = qapply(couple(apply1)); apply2
```

We now collect the terms of the sum, since they share the same limits, and factor the result:

```
subs = []
for sum_term in apply2.atoms(Sum):
    subs.append((sum_term, sum_term.function))
    limits = sum_term.limits
final = Sum(factor(apply2.subs(subs)), limits)
final
```

This gives us the modification of the angular part of the spin-orbit Hamiltonian. We see there is now the new $j$ quantum number in the coupled states, which we see from looking at the equation will have values $l\pm \frac{1}{2}$, and $m_j=m_l + m_s$. We still have the $l$ and $s$ quantum numbers.


# Multi-Armed bandit UCB

Thompson sampling is an ingenious algorithm that implicitly balances exploration and exploitation based on quality and uncertainty. Let's say we sample a 3-armed bandit and model the probability that each arm gives us a positive reward. The goal is of course to maximize our rewards by pulling the most promising arm. Assume at the current timestep arm-3 has mean reward of 0.9 over 800 pulls, arm-2 has mean reward of 0.8 over 800 pulls, and arm-1 has mean reward of 0.78 over 10 pulls. So far, arm-3 is clearly the best. But if we were to explore, would we choose arm-2 or arm-1? An $\epsilon$-greedy algorithm would, with probability $\epsilon$, just as likely choose arm-3, arm-2, or arm-1. However, arm-2 has been examined many times, as many as arm-1, and has a mean reward lower than arm-1. Selecting arm-2 seems like a wasteful exploratory action. Arm-1 however, has a lower mean reward than either arm-2 or arm-3, but has only been pulled a few times. In other words, arm-1 has a higher chance of being a better action than arm-3 when compared to arm-2, since we are more uncertain about its true value. The $\epsilon$-greedy algorithm completely misses this point. Thompson sampling, on the other hand, incorporates uncertainty by modelling the bandit's Bernouilli parameter with a prior beta distribution.

The beauty of the algorithm is that it always chooses the action with the highest expected reward, with the twist that this reward is weighted by uncertainty. It is in fact a Bayesian approach to the bandit problem. In our Bernouilli bandit setup, each action $k$ returns reward of 1 with probability $\theta_k$, and 0 with probability $1-\theta_k$. At the beginning of a simulation, each $\theta_k$ is sampled from a uniform distribution $\theta_k \sim Uniform(0,1)$ with $\theta_k$ held constant for the rest of that simulation (in the stationary case). The agent begins with a prior belief of the reward of each arm $k$ with a beta distribution, where $\alpha = \beta = 1$. The prior probability density of each $\theta_k$ is:

$$
p(\theta_k) = \frac{\Gamma(\alpha_k + \beta_k)}{\Gamma(\alpha_k)\Gamma(\beta_k)} \theta_k^{\alpha_k -1} (1-\theta_k)^{\beta_k-1}
$$
An action is chosen by first sampling from the beta distribution, followed by choosing the action with highest mean reward:$$
x_t = \text{argmax}_k (\hat{\theta}_k), \quad \hat{\theta}_k \sim \text{beta}(\alpha_k, \beta_k)
$$

According to Bayes' rule, an action's posterior distribution is updated depending on the reward $r_t$ received:$$
(\alpha_k, \beta_k) = (\alpha_k, \beta_k) + (r_t, 1-r_t)
$$

Thus the actions' posterior distribution are constantly updated throughout the simulation. We will measure the Thompson algorithm by comparing it with the $\epsilon$-greedy and Upper Confidence Bound (UCB) algorithms using regret. The per-period regret for the Bernouilli bandit problem is the difference between the mean reward of the optimal action minus the mean reward of the selected action:$$
\text{regret}_t(\theta) = \max_k \theta_k - \theta_{x_t}
$$

First we setup the necessary imports and the standard k-armed bandit. The get_reward_regret samples the reward for the given action, and returns the regret based on the true best action.

```python
import numpy as np
import matplotlib.pyplot as plt
from pdb import set_trace

stationary = True
class Bandit():
    def __init__(self, arm_count):
        """
        Multi-armed bandit with rewards 1 or 0
        At initialization, multiple arms are created. The probability of each arm returning reward 1
        if pulled is sample from Bernoulli(p), where randomly chosen from Uniform(0, 1) at initization
        """
        self.arm_count = arm_count
        self.generate_thetas()
        self.timestep = 0
        global stationary
        self.stationary=stationary

    def generate_thetas(self):
        self.thetas = np.random.uniform(0, 1, self.arm_count)

    def get_reward_regret(self, arm):
        """
        Returns random reward for arm action. Assument action are zero-indexed
        Args:
            arg is an int
        """
        self.timestep += 1
        if (self.stationary==False) and (self.timestep%100 == 0) :
          self.generate_thetas()
        # Simulate bernouilli sampling
        sim = np.random.uniform(0,1,self.arm_count)
        rewards = (sim<self.thetas).astype(int)
        reward = rewards[arm]
        regret = self.thetas.max() - self.thetas[arm]

        return reward, regret
```

We implement the two beta algorithms from [1], although we focus only on the Thompson algorithm. For the Bernouilli-greedy algorithm, the Bernouilli parameters are the expected values of the Beta distribution, i.e.:$$
\mathbb{E}(x_k) = \frac{\alpha_k}{(\alpha_k + \beta_k)}
$$

The Thompson algorithm follows the pseudocode below, based on [1]

Algorithm: Thompson($K$,$\alpha$, $\beta$)

for $t$ = 1,2, ..., do

    // sample action parameter from beta distribution

    for $k = 1, \dots, K$ do

        Sample $\hat{\theta}_k \sim \text{beta}(\alpha_k, \beta_k)$

    end for

    // select action, get reward

    $x_t \leftarrow \text{argmax}_k \hat{\theta}_k$

    $r_t \leftarrow \text{observe}(x_t)$

    // update beta parameters

    $(\alpha_{x_t}, \beta_{x_t}) \leftarrow (\alpha_{x_t}, \beta_{x_t})+(r_t, 1-r_t)$

end for

```python
class BetaAlgo():
  """
  The algos try to learn which Bandit arm is the best to maximize reward.

  It does this by modelling the distribution of the Bandit arms with a Beta,
  assuming the true probability of success of an arm is Bernouilli distributed.
  """
  def __init__(self, bandit):
    """
    Args:
      bandit: the bandit class the algo is trying to model
    """
    self.bandit = bandit
    self.arm_count = bandit.arm_count
    self.alpha = np.ones(self.arm_count)
    self.beta = np.ones(self.arm_count)

  def get_reward_regret(self, arm):
    reward, regret = self.bandit.get_reward_regret(arm)
    self._update_params(arm, reward)
    return reward, regret

  def _update_params(self, arm, reward):
    self.alpha[arm] += reward
    self.beta[arm] += 1 - reward

class BernGreedy(BetaAlgo):
  def __init__(self, bandit):
    super().__init__(bandit)

  @staticmethod
  def name():
    return 'beta-greedy'

  def get_action(self):
    """ Bernouilli parameters are the expected values of the beta"""
    theta = self.alpha / (self.alpha + self.beta)
    return theta.argmax()

class BernThompson(BetaAlgo):
  def __init__(self, bandit):
    super().__init__(bandit)

  @staticmethod
  def name():
    return 'thompson'

  def get_action(self):
    """ Bernouilli parameters are sampled from the beta"""
    theta = np.random.beta(self.alpha, self.beta)
    return theta.argmax()
```

For comparison, we also implement the $\epsilon$-greedy algorithm and Upper Confidence Bound (UBC) algorithm. The implementations are based on [2] (pages 24-28). The $\epsilon$-greedy algorithm is straightforward and explained briefly above. Note in this implementation we make use of the incremental update rule. That is, to update the $Q$-value of each action, we maintain a count of each action. For action $k$ taken at time $t$:$$
\begin{align}
r_t &\leftarrow \text{observe}(k) \\
N(k) &\leftarrow N(k) + 1 \\
Q(k) &\leftarrow Q(k) + \frac{1}{N(k)}[r_t-Q(k)] \\
\end{align}
$$

```python
epsilon = 0.1
class EpsilonGreedy():
  """
  Epsilon Greedy with incremental update.
  Based on Sutton and Barto pseudo-code, page. 24
  """
  def __init__(self, bandit):
    global epsilon
    self.epsilon = epsilon
    self.bandit = bandit
    self.arm_count = bandit.arm_count
    self.Q = np.zeros(self.arm_count) # q-value of actions
    self.N = np.zeros(self.arm_count) # action count

  @staticmethod
  def name():
    return 'epsilon-greedy'

  def get_action(self):
    if np.random.uniform(0,1) > self.epsilon:
      action = self.Q.argmax()
    else:
      action = np.random.randint(0, self.arm_count)
    return action

  def get_reward_regret(self, arm):
    reward, regret = self.bandit.get_reward_regret(arm)
    self._update_params(arm, reward)
    return reward, regret

  def _update_params(self, arm, reward):
    self.N[arm] += 1 # increment action count
    self.Q[arm] += 1/self.N[arm] * (reward - self.Q[arm]) # inc. update rule
```

The UCB action selection is different to the $\epsilon$-greedy. Like the Thompson algorithm, it includes a measure of uncertainty. The selected action follows the rule:$$
A_t = \text{argmax}_a \left[ Q_t(a) + c \sqrt{\frac{\ln t}{N_t (a)}} \right]
$$

Where $N_t (a)$ is the number of times action $a$ has been selected up to time $t$. As the denominator grows in the square root expression, the added effect on $Q_t(a)$ diminishes. This uncertainty measure is weighed by the hyperparameter $c$. The disadvantage is that, unlike the Thompson algorithm, this uncertainty hyperparameter requires tuning. Fundamentally, the UCB uncertainty is deterministic and beneficial, whereas in the Thompson case, uncertainty increases the expected reward variance. Since the Thompson algorithm samples the mean rewards from a beta distribution, the actions with high variance may not only have a higher chance of being chosen, but may also have a lower chance.

```python
ucb_c = 2
class UCB():
  """
  Epsilon Greedy with incremental update.
  Based on Sutton and Barto pseudo-code, page. 24
  """
  def __init__(self, bandit):
    global ucb_c
    self.ucb_c = ucb_c
    self.bandit = bandit
    self.arm_count = bandit.arm_count
    self.Q = np.zeros(self.arm_count) # q-value of actions
    self.N = np.zeros(self.arm_count) + 0.0001 # action count
    self.timestep = 1

  @staticmethod
  def name():
    return 'ucb'

  def get_action(self):
    ln_timestep = np.log(np.full(self.arm_count, self.timestep))
    confidence = self.ucb_c * np.sqrt(ln_timestep/self.N)
    action = np.argmax(self.Q + confidence)
    self.timestep += 1
    return action

  def get_reward_regret(self, arm):
    reward, regret = self.bandit.get_reward_regret(arm)
    self._update_params(arm, reward)
    return reward, regret

  def _update_params(self, arm, reward):
    self.N[arm] += 1 # increment action count
    self.Q[arm] += 1/self.N[arm] * (reward - self.Q[arm]) # inc. update rule
```

Below are some helper functions. The function simulate will simulate the learning for a single algorithm and return the mean regrets over a number of trials. The experiment function runs the simulations over all algorithms and plots their mean regrets

```python
def plot_data(y):
  """ y is a 1D vector """
  x = np.arange(y.size)
  _ = plt.plot(x, y, 'o')

def multi_plot_data(data, names):
  """ data, names are lists of vectors """
  x = np.arange(data[0].size)
  for i, y in enumerate(data):
    plt.plot(x, y, 'o', markersize=2, label=names[i])
  plt.legend(loc='upper right', prop={'size': 16}, numpoints=10)
  plt.show()

def simulate(simulations, timesteps, arm_count, Algorithm):
  """ Simulates the algorithm over 'simulations' epochs """
  sum_regrets = np.zeros(timesteps)
  for e in range(simulations):
    bandit = Bandit(arm_count)
    algo = Algorithm(bandit)
    regrets = np.zeros(timesteps)
    for i in range(timesteps):
      action = algo.get_action()
      reward, regret = algo.get_reward_regret(action)
      regrets[i] = regret
    sum_regrets += regrets
  mean_regrets = sum_regrets / simulations
  return mean_regrets

def experiment(arm_count, timesteps=1000, simulations=1000):
  """
  Standard setup across all experiments
  Args:
    timesteps: (int) how many steps for the algo to learn the bandit
    simulations: (int) number of epochs
  """
  algos = [EpsilonGreedy, UCB, BernThompson]
  regrets = []
  names = []
  for algo in algos:
    regrets.append(simulate(simulations, timesteps, arm_count, algo))
    names.append(algo.name())
  multi_plot_data(regrets, names)
```

## Experiments
For all experiments, in each trial the agents are allowed 1000 timesteps to maximize reward. We perform 5000 trials for each experiment.

### Baseline
In this first experiment, we aim for a standard setup, inspired by the bandit testbed in Chapter 2 of [2]. We set $\epsilon=0.1$ for the $\epsilon$-greedy algorithm, and $c=2$ for UCB. As can be seen in the chart below, the Thompson and $\epsilon$-greedy agents quickly converge to a steady regret value after only 200 steps. The UCB agent on the other hand very slowly decreases, lagging behind the two other agents, and continues in its downward trend even at step 1000. This suggests the non-Thompson agents could benefit from parameter tuning, whereas the Thompson agent works well right off the bat.

```python
# Experiment 1
arm_count = 10 # number of arms in bandit
epsilon = 0.1
ucb_c = 2
stationary=True
experiment(arm_count)
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


```python
import numpy as np
from scipy import stats
from matplotlib import pyplot as plt
import sys
```

## What is a $\chi^2$ distribution?

Before we talk about $\chi^2$ tests we need to discuss the eponymous distribution it is based upon. If I want to test how well a model fit some data, the question I am really asking is: what is the chance that the model only works by coincidence? How would I test this? Well, if I have $N$ data points, I could compare my model's performance, at the $N$ points, to a random distribution with $N$ degrees of freedom.

First, I need to decide what to compare. The simplest case is just the "distance" between the model and the data, i.e. $\propto (E - O)^2$, where $O$ is the data value and $E$ the predicted value. I don't want the comparator to have physical units (to ensure it works in any scenario). Therefore, I will define the distance between my model and the data at $x$ via
\begin{equation}
d^2 = \left(\frac{E(x) - O(x)}{\delta(x)}\right)^2 \; ,
\end{equation}
this $\delta(x)$ can be chosen several ways to produce different tests. If we know the uncertainties on our measurements $O$ then $\delta = \delta O$, otherwise we can just use either $O$ or $E$ (I prefer using $O$).

This isn't quite enough yet. How do I deal with multiple data points? I can just sum them, of course! This now defines our $\chi^2$ value
\begin{equation}
\chi^2 = \sum_i^N \left(\frac{E(x_i) - O(x_i)}{\delta(x_i)}\right)^2 \; .
\end{equation}

So, now I know what the comparator is, its the sum of the of N squares. Thus, my random version can be the sum of the squares of N randomly generated numbers. The distribution for this is the $\chi^2$ distribution with $N$ degrees of freedom. There's a caveat though, $\chi^2$ is a distribution for $N$ **independent** random variables. This means it will only be accurate if our measurements at each data point are uncorrelated. Most of the time we can assume this, but often instruments like telescopes do have correlations.

For instance, a gamma-ray telescope categorises photons into energy bins. This means it considers all photons in the range $E_{n}$ to $E_{n+1}$, bin $n$, as having the same energy. Instruments are never 100\% accurate, so photons can be misclassified. This means they can end up in either bin $n-1$ or $n+1$ instead of $n$. Thus the bin $n$ can become correlated with its neighbours, the correlation being defined by the accuracy of the instrument in classifying photon energies.

For now we will neglect the possibility of correlated data and merrily continue with independent data points. First, we can take a look at the $\chi^2$ distribution itself. Probability distributions have two important forms to consider: the probability density and cumulative density. The probability density (pdf) is the probability that a variable takes a value between $x$ and $x+dx$ per unit $x$. The cumulative distribution (cdf) is the probability that a variable is smaller than $x$, its thus an integral over the pdf from $-\infty$ to $x$.

Let's generate a pdf and a cdf for $\chi^2$, we will assume that we have $N=20$ data points. To do this I need to use the scipy.stats package (imported above). scipy.stats.chi2 provides access to all of the properties I need. Note that each method requires I specify the degrees of freedom as well as the values of $\chi^2$ (remember the distribution is assigning probabilities to values of $\chi^2$).

```python
N = 20
x = np.linspace(0, 100,num=100)
plt.plot(x,stats.chi2.pdf(x,N),label="PDF",color="k",linestyle="-")
plt.xlabel(r"$\chi^2$",fontsize=16); plt.ylabel("PDF",fontsize=16)
plt.show()

plt.plot(x,stats.chi2.cdf(x,N),label="CDF",color="k",linestyle="-")
plt.xlabel(r"$\chi^2$",fontsize=16); plt.ylabel("CDF",fontsize=16)
plt.show()
```

From my pdf I can see that the distribution has two tails, or two regions where the pdf becomes vanishingly small. The cdf and pdf are only defined for $\chi^2 > 0$ which makes sense when you think about it. Additionally, as expected, the cdf rises from 0 towards 1 as $\chi^2$ gets larger. It is the probability that $\chi^2 < x$ after all.

## How does this help us?

We know the probability that a set of random numbers will generate various sums of squares. This means we can compare the distance between our model and the data to this. What does this achieve? It tells us how likely any match between our model and data is down to pure chance. To make this comparison we will use the cdf. Sometimes people use the "survival function", or sf, which is 1 - cdf, I will discuss this later.

The cdf is small at small values of $\chi^2$. We can interpret this as meaning that, for a very good match between $O$'s and $E$'s, the probability of the relationship being a statistical fluke is very small. In practice we use several thresholds for this. These are called confidence levels and are often referred to as $\sigma$'s, i.e. $1\sigma$ is a confidence level of $\sim 68\%$, $2\sigma$ is $95\%$, and $3\sigma$ is $99.99\%$. The percentages represent our confidence that the model is accurately predicting the data. Where do these levels come from? Simply put these are integrals over a normal distribution. The scheme is for the ith C.L.
\begin{equation}
P_{CL}(i) = \int_{-i}^{i} dx \, \frac{\mathrm{e}^{-\frac{x^2}{2}}}{\sqrt{2\pi}} \; .
\end{equation}
The normal distribution is used here as it corresponds to a model of normally distributed errors (under assumptions like central limit theorem). You can see that, since $\sigma = 1$, we are integrating from $-i\sigma$ to $i\sigma$. The choice of mean and variance are irrelevant, these results will always be true for a normal distribution.

## Exclusion by data

Here is where we use data to say that some models must be rejected at a given confidence level, because they make the wrong predictions. Again we use the same confidence levels as before, but their meaning is subtly different. Now we are saying we reject the model because it has $95\%$ chance of over-predicting the data, for instance. This being what we would call a $2\sigma$ exclusion. Note that whether we care about under or over-predicting data depends on our scenario. For dark matter hunting, we can only place limits on models that produce more emissions than actually observed (if it were less, these models could just be hidden under the stronger signal). Thus, we only care about over-predictions by the model.

To implement this, I will generate some fake data and a toy model

```python
def modelResult(x):
    mu = 5e1
    sig = 1e1
    #artificial model I created
    return 1e14*np.exp(-(x-mu)**2/sig**2*0.5)

def dataResult(x,y0,alpha):
    #fake data that's just a power-law
    return y0*x**alpha
```

I will now choose some x range to consider them over, I will choose it so the gaussian bump of the model fits into the range

```python
x = np.logspace(1,2,num=20)
model = modelResult(x)
data = dataResult(x,1e-10,-1.5)
```

Now we are at the point you would be in reality. You have some data points and model predictions at those data points. In dark matter hunting we have a free parameter $\langle \sigma V \rangle$, so we must consider the model results for a range of cross-sections. By doing so, we hope to exclude values of the cross-section that over-predict the data. Note that my model function here is like dark matters (assumes $\langle \sigma V \rangle = 1$ cm $^3$ s $^{-1}$).

```python
sigV = np.logspace(-3,3,num=3000)*1e-26
model = np.tensordot(sigV,model,axes=0)
```

What I have done here is generated a 2D array where each element is every possible combination of a sigV value times a model value. I will need to ensure my data is the same shape array (to be able to compare them). However, data does not vary with sigV, it is fixed. Thus, I will do the following

```python
data = np.tensordot(np.ones_like(sigV),data,axes=0)
```

Which generates a 2D array filled with copies of data array. Now I can find their $\chi^2$ values

```python
def chiSquare(model,data,axis=-1,dataUncert=None,alpha=2):
    """
    model and data are the predicted and observed values respectively
    This method assumes data and model are sampled at the same values of the independent variable
    """
    if dataUncert is None:
        dataUncert = data #modify the test if we don't know the uncertainties on the data
    if dataUncert.shape != data.shape or model.shape != data.shape:
        print("chiSquare error: model, data, and dataUncert (if specified) must be the same shape!")
        sys.exit(2)
    return np.sum((model-data)**alpha/dataUncert**alpha,axis=axis) #we always sum over the data points so I need to specify which axis this corresponds to

chi2 = chiSquare(model,data,axis=-1)

plt.plot(sigV,chi2)
plt.ylabel(r"$\chi^2$",fontsize=16)
plt.xlabel(r"$\langle \sigma V\rangle$ (cm$^3$ s$^{-1}$)",fontsize=16)
plt.yscale("log");plt.xscale("log")
plt.show()
```

Note my use of the argument "axis=-1", this is because I multiplied data by a set of ones on the left. Thus, the data axis is the last axis (or -1). We can see our distribution has a minimum. This we could use to find a best-fit dark matter $\langle \sigma V \rangle$. However, I want to exclude over predicting models. Let's begin by normalising over $\chi^2$ by subtracting off the minimum value. This is done because now we are performing a so-called "likelihood ratio test". Where our likelihood is
\begin{equation}
\mathcal{L}_i = \mathrm{e}^{-\chi_i^2/2} \; ,
\end{equation}
for model $i$ and our ratio test is done via
\begin{equation}
2\ln{\left(\frac{\mathcal{L}_i}{\mathcal{L}_{\mathrm{max}}}\right)} = \chi^2_i - \chi^2_{\mathrm{min}} \; .
\end{equation}

How do we know what points are excluded though? Let's use a 95\% confidence exclusion. Now, any model with a $\chi^2$ that corresponds to the cdf being 0.95 or greater has a 95\% chance of not being a good explanation of the data. However, this will include both under and over-predicting models (as $\chi^2$ is a square of the distance between data and model). To exclude the under predicting models we will look for values of $\langle \sigma V\rangle$ that have $E(x_i) > O(x_i)$ for any of the data points $x_i$. Then we will only examine the $\chi^2$ values for the models that survive this test.

```python
chi = (model-data)/data #2D array (len(sigV) by len(data))
chi2Over = np.zeros_like(sigV)
for i in range(len(sigV)):
        if np.any(chi[i]>0.0): #is E > O for any data points for ith sigV value
            chi2Over[i] = chi2[i]
print(chi2Over.shape)
plt.plot(sigV,chi2,color="k",label="All models")
plt.plot(sigV,chi2Over,color="y",linestyle="--",label="Over-predicting only")
plt.xscale("log");plt.yscale("log")
plt.ylim([1e0,1e2])
plt.xlim([sigV[0],sigV[-1]])
plt.ylabel(r"$\chi^2$",fontsize=16)
plt.xlabel(r"$\langle \sigma V\rangle$ (cm$^3$ s$^{-1}$)",fontsize=16)
plt.legend()
plt.show()
```

The plot tells us that $\langle \sigma V \rangle \gtrsim 3\times 10^{-27}$ cm $^3$ s $^{-1}$ result in over-predicting models. What we can now do is determine how badly they over-predict the data. For this we will need to determine the cdf for these $\chi^2$ values. First, we need to normalise $\chi^2$. Then we will feed this into the cdf and ask which is the smallest $\langle \sigma V \rangle$ that yields a cdf $> 0.95$. Note we need to use the original minimum from chi2, not chi2Over. We will plot a comparison of 2*data (roughly like a 95\% exclusion) with this model prediction just as a sanity check.

```python
chiDiffOver = chi2Over - np.min(chi2)
chiDiff = chi2 - np.min(chi2)
cdfOver = stats.chi2.cdf(chiDiffOver,len(x))
cdf = stats.chi2.cdf(chiDiff,len(x))
plt.plot(sigV,cdf,color="k")
plt.xscale("log")
plt.plot(sigV,np.ones_like(sigV)*0.95,linestyle=":",color="y")
plt.ylim([0,1.1])
plt.ylabel("CDF",fontsize=16)
plt.xlabel(r"$\langle \sigma V\rangle$ (cm$^3$ s$^{-1}$)",fontsize=16)
plt.show()

upperLim = np.min(sigV[cdfOver>=0.95])

print("Values of sigV > {:.2e} cm^3 s^-1 are excluded at a 95% confidence level".format(upperLim))

plt.plot(x,dataResult(x,1e-10,-1.5)*2,label="2*data")
plt.plot(x,modelResult(x)*upperLim,label="95% Upper-limit model")
plt.ylabel(r"$y$",fontsize=16)
plt.xlabel(r"$x$",fontsize=16)
plt.yscale("log");plt.xscale("log")
plt.legend()
plt.show()
```

What we see is a small selection of negative $\chi$ values are excluded at a 95\% confidence interval (above yellow dotted line). We found the smallest such value using np.min(sigVNeg[cdf>=0.95]) to select the minimum sigV that has a cdf value above 0.95. Finally, our sanity check is good, as our upper-limit model just exceeds 2*data.

## Incorporating dark matter mass

The above example works for a fixed dark matter mass. I will now modify the model to include a fake version of the effect of varying the mass. Due to my choice of model, I will need to expand the x range as well.

```python
def modelResult(x,m):
    mu = 5e1*m/10.0
    sig = 1e1*m/10
    #artificial model I created
    return 1e14*np.exp(-(x-mu)**2/sig**2*0.5)/(m/10)**2

def dataResult(x,y0,alpha):
    #fake data that's just a power-law
    return y0*x**alpha
```

Now what I aim to do is find a sigV limit for a set of masses. I will choose a few, logarithmically spaced, and run a loop over them

```python
mSet = np.array([1e1,1e2,1e3])
upperLims = []

for m in mSet:
    x = np.logspace(1,4,num=20)
    model = modelResult(x,m)
    data = dataResult(x,1e-10,-1.5)
    sigV = np.logspace(-3,5,num=3000)*1e-26
    model = np.tensordot(sigV,model,axes=0)
    data = np.tensordot(np.ones_like(sigV),data,axes=0)
    chi2 = chiSquare(model,data,axis=-1,alpha=2)
    chi = (model-data)/data #2D array (len(sigV) by len(data))
    chi2Over = np.zeros_like(sigV)
    for i in range(len(sigV)):
        if np.any(chi[i]>0.0): #is E > O for any data points for ith sigV value
            chi2Over[i] = chi2[i]
    chiDiff = chi2Over - np.min(chi2)
    cdf = stats.chi2.cdf(chiDiff,len(x))
    upperLims.append(np.min(sigV[cdf>=0.95]))

plt.plot(mSet,upperLims,linestyle="-",color="k")
plt.plot(mSet,np.ones_like(mSet)*2e-26,linestyle=":",color="k")
plt.yscale("log");plt.xscale("log")
plt.ylabel(r"$\langle \sigma V\rangle$ (cm$^3$ s$^{-1}$)",fontsize=16)
plt.xlabel(r"$m_\chi$",fontsize=16)
plt.show()
```

My final plot is then the exclusion curve, models above the solid line are excluded at 95% C.L. The dotted line is the thermal relic cross-section (the value needed to produce the amount of observed dark matter )


# PRAKTIKUM 6
__Hampiran Polinomial__

*Topik*
1. Hampiran Deret Taylor
2. Analisis Galat Hampiran Deret Taylor

# 1 Hampiran Deret Taylor
Asumsikan bahwa $ f\in C^{N+1}[a,b] $ dan diketahui $ x_0\in[a,b] $. Jika $ x\in[a,b] $, maka

\begin{equation}
f(x)=P_N(x)+E_N(x)
\end{equation}

dimana $ P_N(x) $ adalah fungsi polinomial yang dapat digunakan untuk menghampiri $ f(x) $, yaitu

\begin{equation}
f(x)\approx P_N(x)=\sum_{k=0}^{N}\dfrac{f^{(k)}(x_0)}{k!}(x-x_0)^k
\end{equation}

dan galat $ E_N(x) $ memiliki bentuk

\begin{equation}
E_N(x)=\dfrac{f^{(N+1)}(c)}{(N+1)!}(x-x_0)^{N+1}
\end{equation}

### Contoh Soal 1
Diberikan fungsi $ f(x)=e^x $. Gunakan $ x_0=0 $ dan carilah $ P_5(x) $, $ P_7(x) $, dan $ P_9(x) $.

Jawab:

Untuk menyelesaikan soal tersebut, langkah pertama yang dilakukan adalah mendefinisikan $P_N(x)$ dengan $f(x)=e^x$, yaitu
1. Dicari nilai $f^{(k)}(x_0)$  yaitu turunan ke-$k$ dari $f(x_0)$

Fungsi turunan $f(x)=e^x$ adalah
\begin{align*}
&f'(x)=e^x\\
&f''(x)=e^x\\
&f'''(x)=e^x\\
&f^{(4)}(x)=e^x
\end{align*}
Karena $ x_0=0 $, didapatkan
\begin{align*}
&f'(x_0)=e^0=1\\
&f''(x_0)=e^0=1\\
&f'''(x_0)=e^0=1\\
&f^{(4)}(x_0)=e^0=1
\end{align*}
Dengan memperhatikan pola di atas, maka $f^{(k)}(x_0) = 1$

2. Definisikan $P_N(x)$
\begin{align}
P_N(x)&=\sum_{k=0}^{N}\dfrac{f^{(k)}(x_0)}{k!}(x-x_0)^k\\
      &=\sum_{k=0}^{N}\dfrac{1}{k!}(x)^k
\end{align}
Dengan $ N=5,7,9 $ untuk menghitung $ P_5(x) $, $ P_7(x) $, dan $ P_9(x) $, maka diperoleh
\begin{align*}
& P_5(x)=\sum_{k=0}^{5}\dfrac{1}{k!}\ {x^k} = 1+x+\dfrac{x^2}{2!}+\dfrac{x^3}{3!}+\dfrac{x^4}{4!}+\dfrac{x^5}{5!}\\
& P_7(x)=\sum_{k=0}^{7}\dfrac{1}{k!}\ {x^k} = 1+x+\dfrac{x^2}{2!}+\dfrac{x^3}{3!}+\dfrac{x^4}{4!}+\dfrac{x^5}{5!}+\dfrac{x^6}{6!}+\dfrac{x^7}{7!}\\
& P_9(x)=\sum_{k=0}^{9}\dfrac{1}{k!}\ {x^k} = 1+x+\dfrac{x^2}{2!}+\dfrac{x^3}{3!}+\dfrac{x^4}{4!}+\dfrac{x^5}{5!}+\dfrac{x^6}{6!}+\dfrac{x^7}{7!}+\dfrac{x^8}{8!}+\dfrac{x^9}{9!}
\end{align*}

# 2 Analisis Galat
Pada subbab 1, telah diketahui bahwa galat $ E_N(x) $ memiliki bentuk

\begin{equation}
E_N(x)=\dfrac{f^{(N+1)}(c)}{(N+1)!}(x-x_0)^{N+1}
\end{equation}

Diberikan fungsi $ f(x)=e^x $. Hitung nilai $E_9(x)$ jika $ |x|\le1 $

Jawab:

Galat untuk hampiran $ P_9(x) $ adalah

$$ E_9(x)=\dfrac{f^{(10)}(c)}{10!}\ x^{10} $$

Karena $ |x|\le 1 $ sehingga nilai $ -1\le c\le 1 $. Serta, karena $ f^{(10)}(c)=e^c $ maka $ e^c $ merupakan fungsi naik. Dengan demikian, nilai $ e^c\le e^1<3 $, sehingga nilai batas galat untuk hampiran $ P_9(x) $ adalah

$$ |E_9(x)|=\dfrac{|f^{(10)}(c)|}{10!}\ x^{10}\le\dfrac{e^c}{10!}<\dfrac{3}{10!}<8.26719\times10^{-7} $$

Jadi, hampiran $ E_9(x) $ memiliki batas akurasi hingga 6 angka desimal.

__Pertanyaan__ ? Berapa N minimal agar hampiran $f(x)=e^x$ memiliki galat maksimal $10^{-9}$ untuk selang  $|x|\le1$

Cek Kuliah 6

### Contoh 1
Diberikan fungsi $ f(x)=\exp(x) $. Gunakan teorema deret Taylor untuk
1. menggambarkan hampiran polinomial dari fungsi $ f $ dengan $ n=1,2,3,4 $ pada interval $ [-3,3] $ beserta $ f(x)=\exp(x) $
2. menggambarkan galat dari masing-masing hampiran terhadap fungsi $ f $ pada interval $ [-3,3] $.

Berdasarkan contoh sebelumnya didapatkan bentuk deret Taylor dari $ f(x)=\exp(x) $, yaitu

$$ P_N(x) = \sum_{k=0}^{N}\dfrac{1}{k!}(x)^k $$

```julia
function pn_exp(x,n)
    y = 0;
    for k = 0:n
        y = y + x.^k./factorial(k);
    end
    return y
end
```

```julia
# Definisikan batas interval dan nilai y eksak.
x = -3:0.01:3;
y = exp.(x);
```

```julia
using Plots
```

```julia
# hitung nilai P1 P2 P3 P4 pada interval x yang ditentukan
p1= pn_exp.(x,1);
p2= pn_exp.(x,2);
p3= pn_exp.(x,3);
p4= pn_exp.(x,4);
# gambarkan grafik masing-masing fungsi
p1a = plot(x,y,label = "f(x)",legend=:topleft)
plot!(x,p1,label = "p1")
plot!(x,p2,label = "p2")
plot!(x,p3,label = "p3")
plot!(x,p4,label = "p4")
# Tambahkan grid, title, label dan legend
title!("Plot perbandingan fungsi f(x) dan hampirannya")
xlabel!("x")
ylabel!("y")
```

```julia
# hitung nilai galat P1 P2 P3 P4 pada interval x yang ditentukan
e1 = abs.(y.-p1);
e2 = abs.(y.-p2);
e3 = abs.(y.-p3);
e4 = abs.(y.-p4);
# gambarkan grafik masing-masing galat
p1b = plot(x,e1,label = "E1",yaxis=:log,ylim=(1e-12, 100))
plot!(x,e2,label = "E2",legend=:bottomright)
plot!(x,e3,label = "E3")
plot!(x,e4,label = "E4")

title!("Plot perbandingan galat hampiran fungsi f")
xlabel!("x")
ylabel!("y")

```

#### Pertanyaan
Bagaimana hampiran yang lain? seperti

$\sin(x)$ ?

$\cos(x)$ ?

Setiap fungsi memiliki deret Taylor yang unik.

Jadi, seperti langkah-langkah pada $e^x$, untuk menentukan hampiran $sin(x)$ maka harus mendefinisikan $P_N(x)$ terlebih dulu.

Misalkan, akan dicari hampiran $\cos(x)$ disekitar $x=0$.

Maka, cari $f^{(k)}(x_0)$ dengan $x_0=0$

\begin{align}
\begin{split}
&f(x) = cos(x)\\
&f'(x)= -sin(x)\\
&f''(x)=-cos(x)\\
&f'''(x) = sin(x)\\
&f^{(4)}(x) = cos(x)
\end{split}
\ \ \ \ \ \ \text{maka} \ \ \ \ \ \
\begin{split}
&f(0) = 1\\
&f'(0)= 0\\
&f''(0)=-1\\
&f'''(0) = 0\\
&f^{(4)}(0) = 1
\end{split}
\end{align}
Jadi,
\begin{align}
P_N(x) &= 1+0-\frac{x^2}{2!}+0+\frac{x^4}{4!}+\dots\\
       &= 1-\frac{x^2}{2!}+\frac{x^4}{4!}+\dots\\
       &= \sum_{k=0}^{N}\dfrac{(-1)^k}{(2k)!}x^{2k}
\end{align}

Beberapa bentuk deret Taylor dapat dilihat di
https://id.wikipedia.org/wiki/Deret_Taylor

# Soal Latihan
Kerjakan soal berikut pada saat kegiatan praktikum berlangsung.

`Nama: ________`

`NIM: ________`

### Soal 1

Diberikan fungsi

$$  f(x)=\cos(x)  $$

Ulangi langkah-langkah pada **Contoh 1** untuk membentuk hampiran polinomial dengan derajat $ n=1,2,3,4 $ pada interval $ [-4,4] $.

```julia

```


# Problem Sheet 1 Question 2b

The general form of the population growth differential equation
 \begin{equation} y^{'}=y-t, \ \ (0 \leq t \leq 2)  \end{equation}
with the initial condition
 \begin{equation}y(0)=2\end{equation}
For N=4
with the analytic (exact) solution
 \begin{equation} y= e^{t}+t+1.\end{equation}

Apply __Euler’s Method__ to approximate the solution of the given initial value problems using the indicated number of time steps. Compare the approximate solution with the given exact solution, and compare the actual error with the theoretical error.

```python
## Library
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
import pandas as pd

warnings.filterwarnings("ignore")
```

## General Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
 \begin{equation} h=\frac{b-a}{N}.\end{equation}
## Specific Discrete Interval
Here the interval is $0\leq t \leq 2$ with $N=4$
 \begin{equation} h=\frac{2-0}{4}=0.5.\end{equation}
This gives the 5 discrete points with stepsize h=0.5:
 \begin{equation} t_0=0, \ t_1=0.5, \ ... t_{4}=2. \end{equation}
This is generalised to
 \begin{equation} t_i=0+i0.5, \ \ \ i=0,1,...,4.\end{equation}
The plot below illustrates the discrete time steps from 0 to 2.

```python
### Setting up time
t_end=2
t_start=0
N=4
h=(t_end-t_start)/(N)
t=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.plot(t[0],0*t[0],'o',color='green')

plt.title('Illustration of discrete time points for h=%s'%(h))
plt.plot();
```

## Euler Solution

The Euler difference equation is
 \begin{equation} w_{i+1}=w_i+h(w_i-t_i), \end{equation}
for $i=0,1,2,3$
with
$w_0=2.$

```python
IC=2

INTITIAL_CONDITION=IC
w=np.zeros(N+1)
w[0]=INTITIAL_CONDITION
for i in range (0,N):
    w[i+1]=w[i]+h*(w[i]-t[i])
```

```python
y=np.exp(t)+t+1 # Exact Solution
```

## Global Error
The upper bound of the global error is:
 \begin{equation} |y(t_i)-w_i|\leq\frac{Mh}{2L} |e^{Lt_i}-1| \end{equation}
where
$|y''(t)|\leq M $ for $t \in (0,2)$ and $L$ is the Lipschitz constant.
Generally we do not have access to $M$ or $L$, but as we have the exact solution we can find $M$ and $L$.
### Local Error
The global error consistent of an exponential term and the local error.
The local error for the Euler method is
 \begin{equation} |y(t_1)-w_1|<\tau=\frac{Mh}{2}, \end{equation}

by differentiation the exact solution
$ y= y= e^{t}+t+1$
twice
 \begin{equation}y'= e^{t}+1,\end{equation}
 \begin{equation}y''=e^{t}.\end{equation}
Then we choose the largest possible value of $y''$ on the interval of interest $(0,2)$, which gives,

 \begin{equation}|y''(t)|\leq e^{2}=7.38<8, \end{equation}
therefore
 \begin{equation}M=8.\end{equation}
This gives the local error
 \begin{equation}|y(t_1)-w_1|<\tau=\frac{Mh}{2}=\frac{8h}{2}=4h.\end{equation}
### Lipschitz constant
The Lipschitz constant $L$ is from the Lipschitz condition,
 \begin{equation}\left| \frac{\partial f(t,y)}{\partial y}\right|\leq L. \end{equation}
The constant can be found by taking partical derivative of $f(t,y)=y-t$ with respect to $y$
 \begin{equation}\frac{\partial f(t,y)}{\partial y}=1\end{equation}
 \begin{equation}L=1.\end{equation}

### Specific Global Upper Bound Formula
Puting these values into the upper bound gives:
 \begin{equation} |y(t_i)-w_i|\leq\frac{8h}{2} |e^{t_i}-1|.  \end{equation}

```python
Upper_bound=8*h/(2*1)*(np.exp(t)-1) # Upper Bound
```

## Results

```python
fig = plt.figure(figsize=(12,5))
# --- left hand plot
ax = fig.add_subplot(1,4,1)
plt.plot(t,w,'o:',color='k')
plt.plot(t[0],w[0],'o',color='green')

#ax.legend(loc='best')
plt.title('Numerical Solution (w) h=%s'%(h))

# --- right hand plot
ax = fig.add_subplot(1,4,2)
plt.plot(t,y,color='blue')
plt.title('Analytic Solution (y)')

#ax.legend(loc='best')
ax = fig.add_subplot(1,4,3)
plt.plot(t,np.abs(w-y),'o:',color='red')
plt.title('Exact Error |w-y|')

# --- title, explanatory text and save

ax = fig.add_subplot(1,4,4)
plt.plot(t,np.abs(w-y),'o:',color='red')
plt.plot(t,Upper_bound,color='purple')
plt.title('Upper Bound')

# --- title, explanatory text and save
fig.suptitle(r"$y'=y-t$", fontsize=20)
plt.tight_layout()
plt.subplots_adjust(top=0.85)
```

```python

d = {'time t_i': t,    'Euler (w_i) ':w,    'Exact (y_i) ':y,    'Exact Error( |y_i-w_i|) ':np.abs(y-w),'Upper Bound':Upper_bound}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Euler (w_i)
      Exact (y_i)
      Exact Error( |y_i-w_i|)
      Upper Bound

      0
      0.0
      2.0000
      2.000000
      0.000000
      0.000000

      1
      0.5
      3.0000
      3.148721
      0.148721
      1.297443

      2
      1.0
      4.2500
      4.718282
      0.468282
      3.436564

      3
      1.5
      5.8750
      6.981689
      1.106689
      6.963378

      4
      2.0
      8.0625
      10.389056
      2.326556
      12.778112

```python

```

Chapter 01 - Euler Methods/1_Problem Sheet/102b_Problem_Sheet.ipynb


# Stochastic Processes:
Data Analysis and Computer Simulation

# Distribution function and random number

# 2. Generating random numbers with Gaussian/binomial/Poisson distributions

# 2.1. Preparations

## Python built-in functions for random numbers (numpy.random)

1. `seed(seed)`: Initialize the generator with an integer "seed".
1. `rand(d0,d1,...,dn)`: Return a multi-dimensional array of uniform random numbers of shape (d0, d1, ... , dn).
1. `randn(d0,d1,...,dn)`: The same as above but from the standard normal distribution.
1. `binomial(M,p,size)`: Draw samples from a binomial distribution with "M" and "p".
1. `poisson(a,size)`: Draw samples from a Poisson distribution with "a".
1. `choice([-1,1],size)`: Generates random samples from the two choices, -1 or 1 in this case.
1. `normal(ave,std,size)`: Draw random samples from a normal distribution.
1. `uniform([low,high,size])`: Draw samples from a uniform distribution.

- See the Scipy website for details

https://docs.scipy.org/doc/numpy-dev/reference/routines.random.html

## Import common libraries

```python
% matplotlib inline
import numpy as np # import numpy library as np
import math # use mathematical functions defined by the C standard
import matplotlib.pyplot as plt # import pyplot library as plt
plt.style.use('ggplot') # use "ggplot" style for graphs
```

# 2.2. Normal / Gaussian distribution

## Generate random numbers, $x_0, x_1, \cdots, x_N$

```python
ave = 0.0 # set average
std = 1.0 # set standard deviation
N = 100000 # number of generated random numbers
np.random.seed(0) # initialize the random number generator with seed=0
X = ave+std*np.random.randn(N) # generate random sequence and store it as X
plt.ylim(-10,10) # set y-range
plt.xlabel(r'$i$',fontsize=16) # set x-label
plt.ylabel(r'$x_i$',fontsize=16) # set y-label
plt.plot(X,',') # plot x_i vs. i (i=1,2,...,N) with dots
plt.show() # draw plots
```

## Compare the distribution with the normal distribution function

\begin{equation}
P(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\langle X \rangle)^2}{2\sigma^2}\right]\tag{D1}
\end{equation}

```python
plt.hist(X,bins=25,normed=True) # plot normalized histgram of R using 25 bins
x = np.arange(-10,10,0.01) # create array of x from 0 to 1 with increment 0.01
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # create array of y as y=1
plt.xlim(-10,10) # set x-range
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$x$',fontsize=16) # set x-label
plt.ylabel(r'$P(x)$',fontsize=16) # set y-label
plt.legend([r'Gaussian',r'histgram'], fontsize=16) # set legends
plt.show() # display plots
```

## Calculate the auto-correlation function $\varphi(i)$

### The definition

\begin{equation}
\varphi(i)=\frac{1}{N}\sum_{j=1}^{N} \left(x_j - \langle X\rangle\right)\left(x_{i+j}-\langle X\rangle\right) \tag{D2}
\end{equation}

\begin{equation}
\varphi(i=0)=\frac{1}{N}\sum_{j=1}^{N} \left(x_j - \langle X\rangle\right)^2=\langle x_j - \langle X\rangle\rangle^2=\sigma^2\tag{D3}
\end{equation}

\begin{equation}
\varphi(i\ne 0)= \langle x_j - \langle X\rangle\rangle\langle x_{i\ne j} - \langle X\rangle\rangle=0\hspace{5mm}
(\rightarrow{\rm White\ noise})\hspace{-12mm}
\tag{D4}
\end{equation}

### A code example to calculate auto-correlation

```python
def auto_correlate(x):
    cor = np.correlate(x,x,mode="full")
    return cor[N-1:]
c = np.zeros(N)
c = auto_correlate(X-ave)/N
plt.plot(c)
plt.xlim(-1000,10000)
plt.xlabel(r'$i$',fontsize=16)
plt.ylabel(r'$\varphi(i)$',fontsize=16)
print('\sigma^2  =',std**2)
plt.show()
```

# 2.3. Binomial distribution

## Generate random numbers, $n_0, n_1, \cdots, n_N$

```python
p = 0.5 # set p, propability to obtain "head" from a coin toss
M = 100 # set M, number of tosses in one experiment
N = 100000 # number of experiments
np.random.seed(0) # initialize the random number generator with seed=0
X = np.random.binomial(M,p,N) # generate the number of heads after M tosses, N times, and store it as X
plt.xlabel(r'$i$',fontsize=16) # set x-label
plt.ylabel(r'$n_i$',fontsize=16) # set y-label
plt.plot(X,',') # plot n_i vs. i (i=1,2,...,N) with dots
plt.show() # draw plots
```

## Compare the distribution with the Binomial distribution function

\begin{equation}
P(n)=\frac{M!}{n!(M-n)!}p^{n}(1-p)^{M-n}\tag{D5}
\end{equation}

\begin{equation}
\langle n \rangle=Mp\tag{D6}
\end{equation}

\begin{equation}
\sigma^2=Mp(1-p)\tag{D7}
\end{equation}

```python
def binomial(n,m,p):
    comb=math.factorial(m)/(math.factorial(n)*math.factorial(m-n))
    prob=comb*p**n*(1-p)**(m-n)
    return prob
plt.hist(X,bins=20,normed=True) # plot normalized histgram of R using 22 bins
x = np.arange(M) # generate array of x values from 0 to 100, in intervals of 1
y = np.zeros(M) # generate array of y values, initialized to 0
for i in range(M):
    y[i]=binomial(i,M,p) # compute binomial distribution P(n), Eq. (D5)
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Binomial',r'histgram'], fontsize=16) # set legends
plt.show() # display plots
```

## Calculate the auto-correlation function $\varphi(i)$

### The definition

\begin{equation}
\varphi(i)=\frac{1}{N}\sum_{j=1}^{N} \left(n_j - \langle n\rangle\right)\left(n_{i+j}-\langle n\rangle\right) \tag{D8}
\end{equation}

\begin{equation}
\varphi(i=0)=\frac{1}{N}\sum_{j=1}^{N} \left(n_j - \langle n\rangle\right)^2=\langle n_j - \langle n\rangle\rangle^2=\sigma^2=Mp(1-p)\tag{D9}
\end{equation}

\begin{equation}
\varphi(i\ne 0)= \langle n_j - \langle n\rangle\rangle\langle n_{i\ne j} - \langle n\rangle\rangle=0\hspace{8mm}
(\rightarrow{\rm White\ noise})\hspace{10mm}
\tag{D10}
\end{equation}

### A code example to calculate auto-correlation

```python
def auto_correlate(x):
    cor = np.correlate(x,x,mode="full")
    return cor[N-1:]
c = np.zeros(N)
c = auto_correlate(X-M*p)/N
plt.plot(c)
plt.xlim(-1000,10000)
plt.xlabel(r'$i$',fontsize=16)
plt.ylabel(r'$\varphi(i)$',fontsize=16)
print('\sigma^2  =',M*p*(1-p))
plt.show()
```

# 2.4. Poisson distribution

## Generate random numbers, $n_0, n_1, \cdots, n_N$

```python
a = 10.0 # set a, the expected value
N = 100000 #  number of generated random numbers
np.random.seed(0) # initialize the random number generator with seed=0
X = np.random.poisson(a,N) # generate randon numbers from poisson distribution
plt.xlabel(r'$i$',fontsize=16) # set x-label
plt.ylabel(r'$n_i$',fontsize=16) # set y-label
plt.plot(X,',') # plot n_i vs. i (i=1,2,...,N) with dots
plt.show() # draw plots
```

## Compare the distribution with the binomial distribution function

\begin{equation}
P(n)=\frac{a^n e^{-a}}{n!}\tag{D11}
\end{equation}

\begin{equation}
\langle n \rangle=a\tag{D12}
\end{equation}

\begin{equation}
\sigma^2=a\tag{D13}
\end{equation}

```python
def poisson(n,a):
    prob=a**n*np.exp(-a)/math.factorial(n)
    return prob
plt.hist(X,bins=25,normed=True) # plot normalized histgram of X using 25 bins
x = np.arange(M) # generate array of x values from 0 to 100, in intervals of 1
y = np.zeros(M) # generate array of y values, initialized to zero
for i in range(M):
    y[i]=poisson(i,a) # Compute Poisson distribution for n, Eq. (D11)
plt.plot(x,y,color='b') # plot y vs. x with blue line
plt.xlabel(r'$n$',fontsize=16) # set x-label
plt.ylabel(r'$P(n)$',fontsize=16) # set y-label
plt.legend([r'Poisson',r'histgram'], fontsize=16) # set legends
plt.show() # display plots
```


# SVM

**Classification**

Prerequisite：Linearly separable

Basic idea is Max Margin Classifier, we have to find the widest road between class1 and class2.

\begin{align}
max:margin(w,b)
\\st.
\left\{\begin{matrix}
w^Tx_i +b>0,y_i=+1\\
w^Tx_i +b<0,y_i=-1
\end{matrix}\right.
\\
 \Rightarrow y_i(w^{T}x_{i}+b) > 0\\
 \forall i=1,2,...,N
\end{align}

1. For the hard margin:

   * We set margin equals 1, and $y_i(w^Tx_i+b)\geqslant1$;

2. For the soft margin:

   * We allow some noise, so that we can increase the robustness of our system.

   * We introduced slckness variable $\xi$, which represent **loss**.

   * And now the margin function changes to $y_i(w^Tx_i+b)\geqslant1-\xi_i,s.t.xi\geqslant0$

Suppose we have two classes X and O, for class X:

   1. When X is correctly classified, means X locates outside the margin, then $\xi=0$
   2. WHen X is incorrecly classified and locates on the right side of $y_i(w^Tx_i+b)\geqslant0$,then $0<\xi\leqslant0$
   3. When X is on the other side of margin, which means $y_i(w^Tx_i+b)\leqslant0$, then $\xi>1$

Based on these 3 conditions, we get the new target funtion:

\begin{align}
min_{w,b,\xi }:\frac{1}{2}\left \| w \right \|^{2}+C\sum_{i=1}^{N}\xi_i\\
s.t. \left\{\begin{matrix}y_i(w^Tx_i + b)\geqslant 1-\xi_i
\\ \xi_i\geqslant0\\
\end{matrix}\right.
\\
\end{align}

The loss function here called **Hinge Loss**, basically it uses distance to measure **loss**: $\xi$ represents the distance from a point to its corresponding margin $w^Tx+b=1$ when it is miss-classified.

   1. If $w^Tx+b\geqslant1$, $\xi_i=0$, No loss, correct.
   2. If $w^Tx+b<1$, $\xi_i=1-y_i(w^Tx+b)$

So now we have:
   \begin{align}
\xi_i =max\left \{ 0,1-y_i(w^Tx_i + b) \right \}
\end{align}

Base on lagrange duality and KKT conditions, now we get the new target:

\begin{align}
min: \sum_{i=1}^{N}\alpha_i - \frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\alpha_i\alpha_jy_iy_jX_i^TX_j\\
s.t.  \left\{\begin{matrix}
0< \alpha_i<C\\
\sum_{i=1}^{N}\alpha_iy_i=0
\end{matrix}\right.
\\
\end{align}

Optimal Solutions:

$W^* = \sum_{i=1}^{N}x_iy_i\alpha_i$

$b^* = y_i-w^Tx_i$

#### SMO: Sequential Minimal Optimization

Basic idea: SGD and paired $\alpha$

Each time, we only update tow $\alpha$s, the rest of them were treated as constant **Const**.

Now set:

1. $K_{ij} = X_i^TX_j$
2. $f(x_k)=\sum_{i=1}^{N}\alpha_iy_iX_i^Tx_k+b$
3. Error: $E_i = f(x_i)-y_i$
4. $\xi=K_{mm}+K_{nn}-2K_{mn}$

Then we get:

1. $\alpha_2^{new} = \alpha_2^{old} + y_2\frac{(E_1-E_2)}{\xi}$
2. $\alpha_1^{new}=\alpha_1^{old}+y_1y_2(\alpha_2^{old}-\alpha_2^{new})$

```python
import numpy as np
import pandas as pd
import random
from sklearn.model_selection import train_test_split
from matplotlib import pyplot as plt
import seaborn as sns
from sklearn.svm import LinearSVC
from sklearn.metrics import confusion_matrix
```

```python
# input data: feature and target should be in matrix form
# data_mat, label_mat
```

```python
# define a function to randomly choose a paired alpha j
def select_rand_j(i,maximun_number):
    # i is the input index of alpha, and we need to find a j that is different from i
    # maximun_number is the maximun # of samples or rows
    j = i
    while j==i:
        j = int(random.uniform(0,maximun_number))
    return j

```

Based on KKT condition, $\sum_{i=1}^{N}\alpha_iy_i=0$, and each time we only update two $\alpha$

we have $\alpha_1y_1 + \alpha_2y_2 + \sum_{i=3}^{N}\alpha_iy_i = 0$

and $\alpha_1^{new}y_1 + \alpha_2^{new}y_2 = \alpha_1^{old}y_1 + \alpha_2^{old}y_2 = Constant K$

So, for each pair of $\alpha$, there are upper and lower limits for them

```python
def clip_alpha(alpha, H, L):
    if alpha > H:
        alpha = H
    elif alpha < L:
        alpha = L
    return alpha
```

```python
def feature_scale(df):
    # Normalization
    m,n = np.shape(df)
    for i in range(n):
        X_min = np.min(df.iloc[:,i])
        X_max = np.max(df.iloc[:,i])
        if X_max-X_min == 0:
            df.iloc[:,i] = df.iloc[:,i]
        else:
            df.iloc[:,i] = (df.iloc[:,i] - X_min)/(X_max-X_min)
    return df
```

```python
class svc():
    def __init__(self,data_mat,label_mat,C,tolerance,iteration,kernel):
        self.C = 0.6
        self.data_mat = data_mat
        self.label_mat = label_mat
        self.tolerance = 0.001
        self.iteration = 10
        self.kernel = kernel

    # feature scaling

    # now define smo function
    def smo_simple(self):
        # data_mat: feature matrix m*n
        # label_mat: label matrix m*1
        #label_mat = self.X.copy()
        label_mat[label_mat==0]=-1
        # C: hyperparameter
        b = 0
        m, n = np.shape(self.data_mat)
        alpha = np.asmatrix(np.zeros((m,1)))
        iter = 0
        while iter < self.iteration:
            # randomly choose an alpha i
            # i = rand,uniform(0,m)
            alpha_pairs_changed = 0
            for i in range(m):
                wT = np.dot(np.multiply(alpha,self.label_mat).T,self.data_mat)
                f_xi = np.dot(wT,self.data_mat[i,:].T) + b
                E_i = f_xi - float(self.label_mat[i])
                # if the error is less than tolerance, we stop.
                # if E_i * y_i > tolerance, this point locates on the right side
                # if E_i * y_i < tolerance, this point locates on the wrong side, and alpha needs to be updated
                if ((E_i * self.label_mat[i,:] < -self.tolerance) and (alpha[i]<self.C)) or \
                (( E_i * self.label_mat[i] > self.tolerance) and (alpha[i] > 0)):
                    ####?????????????????????????????????
                    # Now we need find its paired alpha j
                    j = select_rand_j(i,m)
                    f_xj = float(np.dot(wT,self.data_mat[j,:].T)) + b
                    E_j = f_xj - float(self.label_mat[j])
                    alpha_j_old = alpha[j].copy()
                    alpha_i_old = alpha[i].copy()
                    #print('now {}{}{}{}'.format(E_i,E_j,alpha_i_old,alpha_j_old))

                    #
                    if (self.label_mat[i] != self.label_mat[j]):
                        # alpha1 - alpha2 = Constant
                        # alpha1 = C - alpha2
                        # 0 < alpha < C
                        L = max(0,alpha[j] - alpha[i])
                        H = min(self.C ,self.C + alpha[j] - alpha[i])
                    else:
                        # alpha1+alpha2 = constant
                        L = max(0,alpha[j] + alpha[i] - self.C)
                        H = min(self.C, alpha[j] + alpha[i])
                    if L==H:

                        continue
                    #print('L{}'.format(L))
                    #print('H{}'.format(H))

                    # 𝜉=𝐾𝑚𝑚+𝐾𝑛𝑛−2𝐾𝑚𝑛
#                     eta = np.dot(self.data_mat[i,:],self.data_mat[i,:].T) + np.dot(self.data_mat[j,:],self.data_mat[j,:].T) -\
#                     2.0 * np.dot(self.data_mat[i,:],self.data_mat[j,:].T)
                    eta = -np.dot(self.data_mat[i,:],self.data_mat[i,:].T) - np.dot(self.data_mat[j,:],self.data_mat[j,:].T) +\
                    2.0 * np.dot(self.data_mat[i,:],self.data_mat[j,:].T)
                    if eta >= 0:
                        #print('eta')
                        continue
                    #print('eat{}'.format(eta))
                    alpha[j] = alpha[j] - self.label_mat[j]*(E_i-E_j)/eta
                    #print('alpha{},label{}'.format(alpha[j],label_mat[j]))
                    alpha[j] = clip_alpha(alpha[j],H,L)
                    #print(alpha[j])

                    if (abs(alpha[j] - alpha_j_old) < 0.00001): #self.tolerance
                        #print('tolerance')
                        continue

                    #𝛼𝑛𝑒𝑤1=𝛼𝑜𝑙𝑑1+𝑦1𝑦2(𝛼𝑜𝑙𝑑2−𝛼𝑛𝑒𝑤2)
                    alpha[i] = alpha[i] + self.label_mat[i]*self.label_mat[j]*(alpha_j_old-alpha[j])

                    # calculate b

                    b1_new = b - E_i - self.label_mat[i]*(alpha[i] - alpha_i_old)*np.dot(self.data_mat[i,:],self.data_mat[i,:].T)\
                                - self.label_mat[j]*(alpha[j]-alpha_j_old)*np.dot(self.data_mat[i,:],self.data_mat[j,:].T)
                    b2_new = b - E_j - self.label_mat[i]*(alpha[i] - alpha_i_old)*np.dot(self.data_mat[i,:],self.data_mat[j,:].T)\
                                - self.label_mat[j]*(alpha[j]-alpha_j_old)*np.dot(self.data_mat[j,:],self.data_mat[j,:].T)
                    if (0 < alpha[i]) and (alpha[i] < self.C):
                        b = b1_new
                    elif (0 < alpha[j]) and (alpha[j]< self.C):
                        b = b2_new
                    else:
                        b = (b1_new + b2_new)/2.0
                    #print(b)
                    alpha_pairs_changed += 1

            if (alpha_pairs_changed == 0): iter += 1
            else: iter = 0

        return b, alpha

```

```python
df = pd.read_csv('ex.csv')
# feature_scale(df.drop('target',axis=1))

```

```python
feature = feature_scale(df.drop('target',axis=1))
target = df['target'].to_frame()
df = pd.concat([feature,target],axis=1)

```

```python

feature = feature.to_numpy()
data_mat = np.asmatrix(feature)
target = target.to_numpy()
label_mat = np.asmatrix(target)

```

```python

positive = df[df['target'].isin([1])]
negative = df[df['target'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['f1'], positive['f2'], s=50, marker='x', label='Positive')
ax.scatter(negative['f1'], negative['f2'], s=50, marker='o', label='Negative')
ax.legend()
plt.show()

```

```python
X = data_mat
y = label_mat
svm = svc(X,y, C = 0.6, tolerance = 0.001,iteration = 10,kernel= 'RBF')
```

```python
b, alphas = svm.smo_simple()
```

```python
w_opt = np.dot(np.multiply(alphas,label_mat).T,data_mat)
```

```python
prediction = data_mat * w_opt.T + b
def sign(x):
    ls=[]
    for i in range(len(x)):
        if x[i] >= 0:
            ls.append(1)
        else:
            ls.append(0)
    return ls
pred = sign(prediction[:,0])
```

```python
b = b[0,0]
w = w_opt

m = np.linspace(0,1)

n = (-b - w[0,0]*m)/w[0,1]
```

```python

positive = df[df['target'].isin([1])]
negative = df[df['target'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['f1'], positive['f2'], s=50, marker='x', label='Positive')
ax.scatter(negative['f1'], negative['f2'], s=50, marker='o', label='Negative')
ax.plot(m,n)
ax.legend()
plt.show()

```

```python
s = LinearSVC()
```

```python
s.fit(X= df.drop('target',axis=1), y=df['target'].to_frame())
```

    /Users/qijin/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py:760: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
      y = column_or_1d(y, warn=True)

    LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,
              intercept_scaling=1, loss='squared_hinge', max_iter=1000,
              multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,
              verbose=0)

```python
b = s.intercept_[0]
w = s.coef_[0]

x = np.linspace(0,1)

y = (-b - w[0]*x)/w[1]
```

```python

positive = df[df['target'].isin([1])]
negative = df[df['target'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['f1'], positive['f2'], s=50, marker='x', label='Positive')
ax.scatter(negative['f1'], negative['f2'], s=50, marker='o', label='Negative')
ax.plot(x,y)
ax.legend()
plt.show()

```

```python
from sklearn import datasets
iris = datasets.load_iris()

feature = pd.DataFrame(iris.data,columns=iris.feature_names)
target = pd.DataFrame(iris.target,columns=['target'])
df = pd.concat([feature,target],axis=1)
df = df[df['target']!=2]
```

```python
sns.countplot(df['target'])
```

```python
df =df.reset_index()
target = df['target'].to_frame()
feature = df.drop('target',axis=1)
type(feature)
```

    pandas.core.frame.DataFrame

```python
feature = feature.to_numpy()
data_mat = np.asmatrix(feature)
target = target.to_numpy()
label_mat = np.asmatrix(target)

```

```python
svm = svc(data_mat = data_mat,label_mat = label_mat, C = 0.6, tolerance = 0.001,iteration = 10,kernel= 'RBF')
```

```python
b,alphas = svm.smo_simple()
w_opt = np.dot(np.multiply(alphas,label_mat).T,data_mat)
```

```python
from sklearn.metrics import confusion_matrix
```

```python
prediction = data_mat * w_opt.T + b
def sign(x):
    ls=[]
    for i in range(len(x)):
        if x[i] >= 0:
            ls.append(1)
        else:
            ls.append(0)
    return ls
pred = sign(prediction[:,0])
```

```python
label_mat[label_mat==-1]=0
np.array(label_mat.T)
```

    array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
            0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
            1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
            1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])

```python
np.array(pred)
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])

```python
confusion_matrix(label_mat,pred)
```

    array([[50,  0],
           [ 0, 50]])