# Linear Regression

Machine learning is about using data to infer patterns.

Linear regression is the basic unit of Deep Learning. In fact personally I prefer to think of it as stacking linear regression modules.

Deep Learning has proven itself over and over again to be far superior to traditional machine learning techiques. It's strength comes from the fact that it can be scalable to large amounts of data.

Some of the most impressive applications are in voice recognition, self driving cars and in chatbots.

If you are finding this tutorial challenging try playing around with https://www.blog.google/topics/machine-learning/now-anyone-can-explore-machine-learning-no-coding-required/ (and make sure you watch the video as well) . For a bit more mathematical take, try reading: https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/. Also watch the

```python
from IPython.display import YouTubeVideo
YouTubeVideo("jpmGxOB2YPo")
```

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline
```

Consider the line $y=2x+3$. See below for a light introduction to plotting and numpy arrays. Numpy arrays are really important to understand. If you have experience with Matlab once you take out the `np.` part it should be familiar to the relevant Matlab function.

```python
x = np.linspace(-5,5,num=100)
y = 2*x+3

plt.plot(x,y)
plt.show()
```

This is how you define a function in Python. Remember to return the value.

```python
def f(w,b):
    """
    We will limit ourselves to a linear function
    """
    return w*x+b

def loss_function(w,b):
    """
    Loss function (squared error)
    Note that w and b are the ONLY variables
    """
    e = y - f(w,b)
    return np.mean(np.square(e))
```

## Find the function

Given the observation $x,y$ the goal is to find the best parameters, $w,b$ that fits the data. Ignore the fact that you can solve this as an equation. Most of the time the data isn't as linear or noiseless as what is being presented here.

The most naiive way of finding $w$ and $b$ is to do a grid search over $w$ and $b$ for the smallest loss function value. In this case the loss function is: $$\frac{1}{N}\sum_{i=1}^N \xi_i^2$$ where, $\xi_i = y_i - f(x_i|w,b)$ and $f(x_i|w,b)=wx_i+b$.

Another way of stating the above is that we need to find the best fitting function $f(x_i|w,b)$ such that we minimising the __loss function__ by varying $w$ and $b$. An illustration is shown below:

```python
y
```

    array([ -7.        ,  -6.7979798 ,  -6.5959596 ,  -6.39393939,
            -6.19191919,  -5.98989899,  -5.78787879,  -5.58585859,
            -5.38383838,  -5.18181818,  -4.97979798,  -4.77777778,
            -4.57575758,  -4.37373737,  -4.17171717,  -3.96969697,
            -3.76767677,  -3.56565657,  -3.36363636,  -3.16161616,
            -2.95959596,  -2.75757576,  -2.55555556,  -2.35353535,
            -2.15151515,  -1.94949495,  -1.74747475,  -1.54545455,
            -1.34343434,  -1.14141414,  -0.93939394,  -0.73737374,
            -0.53535354,  -0.33333333,  -0.13131313,   0.07070707,
             0.27272727,   0.47474747,   0.67676768,   0.87878788,
             1.08080808,   1.28282828,   1.48484848,   1.68686869,
             1.88888889,   2.09090909,   2.29292929,   2.49494949,
             2.6969697 ,   2.8989899 ,   3.1010101 ,   3.3030303 ,
             3.50505051,   3.70707071,   3.90909091,   4.11111111,
             4.31313131,   4.51515152,   4.71717172,   4.91919192,
             5.12121212,   5.32323232,   5.52525253,   5.72727273,
             5.92929293,   6.13131313,   6.33333333,   6.53535354,
             6.73737374,   6.93939394,   7.14141414,   7.34343434,
             7.54545455,   7.74747475,   7.94949495,   8.15151515,
             8.35353535,   8.55555556,   8.75757576,   8.95959596,
             9.16161616,   9.36363636,   9.56565657,   9.76767677,
             9.96969697,  10.17171717,  10.37373737,  10.57575758,
            10.77777778,  10.97979798,  11.18181818,  11.38383838,
            11.58585859,  11.78787879,  11.98989899,  12.19191919,
            12.39393939,  12.5959596 ,  12.7979798 ,  13.        ])

```python
x
```

    array([-5.        , -4.8989899 , -4.7979798 , -4.6969697 , -4.5959596 ,
           -4.49494949, -4.39393939, -4.29292929, -4.19191919, -4.09090909,
           -3.98989899, -3.88888889, -3.78787879, -3.68686869, -3.58585859,
           -3.48484848, -3.38383838, -3.28282828, -3.18181818, -3.08080808,
           -2.97979798, -2.87878788, -2.77777778, -2.67676768, -2.57575758,
           -2.47474747, -2.37373737, -2.27272727, -2.17171717, -2.07070707,
           -1.96969697, -1.86868687, -1.76767677, -1.66666667, -1.56565657,
           -1.46464646, -1.36363636, -1.26262626, -1.16161616, -1.06060606,
           -0.95959596, -0.85858586, -0.75757576, -0.65656566, -0.55555556,
           -0.45454545, -0.35353535, -0.25252525, -0.15151515, -0.05050505,
            0.05050505,  0.15151515,  0.25252525,  0.35353535,  0.45454545,
            0.55555556,  0.65656566,  0.75757576,  0.85858586,  0.95959596,
            1.06060606,  1.16161616,  1.26262626,  1.36363636,  1.46464646,
            1.56565657,  1.66666667,  1.76767677,  1.86868687,  1.96969697,
            2.07070707,  2.17171717,  2.27272727,  2.37373737,  2.47474747,
            2.57575758,  2.67676768,  2.77777778,  2.87878788,  2.97979798,
            3.08080808,  3.18181818,  3.28282828,  3.38383838,  3.48484848,
            3.58585859,  3.68686869,  3.78787879,  3.88888889,  3.98989899,
            4.09090909,  4.19191919,  4.29292929,  4.39393939,  4.49494949,
            4.5959596 ,  4.6969697 ,  4.7979798 ,  4.8989899 ,  5.        ])

```python
loss_function(2,2)
```

    1.0

```python
loss_function(2,3)
```

    0.0

```python
w_try, b_try = np.meshgrid(np.linspace(-5,5), np.linspace(-5,5))
```

```python
w_try
```

    array([[-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           ...,
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ],
           [-5.        , -4.79591837, -4.59183673, ...,  4.59183673,
             4.79591837,  5.        ]])

```python
b_try
```

    array([[-5.        , -5.        , -5.        , ..., -5.        ,
            -5.        , -5.        ],
           [-4.79591837, -4.79591837, -4.79591837, ..., -4.79591837,
            -4.79591837, -4.79591837],
           [-4.59183673, -4.59183673, -4.59183673, ..., -4.59183673,
            -4.59183673, -4.59183673],
           ...,
           [ 4.59183673,  4.59183673,  4.59183673, ...,  4.59183673,
             4.59183673,  4.59183673],
           [ 4.79591837,  4.79591837,  4.79591837, ...,  4.79591837,
             4.79591837,  4.79591837],
           [ 5.        ,  5.        ,  5.        , ...,  5.        ,
             5.        ,  5.        ]])

```python
w_try = np.linspace(-4,4,num=100)
b_try = np.linspace(-4,4,num=100)

w_try, b_try = np.meshgrid(w_try,b_try)
e_obs = np.zeros((100,100))

for i in range(100):
    for j in range(100):
        e_obs[i,j] = loss_function(w_try[i,j],b_try[i,j])

plt.pcolormesh(w_try, b_try, np.log(e_obs)) #using log here helps visualise better, you may take it out
plt.xlabel('w')
plt.ylabel('b')
plt.title('Mean Squared Error')
plt.show()
```

You can see that the error in the plot is minimised at $w=2$ and $b=3$.

```python
e_obs.max()
```

    355.06060606060601

```python
e_obs.min()
```

    0.0043879944910451526

Algorithms try to be a bit smarter by following the gradients to find the minimum. Remember we can only influence the error via the weights (and bias) `w,b`.

## Maths (optional - no really)
\begin{align}
\mathcal{L} & = \sum_{i=1}^N (y_i-f(x_i,w,b))^2 \\
\frac{\delta\mathcal{L}}{\delta w} & = \sum_{i=1}^N 2(y_i-f(x_i,w,b))\frac{\delta f(x_i,w,b)}{\delta w} \\
& = \sum_{i=1}^N 2\xi_i\frac{\delta f(x_i,w,b)}{\delta w}
\end{align}
where $\xi_i$ is the error term $y_i-f(x_i,w,b)$ and
$$
\frac{\delta f(x_i,w,b)}{\delta w} = x_i
$$

Similar expression can be found for $\frac{\delta\mathcal{L}}{\delta b}$ (exercise).

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
from IPython.display import YouTubeVideo
YouTubeVideo("EvnpoUTXA0E")
```

```python
from sklearn.linear_model import LinearRegression
```

Scikit learn is another Machine Learning that is quite powerful on itself. However it may not scale to millions of data points same way that DL does.

## Noisy Data
We see how to find the weights using scikit learn below. However, the data isn't as perfect as before. We add a tiny bit of noise.

```python
x = np.random.randn(10,1)
y = 2*x+3 + 0.1*np.random.randn(10,1)

plt.scatter(x,y)
plt.show()
```

```python
x
```

    array([[-0.0710438 ],
           [-1.63204099],
           [ 0.61393732],
           [-0.72602171],
           [ 0.56296216],
           [-1.19009865],
           [ 0.21120869],
           [ 1.11886921],
           [-0.21082568],
           [-2.44257088]])

```python
model = LinearRegression() #
model.fit(x,y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

```python
model.coef_
```

    array([[ 2.00976455]])

```python
model.intercept_
```

    array([ 3.03645155])

```python
model.predict(20)
```

    array([[ 43.2317426]])

## Note:
1. `model.fit(...)` and `model.predict` will be two of the most important functions you will use all throughout. Worth remembering!

```python
2*20+3
```

    43

```python
x_test = np.linspace(-3,3)
y_pred = model.predict(x_test[:,None])

plt.scatter(x,y)
plt.plot(x_test,y_pred,'r')
plt.legend(['Predicted line','Observed data'])
plt.show()
```

## A more complicated function
Remember that when we say linear regression it mean linear in the parmaters $w, b$ not $x$! See illustration below:

\begin{align}
y = b +w^T X \\
X = [x\quad x^2\quad x^3] \\
y = b +wx +w_2 x^2 +w_3 x^3 \\
\end{align}
The shapes of the matrices are:
- X = NxD matrix
- y = Nx1 vector
- w = Dx1 vector

Note in the example below that there is no $x^2$ term. i.e. it's coefficient is zero!

```python
x = np.linspace(-5,5,num=100)[:,None]
y = -0.5 + 2.2*x +0.3*x**3+ 2*np.random.randn(100,1)

plt.plot(x,y)
plt.show()
```

```python
x.shape
```

    (100, 1)

```python
x_new = np.hstack([x,x**2,x**3,x**4])
x_new.shape
```

    (100, 4)

```python
x.shape
```

    (100, 1)

```python
model = LinearRegression()
model.fit(x_new,y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

```python
model.coef_
```

    array([[  2.22229669e+00,  -1.39017445e-02,   2.96369412e-01,
              2.16217922e-03]])

## Note:
- The coefficient of $x^2$ and $x^4$ is really close to zero.

```python
model.intercept_
```

    array([-0.65870273])

```python
y_pred = model.predict(x_new)

plt.scatter(x,y)
plt.plot(x_new[:,0],y_pred,'r')
plt.legend(['Predicted line','Observed data'])
plt.show()
```

```python
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=4,include_bias=False)
x_new2 = poly.fit_transform(x)
x_new2[:2]
```

    array([[  -5.        ,   25.        , -125.        ,  625.        ],
           [  -4.8989899 ,   24.00010203, -117.57625742,  576.00489747]])

```python
x_new.shape
```

    (100, 4)

```python
x_new[:2]
```

    array([[  -5.        ,   25.        , -125.        ,  625.        ],
           [  -4.8989899 ,   24.00010203, -117.57625742,  576.00489747]])

```python

```

deepschool.io/Lesson 0 - LinRegression.ipynb


# Plotting

```python
from IPython.display import Latex
from matplotlib.pyplot import *
from numpy import *
from sympy import latex, simplify, solve, symbols
```

```python
x = symbols("x")
X = linspace(-4, 4.05, 128)
F = [x**2, x + 4]
s = r'$$\begin{align}' + '\n'
for i, f in enumerate(F):
    s += f"f_{i+1}={latex(F[i])}." + '\n' + (r'\\' if len(F) > i+1 else r'\end{align}$$')
Latex(s)
```

$$\begin{align}
f_1=x^{2}.
\\f_2=x + 4.
\end{align}$$

```python
xlabel("$x$")
ylabel("$y$")
grid(True)
axis('equal')
axis([-5, 5, 0, 16])
axis(False)
labels = list()
for f in F:
    Y = [f.evalf(subs={x:x0}) for x0 in X]
    labels.append(f"${latex(f)}$")
    plot(X,Y)
legend(labels=labels)
S = solve(F[0]-F[1])
for i in range(len(S)):
    annotate(rf"$\left({latex(S[i])}, {latex(F[i].subs({x:S[i]}))}\right)$",
             xy=(S[i].evalf(), F[i].evalf(subs={x:S[i]})))
Latex(f'$$S={latex(S)}.$$')
```

$$S=\left[ \frac{1}{2} - \frac{\sqrt{17}}{2}, \  \frac{1}{2} + \frac{\sqrt{17}}{2}\right].$$

```python
show()
```

```python
public(plt)
```

```python

```


```python
import numpy as np
from sympy import *
from sympy.calculus.util import continuous_domain
import pandas as pd
import copy

init_printing(use_unicode=True)
```

```python
class Ponto:
    def __init__(self,x, fx, tipo):
        self.x = x
        self.fx = fx
        self.tipo = tipo

    def __lt__(self, other):
        return self.x < other.x

def acha_intervalo_newton(f, symbol):
    f_ = diff(f)
    f__ = diff(f_)
    _inf = limit(f,symbol,-oo).evalf()
    Pontos = []
    Pontos.append(Ponto(-oo, _inf,'- infinito: '))
    inf_mais = limit(f,symbol,+oo).evalf()
    Pontos.append(Ponto(oo, inf_mais,'+ infinito: '))

    f_zeros = solve(f_)

    for zero in f_zeros:
        if str(type(zero)) != "<class 'sympy.core.add.Add'>":
            fvalue = f.subs(symbol, zero.evalf())
            if fvalue < 0:
                Pontos.append(Ponto(zero.evalf(), fvalue,'minimo local: '))
            else:
                Pontos.append(Ponto(zero.evalf(), fvalue,'maximo local: '))

    f__zeros = solve(f__)
    for zero in f__zeros:
        if str(type(zero)) != "<class 'sympy.core.add.Add'>":
            fvalue = f.subs(symbol, zero.evalf())
            Pontos.append(Ponto(zero.evalf(), fvalue, 'mudanca concav: '))
    df = pd.DataFrame(columns=['Tipo','x','f(x)'])
    Pontos.sort()
    for i, ponto in enumerate(Pontos):
        df.loc[i, 'x'] = ponto.x
        df.loc[i, 'f(x)'] = ponto.fx
        df.loc[i, 'Tipo'] = ponto.tipo

    return df

def visual_check(f,interval,symbol):
    plot(f, (symbol, interval.args[0], interval.args[1]))
    inp = input('Vazio para confirmar')
    if inp == '':
        return True
    else:
        return False

def check_continuity(f, interval, symbol, visual=False):
    cont_domain = continuous_domain(f,symbol,interval)
    if visual:
        check = visual_check(f,interval,symbol)
        if not check:
            raise NameError('Aborted at visual check')
    if len(cont_domain.args) == 1:
        return True
    elif type(cont_domain) == type(Interval(0,1)):
        return True
    else:
        return False

def check_maximum(f,interval,symbol):
    possiveis_max = []
    borda1 = (f.subs(symbol,interval.args[0]).evalf())
    borda2 = (f.subs(symbol,interval.args[1]).evalf())

    possiveis_max.append(borda1)
    possiveis_max.append(borda2)
    f_ = diff(f)
    zeros = solve(f_)
    for zero in zeros:
        if str(type(zero)) == "<class 'sympy.core.add.Add'>":
            zero = zero.evalf()
            if zero in interval:
                possiveis_max.append(f.subs(symbol,zero).evalf())

    possiveis_sem_complex = []
    for ele in possiveis_max:
        if str(type(ele)) != "<class 'sympy.core.add.Add'>":
            possiveis_sem_complex.append(float(ele))
    return possiveis_sem_complex

def check_bolz(f, intervalo, symbol):
    t0 = float(intervalo.args[0])
    t1 = float(intervalo.args[1])
    boz = f.subs(symbol,t0)*f.subs(symbol,t1) < 0
    return boz

def check_monotonia(f, intervalo, symbol):
    f_ = diff(f)
    maximos = check_maximum(f_,intervalo, symbol)
    f_max = np.max(maximos)
    f_mim = np.min(maximos)
    if f_max * f_mim > 0:
        if f_max > 0:
            return True, 1
        else:
            return True, -1
    else:
        return False, False

def muda_concav(f,intervalo):
    f_ = diff(f)
    f__ = diff(f_)
    sols = solve(f__)
    for sol in sols:
        if sol in intervalo:
            return False
    return True

def acha_intervalo_aprx_suc(phi):
    ''' Da uma dica pra achar o intervalo'''
    phi_ = diff(phi)
    sol1 = solve(phi_)
    sol0 = solve(phi_ + 1)
    sol2 = solve(phi_ - 1)

    if len(sol1) != len(sol0) or len(sol0) != len(sol2):
        print(sol0)
        print(sol1)
        print(sol2)
    else:
        for i in range(len(sol0)):
            print(sol0[i].evalf(), '|', sol1[i].evalf(), '|', sol2[i].evalf())

def check_intervalo_aprx_suc(f,phi, symbol, intervalo):
    '''Verifica se o intervalo serve'''
    t0 = float(intervalo.args[0])
    t1 = float(intervalo.args[1])
    fcond = f.subs(symbol,t0)*f.subs(symbol,t1) < 0
    phi_ = abs(diff(phi))
    phi_cond = phi_.subs(symbol,t0) < 1 and phi_.subs(symbol,t1) < 1
    if not phi_cond:
        print('Intervalo invalido, modulo de phi_ nao eh menor que 1')
    if not fcond:
        print('raizes tao do msm lado, cade bolzano???')
    if fcond and phi_cond:
        print('Intervalo valido')
    return fcond and phi_cond

class aproximacoes_sucessivas:
    ''''Classe para executar todas as variacoes de aproximacoes sucessivas'''
    def __init__(self,f, phi,intervalo, erro, symbol, visual = False):
        self.erro = erro
        self.phi = phi
        self.phi_ = diff(phi)
        self.intervalo = intervalo
        self.symbol = symbol
        self.visual = visual
        self.f = f

        inter_check = check_intervalo_aprx_suc(f, phi, symbol, intervalo)
        if not inter_check:
            print('intervalo incorreto')

        condicoes_basicas = self.__verifica_condicoes()
        if not condicoes_basicas:
            print('condicoes basicas nao são suficientes pelo(s) motivo(s) acima')

        convergente_em_phi = self.__converg_phi()
        if not convergente_em_phi:
            print('Não podemos garantir convergencia em phi, pelo(s) motivo(s) acima')

        self.x0 = self.__escolhe_x0()
        self.iters = 100
        self.df = pd.DataFrame()

#################################################################################################
#metodo monotonic
    def exec_monotonic(self):
        print()
        print('Exececutando monotonic')
        phi = self.phi
        intervalo = self.intervalo
        symbol = self.symbol
        mono = check_monotonia(phi,intervalo,symbol)
        if mono[1] < 0:
            print('Metodo invalido, sequencia nao monotonica')
            return
        column1, column2 = self.monotonic_type()

        self.df[column1] = np.nan
        self.df[column2] = np.nan
        self.df.loc[0, column2] = self.meio_intervalo
        self.resps = [self.x0]
        self.iters_ = 0
        self.iters = 100
        self.__recursi(column1,column2,'monotonic_criteria')

    def monotonic_criteria(self):
        if np.abs(self.resps[-1] - self.resps[-2]) < 2*self.erro:
            return True

    def monotonic_type(self):
        symbol = self.symbol
        phi = self.phi
        resps = self.resps
        resp = float(phi.subs(symbol , resps[-1]))
        resps.append(resp)
        self.sinal = (resps[-1]-resps[-2])
        self.sinal = self.sinal/abs(self.sinal)
        if self.sinal > 0:
            print('monotonica crescente')
            return 'Xn_Mono_cres', 'Erro_Xn_Mono_cres'
        else:
            print('monotonica decrescente')
            return 'Xn_Mono_decres', 'Erro_Xn_Mono_decres'

#################################################################################################
#metodo alternado
    def exec_alternado(self):
        print()
        print('Exececutando alternado')
        phi = self.phi
        intervalo = self.intervalo
        symbol = self.symbol
        mono = check_monotonia(phi,intervalo,symbol)
        if mono[1] > 0:
            print('Método invalido, sequencia nao alternada')
            return False
        if not self.M/(1-self.M) > 1:
            print('parada invalida, M/(1-M) < 1')
            return False
        column1, column2 = 'Xn_Alternado', 'Erro_Xn_Alternado'
        self.df[column1] = np.nan
        self.df[column2] = np.nan
        self.df.loc[0, column2] = self.meio_intervalo
        self.resps = [self.x0]
        self.iters_ = 0
        self.iters = 100
        self.__recursi(column1, column2, 'alternado_criteria')

    def alternado_criteria(self):
        if np.abs(self.resps[-1] - self.resps[-2]) < self.erro:
            return True
        else:
            return False

#################################################################################################
#metodo parada (I)
    def exec_parada_I(self):
        print()
        print('Exececutando parada I')
        if self.M >=1:
            print('Metodo invalido, M = {} > 1'.format(self.M))
            return
        if not self.M/(1-self.M) < 1:
            print('metodo invalido, M/(1-M) > 1')
            return
        column1, column2 = 'Xn_Parada_I', 'Erro_Xn_Parada_I'
        self.df[column1] = np.nan
        self.df[column2] = np.nan
        self.df.loc[0, column2] = self.meio_intervalo
        self.resps = [self.x0]
        self.iters_ = 0
        self.iters = 100
        self.__recursi(column1, column2, 'parada_I_criteria')

    def parada_I_criteria(self):
        M = self.M
        if (M/(1-M))*np.abs(self.resps[-1] - self.resps[-2]) < self.erro:
            return True

#################################################################################################
#metodo padrao
    def exec_padrao(self):
        print()
        print('Exececutando padrao')
        n = self.numero_de_passos()
        column1, column2 = 'Xn_Padrao', 'Erro_Xn_Padrao'
        self.df[column1] = np.nan
        self.df[column2] = np.nan
        self.df.loc[0, column2] = self.meio_intervalo
        self.resps = [self.x0]
        self.iters_ = 0
        self.iters = n
        print('Serao , ', n, ' iteracoes')
        self.__recursi(column1, column2, 'padrao_criteria')

    def padrao_criteria(self):
        if self.iters <= 0:
            return True

    def numero_de_passos(self):
        c = float(self.meio_intervalo)
        M = float(self.M)
        erro = self.erro
        n = np.ceil(np.log(erro/c)/np.log(M))
        return n

################################################################################################
#funcs auxiliares
    def __escolhe_x0(self):
        intervalo = self.intervalo
        symbol = self.symbol
        phi = self.phi

        a = intervalo.args[0]
        b = intervalo.args[1]
        c = float((a+b)/2)
        phi_de_c = float(phi.subs(symbol, c))
        if phi_de_c < c:
            x0 = a
        if phi_de_c > c:
            x0 = b
        if phi_de_c == c:
            x0 = c
        self.meio_intervalo = (b-a)/2
        return float(x0)

    def __verifica_condicoes(self):
        phi = self.phi
        f = self.f
        symbol = self.symbol
        intervalo = self.intervalo

        phicontinua = check_continuity(phi, intervalo, symbol, self.visual)
        fcontinua = check_continuity(f, intervalo, symbol, self.visual)
        fmonotonica = check_monotonia(f, intervalo, symbol)[0]
        fbolz = check_bolz(f, intervalo, symbol)

        if not phicontinua:
            print('Phi não continua no intervalo')
        if not fcontinua:
            print('f não continua no intervalo')
        if not fmonotonica:
            print('f não monotonica no intervalo')
        if not fbolz:
            print('f não satifaz bolzano')
        return phicontinua and fcontinua and fmonotonica and fbolz

    def __converg_phi(self):
        intervalo = self.intervalo
        symbol = self.symbol
        phi = self.phi
        phi_ = self.phi_
        visual = self.visual
        phicontinua = check_continuity(phi,intervalo,symbol,visual)
        phi_continua = check_continuity(phi_,intervalo,symbol,visual)
        phi_max = np.max(np.abs(np.array(check_maximum(phi_,intervalo,symbol))))
        phiarr = np.array(check_maximum(phi,intervalo,symbol))
        phimax = np.max(phiarr)
        phimim = np.min(phiarr)
        self.M = float(phi_max)
        phimax_intervalo = True
        phimin_intervalo = True
        phi_maxmod = True
        if not phicontinua:
            print('Phi não é continua no intervalo')
        if not phi_continua:
            print("Phi' não é continua no intervalo")
        if phimax >= intervalo.args[1]:
            phimax_intervalo = False
            print('Nao podemos garantir que Xn+1 estara no intervalo Phimax <= Intmax'
                  ' = {:.2f} < {:.2f}'.format(float(phimax),float(intervalo.args[1])))
        if phimim <= intervalo.args[0]:
            phimin_intervalo = False
            print('Nao podemos garantir que Xn+1 estara no intervalo Phimin'
                  ' <= Intmin = {:.2f} < {:.2f}'.format(float(phimim),float(intervalo.args[0])))
        if phi_max >= 1:
            phi_maxmod = False
            print('Nao podemos mostrar que Phi é uma n-contração, |Phi_| = {} >= 1'.format(phi_max))
        return phicontinua and phi_continua and phimax_intervalo and phimin_intervalo and phi_maxmod

    def exec(self):
        self.exec_padrao()#tanto faz
        self.exec_alternado()
        self.exec_parada_I()
        self.exec_monotonic()#xn+1

    def __calc_erro(self, column):
        M = self.M
        if column == 'Erro_Xn_Padrao':
            return np.abs(self.resps[-2]-self.resps[-1])*(M/(1-M))
        if column == 'Erro_Xn_Alternado':
            return np.abs(self.resps[-2]-self.resps[-1])
        if column == 'Erro_Xn_Parada_I':
            return (M/(1-M))*np.abs(self.resps[-1] - self.resps[-2])
        if column == 'Erro_Xn_Mono_cres' or column == 'Erro_Xn_Mono_decres':
            return np.abs(self.resps[-1] - self.resps[-2])

    def __recursi(self, column1, columns2, stop_criteria):
        symbol = self.symbol
        phi = self.phi
        resps = self.resps
        resp = float(phi.subs(symbol , resps[-1]))
        resps.append(resp)

        self.df.loc[self.iters_ , column1] = resps[-2]
        self.iters_ += 1
        self.df.loc[self.iters_ , column1] = resp
        self.df.loc[self.iters_ , columns2] = self.__calc_erro(columns2)
        self.iters -= 1
        validation = getattr(self, stop_criteria)
        result = validation()
        if result:
            print('Metodo '+ column1 + ' achou')
            print()
            if stop_criteria == 'monotonic_criteria':
                self.df.loc[self.iters_ , column1] = np.nan
                self.df.loc[self.iters_ , columns2] = np.nan
            return True
        elif self.iters == 0:
            print('esgotou iteracoes')
            print()
            return False
        else:
            self.__recursi(column1, columns2, stop_criteria)

class Newton:
    def __init__(self,f, symbol, intervalo, erro, visual = False):
        self.f = f
        f_ = diff(f)
        self.f_ = f_
        f__ = diff(f_)
        self.f__ = f__
        self.erro = erro
        self.visual = visual
        self.phi = symbol - f/(diff(f))
        self.phi_ = diff(self.phi)
        self.intervalo = intervalo
        self.symbol = symbol

        inter_check = self.__verifica_intervalo()
        if not inter_check:
            print('Intervalo contem irregularidades')

        self.x0 = self.__escolhex0()
        self.df = pd.DataFrame()

    def __verifica_intervalo(self):
        f = self.f
        intervalo = self.intervalo
        symbol = self.symbol

        bolz = check_bolz(f, intervalo, symbol)
        mono = check_monotonia(f, intervalo, symbol)[0]
        concav = muda_concav(f,intervalo)
        fcont = check_continuity(f,intervalo,symbol,self.visual)

        if not bolz:
            print('intervalo nao satisfaz bolzano')

        if not mono:
            print('Intervalo invalido, f nao e monotonica')

        if not concav:
            print('Intervalo invalido, f muda concavidade')

        if not fcont:
            print('Intervalo invalido, f n eh continua')

        return bolz and mono and concav and fcont

    def __escolhex0(self):
        f__ = self.f__
        f_ = self.f_
        symbol = self.symbol
        intervalo = self.intervalo
        a = intervalo.args[0]
        b = intervalo.args[1]
        c = (a+b)/2
        sinalf_ = f_.subs(symbol, c).evalf()
        sinalf__ = f__.subs(symbol, c).evalf()
        self.meio_intervalo = (b-a)/2

        if sinalf__*sinalf_ > 0:
            x0 = b
        if sinalf__*sinalf_ < 0:
            x0 = a
        return x0

    def __tipo_conv(self):
        phi = self.phi
        symbol = self.symbol
        phi_list = [self.x0, phi.subs(symbol,self.x0)]
        phi_list.append(phi.subs(symbol,phi_list[-1]))
        sinal1 = phi_list[1] - phi_list[0]
        sinal2 = phi_list[2] - phi_list[1]

        if sinal1*sinal2 > 0:
            self.tipo = 'monotonica'
            self.mono_tipo = self.__check_cres()
        else:
            print('Phi eh  oscilante')
            self.tipo = 'oscilante'
    #     phi = self.phi
    #     intervalo = self.intervalo
    #     symbol = self.symbol
    #
    #     mono = check_monotonia(phi, intervalo, symbol)
    #     if not mono[0]:
    #         print('phi_ troca de sinal no intervalo, n eh possivel determinar '
    #               'se a sequencia sera monotonica ou alternante')
    #         print()
    #         print('Fazendo que nem o prof, assumindo monotonica e andando com 2Erro')
    #         self.__estado_pelo_ponto()
    #
    #     elif mono[0]:
    #         if mono[1] > 0:
    #
    #         if mono[1] < 0:
    #
    #
    # def __estado_pelo_ponto(self):
    #     # phi_ = self.phi_
    #     # a = phi_.subs(self.symbol, self.x0).evalf()
    #     # if a > 0:
    #     self.tipo = 'monotonica'
    #     self.mono_tipo = self.__check_cres()
    #     # elif a < 0:
    #     #     print('Phi eh  oscilante')
    #     #     self.tipo = 'oscilante'

    def __check_cres(self):
        symbol = self.symbol
        phi = self.phi
        resps = [self.x0]
        resp = float(phi.subs(symbol , resps[-1]))
        resps.append(resp)
        self.sinal = (resps[-1]-resps[-2])
        self.sinal = self.sinal/abs(self.sinal)
        if self.sinal > 0:
            print('Phi eh monotonica crescente')
            return 'crescente'
        else:
            print('Phi eh monotonica decrescente')
            return 'decrescente'

    def exec(self):
        print()
        print('Exececutando Newton')
        self.sinal = 0

        self.__tipo_conv()
        self.df['Erro'] = np.nan
        self.df.loc[0,'Erro'] = self.meio_intervalo
        self.df['Xn'] = np.nan
        if self.tipo == 'oscilante':
            column2 = 'phi(Xn)'
            self.df.loc[0, 'Erro'] = self.meio_intervalo
        else:

            if self.mono_tipo == 'crescente':
                self.df['Xn + 2E'] = np.nan
                self.df['f(Xn + 2E)'] = np.nan
                column2 = 'phi(Xn+2E)'
            else:
                self.df['f(Xn - 2E)'] = np.nan
                column2 = 'phi(Xn-2E)'

        self.df[column2] = np.nan
        self.resps = [self.x0]
        self.iters_ = 0
        self.iters = 100
        self.__recursi(column2)
        self.__finalize()

    def __finalize(self):
        resps = self.resps
        if self.tipo == 'oscilante':
            _x_ = (resps[-1] + resps[-2])/2
            print('xbarra = {}'.format(_x_))
        elif self.mono_tipo == 'crescente':
            _x_ = (resps[-2] + self.erro)
            print('xbarra = {}'.format(_x_))
        else:
            _x_ = (resps[-2] - self.erro)
            print('xbarra = {}'.format(_x_))
        self.xbarra = _x_

    def __calc_erro(self):
        return abs(self.resps[-2]-self.resps[-1])/2

    def stop_criteria(self):
        if self.__calc_erro() < self.erro:
            return True
        else:
            return False

    def __recursi(self, columns2):
        symbol = self.symbol
        phi = self.phi
        resps = self.resps
        resp = float(phi.subs(symbol , resps[-1] + self.sinal*2*self.erro))
        resps.append(resp)
        result = self.stop_criteria()
        if result:
            print('Metodo de newton achou')
            print()
            return True
        elif self.iters == 0:
            print('esgotou iteracoes')
            print()
            return False
        else:
            self.df.loc[self.iters_ , 'Xn'] = resps[-2]

            if self.tipo == 'monotonica':
                if self.mono_tipo == 'crescente':
                    self.df.loc[self.iters_ ,'Xn + 2E'] = resps[-2] + 2*self.erro
                    self.df.loc[self.iters_ ,'f(Xn + 2E)'] = self.f.subs(self.symbol, resps[-2] + 2*self.erro).evalf()
                else:
                    self.df.loc[self.iters_ ,'Xn - 2E'] = resps[-2] - 2*self.erro
                    self.df.loc[self.iters_ ,'f(Xn - 2E)'] = self.f.subs(self.symbol, resps[-2] - 2*self.erro).evalf()

            self.df.loc[self.iters_ , columns2] = resps[-1]
            self.iters_ += 1
            self.df.loc[self.iters_ , 'Xn'] = resp
            self.df.loc[self.iters_ , 'Erro'] = self.__calc_erro()

            if self.tipo == 'monotonica':
                if self.mono_tipo == 'crescente':
                    self.df.loc[self.iters_ ,'Xn + 2E'] = resps[-1] + 2*self.erro
                    self.df.loc[self.iters_ ,'f(Xn + 2E)'] = self.f.subs(self.symbol, resps[-1] + 2*self.erro).evalf()
                else:
                    self.df.loc[self.iters_ ,'Xn - 2E'] = resps[-1] - 2*self.erro
                    self.df.loc[self.iters_ ,'f(Xn - 2E)'] = self.f.subs(self.symbol, resps[-1] - 2*self.erro).evalf()

            self.iters -= 1
            self.__recursi(columns2)

class Bisseccao:
    def __init__(self,f, symbol, intervalo, erro, visual = False):
        self.f = f
        self.symbol = symbol
        self.intervalo = intervalo
        self.erro = erro
        self.visual = visual
        self.df = pd.DataFrame(columns=['a','c','b','erro'])

        self.numero_de_passos()
        self.iters = 0
        self.check_inter()
        self.c = 0

    def recursi(self):
        a = self.a
        b = self.b
        self.df.loc[self.iters, 'a'] = a
        self.df.loc[self.iters, 'b'] = b
        self.c = (self.a + self.b)/2
        c = self.c
        self.df.loc[self.iters, 'c'] = c
        self.df.loc[self.iters, 'erro'] = b-c
        f = self.f
        symbol = self.symbol
        if f.subs(symbol, a) * f.subs(symbol, c) < 0:
            self.b = c
        else:
            if f.subs(symbol, c) * f.subs(symbol, b) < 0:
                self.a = c
            else:
                self.xbarra = c
                return
        self.n -= 1
        self.iters += 1
        if self.n <= 0:
            return
        else:
            self.recursi()

    def exec(self):
        self.numero_de_passos()
        self.recursi()

    def numero_de_passos(self):
        intervalo = self.intervalo
        a = float(intervalo.args[0])
        b = float(intervalo.args[1])
        self.a = a
        self.b = b
        n = np.ceil(np.log2((b-a)/(2*self.erro)))
        self.n = n + 1

    def check_inter(self):
        fcont = check_continuity(self.f,self.intervalo,self.symbol,self.visual)

        if not fcont:
            print('Intervalo invalido, f nao eh continua')

        return fcont

class Secantes:
    def __init__(self, f , symbols, x, y, erro):
        ''''x e y sao valores iniciais, e os symbols tao na ordem obvia'''
        self.symbol1 = symbols[0]
        self.symbol2 = symbols[1]
        symbol1 = self.symbol1
        symbol2 = self.symbol2
        self.erro = erro
        self.f = f
        self.x0 = y
        self.x1 = x
        self.df = pd.DataFrame()

        self.phi = symbol1 - (f*(symbol1-symbol2)/(f-f.subs(symbol1,symbol2)))

    def exec(self):
        self.df['Xn'] = np.nan
        self.df['Erro'] = np.nan
        self.resps = [self.x0, self.x1]
        self.x = self.resps[1]
        self.y = self.resps[0]
        self.iters = 100
        self.iters_ = 1
        self.df.loc[0, 'Xn'] = self.x0
        self.df.loc[1, 'Xn'] = self.x1
        self.df.loc[1, 'Erro'] = abs(self.x1 - self.x0)
        self.recursi()

    def recursi(self):
        x = self.x
        y = self.y
        resps = self.resps
        symbol1 = self.symbol1
        symbol2 = self.symbol2
        phi = self.phi
        z = phi.subs(symbol1,x).subs(symbol2,y).evalf()
        resps.append(z)
        self.y = x
        self.x = z
        self.iters -= 1
        self.iters_ += 1
        self.df.loc[self.iters_, 'Erro'] = abs(resps[-1]- resps[-2])
        self.df.loc[self.iters_, 'Xn'] = z
        if abs(resps[-2]- resps[-1]) < self.erro:# and abs(resps[-2]- resps[-3]) < self.erro:
            print(self.iters_)
        if abs(self.f.subs(symbol1, resps[-1])) < self.erro:
            print('achou')
            return True
        elif self.iters <= 0:
            print('nao achou')
            return False
        else:
            return self.recursi()

def arredonda(x, R):
    if R is None:
        return x
    if x != 0:
        sinalx = x/abs(x)
        if sinalx < 0:
            x *= -1
        digs = 0
        if x > 1:
            sinal = 1
            while x > 1:
                x /= 10
                digs += 1
            digs -= R
        else:
            sinal = -1
            while x < 1:
                x *= 10
                digs += 1
            digs -= 1
            digs += R
            x/= 10
        x = x*10**R + 0.5000001
        x = x//1
        x = x*10**(digs*sinal)
        if sinalx < 0:
            x*=-1
        return x
    else:
        return 0

def escalona_gauss(matrix, R, cond):
    matrix = copy.deepcopy(matrix)
    multipliers = zeros(matrix.shape[0], matrix.shape[1])
    swaps = []
    line = 0
    for a in range(matrix.cols - 1):
        # print()
        # print()
        # print('Interacao ', a)
        # print('Antes do swap')
        # pprint(matrix)
        if cond:
            line_ = 0
            refcol = copy.deepcopy(matrix.col(a)[line:])
            refcol = special_sort(refcol)
            while refcol != matrix.col(a)[line:]:
                colA = matrix.col(a)[line + line_:]
                Max = max(colA)
                Min = min(colA)
                if abs(Min) > abs(Max):
                    larger = Min
                else:
                    larger = Max
                indi = index(colA, larger) + line + line_
                if indi != line + line_:
                    matrix.row_swap(line + line_, indi)
                    swaps.append((line + line_, indi, 'Col ' + str(a)))
                line_ += 1

        # print('Depois do swap')
        # pprint(matrix)
        # print(swaps)
        colA = matrix.col(a)[line:]
        Pivot = float(colA[0])
        multis = colA[1:]#/Pivot
        if Pivot != 0:
            for i, multi in enumerate(multis):
                multis[i] = arredonda(multi/Pivot, R)
                multipliers[line + i + 1, a] = multis[i]
                Pivot_row = matrix.row(line)
                Target_row = matrix.row(line + i + 1)
                Result_row = Target_row - Pivot_row*multis[i]
                Result_row = Matrix([[arredonda(number, R) for number in Result_row]])
                # pprint(matrix)
                matrix.row_del(line + i + 1)
                # pprint(matrix)
                matrix = matrix.row_insert(line + i + 1, Result_row)
                # pprint(matrix)
                matrix[line + i + 1, a] = 0
        else:
            if cond:
                pass
            else:
                Max = max(colA)
                Min = min(colA)
                if abs(Max) < 0.0001 and abs(Min) < 0.0001:
                    pass
                else:
                    if abs(Min) > abs(Max):
                        larger = Min
                    else:
                        larger = Max
                    indi = index(colA, larger) + line
                    if indi != line :
                        matrix.row_swap(line , indi)
                        swaps.append((line , indi, 'Col ' + str(a)))

                    colA = matrix.col(a)[line:]
                    Pivot = float(colA[0])
                    multis = colA[1:]#/Pivot
                    for i, multi in enumerate(multis):
                        multis[i] = arredonda(multi/Pivot, R)
                        multipliers[line + i + 1, a] = multis[i]
                        Pivot_row = matrix.row(line)
                        Target_row = matrix.row(line + i + 1)
                        Result_row = Target_row - Pivot_row*multis[i]
                        Result_row = Matrix([[arredonda(number, R) for number in Result_row]])
                        # pprint(matrix)
                        matrix.row_del(line + i + 1)
                        # pprint(matrix)
                        matrix = matrix.row_insert(line + i + 1, Result_row)
                        # pprint(matrix)
                        matrix[line + i + 1, a] = 0

        line += 1
        # print('multiplicadore')
        # pprint(multis)
        # print('Apos CL')
        # pprint(matrix)
    return matrix, multipliers, matrix+multipliers, swaps

def special_sort(Orig_lis, Targ_lis = None, targ_size = -10):
    if targ_size < 0:
        Targ_lis = []
        targ_size = len(Orig_lis)
    Max = max(Orig_lis)
    Min = min(Orig_lis)
    if abs(Min) > abs(Max):
        larger = Min
    else:
        larger = Max
    Targ_lis.append(larger)
    Orig_lis.remove(larger)
    if len(Targ_lis) == targ_size:
        return Targ_lis
    else:
        return special_sort(Orig_lis, Targ_lis, targ_size)

def index(line,target):
    for i in range(len(line)):
        if line[i] == target:
            return i
    return 0

def resolve_U(matrix, b, R):
    matrix = copy.deepcopy(matrix)
    b = copy.deepcopy(b)
    A = matrix
    x = zeros(b.shape[0],1)
    var = symbols('var')
    for j in range(A.shape[0]-1, -1, -1):
        A_row = A[j,:]
        x[j] = var
        multi = A_row*x
        sol = arredonda(solve(multi[0]-b[j])[0],R)
        x[j] = sol
    return x

def resolve_L(matrix, b, R):
    matrix = copy.deepcopy(matrix)
    b = copy.deepcopy(b)
    A = matrix
    x = zeros(b.shape[0],1)
    var = symbols('var')
    for j in range(A.shape[0]):
        A_row = A[j,:]
        x[j] = var
        multi = A_row*x
        sol = arredonda(solve(multi[0]-b[j])[0],R)
        x[j] = sol
    return x

def executa_swaps(swaps, vector):
    vector = copy.deepcopy(vector)
    vector = copy.deepcopy(vector)
    if swaps == []:
        print('Nao houveram swaps')
        return 'Nao houveram swaps'
    lines = []
    for swap in swaps:
        lines.append(swap[0])
        lines.append(swap[1])
    maxline = max(lines)
    column = Matrix([i for i in range(1, maxline+2)])
    for swap in swaps:
        column.row_swap(swap[0], swap[1])
        vector.row_swap(swap[0], swap[1])
    return column, vector

def desexecuta_swaps(swaps, vector):
    swaps = copy.deepcopy(swaps)
    vector = copy.deepcopy(vector)
    vector = copy.deepcopy(vector)
    if swaps == []:
        print('Nao houveram swaps')
        return 'Nao houveram swaps'
    lines = []
    for swap in swaps:
        lines.append(swap[0])
        lines.append(swap[1])
    maxline = max(lines)
    column = Matrix([i for i in range(1, maxline+2)])
    swaps.reverse()
    for swap in swaps:
        column.row_swap(swap[1], swap[0])
        vector.row_swap(swap[1], swap[0])
    return column, vector

class gauss_elimi:
    def __init__(self, matrix_exted, R, condensa, erro):
        self.erro = erro
        self.matrix = copy.deepcopy(matrix_exted)
        self.b = self.matrix.col(-1)
        self.R = R
        self.iters = 15
        self.xs = []
        self.rs = []
        self.cs = []
        self.A, self.multipliers, self.A_escal_bonita, self.swaps = escalona_gauss(self.matrix, R, condensa)
        self.A_escal = copy.deepcopy(self.A)
        self.b_ = self.A.col(-1)
        self.A = self.A[:,:-1]
        self.matrix = self.matrix[:,:-1]
        self.iters_ = 0
        x0 = resolve_U(self.A, self.b_, R)
        self.xs.append(x0)
        self.erros = [np.nan]

    def recursi(self):
        self.iters -= 1

        r = copy.deepcopy(self.b - self.matrix*self.xs[-1])
        r = Matrix([[arredonda(number, self.R*2) for number in r]])
        r = r.transpose()
        self.rs.append(r)
        c = resolve_U(self.matrix, self.rs[-1], self.R)
        self.cs.append(c)
        x = c + self.xs[-1]
        x = Matrix([arredonda(number, self.R) for number in x])
        self.xs.append(x)
        self.iters_ += 1
        stop = self.stop_criteria()
        if stop:
            print('achou em {} iteracoes'.format(self.iters_))
            return True
        elif self.iters <= 0:
            print('nao achou')
            return True
        else:
            self.recursi()

    def stop_criteria(self):
        resps = self.xs
        vk = copy.deepcopy(resps[-1])
        vk_1 = copy.deepcopy(resps[-2])
        lista = []
        for i, x in enumerate(vk):
            if abs(vk[i]) > 0.00000001:
                v = abs(vk[i] - vk_1[i])/vk[i]
                lista.append(v)
            elif abs(vk[i]) <  0.00000001 and abs(vk_1[i]) <  0.00000001:
                lista.append(0)
            else:
                lista.append(1)
        var = max(lista).evalf()
        # print(lista)
        self.erros.append(var)
        if var < self.erro:
            return True
        else:
            return False

class decomposicao_LU:
    def __init__(self, matrix, R, condensa, erro):
        self.erro = erro
        self.matrix = copy.deepcopy(matrix)
        self.b = self.matrix.col(-1)
        self.R = R
        self.U, self.L, self.A_escal_bonita, self.swaps = escalona_gauss(self.matrix, R, condensa)
        self.U = copy.deepcopy(self.U)
        self.b_ = self.U.col(-1)
        self.U = self.U[:,:-1]
        self.L = self.L[:,:-1]
        self.L += eye(self.L.shape[0])
        self.matrix = self.matrix[:,:-1]
        self.P = eye(self.L.shape[0])
        self.P = executa_swaps(self.swaps, self.P)[1]
        self.iters = 15
        self.xs = []
        self.rs = []
        self.cs = []
        self.iters_ = 0
        x0 = resolve_L(self.L, self.b, self.R)
        x0 = resolve_U(self.U, x0, self.R)
        self.xs.append(x0)
        self.erros = [np.nan]

    def recursi(self):
        self.iters -= 1
        #LU = PB
        r = self.P*copy.deepcopy(self.b - self.matrix*self.xs[-1])
        r = Matrix([[arredonda(number, self.R*2) for number in r]])
        r = r.transpose()
        self.rs.append(r)
        c = resolve_L(self.L, self.rs[-1], self.R)
        c = resolve_U(self.U, c,  self.R)
        self.cs.append(c)
        x = c + self.xs[-1]
        x = Matrix([arredonda(number, self.R) for number in x])
        self.xs.append(x)
        self.iters_ += 1
        stop = self.stop_criteria()
        if stop:
            print('achou em {} iteracoes'.format(self.iters_))
            return True
        elif self.iters <= 0:
            print('nao achou')
            return True
        else:
            self.recursi()

    def stop_criteria(self):
        resps = self.xs
        vk = copy.deepcopy(resps[-1])
        vk_1 = copy.deepcopy(resps[-2])
        lista = []
        for i, x in enumerate(vk):
            if abs(vk[i]) > 0.00000001:
                v = abs(vk[i] - vk_1[i])/vk[i]
                lista.append(v)
            elif abs(vk[i]) <  0.00000001 and abs(vk_1[i]) <  0.00000001:
                lista.append(0)
            else:
                lista.append(1)
        var = max(lista).evalf()
        # print(lista)
        self.erros.append(var)
        if var < self.erro:
            return True
        else:
            return False

# A = Matrix([
#     [4,1,0,1],
#     [1,4,1,1],
#     [0,1,4,1]
# ])
#
# a = decomposicao_LU(A,5,True,0.01)
#
# a.L

class jacobi:
    def __init__(self, matrix, R, erro, x0 = None):
        if x0 is None:
            x0 = zeros(matrix.shape[0],1)

        self.matrix = copy.deepcopy(matrix)
        self.b = self.matrix.col(-1)
        self.matrix = self.matrix[:,:-1]
        self.R = R
        self.erro = erro
        self.xs = [x0]
        self.iters_ = 0
        self.iters = 15
        self.criterios_conv()
        self.erros = [np.nan]
        self.errosk = [np.nan]

    def recursi(self):
        matrix = self.matrix
        b = self.b
        xk_1 = self.xs[-1]
        xk = zeros(matrix.shape[0],1)
        for i in range(len(xk)):
            somatorio = 0
            for j in range(len(xk)):
                if j != i:
                    somatorio += matrix[j,i]*xk_1[j]
            xk[i] = arredonda((b[i] - somatorio)/matrix[i, i], self.R)
        self.iters_ += 1
        self.iters -= 1
        self.xs.append(xk)
        stop = self.stop_criteria()
        if stop:
            print('achou em {} iteracoes'.format(self.iters_))
            return True
        elif self.iters <= 0:
            print('nao achou')
            return True
        else:
            self.recursi()

    def stop_criteria(self):
        resps = self.xs
        vk = copy.deepcopy(resps[-1])
        vk_1 = copy.deepcopy(resps[-2])
        lista = []
        for i, x in enumerate(vk):
            if abs(vk[i]) > 0.00000001:
                v = abs(vk[i] - vk_1[i])/vk[i]
                lista.append(v)
            elif abs(vk[i]) <  0.00000001 and abs(vk_1[i]) <  0.00000001:
                lista.append(0)
            else:
                lista.append(1)
        var = max(lista).evalf()
        M = self.M
        vec_dif = vk - vk_1
        max_error = max(abs(vec_dif))
        errok = (M)/(1-M)*max_error
        vec_dif = (self.xs[0] - self.xs[1])
        max_error = max(abs(vec_dif))
        erro = (M**(self.iters_+1))/(1-M)*max_error
        self.erros.append(erro)
        self.errosk.append(errok)

        if var < self.erro:
            return True
        else:
            return False

    def criterios_conv(self):
        matrix = self.matrix
        eig = matrix.eigenvals()
        size = matrix.shape[0]
        vals = []
        for val in eig:
            vals.append(val.evalf())
        maior = max(vals)
        self.Maior_eig = maior
        if maior > 1:
            print('Talvez n convirja, M = {} > 1'.format(maior))
        diag_domi = False
        diag_domi_num = []
        for i in range(size):
            soma_diag = 0
            for j in range(size):
                if i != j:
                    soma_diag += abs(matrix[i,j])
            if abs(matrix[i,i]) > soma_diag:
                diag_domi = True
                diag_domi_num.append(i)

        if not diag_domi:
            print('Talvez n convirja, Diagonal {} nao dominante',format(diag_domi_num))

        Alfa = ones(size,1)
        for j in range(size):
            somato = 0
            for i in range(size):
                if i != j:
                    somato += abs(matrix[j,i])
            Alfa[j] = arredonda(somato/matrix[j,j], self.R)
        self.Alfa = Alfa
        self.M = max(Alfa).evalf()

        if self.M > 1:
            print('Talvez n convirja, criterio das linhas, M = {} > 1'.format(self.M))

class gauss_seidel:
    def __init__(self, matrix, R, erro, x0 = None):
        if x0 is None:
            x0 = zeros(matrix.shape[0],1)

        self.matrix = copy.deepcopy(matrix)
        self.b = self.matrix.col(-1)
        self.matrix = self.matrix[:,:-1]
        self.R = R
        self.erro = erro
        self.xs = [x0]
        self.iters_ = 0
        self.iters = 15
        self.criterios_conv()
        self.errosk = [np.nan]
        self.erros = [np.nan]

    def recursi(self):
        matrix = self.matrix
        b = self.b
        xk_1 = self.xs[-1]
        xk = zeros(matrix.shape[0],1)
        for i in range(len(xk)):
            somatorio = 0
            for j in range(len(xk)):
                if j != i:
                    if j < i:
                        somatorio += matrix[j,i]*xk[j]
                    else:
                        somatorio += matrix[j,i]*xk_1[j]
            xk[i] = arredonda((b[i] - somatorio)/matrix[i, i], self.R)
        self.iters_ += 1
        self.iters -= 1
        self.xs.append(xk)
        stop = self.stop_criteria()
        if stop:
            print('achou em {} iteracoes'.format(self.iters_))
            return True
        elif self.iters <= 0:
            print('nao achou')
            return True
        else:
            self.recursi()

    def stop_criteria(self):
        resps = self.xs
        vk = copy.deepcopy(resps[-1])
        vk_1 = copy.deepcopy(resps[-2])
        lista = []
        for i, x in enumerate(vk):
            if abs(vk[i]) > 0.00000001:
                v = abs(vk[i] - vk_1[i])/vk[i]
                lista.append(v)
            elif abs(vk[i]) <  0.00000001 and abs(vk_1[i]) <  0.00000001:
                lista.append(0)
            else:
                lista.append(1)
        var = max(lista).evalf()
        # print(lista)
        M = self.M
        vec_dif = vk - vk_1
        max_error = max(abs(vec_dif))
        errok = (M)/(1-M)*max_error
        vec_dif = (self.xs[0] - self.xs[1])
        max_error = max(abs(vec_dif))
        erro = (M**(self.iters_+1))/(1-M)*max_error
        self.erros.append(erro)
        self.errosk.append(errok)
        if var < self.erro:
            return True
        else:
            return False

    def criterios_conv(self):
        matrix = copy.deepcopy(self.matrix)
        eig = matrix.eigenvals()
        size = matrix.shape[0]
        vals = []
        for val in eig:
            vals.append(val.evalf())
        maior = max(vals)
        self.Maior_eig = maior
        if maior > 1:
            print('Talvez n convirja, raio espectral = {} > 1'.format(maior))
        diag_domi = False
        diag_domi_num = []
        for i in range(size):
            soma_diag = 0
            for j in range(size):
                if i != j:
                    soma_diag += abs(matrix[i,j])
            if abs(matrix[i,i]) > soma_diag:
                diag_domi = True
                diag_domi_num.append(i)

        if not diag_domi:
            print('Talvez n convirja, Diagonal {} nao dominante'.format(diag_domi_num))
        matrix = copy.deepcopy(self.matrix)
        Beta = ones(size,1)
        for j in range(size):
            somato = 0
            for i in range(size):
                if i != j:
                    somato += abs(Beta[i]*matrix[j,i])
            Beta[j] = arredonda(somato/matrix[j,j], self.R)
        self.Beta = Beta
        self.M = max(Beta).evalf()

        if self.M > 1:
            print('Talvez n convirja, sassenfeld, M = {} > 1'.format(self.M))

class Sor:
    def __init__(self, matrix, R, erro, w, x0 = None):
        if x0 is None:
            x0 = zeros(matrix.shape[0],1)
        self.w = w
        self.matrix = copy.deepcopy(matrix)
        self.b = self.matrix.col(-1)
        self.matrix = self.matrix[:,:-1]
        self.R = R
        self.erro = erro
        self.xs = [x0]
        self.iters_ = 0
        self.iters = 15
        self.criterios_conv()
        self.errosk = [np.nan]
        self.erros = [np.nan]

    def recursi(self):
        matrix = self.matrix
        b = self.b
        xk_1 = self.xs[-1]
        xk = zeros(matrix.shape[0],1)
        for i in range(len(xk)):
            somatorio = 0
            for j in range(len(xk)):
                if j != i:
                    if j < i:
                        somatorio += matrix[j,i]*xk[j]
                    else:
                        somatorio += matrix[j,i]*xk_1[j]
            xk[i] = arredonda( self.w * (b[i] - somatorio)/matrix[i, i] + (1-self.w)*xk_1[i] , self.R)
        self.iters_ += 1
        self.iters -= 1
        self.xs.append(xk)
        stop = self.stop_criteria()
        if stop:
            print('achou em {} iteracoes'.format(self.iters_))
            return True
        elif self.iters <= 0:
            print('nao achou')
            return True
        else:
            self.recursi()

    def stop_criteria(self):
        resps = self.xs
        vk = copy.deepcopy(resps[-1])
        vk_1 = copy.deepcopy(resps[-2])
        lista = []
        for i, x in enumerate(vk):
            if abs(vk[i]) > 0.00000001:
                v = abs(vk[i] - vk_1[i])/vk[i]
                lista.append(v)
            elif abs(vk[i]) <  0.00000001 and abs(vk_1[i]) <  0.00000001:
                lista.append(0)
            else:
                lista.append(1)
        var = max(lista).evalf()
        # print(lista)
        M = self.M
        vec_dif = vk - vk_1
        max_error = max(abs(vec_dif))
        errok = (M)/(1-M)*max_error
        vec_dif = (self.xs[0] - self.xs[1])
        max_error = max(abs(vec_dif))
        erro = (M**(self.iters_+1))/(1-M)*max_error
        self.erros.append(erro)
        self.errosk.append(errok)
        if var < self.erro:
            return True
        else:
            return False

    def criterios_conv(self):
        matrix = copy.deepcopy(self.matrix)
        eig = matrix.eigenvals()
        size = matrix.shape[0]
        vals = []
        for val in eig:
            vals.append(val.evalf())
        maior = max(vals)
        self.Maior_eig = maior
        if maior > 1:
            print('Talvez n convirja, raio espectral = {} > 1'.format(maior))
        diag_domi = False
        diag_domi_num = []
        for i in range(size):
            soma_diag = 0
            for j in range(size):
                if i != j:
                    soma_diag += abs(matrix[i,j])
            if abs(matrix[i,i]) > soma_diag:
                diag_domi = True
                diag_domi_num.append(i)

        if not diag_domi:
            print('Talvez n convirja, Diagonal {} nao dominante'.format(diag_domi_num))
        matrix = copy.deepcopy(self.matrix)
        Beta = ones(size,1)
        for j in range(size):
            somato = 0
            for i in range(size):
                if i != j:
                    somato += abs(Beta[i]*matrix[j,i])
            Beta[j] = arredonda(somato/matrix[j,j], self.R)
        self.Beta = Beta
        self.M = max(Beta).evalf()

        if self.M > 1:
            print('Talvez n convirja, sassenfeld, M = {} > 1'.format(self.M))

class MMQ_discri:
    def __init__(self, GS, symbol, Xvector, Yvector, R = None):
        ''''por funcao 1 dentro da GS, GS é uma lista de g's '''
        R = R
        symbol = symbol
        GS = GS
        GSvectors = []
        dim = len(GS)
        B = []
        for g in GS:
            gvector = []
            for X in Xvector:
                gvector.append(arredonda(g.subs(symbol,X).evalf(), R))
            gvector = Matrix(gvector)
            GSvectors.append(gvector)
            b = gvector.dot(Yvector)
            B.append(b)
        B = Matrix(B)

        matrix = zeros(dim)

        for i in range(dim):
            for j in range(dim):
                matrix[i,j] =  GSvectors[i].dot(GSvectors[j])
        self.matrix = matrix
        self.B = B
        self.sol = self.matrix.inverse_LU() * self.B
        Funcvecs = Matrix(GS)
        self.GS = Funcvecs
        self.sol_multiplied = self.sol.dot(Funcvecs)
        self.sol_multiplied_matrix = Funcvecs.multiply_elementwise(self.sol)

# funcao que nao 'e do tipo a *x + b, deve ser linearizada
# A linearizacao vai deixar algo do tipo F(f(x),x) = a_  *x + B_
# (ou seja, encontramos a_ e b_, tem que converter pro original de volta)
# Com esse F, temos que converter o Yvector, como no exemplo abaixo.
#
#
# Xvector = [2, 4, 6, 8, 10]
# Yvector = [51.34, 52.72, 40.60, 27.79, 17.84]
# x_, y_ = symbols('x* y*')
# F = log(y_/x_)
#
# convert_lienarizado(Xvector, Yvector, F, x_, y_)

def convert_lienarizado(Xvector, Yvector,F, symbolx,symboly):
    Y_vector = []
    for i in range(len(Xvector)):
        temp = F.subs({
            symbolx:Xvector[i],
            symboly:Yvector[i]
        }).evalf()
        Y_vector.append(temp)
    return Matrix(Y_vector)

# x = symbols('x')
#
# g0 = x*0 + 1
# g1 = x
# GS = [g0, g1]
# Xvector = [2, 4, 6, 8, 10]
# Yvector = [51.34, 52.72, 40.60, 27.79, 17.84]
# x_, y_ = symbols('x* y*')
# F = log(y_/x_)
#
# Yvector = convert_lienarizado(Xvector, Yvector, F, x_, y_)
#
# a = MMQ_discri(GS, x, Xvector, Yvector)
#
# print(solveset(log(400*x)-float(a.sol[0])))

class MMQ_conti:
    def __init__(self, GS, symbol, Func, intervalo, R = None):
        ''''por funcao 1 dentro da GS, GS é uma lista de g's '''
        inf = intervalo.args[0]
        sup = intervalo.args[1]
        symbol = symbol
        GS = GS
        dim = len(GS)
        B = []
        for g in GS:
            inte = Func * g
            b = integrate(inte,(symbol,inf,sup))
            b = b.evalf()
            B.append(b)

        B = Matrix(B)
        matrix = zeros(dim)
        for i in range(dim):
            for j in range(dim):
                inte = GS[i] *GS[j] #GSvectors[i].dot(GSvectors[j])
                prod = integrate(inte,(symbol,inf,sup))
                prod = prod.evalf()
                matrix[i,j] =  prod

        self.matrix = matrix
        self.B = B
        self.sol = self.matrix.inverse_LU() * self.B
        Funcvecs = Matrix(GS)
        self.GS = Funcvecs
        self.sol_multiplied = self.sol.dot(Funcvecs)
        self.sol_multiplied_matrix = Funcvecs.multiply_elementwise(self.sol)

# x = symbols('x')
# g0 = x*0 + 1
# g1 = x
# g2 = x**2 - 1/3
# g3 = x**3 - 3*x/5
#
#
# Func = exp(2*x) + exp(-2*x)
# GS = [g0, g1, g2, g3]
# intervalo = Interval(-0.5,0.5)
#
# a = MMQ_conti(GS, x, Func, intervalo)

class serie_harmonic:
    def __init__(self, ordem, symbol, Func, intervalo, R = None):
        ''''por funcao 1 dentro da GS, GS é uma lista de g's '''
        self.ordem = ordem
        inf = intervalo.args[0]
        sup = intervalo.args[1]
        self.symbol = symbol
        self.gera_gs()
        GS = self.GS
        dim = len(GS)
        B = []
        for g in GS:
            inte = Func * g
            b = integrate(inte,(symbol,inf,sup))
            b = b.evalf()
            B.append(b)

        B = Matrix(B)
        matrix = zeros(dim)
        for i in range(dim):
            for j in range(dim):
                inte = GS[i] *GS[j] #GSvectors[i].dot(GSvectors[j])
                prod = integrate(inte,(symbol,inf,sup))
                prod = prod.evalf()
                matrix[i,j] =  prod

        self.matrix = matrix
        self.B = B
        self.sol = self.matrix.inverse_LU() * self.B
        Funcvecs = Matrix(GS)
        self.GS = Funcvecs
        self.sol_multiplied = self.sol.dot(Funcvecs)
        self.sol_multiplied_matrix = Funcvecs.multiply_elementwise(self.sol)

    def gera_gs(self):
        ordem = self.ordem
        symbol = self.symbol
        a0 = symbol*0 + 1
        GS = [a0]
        for i in range(1, ordem + 1):
            g = cos(i*symbol)
            GS.append(g)
            g_ = sin(i*symbol)
            GS.append(g_)
        self.GS = GS

#Para converter intervalo funcao F
#Nos extremos tem que bater
# x(t) tal que x(fim inter novo) é igual F(fim inter antigo)

# x = symbols('x')
# Func = abs(abs(x)-1/4)
# intervalo = Interval(-1,1)
#
# a = serie_harmonic(1, x, Func, intervalo)
# a.sol_multiplied_matrix
#
# p1 = plot(Func,(x,-3,3), show=False)
# p2 = plot(cos(x),(x,-3,3), show=False)
# p1.append(p2[0])
# p1.show()
```

```python
A = Matrix([
    [5.1, 3.2, -7.3, -136],
    [-8.4, -5.2, 8.1, 186],
    [4, 7.1, 0.4, -74]
])

Real = Matrix([-9.390888310, -5.671787784, 9.583116262])

a = gauss_elimi(A, 3, True, 0)
a.recursi()

print(max(abs(a.xs[0] - Real)))
```

    achou em 4 iteracoes
    0.0282122160000000

```python
a.xs[0] - Real

```

$\displaystyle \left[\begin{matrix}0.0208883099999984\\-0.028212216\\0.00688373799999908\end{matrix}\right]$


# Discriminant Analysis

* So far, we have seen methods for regression, classification, and clustering. Today, let's look again at classification.
* The goal in *classification* is to take an input vector, **x**, and assign it to one of K discrete classes.
* Usually this is done by learning *decision boundaries* or *decision surfaces* that divide up the feature space and assign a class to each partition of the feature space created by the decision boundary.
* Data whose classes can be separated perfectly with a linear decision surface (i.e., a line in 2-D, a plane in 3-D, etc) are called *linearly separable.* These are the easiest classification problems to address.
* You can encode class labels in a number of ways: binary vectors, class number, etc.  Different encodings make sense in different situations.  But you want to select carefully - depending on the algorithm you use, it can make a difference.

## Fisher's Linear Discriminant

* We will begin with linear discriminant functions:
\begin{equation}
y(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + w_0
\end{equation}

* *Looks pretty familiar, right?* If you are on one side of the line, then you are in class 1.  If you are on the other side of the line, then you are in class 2.  So, the decision boundary is $y(\mathbf{x}) = 0$
* The distance of a point to the decision boundary is: $\frac{y(\mathbf{x})}{\left\| \mathbf{w} \right\|}$  See Figure 4.1 from textbook.

* We could use a least squares error function to solve for $\mathbf{w}$ and $w_0$ as we did in regression.  But, there are some issues.  *Can you think of any?*
* A more effective (and well-known) approach is *Fisher's Linear Discriminant*
* Given two classes, we can compute the mean of each class:
\begin{equation}
\mathbf{m}_1 = \frac{1}{{N}_1}\sum_{n\in C_1} \mathbf{x}_n, \quad \mathbf{m}_2 = \frac{1}{N_2}\sum_{n\in C_2} \mathbf{x}_n
\end{equation}
* So, we could try to maximize the separation of the means:
\begin{equation}
m_2 - m_1 = \mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)
\end{equation}
* This is like an optimal *dimensionality reduction*. $\mathbf{w}^T\mathbf{x}$ takes a $D$ dimensional data point and projects it down to 1-D with a weight sum of the original features.  We want to find a weighting that maximizes the separation of the class means.
* Not only do we want well separate means for each class, but we also want each class to be *compact* to minimize overlap between the classes.
* Consider the *within class variance:*

\begin{equation}
s_k^2 = \sum_{n \in C_k} (y_n - m_k)^2 = \sum_{n \in C_k} (\mathbf{w}^T\mathbf{x}_n - m_k)^2
\end{equation}
* So, we want to minimize within class variance and maximize between class separability.  How about the following objective function:

\begin{eqnarray}
J(\mathbf{w}) &=& \frac{(m_2 - m_1)^2}{s_1^2 + s_2^2}\\
      &=& \frac{\mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)(\mathbf{m}_2 - \mathbf{m}_1)^T\mathbf{w}}{\sum_{n \in C_1} (\mathbf{w}^T\mathbf{x}_n - m_1)^2 + \sum_{n \in C_2} (\mathbf{w}^T\mathbf{x}_n - m_2)^2}\\
	  &=& \frac{\mathbf{w}^T(\mathbf{m}_2 - \mathbf{m}_1)(\mathbf{m}_2 - \mathbf{m}_1)^T\mathbf{w}}{\mathbf{w}^T\left(\sum_{n \in C_1} (\mathbf{x}_n - \mathbf{m}_1)(\mathbf{x}_n - \mathbf{m}_1)^T + \sum_{n \in C_2} (\mathbf{x}_n - \mathbf{m}_2)(\mathbf{x}_n - \mathbf{m}_2)^T\right)\mathbf{w}}\\
	  &=& \frac{\mathbf{w}^T\mathbf{S}_B\mathbf{w}}{\mathbf{w}^T\mathbf{S}_W\mathbf{w}}
\end{eqnarray}

* Ok, so lets optimize

\begin{eqnarray}
\frac{\partial J(\mathbf{w})}{\partial \mathbf{w}} &=& \frac{2(\mathbf{w}^T\mathbf{S}_W\mathbf{w})\mathbf{S}_B\mathbf{w} - 2(\mathbf{w}^T\mathbf{S}_B\mathbf{w})\mathbf{S}_W\mathbf{w}}{(\mathbf{w}^T\mathbf{S}_W\mathbf{w})^2} = 0 \\
0 &=& \ \frac{\mathbf{S}_B\mathbf{w}}{(\mathbf{w}^T\mathbf{S}_W\mathbf{w})} -\frac{(\mathbf{w}^T\mathbf{S}_B\mathbf{w})\mathbf{S}_W\mathbf{w}}{(\mathbf{w}^T\mathbf{S}_W\mathbf{w})^2} \\
 (\mathbf{w}^T\mathbf{S}_W\mathbf{w})\mathbf{S}_B\mathbf{w} &=& (\mathbf{w}^T\mathbf{S}_B\mathbf{w})\mathbf{S}_W\mathbf{w} \\
 \mathbf{S}_W^{-1}\mathbf{S}_B\mathbf{w} &=& \lambda\mathbf{w}
\end{eqnarray}

where the scalar $\lambda = \frac{\mathbf{w}^T\mathbf{S}_B\mathbf{w}} { \mathbf{w}^T\mathbf{S}_W\mathbf{w}}$

* *Does this look familiar?* This is the eigenvalue/eigenvector equation.

## Eigenvalues and Eigenvectors

* For a square matrix $\mathbf{A}$ of size $M\times M$, the eigenvector equation is:
\begin{equation}
\mathbf{A}\mathbf{u}_i = \lambda\mathbf{u}_i
\end{equation}
* This can be viewed as a set of $M$ linear equations with the condition (i.e., the *characteristic equation*:
\begin{equation}
|\mathbf{A} - \lambda_i\mathbf{I}| = 0
\end{equation}
* Eigenvalues and eigenvectors play a big role in this course and this area.  We will return to this topic repeatedly.
* For now, we can recognize the equation and know that there are many solvers that will provide the eigenvalues and orthogonal eigenvectors for a matrix.

## Reading Assignment

* Section 4.1.7
* Section 5.1

```python
import numpy as np
import numpy.matlib
import matplotlib.pyplot as plt
import math

def fisherDiscriminant(data,t):
    data1 = data[t==0,:]
    data2 = data[t==1,:]
    mean1 = np.atleast_2d(np.mean(data1,0))
    mean2 = np.atleast_2d(np.mean(data2,0))
    #Sb  = (mean2 - mean1).T@(mean2 - mean1)
    Sw1 = np.dstack([(data1[i,:]-mean1).T@(data1[i,:]-mean1) for i in range(data1.shape[0])])
    Sw2 = np.dstack([(data2[i,:]-mean2).T@(data2[i,:]-mean2) for i in range(data2.shape[0])])
    Sw = np.sum(Sw1,2) + np.sum(Sw2,2)
    w = np.linalg.inv(Sw)@(mean2 - mean1).T
    w = w/np.linalg.norm(w)
    #[l, v] = np.linalg.eig(np.linalg.inv(Sw)@Sb)
    #w = v[:,l.argsort()[1]]
    data_t = data@w
    return w, data_t

def PCA(data,d):
    N = np.size(data,0)
    data = data - data.mean(axis=0)
    cov_mat = (data - data.mean(axis=0)).T.dot(data - data.mean(axis=0))/N
    eigen_vals, eigen_vecs = np.linalg.eig(cov_mat)
    index = eigen_vals.argsort()[::-1]
    sorted_eigevecs = eigen_vecs[:,index]
    w = sorted_eigevecs[:,0:d]
    data_t = data@w
    return w, data_t
```

```python
# Fisher's Linear Discriminant

s     = .01
mean1 = [0, 0] #mean of class 1
mean0 = [1, 1] #mean of class 0
cov   = [1,0.01]*np.eye(2) #diagonal covariance
print(np.linalg.cond(cov))
numPnt1 = 100 #number of points for class1
numPnt0 = 100 #number of points for class0
data    = np.random.multivariate_normal(mean1, cov, numPnt1) #generate points  for class 1
t       = np.ones([1,numPnt1]) #labels for class
data    = np.vstack((data, np.random.multivariate_normal(mean0, cov, numPnt0))) #generate points for class 0
t       = np.hstack((t,np.zeros([1,numPnt0])))

plt.scatter(data[:,0],data[:,1],c=t[0]); plt.show();

```

```python
v,data_t = fisherDiscriminant(data,t[0])

fig = plt.figure()
plt.scatter(data[:,0],data[:,1],c=t[0,:])

v_perp = np.array([v[1], -v[0]])
b = ((np.mean(data[t[0]==0,:],axis=0)+np.mean(data[t[0]==1,:],axis=0))/2)@v
lambda_vec = np.linspace(-3,3,100)
v_line = lambda_vec * v
decision_boundary = b * v + lambda_vec * v_perp
plt.plot(v_line[0], v_line[1],label='Direction of Projection')
plt.plot(decision_boundary[0,:], decision_boundary[1,:],'k',linewidth=2,label='Decision Boundary')
plt.title("LDA")
plt.axis('equal')
plt.legend(loc='best')
plt.show()

fig = plt.figure()
plt.scatter(data_t[:,np.newaxis],np.zeros(len(data_t)),c=t[0,:],linewidth=0,s=100);
```

```python
v,data_t = PCA(data,1)

fig = plt.figure()
plt.scatter(data[:,0],data[:,1],c=t[0,:])

v_perp = np.array([v[1], -v[0]])
b = ((np.mean(data[t[0]==0,:],axis=0)+np.mean(data[t[0]==1,:],axis=0))/2)@v
lambda_vec = np.linspace(-3,3,100)
v_line = lambda_vec * v
decision_boundary = b * v + lambda_vec * v_perp
plt.plot(v_line[0], v_line[1],label='Direction of Projection')
plt.plot(decision_boundary[0,:], decision_boundary[1,:],'k',linewidth=2,label='Decision Boundary')
plt.title("PCA")
plt.axis('equal')
plt.legend(loc='best')
plt.show()

fig = plt.figure()
plt.scatter(data_t[:,np.newaxis],np.zeros(len(data_t)),c=t[0,:],linewidth=0,s=100);
```

```python

```

```python

```


# Define the flow potential

```python
import sympy as sp
sp.init_printing()
```

## Logic behind the flow potential extension

To account for a broad range of applications we propose
an extension of the flow potential with an adjustable
level of interaction between the damage in the normal
and tangential directions. Recall that the evolution equations
are defined as
\begin{align}
 \dot{\omega}_\mathrm{N} &= \lambda \frac{\partial \varphi}{ \partial {\omega}_\mathrm{N} } \\
 \dot{\omega}_\mathrm{T} &= \lambda \frac{\partial \varphi}{ \partial {\omega}_\mathrm{T} }
\end{align}
If the damage due to normal direction directly affects the tangential version we assume
that also the opposite affect is the same. The respective derivatives of the potential must be equal, i.e.
\begin{align}
\frac{\partial \varphi}{ \partial {\omega}_\mathrm{N} } = \frac{\partial \varphi}{ \partial {\omega}_\mathrm{T} }
\end{align}

```python
S_N, S_T, eta = sp.symbols('S_N, S_T, eta')
Y_N, Y_T = sp.symbols('Y_N, Y_T')
omega_N, omega_T = sp.symbols('omega_N, omega_T')
c_N, c_T = sp.symbols('c_N, c_T')
phi_N_ext = (  # (1-omega_N)**c_N * (
    (1*Y_N*(Y_N + eta * Y_T)) /
    (2*(S_N - eta * (S_N - (S_N + S_T)/2)))
)
phi_T_ext = (  # (1-omega_T)**c_T * (
    (1*Y_T*(Y_T + eta * Y_N)) /
    (2*(S_T - eta * (S_T - (S_N + S_T)/2)))
)
phi_ext = sp.simplify(phi_N_ext) + sp.simplify(phi_T_ext)
phi_ext
```

```python
phi_ext.subs(eta,1)
```

```python
sp.simplify(phi_ext.diff(Y_N).subs(eta,1))
```

```python
sp.simplify(phi_ext.diff(Y_T).subs(eta,1))
```

```python
sp.simplify(phi_ext.diff(Y_N).subs(eta,0))
```

## Backward derivation of the potential from desired flow direction

Find a potential function
\begin{align}
\varphi = a Y_\mathrm{N}^2 + \eta b Y_\mathrm{N}(Y_\mathrm{N} + Y_\mathrm{T})+ c Y_\mathrm{T}^2 + \eta d
Y_\mathrm{T}(Y_\mathrm{T} + Y_\mathrm{N})
\end{align}
which fulfills
\begin{align}
\left. \frac{\partial \varphi}{\partial Y_\mathrm{N}}\right|_{\eta=0} &=
(1 - \omega_\mathrm{N})^{c_\mathrm{N}} \frac{Y_\mathrm{N}}{S_\mathrm{N}} \\
\left. \frac{\partial \varphi}{\partial Y_\mathrm{T}}\right|_{\eta=0} &=
(1 - \omega_\mathrm{T})^{c_\mathrm{T}} \frac{Y_\mathrm{T}}{S_\mathrm{T}} \\
\left. \frac{\partial \varphi}{\partial Y_\mathrm{N}}\right|_{\eta=1} =
\left. \frac{\partial \varphi}{\partial Y_\mathrm{T}}\right|_{\eta=1} &=
\left(1 - \frac{1}{2}(\omega_\mathrm{N} + \omega_\mathrm{T})\right)^{\frac{c_\mathrm{N}+c_\mathrm{T}}{2}}
\frac{Y_\mathrm{N} + Y_\mathrm{T}}{S_\mathrm{N} + S_\mathrm{T}}
\end{align}

```python
a, b, c, d = sp.symbols('a,b,c,d')
H = lambda x: sp.Piecewise( (0, x <=0 ), (1, True) )
H_switch, sig_pi = sp.symbols(r'H(\sigma^{\pi}), \sigma^\pi', real=True)
phi_ext = a * Y_N**2 + b * eta * Y_N*(Y_N + Y_T) + c * Y_T**2 + d * eta * Y_T*(Y_T+Y_N)
```

```python
d_phi_N_0 = phi_ext.diff(Y_N).subs(eta,0)
a_solved = sp.solve(sp.Eq(d_phi_N_0, (1 - omega_N)**c_N * Y_N / S_N * H_switch ), a )[0]
#a_solved = sp.solve( sp.Eq( d_phi_N_0, (1 - omega_N)**c_N * Y_N / S_N ), a )[0]
d_phi_N_0, a_solved
```

```python
d_phi_T_0 = phi_ext.diff(Y_T).subs(eta,0)
c_solved = sp.solve( sp.Eq( d_phi_T_0, (1 - omega_T)**c_T * Y_T / S_T ), c )[0]
d_phi_T_0, c_solved
```

```python
phi_ext_ac = phi_ext.subs({a: a_solved, c: c_solved})
phi_ext_ac
```

```python
d_phi_N_1 = phi_ext_ac.diff(Y_N).subs(eta,1)
d_phi_N_1
```

```python
d_phi_T_1 = phi_ext_ac.diff(Y_T).subs(eta,1)
d_phi_T_1
```

```python
d_phi_1_req = (1 - (omega_N + omega_T)/2)**((c_N+c_T)/2) * (Y_N + Y_T) / (S_N + S_T)
d_phi_1_req
```

```python
bd_solved = sp.solve({sp.Eq(d_phi_N_1, d_phi_1_req), sp.Eq(d_phi_T_1, d_phi_1_req)},[b,d])
sp.simplify(bd_solved[b])
```

```python
phi_abcd = phi_ext_ac.subs(bd_solved)
sp.simplify(phi_abcd)
```

```python
sp.simplify(phi_abcd.diff(Y_T).subs(eta,1))
```

```python
sp.simplify(phi_abcd.diff(Y_N).subs(eta,sp.Rational(1,2)))
```

# Geometric mean

```python
a,b,c,d = sp.symbols('a,b,c,d')
H = lambda x: sp.Piecewise( (0, x <=0 ), (1, True) )
H_switch, sig_pi = sp.symbols(r'H(\sigma^{\pi}), \sigma^\pi', real=True)
phi2_ext = a * Y_N**2 + b * eta * Y_N*(Y_N+Y_T) + c * Y_T**2 + d * eta * Y_T*(Y_N+Y_T)
phi2_ext
```

```python
d_phi2_N_0 = phi2_ext.diff(Y_N).subs(eta,0)
a2_solved = sp.solve( sp.Eq( d_phi2_N_0, (1 - omega_N)**c_N * Y_N / S_N * H_switch ), a )[0]
#a_solved = sp.solve( sp.Eq( d_phi_N_0, (1 - omega_N)**c_N * Y_N / S_N ), a )[0]
d_phi2_N_0, a2_solved
```

```python
d_phi2_T_0 = phi2_ext.diff(Y_T).subs(eta,0)
c2_solved = sp.solve( sp.Eq( d_phi2_T_0, (1 - omega_T)**c_T * Y_T / S_T ), c )[0]
d_phi2_T_0, c2_solved
```

```python
phi2_ext_ac = phi2_ext.subs({a: a2_solved, c: c2_solved})
phi2_ext_ac
```

```python
d_phi2_N_1 = phi2_ext_ac.diff(Y_N).subs(eta,1)
d_phi2_N_1
```

```python
d_phi2_T_1 = phi2_ext_ac.diff(Y_T).subs(eta,1)
d_phi2_T_1
```

```python
c_NT = sp.sqrt(c_N*c_T)
S_NT = sp.sqrt(S_N*S_T)
d_phi_2_req = (1 - sp.sqrt(omega_N*omega_T))**(c_NT) * (Y_N+Y_T) / (2*S_NT)
d_phi_2_req
```

```python
bd2_solved = sp.solve({sp.Eq(d_phi2_N_1, d_phi_2_req), sp.Eq(d_phi2_T_1, d_phi_2_req)},[b,d])
sp.simplify(bd2_solved[b])
```

```python
phi2_abcd = phi2_ext_ac.subs(bd2_solved)
sp.simplify(phi2_abcd).subs(eta,0)
```

```python
omega_T_solved_0 = sp.simplify(phi2_abcd.diff(Y_T).subs(eta,0))
omega_T_solved_0
```

```python
omega_N_solved_0 = sp.simplify(phi2_abcd.diff(Y_N).subs(eta,0))
omega_N_solved_0
```

```python
omega_N_solved_1 = sp.simplify(phi2_abcd.diff(Y_N).subs(eta,1))
omega_N_solved_1
```

```python
omega_T_solved_1 = sp.simplify(phi2_abcd.diff(Y_T).subs(eta,1))
omega_T_solved_1
```

**Archived compact version:** derived by Mirek for the case of dropped fundamental damage terms $(1 - \omega_\mathrm{N})^{c_\mathrm{N}}$ and $(1 - \omega_\mathrm{T})^{c_\mathrm{T}}$

```python
phi_simplified = (
    (( 1 - eta ) * ( Y_T**2 * S_N**2 + Y_N**2 * S_T**2 ) +
    S_N * S_T * ( Y_T**2 + Y_N**2 + 2 * Y_T * Y_N * eta )) /
    (2* S_N * S_T * ( S_N + S_T ))
)
phi_simplified
```

# Scaling of phi between coupled and uncoupled cases

```python
def avg(var1, var2):
    return (var1 + var2) / 2
r = sp.symbols(r'r', positive=True)
c_NT = avg(c_N, c_T)
S_NT = avg(S_N, S_T)
omega_NT = avg(omega_N, omega_T)
Y_NT = avg(Y_N, Y_T)
phi_N = (1 - omega_N)**(c_N) * S_N/(r+1) * (Y_N/S_N)**(r+1) # * H_switch
phi_T = (1 - omega_T)**(c_T) * S_T/(r+1) * (Y_T/S_T)**(r+1)
phi_NT  = (1 - omega_NT)**(c_NT) * 2*S_NT/(r+1) * (Y_NT/S_NT)**(r+1)
phi_ = (1 - eta) * (phi_N + phi_T) + eta * phi_NT
```

```python
sp.simplify(phi_).subs(r,1).subs(eta,0).diff(Y_T)
```

```python
sp.simplify(phi_).subs(r,1).subs(eta,0).diff(Y_N)
```

```python

```

# Derivation of potential from a damage function

```python
A_d, Y_N, R_N, lambda_N = sp.symbols('A_d, Y_N, R_N, lambda_N')
dot_Y_N, dot_R_N = sp.symbols(r'\dot{Y}_N, \dot{R}_N')
E_N , eps_N = sp.symbols(r'E_N, \epsilon_N')
dot_eps_N = sp.symbols(r'\dot{\epsilon}_N')
r_N, dot_r_N = sp.symbols(r'r_N, \dot{r}_N')
f_N = sp.Function('f_N')(r_N)
omega_N = sp.symbols(r'\omega_N')
dot_omega_N = sp.symbols(r'\dot{\omega}_N')
```

```python
psi_omega = sp.Rational(1,2)*(1 - omega_N)*E_N*eps_N**2 + f_N
psi_omega
```

```python
sigma_N_ = psi_omega.diff(eps_N)
Y_N_ = -psi_omega.diff(omega_N)
R_N_ = psi_omega.diff(r_N)
Y_N_
```

```python
dot_R_N_ = R_N_.diff(r_N) * dot_r_N
dot_R_N_
```

```python
dot_Y_N_ = Y_N_.diff(eps_N) * dot_eps_N + Y_N_.diff(omega_N) * dot_omega_N
dot_Y_N_
```

```python
f_omega = Y_N - R_N
```

```python
f_omega
```

```python
dot_r_N_ = lambda_N * f_omega.diff(R_N)
dot_r_N_
```

```python
dot_omega_N_ = lambda_N * f_omega.diff(Y_N)
dot_omega_N_
```

```python
dot_f_omega_ = f_omega.diff(Y_N) * dot_Y_N_ + f_omega.diff(R_N) * dot_R_N_
```

```python
dot_f_omega_
```

```python
dot_f_omega_lambda_ = dot_f_omega_.subs(dot_r_N, dot_r_N_)
```

```python
lambda_N_solved = sp.solve(dot_f_omega_lambda_, lambda_N)[0]
```

```python
lambda_N_solved
```

```python
dot_omega_N_solved = dot_omega_N_.subs(lambda_N, lambda_N_solved)
dot_omega_N_solved
```

# Damage function from MS1 - N

**reference [74]**

```python
omega_N_paper = 1 - 1 / (1 + A_d * Y_N_)
```

```python
omega_N_paper
```

```python
dot_omega_N_paper = omega_N_paper.diff(eps_N) * dot_eps_N
dot_omega_N_paper
```

**from potential**

```python
f_prime_r_N = 1 / A_d * -r_N / (1 + r_N)
f_prime_r_N
```

```python
f_prime_r_N.diff(r_N)
```

```python
dot_omega_N_potential = sp.simplify(- E_N * eps_N * dot_eps_N / f_prime_r_N.diff(r_N))
dot_omega_N_potential
```

```python
r_N_solved = sp.solve( sp.Eq( dot_omega_N_paper, dot_omega_N_potential ), r_N )
r_N_solved
```

```python
dot_omega_N_potential_solved = sp.simplify(dot_omega_N_potential.subs(r_N, r_N_solved[1]))
dot_omega_N_potential_solved
```

```python
sp.simplify(dot_omega_N_paper)
```

```python

```


# 2D Ising Model

In this exercise, we will simulate a 2D Ising model on a square lattice. The Ising model is a simple model that can be used to demostrate magnetism. It consists of spins $1/2$s arranged on a lattice with nearest-neighbor (NN) interactions. The model can also include next-to-nearest neighbor (NN) interactions as well as be extended to higher dimensions.

### Hamiltonian

The Hamiltonian of the NN spin $1/2$ Ising model can be written as
\begin{align}
𝐻 = 𝐽 \ \sum_{\langle 𝑖𝑗 \rangle} \sigma^𝑧_𝑖 \sigma^𝑧_𝑗 − ℎ\ \Sigma_i \sigma^𝑧_𝑖,
\end{align}
where $\langle ij \rangle$ denotes nearest neighbor couplings. For simplicity, let us assume that $\sigma_z = \pm 1$.
$J$ is the interaction strength ($J<0$ for ferromagnetic interactions and $J>0$ for antiferromagnetic interactions).
$h$ is the magnetic field.

The spin lattice can be realized using a 2d (Integer) matrix of size $L$ (indices 0,1,..,L-1). To begin with, let us choose $L=20$. We choose periodic boundary conditions, such that the the spin corresponding to $[i,L+1] = [i,0]$ and $[L+1,i] = [0,i]$ for $0 \leq i \leq L-1$. Choose $k_B=1$, such that $\beta = 1/T$.

### Metropolis Algorithm:

At a given temperature $T$, a possible state of the lattice can be found out using the Metropolis algorithm where a Markov-Chain Monte-Carlo is realized through changing the configuration of a single spin. it consists of the following steps-

a) Choose a random site $[i,j]$;

b) Calculate $\delta E$, the energy cost of flipping the spin at site $[i,j]$;

c) Choose the new configuration if $\delta E<0$, or if $\delta E >0$, then choose the new state with a probability of $\exp(-\beta \delta E)$.

d) Repeat steps a) through c) for at least $L^3$ iterations. It is better to choose even more iterations e.g., $10 L^3$, such that there are enough Monte-Carlo steps for the system to thermalize.

#### Note:
One needs to be careful near the edge of the lattie, eg, if a spin is choosen at site $[L-1,j]$ and $[i,L-1]$. In this case the NN site corresponds to $[0,j]$ and $[i,0]$ respectively.

### Measurement of physical quantities

In order to measure a physial quantity, one needs to sample over a number of configurations. In practice, we will use the Metroplois algorithm to thermalize the system (using e.g., $L^3$ Monte-Carlo steps). After that, a measurement is performed on the system after every 'mstep' Monte-Carlo steps. Subsequently, one can take the average of all these measurements. We will be calculating the following opservables.

#### Magnetization per unit site:
\begin{align}
m = \langle \sigma^z \rangle = \sum_i \sigma_i^z / L^2
\end{align}

#### Magnetic Susceptibility:

\begin{align}
\chi =  \lim_{h\rightarrow 0}  \frac{\partial m}{\partial h} \approx \frac{\langle m^2 \rangle  - \langle m \rangle ^2}{T}
\end{align}

#### Energy per site:

\begin{align}
E = \frac{\langle H \rangle }{L^2}
\end{align}

#### Specific heat:

\begin{align}
C_V = \frac{\langle E^2 \rangle - \langle E \rangle ^2}{T^2}
\end{align}

#### Spin correlation function:

\begin{align}
C(r) = \langle \sigma^z(r) \sigma^z(0) \rangle
\end{align}

#### Nearest-neighbor Spin correlation:

\begin{align}
C_1 = \langle \sigma^z_i \sigma^z_j \rangle
\end{align}

```
import numpy as np
import matplotlib.pyplot as plt
from numpy import random
```

```
# Run Parameters
# L: Lattice size
# j_nn, j_nnn : nearest-neighbor, next-nearest-neighbor interactions
# h : applied uniform magnetic field
# thi, tlo, tstep : Highest, Lowest, decrement temperature
# ntemp : No. of temperature steps
# Iterations : No. of MC steps for thermalization
# n_meas : No. of measurements of observables
# msteps : No. of MC steps before each measurement

L=40;
j_nn = -1.0; j_nnn = 0.0; h = 0.0;
thi = 5.0; tlo = 0.5; tstep = -0.1; ntemp = -int((thi-tlo)/tstep);
Iterations = 10*L**3;
n_meas = 10; mstep = 50;
```

```
def Magnetization_Op(Lattice,L):
    "calculate <m> and <m^2> of a lattice configuration"
    return sum(sum(Lattice))/L**2, sum(sum(Lattice**2))/L**2

def Energy_Op(Lattice,L,j_nn,h):
    "Calculate the <Energy> and <Energy^2> per lattice site"
    e = 0.0; e2=0.0
    for x in range(L):
        for y in range(L):
            # nearest neighbors are (x,y1)(x,y2)(x1,y)(x2,y)
            x1,x2,y1,y2=(x-1)%L,(x+1)%L,(y-1)%L,(y+1)%L;
            e += h*Lattice[x,y] +j_nn*Lattice[x,y]*(Lattice[x,y1]+Lattice[x,y2]+Lattice[x1,y]+Lattice[x2,y]);
            e2+= (h*Lattice[x,y] +j_nn*Lattice[x,y]*(Lattice[x,y1]+Lattice[x,y2]+Lattice[x1,y]+Lattice[x2,y]))**2;

    return e/(2*L**2), e2/(4*L**2)

def Spin_Correlation_NN_Op(Lattice,L):
    Spin_corr_nn = 0
    for x in range(L):
        for y in range(L):
            # nearest neighbors are (x,y1)(x,y2)(x1,y)(x2,y)
            x1,x2,y1,y2=(x-1)%L,(x+1)%L,(y-1)%L,(y+1)%L;
            Spin_corr_nn += 0.25*Lattice[x,y]*(Lattice[x,y1]+Lattice[x,y2]+Lattice[x1,y]+Lattice[x2,y]);
    return Spin_corr_nn/L**2

def Spin_Correlation_Op(Lattice,L):
    Spin_corr = np.zeros(int(L/2)+1);
    x0 = int(L/2)-1; y0 = int(L/2)-1;
    for i in range(int(L/2)+1):
        Spin_corr[i] = 0.25*Lattice[x0,y0]*( Lattice[x0+i,y0] + Lattice[x0-i,y0] + Lattice[x0,y0+i] + Lattice[x0,y0-i])
    return Spin_corr

```

```
# Initialization of Arrays for Lattice, Observables and temperatures

# Random Assignment of lattice
lattice = np.zeros((L,L),dtype=int);
lattice = random.randint(2,size=(L,L))
lattice = 2*lattice-1

temperatures = np.linspace(thi,tlo,ntemp);

magnetization  = np.zeros(ntemp);
temperature    = np.zeros(ntemp);
susceptibility = np.zeros(ntemp);
energy         = np.zeros(ntemp);
specific_heat  = np.zeros(ntemp);
spin_corr      = np.zeros((ntemp,int(L/2)+1));
sp_corr        = np.zeros(int(L/2)+1);
spin_corr_nn   = np.zeros(ntemp);
lattice_save   = np.zeros((ntemp,L,L),dtype=int);
```

```
# Cooling down the Lattice
step = 0;
for temp in temperatures:

    m = 0.0; chi = 0.0; e = 0.0; cv = 0.0; sp_corr[:] = 0.0; sp_corr_nn = 0.0;

    for i in range(Iterations):

        # Perform a MC step

        # Choose a random site
        x=random.randint(L); y=random.randint(L);
        # nearest neighbors are (x,y1)(x,y2)(x1,y)(x2,y)
        x1,x2,y1,y2=(x-1)%L,(x+1)%L,(y-1)%L,(y+1)%L

        # Change in energy for a spin-flip
        de = 2*h*lattice[x,y]- 2*j_nn*lattice[x,y]*(lattice[x,y1]+lattice[x,y2]+lattice[x1,y]+lattice[x2,y])
        r = random.random();
        if (de < 0.0 or r < np.exp(-de/temp)): # Choose new configuration with probablity ~ exp(-beta dE)
            lattice[x,y] = -1 * lattice[x,y]   # Change the spin

    for i in range((n_meas-1)*mstep+1):
        # Perform a MC step

        # Choose a random site
        x=random.randint(L); y=random.randint(L);
        # nearest neighbors are (x,y1)(x,y2)(x1,y)(x2,y)
        x1,x2,y1,y2=(x-1)%L,(x+1)%L,(y-1)%L,(y+1)%L

        # Change in energy for a spin-flip
        de = 2*h*lattice[x,y]- 2*j_nn*lattice[x,y]*(lattice[x,y1]+lattice[x,y2]+lattice[x1,y]+lattice[x2,y])
        r = random.random();
        if (de<0.0 or r < np.exp(-de/temp)): # Choose new configuration with probablity ~ exp(-beta dE)
            lattice[x,y]=-1*lattice[x,y]     # Change the spin

        if (i%mstep == 0):
            m1, m2 = Magnetization_Op(lattice,L); e1,e2 = Energy_Op(lattice,L,j_nn,h)
            m += m1; chi += (m2 - m1**2)/temp
            e += e1; cv  += (e2 - e1**2)/temp**2
            sp_corr[:] += Spin_Correlation_Op(lattice,L)
            sp_corr_nn += Spin_Correlation_NN_Op(lattice,L)

    magnetization[step] = m/n_meas; susceptibility[step] = chi/n_meas;
    energy[step] = e/n_meas; specific_heat[step] = cv / n_meas;
    spin_corr[step,:] = sp_corr[:] /n_meas;
    spin_corr_nn[step] = sp_corr_nn /n_meas;
    lattice_save[step,:,:] = lattice[:,:];
    step +=1

```

```
# Plot Magnetization, Magnetic Susceptibility, Energy and Specific Heat

plt.figure(figsize=(12,12))

plt.subplot(221)
plt.title("Magnetization")
plt.grid()
plt.xlabel(r'$T$'); plt.ylabel(r'$\langle m \rangle$')
plt.plot(temperatures,magnetization,'-o')

plt.subplot(222)
plt.title("Magnetic susceptibility")
plt.grid()
plt.xlabel(r'$T$'); plt.ylabel(r'$\chi$')
plt.plot(temperatures,susceptibility,'-o')

plt.subplot(223)
plt.title("Energy")
plt.grid()
plt.xlabel(r'$T$'); plt.ylabel(r'$\langle E \rangle$')
plt.plot(temperatures,energy,'-o')

plt.subplot(224)
plt.title("Specific Heat")
plt.grid()
plt.xlabel(r'$T$'); plt.ylabel(r'$C_v$')
plt.plot(temperatures,specific_heat,'-o')

plt.savefig('Observables.pdf',format='pdf')
```

```
# Plot Nearest-neighbor Spin-Correlation

plt.title("Nearest-neighbor Spin-Correlation")
plt.grid()
plt.xlim(0,5.0); plt.ylim(0,1.2)
plt.xlabel(r'$T$'); plt.ylabel(r'$\langle S_{\langle ij \rangle} \rangle$')
plt.plot(temperatures,spin_corr_nn,'o-')
plt.savefig('SC_nn.pdf',format='pdf')
```

```
# Plot Spin-Correlation

# plt.figure()
# plt.title("Spin-Correlation")
# plt.ylabel(r'$ \langle S(0).S(r) \rangle $'); plt.xlabel(r'$r$')
# plt.grid()
# plt.plot(spin_corr[0,:],  label = '%.1f' % temperatures[0])
# plt.plot(spin_corr[10,:], label = '%.1f' % temperatures[10])
# plt.plot(spin_corr[20,:], label = '%.1f' % temperatures[20])
# plt.plot(spin_corr[25,:], label = '%.1f' % temperatures[25])
# plt.plot(spin_corr[29,:],  label = '%.1f' % temperatures[29])
# plt.plot(spin_corr[39,:], label = '%.1f' % temperatures[39])
# plt.legend()
```

```
# Plot Lattice Configurations

plt.figure(figsize=(18,12))

plt.subplot(231)
plt.title('T =' + '%.1f' % temperatures[0])
plt.imshow(lattice_save[0,:,:])

plt.subplot(232)
plt.title('T = ' + '%.1f' % temperatures[10])
plt.imshow(lattice_save[10,:,:])

plt.subplot(233)
plt.title('T = ' + '%.1f' % temperatures[20])
plt.imshow(lattice_save[20,:,:])

plt.subplot(234)
plt.title('T = Tc = ' + '%.1f' % temperatures[25])
plt.imshow(lattice_save[25,:,:])

plt.subplot(235)
plt.title('T = ' + '%.1f' % temperatures[29])
plt.imshow(lattice_save[29,:,:])

plt.subplot(236)
plt.title('T = ' + '%.1f' % temperatures[39])
plt.imshow(lattice_save[39,:,:])

plt.savefig('Lattice_config.pdf',format='pdf')
```