#### Zadanie 1
Dany jest słownik student={'imie':'Jan', 'nazwisko':'kowalski','wiek':25, 'wzrost':188, 'waga':80, 'miasto':'Toruń'}

* Wypisz jego elementy w postaci "klucz -> wartość "
* Zmień 'wzrost' na 182
* Dodaj klucz 'wynik matury' z dowolną wartością
* Usuń klucz (i wartość) 'miasto'

```python
student={'imie':'Jan', 'nazwisko':'kowalski','wiek':25, 'wzrost':188, 'waga':80, 'miasto':'Toruń'}
```

```python
for klucz in student.keys():
  print(klucz+'->'+str(student[klucz]))
```

    imie->Jan
    nazwisko->kowalski
    wiek->25
    wzrost->188
    waga->80
    miasto->Toruń

```python
student['wzrost']=182
```

```python
student
```

    {'imie': 'Jan',
     'miasto': 'Toruń',
     'nazwisko': 'kowalski',
     'waga': 80,
     'wiek': 25,
     'wzrost': 182}

```python
student['wynik matury']=20
```

```python
del student['miasto']
```

```python
student
```

    {'imie': 'Jan',
     'nazwisko': 'kowalski',
     'waga': 80,
     'wiek': 25,
     'wynik matury': 20,
     'wzrost': 182}

```python

```

```python

```

####Zadanie 2
Dane są dwie listy równej długości

klucze=['klucz1','klucz2','inny klucz', 'test']
wartosci=[1,2,5,1]
Utwórz w sposób zautomatyzowany (nie ręcznie) słownik, który kolejnym kluczom przypisze kolejne wartości.

```python
klucze=['klucz1','klucz2','inny klucz', 'test']
wartosci=[1,2,5,1]
nowy=zip(klucze,wartosci)
```

```python
list(lista)
```

    [('klucz1', 1), ('klucz2', 2), ('inny klucz', 5), ('test', 1)]

```python
dict(nowy)
```

    {'inny klucz': 5, 'klucz1': 1, 'klucz2': 2, 'test': 1}

```python

```

#### Zadanie 3
Napisz funkcję obliczającą $n$-ty wyraz ciągu zadanego rekurencyjnie
1. $a_0=1$, $a_{k+1}=2a_k$ dla $k\geq 0$
2. $b_0=0$, $b_1=1$, $b_{k+2}=b_{k+1}+b_k$ dla $k\geq 0$

```python
def ciag(n):
  if n==0:
    return 1
  else:
    return 2*ciag(n-1)
```

```python
ciag(5)
```

    32

```python

```

# Sympy

```python
from sympy import *

```

```python
sin(30)
```

$\displaystyle \sin{\left(30 \right)}$

```python
import math
```

```python
math.sin(30)
```

    -0.9880316240928618

```python
x,y=symbols('x,y')
```

```python
x
```

$\displaystyle x$

```python
y
```

$\displaystyle y$

```python
cos(x)**2+sin(x)**2
```

$\displaystyle \sin^{2}{\left(x \right)} + \cos^{2}{\left(x \right)}$

```python
simplify(_)
```

$\displaystyle 1$

```python
solve(2*x-5)
```

    [5/2]

```python
wyr=x**2+5*x+3
```

```python
wyr.subs(x,1)
```

$\displaystyle 9$

```python
sin(30).evalf()
```

$\displaystyle -0.988031624092862$

```python
wyr.diff(x)
```

$\displaystyle 2 x + 5$

```python
f=symbols('f',cls=Function)
```

```python
g=Function('g')
```

```python
ODE=f(x).diff(x)-f(x)
```

```python
dsolve(ODE)
```

$\displaystyle f{\left(x \right)} = C_{1} e^{x}$

```python
integrate(wyr,(x,0,1))
```

$\displaystyle \frac{35}{6}$

```python
plot(wyr,(x,0,5))
```

```python

```

#### Zadanie 4
Napisz funkcję $pochodna(funkcja,punkt)$, która oblicza z definicji pochodną danej funkcji w punkcie. Korzystając ze swojej funkcji oblicz

* pochodną funkcji $e^{x^2}$ w punkcie $1$
* pochodną funkcji $x*\ln(\sqrt(x))$ w punkcie $1$

Porównaj wyniki do tych uzyskanych za pomocą metody diff

```python
from sympy import *
def pochodna(f,x,a):
  h=symbols('h')
  return limit((f.subs(x,a+h)-f.subs(x,a))/h,h,0)

pochodna(y**2,y,1)
```

$\displaystyle 2$

```python

```

## Zadanie domowe
a) Napisz funkcję obliczającą wartość wielomianu interpolacyjnego Lagrange'a w zadanym punkcie. Funkcja powinna przyjmować następujące argumenty:

X - tablicę wartości  xi ,
Y - tablicę wartości  yi ,
x - punkt, w którym liczymy wartość wielomianu.
b) Dodaj opcjonalny argument do powyższej funkcji, który pozwoli wyświetlić wzór interpolacyjny Lagrange'a w postaci symbolicznej.

c) sporządź wykres otrzymanego wielomianu na podstawie symbolicznego wzoru z b)

d) wybierz dwie funkcje: wielomian stopnia 5 i inną funkcję niebędącą wielomianem. Sporządź ich aproksymacje za pomocą 4 wybranych punktów. Porównaj wykresy oryginalnych funkcji i ich przybliżeń. Sformułuj wnioski.

```python

```


 INF285 - Computación Científica

 Polynomial Interpolation: Vandermonde matrix, Lagrange interpolation, Barycentric interpolation, and Chebyshev points.

  [S]cientific [C]omputing [T]eam

 Version: 1.33

## Table of Contents
* [Introduction](#intro)
* [Vandermonde Matrix](#vander)
    * [Is the Vandermonde matrix really ill-conditioned?](#vandermonde_illconditioned)
* [Lagrange Interpolation](#lagrange)
    * [Plotting $L_i(x)$](#LiLagrange)
* [Barycentric interpolation](#barycentric)
* [Interpolation Error](#Error)
* [Runge Phenomenon](#runge)
* [Interpolation with Chebyshev points](#cheby)
* [Python Modules and Functions](#py)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
from functools import reduce
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
%matplotlib inline
from ipywidgets import interact, fixed, IntSlider, interact_manual, Checkbox, RadioButtons
sym.init_printing()
import time
```

## Introduction
[Back to TOC](#toc)

Hello! In this notebook we will learn how to interpolate 1D data with polynomials.
A polynomial interpolation consists in finding a polynomial that fits a discrete set of known data points, allowing us to construct new data points within the range of the data.
Formally, a polynomial $p(x)$ interpolate the data $(x_1,y_1),...,(x_n,y_n)$ if $p(x_i)=y_i$ for all $i$ in $1,...,n$.

```python
# Function that evaluates the xi's points in the polynomial
def EvaluateInterpolant(D, xx):
    if D['Method']=='Vandermonde':
        p = lambda x: np.dot(D['PrecomputedData'],np.power(x,np.arange(D['n'])))
    elif D['Method']=='Lagrange':
        p = lambda x: np.dot(D['PrecomputedData'],[np.prod(x - np.delete(D['x'],j)) for j in range(D['n'])])
    elif D['Method']=='Barycentric':
        wi = D['PrecomputedData']
        xi = D['x']
        yi = D['y']
        p = lambda x: np.sum(yi*wi/(x-xi))/np.sum(wi/(x-xi)) if len(xi[xi == x])==0 else np.ndarray.item(yi[xi == x])
        # We add these three alternatives but they are analyzed in the file 'Bonus - 05 - Sinc and piecewiselinear interpolations'.
    elif D['Method']=='Newton':
        p = lambda x: np.dot(D['PrecomputedData'],np.append([1],[np.prod(x-D['x'][:j]) for j in range(1,D['n'])]))
    elif D['Method']=='SincInterpolator':
        p = lambda x: np.dot([np.sinc((x-xi)/D['h']) for xi in D['x']],D['y'])
    elif D['Method']=='PiecewiseLinear':
        m = D['m']
        n = D['n']
        xi = D['x']
        yi = D['y']
        def p(x):
            for i in range(n-1):
                if xi[i]<= x and x<xi[i+1]:
                    return yi[i]+m[i]*(x-xi[i])
            # This is the last case, if x==x[-1]
            return yi[-1]
    return np.array([p(x) for x in xx])

# Function that shows the data points and the function that interpolates them.
def PlotInterpolant(D,ylim=None):
    xi = np.linspace(min(D['x']),max(D['x']),1000)
    yi = EvaluateInterpolant(D,xi)
    fig = plt.figure(figsize=(8,8))
    plt.plot(D['x'],D['y'],'ro',label='Interpolation points')
    plt.plot(xi,yi,'b-',label='$p(x)$')
    plt.xlim(min(xi)-0.5, max(xi)+0.5)
    if ylim:
        plt.ylim(ylim[0], ylim[1])
    else:
        plt.ylim(min(yi)-0.5, max(yi)+0.5)
    plt.grid(True)
    plt.legend(loc='best')
    plt.xlabel('$x$')
    plt.show()
```

## Vandermonde Matrix
[Back to TOC](#toc)

First, we are going to learn the Vandermonde matrix method.
This is a $n \times n$ matrix (with $n$ being the cardinality of the set of known data points) with the terms of a geometric progression in each row.
It allows us to construct a system of linear equations with the objective of find the coefficients of the polynomial function that interpolates our data.

Example:

Given the set of known data points: $(x_1,y_1),(x_2,y_2),(x_3,y_3)$

Our system of linear equations will be:
$$ \begin{bmatrix}
    1 & x_1 & x_1^2  \\[0.3em]
    1 & x_2 & x_2^2  \\[0.3em]
    1 & x_3 & x_3^2  \end{bmatrix}
    \begin{bmatrix}
    a_0  \\[0.3em]
    a_1  \\[0.3em]
    a_2  \end{bmatrix} =
    \begin{bmatrix}
    y_1  \\[0.3em]
    y_2  \\[0.3em]
    y_3  \end{bmatrix}$$
And by solving it we will find the coefficients $a_0,a_1,a_2$ that we need to construct the polynomial, $$p(x)=a_0+a_1\,x+a_2\,x^2,$$
that interpolates our data.

```python
def Vandermonde(x, y, show=False):
    # We construct the matrix and solve the system of linear equations
    # A = np.array([xi**np.arange(len(x)) for xi in x]) # OLDER VERSION
    A = np.vander(x, increasing=True)
    b = y
    coefficients = np.linalg.solve(A,b)
    n = len(x)

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(n)])
        print('A = '); print(np.array_str(A, precision=2, suppress_small=True))
        print("cond(A) = "+str(np.linalg.cond(A)))
        print('b = '); print(np.array_str(b, precision=2, suppress_small=True))
        print('x = '); print(np.array_str(coefficients, precision=2, suppress_small=True))
        xS = sym.Symbol('x')
        F = np.dot(xS**np.arange(len(x)),coefficients)
        print('Interpolation Function: ')
        print('F(x) = ')
        print(F)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'Method': 'Vandermonde',
         'PrecomputedData': coefficients,
         'x': x,
         'y': y,
         'n': n}
    return D
```

The following example shows the output of a polynomial interpolation with the provided data.
We use an epsilon value on the thrid interpolation point ($x_3=3+\varepsilon$) and then we control it with a intercat widget, this way we can evaluate what will happen to the interpolation when two collocation points gets close to each other.
This behavior is obtained as soon as the third point is to close to $x_2=2$ by setting the value of $\varepsilon$ equal to $-1$, or making it close to the forth point $x_4$ by setting $\varepsilon$ equal to $2$.

```python
def show_time_Vandermonde(epsilon=0):
    x = np.array([0.0,2.0,3.0+epsilon,4.0,5.0,6.0])
    y = np.array([1.0,3.0,0.0,6.0,8.0,4.0])
    D = Vandermonde(x,y,True)
    PlotInterpolant(D)
interact(show_time_Vandermonde,epsilon=(-1,1,0.1))
```

    interactive(children=(FloatSlider(value=0.0, description='epsilon', max=1.0, min=-1.0), Output()), _dom_classe…

    <function __main__.show_time_Vandermonde(epsilon=0)>

As the numerical experiment show, the interpolated polynomial shows larger and larger oscilations as two points get close to each other.
Notice however that the function still interpolates the data.

### Is the Vandermonde matrix really ill-conditioned?

[Back to TOC](#toc)

```python
N=20
my_range = np.arange(4,N+4)
F_errors=np.zeros(N)
B_errors=np.zeros(N)
kappas=np.zeros(N)

np.random.seed(0)

for i,n in zip(np.arange(N),my_range):
    # Generating 'n' equalspaced points on [0,100]
    x = np.linspace(0,100,n)
    # Building Vandermonde matrix
    V = np.vander(x, increasing=True)
    # Generating a random solution
    coefficients_exact = np.random.random(n)
    # Building the RHS for the linear system of equations to be solved
    b = V @ coefficients_exact
    # Solving the associated linear system of equations
    coefficients = np.linalg.solve(V,b)
    # Computing Forward error:  ||coefficients_exact-coefficients||_2/||coefficients_exact||_2
    F_errors[i] = np.linalg.norm(coefficients-coefficients_exact)/np.linalg.norm(coefficients_exact)
    # Computing Backward error: ||b-A*coefficients||_2/||b||_2
    B_errors[i] = np.linalg.norm(b - V @ coefficients)/np.linalg.norm(b)
    # Computing the condition number
    kappas[i]   = np.linalg.cond(V,2)

plt.figure(figsize=(5,10))
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5), sharey = False)

ax1.semilogy(my_range, F_errors, marker='o', linestyle='--', color='r',label=r'Forward error: $||\mathbf{x}-\mathbf{x}_a||_2/||\mathbf{x}||$')
ax1.semilogy(my_range, B_errors, marker='o', linestyle='--', color='g',label=r'Backward error: $||\mathbf{b}-A\,\mathbf{x}_a||_2/||\mathbf{b}||$')
ax1.semilogy(my_range, kappas*B_errors, marker='o', linestyle='--', color='m', label=r'$\kappa(A_n)\,||\mathbf{b}-A\,\mathbf{x}_a||_2/||\mathbf{b}||$')
ax1.set_title('Errors')
ax1.grid(True)
ax1.set_xlabel('$n$')
ax1.legend(loc='best')
ax1.set_xticks(my_range)
ax1.set_xticklabels(my_range)

ax2.semilogy(my_range, kappas, marker='s', linestyle='--', color='k',label=r'$\kappa(V_n)$')
ax2.set_title(r'$\kappa(V_n)$')
ax2.set_xlabel('$n$')
ax2.grid(True)
ax2.set_xticks(my_range)
ax2.set_xticklabels(my_range)
plt.show()
```

What we see in the previous plots is the fact that solving a linear system with an ill condition matrix, like the Vandermonde matrix, it is definitely not a good idea.
Suggestion: What about if you interpolate the constant $1$ with the Vandermonde matrix, which we know that the exact solution will be the $p(x)=1$.
This means that the constant term will be $1$ and all the other terms must be $0$.
Do you observe the same behavior?

## Lagrange Interpolation
[Back to TOC](#toc)

With this method, we can interpolate data thanks to the Lagrange basis polynomials. Given a set of $n$ data points $(x_1,y_1),...,(x_n,y_n)$, the Lagrange interpolation polynomial is the following:
$$ p(x) = \sum^n_{i=1} y_i\,L_i(x),$$
where $L_i(x)$ are the Lagrange basis polynomials:
$$ L_i(x) = \prod^n_{j=1,j \neq i} \frac{x-x_j}{x_i-x_j} = \frac{x-x_1}{x_i-x_1} \cdot ... \cdot \frac{x-x_{i-1}}{x_i-x_{i-1}} \cdot \frac{x-x_{i+1}}{x_i-x_{i+1}} \cdot ... \cdot \frac{x-x_n}{x_i-x_n}$$
or simply $L_i(x)=\dfrac{l_i(x)}{l_i(x_i)}$, where $l_i(x)=\displaystyle{\prod^n_{j=1,j \neq i} (x-x_j)}$.
The key properties of these basis polynomials are the followings:
$$ L_i(x_i) = 1 $$
$$ L_{j \neq i}(x_i) = 0 $$
So, we assure that $L(x_i) = y_i$, which indeed interpolates the data.

```python
def Lagrange(x, y, show=False):
    # We calculate the li's
    n = len(x)
    p = np.array([y[i]/np.prod(x[i] - np.delete(x,i)) for i in range(n)])

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(n)])
        xS = sym.Symbol('x')
        L = np.dot(np.array([np.prod(xS - np.delete(x,i))/np.prod(x[i] - np.delete(x,i)) for i in range(n)]),y)
        print('Interpolation Function: ');
        print(L)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'Method': 'Lagrange',
         'PrecomputedData': p,
         'x': x,
         'y': y,
         'n': n}
    return D
```

The following numerical experiment follows the same idea of the previous numerical experiment with the Vandermonde matrix.

```python
def show_time_Lagrange(epsilon=0):
    x = np.array([0.0,2.0,3.0+epsilon,4.0,5.0,6.0])
    y = np.array([1.0,3.0,0.0,6.0,8.0,4.0])
    D = Lagrange(x,y,True)
    PlotInterpolant(D)
interact(show_time_Lagrange,epsilon=(-1,1,0.1))
```

    interactive(children=(FloatSlider(value=0.0, description='epsilon', max=1.0, min=-1.0), Output()), _dom_classe…

    <function __main__.show_time_Lagrange(epsilon=0)>

### Plotting $L_i(x)$
[Back to TOC](#toc)

The next cell shows what $L_i(x)$ looks like.
It basically shows how a polynomial looks like when it is one at $x_i$ and zero at all the other interpolation nodes.

```python
def show_time_Li(i=0, N=7):
    x = np.arange(N+1)
    y = np.zeros(N+1)
    y[i]=1
    D = Lagrange(x,y,True)
    PlotInterpolant(D,[-1,2])

i_widget = IntSlider(min=0, max=7, step=1, value=0)
N_widget = IntSlider(min=1, max=20, step=1, value=7)
def update_i_range(*args):
    i_widget.max = N_widget.value
N_widget.observe(update_i_range, 'value')

interact(show_time_Li,i=i_widget,N=N_widget)
```

    interactive(children=(IntSlider(value=0, description='i', max=7), IntSlider(value=7, description='N', max=20, …

    <function __main__.show_time_Li(i=0, N=7)>

The following example show all the $L_i(x)$ for $i\in\{1,2,\dots,N\}$ at once.
This was included thank to an inclass question!

```python
def show_time_all_Li(N=3, ShowLegend=False):
    x = np.arange(1,N+1)
    xx = np.linspace(1,N,1000)
    plt.figure()
    for i in range(N):
        y = np.zeros(N)
        y[i]=1
        D = Lagrange(x,y)
        yy=EvaluateInterpolant(D, xx)
        plt.plot(xx,yy,label=r'$L_{0}(x)$'.format(i+1))
    if ShowLegend:
        plt.legend(loc='best')
    plt.grid(True)
    plt.show()
interact(show_time_all_Li,N=(2,20,1), ShowLegend=Checkbox(value=False, description="Include legend"))
```

    interactive(children=(IntSlider(value=3, description='N', max=20, min=2), Checkbox(value=False, description='I…

    <function __main__.show_time_all_Li(N=3, ShowLegend=False)>

Here you get some questions about Lagrange Interpolation:
- Explain what happens with the interpolator polynomial when you add a new point to the set of points to interpolate. **Answer: We need to recalculate the polynomial**
- Why it is not a good idea to use Lagrange interpolation for a set of points which is constantly changing? **A: Because we need to compute the whole interpolation again**
- What is the operation count of obtaining the interpolator polynomial using Lagrange? What happens with the error?

#### As an additional example we show the $\text{sinc}(x)$ function.
This function is a relative of $L_i(x)$ from Lagrange interpolation, the only difference is that it is one at one point and zero over an infinite grid!
This could be also use for interpolation, you only need to shift it a bit and you are good to go!
This is not an polynomial interpolation but it may be a good alternative.

```python
xD = np.arange(-10,11)
yD = np.zeros_like(xD)
yD[10] = 1
xi = np.linspace(-10,10,1000)
yi = np.sinc(xi)
plt.figure(figsize=(8,8))
plt.plot(xD,yD,'ro',label='Interpolation points')
plt.plot(xi,yi,'b-',label='sinc$(x)$')
plt.xlim(min(xi)-0.5, max(xi)+0.5)
plt.ylim(min(yi)-0.5, max(yi)+0.5)
plt.grid(True)
plt.legend(loc='best')
plt.xlabel('$x$')
#plt.ylabel('$P(x)$')
plt.show()
```

### Bonus: Sinc and Piecewise-linear interpolations
We add these interpolations based on the following question from class 2021-2: why don't we just use a linear or a sinc interpolator?

**See file 'Bonus - 05 - Newton's divided differences, Sinc and piecewiselinear interpolations' for a complete analysis.**

## Barycentric interpolation
[Back to TOC](#toc)

The barycentric interpolation for interpolation $n$ data points $(x_1,y_1),...,(x_n,y_n)$ is of the following form:
$$ p(x) = \dfrac{\displaystyle{\sum_{i=1}^n y_i\dfrac{w_i}{(x-x_i)}}}{\displaystyle{\sum_{i=1}^n \dfrac{w_i}{(x-x_i)}}},$$
where $w_i=\dfrac{1}{l_i(x)}$ and $l_i(x)=\displaystyle{\prod_{k=1,i\neq k}^n(x-x_k)}$.

```python
def Barycentric(x, y, show=False):
    W  = np.subtract.outer(x, x)
    wi  = 1/np.prod(W, axis=1, where = W!=0)
    n = len(x)

    # The function shows the data if the flag is true
    if show:
        print('Data Points: '); print([(x[i],y[i]) for i in range(n)])
        xS = sym.Symbol('x')
        N = np.sum(y*wi/(xS-x))/np.sum(wi/(xS-x))
        print('Interpolation Function: ');
        print(N)

    # Finally, we return a data structure with our interpolating polynomial
    D = {'Method':'Barycentric',
         'PrecomputedData': wi,
         'x': x,
         'y': y,
         'n': n}
    return D
```

```python
def show_time_Barycentric(epsilon=0):
    x = np.array([0.0,2.0,3.0+epsilon,4.0,5.0,6.0])
    y = np.array([1.0,3.0,0.0,6.0,8.0,4.0])
    D = Barycentric(x,y,True)
    PlotInterpolant(D)
interact(show_time_Barycentric,epsilon=(-1,1,0.1))
```

    interactive(children=(FloatSlider(value=0.0, description='epsilon', max=1.0, min=-1.0), Output()), _dom_classe…

    <function __main__.show_time_Barycentric(epsilon=0)>

## Polynomial Interpolation Error
[Back to TOC](#toc)

The interpolation error is given by:

$$ f(x)-p(x) = \frac{(x-x_1) \, (x-x_2) \, \dots \, (x-x_n)}{n!} \, f^{(n)}(c),$$

where $c$ is within the interval defined by $[\min(x,x_1,\dots,x_n),\max(x,x_1,\dots,x_n)]$.

```python
def Error(f, n, xmin, xmax, method=Lagrange, points=np.linspace, plot_flag=True):
    # This function plots f(x), the interpolating polynomial, and the associated error
    # points can be np.linspace to equidistant points or Chebyshev to get Chebyshev points

    # These two lines generate the data that will be used in the interpolation
    x = points(xmin,xmax,n)
    y = f(x)

    # These two lines compute the exact value of the function being interpolated on a finer grid
    xe = np.linspace(xmin,xmax,100)
    ye = f(xe)

    # This two lines build the interpolant chosen, where the name of the method is
    # passed as parameter usign the variable 'method'.
    D = method(x,y)
    yi = EvaluateInterpolant(D, xe)

    if plot_flag:
        plt.figure(figsize=(5,10))
        f, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5), sharey = False)

        ax1.plot(xe, ye,'k-', label='f(x)')
        ax1.plot(x, y,'ro', label='Interpolation points',markersize=10)
        ax1.plot(xe, yi,'b-', label='Interpolation')
        ax1.set_xlim(xmin-0.5,xmax+0.5)
        ax1.set_ylim(min(yi)-0.5,max(yi)+0.5)
        ax1.set_title('Interpolation')
        ax1.grid(True)
        ax1.set_xlabel('$x$')
        ax1.legend(loc='best')

        ax2.semilogy(xe, abs(ye-yi),'c-', label='Absolute Error')
        ax2.set_xlim(xmin-0.5,xmax+0.5)
        ax2.set_title('Absolute Error')
        ax2.set_xlabel('$x$')
        ax2.grid(True)
        plt.show()

    return max(abs(ye-yi))
```

```python
def test_error_Lagrange(n=5):
    # Example 1
    #me = Error(lambda x: np.sin(x)**3, n, 1, 7, Lagrange)
    # Example 2
    me = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, Lagrange)
    print("Max Error:", me)
```

```python
interact(test_error_Lagrange,n=(5,25))
```

    interactive(children=(IntSlider(value=5, description='n', max=25, min=5), Output()), _dom_classes=('widget-int…

    <function __main__.test_error_Lagrange(n=5)>

##  **Runge's Phenomenon**: It is a problem of oscillation of polynomials at the edges of the interval.
[Back to TOC](#toc)

We are interpolating a data that is 0 almost everywhere and 1 at the middle point.
Notice that as $n$ increases the oscilations increase and all the red dots seems to be at 0 in the plot but this is just an artifact due to the scale on the left.
The oscillations you see at the end of the interval is the Runge phenomenon.
This is due to the use of equalspaced point.

```python
def Runge(n=9):
    x = np.linspace(0,1,n)
    y = np.zeros(n)
    y[int((n-1.0)/2.)]=1
    D = Barycentric(x,y,False)
    PlotInterpolant(D)
interact(Runge,n=(5,25,2))
```

    interactive(children=(IntSlider(value=9, description='n', max=25, min=5, step=2), Output()), _dom_classes=('wi…

    <function __main__.Runge(n=9)>

## Interpolation with Chebyshev points
[Back to TOC](#toc)

From the interpolation error defined previously, we notice on the right hand side that the only term we can actually control (besides $n$) is the actual value used for the $x_i$, for $i\in\{1,2,\dots,n\}$.
Thus, with the objective of reducing the error of the polynomial interpolation, we need to find the values of $x_1,x_2,...,x_n$ that minimize the following expression:
$$(x-x_1) \, (x-x_2) \, ... \, (x-x_n).$$

To choose these values we select the canonical interval $[-1,1]$, in particular, we expect the points to be in $-1 \leq x_1,x_2,...,x_n \leq 1$ (to use another interval we just need to do a change of variables).
This probles was already studied by Chebyshev, thus we will use the roots of the Chebyshev polynomials, also called **Chebyshev nodes** (of the first kind), which are defined by:

$$ x_i = \cos\left(\frac{(2i-1)\pi}{2n}\right), i\in\{1,2,\dots,n\} $$

```python
def Chebyshev(xmin,xmax,n=5):
    # This function calculates the n Chebyshev points and plots or returns them depending on ax
    ns = np.arange(1,n+1)
    x = np.cos((2*ns-1)*np.pi/(2*n))
    y = np.sin((2*ns-1)*np.pi/(2*n))
    plt.figure(figsize=(10,5))
    plt.ylim(-0.1,1.1)
    plt.xlim(-1.1,1.1)
    plt.plot(np.cos(np.linspace(0,np.pi)),np.sin(np.linspace(0,np.pi)),'k-')
    plt.plot([-2,2],[0,0],'k-')
    plt.plot([0,0],[-1,2],'k-')
    for i in range(len(y)):
        plt.plot([x[i],x[i]],[0,y[i]],'r-')
        plt.plot([0,x[i]],[0,y[i]],'r-')
    plt.plot(x,[0]*len(x),'bo',label='Chebyshev points')
    plt.plot(x,y,'ro')
    plt.xlabel('$x$')
    plt.title('n = '+str(n))
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()
def Chebyshev_points(xmin,xmax,n):
    ns = np.arange(1,n+1)
    x = np.cos((2*ns-1)*np.pi/(2*n))
    #y = np.sin((2*ns-1)*np.pi/(2*n))
    return (xmin+xmax)/2 + (xmax-xmin)*x/2
def Chebyshev_points_histogram(n=50,nbins=20):
    xCheb=Chebyshev_points(-1,1,n)
    #plt.figure()
    #plt.hist(xCheb,bins=nbins,density=True,cumulative=True)
    #plt.grid(True)
    #plt.show()

    plt.figure(figsize=(5,10))
    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5), sharey = False)
    ax1.hist(xCheb,bins=nbins,density=True)
    ax1.grid(True)
    ax2.hist(xCheb,bins=nbins,density=True,cumulative=True)
    ax2.grid(True)
    plt.show()
```

The next plot shows a graphical interpretation of the Chebyshev nodes, in particular, they are the projection on the $x$-axis from equalspaced point on the unit sphere.
The angle used for defining the equalspaced point on the unit sphere are: $\theta_i=\frac{(2i-1)\pi}{2n}$, for $i\in\{1,2,\dots,n\}$.

```python
interact(Chebyshev,xmin=fixed(-1),xmax=fixed(1),n=(2,50))
```

    interactive(children=(IntSlider(value=5, description='n', max=50, min=2), Output()), _dom_classes=('widget-int…

    <function __main__.Chebyshev(xmin, xmax, n=5)>

The next plot shows an histogram to explicitly show that the Chebychev nodes cluster at each end of the interval.
The left plot is an explicit histogram and the plot on the right is the cumulative distribution function.

```python
interact_manual(Chebyshev_points_histogram,n=(20,10000),nbins=(20,200))
```

    interactive(children=(IntSlider(value=50, description='n', max=10000, min=20), IntSlider(value=20, description…

    <function __main__.Chebyshev_points_histogram(n=50, nbins=20)>

Moreover, these points have the an explicit expresion for the term we want to minimize,
this means,
$$ (x-x_1) \, (x-x_2) \, ... \, (x-x_n) = \dfrac{1}{2^{n-1}} \, T_n(x), $$
where $T_n(x) = \cos (n \, \arccos (x))$ is the n-th Chebyshev polynomial.
$$
\begin{align*}
T_0(x) &= 1, \\
T_1(x) &= x, \\
T_2(x) &= 2x^2 -1,\\
\vdots
\end{align*}
$$

It also has a recurrence relation,
$$ T_{n+1}(x) = 2 \cdot x \cdot T_n(x) - T_{n-1}(x).$$
The importance of the first identity showed in this cell is that we can bound the left-hand-side when using Chebychev points.
This is not possible when using equalspaced points.
For instance, when using Chebychehv points we have the following bound,
$$|(x-x_1) \, (x-x_2) \, ... \, (x-x_n)| = \left|\dfrac{1}{2^{n-1}} \, T_n(x)\right|\leq \dfrac{1}{2^{n-1}}.$$
This is true since $T(x) = \cos (n \, \arccos (x))$ and $|\cos(x)|\leq 1$.
This is useful when one wants to find the number of points required to interpolate a funcion to a required error.
In this case the error would become,
$$
\begin{align*}
|f(x)-p(x)| &= \left|\frac{(x-x_1) \, (x-x_2) \, \dots \, (x-x_n)}{n!} \, f^{(n)}(c)\right|\\
            &\leq \frac{1}{2^{n-1}\,n!} \, \left|f^{(n)}(c)\right|,
\end{align*}
$$
which is very nice!
We only need to pay attention to find a bound for $\left|f^{(n)}(c)\right|$ on the interval we want to interpolate the function.

In the next cell, for completeness, we plot the Chebyshev polynomials $T_n(x)$.

```python
# Recursive function that returns the n-th Chebyshev polynomial evaluated at x
def T(n,x):
    if n == 0:
        return x**0
    elif n == 1:
        return x
    else:
        return 2*x*T(n-1,x)-T(n-2,x)

# This function plots the first n Chebyshev polynomials
def Chebyshev_Polynomials(n=2, Flag_All_Tn=False):
    x = np.linspace(-1,1,1000)
    plt.figure(figsize=(10,5))
    plt.xlim(-1, 1)
    plt.ylim(-1.1, 1.1)
    if Flag_All_Tn:
        for i in np.arange(n+1):
            y = T(i,x)
            plt.plot(x,y,label='$T_{'+str(i)+'}(x)$')
    else:
        y = T(n,x)
        plt.plot(x,y,label='$T_{'+str(n)+'}(x)$')
    plt.legend(loc='right')
    plt.grid(True)
    plt.xlabel('$x$')
    plt.show()
```

```python
interact(Chebyshev_Polynomials,n=(0,12),Flag_All_Tn=True)
```

    interactive(children=(IntSlider(value=2, description='n', max=12), Checkbox(value=True, description='Flag_All_…

    <function __main__.Chebyshev_Polynomials(n=2, Flag_All_Tn=False)>

The follwing function compares the error obtain by interpolation a function with equalspaced points and Chebychev points.

```python
def test_error_Chebyshev(n=5):
    mee = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, Lagrange)
    mec = Error(lambda x: (1/(1+12*x**2)), n, -1, 1, method=Lagrange, points=Chebyshev_points)
    print("Max error (equidistants points):", mee)
    print("Max error (Chebyshev nodes):", mec)
```

The plots on the left show the actual function and the corresponding interpolation.
The plots in the right show the error in $\log_{10}$ scale.
Notice that we interpolation polynomial goes by the interpolation nodes exactly, this induce an error equal to $0$ and these nodes.
This behaviour explain the down spikes on the logaritmic scales, but we care about the worst case.
In the worst case the equalspaced polynomial behaves very badly, this is way polynomial interpolation with Chebyshev points is so useful.

```python
interact(test_error_Chebyshev,n=(5,100,2))
```

    interactive(children=(IntSlider(value=5, description='n', min=5, step=2), Output()), _dom_classes=('widget-int…

    <function __main__.test_error_Chebyshev(n=5)>

Questions about Chebyshev:
- How can you calculate the Chebyshev points in the interval [a,b] instead of [-1,1]? **A: Using a change of variable**

## Convergence analysis

The next set of experiments show a convergence analysis comparing equalspaced points to Chebychev points.
The $x$-axis is the number of points used and the $y$-axis is the maxmimum error obtain on the interval in logarithmic scale.
The experiment are the followings:
1. $f(x)=x^{10}$. In this case both sets of points perform similarly, however there is a significant drop in the error when $n$ is equal to $11$. This means, the error reach a minimum of $10^{-14}$ app at that number points. Why?
2. $f(x)=|x^3|$. This looks like a naive function but it is not. The equalspace points start to decrease from an error of $10^{0}$ until an error app $10^{-3}$ with approximately 10 points, it then start to increase the error. On the other hand, the Chebychev points show a decresing behavior upto $n=50$ but it only reaches an error of app $10^{-5}$, which is too high. What is the issue with this funcion?
3. $f(x)=\exp(-(x^{-2}))$. This is also a tricky function bevcause involves a division by $0$ when we evaluate it at $x=0$, but it is well defined at that points. Similarly to the first example, the equalspaced points reduce the error upto $n=10$, and then start to increase. For the Chebychev points, they decrease upto 40 points and the also show a slight increment in the error. What is the value of $f(x)$ at $x=0$?
4. $f(x)=\dfrac{1}{1+x^2}$. In this case both cases decrease the error but faster for the Chebyshev points. The main difference is that the minimum error reached for equalspaced points is about $10^{-8}$ and $10^{-13}$ for the Chebyshev points, in both cases about $n=37$.
5. $f(x)=\sin^3(x)$. In this case both cases reached an error $10^{-14}$ with $n=20$, then for equalspaced points start to increase the error right away but for Chebyshev points it starts to increase at $n=40$.

Answers to the previous questions:
1. The reason is that when $n=11$ we will be interpolation an polynomial of degree $10$ with a polynomial of degree $10$, so we can reproduce it exactly! This is why the error is $0$. When $n$ was lower, we only had an approximation. It is interesting that the error increases as we increase the number of points after $n=10$. Would it be useful to change the algorihtm used?
2. The issue is that the function is not analytic, this is because it involves the absolute value. Recall that expression for the error requires the computation of the $n$-th derivative of $f(x)$ at a $c$ point, so if the function is not differentiable, in this case the issues is at $x=0$, we could expect a bad behavior. The probles can be easily solve if we interpolate the function with a polynomial defined on $[-1,0]$ and another on $[0,1]$, ensurring the are equal at $x=0$. In this case we only need two polynomials of degree $3$ and we will be able to reproduce the function exactly! The issue is to notice this when we only have access to the computational implementation of $f(x)$ and not to the function itself.
3. 0
4. -
5. -

**We strongly suggest you to try different functions and play around!**

```python
my_functions=[lambda x: x**10,
              lambda x: np.abs(x**3),
              lambda x: np.exp(-(x**-2)),
              lambda x: 1/(1+x**2),
              lambda x: np.sin(x)**3]
labels = ["x^{10}",
          "|x^3|",
          "\exp(-x^{-2})",
          "1/(1+x^2)",
          "\sin^3(x)"]
data=zip(labels,my_functions)

radio_button_function=RadioButtons(
    options=list(data),
    description='Function:',
    disabled=False
)

radio_button_interpolation_algorithm=RadioButtons(
    options=[('Vandermonde',Vandermonde),('Lagrange',Lagrange),('Barycentric',Barycentric)],
    value=Lagrange,
    description='Algorithm:',
    disabled=False
)

def convergence_study(f, method):
    n=70
    shift=2
    n_points=np.arange(shift,n)
    max_error=np.zeros(n-shift)
    max_error_es=np.zeros(n-shift)
    for i in n_points:
        max_error[i-shift] = Error(f, i, -1, 1, method, Chebyshev_points, plot_flag=False)
        max_error_es[i-shift] = Error(f, i, -1, 1, method, points=np.linspace, plot_flag=False)
    axis=plt.figure(figsize=(8,8))
    plt.semilogy(n_points,max_error,'bd',label='Chebyshev points',markersize=10)
    plt.semilogy(n_points,max_error_es,'r.',label='Equalspaced poins',markersize=16)
    plt.ylim(10**-16,10**4)
    plt.grid(True)
    plt.title('Interpolation Error')
    plt.xlabel('Number of points used in the interpolation')
    plt.ylabel('Max error on domain')
    plt.legend(loc='best')
    plt.show()
interact(convergence_study,f=radio_button_function, method=radio_button_interpolation_algorithm)
```

    interactive(children=(RadioButtons(description='Function:', options=(('x^{10}', <function <lambda> at 0x7fc149…

    <function __main__.convergence_study(f, method)>

## Analysis of time vs error for interpolation algorithms
The idea here is to understand the relationship between the error we get for each algorithm and the computation time associated.
This analysis highlights the advantage of Barycentric interpolation with Chebyshev points when the function being interpolated is analytic.
Notice that the first plot shows the computation time required for the construction of the interpolant, the second the cost of evaluation of the interpolant in a finer grid and the third one shows the sum of both times.

```python
def get_error_and_time(f, xmin=-1, xmax=1, method=Lagrange, points=np.linspace):

    construction_times   = []
    evaluation_times     = []
    interpolation_errors = []

    n_range = np.array([2,4,8,16,30,50])

    for n in n_range:

        # This function plots f(x), the interpolating polynomial, and the associated error
        # points can be np.linspace to equidistant points or Chebyshev to get Chebyshev points

        # These two lines generate the data that will be used in the interpolation
        x = points(xmin,xmax,n)
        y = f(x)

        # These two lines compute the exact value of the function being interpolated on a finer grid
        xe = np.linspace(xmin,xmax,100)
        ye = f(xe)

        # This line build the interpolant chosen, where the name of the method is
        # passed as parameter usign the variable 'method'.
        time_construction = %timeit -o -q -n 10 -r 5 method(x,y)
        construction_times.append(time_construction.average)
        D = method(x,y)

        time_evaluation = %timeit -o -q -n 10 -r 5 EvaluateInterpolant(D, xe)
        evaluation_times.append(time_evaluation.average)
        yi = EvaluateInterpolant(D, xe)

        if max(abs(ye-yi))<1e-16:
            interpolation_errors.append(1e-16)
        else:
            interpolation_errors.append(max(abs(ye-yi)))

    return construction_times, evaluation_times, interpolation_errors

def analysis_of_time_vs_error_of_interpolation_algorihtms(f):
    # List of methods
    list_of_methods = [('Vandermonde',Vandermonde),('Lagrange',Lagrange),('Barycentric',Barycentric)]
    # List of type of points
    list_of_points = [('equalspaced',np.linspace), ('Chebyshev',Chebyshev_points)]
    list_output_data = []
    k = 1
    for name_method, method in list_of_methods:
        for points_name, points in list_of_points:
            # We exclude SincInterpolator for Chebyshev points because it need equalspaced data
            if (name_method=='SincInterpolator' and points_name=='Chebyshev'):
                continue
            tic = time.time()
            c_time, e_time, errors = get_error_and_time(f, xmin=-1, xmax=1, method=method, points=points)
            print(k,'/ 6: ',name_method, ',' , points_name, ', time:',time.time()-tic,'[s]')
            k=k+1
            list_output_data.append([name_method, points_name, c_time, e_time, errors])

    markers = ['8','>','<','^','v','o','d','D','H','h','*','p','s']

    fig = plt.figure(figsize=(16,16))
    plt.rcParams.update({'font.size': 12})

    k = 0
    for data in list_output_data:
        name_method, points_name, c_time, e_time, errors = data

        ax = plt.subplot(2,2,1)
        plt.title('Construction time')
        plt.loglog(errors, c_time, markers[k], lw=2.0, ms=10, label=name_method+', '+points_name)
        ax = plt.subplot(2,2,2)
        plt.title('Evaluation time')
        plt.loglog(errors, e_time, markers[k], lw=2.0, ms=10, label=name_method+', '+points_name)
        ax = plt.subplot(2,2,3)
        plt.title('Construction + evaluation times')
        plt.loglog(errors, np.array(c_time)+np.array(e_time), markers[k], lw=2.0, ms=10, label=name_method+', '+points_name)
        k=k+1

    plt.legend(loc='lower left', ncol=1, fancybox=True, shadow=True, numpoints=1, bbox_to_anchor=(1,0))

    for i in range(3):
        ax = plt.subplot(2,2,i+1)
        plt.xlabel("Absolute Error")
        plt.ylabel("Time in seconds")
        plt.grid('on')
    plt.show()

interact(analysis_of_time_vs_error_of_interpolation_algorihtms,f=radio_button_function)
```

    interactive(children=(RadioButtons(description='Function:', options=(('x^{10}', <function <lambda> at 0x7fc149…

    <function __main__.analysis_of_time_vs_error_of_interpolation_algorihtms(f)>

**Warning**: Notice that this experiment shows a _competitive_ behavior of the Vandermonde matrix but don't forget that the Vandermonde matrix is very ill-conditioned!

## Python Modules and Functions
[Back to TOC](#toc)

Interpolation:

https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html

Vandermonde Matrix:

https://numpy.org/doc/stable/reference/generated/numpy.vander.html

Lagrange Interpolator:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.lagrange.html

Barycentric Interpolator:

https://people.maths.ox.ac.uk/trefethen/barycentric.pdf

https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.BarycentricInterpolator.html

Chebyshev Points for the First Kind:

https://numpy.org/doc/stable/reference/generated/numpy.polynomial.chebyshev.chebroots.html

# Acknowledgements
[Back to TOC](#toc)

* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. April 2016._
* _Material modified by Cristopher Arenas. May 2017._
* _Material modified by Claudio Torres. May 2017._
* _Bug fixed by Cristobal Carmona. April 2018._
* _Update June 2020 - v1.25 - C.Torres_ : Fixing formatting issues.
* _Update June 2020 - v1.26 - C.Torres_ : Adding "ylim" argumento to Interpolation_Plot(D,ylim=None) and addint "show_time_Li".
* _Update June 2020 - v1.27 - C.Torres_ : Adding comment that the Chebyshev nodes used are of the first kind and "Chebyshev_points_histogram".
* _Update May 2021 - v1.28 - C.Torres_ : Improving format, adding cumulative histogram in "Chebyshev_points_histogram", updating Python modules links, implementing Barycentric interpolation, changing functions names: Interpolation_Plot->PlotInterpolant, Y->EvaluateInterpolant; adding field 'n' to data structure 'D' used for building interpolant, several changes everywhere adding more explanations.
* _Update May 2021 - v1.29 - C.Torres_ : Adding "show_time_all_Li" thanks to Felipe Samur question during class. Thanks Felipe! Changing reference tu sympy from 'sp' to 'sym'.
* _Update May 2021 - v1.30 - C.Torres_ : Adding radio buttons to last examples.
* _Update October 2021 - v1.31 - C.Torres_ : Adding Vandermonde matrix analysis about being ill-conditioned.
* _Update October 2021 - v1.32 - C.Torres_ : Fixing typos.
* _Update October 2021 - v1.33 - C.Torres_ : Removing Newton's divided differences and moving it to 'Bonus - 05 - Newton's divided differences, Sinc and piecewiselinear interpolations'.

```python

```


# Analiza i projektiranje računalom - 2. domaća zadaća

U okviru ove domaće zadaće potrebno je implementirati sljedeće metode optimiranja:

## Postupak zlatnog reza

Algoritam je dan na [web stranici predmeta](http://www.fer.unizg.hr/_download/repository/zlatni_rez.txt) i na predavanjima. U okviru ovog postupka potrebno je također implementirati metodu pronalaženja unimodalnog intervala (**napomena**: *kao što je pokazano na predavanjima, po [ovome](http://www.fer.unizg.hr/_download/repository/unimodalni.txt)*). Postupak zlatnog reza mora se moći pokrenuti ili s početnom točkom, s pomoću koje se tada određuje unimodalni interval, ili s predefiniranim intervalom, u kojem slučaju se preskače postupak traženja unimodalnog intervala i koristi zadani interval. U svakom koraku postupka potrebno je omogućiti ispis sve četiri točke i vrijednosti funkcije cilja u njima. Potrebno je omogućiti da se bez prevođenja programa mogu definirati preciznost *e* te početna točka ili početni interval pretraživanja (npr.  učitavanjem iz datoteke ili ulaznog toka). Pretpostavljene vrijednosti: $10^{-6}$ za *e*. (**napomena**: *za iznos konstante k uzmite što precizniju vrijednost, računanjem kao* $0.5 \left( \sqrt{5} - 1 \right)$.

## Pretraživanje po koordinatnim osima

Algoritam je opisan u skripti (str. 4-33, 4-34) i na predavanjima. U sklopu ove metode iskoristite metodu zlatnog reza za minimizaciju u jednoj dimenziji. Potrebno je omogućiti da se bez prevođenja programa mogu definirati vektor granične preciznosti $\vec{e}$ te početna točka pretraživanja (npr. učitavanjem iz datoteke). Pretpostavljene vrijednosti: $10^{-6}$ za sve elemente $\vec{e}$.

## Simpleks postupak po Nelderu i Meadu

lgoritam je dan na [web stranici predmeta](http://www.fer.unizg.hr/_download/repository/simplex.html). Početni simpleks generirajte na način da se za jednu konkretnu novu točku pomaknete od početne točke za određeni pomak u svakoj dimenziji (**primjer**: *ako je početna točka $\begin{bmatrix}0 & 0\end{bmatrix}$ i pomak $1$, tada ćete generirati točke $\begin{bmatrix}1 & 0\end{bmatrix}$ i $\begin{bmatrix}0 & 1\end{bmatrix}$*). U svakom koraku postupka potrebno je ispisati centroid točaka i vrijednost funkcije cilja u njoj. Potrebno je omogućiti da se bez prevođenja mogu definirati početna točka pretraživanja, preciznost *e*, pomak koji se koristi za generiranje simpleksa, parametri $\alpha$, $\beta$, $\gamma$ te vrijednost za koju se sve točke pomiču prema najboljoj točki simpleksa ($\sigma$). Pretpostavljene vrijednosti za parametre: $10^{-6}$ za *e*, $1$ za pomak koji se koristi kod početnog generiranja simpleksa, $\alpha = 1$, $\beta = 0.5$, $\gamma = 2$ i $\sigma = 0.5$.

### Postupak Hooke-Jeeves

Algoritam je dan na [web stranici predmeta](http://www.fer.hr/_download/repository/hj.html) i opisan na predavanjima (**napomena**: *ne implementirati algoritam u skripti*, koji ima nešto drugačije ponašanje!). U svakom koraku postupka potrebno je ispisati baznu točku, početnu točku pretraživanja i točku dobivenu pretraživanjem, kao i vrijednosti funkcije cilja u tim točkama. Potrebno je omogućiti da se bez prevođenja programa mogu definirati: vektor pomaka $\vec{dx}$, vektor granične preciznosti $\vec{e}$ te početna točka pretraživanja (npr. učitavanjem iz datoteke). Pretpostavljene vrijednosti: $0.5$ za sve elemente $\vec{dx}$ te $10^{-6}$ za sve elemente $\vec{e}$.

---

Svi algoritmi (osim postupka zlatnog reza) moraju podržavati proizvoljno veliku dimenzionalnost rješenja. Funkciju cilja potrebno je implementirati tako da vodi evidenciju o broju pozivanja, jer će algoritme biti potrebno usporediti na temelju broja evaluacija (preporučuje se ostvariti kao apstraktni razred).

## Funkcije cilja

### Funkcija 1 - Rosenbrockova "banana" funkcija

$$
\begin{equation}
    f_1 \left( \vec{x} \right) = 100 \left( x_1 - {x_0}^2 \right)^2 + \left( 1 - x_0 \right)^2
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        -1.9 & 2
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        1 & 1
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_1 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 2

$$
\begin{equation}
    f_2 \left( \vec{x} \right) = \left( x_0 - 4 \right)^2 + 4 \left( x_1 - 2 \right)^2
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        -0.1 & 0.3
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        4 & 2
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_2 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 3

$$
\begin{equation}
    f_3 \left( \vec{x} \right) = \sum_i{\left( x_i - i \right)^2}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} = \vec{0}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        1 & 2 & 3 & \dots & n
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_3 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 4 - Jakobovićeva funkcija

$$
\begin{equation}
    f_4 \left( \vec{x} \right) = \left| \left( x_0 - x_1 \right) \left( x_0 + x_1 \right) \right| + \sqrt{{x_0}^2 + {x_1}^2}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_0} =
    \begin{bmatrix}
        5.1 & 1.1
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} =
    \begin{bmatrix}
        0 & 0
    \end{bmatrix}
\end{equation}
$$

$$
\begin{equation}
    f_4 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

### Funkcija 6 - Schafferova F6 funkcija

$$
\begin{equation}
    f_6 \left( \vec{x} \right) = 0.5 + \frac{%
        sin^2 \left( \sqrt{\sum{{x_i}^2}} \right) - 0.5
    }{%
        \left( 1 + 0.001 \sum{{x_i}^2}\right)^2
    }
\end{equation}
$$

$$
\begin{equation}
    \vec{x_{min}} = \vec{0}
\end{equation}
$$

$$
\begin{equation}
    f_6 \left( \vec{x_{min}} \right) = 0
\end{equation}
$$

## Priprema za izvođenje

```python
import os

CD_KEY = "--HW02_IN_ROOT"
```

```python
if (
    CD_KEY not in os.environ
    or os.environ[CD_KEY] is None
    or len(os.environ[CD_KEY]) == 0
    or os.environ[CD_KEY] == "false"
):
    %cd ..
else:
    print(os.getcwd())

os.environ[CD_KEY] = "true"
```

    /mnt/data/projekti/faks/AIPR/dz/dz-02

## Učitavanje paketa

```python
import copy
import sys

import array_to_latex as a2l
from IPython.display import display, Markdown
from matplotlib import pyplot as plt
import numpy as np
from tqdm import tqdm

from src.searches.axis import axis_search
from src.searches.function import Function
from src.searches.golden_section import golden_section_search
from src.searches.hooke_jeeves import (
    hooke_jeeves_search, get_survey_points
)
from src.searches.nelder_mead import nelder_mead_simplex_search
from src.utils.function import get_f3, get_f3_scalar
```

## Definicija pomoćnih funkcija

```python
def array_to_latex(array):
    return a2l.to_ltx(
        array,
        frmt="{:g}",
        arraytype="bmatrix",
        print_out=False
    )
```

## Definicija funkcija i početnih točaka

```python
f1 = Function(lambda x: 100 * (x[1] - x[0] ** 2) ** 2 + (1 - x[0]) ** 2)
f1_start = np.array([-1.9, 2])
```

```python
f2 = Function(lambda x: (x[0] - 4) ** 2 + 4 * (x[1] - 2) ** 2)
f2_start = np.array([0.1, 0.3])
```

```python
f3 = Function(get_f3(1))
```

```python
f4 = Function(lambda x: np.abs((x[0] - x[1]) * (x[0] + x[1])) + np.sqrt(x[0] ** 2 + x[1] ** 2))
f4_start = np.array([5.1, 1.1])
```

```python
f6 = Function(
    lambda x: (
        0.5
        + ((np.sin(np.sqrt(np.sum(x ** 2)))) ** 2 - 0.5)
        / ((1 + 0.001 * np.sum(x ** 2)) ** 2)
    )
)
```

## Zadatci

### Zadatak 1

Definirajte jednodimenzijsku funkciju br. 3, koja će imati minimum u točki 3.

```python
t1_function = Function(get_f3(3))
t1_function_scalar = Function(get_f3_scalar(3))
```

Kao početnu točku pretraživanja postavite točku 10.

```python
t1_start = np.array([10])
t1_start_scalar = 10
```

Primijenite sve postupke na rješavanje ove funkcije te ispišite pronađeni minimum i broj evaluacija funkcije za svaki pojedini postupak.

#### Zlatni rez

```python
t1_golden_section_function = t1_function_scalar.get_new()

t1_golden_section_result = golden_section_search(
    function=t1_golden_section_function,
    start=t1_start_scalar,
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **postupak zlatnog reza** "
        f"je ${t1_golden_section_result:g}$ "
        f"(uz ${t1_golden_section_function.call_count}$ poziva funkcije)."
    )
)
```

Minimum koji je pronašao **postupak zlatnog reza** je $3$ (uz $43$ poziva funkcije).

#### Pretraživanje po koordinatnim osima

```python
t1_axis_search_function = t1_function.get_new()

t1_axis_search_result = axis_search(
    function=t1_axis_search_function,
    start=t1_start,
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašlo **pretraživanje po koordinatnim osima** "
        f"je ${t1_axis_search_result[0]:g}$ "
        f"(uz ${t1_axis_search_function.call_count}$ poziva funkcije)."
    )
)
```

Minimum koji je pronašlo **pretraživanje po koordinatnim osima** je $3$ (uz $128$ poziva funkcije).

#### Nelder Mead simpleks

```python
t1_nelder_mead_simplex_search_function = t1_function.get_new()

t1_nelder_mead_simplex_search_result = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_function,
    start=t1_start
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_result[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_function.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $2$ (uz $11$ poziva funkcije).

#### Nelder Mead simpleks (Jakobovićeva verzija)

```python
t1_nelder_mead_simplex_search_function_j = t1_function.get_new()

t1_nelder_mead_simplex_search_result_j = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_function_j,
    start=t1_start,
    use_jakobovic_expand=True
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_result_j[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_function_j.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $3$ (uz $53$ poziva funkcije).

#### Hooke-Jeeves postupak

```python
t1_hooke_jeeves_search_function = t1_function.get_new()

t1_hooke_jeeves_search_result = hooke_jeeves_search(
    function=t1_hooke_jeeves_search_function,
    start=t1_start
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Hooke-Jeeves postupak** "
        f"je ${t1_hooke_jeeves_search_result[0]:g}$ "
        f"(uz ${t1_hooke_jeeves_search_function.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Hooke-Jeeves postupak** je $3$ (uz $124$ poziva funkcije).

Probajte sve više udaljavati početnu točku od minimuma i probajte ponovo pokrenuti navedene postupke. Što možete zaključiti?

```python
t1_start_2 = np.array([15])
t1_start_scalar_2 = 15
```

#### Zlatni rez 2

```python
t1_golden_section_function_2 = t1_function_scalar.get_new()

t1_golden_section_result_2 = golden_section_search(
    function=t1_golden_section_function_2,
    start=t1_start_scalar_2,
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **postupak zlatnog reza** "
        f"je ${t1_golden_section_result_2:g}$ "
        f"(uz ${t1_golden_section_function_2.call_count}$ poziva funkcije)."
    )
)
```

Minimum koji je pronašao **postupak zlatnog reza** je $3$ (uz $43$ poziva funkcije).

#### Pretraživanje po koordinatnim osima 2

```python
t1_axis_search_function_2 = t1_function.get_new()

t1_axis_search_result_2 = axis_search(
    function=t1_axis_search_function_2,
    start=t1_start_2,
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašlo **pretraživanje po koordinatnim osima** "
        f"je ${t1_axis_search_result_2[0]:g}$ "
        f"(uz ${t1_axis_search_function_2.call_count}$ poziva funkcije)."
    )
)
```

Minimum koji je pronašlo **pretraživanje po koordinatnim osima** je $3$ (uz $213$ poziva funkcije).

#### Nelder Mead simpleks 2

```python
t1_nelder_mead_simplex_search_function_2 = t1_function.get_new()

t1_nelder_mead_simplex_search_result_2 = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_function_2,
    start=t1_start_2
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_result_2[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_function_2.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $1$ (uz $14$ poziva funkcije).

#### Nelder Mead simpleks 2 (Jakobovićeva verzija)

```python
t1_nelder_mead_simplex_search_function_j_2 = t1_function.get_new()

t1_nelder_mead_simplex_search_result_j_2 = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_function_j_2,
    start=t1_start_2,
    use_jakobovic_expand=True
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_result_j_2[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_function_j_2.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $3$ (uz $68$ poziva funkcije).

#### Hooke-Jeeves postupak 2

```python
t1_hooke_jeeves_search_function_2 = t1_function.get_new()

t1_hooke_jeeves_search_result_2 = hooke_jeeves_search(
    function=t1_hooke_jeeves_search_function_2,
    start=t1_start_2
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Hooke-Jeeves postupak** "
        f"je ${t1_hooke_jeeves_search_result_2[0]:g}$ "
        f"(uz ${t1_hooke_jeeves_search_function_2.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Hooke-Jeeves postupak** je $3$ (uz $146$ poziva funkcije).

**Komentar**: Vidimo da su sve metode osim **Nelder Mead simpleksa** pronašle minimum uz više poziva, izuzev **zlatnog reza** koji je pronašao minimum u istom broju koraka.

---

```python
t1_start_3 = np.array([100])
t1_start_scalar_3 = 100
```

#### Zlatni rez 3

```python
t1_golden_section_function_3 = t1_function_scalar.get_new()

t1_golden_section_result_3 = golden_section_search(
    function=t1_golden_section_function_3,
    start=t1_start_scalar_3,
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **postupak zlatnog reza** "
        f"je ${t1_golden_section_result_3:g}$ "
        f"(uz ${t1_golden_section_function_3.call_count}$ poziva funkcije)."
    )
)
```

Minimum koji je pronašao **postupak zlatnog reza** je $3$ (uz $53$ poziva funkcije).

#### Pretraživanje po koordinatnim osima 3

```python
t1_axis_search_function_3 = t1_function.get_new()

t1_axis_search_result_3 = axis_search(
    function=t1_axis_search_function_3,
    start=t1_start_3,
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašlo **pretraživanje po koordinatnim osima** "
        f"je ${t1_axis_search_result_3[0]:g}$ "
        f"(uz ${t1_axis_search_function_3.call_count}$ poziva funkcije)."
    )
)
```

Minimum koji je pronašlo **pretraživanje po koordinatnim osima** je $3$ (uz $136$ poziva funkcije).

#### Nelder Mead simpleks 3

```python
t1_nelder_mead_simplex_search_function_3 = t1_function.get_new()

t1_nelder_mead_simplex_search_result_3 = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_function_3,
    start=t1_start_3
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_result_3[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_function_3.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $4$ (uz $35$ poziva funkcije).

#### Nelder Mead simpleks 3 (Jakobovićeva verzija)

```python
t1_nelder_mead_simplex_search_function_j_3 = t1_function.get_new()

t1_nelder_mead_simplex_search_result_j_3 = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_function_j_3,
    start=t1_start_3,
    use_jakobovic_expand=True
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_result_j_3[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_function_j_3.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $3$ (uz $323$ poziva funkcije).

#### Hooke-Jeeves postupak 3

```python
t1_hooke_jeeves_search_function_3 = t1_function.get_new()

t1_hooke_jeeves_search_result_3 = hooke_jeeves_search(
    function=t1_hooke_jeeves_search_function_3,
    start=t1_start_3
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Hooke-Jeeves postupak** "
        f"je ${t1_hooke_jeeves_search_result_3[0]:g}$ "
        f"(uz ${t1_hooke_jeeves_search_function_3.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Hooke-Jeeves postupak** je $3$ (uz $212$ poziva funkcije).

**Komentar**: I ovog puta su svi osim **Nelder Mead simpleksa** uspješno našli rješenje, uz nešto više poziva funkcije. Za razliku od prošlog eksperimenta, ovog puta je i **zlatni rez** više puta pozivao funkciju.

E sad, zašto usprkos promjenama **Nelder Mead simpleks** ne uspijeva pronaći minimum? Jedan razlog za to je što taj postupak nije adekvatan za jednodimenzionalne funkcije. Razlog za to je što su sve točke kolinearne u jednoj dimenziji - razlika je jedino u normi vektora neke točke, ali sve imaju isti vektor smjera. Ovo će generirati nekvalitetne simplekse s kojima će biti preteško pronaći optimum čak i za trivijalne funkcije.

Isto tako, onog trena kada **Nelder Mead simpleks** naiđe na stanje gdje lošija točka ima istu funkciju gubitka kao i bolja, algoritam završava. Ovo je vrlo jednostavno kad pretražujemo jednu dimenziju, jer je potrebno samo jedno poklapanje. Kod više dimenzija ovakav događaj se otežava.

Ovakvo ponašanje može se kontrirati približavanjem početne točke optimumu ili povećavanjem koraka. Demonstrirajmo što će učiniti povećanje koraka.

```python
t1_nelder_mead_simplex_search_special_function = t1_function.get_new()

t1_nelder_mead_simplex_search_special_result = nelder_mead_simplex_search(
    function=t1_nelder_mead_simplex_search_special_function,
    start=t1_start_3,
    stride=5
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t1_nelder_mead_simplex_search_special_result[0]:g}$ "
        f"(uz ${t1_nelder_mead_simplex_search_special_function.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $3.00049$ (uz $59$ poziva funkcije).

Vidimo da smo sad ipak dobili nešto točniju brojku, uz malo više poziva funkcije.

### Zadatak 2

Primijenite simpleks po Nelderu i Meadu, Hooke-Jeeves postupak te pretraživanje po koordinatnim osima na funkcije 1 - 4 uz zadane parametre i početne točke (broj varijabli funkcije 3 najmanje $5$). Za svaki postupak i svaku funkciju odredite minimum koji su postupci pronašli i potrebni broj evaluacija funkcije cilja koji je potreban do konvergencije (prikažite tablično).

```python
t2_f1 = f1.get_new()
t2_f2 = f2.get_new()
t2_f3 = f3.get_new()
t2_f4 = f4.get_new()
t2_functions = (t2_f1, t2_f2, t2_f3, t2_f4)

t2_f1_start = copy.deepcopy(f1_start)
t2_f2_start = copy.deepcopy(f2_start)
t2_f3_start = np.zeros(5)
t2_f4_start = copy.deepcopy(f4_start)
t2_starts = (t2_f1_start, t2_f2_start, t2_f3_start, t2_f4_start)
```

#### Pretraživanje po koordinatnim osima

```python
t2_axis_search_functions = [x.get_new() for x in t2_functions]
t2_axis_search_results = [
    axis_search(function=f, start=s)
    for f, s in zip(t2_axis_search_functions, t2_starts)
]
```

```python
t2_axis_search_results_formatted = (
    "\n".join(
        [
            f"{i + 1}."
            " $x_{min}$ = "
            f"${array_to_latex(x)}$ "
            f"(uz ${f.call_count}$ poziva)"
            for i, (x, f) in enumerate(
                zip(
                    t2_axis_search_results,
                    t2_axis_search_functions
                )
            )
        ]
    )
)

display(
    Markdown(
        "**Pretraživanje po koordinatnim osima** daje "
        "sljedeće rezultate:\n"
        f"{t2_axis_search_results_formatted}"
    )
)
```

**Pretraživanje po koordinatnim osima** daje sljedeće rezultate:
1. $x_{min}$ = $\begin{bmatrix}
  0.9998 &  0.999601
\end{bmatrix}$ (uz $292904$ poziva)
2. $x_{min}$ = $\begin{bmatrix}
  4  &  2
\end{bmatrix}$ (uz $249$ poziva)
3. $x_{min}$ = $\begin{bmatrix}
  1  &  2  &  3  &  4  &  5
\end{bmatrix}$ (uz $619$ poziva)
4. $x_{min}$ = $\begin{bmatrix}
 -1.1 &  1.1
\end{bmatrix}$ (uz $198$ poziva)

#### Nelder Mead simpleks

```python
t2_nelder_mead_simplex_search_functions = [
    x.get_new() for x in t2_functions
]
t2_nelder_mead_simplex_search_results = [
    nelder_mead_simplex_search(function=f, start=s)
    for f, s in zip(t2_nelder_mead_simplex_search_functions, t2_starts)
]
```

```python
t2_nelder_mead_simplex_search_results_formatted = (
    "\n".join(
        [
            f"{i + 1}."
            " $x_{min}$ = "
            f"${array_to_latex(x)}$ "
            f"(uz ${f.call_count}$ poziva)"
            for i, (x, f) in enumerate(
                zip(
                    t2_nelder_mead_simplex_search_results,
                    t2_nelder_mead_simplex_search_functions
                )
            )
        ]
    )
)

display(
    Markdown(
        "**Nelder Mead simpleks** daje "
        "sljedeće rezultate:\n"
        f"{t2_nelder_mead_simplex_search_results_formatted}"
    )
)
```

**Nelder Mead simpleks** daje sljedeće rezultate:
1. $x_{min}$ = $\begin{bmatrix}
  1.00015 &  1.00028
\end{bmatrix}$ (uz $220$ poziva)
2. $x_{min}$ = $\begin{bmatrix}
  4.00001 &  2.00008
\end{bmatrix}$ (uz $94$ poziva)
3. $x_{min}$ = $\begin{bmatrix}
  1.00054 &  2.00055 &  3.00073 &  4.00113 &  4.99916
\end{bmatrix}$ (uz $277$ poziva)
4. $x_{min}$ = $\begin{bmatrix}
  3.19099\times 10^{-07} &  1.06134\times 10^{-07}
\end{bmatrix}$ (uz $190$ poziva)

#### Hooke-Jeeves postupak

```python
t2_hooke_jeeves_search_functions = [
        x.get_new() for x in t2_functions
    ]

t2_hooke_jeeves_search_results = [
    hooke_jeeves_search(function=f, start=s)
    for f, s in zip(t2_hooke_jeeves_search_functions, t2_starts)
]
```

```python
t2_hooke_jeeves_search_results_formatted = (
    "\n".join(
        [
            f"{i + 1}."
            " $x_{min}$ = "
            f"${array_to_latex(x)}$ "
            f"(uz ${f.call_count}$ poziva)"
            for i, (x, f) in enumerate(
                zip(
                    t2_hooke_jeeves_search_results,
                    t2_hooke_jeeves_search_functions
                )
            )
        ]
    )
)

display(
    Markdown(
        "**Hooke-Jeeves postupak** daje "
        "sljedeće rezultate:\n"
        f"{t2_hooke_jeeves_search_results_formatted}"
    )
)
```

**Hooke-Jeeves postupak** daje sljedeće rezultate:
1. $x_{min}$ = $\begin{bmatrix}
  1  &  0.999999
\end{bmatrix}$ (uz $726$ poziva)
2. $x_{min}$ = $\begin{bmatrix}
  4  &  2
\end{bmatrix}$ (uz $320$ poziva)
3. $x_{min}$ = $\begin{bmatrix}
  1  &  2  &  3  &  4  &  5
\end{bmatrix}$ (uz $444$ poziva)
4. $x_{min}$ = $\begin{bmatrix}
  3.1 &  3.1
\end{bmatrix}$ (uz $180$ poziva)

Što možete zaključiti iz rezultata?

**Odgovor**: Krenimo funkciju po funkciju.

**Funkcija 1**

Sve funkcije uspjele su pronaći minimum (koliko-toliko). Stoga ćemo analizirati njihove devijacije od rješenja:

- **Pretraživanje po koordinatnim osima**: $\begin{bmatrix}0.0002 & 0.000399\end{bmatrix}$
- **Nelder Mead simplex**: $\begin{bmatrix}0.00015 & 0.00028\end{bmatrix}$
- **Hooke-Jeeves postupak**: $\begin{bmatrix}0 & 0.0000001 \end{bmatrix}$

Vidimo da najmanju razliku ima **Hooke-Jeeves postupak**, pa **Nelder Mead simpleks**, pa **pretraživanje po koordinatnim osima**. **Hooke-Jeeves postupak**, međutim, više puta poziva funkciju od **Nelder Mead simpleksa**, tako da nije striktno bolji. **Pretraživanje po koordinatnim osima** uz najgore rezultate ima i najgore performanse - zahtjevajući 3 reda veličina više poziva od **Nelder Mead simpleksa**.

**Funkcija 2**

I ovdje su sve funkcije uspjele pronaći minimum, samo je **Nelder Mead simpleks** bio manje precizan od ostalih. Međutim, bitno je za napomenuti da je on našao rješenje u najmanje koraka (opet), te da bi nam ono možda bilo prihvatljivi trade-off za performanse.

**Funkcija 3**

Ovdje su stvari isto prilično jasne. Naime, **pretraživanje po koordinatnim osima** pronalazi rješenje bez problema, uz razuman broj poziva. **Nelder Mead simpleks** pronalazi rješenje uz $10^{-3}$ ili manju apsolutnu pogrešku, opet uz najmanji broj poziva. **Hooke-Jeeves postupak** pronalazi točno rješenje uz nešto više poziva od **Nelder Mead sipleksa**.

**Funkcija 4**

Globalni minimum ove funkcije pronalazi samo **Nelder Mead sipleks**, uz apsolutnu grešku reda $10^{-7}$. Razlog za ovo je što funkcija 4 izravno kontrira ostale postupke. **Pretraživanje po koordinatnim osima** pokušava optimirati točku pretraživanja koordinatu po koordinatu. Međutim, ovdje je funkcija 4 problematična zbog oblika - potrebno je pomicati se u svim smjerovima hiperprostora odjednom jer se oštre brazde nalaze točno na dijagonalama hiperprostora (ovo uzrokuje operator apsolutno u formuli). Radi sličnog problema globalni minimum ne nalazi ni **Hooke-Jeeves postupak**; on je čak manje efikasan. Razlog za to je relativno jednostavan: **Hooke-Jeeves postupak** ima manji pomak, što ga hendikepira da nađe bolji minimum pomacima po koordinatnim osima. Oba postupka zapnu u dijagonalama hiperprostora.

Ovaj problem je relativno lagano rješiv povećanjem pomaka algoritma. Zbog strukture **pretraživanja po koordinatnim osima** ne možemo popraviti algoritam (jer je pomak ovisan o $\epsilon$), ali to možemo za **Hooke-Jeeves postupak**. Demonstrirajmo tvrdnju.

```python
t2_hooke_jeeves_search_special_function = t2_f4.get_new()

t2_hooke_jeeves_search_special_result = hooke_jeeves_search(
    t2_hooke_jeeves_search_special_function,
    t2_f4_start,
    stride=np.array([5.] * len(t2_f4_start))
)
```

```python
display(
    Markdown(
        "Pretraživanje po koordinatnim osima našlo je minimum "
        f"${array_to_latex(t2_hooke_jeeves_search_special_result)}$ uz "
        f"${t2_hooke_jeeves_search_special_function.call_count}$ poziva "
        "funkcije."

    )
)
```

Pretraživanje po koordinatnim osima našlo je minimum $\begin{bmatrix}
  9.53674\times 10^{-08} & -1.43051\times 10^{-07}
\end{bmatrix}$ uz $335$ poziva funkcije.

Vidimo da je sad **Hooke-Jeeves postupak** uspio pronaći minimum, i to nešto kvalitetniji od **Nelder Mead simpleksa**, ali uz nešto više poziva - gotovo duplo više od inicijalne pretrage. Konkretno zašto je veći pomak pomogao razmotrit ćemo u sljedećem zadatku.

Način na koji bismo popravili **pretraživanje po koordinatnim osima** je da uvedemo dodatan hiperparametar (npr. $\alpha$) koji bi skalirao korak, pa bi surogatna funkcija glasila:

$$
\begin{equation}
    \begin{array}
        & i, j, k \in \mathbb{N} & i, k \in \left[ 1, j \right] &
    \end{array} \\
%
\vec{\epsilon}_i : {\mathbb{R}}^j =
\begin{cases}
    0 & k \neq i \\
    \epsilon_k & k = i
\end{cases} \\
%
f_i \left( \lambda, \alpha \right) = f \left( \vec{x} + \alpha \lambda \cdot \vec{\epsilon}_{i} \right) \\
\end{equation}
$$

### Zadatak 3

Primijenite postupak Hooke-Jeeves i simpleks po Nelderu i Meadu na funkciju 4 uz početnu točku $\begin{bmatrix}5 & 5\end{bmatrix}$.

```python
t3_function = f4.get_new()
t3_start = np.array([5, 5])
```

#### Nelder Mead simpleks

```python
t3_nelder_mead_simplex_search_function = t3_function.get_new()

t3_nelder_mead_simplex_search_result = nelder_mead_simplex_search(
    function=t3_nelder_mead_simplex_search_function,
    start=t3_start
)
t3_nelder_mead_simplex_search_result_pretty = (
    array_to_latex(t3_nelder_mead_simplex_search_result)
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Nelder Mead simpleks** "
        f"je ${t3_nelder_mead_simplex_search_result_pretty}$ "
        f"(uz ${t3_nelder_mead_simplex_search_function.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Nelder Mead simpleks** je $\begin{bmatrix}
 -7.67649\times 10^{-07} & -4.89511\times 10^{-07}
\end{bmatrix}$ (uz $194$ poziva funkcije).

#### Hooke-Jeeves postupak

```python
t3_hooke_jeeves_search_function = t3_function.get_new()

t3_hooke_jeeves_search_result = hooke_jeeves_search(
    function=t3_hooke_jeeves_search_function,
    start=t3_start,
)
t3_hooke_jeeves_search_result_pretty = (
    array_to_latex(t3_hooke_jeeves_search_result)
)
```

```python
display(
    Markdown(
        "Minimum koji je pronašao **Hooke-Jeeves postupak** "
        f"je ${t3_hooke_jeeves_search_result_pretty}$ "
        f"(uz ${t3_hooke_jeeves_search_function.call_count}$ "
        "poziva funkcije)."
    )
)
```

Minimum koji je pronašao **Hooke-Jeeves postupak** je $\begin{bmatrix}
  5  &  5
\end{bmatrix}$ (uz $160$ poziva funkcije).

Objasnite rezultate!

**Odgovor**: U pitanju je bila funkcija 4, tj. Jakobovićeva funkcija. **Nelder Mead simpleks** je dobro aproksimirao minimum, skoro ga doveo do nul-vektora. Međutim, izgleda da je **Hooke-Jeeves postupak** bio neefektivan. Razlog za to je jednostavan, no olakšajmo si prvo razmatranje tako da nacrtamo kako funkcija 4 izgleda.

```python
t3_plot_x = np.linspace(-7, 7, 201)
t3_plot_y = np.linspace(-7, 7, 201)
t3_plot_z = np.array(
    [
        t3_hooke_jeeves_search_function(np.array([x, y]), dont_count=True)
        for x in t3_plot_x
        for y in t3_plot_y
    ]
)

t3_plot_x, t3_plot_y = np.meshgrid(t3_plot_x, t3_plot_y)
t3_plot_z = t3_plot_z.reshape(201, 201)

t3_fig, t3_ax = plt.subplots(1, 1, figsize=(16,16))
t3_ax.pcolor(
    t3_plot_x,
    t3_plot_y,
    t3_plot_z,
    shading="nearest",
    cmap="inferno"
);
```

Sada prikažimo gdje se nalazi naša početna točka, i sve točke s kojima će **Hooke-Jeeves postupak** inicijalno pokušati pronaći minimum.

```python
t3_survey_points = get_survey_points(
    t3_hooke_jeeves_search_function,
    t3_start,
    np.array([0.5] * len(t3_start))
)
```

```python
t3_special_fig, t3_special_ax = plt.subplots(1, 1, figsize=(16,16))

t3_special_ax.pcolor(
    t3_plot_x,
    t3_plot_y,
    t3_plot_z,
    shading="nearest",
    cmap="inferno"
)

t3_special_ax.scatter(
    [float(t3_start[0])],
    [float(t3_start[1])],
    c="g",
    s=150,
    marker="o"
)

t3_special_ax.scatter(
    [float(x[0]) for x in t3_survey_points],
    [float(x[1]) for x in t3_survey_points],
    c="w",
    s=40,
    marker="x"
);
```

Vidimo da prva pretraga ne može pronaći bolju točku. Pretrage gdje je smanjen pomak služe samo kako bi se našlo "finije" rješenje, tj. kako bi trenutna točka potonula u rupicu koja bi trebala biti minimum. Sljedećim pretragama ovi će se bijeli markeri samo približavati zelenom markeru, tj. početnoj točki. To neće pomoći pronalaženju optimuma jer dana funkcija nema oblik bazenčića. Način kako bi se mogli izbaviti iz minimuma je smanjenjem epsilona - na taj način ćemo dopustiti ovim "sonda" točkama da dođu dovoljno blizu da nađu minimum. Međutim, ovakva konvergencija će biti iznimno spora jer ne postoji mehanizam vraćanja pomaka na inicijalnu vrijednost, čime bi se ubrzala pretraga jednom kad natjeramo točku u bazen blizu ishodišta.

Bolja metoda bi bila sljedeća

- smanjiti koeficijent s kojim se dijeli epsilon
- uvesti rotaciju u sljedećem pozivu `pretraži` ako nije pronađena bolja točka

Ovim bi postigli to da imamo manje problema s topologijom hiperplohe funkcije, a smanjili bismo koeficijent tako da damo vremena toj rotaciji da provjeri dovoljno guste točke. Specifično kod ovog primjera, jednom kad bi rotacija dosegla $\frac{\pi}{4}$, algoritam bi vrlo brzo iskonvergirao u $\begin{bmatrix}0 & 0 \end{bmatrix}$.

Također, u prošlom zadatku smo pokazali da pomoći može i povećanje pomaka - time će algoritam napraviti nešto više poziva funkcije, ali barem može pronaći minimum.

### Zadatak 4

Primijenite simpleks po Nelderu i Meadu na funkciju 1.

```python
t4_function = f1.get_new()
```

Kao početnu točku postavite točku $\begin{bmatrix}0.5 & 0.5\end{bmatrix}$.

```python
t4_start = np.array([0.5, 0.5])
```

Provedite postupak s nekoliko različitih koraka za generiranje početnog simpleksa (primjerice iz intervala od $1$ do $20$) i zabilježite potreban broj evaluacija funkcije cilja i pronađene točke minimuma.

```python
t4_result_dict = dict()
t4_range = list(range(1, 20 + 1))
t4_iterator = tqdm(t4_range, file=sys.stdout)

for i in t4_iterator:
    t4_iterator.set_description(f"Računam Nelder Mead Simpleks uz pomak {i}")

    _t4_function = t4_function.get_new()

    result = nelder_mead_simplex_search(
        function=_t4_function,
        start=t4_start,
        stride=i,
    )

    t4_result_dict[i] = {
        "minimum": result,
        "call_count": _t4_function.call_count
    }

t4_iterator.close()
```

    Računam Nelder Mead Simpleks uz pomak 20: 100%|██████████| 20/20 [00:00<00:00, 105.55it/s]

Možemo prikazati ovisnost broja poziva funkcije o koracima, kao i ovisnost norme razlike dobivenog i točnog rješenja o koracima

```python
t4_calls = [t4_result_dict[x]["call_count"] for x in t4_range]
t4_result_norm = [
    np.linalg.norm(t4_result_dict[x]["minimum"] - np.array([1, 1]))
    for x in t4_range
]

t4_calls_a, t4_calls_b = (
    np.polyfit(t4_range, t4_calls, 1)
)
t4_result_norm_a, t4_result_norm_b = (
    np.polyfit(t4_range, t4_result_norm, 1)
)
```

```python
t4_fig, t4_ax = plt.subplots(2, 1, figsize=(16, 18))

t4_ax[0].plot(
    t4_range, t4_calls, "-o", label="Pozivi funkcije",
)
t4_ax[0].plot(
    t4_range,
    t4_calls_a * np.array(t4_range) + t4_calls_b,
    linestyle="dotted",
    label="Regresija poziva funkcije"
)
t4_ax[0].set_xlabel("Korak")
t4_ax[0].set_ylabel("Broj poziva funkcije")
t4_ax[0].xaxis.set_ticks(t4_range)
t4_ax[0].legend()

t4_ax[1].plot(
    t4_range,
    t4_result_norm,
    "-o",
    label="Norma razlike rješenja"
)
t4_ax[1].plot(
    t4_range,
    t4_result_norm_a * np.array(t4_range) + t4_result_norm_b,
    linestyle="dotted",
    label="Regresija norme razlike rješenja"
)
t4_ax[1].set_xlabel("Korak")
t4_ax[1].set_ylabel("Norma razlike rješenja")
t4_ax[1].xaxis.set_ticks(t4_range)
t4_ax[1].legend();
```

**Komentar**: Regresijske krivulje oba grafa nam govore da korak ne čini znatnu razliku. Kod norme je varijanca pogotovo velika (npr. prosječna vrijednost je oko $0.0006$, a raspon od $0.0002$ do $0.0012$). Kod broja poziva funkcije imamo mirniju funkciju koja doživljava šiljke tek kod par koraka, ali za većinu ostalih koraka imamo oko $150$ poziva funkcije.

Potom probajte kao početnu točku postaviti točku $\begin{bmatrix}20 & 20\end{bmatrix}$ i ponovo provesti eksperiment.

```python
t4_special_start = np.array([20, 20])
```

```python
t4_special_result_dict = dict()
t4_special_range = list(range(1, 20 + 1))
t4_special_iterator = tqdm(t4_special_range, file=sys.stdout)

for i in t4_special_iterator:
    t4_special_iterator.set_description(f"Računam Nelder Mead Simpleks uz pomak {i}")

    _t4_function = t4_function.get_new()

    result = nelder_mead_simplex_search(
        function=_t4_function,
        start=t4_special_start,
        stride=i,
    )

    t4_special_result_dict[i] = {
        "minimum": result,
        "call_count": _t4_function.call_count
    }

t4_special_iterator.close()
```

    Računam Nelder Mead Simpleks uz pomak 20: 100%|██████████| 20/20 [00:00<00:00, 45.80it/s]

Isto kao i prije ćemo nacrtati dva grafa ovisnosti

```python
t4_special_calls = [t4_special_result_dict[x]["call_count"] for x in t4_special_range]
t4_special_result_norm = [
    np.linalg.norm(t4_special_result_dict[x]["minimum"] - np.array([1, 1]))
    for x in t4_special_range
]

t4_special_calls_a, t4_special_calls_b = (
    np.polyfit(t4_special_range, t4_special_calls, 1)
)
t4_special_result_norm_a, t4_special_result_norm_b = (
    np.polyfit(t4_special_range, t4_special_result_norm, 1)
)
```

```python
t4_special_fig, t4_special_ax = plt.subplots(2, 1, figsize=(16, 18))

t4_special_ax[0].plot(
    t4_special_range,
    t4_special_calls,
    "-o",
    label="Pozivi funkcije"
)
t4_special_ax[0].plot(
    t4_special_range,
    t4_special_calls_a * np.array(t4_special_range) + t4_special_calls_b,
    linestyle="dotted",
    label="Regresija poziva funkcije"
)
t4_special_ax[0].set_xlabel("Korak")
t4_special_ax[0].set_ylabel("Broj poziva funkcije")
t4_special_ax[0].xaxis.set_ticks(t4_special_range)
t4_special_ax[0].legend()

t4_special_ax[1].plot(
    t4_special_range,
    t4_special_result_norm,
    "-o",
    label="Norma razlike rješenja"
)
t4_special_ax[1].plot(
    t4_special_range,
    t4_special_result_norm_a * np.array(t4_special_range) + t4_special_result_norm_b,
    linestyle="dotted",
    label="Regresija norme razlike rješenja"
)
t4_special_ax[1].set_xlabel("Korak")
t4_special_ax[1].set_ylabel("Norma razlike rješenja")
t4_special_ax[1].xaxis.set_ticks(t4_special_range)
t4_special_ax[1].legend();
```

Što možete zaključiti?

**Odgovor**: Opet imamo slično ponašanje, ali su ovog puta pozivi funkcija učestaliji. Vidimo da sada jasnije opadaju povećanjem koraka. Racionalizacija za ovo je relativno jednostavna - veći pomak stvara veći simpleks. Veći simpleks može brže putovati prema optimumu. Za razliku od prije, ovdje postoji mali skup vrijednosti kod kojih broj poziva funkcije znatno opada. Ako pogledamo normu razlike rješenja, vidimo da je za pomak od $11$ iznadprosječna greška, dok je za pomak $16$ ispodprosječna.

### Zadatak 5

Primijenite jedan postupak optimizacije na funkciju 6 u dvije dimenzije, tako da postupak pokrećete više puta iz slučajno odabrane početne točke u intervalu $\left[ -50, 50 \right]$.

```python
t5_n_repeats = 10000
t5_function = f6.get_new()
t5_solution = np.zeros(shape=(2,))
```

```python
t5_results = list()
t5_range = list(range(t5_n_repeats))
t5_iterator = tqdm(t5_range, file=sys.stdout)
t5_iterator.set_description("Računam Nelder Mead simplekse")

for _ in t5_iterator:
    t5_start = np.random.uniform(-50, 50, size=2)

    t5_result = nelder_mead_simplex_search(
        function=t5_function,
        start=t5_start
    )
    t5_results.append(t5_result)

t5_iterator.close()
```

    Računam Nelder Mead simplekse: 100%|██████████| 10000/10000 [00:30<00:00, 323.82it/s]

Možete li odrediti vjerojatnost pronalaženja *globalnog* optimuma na ovaj način? (smatramo da je algoritam locirao globalni minimum ako je nađena vrijednost funkcije cilja manja od $10^{-4}$).

**Odgovor**: Možemo aproksimirati tu vrijednost

```python
t5_epsilon = 1e-4
t5_values = np.array([t5_function(x) for x in t5_results])

t5_optimum_found = abs(t5_values) < t5_epsilon
t5_n_optima_found = sum(t5_optimum_found)
t5_global_optimum_probability = float(t5_n_optima_found / t5_n_repeats)
```

```python
display(
    Markdown(
        f"U ${t5_n_repeats}$ ponavljanja globalni optimum je "
        f"pronađen ${t5_n_optima_found}$ puta. Ovo bi sugeriralo "
        "da je vjerojatnost pronalaženja globalnog optimuma za "
        "početnu točku iz intervala $ [ -50, 50 ] $ "
        f"približno ${t5_global_optimum_probability * 100:.02f}\%$."
    )
)
```

U $10000$ ponavljanja globalni optimum je pronađen $9$ puta. Ovo bi sugeriralo da je vjerojatnost pronalaženja globalnog optimuma za početnu točku iz intervala $ [ -50, 50 ] $ približno $0.09\%$.

---

#### Bonus: aproksimacija pretraživanjem po koordinatnim osima

```python
t5_function_bonus = t5_function.get_new()
```

```python
t5_results_bonus = list()
t5_range_bonus = list(range(t5_n_repeats))
t5_iterator_bonus = tqdm(t5_range_bonus, file=sys.stdout)
t5_iterator_bonus.set_description("Pretražujem po koordinatnim osima")

for _ in t5_iterator_bonus:
    t5_start = np.random.uniform(-50, 50, size=2)

    t5_result_bonus = axis_search(
        function=t5_function_bonus,
        start=t5_start
    )
    t5_results_bonus.append(t5_result_bonus)

t5_iterator_bonus.close()
```

    Pretražujem po koordinatnim osima: 100%|██████████| 10000/10000 [01:02<00:00, 159.05it/s]

```python
t5_values_bonus = np.array(
    [t5_function_bonus(x) for x in t5_results_bonus]
)

t5_optimum_found_bonus = abs(t5_values_bonus) < t5_epsilon
t5_n_optima_found_bonus = sum(t5_optimum_found_bonus)
t5_global_optimum_probability_bonus = float(t5_n_optima_found_bonus / t5_n_repeats)
```

```python
display(
    Markdown(
        f"U ${t5_n_repeats}$ ponavljanja globalni optimum je "
        f"pronađen ${t5_n_optima_found_bonus}$ puta. Ovo bi sugeriralo "
        "da je vjerojatnost pronalaženja globalnog optimuma za "
        "početnu točku iz intervala $ [ -50, 50 ] $ "
        f"približno ${t5_global_optimum_probability_bonus * 100:.02f}\%$."
    )
)
```

U $10000$ ponavljanja globalni optimum je pronađen $7$ puta. Ovo bi sugeriralo da je vjerojatnost pronalaženja globalnog optimuma za početnu točku iz intervala $ [ -50, 50 ] $ približno $0.07\%$.

---

#### Bonus: aproksimacija Hooke-Jeeves postupkom

```python
t5_function_bonus_2 = t5_function.get_new()
```

```python
t5_results_bonus_2 = list()
t5_range_bonus_2 = list(range(t5_n_repeats))
t5_iterator_bonus_2 = tqdm(t5_range_bonus_2, file=sys.stdout)
t5_iterator_bonus_2.set_description("Provodim Hooke-Jeeves postupke")

for _ in t5_iterator_bonus_2:
    t5_start = np.random.uniform(-50, 50, size=2)

    t5_result_bonus_2 = axis_search(
        function=t5_function_bonus_2,
        start=t5_start
    )
    t5_results_bonus_2.append(t5_result_bonus_2)

t5_iterator_bonus_2.close()
```

    Provodim Hooke-Jeeves postupke: 100%|██████████| 10000/10000 [01:01<00:00, 162.85it/s]

```python
t5_values_bonus_2 = np.array(
    [t5_function_bonus_2(x) for x in t5_results_bonus_2]
)

t5_optimum_found_bonus_2 = abs(t5_values_bonus_2) < t5_epsilon
t5_n_optima_found_bonus_2 = sum(t5_optimum_found_bonus_2)
t5_global_optimum_probability_bonus_2 = float(t5_n_optima_found_bonus_2 / t5_n_repeats)
```

```python
display(
    Markdown(
        f"U ${t5_n_repeats}$ ponavljanja globalni optimum je "
        f"pronađen ${t5_n_optima_found_bonus_2}$ puta. Ovo bi sugeriralo "
        "da je vjerojatnost pronalaženja globalnog optimuma za "
        "početnu točku iz intervala $ [ -50, 50 ] $ "
        f"približno ${t5_global_optimum_probability_bonus_2 * 100:.02f}\%$."
    )
)
```

U $10000$ ponavljanja globalni optimum je pronađen $11$ puta. Ovo bi sugeriralo da je vjerojatnost pronalaženja globalnog optimuma za početnu točku iz intervala $ [ -50, 50 ] $ približno $0.11\%$.

---

**Zaključak**: Vidimo da očekivana učestalost pronalaska minimuma za ove parametre ne ovisi previše o algoritmu, te je oko $0.1 \%$.


Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your name and collaborators below:

```python
NAME = "Prabal Chowdhury"
COLLABORATORS = ""
```

---

## CSE330 Lab: Richardson Extrapolation
---

## Instructions

Today's assignment is to:
1. Implement Richardson Extrapolation method using Python

## Richardson Extrapolation:
We used central difference method to calculate derivatives of functions last task. In this task we will use Richardson extrapolation to get a more accurate result.
Let,
$$ D_h = \frac{f(x_1+h) -f(x_1-h)}{2h}\tag{5.1}$$

General Taylor Series formula:
$$ f(x) = f(x_1) + f'(x_1)(x - x_1) + \frac{f''(x_1)}{2}(x - x_1)^2+... $$
Using Taylor's theorem to expand we get,

\begin{align}
f(x_1+h) &= f(x_1) + f^{\prime}(x_1)h + \frac{f^{\prime \prime}(x_1)}{2}h^2 + \frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + \frac{f^{(4)}(x_1)}{4!}h^4 + \frac{f^{(5)}(x_1)}{5!}h^5 + O(h^6)\tag{5.2} \\
f(x_1-h) &= f(x_1) - f^{\prime}(x_1)h + \frac{f^{\prime \prime}(x_1)}{2}h^2 - \frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + \frac{f^{(4)}(x_1)}{4!}h^4 - \frac{f^{(5)}(x_1)}{5!}h^5 + O(h^6)\tag{5.3}
\end{align}
Subtracting $5.3$ from $5.2$ we get,
$$ f(x_1+h) - f(x_1-h) = 2f^{\prime}(x_1)h + 2\frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + 2\frac{f^{(5)}(x_1)}{5!}h^5 + O(h^7)\tag{5.4}$$
So,
\begin{align}
D_h &= \frac{f(x_1+h) - f(x_1-h)}{2h} \\
&= \frac{1}{2h} \left( 2f^{\prime}(x_1)h + 2\frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + 2\frac{f^{(5)}(x_1)}{5!}h^5 + O(h^7) \right) \\
&= f^{\prime}(x_1) + \frac{f^{\prime \prime \prime}(x_1)}{6}h^2 + \frac{f^{(5)}(x_1)}{120}h^4 + O(h^6) \tag{5.5}
\end{align}
We get our derivative $f'(x)$ plus some error terms of order $>= 2$ Now, we want to bring our error order down to 4.

If we use $h, \text{and} \frac{h}{2}$ as step size in $5.5$, we get,
\begin{align}
D_h &= f^{\prime}(x_1) + f^{\prime \prime \prime}(x_1)\frac{h^2}{6} + f^{(5)}(x_1) \frac{h^4}{120} + O(h^6) \tag{5.6} \\
D_{h/2} &= f^{\prime}(x_1) + f^{\prime \prime \prime}(x_1)\frac{h^2}{2^2 . 6} + f^{(5)}(x_1) \frac{h^4}{2^4 . 120} + O(h^6) \tag{5.7}
\end{align}
Multiplying $5.7$ by $4$ and subtracting from $5.6$ we get,
\begin{align}
D_h - 4D_{h/2} &= -3f^{\prime}(x) + f^{(5)}(x_1) \frac{h^4}{160} + O(h^6)\\
\Longrightarrow D^{(1)}_h = \frac{4D_{h/2} - D_h}{3} &= f^{\prime}(x) - f^{(5)}(x_1) \frac{h^4}{480} + O(h^6) \tag{5.8}
\end{align}
Let's calculate the derivative using $5.8$

### 1. Let's import the necessary headers

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from numpy.polynomial import Polynomial
```

### 2. Let's create a function named `dh(f, h, x)`
function `dh(f, h, x)` takes three parameters as input: a function `f`, a value `h`, and a set of values `x`. It returns the derivatives of the function at each elements of array `x` using the Central Difference method. This calculates equation $(5.1)$.

```python
def dh(f, h, x):
    '''
    Input:
        f: np.polynomial.Polynonimial type data.
        h: floating point data.
        x: np.array type data.
    Output:
        return np.array type data of slope at each point x.
    '''
    # --------------------------------------------
    return (f(x+h) - f(x-h)) / (2*h)
    # --------------------------------------------
```

### 3. Let's create another funtion `dh1(f, h, x)`.
`dh1(f, h, x)` takes the same type of values as `dh(f, h, x)` as input. It calculates the derivative using previously defined `dh(f, h, x)` function and using equation $5.8$ and returns the values.

```python
def dh1(f, h, x):
    '''
    Input:
        f: np.polynomial.Polynonimial type data.
        h: floating point data.
        x: np.array type data.
    Output:
        return np.array type data of slope at each point x.
    '''
    # --------------------------------------------
    # YOUR CODE HERE
    return (4 * dh(f, h/2, x) - dh(f, h, x)) / 3
    # --------------------------------------------
```

### 4. Now let's create the `error(f, hs, x_i)` function
The `error(f, hs, x_i)` function takes a function `f` as input. It also takes a list of different values of h as `hs` and a specific value as `x_i` as input. It calculates the derivatives as point `x_i` using both functions described in **B** and **C**, i.e. `dh` and `dh1`

```python
def error(f, hs, x_i):  #Using the functions we wrote dh() my c_diff  and dh1() which is my first order c diff, we find the error through appending their diffrences with Y_actual ny f(x)
    '''
    Input:
        f  : np.polynomial.Polynonimial type data.
        hs : np.array type data. list of h.
        x_i: floating point data. single value of x.
    Output:
        return two np.array type data of errors by two methods..
    '''
    f_prime = f.deriv(1)   #first order derivitive f^1(x)
    Y_actual = f_prime(x_i)

    diff_error = []
    diff2_error = []

    for h in hs: #where h is my loop counter iterating through hs
        # for each values of hs calculate the error using both methods
        # and append those values into diff_error and diff2_error list.

        # --------------------------------------------
        # YOUR CODE HERE
        e1 = Y_actual - dh(f, hs, x_i)
        diff_error.append(e1)

        e2 = Y_actual - dh1(f, hs, x_i)
        diff2_error.append(e2)
        # --------------------------------------------

    print(pd.DataFrame({"h": hs, "Diff": diff_error, "Diff2": diff2_error}))

    return diff_error, diff2_error
```

### 5. Finally let's run some tests
function to draw the actual function

```python
def draw_graph(f, ax, domain=[-10, 10], label=None):
    data = f.linspace(domain=domain)
    ax.plot(data[0], data[1], label='Function')
```

### Draw the polynomial and it's actual derivative function

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')

p = Polynomial([2.0, 1.0, -6.0, -2.0, 2.5, 1.0])
p_prime = p.deriv(1)
draw_graph(p, ax, [-2.4, 1.5], 'Function')
draw_graph(p_prime, ax, [-2.4, 1.5], 'Derivative')

ax.legend()
```

### Draw the actual derivative and richardson derivative using `h=1` and `h=0.1` as step size.

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')

draw_graph(p_prime, ax, [-2.4, 1.5], 'actual')

h = 1
x = np.linspace(-2.4, 1.5, 50, endpoint=True)
y = dh1(p, h, x)
ax.plot(x, y, label='Richardson; h=1')

h = 0.1
x = np.linspace(-2.4, 1.5, 50, endpoint=True)
y = dh1(p, h, x)
ax.plot(x, y, label='Richardson; h=0.1')

ax.legend()

```

### Draw error-vs-h cuve

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')
hs = np.array([1., 0.55, 0.3, .17, 0.1, 0.055, 0.03, 0.017, 0.01])
e1, e2 = error(p, hs, 2.0)
ax.plot(hs, e1, label='e1')
ax.plot(hs, e2, label='e2')

ax.legend()

```

```python

```


Ejericicio 1.

Creen un vector a entre 5 y 6 con intervalo de 0.2 y guardenlo en formato datos.npy y luego carguenlo con nombre b y muestrelo

```python
from google.colab import drive
import os
drive.mount('/content/gdrive')
```

    Mounted at /content/gdrive

```python
%cd '/content/gdrive/MyDrive/bootcamp/Semana 1-20211126T000634Z-001'
```

    /content/gdrive/MyDrive/bootcamp/Semana 1-20211126T000634Z-001

```python
import numpy as np

a=np.arange(5,6,0.2)
print(a.tolist())
a.tofile('datos.npy')
b=np.fromfile('datos.npy')
print(b)

```

    [5.0, 5.2, 5.4, 5.6000000000000005, 5.800000000000001]
    [5.  5.2 5.4 5.6 5.8]

Ejercicio 2
Cree un una matriz A con np.array

\begin{equation}
\begin{bmatrix}
1 & 3 & 5\\
11 & 9 & 7\\
13 & 15 & 17
\end{bmatrix}
\end{equation}

Luego de eso utilize el metodo np.sort con el argumento axis =1 or axis=0

A que conclusiones llegan?

```python
A=np.array([[1,3,5],[11,9,7],[13,15,17]])
print(A)
B=np.sort(A,axis=0)
print(B)
C=np.sort(A,axis=1)
print(C)

```

    [[ 1  3  5]
     [11  9  7]
     [13 15 17]]
    [[ 1  3  5]
     [11  9  7]
     [13 15 17]]
    [[ 1  3  5]
     [ 7  9 11]
     [13 15 17]]

Ejercicio 3
Cree una matriz de 3x3 B, como la siguiente:
\begin{equation}
\begin{bmatrix}
9 & 1 & 2\\
1 & 9 & 5\\
2 & 5 & 9
\end{bmatrix}
\end{equation}

1. Convierta la matriz a un vector de 1x9
2. Ordene de de menor a mayor el vector
3. Aplique un filtro para extraer solo los numeros menores que 9

```python
A=np.array([[9,1,2],[1,9,5],[2,5,9]])
print(A)
print('parte 1')
print(A.flatten())
print('parte 2')
b=np.sort(A.flatten())
print(b)
print('parte 3')
c=numpy.where(b<9)
print(c)

```

    [[9 1 2]
     [1 9 5]
     [2 5 9]]
    parte 1
    [9 1 2 1 9 5 2 5 9]
    parte 2
    [1 1 2 2 5 5 9 9 9]
    parte 3
    (array([0, 1, 2, 3, 4, 5]),)

Ejercicio 4
Cree una matriz de 3x3 C, como la siguiente:
\begin{equation}
\begin{bmatrix}
1 & 4 & 7\\
2 & 5 & 8\\
3 & 6 & 9
\end{bmatrix}
\end{equation}

1. Calcule la media por cada columna
2. Calcule la media por cada fila
3. Obtenga la media por columnas dandole el doble de peso a la columna con mayores valores de la variable
5. Calcule el percentil 25 por cada columna

```python
C=np.array([[1,4,7],[2,5,8],[3,6,9]])
print(C)
print('parte 1')
print(np.mean(C,axis=1))
print('parte 2')
print(np.mean(C,axis=0))
print('parte 3')
print(np.average(C,axis=1, weights=(1,1,2)))
print('parte 4')
print(np.percentile(C,25,axis=1))

```

    [[1 4 7]
     [2 5 8]
     [3 6 9]]
    parte 1
    [4. 5. 6.]
    parte 2
    [2. 5. 8.]
    parte 3
    [4.75 5.75 6.75]
    parte 4
    [2.5 3.5 4.5]