```python
#remove cell visibility
from IPython.display import HTML
tag = HTML('''
Promijeni vidljivost <a href="javascript:code_toggle()">ovdje</a>.''')
display(tag)
```

Promijeni vidljivost ovdje.

## Upravljanje bočnog položaja lunarnog prizemljivača

Ovaj primjer ilustrira razvoj regulatora (promatrača i kontrolera u formi povratne veze stanja) za upravljanje bočnim položajem lunarnog prizemljivača, počevši od jednadžbi sustava.

Sustav je predstavljen na gornjoj slici, njegov vertikalni spust usporen je pomoću vertikalnog potisnika koji proizvodi konstantnu silu $F$. Horizontalno kretanje može se postići laganim naginjanjem prizemljivača za kut $\theta$; naginjanjem se stvara bočna sila koja je približno jednaka $F\theta$. Nagib se postiže generiranjem momenta $T$ pomoću skupa upravljačkih raketa (maksimalni moment = 500 Nm). Kut nagiba mora biti unutar$\pm15$ stupnjeva kako bi se izbjeglo opasno povećanje brzine vertikalnog spusta. Izmjerene veličine su bočni položaj i brzina. Pretpostavlja se da je atmosferski otpor zanemariv, a vrijednosti parametara navedene su u donjoj tablici.

| Parametar |                     Vrijednost |
|-----------|-------------------------------:|
|$m$        |                        1000 kg |
|$J$        |           1000 kg$\text{m}^2$ |
|$F$        |                         1500 N |

Cilj dizajna upravljačkog sustava je postići sljedeće performanse za regulaciju vodoravnog položaja $z$:
1. Maksimalno prekoračenje od 30%.
2. Vrijeme smirivanja (za 5% pojasa tolerancije) manje od 15 sekundi.
3. Kut $\theta$ uvijek unutar svojih granica za željenu maksimalnu bočnu promjenu od 10 metara.
4. Nulta pogreška kao odziv na naredbeni korak.

Jednadžbe sustava su:

\begin{cases}
J\ddot{\theta}=T \\
m\ddot{z}=F\theta
\end{cases}
i definiranjem $\textbf{x}=[x_1,x_2,x_3,x_4]^T=[z,\dot{z},\theta,\dot{\theta}]^T$ kao vektora stanja te $u=T$ kao ulaza, u formi prostora stanja dobivamo:

\begin{cases}
\dot{\textbf{x}}=\underbrace{\begin{bmatrix}0&1&0&0 \\ 0&0&F/m&0 \\ 0&0&0&1 \\ 0&0&0&0\end{bmatrix}}_{A}\textbf{x}+\underbrace{\begin{bmatrix}0\\0\\0\\1/J\end{bmatrix}}_{B}u \\ \\
\textbf{y}=\underbrace{\begin{bmatrix}1&0&0&0 \\ 0&1&0&0\end{bmatrix}}_{C}\textbf{x}.
\end{cases}

### Dizajn kontrolera
Kako bi postigli nultu pogrešku za odziv na referentni korak, sustav se proširuje dodavanjem novog stanja $\dot{x_5}=y_1-y_d$ gdje je $y_1$ izmjereni bočni položaj, a $y_d$ je vrijednost željenog položaja. Stoga je prošireni sustav:

\begin{cases}
\dot{\textbf{x}_a}=\underbrace{\begin{bmatrix}0&1&0&0&0 \\ 0&0&F/m&0&0 \\ 0&0&0&1&0 \\ 0&0&0&0&0 \\ 1&0&0&0&0 \end{bmatrix}}_{A_a}\textbf{x}_a+\underbrace{\begin{bmatrix} 0&0\\0&0\\0&0\\1/J&0\\0&-1 \end{bmatrix}}_{B_a}\underbrace{\begin{bmatrix} u\\y_d \end{bmatrix}}_{u_a} \\ \\
\textbf{y}_a=\underbrace{\begin{bmatrix}1&0&0&0&0\\0&1&0&0&0\\0&0&0&0&1\end{bmatrix}}_{C_a}\textbf{x}_a
\end{cases}

koji se može kontrolirati (upravljiv) s prvim stupcem od $B_a$, tako da je moguće koristiti metodu postavljanja polova. Imajte na umu da je, kako bi se održala osmotrivost sustava, dodan red u matrici $C$ jer je poznato novo stanje $x_5$.

Matrica pojačanja $K_a$ koja zadovoljava sve zadane uvjete je:
$$
K_a=\begin{bmatrix}2225.0&6244.0&13861.0&5275.0&316.0\end{bmatrix}
$$
čime se polovi od $(A_a-B_aK_a)$ pozicioniraju na $-0.28$, $-2.24+2.23i$, $-2.24-2.23i$, $-0.26+0.32i$ i $-0.26-0.32i$.

### Dizajn promatrača
Sustav je osmotriv, a, budući da se mjere tri stanja, moguće je dizajnirati promatrač s reduciranim stanjima (za $\theta$ i $\dot{\theta}$) koji ima strukturu:
$$
\dot{\hat{\textbf{v}}}=(A_{11}+L_aA_{21})\hat{\textbf{v}}+(A_{12}+L_aA_{22}-A_{11}L_a-L_aA_{21}L_a)\textbf{y}_a+(B_1+L_aB_2)u_a,
$$
gdje je
$$
T^{-1}A_aT=\begin{bmatrix}A_{11}&A_{12} \\ A_{21}&A_{22}\end{bmatrix},
\quad T^{-1}B_a=\begin{bmatrix}B_1 \\ B_2\end{bmatrix},
\quad \overline{\textbf{x}_a}=T^{-1}\textbf{x}_a=\begin{bmatrix}V \\ C\end{bmatrix}\textbf{x}_a,
\quad V=\begin{bmatrix}0&0&1&0&0 \\ 0&0&0&1&0\end{bmatrix},
\quad \hat{\textbf{x}_a}=\begin{bmatrix}\hat{\textbf{v}}-L_a\textbf{y}_a \\ \textbf{y}_a\end{bmatrix}.
$$

Odabir svojstvenih vrijednosti promatrača vrši se na način da dinamika pogrešaka konvergira brže od dinamike sustava specificirane zahtjevima. Svojstvene vrijednosti odabrane za $A_{11}+L_aA_{21}$ su $\lambda_i=-10$ rad/s, $i=1,2$ uz
$$ L_a=\begin{bmatrix}0&-\frac{40}{3}&0 \\ 0&-\frac{200}{3}&0\end{bmatrix} $$

### Kako koristiti ovaj interaktivni primjer?
Učinkovitost sustava možete provjeriti s razvijenim regulatorom i izravno modificirati kontroler i/ili promatrač. Simulacija započinje početnom pogreškom promatrača.

```python
#Preparatory Cell

%matplotlib notebook
import control as ctrl
import numpy
import sympy as sym
from IPython.display import display, Markdown
import ipywidgets as widgets
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import matplotlib.patches as patches
import matplotlib.transforms as transforms
import matplotlib.lines as lines

#print a matrix latex-like
def bmatrix(a):
     """Returns a LaTeX bmatrix - by Damir Arbula (ICCT project)

     :a: numpy array
     :returns: LaTeX bmatrix as a string
     """
     if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
     lines = str(a).replace('[', '').replace(']', '').splitlines()
     rv = [r'\begin{bmatrix}']
     rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
     rv +=  [r'\end{bmatrix}']
     return '\n'.join(rv)

# Display formatted matrix:
def vmatrix(a):
    if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{vmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{vmatrix}']
    return '\n'.join(rv)

#matrixWidget is a matrix looking widget built with a VBox of HBox(es) that returns a numPy array as value !
class matrixWidget(widgets.VBox):
    def updateM(self,change):
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.M_[irow,icol] = self.children[irow].children[icol].value
                #print(self.M_[irow,icol])
        self.value = self.M_

    def dummychangecallback(self,change):
        pass

    def __init__(self,n,m):
        self.n = n
        self.m = m
        self.M_ = numpy.matrix(numpy.zeros((self.n,self.m)))
        self.value = self.M_
        widgets.VBox.__init__(self,
                             children = [
                                 widgets.HBox(children =
                                              [widgets.FloatText(value=0.0, layout=widgets.Layout(width='90px')) for i in range(m)]
                                             )
                                 for j in range(n)
                             ])

        #fill in widgets and tell interact to call updateM each time a children changes value
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
                self.children[irow].children[icol].observe(self.updateM, names='value')
        #value = Unicode('example@example.com', help="The email value.").tag(sync=True)
        self.observe(self.updateM, names='value', type= 'All')

    def setM(self, newM):
        #disable callbacks, change values, and reenable
        self.unobserve(self.updateM, names='value', type= 'All')
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].unobserve(self.updateM, names='value')
        self.M_ = newM
        self.value = self.M_
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].observe(self.updateM, names='value')
        self.observe(self.updateM, names='value', type= 'All')

                #self.children[irow].children[icol].observe(self.updateM, names='value')

#overlaod class for state space systems that DO NOT remove "useless" states (what "professor" of automatic control would do this?)
class sss(ctrl.StateSpace):
    def __init__(self,*args):
        #call base class init constructor
        ctrl.StateSpace.__init__(self,*args)
    #disable function below in base class
    def _remove_useless_states(self):
        pass
```

```python
# Define matrixes

A = numpy.matrix('0 1 0 0; 0 0 1.5 0; 0 0 0 1; 0 0 0 0')
B = numpy.matrix('0;0;0;0.001')
C = numpy.matrix('1 0 0 0; 0 1 0 0')
Aa = numpy.matrix('0 1 0 0 0; 0 0 1.5 0 0; 0 0 0 1 0; 0 0 0 0 0; 1 0 0 0 0')
Ba = numpy.matrix('0 0;0 0;0 0;0.001 0;0 -1')
Ca = numpy.matrix('1 0 0 0 0; 0 1 0 0 0; 0 0 0 0 1')
Ka1 = numpy.matrix('[2225.0, 6244.0, 13861.0, 5275.0, 316.0') #318.9333 835 2012.5 2000 59.6
Ta = (numpy.matrix('0 0 1 0 0; 0 0 0 1 0; 1 0 0 0 0;0 1 0 0 0; 0 0 0 0 1'))**(-1)
Ar = Ta**(-1)*Aa*Ta
Br = Ta**(-1)*Ba
A11 = Ar[0:2,0:2]
A12 = Ar[0:2,2:5]
A21 = Ar[2:5,0:2]
A22 = Ar[2:5,2:5]
B1 = Br[0:2,:]
B2 = Br[2:5,:]
La1 = numpy.matrix([[0, -4*10/3, 0],[0, -3/8*(-4*10/3)**2, 0]])
X0a = numpy.matrix('0;0;0;0;0;0;0;0;0;0;0.002;0.002;0;0;0;0;0;0.002;0.002;0;0;0;0;0')
# X0a = numpy.matrix('0;0;0;0;0')
# V0 = numpy.matrix('0;0')
```

```python
# Define matrixes widget
Kaw = matrixWidget(1,5)
Law = matrixWidget(2,3)
eig1 = matrixWidget(1,1)
eig2 = matrixWidget(2,1)
eig3 = matrixWidget(2,1)
eig4 = matrixWidget(1,1)
eig5 = matrixWidget(1,1)
eig1o = matrixWidget(1,1)
eig2o = matrixWidget(2,1)

Ydw = widgets.FloatSlider(
                         value=10,
                         min=0,
                         max=10.0,
                         step=0.1,
                         description='$y_d$:',
                         disabled=False,
                         continuous_update=False,
                         orientation='horizontal',
                         readout=True,
                         readout_format='.1f',
                        )

# Init matrix widgets
Kaw.setM(Ka1)
Law.setM(La1)
#[-0.6,-0.5-0.35j,-0.5+0.35j,-0.2-0.6j,-0.2+0.6j]
eig1.setM(numpy.matrix([-0.28]))
eig2.setM(numpy.matrix([[-2.24],[-2.23]]))
eig3.setM(numpy.matrix([[-0.26],[-0.32]]))
eig4.setM(numpy.matrix([-1]))
eig5.setM(numpy.matrix([-1]))
eig1o.setM(numpy.matrix([-10]))
eig2o.setM(numpy.matrix([[-10],[0]]))
```

```python
# Support functions
# Simulation function
def simulation(Aa, Baa, Ca, A11, A12, A21, A22, B1, B2, La, Ka, Ta):
    Aa, Baa, Ca = sym.Matrix(Aa), sym.Matrix(Baa), sym.Matrix(Ca)
    A11, A12, A21, A22 = sym.Matrix(A11), sym.Matrix(A12), sym.Matrix(A21), sym.Matrix(A22)
    B1, B2 = sym.Matrix(B1), sym.Matrix(B2)
    La, Ka = sym.Matrix(La), sym.Matrix(Ka)
    Ta = sym.Matrix(Ta)
    sysS = sss(Aa, Baa, Ca, sym.zeros(3,2))
    sysX = sss(Aa, Baa, sym.eye(5), sym.zeros(5,2))
    sysO1 = sss((A11+La*A21), (B1+La*B2).row_join(A12+La*A22-A11*La-La*A21*La), sym.eye(2), sym.zeros(2,5))
    sysO2 = ctrl.append(sysO1, sysS)
    sysO3 = ctrl.connect(sysO2, [[3, 3], [4, 4], [5, 5]], [1, 2, 6, 7], [1, 2, 3, 4, 5])
    sysO = sss(sysO3.A,
               sysO3.B*sym.eye(2).col_join(sym.eye(2)),
               Ta*(sym.eye(2).row_join(-La)).col_join(sym.zeros(3, 2).row_join(sym.eye(3)))*sysO3.C,
               sym.zeros(5,2))
    sysU = sss(sysO.A, sysO.B, -Ka*sysO.C, sym.zeros(1,2))
    sysT = ctrl.append(sysS, sysX, sysO, sysU)
    sysT1 = ctrl.connect(sysT, [[1, 14], [3, 14], [5, 14], [7, 14]], [2, 4, 6, 8], [i for i in range(1, 15)])
    sys = sss(sysT1.A, sysT1.B*sym.Matrix([1, 1, 1, 1]), sysT1.C, sym.zeros(14, 1))
    return sys

# check functions
def eigen_choice(selc,selo):
    if selc == '0 kompleksnih svojstvenih vrijednosti':
        eig2.children[1].children[0].disabled = True
        eig3.children[1].children[0].disabled = True
        eig3.children[0].children[0].disabled = False
        eig4.children[0].children[0].disabled = False
        eig5.children[0].children[0].disabled = False
        eigc = 0
    if selc == '2 kompleksne svojstvene vrijednosti':
        eig2.children[1].children[0].disabled = False
        eig3.children[1].children[0].disabled = True
        eig3.children[0].children[0].disabled = True
        eig4.children[0].children[0].disabled = False
        eig5.children[0].children[0].disabled = False
        eigc = 2
    if selc == '4 kompleksne svojstvene vrijednosti':
        eig2.children[1].children[0].disabled = False
        eig3.children[1].children[0].disabled = False
        eig3.children[0].children[0].disabled = False
        eig4.children[0].children[0].disabled = True
        eig5.children[0].children[0].disabled = True
        eigc = 4
    if selo == '0 kompleksnih svojstvenih vrijednosti':
        eig1o.children[0].children[0].disabled = False
        eig2o.children[1].children[0].disabled = True
        eigo = 0
    if selo == '2 kompleksne svojstvene vrijednosti':
        eig1o.children[0].children[0].disabled = True
        eig2o.children[1].children[0].disabled = False
        eigo = 2
    return (eigc, eigo)

def method_choice(selm):
    if selm == 'Postavi Ka i La':
        method = 1
        selc.disabled = True
        selo.disabled = True
    if selm == 'Postavi svojstvene vrijednosti':
        method = 2
        selc.disabled = False
        selo.disabled = False
    return method

# Animation functions
def fun_animation(index):
    global Ydw, yout, T
    yd = Ydw.value
    frame = 1

    linez.set_data(T[0:index*frame],yd*yout[0][0:index*frame])
    linezv.set_data(T[0:index*frame],yd*yout[1][0:index*frame])
    lined.set_data(T,[yd for i in range(0,len(T))])
    lineu.set_data(T[0:index*frame],yd*yout[13][0:index*frame])
    linelimu1.set_data(T,[500 for j in range(0,len(T))])
    linelimu2.set_data(T,[-500 for j in range(0,len(T))])
    linethetaest.set_data(T[0:index*frame],yd*yout[10][0:index*frame]*180/numpy.pi)
    linetheta.set_data(T[0:index*frame],yd*yout[5][0:index*frame]*180/numpy.pi)

    rotation_transform.clear().translate(yd*yout[0][index*frame]*numpy.cos(float(yd*yout[6][index*frame])), yd*yout[0][index*frame]*numpy.sin(float(yd*yout[6][index*frame]))).rotate(float(-yd*yout[6][index*frame]))

    return (linez,linezv,lined,lineu,linelimu1,linelimu2,linethetaest,linetheta)

def anim_init():
    linez.set_data([], [])
    linezv.set_data([], [])
    lined.set_data([], [])
    lineu.set_data([], [])
    linelimu1.set_data([], [])
    linelimu2.set_data([], [])
    linethetaest.set_data([], [])
    linetheta.set_data([], [])
    return (linez,linezv,lined,lineu,linelimu1,linelimu2,linethetaest,linetheta)

```

```python
# Main cell
# Data
global yd, T, yout
yd = 10.
T = []
yout = []

# Figures
fig = plt.figure(num='Simulacija sustava za upravljanje bočnog položaja lunarnog prizemljivača')
fig.set_size_inches((9.8, 6))
fig.set_tight_layout(True)

ax0 = fig.add_subplot(221)
ax0.set_title('Lunarni prizemljivač')
ax0.set_xlim(-12,12)
ax0.set_ylim(-4,4)
ax0.grid()
# ax0.axis('off')

ax1 = fig.add_subplot(222)
linez = ax1.plot([],[])[0]
linezv = ax1.plot([],[])[0]
lined = ax1.plot([],[])[0]
ax1.set_title('Bočni položaj i brzina')
ax1.set_xlabel('$t$ [s]')
ax1.set_ylabel('y [m], $\dot y$ [m/s]')
ax1.set_xlim([0,17])
ax1.axvline(x=0,color='black',linewidth=0.8)
ax1.axhline(y=0,color='black',linewidth=0.8)
ax1.grid()
ax1.legend(['Nočni položaj','Bočna brzina','Željena vrijednost'])

ax2 = fig.add_subplot(223)
lineu = ax2.plot([],[])[0]
linelimu1 = ax2.plot([],[],'r')[0]
linelimu2 = ax2.plot([],[],'r')[0]
ax2.set_title('Ulazni moment T')
ax2.set_xlabel('$t$ [s]')
ax2.set_ylabel('$T$ [Nm]')
ax2.set_xlim([0,17])
ax2.axvline(x=0,color='black',linewidth=0.8)
ax2.axhline(y=0,color='black',linewidth=0.8)
ax2.grid()
ax2.legend(['T','Limit'])

ax3 = fig.add_subplot(224)
linethetaest = ax3.plot([],[])[0]
linetheta = ax3.plot([],[])[0]
ax3.set_title(r'$\theta_{est}$ vs $\theta$')
ax3.set_xlabel('$t$ [s]')
ax3.set_ylabel(r'$\theta$ [deg]')
ax3.axvline(x=0,color='black',linewidth=0.8)
ax3.axhline(y=0,color='black',linewidth=0.8)
ax3.set_xlim([0,17])
ax3.grid()

# Patches
rotation_transform = transforms.Affine2D()
circle = patches.Circle((0, 0.6), fill=True, radius=0.5, ec='black', fc='gray', lw=1, zorder=20,
                        transform=rotation_transform + ax0.transData)
rect = patches.Rectangle((-1, -0.4), 2, 0.5, fill=True, ec='black', fc='gray', lw=1, zorder=20,
                         transform=rotation_transform + ax0.transData)
poly = patches.Polygon(numpy.stack(([-0.25, -0.15, 0.15, 0.25], [-0.8, -0.4, -0.4, -0.8])).T,
                       closed=True, fill=True, ec='black', fc='black', lw=1, zorder=20,
                       transform=rotation_transform + ax0.transData)
lleg = patches.Rectangle((-1, -1.2), 0.05, 1, angle=-15, fill=True, ec='black', fc='black', lw=1, zorder=10,
                         transform=rotation_transform + ax0.transData)
rleg = patches.Rectangle((1, -1.2), 0.05, 1, angle=15, fill=True, ec='black', fc='black', lw=1, zorder=10,
                         transform=rotation_transform + ax0.transData)
lfoot = patches.Rectangle((-1.1, -1.2), 0.2, 0.05, fill=True, ec='black', fc='black', lw=1, zorder=20,
                         transform=rotation_transform + ax0.transData)
rfoot = patches.Rectangle((0.9, -1.2), 0.2, 0.05, fill=True, ec='black', fc='black', lw=1, zorder=20,
                         transform=rotation_transform + ax0.transData)
ax0.add_patch(circle)
ax0.add_patch(rect)
ax0.add_patch(poly)
ax0.add_patch(lleg)
ax0.add_patch(rleg)
ax0.add_patch(lfoot)
ax0.add_patch(rfoot)
plt.show()

# Functions
def main_function(Ka,La,Ydw,eig1,eig2,eig3,eig4,eig5,eig1o,eig2o,selm,selc,selo,DW):
    global T, yout, yd, Aa, Ba, A11, A21
    method = method_choice(selm)
    eigc, eigo = eigen_choice(selc,selo)
    yd = Ydw
    ax1.set_ylim([-0.1*yd,yd*1.5])
    ax2.set_ylim([-51*yd,51*yd])
    ax3.set_ylim([-15,15])

    if method == 1: #Setted matrix gain
        sol = numpy.linalg.eig((Aa-Ba[:,0]*Ka))
        print('Svojstvene vrijednosti od Aa su: '+str(round(sol[0][0],3))+', '+str(round(sol[0][1],3))+', '+str(round(sol[0][2],3))+', '+str(round(sol[0][3],3))+' i '+str(round(sol[0][4],3)))
        sol = numpy.linalg.eig(A11+La*A21)
        print('Svojstvene vrijednosti od A11+La*A21 su: '+str(round(sol[0][0],3))+' i '+str(round(sol[0][1],3)))
        sys = simulation(Aa, Ba, Ca, A11, A12, A21, A22, B1, B2, La, Ka, Ta)
        T = numpy.linspace(0, 17, 100)
        T, yout = ctrl.step_response(sys, T, X0a)
    if method == 2: #Setted eigenvalues
        if eigc == 0:
            Ka = ctrl.acker(Aa, Ba[:,0], [eig1[0,0], eig2[0,0], eig3[0,0], eig4[0,0], eig5[0,0]])
            Kaw.setM(Ka)
        if eigc == 2:
            Ka = ctrl.acker(Aa, Ba[:,0], [eig1[0,0], numpy.complex(eig2[0,0],eig2[1,0]), numpy.complex(eig2[0,0],-eig2[1,0]), eig4[0,0], eig5[0,0]])
            Kaw.setM(Ka)
        if eigc == 4:
            Ka = ctrl.acker(Aa, Ba[:,0], [eig1[0,0], numpy.complex(eig2[0,0],eig2[1,0]), numpy.complex(eig2[0,0],-eig2[1,0]), numpy.complex(eig3[0,0],eig3[1,0]), numpy.complex(eig3[0,0],-eig3[1,0])])
            Kaw.setM(Ka)
        if eigo == 0:
            La = numpy.matrix([[0, 2*eig1o[0,0]/3 + 2*eig2o[0,0]/3, 0], [0, -2*eig1o[0,0]*eig2o[0,0]/3, 0]])
            Law.setM(La)
        if eigo == 2:
            La = numpy.matrix([[0, 2*numpy.complex(eig2o[0,0],eig2o[1,0])/3 + 2*numpy.complex(eig2o[0,0],-eig2o[1,0])/3, 0], [0, -2*numpy.complex(eig2o[0,0],eig2o[1,0])*numpy.complex(eig2o[0,0],-eig2o[1,0])/3, 0]])
            Law.setM(La)
        sol = numpy.linalg.eig((Aa-Ba[:,0]*Ka))
        print('Svojstvene vrijednosti od Aa su: '+str(round(sol[0][0],3))+', '+str(round(sol[0][1],3))+', '+str(round(sol[0][2],3))+', '+str(round(sol[0][3],3))+' i '+str(round(sol[0][4],3)))
        sol = numpy.linalg.eig(A11+La*A21)
        print('Svojstvene vrijednosti od A11+La*A21 su: '+str(round(sol[0][0],3))+' i '+str(round(sol[0][1],3)))
        sys = simulation(Aa, Ba, Ca, A11, A12, A21, A22, B1, B2, La, Ka, Ta)
        T = numpy.linspace(0, 17, 100)
        T, yout = ctrl.step_response(sys, T, X0a)

ani = animation.FuncAnimation(fig, fun_animation, init_func=anim_init, frames=100, repeat=True, interval=170, blit=True)

#create dummy widget
DW = widgets.FloatText(layout=widgets.Layout(width='0px', height='0px'))

#create button widget
START = widgets.Button(
    description='Test',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Test',
    icon='check'
)

def on_start_button_clicked(b):
    #This is a workaround to have intreactive_output call the callback:
    #   force the value of the dummy widget to change
    if DW.value> 0 :
        DW.value = -1
    else:
        DW.value = 1
    pass
START.on_click(on_start_button_clicked)

# Define type of method
selm = widgets.Dropdown(
    options= ['Postavi Ka i La', 'Postavi svojstvene vrijednosti'],
    value= 'Postavi Ka i La',
    description='',
    disabled=False
)

# Define the number of complex eigenvalues for the controller
selc = widgets.Dropdown(
    options= ['0 kompleksnih svojstvenih vrijednosti', '2 kompleksne svojstvene vrijednosti', '4 kompleksne svojstvene vrijednosti'],
    value= '4 kompleksne svojstvene vrijednosti',
    description='Aa:',
    disabled=False
)

# Define the number of complex eigenvalues for the observer
selo = widgets.Dropdown(
    options= ['0 kompleksnih svojstvenih vrijednosti', '2 kompleksne svojstvene vrijednosti'],
    value= '0 kompleksnih svojstvenih vrijednosti',
    description='Aobs:',
    disabled=False
)

alltogether = widgets.VBox([
    widgets.HBox([
        selm,
        selc,
        selo
    ]),
    widgets.Label('',border=3),
    widgets.HBox([
        widgets.Label('Ka:',border=3),
        Kaw,
        widgets.Label('',border=3),
        widgets.Label('',border=3),
        widgets.Label('La:',border=3),
        Law
    ]),
    widgets.Label('',border=3),
    widgets.HBox([
        widgets.Label('Svojstvene vrijednosti od Aa:',border=3),
        eig1, eig2, eig3, eig4, eig5,
        widgets.Label('',border=3),
        widgets.Label('',border=3),
        widgets.Label('Svojstvene vrijednosti od Aobs:',border=3),
        eig1o, eig2o
    ]),
    widgets.Label('',border=3),
    widgets.HBox([
        Ydw,
        widgets.Label('',border=3),
        widgets.Label('',border=3),
        widgets.Label('',border=3),
        START
    ])
])

out = widgets.interactive_output(main_function,{'Ka':Kaw, 'La':Law, 'Ydw':Ydw, 'eig1':eig1, 'eig2':eig2,
                                                'eig3':eig3, 'eig4':eig4, 'eig5':eig5,
                                                'eig1o':eig1o, 'eig2o':eig2o,
                                                'selm':selm, 'selc':selc, 'selo':selo, 'DW':DW})
display(out, alltogether)
```

    <IPython.core.display.Javascript object>

    Output()

    VBox(children=(HBox(children=(Dropdown(options=('Postavi Ka i La', 'Postavi svojstvene vrijednosti'), value='P…

```python

```


Processamento de Linguagem Natural

Gustavo C. A Corradi

robotreport.me

@gustavocorradi, @data4sci

# Lesson I - Text representation

In this lesson we will see in some details how we can best represent text in our application. Let's start by importing the modules we will be using:

```python
import string
from collections import Counter
from pprint import pprint
import gzip
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
```

We choose a well known nursery rhyme, that has the added distinction of having been the first audio ever recorded, to be the short snippet of text that we will use in our examples:

```python
text = """Mary had a little lamb, little lamb,
    little lamb. Mary had a little lamb
    whose fleece was white as snow.
    And everywhere that Mary went
    Mary went, Mary went. Everywhere
    that Mary went,
    The lamb was sure to go"""
```

## Tokenization

The first step in any analysis is to tokenize the text. What this means is that we will extract all the individual words in the text. For the sake of simplicity, we will assume that our text is well formed and that our words are delimited either by white space or punctuation characters.

```python
def extract_words(text):
    temp = text.split() # Split the text on whitespace
    text_words = []

    for word in temp:
        # Remove any punctuation characters present in the beginning of the word
        while word[0] in string.punctuation:
            word = word[1:]

        # Remove any punctuation characters present in the end of the word
        while word[-1] in string.punctuation:
            word = word[:-1]

        # Append this word into our list of words.
        text_words.append(word.lower())

    return text_words
```

After this step we now have our text represented as an array of individual, lowercase, words:

```python
text_words = extract_words(text)
print(text_words)
```

    ['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb', 'mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went', 'everywhere', 'that', 'mary', 'went', 'the', 'lamb', 'was', 'sure', 'to', 'go']

As we saw during the video, this is a wasteful way to represent text. We can be much more efficient by representing each word by a number

```python
word_dict = {}
word_list = []
vocabulary_size = 0
text_tokens = []

for word in text_words:
    # If we are seeing this word for the first time, create an id for it and added it to our word dictionary
    if word not in word_dict:
        word_dict[word] = vocabulary_size
        word_list.append(word)
        vocabulary_size += 1

    # add the token corresponding to the current word to the tokenized text.
    text_tokens.append(word_dict[word])
```

When we were tokenizing our text, we also generated a dictionary **word_dict** that maps words to integers and a **word_list** that maps each integer to the corresponding word.

```python
print("Word list:", word_list, "\n\n Word dictionary:")
pprint(word_dict)
```

    Word list: ['mary', 'had', 'a', 'little', 'lamb', 'whose', 'fleece', 'was', 'white', 'as', 'snow', 'and', 'everywhere', 'that', 'went', 'the', 'sure', 'to', 'go']

     Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

These two datastructures already proved their usefulness when we converted our text to a list of tokens.

```python
print(text_tokens)
```

    [0, 1, 2, 3, 4, 3, 4, 3, 4, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 0, 14, 0, 14, 0, 14, 12, 13, 0, 14, 15, 4, 7, 16, 17, 18]

Unfortunately, while this representation is convenient for memory reasons it has some severe limitations. Perhaps the most important of which is the fact that computers naturally assume that numbers can be operated on mathematically (by addition, subtraction, etc) in a way that doesn't match our understanding of words.

## One-hot encoding

One typical way of overcoming this difficulty is to represent each word by a one-hot encoded vector where every element is zero except the one corresponding to a specific word.

```python
def one_hot(word, word_dict):
    """
        Generate a one-hot encoded vector corresponding to *word*
    """

    vector = np.zeros(len(word_dict))
    vector[word_dict[word]] = 1

    return vector
```

So, for example, the word "fleece" would be represented by:

```python
fleece_hot = one_hot("fleece", word_dict)
print(fleece_hot)
```

    [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

This vector has every element set to zero, except element 6, since:

```python
print(word_dict["fleece"])
fleece_hot[6] == 1
```

    6

    True

## Bag of words

We can now use the one-hot encoded vector for each word to produce a vector representation of our original text, by simply adding up all the one-hot encoded vectors:

```python
text_vector1 = np.zeros(vocabulary_size)

for word in text_words:
    hot_word = one_hot(word, word_dict)
    text_vector1 += hot_word

print(text_vector1)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

In practice, we can also easily skip the encoding step at the word level by using the *word_dict* defined above:

```python
text_vector = np.zeros(vocabulary_size)

for word in text_words:
    text_vector[word_dict[word]] += 1

print(text_vector)
```

    [6. 2. 2. 4. 5. 1. 1. 2. 1. 1. 1. 1. 2. 2. 4. 1. 1. 1. 1.]

Naturally, this approach is completely equivalent to the previous one and has the added advantage of being more efficient in terms of both speed and memory requirements.

This is known as the __bag of words__ representation of the text. It should be noted that these vectors simply contains the number of times each word appears in our document, so we can easily tell that the word *mary* appears exactly 6 times in our little nursery rhyme.

```python
text_vector[word_dict["mary"]]
```

    6.0

A more pythonic (and efficient) way of producing the same result is to use the standard __Counter__ module:

```python
word_counts = Counter(text_words)
pprint(word_counts)
```

    Counter({'mary': 6,
             'lamb': 5,
             'little': 4,
             'went': 4,
             'had': 2,
             'a': 2,
             'was': 2,
             'everywhere': 2,
             'that': 2,
             'whose': 1,
             'fleece': 1,
             'white': 1,
             'as': 1,
             'snow': 1,
             'and': 1,
             'the': 1,
             'sure': 1,
             'to': 1,
             'go': 1})

From which we can easily generate the __text_vector__ and __word_dict__ data structures:

```python
items = list(word_counts.items())

# Extract word dictionary and vector representation
word_dict2 = dict([[items[i][0], i] for i in range(len(items))])
text_vector2 = [items[i][1] for i in range(len(items))]
```

And let's take a look at them:

```python
print("Text vector:", text_vector2, "\n\nWord dictionary:")
pprint(word_dict2)
```

    Text vector: [6, 2, 2, 4, 5, 1, 1, 2, 1, 1, 1, 1, 2, 2, 4, 1, 1, 1, 1]

    Word dictionary:
    {'a': 2,
     'and': 11,
     'as': 9,
     'everywhere': 12,
     'fleece': 6,
     'go': 18,
     'had': 1,
     'lamb': 4,
     'little': 3,
     'mary': 0,
     'snow': 10,
     'sure': 16,
     'that': 13,
     'the': 15,
     'to': 17,
     'was': 7,
     'went': 14,
     'white': 8,
     'whose': 5}

The results using this approach are slightly different than the previous ones, because the words are mapped to different integer ids but the corresponding values are the same:

```python
for word in word_dict.keys():
    if text_vector[word_dict[word]] != text_vector2[word_dict2[word]]:
        print("Error!")
```

As expected, there are no differences!

## Term Frequency

The bag of words vector representation introduced above relies simply on the frequency of occurence of each word. Following a long tradition of giving fancy names to simple ideas, this is known as __Term Frequency__.

Intuitively, we expect the the frequency with which a given word is mentioned should correspond to the relevance of that word for the piece of text we are considering. For example, **Mary** is a pretty important word in our little nursery rhyme and indeed it is the one that occurs the most often:

```python
sorted(items, key=lambda x:x[1], reverse=True)
```

    [('mary', 6),
     ('lamb', 5),
     ('little', 4),
     ('went', 4),
     ('had', 2),
     ('a', 2),
     ('was', 2),
     ('everywhere', 2),
     ('that', 2),
     ('whose', 1),
     ('fleece', 1),
     ('white', 1),
     ('as', 1),
     ('snow', 1),
     ('and', 1),
     ('the', 1),
     ('sure', 1),
     ('to', 1),
     ('go', 1)]

However, it's hard to draw conclusions from such a small piece of text. Let us consider a significantly larger piece of text, the first 100 MB of the english Wikipedia from: http://mattmahoney.net/dc/textdata. For the sake of convenience, text8.gz has been included in this repository in the **data/** directory. We start by loading it's contents into memory as an array of words:

```python
data = []

for line in gzip.open("data/text8.gz", 'rt'):
    data.extend(line.strip().split())
```

Now let's take a look at the most common words in this large corpus:

```python
counts = Counter(data)

sorted_counts = sorted(list(counts.items()), key=lambda x:x[1], reverse=True)

for word, count in sorted_counts[:10]:
    print(word, count)
```

    the 1061396
    of 593677
    and 416629
    one 411764
    in 372201
    a 325873
    to 316376
    zero 264975
    nine 250430
    two 192644

Surprisingly, we find that the most common words are not particularly meaningful. Indeed, this is a common occurence in Natural Language Processing. The most frequent words are typically auxiliaries required due to gramatical rules.

On the other hand, there is also a large number of words that occur very infrequently as can be easily seen by glancing at the word freqency distribution.

```python
dist = Counter(counts.values())
dist = list(dist.items())
dist.sort(key=lambda x:x[0])
dist = np.array(dist)

norm = np.dot(dist.T[0], dist.T[1])

plt.loglog(dist.T[0], dist.T[1]/norm)
plt.xlabel("count")
plt.ylabel("P(count)")
plt.title("Word frequency distribution")
```

## Stopwords

One common technique to simplify NLP tasks is to remove what are known as Stopwords, words that are very frequent but not meaningful. If we simply remove the most common 100 words, we significantly reduce the amount of data we have to consider while losing little information.

```python
stopwords = set([word for word, count in sorted_counts[:100]])

clean_data = []

for word in data:
    if word not in stopwords:
        clean_data.append(word)

print("Original size:", len(data))
print("Clean size:", len(clean_data))
print("Reduction:", 1-len(clean_data)/len(data))
```

    Original size: 17005207
    Clean size: 9006229
    Reduction: 0.470384041782026

Wow, our dataset size was reduced almost in half!

In practice, we don't simply remove the most common words in our corpus but rather a manually curate list of stopwords. Lists for dozens of languages and applications can easily be found online.

## Term Frequency/Inverse Document Frequency

One way of determining of the relative importance of a word is to see how often it appears across multiple documents. Words that are relevant to a specific topic are more likely to appear in documents about that topic and much less in documents about other topics. On the other hand, less meaningful words (like **the**) will be common across documents about any subject.

To measure the document frequency of a word we will need to have multiple documents. For the sake of simplicity, we will treat each sentence of our nursery rhyme as an individual document:

```python
corpus_text = text.split('.')
corpus_words = []

for document in corpus_text:
    doc_words = extract_words(document)
    corpus_words.append(doc_words)
```

Now our corpus is represented as a list of word lists, where each list is just the word representation of the corresponding sentence:

```python
pprint(corpus_words)
```

    [['mary', 'had', 'a', 'little', 'lamb', 'little', 'lamb', 'little', 'lamb'],
     ['mary',
      'had',
      'a',
      'little',
      'lamb',
      'whose',
      'fleece',
      'was',
      'white',
      'as',
      'snow'],
     ['and', 'everywhere', 'that', 'mary', 'went', 'mary', 'went', 'mary', 'went'],
     ['everywhere',
      'that',
      'mary',
      'went',
      'the',
      'lamb',
      'was',
      'sure',
      'to',
      'go']]

Let us now calculate the number of documents in which each word appears:

```python
document_count = {}

for document in corpus_words:
    word_set = set(document)

    for word in word_set:
        document_count[word] = document_count.get(word, 0) + 1

pprint(document_count)
```

    {'a': 2,
     'and': 1,
     'as': 1,
     'everywhere': 2,
     'fleece': 1,
     'go': 1,
     'had': 2,
     'lamb': 3,
     'little': 2,
     'mary': 4,
     'snow': 1,
     'sure': 1,
     'that': 2,
     'the': 1,
     'to': 1,
     'was': 2,
     'went': 2,
     'white': 1,
     'whose': 1}

As we can see, the word __Mary__ appears in all 4 of our documents, making it useless when it comes to distinguish between the different sentences. On the other hand, words like __white__ which appear in only one document are very discriminative. Using this approach we can define a new quantity, the ___Inverse Document Frequency__ that tells us how frequent a word is across the documents in a specific corpus:

```python
def inv_doc_freq(corpus_words):
    number_docs = len(corpus_words)

    document_count = {}

    for document in corpus_words:
        word_set = set(document)

        for word in word_set:
            document_count[word] = document_count.get(word, 0) + 1

    IDF = {}

    for word in document_count:
        IDF[word] = np.log(number_docs/document_count[word])

    return IDF
```

Where we followed the convention of using the logarithm of the inverse document frequency. This has the numerical advantage of avoiding to have to handle small fractional numbers.

We can easily see that the IDF gives a smaller weight to the most common words and a higher weight to the less frequent:

```python
IDF = inv_doc_freq(corpus_words)

pprint(IDF)
```

    {'a': 0.6931471805599453,
     'and': 1.3862943611198906,
     'as': 1.3862943611198906,
     'everywhere': 0.6931471805599453,
     'fleece': 1.3862943611198906,
     'go': 1.3862943611198906,
     'had': 0.6931471805599453,
     'lamb': 0.28768207245178085,
     'little': 0.6931471805599453,
     'mary': 0.0,
     'snow': 1.3862943611198906,
     'sure': 1.3862943611198906,
     'that': 0.6931471805599453,
     'the': 1.3862943611198906,
     'to': 1.3862943611198906,
     'was': 0.6931471805599453,
     'went': 0.6931471805599453,
     'white': 1.3862943611198906,
     'whose': 1.3862943611198906}

As expected **Mary** has the smallest weight of all words 0, meaning that it is effectively removed from the dataset. You can consider this as a way of implicitly identify and remove stopwords. In case you do want to keep even the words that appear in every document, you can just add a 1. to the argument of the logarithm above:

\begin{equation}
\log\left[1+\frac{N_d}{N_d\left(w\right)}\right]
\end{equation}

When we multiply the term frequency of each word by it's inverse document frequency, we have a good way of quantifying how relevant a word is to understand the meaning of a specific document.

```python
def tf_idf(corpus_words):
    IDF = inv_doc_freq(corpus_words)

    TFIDF = []

    for document in corpus_words:
        TFIDF.append(Counter(document))

    for document in TFIDF:
        for word in document:
            document[word] = document[word]*IDF[word]

    return TFIDF
```

```python
tf_idf(corpus_words)
```

    [Counter({'mary': 0.0,
              'had': 0.6931471805599453,
              'a': 0.6931471805599453,
              'little': 2.0794415416798357,
              'lamb': 0.8630462173553426}),
     Counter({'mary': 0.0,
              'had': 0.6931471805599453,
              'a': 0.6931471805599453,
              'little': 0.6931471805599453,
              'lamb': 0.28768207245178085,
              'whose': 1.3862943611198906,
              'fleece': 1.3862943611198906,
              'was': 0.6931471805599453,
              'white': 1.3862943611198906,
              'as': 1.3862943611198906,
              'snow': 1.3862943611198906}),
     Counter({'and': 1.3862943611198906,
              'everywhere': 0.6931471805599453,
              'that': 0.6931471805599453,
              'mary': 0.0,
              'went': 2.0794415416798357}),
     Counter({'everywhere': 0.6931471805599453,
              'that': 0.6931471805599453,
              'mary': 0.0,
              'went': 0.6931471805599453,
              'the': 1.3862943611198906,
              'lamb': 0.28768207245178085,
              'was': 0.6931471805599453,
              'sure': 1.3862943611198906,
              'to': 1.3862943611198906,
              'go': 1.3862943611198906})]

Now we finally have a vector representation of each of our documents that takes the informational contributions of each word into account. Each of these vectors provides us with a unique representation of each document, in the context (corpus) in which it occurs, making it posssible to define the similarity of two documents, etc.

## Porter Stemmer

There is still, however, one issue with our approach to representing text. Since we treat each word as a unique token and completely independently from all others, for large documents we will end up with many variations of the same word such as verb conjugations, the corresponding adverbs and nouns, etc.

One way around this difficulty is to use stemming algorithm to reduce words to their root (or stem) version. The most famous Stemming algorithm is known as the **Porter Stemmer** and was introduced by Martin Porter in 1980 [Program 14, 130 (1980)](https://dl.acm.org/citation.cfm?id=275705)

The algorithm starts by defining consonants (C) and vowels (V):

```python
V = set('aeiouy')
C = set('bcdfghjklmnpqrstvwxz')
```

The stem of a word is what is left of that word after a speficic ending has been removed. A function to do this is easy to implement:

```python
def get_stem(suffix, word):
    """
        Extract the stem of a word
    """

    if word.lower().endswith(suffix.lower()): # Case insensitive comparison
        return word[:-len(suffix)]

    return None
```

It also defines words (or stems) to be sequences of vowels and consonants of the form:

\begin{equation}
[C](VC)^m[V]
\end{equation}

where $m$ is called the **measure** of the word and [] represent optional sections.

```python
def measure(orig_word):
    """
        Calculate the "measure" m of a word or stem, according to the Porter Stemmer algorthim
    """

    word = orig_word.lower()

    optV = False
    optC = False
    VC = False
    m = 0

    pos = 0

    # We can think of this implementation as a simple finite state machine
    # looks for sequences of vowels or consonants depending of the state
    # in which it's in, while keeping track of how many VC sequences it
    # has encountered.
    # The presence of the optional V and C portions is recorded in the
    # optV and optC booleans.

    # We're at the initial state.
    # gobble up all the optional consonants at the beginning of the word
    while pos < len(word) and word[pos] in C:
        pos += 1
        optC = True

    while pos < len(word):
        # Now we know that the next state must be a vowel
        while pos < len(word) and word[pos] in V:
            pos += 1
            optV = True

        # Followd by a consonant
        while pos < len(word) and word[pos] in C:
            pos += 1
            optV = False

        # If a consonant was found, then we matched VC
        # so we should increment m by one. Otherwise,
        # optV remained true and we simply had a dangling
        # V sequence.
        if not optV:
            m += 1

    return m
```

Let's consider a simple example. The word __crepusculars__ should have measure 4:

[cr] (ep) (usc) (ul) (ars)

and indeed it does.

```python
word = "crepusculars"
print(measure(word))
```

    4

The Porter algorithm sequentially applies a series of transformation rules over a series of 5 steps (step 1 is divided in 3 substeps and step 5 in 2). The rules are only applied if a certain condition is true.

In addition to possibily specifying a requirement on the measure of a word, conditions can make use of different boolean functions as well:

```python
def ends_with(char, stem):
    """
        Checks the ending of the word
    """
    return stem[-1] == char

def double_consonant(stem):
    """
        Checks the ending of a word for a double consonant
    """
    if len(stem) < 2:
        return False

    if stem[-1] in C and stem[-2] == stem[-1]:
        return True

    return False

def contains_vowel(stem):
    """
        Checks if a word contains a vowel or not
    """
    return len(set(stem) & V) > 0
```

Finally, we define a function to apply a specific rule to a word or stem:

```python
def apply_rule(condition, suffix, replacement, word):
    """
        Apply Porter Stemmer rule.
        if "condition" is True replace "suffix" by "replacement" in "word"
    """

    stem = get_stem(suffix, word)

    if stem is not None and condition is True:
        # Remove the suffix
        word = stem

        # Add the replacement suffix, if any
        if replacement is not None:
            word += replacement

    return word
```

Now we can see how rules can be applied. For example, this rule, from step 1b is successfully applied to __pastered__:

```python
word = "plastered"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'plaster'

While try applying the same rule to **bled** will fail to pass the condition resulting in no change.

```python
word = "bled"
suffix = "ed"
stem = get_stem(suffix, word)
apply_rule(contains_vowel(stem), suffix, None, word)
```

    'bled'

For a more complex example, we have, in Step 4:

```python
word = "adoption"
suffix = "ion"
stem = get_stem(suffix, word)
apply_rule(measure(stem) > 1 and (ends_with("s", stem) or ends_with("t", stem)), suffix, None, word)
```

    'adopt'

In total, the Porter Stemmer algorithm (for the English language) applies several dozen rules (see https://tartarus.org/martin/PorterStemmer/def.txt for a complete list). Implementing all of them is both tedious and error prone, so we abstain from providing a full implementation of the algorithm here. High quality implementations can be found in all major NLP libraries such as [NLTK](http://www.nltk.org/howto/stem.html).

The dificulties of defining matching rules to arbitrary text cannot be fully resolved without the use of Regular Expressions (typically implemented as Finite State Machines like our __measure__ implementation above), a more advanced topic that is beyond the scope of this course.


# The Jupyter Notebook

It is a **```web application```** that allows you to create and share documents that contain
- live code
- equations
- visualizations
- explanatory text

Table of Contents

The Jupyter Notebook

搜狗输入法表情和符号

一级标题

二级标题

运行C代码

Jupyter 魔术命令

END

```python
# my first python script
print("hello world! \n I am Cheng-Jun Wang.")
```

    hello world!
     I am Cheng-Jun Wang.

Uses include:
- data cleaning and transformation,
- numerical simulation,
- statistical modeling,
- machine learning
- and much more.

```python
print('hello world')
```

    hello world

```python
1 + 1
```

    2

$E = MC^2$

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

# 搜狗输入法表情和符号

- ∫ ∑ ※ ➕➖✖➗ ❎ √ ×
- 😪😠😡😎☺😁📚🌲
- 👌👍👎👂👃👀✋❌💰🌂
- 0⃣1⃣2⃣3⃣4⃣5⃣6⃣7⃣8⃣9⃣②🔟
- 🐶🐱🐔🐷🐖🐴🐎🐂🐑🐯🐧🐺🐒🐵🐻🐦🐲
- 💻 🌈🌎☁❄🏃♀👩👱✨
- 🆚🔥🌹✈🌉🎄

(✿◡‿◡)害羞 ⁄(⁄ ⁄•⁄ω⁄•⁄ ⁄)⁄ d=====(￣▽￣*)b厉害

我是我，不一样花火。～(￣▽￣～)(～￣▽￣)～ 矜持

```python
%matplotlib inline
import matplotlib.pyplot as plt

xi = [1, 2, 3, 4, 5]
y = [3, 5, 9, 13, 16]

plt.plot(xi, y, 'g-s')
plt.xlabel('$x_i$', fontsize = 20)
plt.ylabel('$y$', fontsize = 20)
plt.title('$Scatter\,Plot$', fontsize = 20)
plt.show()
```

# 一级标题
## 二级标题
[复旦大学](http://www.fdu.edu.cn)是一个*非常棒*的大学！

1. point 1
1. point 2
1. point 3

# 运行C代码

C functions are typically split into header files (.h) where things are declared but not defined, and implementation files (.c) where they are defined. http://people.duke.edu/~ccc14/sta-663/CrashCourseInC.html#a-tutorial-example-coding-a-fibonacci-function-in-c

```python
%%file hello.c
#include <stdio.h>

int main() {
    printf("Hello, world!");
}
```

    Overwriting hello.c

```python
! gcc hello.c -o hello # 编译
```

    xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

```python
! ./hello # 执行
```

    Hello, world!

# Jupyter 魔术命令

```python
%lsmagic
```

    Available line magics:
    %alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

    Available cell magics:
    %%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%markdown  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

    Automagic is ON, % prefix IS NOT needed for line magics.

>  pip install version_information

```python
!pip install version_information
```

    Requirement already satisfied: version_information in /Users/datalab/Applications/anaconda/lib/python3.5/site-packages (1.0.3)
    You are using pip version 19.0.3, however version 19.1.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.

```python
# install version_information in the terminal first.
%reload_ext version_information
%version_information numpy, matplotlib, pandas, scipy, statsmodels
```

Software	Version

Python	3.5.4 64bit [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]

IPython	6.2.1

OS	Darwin 18.6.0 x86_64 i386 64bit

numpy	1.16.3

matplotlib	3.0.1

pandas	0.23.4

scipy	1.1.0

statsmodels	0.9.0

Fri Jun 07 14:29:42 2019 CST

# END

This is the end.

```python

```


|
Name
               |
Date
 |
| ---------------------------------------------------| ------------------------------------- |
|
Diaaeldin SHALABY
   | 18.06.2021                            |

Hands-on AI II

Unit 7 — Introduction to Reinforcement Learning (Assignment)

Authors: B. Schäfl, S. Lehner, J. Brandstetter

Date: 11-06-2021

This file is part of the "Hands-on AI II" lecture material. The following copyright statement applies to all code within this file.

Copyright statement:

This  material,  no  matter  whether  in  printed  or  electronic  form,  may  be  used  for personal  and non-commercial educational use only.  Any reproduction of this manuscript, no matter whether as a whole or in parts, no matter whether in printed or in electronic form, requires explicit prior acceptance of the authors.

Table of contents

Dissection of an Environment

States and actions

Tackling the Environment with Random Exploration

Implementing random search

The problem with random search

Tackling the Environment with $Q$-Learning

First approach of learning a $Q$-table

The problem with missing exploration

Evaluate the agent's performance

The role of randomness in the environment

How to use this notebook

This notebook is designed to run from start to finish. There are different tasks (displayed in orange boxes) which require your contribution (in form of code, plain text, ...). Most/All of the supplied functions are imported from the file u7_utils.py which can be seen and treated as a black box. However, for further understanding, you can look at the implementations of the helper functions. In order to run this notebook, the packages which are imported at the beginning of u7_utils.py need to be installed.

```python
# Import pre-defined utilities specific to this notebook.
import u7_utils as u7

# Import additional utilities needed in this notebook.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import sys
import time

from IPython import display
from typing import Any, Dict, Tuple

# Setup Jupyter notebook (warning: this may affect all Jupyter notebooks running on the same Jupyter server).
u7.setup_jupyter()
```

Setting up notebook ... finished.

Module versions

As mentioned in the introductory slides, specific minimum versions of Python itself as well as of used modules is recommended.

```python
u7.check_module_versions()
```

    Installed Python version: 3.8 (✓)
    Installed numpy version: 1.19.1 (✓)
    Installed pandas version: 1.1.3 (✓)
    Installed PyTorch version: 1.7.1 (✓)
    Installed scikit-learn version: 0.23.2 (✓)
    Installed scipy version: 1.5.0 (✓)
    Installed matplotlib version: 3.3.1 (✓)
    Installed seaborn version: 0.11.0 (✓)
    Installed PIL version: 8.0.0 (✓)
    Installed rdkit version: 2020.09.1 (✓)
    Installed gym version: 0.18.3 (✓)

Dissection of an Environment

All exercises in this assignment are referring to the FrozenLake-v0 environment of OpenAI Gym. This environment is descibed according to its official OpenAI Gym website as follows:

    Winter is here. You and your friends were tossing around a frisbee at the park when you made a wild throw that left the frisbee out in the middle of the lake. The water is mostly frozen, but there are a few holes where the ice has melted. If you step into one of those holes, you'll fall into the freezing water. At this time, there's an international frisbee shortage, so it's absolutely imperative that you navigate across the lake and retrieve the disc. However, the ice is slippery, so you won't always move in the direction you intend.

There are four types of surfaces described in this environment:

S $\rightarrow$ starting point (safe)

F $\rightarrow$ frozen surface (safe)

H $\rightarrow$ hole (fall to your doom)

G $\rightarrow$ goal (frisbee location)

If not already done, more information on how to install and import the gym module is available in the lecture's notebook.

    Execute the notebook until here and try to solve the following tasks:

Create a new instance of <tt>FrozenLakeEnv</tt> with the seed set to <tt>42</tt> and render the current state in a human-readable way.

Gather and print the amount of different actions as well as states of the <tt>FrozenLakeEnv</tt> instance. Discuss the results.

Display the reward table entry for the current state. Discuss the different elements of the resulting dictionary.

```python
enviroment_lake = u7.FrozenLakeEnv()
u7.set_environment_seed(environment=enviroment_lake, seed=42)
```

```python
enviroment_lake.render(mode=r'human')
current_state_id = enviroment_lake.s
print(f'Current state ID: {current_state_id}')
```

    SFFF
    FHFH
    FFFH
    HFFG
    Current state ID: 0

The first and naïve approach to solve this task is by simply brute forcing it applying random search. The outline of this approach is the following:

        Step
        Description

        0
        Choose a random action with respect to the current state.

        1
        Execute previously chosen action and transition into a new state.

        2
        Repeat the previous steps as long as the current episode is still ongoing.

For such an approach to be at least remotely applicable, the number of possible actions and states is of utter importance. Otherwise, we are lost in the depth of combinatorial explosion. The property n of the action_space and observation_space of the respective environment gives the amount of actions as well as states.

```python
num_actions = enviroment_lake.action_space.n
num_states = enviroment_lake.observation_space.n
print(f'The FrozenLake-v0 environment comprises <{num_actions}> actions and <{num_states}> states.')
```

    The FrozenLake-v0 environment comprises <4> actions and <16> states.

```python
current_state_id = enviroment_lake.s
enviroment_lake.P[current_state_id]
```

    {0: [(0.3333333333333333, 0, 0.0, False),
      (0.3333333333333333, 0, 0.0, False),
      (0.3333333333333333, 4, 0.0, False)],
     1: [(0.3333333333333333, 0, 0.0, False),
      (0.3333333333333333, 4, 0.0, False),
      (0.3333333333333333, 1, 0.0, False)],
     2: [(0.3333333333333333, 4, 0.0, False),
      (0.3333333333333333, 1, 0.0, False),
      (0.3333333333333333, 0, 0.0, False)],
     3: [(0.3333333333333333, 1, 0.0, False),
      (0.3333333333333333, 0, 0.0, False),
      (0.3333333333333333, 0, 0.0, False)]}

Each entry of the reward table contains a dictionary of the form s: {a: [] for a in range(nA)} for s in range(nS).

        Element
        Description

        nS=nrows*ncols
        The number of possible moves.

        nA=4
        Number of surfaces. [S,F,H,G]

        a
        Potential state based on the surface of the next move.

Tackling the Environment with Random Exploration

Previously, we talked about solving this task in a naïve way by simply applying brute force: using random search. In the meantime we analyzed the action as well as the state space and came to the conclusion, that such an approach is more than feasible. To repeat the outline of such an approach:

I $\rightarrow$ choose a random action with respect to the current state.

II $\rightarrow$ execute previously chosen action and transition into a new state.

III $\rightarrow$ if the episode is finished, but the goal not reached, reset the position of the disc retrieving entity.

This procedure is repeated as long as the task is not solved or a defined maximum of steps is reached, whatever triggers first (IV). Adapt the function apply_random_search as discussed during the lecture. Mark the corresponding sections of the code using I, II, III and IV. Note that our random search is not guaranteed to find  the solution of a task in finite time, hence an upper border on the runtime is often applied as a safety net (in our case the number of allowed steps).

    Execute the notebook until here and try to solve the following tasks:

Implement the random search algorithm as outlined above (equivalently to the one discussed during the exercise).

Apply your random search implementation on a freshly seeded <tt>FrozenLakeEnv</tt> instance, with an animation delay of $0.1$.

How many steps are necessary to reach the goal at least once and how often did an involuntary dive happen?

```python
def apply_random_search(environment: u7.FrozenLakeEnv, animate: bool = False,
                        delay: float = 0.01, max_steps: int = 1000) -> Tuple[int, int, Dict[str, Any]]:
    """
    Solve specified environment by applying random search.

    :param environment: the environment on which to apply random search
    :param animate: animate the random search process
    :param delay: the minimum delay in milliseconds between each rendered frame (ignored if not animated)
    :param max_steps: maximum amount of steps to perform
    :return: amount of steps performed, involuntary dives and captured frames
    """
    num_steps, num_penalties, final_reward, captured_frames = 0, 0, 0, []

    # <III>: repeat random search procedure as long as the episode is still ongoing.

    u7.set_environment_seed(environment=enviroment_lake, seed=42)

    done = False
    while not done:
        if num_steps == max_steps:
            break

        # <I>: choose a random action with respect to the current state.
        current_action = environment.action_space.sample()

        # <II>: execute previously chosen action and transition into a new state.
        current_state, current_reward, done, info = environment.step(current_action)

        # Update counter for inflicted penalties.
        final_reward += current_reward
        if current_reward <= 0:
            num_penalties += 1
        num_steps += 1

        # Save rendering of current state.
        captured_frames.append({
            r'frame': environment.render(mode=r'ansi'),
            r'state': current_state,
            r'action': current_action,
            r'reward': current_reward
        })

        # Optionally display current state.
        if animate:
            display.clear_output(wait=True)
            print(captured_frames[-1][r'frame'])
            print(f'Step No.: {num_steps}'
                  f'\nState ID: {current_state}'
                  f'\nAction ID: {current_action}'
                  f'\nReward: {current_reward}')
            time.sleep(delay)

        if not done and current_reward == 0:
            enviroment_lake.reset()

    return num_steps, num_penalties, final_reward, captured_frames
```

```python
u7.set_environment_seed(environment=enviroment_lake, seed=42)
num_steps, num_penalties, final_reward, _ = apply_random_search(
    environment=enviroment_lake,
    animate=True,
    delay=0.1
)
```

      (Right)
    SFFF
    FHFH
    FFFH
    HFFG

    Step No.: 1000
    State ID: 1
    Action ID: 2
    Reward: 0.0

Unfortunatly the function did not find a solution.

To drill down on the drawbacks of plain random search, we are designing the following experimental setup (hint: it is actually the same experimental setup as already discussed during the exercise, so you might orient yourself on the implementation presented during class):

Repeat the previous random search procedure a specified amount of times.

Aggregate the results of each run for later analysis.

Visualise the aggegrated results using box- and swarm-plots.

Once again, we are setting the random seed, but take care of setting it outside the loop, otherwise the same result is reported with each iteration (and an aggregation of the results would not give us any more insights).

    Execute the notebook until here and try to solve the following tasks:

Conduct a random search experiment as outlined above, using $100$ repetitions and the random seed set to $42$.

Interpret the visualization (e.g. the span of the boxes) and keep the scaling of the x-axis in mind.

In comparison with the <tt>Taxi-v3</tt> environment, what might be the problem with <tt>FrozenLakeEnv</tt> w.r.t. random exploration?

```python
u7.set_environment_seed(environment=enviroment_lake, seed=42)
num_steps_total, num_penalties_total, final_reward_total = [], [], []
num_repetitions = 100

# Collect information over multiple repetitions.
for repetition in range(num_repetitions):
    enviroment_lake.reset()
    num_steps, num_penalties, final_reward, _ = apply_random_search(environment=enviroment_lake)
    num_steps_total.append(num_steps)
    num_penalties_total.append(num_penalties)
    final_reward_total.append(final_reward)

# Combine collected information to a data frame for further downstream analysis.
collected_experiment_info = pd.DataFrame(zip(
    num_steps_total, num_penalties_total, final_reward_total
), columns=(r'Steps performed', r'Penalties inflicted', r'Final reward'))
```

```python
# Set default plotting style.
sns.set()

# Visualize aggregated results of the random search procedure.
fig, ax = plt.subplots(nrows=1, ncols=1, squeeze=False, figsize=(20, 7))
ax[0, 0].set_xscale(r'symlog')
_ = sns.boxplot(data=collected_experiment_info, ax=ax[0, 0], orient=r'h')
_ = sns.swarmplot(data=collected_experiment_info, ax=ax[0, 0], color=r'0.3', orient=r'h')
```

Tackling the Environment with $Q$-Learning

In a simplified version of $Q$-learning, the $\boldsymbol{Q}$-value
\begin{equation}
    Q(s,a)
\end{equation}

is the expected future reward of being in state $s$ and taking action $a$. Intuitively, if the the $Q$-values are learned correctly, a good policy would be to take the action which maximizes the expected future reward. This is what $Q$-learning is doing. $Q$-learning lets the agent use the environment's rewards to learn, over time, the best action to take in a given state. $Q$-values are initialized to an arbitrary value, and as the agent exposes itself to the environment and receives different rewards by executing different actions, the $Q$-values are updated using the equation:
\begin{equation}
    Q(s_t,a_t) \leftarrow (1 - \alpha) \cdot Q(s_t,a_t) + \alpha \cdot \left( r + \max_{a_{t+1}} Q(s_{t+1}, a_{t+1})\right)
\end{equation}

We are assigning $\leftarrow$, or updating, the $Q$-value of the agent's current state and action, denoted as $Q(s_t,a_t)$ with $\alpha$ as the learning rate, i.e the extent to which our $Q$-values are being updated in every iteration.

The $\boldsymbol{Q}$-table is a matrix where we have a row for every state and a column for every action – $500$ and $6$, respectively, when referring to the Taxi-v3 example, as discussed during class. It's first initialized to $0$, and then values are updated after training.

Previously, we talked about solving this task in a naïve way by simply applying brute force: using random search. This time we want to apply a more sophisticated algorithm – $Q$-learning:

I $\rightarrow$ Choose action $a_t$.

II $\rightarrow$ Go from state $s_t$ to state $s_{t+1}$ by taking action $a_{t}$.

III $\rightarrow$ For all possible $Q$-values from the state $s_{t+1}$, select the highest.

IV $\rightarrow$ Update $Q$-table values using the equation from above.

V $\rightarrow$ Set the next state as the current state.

This procedure is repeated for as many episodes as specified (VI).

    Execute the notebook until here and try to solve the following tasks:

Implement $Q$-learning as outlined above (equivalently to the one discussed during the exercise).

Apply $Q$-learning on a freshly seeded <tt>FrozenLakeEnv</tt> instance for $10^4$ episodes, with $10^3$ delay steps and $\alpha{}=0.1$.

Interpret the visualization of the resulting $Q$-table. What do you observe?

    The following code snippet is taken from the accompanying exercise notebook. You do not need to modify it for this assignment.

```python
def visualize_q_table(q_table: np.ndarray, title: str = r'') -> None:
    """
    Visualize Q-table using a heatmap plot.

    :param q_table: Q-table to visualize
    :return: None
    """
    sns.set()
    fig, ax = plt.subplots(nrows=1, ncols=1, squeeze=False, figsize=(20, 7))
    _ = sns.heatmap(data=q_table, ax=ax[0, 0])
    _ = ax[0, 0].set(xlabel=r'Action', ylabel=r'State', title=title)
    display.clear_output(wait=True)
    display.display(fig)
    plt.close(fig=fig)

def apply_q_learning(environment: u7.FrozenLakeEnv, num_episodes: int = 1000, alpha: float = 0.1,
                     animate: bool = False, delay_steps: int = 100) -> np.ndarray:
    """
    Solve specified environment by applying Q-learning.

    :param environment: the environment on which to apply Q-learning
    :param num_episodes: the total amount of episodes used to adapt the Q-table
    :param alpha: the learning rate to be applied by Q-learning
    :param animate: animate the Q-learning process
    :param delay_steps: the steps between each Q-table visualization (ignored if not animated)
    """
    q_table = np.zeros(shape=(environment.observation_space.n, environment.action_space.n))

    # <VI>: repeat Q-learning as long as the total amount of episodes is not yet reached.
    for episode in range(num_episodes):
        state = environment.reset()

        done = False
        while not done:

            # <I>: choose next action according to current Q-table.
            action = np.argmax(q_table[state])

            # <II>: go from the current state to the next by applying chosen action.
            next_state, reward, done, info = environment.step(action)

            # <III>: from all possible Q-values w.r.t. the new state, select the highest.
            next_max = np.max(q_table[next_state])

            # <IV>: update the Q-table accordingly.
            old_value = q_table[state, action]
            new_value = (1 - alpha) * old_value + alpha * (reward + next_max)
            q_table[state, action] = new_value

            # <V>: update the next step with the current one.
            state = next_state

        # Optionally visualize the current Q-table.
        if animate and any(((episode + 1) % delay_steps == 0, (episode + 1) == num_episodes)):
            visualize_q_table(q_table=q_table, title=f'Episode {episode + 1}')

    return q_table
```

```python
u7.set_environment_seed(environment=enviroment_lake, seed=42)
q_table = apply_q_learning(
    environment=enviroment_lake,
    num_episodes=10000,
    alpha=0.1,
    animate=True,
    delay_steps=1000
)
```

I obsere that it's all the same color which indicates that it's the same level.

Very likely the $Q$-table of the previous experiment looked a little bit odd. Try to add exploration to your algorithm by adapting your $Q$-learning implementation:

I $\rightarrow$ Throw a random uniform number between $0$ and $1$.

II $\rightarrow$ If the number is smaller than $0.1$, sample a random action.

III $\rightarrow$ Otherwise, choose your action as usual.

    Execute the notebook until here and try to solve the following tasks:

Modify the $Q$-learning implementation from the previous tasks as outlined above (mark the corresponding code sections).

Apply $Q$-learning on a freshly seeded <tt>FrozenLakeEnv</tt> instance for $10^4$ episodes, with $10^3$ delay steps and $\alpha{}=0.1$.

Interpret the visualization of the resulting $Q$-table. What do you observe (compare with the previous visualization)?

```python
def apply_q_learning(environment: u7.FrozenLakeEnv, num_episodes: int = 1000, alpha: float = 0.1,
                    animate: bool = False, delay_steps: int = 100, threshold: float = 0.125) -> np.ndarray:
    """
    Solve specified environment by applying Q-learning.

    :param environment: the environment on which to apply Q-learning
    :param num_episodes: the total amount of episodes used to adapt the Q-table
    :param alpha: the learning rate to be applied by Q-learning
    :param animate: animate the Q-learning process
    :param delay_steps: the steps between each Q-table visualization (ignored if not animated)
    :param threshold: threshold for randomly sampling next action
    :return: adapted Q-table
    """
    import random

    q_table = np.zeros(shape=(environment.observation_space.n, environment.action_space.n))

    # <VI>: repeat Q-learning as long as the total amount of episodes is not yet reached.
    for episode in range(num_episodes):
        state = environment.reset()

        done = False
        while not done:
            rnd_num = random.uniform(0, 1)
            action = np.argmax(q_table[state])

            if rnd_num < 0.1:
                action = environment.action_space.sample()
            else:
                action = np.argmax(q_table[state])

            # <II>: go from the current state to the next by applying chosen action.
            next_state, reward, done, info = environment.step(action)

            # <III>: from all possible Q-values w.r.t. the new state, select the highest.
            next_max = np.max(q_table[next_state])

            # <IV>: update the Q-table accordingly.
            old_value = q_table[state, action]
            new_value = (1 - alpha) * old_value + alpha * (reward + next_max)
            q_table[state, action] = new_value

            # <V>: update the next step with the current one.
            state = next_state

        # Optionally visualize the current Q-table.
        if animate and any(((episode + 1) % delay_steps == 0, (episode + 1) == num_episodes)):
            visualize_q_table(q_table=q_table, title=f'Episode {episode + 1}')

    return q_table
```

```python
u7.set_environment_seed(environment=enviroment_lake, seed=42)
q_table = apply_q_learning(
    environment=enviroment_lake,
    num_episodes=10000,
    alpha=0.1,
    animate=True,
    delay_steps=1000
)

```

    Execute the notebook until here and try to solve the following tasks:

Implement a function for applying a pre-trained $Q$-table on a <tt>FrozenLakeEnv</tt> instance (like discussed during class).

Conduct a $Q$-table guided search on a freshly seeded <tt>FrozenLakeEnv</tt> instance, with an animation delay of $0.1$.

How many steps are necessary to reach the goal at least once and how often did an involuntary dive happen?

```python
def apply_q_table(environment: u7.FrozenLakeEnv, q_table: np.ndarray, animate: bool = False,
                  delay: float = 0.01, max_steps: int = 1000) -> Tuple[int, int, Dict[str, Any]]:
    """
    Solve specified environment by applying specified Q-table.

    :param environment: the environment on which to apply Q-table guided search
    :param q_table: the Q-table used during Q-table guided search
    :param animate: animate the Q-table guided search process
    :param delay: the minimum delay in milliseconds between each rendered frame (ignored if not animated)
    :param max_steps: maximum amount of steps to perform
    :return: amount of steps performed, involuntary dives and captured frames
    """
    raise NotImplementedError(r'Exchange this error with your implementation.')
```

```python

```

```python

```

    Execute the notebook until here and try to solve the following tasks:

Conduct a $Q$-table guided search experiment as outlined previously, using $100$ repetitions and the random seed set to $42$.

Interpret the visualization (e.g. the span of the boxes) and keep the scaling of the x-axis in mind.

In comparison with the random search experiment, how does the $Q$-table guided search perform? Discuss the results.

```python

```

```python

```

Introduction to Reinforcement Learning.ipynb