# Programovanie

Letná škola FKS 2018

Maťo Gažo, Fero Dráček
(& vykradnuté materiály od Mateja Badina, Feriho Hermana, Kuba, Peťa, Jarných škôl FX a kade-tade po internete)

V tomto kurze si ukážeme základy programovania a naučíme sa programovať matematiku a fyziku.
Takéto vedomosti sú skvelé a budete vďaka nim:
* vedieť efektívnejšie robiť domáce úlohy
* kvalitnejšie riešiť seminárové a olympiádové príklady
* lepšie rozumieť svetu (IT je dnes na trhu najrýchlejšie rozvíjajúcim sa odvetvím)

Počítač je blbý a treba mu všetko povedať a vysvetliť. Komunikovať sa s ním dá na viacerých úrovniach, my budeme používať Python. Python (názov odvodený z Monty Python's Flying Circus) je všeobecný programovací jazyk, ktorým sa dajú vytvárať webové stránky ako aj robiť seriózne vedecké výpočty. To znamená, že naučiť sa ho nie je na škodu a možno vás raz bude živiť.

Rozhranie, v ktorom píšeme kód, sa volá Jupyter Notebook. Je to prostredie navrhnuté tak, aby sa dalo programovať doslova v prehliadači a aby sa kód dal kúskovať. Pre zbehnutie kúskov programu stačí stlačiť Shift+Enter.

# Dátové typy a operátory

### Čísla

 podľa očakávaní, vracia trojku

```python
3
```

    3

```python
2+3 # scitanie
```

    5

```python
6-2 # odcitanie
```

    4

```python
10*2 # nasobenie
```

    20

```python
35/5 # delenie
```

    7.0

```python
5//3 # celociselne delenie TODO je toto treba?
```

    1

```python
7%3 # modulo
```

    1

```python
2**3 # umocnovanie
```

    8

```python
4 * (2 + 3) # poradie dodrzane
```

    20

### Logické výrazy

```python
1 == 1 # logicka rovnost
```

    True

```python
2 != 3 # logicka nerovnost
```

    True

```python
1 < 10
```

    True

```python
1 > 10
```

    False

```python
2 <= 2
```

    True

# Premenné

Toto je premenná.

Po stlačení Shift+Enter program v okienku zbehne a premenná sa uloží do pamäte (RAMky, všetko sa deje na RAMke).

```python
a = 2
```

Teraz s ňou možno pracovať ako s bežným číslom.

```python
2 * a
```

    4

```python
a + a
```

    4

```python
a + a*a
```

    6

Možno ju aj umocniť.

```python
a**3
```

    8

Pridajme druhú premennú.

```python
b = 5
```

Nasledovné výpočty dopadnú podľa očakávaní.

```python
a + b
```

    7

```python
a * b
```

    10

```python
b**a
```

    25

Reálne čísla môžeme zobrazovať aj vo vedeckej forme: $2.3\times 10^{-3}$.

```python
d = 2.3e-3
```

# Funkcie

Spravme si jednoduchú funkciu, ktorá za nás sčíta dve čísla, aby sme sa s tým už nemuseli trápiť my:

```python
def scitaj(a, b):
    return a + b
```

```python
scitaj(10, 12) # vrati sucet
```

    22

Funkcia funguje na celých aj reálnych číslach.

Naša sčítacia funkcia má __štyri podstatné veci__:
1. `def`: toto slovo definuje funkciu.
2. dvojbodka na konci prvého riadku, odtiaľ začína definícia.
3. Odsadenie kódu vnútri funkcie o štyri medzery.
4. Samotný kód. V ňom sa môže diať čokoľvek, Python ho postupne prechádza.
5. `return`: kľúčová vec. Za toto slovo sa píše, čo je output funkcie.

### Úloha 1
Napíšte funkciu `priemer`, ktorá zoberie dve čísla (výšky dvoch chlapcov) a vypočíta ich priemernú výšku.

Ak máš úlohu hotovú, prihlás sa vedúcemu.

```python
# Tvoje riesenie:
def priemer(prvy, druhy):
    return ((prvy+druhy)/2)

priemer(90,20)

```

    55.0

# Poďme na fyziku

V tomto momente môžeme začať používať Python ako sofistikovanejšiu kalkulačku a počítať ňou základné fyzikálne problémy.
Jednoduchý príklad s ktorým začneme: dostanete zadaných niekoľko fyziklánych konštánt ako **premenné**.
Predstavte si, že máte za úlohou vypočítať nejakú fyzikálnu veličinu pre niekoľko zadaných hodnôt. Veľmi pohodné je napísať si funkciu, do ktorej vždy zadáme počiatočné hodnoty.

Zadané konštanty

```python
kb=1.38064852e-23           # Boltzmanova konštanta
G=6.67408e-11               # Gravitačná konštanta
```

## Úloha 2
Napíšte funkciu kotrá spočíta gravitačnú silu medzi dvomi telesami pre zadanú vzdialenosť $r$ a hmotnosti $m_1$ a $m_2$.

Pripomíname, vzorec pre výpočet gravitačnej  sily je
$F=G \frac{m_1 m_2}{r^2}$

```python
# Tvoje riesenie:

def Sila(m_1, m_2, r):
    F=G* m_1*m_2/r**2
    return (F)

Sila(10,10,100)
```

    6.67408e-13

## Úloha 3
Napíšte funkciu kotrá spočíta tlak v nádobe s objemom $V$, teplotou $T$ v ktorej je $N$ častíc.

Pripomíname, vzorec pre výpočet tlaku je
$p=\frac{N kb T}{V}$

```python
# tvoje riesenie:
def tlak(N,T,V):
    p=N*kb*T/V
    return p

tlak(6e23, 270,1)
```

    2236.6506024

## Úloha 4
Napíšte funkciu ktorá vráti výslednú rýchlosť dvoch guličiek po dokonale pružnej zrážke. Funkciu má mať ako vstupné argumenty hmotnosti $m_1$, $m_2$ a rýchlosti $u_1$ a $u_2$ guličiek pred zrážkou. Výstupom budú nové rýchlosti $v_1$ a $v_2$.

Hint: Využitím zákonu zachovania energie prídeme ku nasledujúcim výrazom pre nové rýchlosti.

$v_1=\frac{u_1 (m_1-m_2)+2 m_2u_2}{m_1+m_2}$

$v_2=\frac{u_2 (m_2-m_1)+2 m_1u_1}{m_1+m_2}$

```python
# tvoje riesenie:
def zrazka(m_1,m_2,u_1,u_2):
    return ((u_1 *(m_1-m_2)+2*m_2*u_2)/(m_1+m_2),(u_2 *(m_2-m_1)+2* m_1*u_1)/(m_1+m_2))

zrazka(1,1,10,-10)
```

    (-10.0, 10.0)

# Zoznamy

Zatiaľ sme sa zoznámili s číslami (celé, reálne), stringami a trochu aj logickými hodnotami.
Zo všetkých týchto prvkov vieme vytvárať množiny, v informatickom jazyku `zoznamy`.

Na úvod sa teda pozrieme, ako s vytvára zoznam (po anglicky `list`). Takúto vec všeobecne nazývame dátová štruktúra.

```python
li = [] # prazdny list
```

```python
ve = [4, 2, 3] # list s cislami
```

```python
ve
```

    [4, 2, 3]

```python
ve[0] # indexovat zaciname nulou!
```

    4

```python
ve[1]
```

    2

```python
ve[-1] # vybratie posledneho prvku
```

    3

```python
w = [5, 10, 15]
```

Čo sa sa stane, ak zoznamy sčítame? Spoja sa.

```python
ve + w
```

    [4, 2, 3, 5, 10, 15]

Môžeme ich násobiť?

```python
ve * ve
```

Smola, nemôžeme. Ale všimnime si, aká užitočná je chybová hláška. Jasne nám hovorí, že nemožno násobiť `list`y.

So zoznamami môžeme robiť rôzne iné užitočné veci. Napríklad ich sčítať.

```python
sum(ve)
```

    9

Alebo zistiť dĺžku:

```python
len(ve)
```

    3

Alebo ich utriediť:

Alebo na koniec pridať nový prvok:

```python
ve.append(10)
ve
```

    [4, 2, 3, 10]

Alebo odobrať:

### Interval

Zoznam možno zadefinovať cez rozsah:

```python
range(10)
type(range(10))
```

    range

```python
list(range(10))
```

    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

```python
list(range(3, 9))
```

    [3, 4, 5, 6, 7, 8]

## Úloha 5
Spočítajte:
* súčet všetkých čísel od 1 do 1000.

Vytvorte zoznam `letnaskola`, ktorý bude obsahovať vašich 5 obľúbených celých čísel.
* Pridajte na koniec zoznamu číslo 100
* Prepíšte prvé číslo v zozname tak, aby sa rovnalo poslednému v zozname.
* Vypočítajte súčet prvého čísla, posledného čísla a dĺžky zoznamu.

```python
# Tvoje riesenie:
zoznam = list(range(1,1001))
print(sum(zoznam))

letnaskola = [1,1995,12,6,42]
print(letnaskola)
letnaskola.append(100)
print(letnaskola)
letnaskola[0] = letnaskola[len(letnaskola)-1]
print(letnaskola)

print(letnaskola[0]+letnaskola[len(letnaskola)-1], len(letnaskola))
```

    500500
    [1, 1995, 12, 6, 42]
    [1, 1995, 12, 6, 42, 100]
    [100, 1995, 12, 6, 42, 100]
    200 6

# For cyklus

Indexy zoznamu môžeme postupne prechádzať. For cyklus je tzv. `iterátor`, ktorý iteruje cez zoznam.

```python
for i in [3,2,5,6]:
    print(i)
```

    2
    5
    7
    8
    10
    11
    14
    18
    20
    25

```python
for i in  [3,2,5,6]:
    print(i**2)
```

    9
    4
    25
    36

Ako úspešne vytvoriť For cyklus? Podobne, ako pri funkciách:

* `for`: toto slovo je na začiatku.
* `i`: iterovana velicina
* `in`: pred zoznamom, cez ktorý prechádzame (iterujeme).
* dvojbodka na konci prvého riadku.
* kod, ktory sa cykli sa odsadzuje o štyri medzery.

Za pomoci for cyklu môžeme takisto sčítať čísla. Napr. čísla od 0 do 100:

```python
suma = 0

for i in range(101):   # uvedomme si, preco tam je 101 a nie 100
    suma = suma + i      # skratene sum += i

print(suma)
```

    5050

## Hľadanie hodnoty zlatého rezu $\varphi$

Jednoduché cvičenie na oboznámenie sa s tzv. selfkonzistentným problémom a for cyklom.
Zlatý rez je možné nájsť ako riešienie rovnice
$x=1+1/x$
Jej riešenie vieme hladať postupným iterovaním

```python
x = 1;

for i in range (0,20):
    x = 1+1/x
    print (x)
```

    2.0
    1.5
    1.6666666666666665
    1.6
    1.625
    1.6153846153846154
    1.619047619047619
    1.6176470588235294
    1.6181818181818182
    1.6179775280898876
    1.6180555555555556
    1.6180257510729614
    1.6180371352785146
    1.6180327868852458
    1.618034447821682
    1.618033813400125
    1.6180340557275543
    1.6180339631667064
    1.6180339985218035
    1.618033985017358

## Úloha 6
Spočítajte súčet druhých mocnín všetkých nepárnych čísel od 1 do 100 s využitím for cyklu.

```python
# Tvoje riesenie:
suma = 0
for i in range(50):
    suma = suma + (2*i+1)**2
print(suma)
```

    166650

## Úloha 7
### Dvojhlavý tank (FKS 30.2.2.A2)
Nevieme odkiaľ, no máme bombastický tank, ktorý má dve hlavne namierené opačným smerom – samozrejme tak, že nemieria proti sebe ;-). V tanku je
$N = 42$ nábojov s hmotnosťou $m= 20$ kg. Tank s nábojmi váži dokopy
$M= 43$ t. Potom tank začne strieľať striedavo z hlavní náboje rýchlosťou v= 1 000 m s frekvenciou strieľania $f= 0.2$ Hz. Keďže tank je nezabrzdený a dobre naolejovaný, začne sa pohybovať.
Ako ďaleko od pôvodného miesta vystrelí posledný náboj? Akej veľkej chyby by sme sa dopustili,
ak by sme zanedbali zmenu celkovej hmotnosti tanku počas strieľania?

Hint: http://old.fks.sk/archiv/2014_15/30vzorakyLeto2.pdf , strana 10

```python
x=0
m=20
Mtank=[43000]
vtank=[0]
v=-1000
f=0.2
for i in range(43):
    x=x+vtank[-1]*1/f
    vtank.append(vtank[-1]-m*v/(Mtank[-1]-m))
    Mtank.append(Mtank[-1]-m)
    v=-v
print(x)

```

    48.83697212654822

# Podmienky

Pochopíme ich na príklade. Zmeňte `a` a zistite, čo to spraví.

```python
a = 5

if a == 3:
    print("cislo a je rovne trom.")
elif a == 5:
    print("cislo a je rovne piatim")
else:
    print("cislo a nie je rovne trom ani piatim.")
```

    cislo a je rovne piatim

Za pomoci podmienky teraz môžeme z for cyklu vypísať napr. len párne čísla. Párne číslo identifikujeme ako také, ktoré po delení dvomi dáva zvyšok nula.

Pre zvyšok po delení sa používa percento:

```python
for i in range(10):
    if i % 2 == 0:
        print(i)
```

    0
    2
    4
    6
    8

Cyklus mozeme zastavit, ak sa porusi nejaka podmienka

```python
for i in range(20):
    print(i)
    if i>10:
        print('Koniec.')
        break
```

    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    Koniec.

## Hľadanie hodnoty Ludolfovho čísla $\pi$
Pomocou Monte Carlo metódy integrovania sa naučíme ako napríklad vypočítať $\pi$.
Nasledujúce príkazy vygenerujú zoznam náhodných čísel od nula po jeden

```python
import random as rnd
import numpy as np

NOP = 50000

CoordXList = [];
CoordYList = [];

for j in range (NOP):
    CoordXList.append(rnd.random())
    CoordYList.append(rnd.random())
```

Tieto dva zoznamy použijeme ako $x$-ové a $y$-ové súradnice bodov v rovnine. Kedže náhodné rozdelenie bodov je rovnomerné, tak pomer bodov, ktoré sa nachádzajú vnútri štvrťkružnice s polomerom jedna ku všetkým bodom musí byť rovnaký ako pomer plochy štvrťkruhu a štvroca.
Teda $$\frac{\frac{1}{4}\pi 1^2}{1^2}\stackrel{!}{=}\frac{N_{in}}{NOP}.$$
Nasledujúce dve bunky vygenerujú obrázok rozloženia bodov a stvrťkružnicu

```python
CircPhi = np.arange(0,np.pi/2,0.01)
```

```python
import matplotlib.pyplot as plt
f1=plt.figure(figsize=(7,7))

plt.plot(
    CoordXList,
    CoordYList,
    color = "red",
    linestyle= "none",
    marker = ","
)
plt.plot(np.cos(CircPhi),np.sin(CircPhi))
#plt.axis([0, 1, 0, 1])
#plt.axes().set_aspect('equal', 'datalim')
plt.show(f1)
```

## Úloha 8
Teraz je vašou úlohou spočítať $\pi$. Hint: bod je vnútri štvrťkružnice pokiaľ platí $x^2+y^2<1.$

```python
#vase riesenie
NumIn = 0

for j in range (NOP):

    #if (CoordXList[j] - 0.5)*(CoordXList[j] - 0.5) + (CoordYList[j] - 0.5)*(CoordYList[j] - 0.5) < 0.25:
    if CoordXList[j]*CoordXList[j] + CoordYList[j]*CoordYList[j] <= 1:

        NumIn = NumIn + 1;
```

```python
NumIn/NOP*4
```

    3.14768

# Numerické sčítavanie

Vo fyzike je často užitočné rozdeliť si problém na malé časti.

## Úloha 9
Teraz je vašou úlohou vymyslieť ako spočítať gravitačné pole od jednorozmernej usečky o výške h nad stredom úsečky.
 Úsečka má hmotnosť  $M$ a dĺžku $L$.

Úsečku si rozdelíte na $N$ malých dielov. Hmotnosť jedného takého dieliku je potom
$$dm=\frac{M}{N}$$

Vzdialenosť takéhoto bodu od stedu úsečky $x$.
Potom gravitačná pole od toho malého kúsku v žiadanom bode je:
$$\vec{\Delta g}=-G \frac{\Delta m}{r^3}\vec{r}.$$
Rozdrobené na $y$-ovú a $x$-ovú zložku:
$$\Delta g_y=-G \frac{\Delta m}{(x^2+h^2)}\cos(\phi)=-G \frac{\Delta m}{(x^2+h^2)}\frac{h}{\sqrt{x^2=h^2}},$$
respektíve
$$\Delta g_x=-G \frac{\Delta m}{(x^2+h^2)}\sin(\phi)=-G \frac{\Delta m}{(x^2+h^2)}\frac{x}{\sqrt{x^2+h^2}},$$
Vašou úlohou je rozdeliť takúto úsečku a sčítať príspevky od všetkých malých kúskov. Premyslite si že kedže sme v strede úsečky, $x$-ové príspevky sa navzájom vynulujú.
Ak vám to príde príliš jednoduché, tak možte naprogramovať program ktorých spočíta gravitačné pole nad lubovoľným bodom

```python
N=1000
M=1000
L=2
h=1

#vase riesenie
g=0
for i in range(-int(N/2),int(N/2)):
    g=g+G*M/N*(i/N)/((i/N)**2+h**2)**(3/2)
g

```

    -2.3877914507634306e-11

# Obiehanie Zeme okolo Slnka

Fyziku (dúfam!) všetci poznáme.

* gravitačná sila:
$$ \mathbf F(\mathbf r) = -\frac{G m M}{r^3} \mathbf r $$

### Eulerov algoritmus (zlý)
$$\begin{align}
a(t) &= F(t)/m \\
v(t+dt) &= v(t) + a(t) dt \\
x(t+dt) &= x(t) + v(t) dt \\
\end{align}$$

### Verletov algoritmus (dobrý)
$$ x(t+dt) = 2 x(t) - x(t-dt) + a(t) dt^2 $$

```python
from numpy.linalg import norm

G = 6.67e-11
Ms = 2e30
Mz = 6e24
dt = 86400.0
N = int(365*86400.0/dt)
#print(N)

R0 = 1.5e11
r_list = np.zeros((N, 2))
r_list[0] = [R0, 0.0]     # mozno miesat listy s ndarray

v0 = 29.7e3
v_list = np.zeros((N, 2))
v_list[0] = [0.0, v0]

# sila medzi planetami
def force(A, r):
    return -A / norm(r)**3 * r

# Verletova integracia
def verlet_step(r_n, r_nm1, a, dt):  # r_nm1 -- r n minus 1
    return 2*r_n - r_nm1 + a*dt**2

# prvy krok je specialny
a = force(G*Ms, r_list[0])
r_list[1] = r_list[0] + v_list[0]*dt + a*dt**2/2

# riesenie pohybovych rovnic
for i in range(2, N):
    a = force(G*Ms, r_list[i-1])
    r_list[i] = verlet_step(r_list[i-1], r_list[i-2], a, dt)

plt.plot(r_list[:, 0], r_list[:, 1])
plt.xlim([-2e11, 2e11])
plt.ylim([-2e11, 2e11])
plt.xlabel("$x$", fontsize=20)
plt.ylabel("$y$", fontsize=20)
plt.gca().set_aspect('equal', adjustable='box')
#plt.axis("equal")
plt.show()
```

## Pridajme Mesiac

```python
Mm = 7.3e22
R0m = R0 + 384e6
v0m = v0 + 1e3
rm_list = np.zeros((N, 2))
rm_list[0] = [R0m, 0.0]
vm_list = np.zeros((N, 2))
vm_list[0] = [0.0, v0m]

# prvy Verletov krok
am = force(G*Ms, rm_list[0]) + force(G*Mz, rm_list[0] - r_list[0])
rm_list[1] = rm_list[0] + vm_list[0]*dt + am*dt**2/2

# riesenie pohybovych rovnic
for i in range(2, N):
    a = force(G*Ms, r_list[i-1]) - force(G*Mm, rm_list[i-1]-r_list[i-1])
    am = force(G*Ms, rm_list[i-1]) + force(G*Mz, rm_list[i-1]-r_list[i-1])
    r_list[i] = verlet_step(r_list[i-1], r_list[i-2], a, dt)
    rm_list[i] = verlet_step(rm_list[i-1], rm_list[i-2], am, dt)

plt.plot(r_list[:, 0], r_list[:, 1])
plt.plot(rm_list[:, 0], rm_list[:, 1])
plt.xlabel("$x$", fontsize=20)
plt.ylabel("$y$", fontsize=20)
plt.gca().set_aspect('equal', adjustable='box')
plt.xlim([-2e11, 2e11])
plt.ylim([-2e11, 2e11])
plt.show()         # mesiac moc nevidno, ale vieme, ze tam je
```

## Úloha pre Vás: Treba pridať Mars :)
Pridajte Mars!

## Matematické kyvadlo s odporom
Nasimulujte matematické kyvadlo s odporom $\gamma$,
$$ \ddot \theta = -\frac g l \sin\theta -\gamma \theta^2,$$
za pomoci metódy `odeint`.

Alebo pád telesa v odporovom prostredí:
$$ a = -g - kv^2.$$

```python
from scipy.integrate import odeint

def F(y, t, g, k):
    return [y[1], g -k*y[1]**2]

N = 101
k = 1.0
g = 10.0
t = np.linspace(0, 1, N)
y0 = [0.0, 0.0]
y = odeint(F, y0, t, args=(g, k))

plt.plot(t, y[:, 1])
plt.xlabel("$t$", fontsize=20)
plt.ylabel("$v(t)$", fontsize=20)
plt.show()
```

## Harmonický oscilátor pomocou metódy Leapfrog (modifikácia Verletovho algoritmu)

```python
N = 10000
t = linspace(0,100,N)
dt = t[1] - t[0]

# Funkcie
def integrate(F,x0,v0,gamma):
    x = zeros(N)
    v = zeros(N)
    E = zeros(N)

    # Počiatočné podmienky
    x[0] = x0
    v[0] = v0

    # Integrovanie rovníc pomocou metódy Leapfrog (wiki)
    fac1 = 1.0 - 0.5*gamma*dt
    fac2 = 1.0/(1.0 + 0.5*gamma*dt)

    for i in range(N-1):
         v[i + 1] = fac1*fac2*v[i] - fac2*dt*x[i] + fac2*dt*F[i]
         x[i + 1] = x[i] + dt*v[i + 1]
         E[i] += 0.5*(x[i]**2 + ((v[i] + v[i+1])/2.0)**2)

    E[-1] = 0.5*(x[-1]**2 + v[-1]**2)

    # Vrátime riešenie
    return x,v,E
```

```python
# Pozrime sa na tri rôzne počiatočné podmienky
F = zeros(N)
x1,v1,E1 = integrate(F,0.0,1.0,0.0) # x0 = 0.0, v0 = 1.0, gamma = 0.0
x2,v2,E2 = integrate(F,0.0,1.0,0.05) # x0 = 0.0, v0 = 1.0, gamma = 0.01
x3,v3,E3 = integrate(F,0.0,1.0,0.4) # x0 = 0.0, v0 = 1.0, gamma = 0.5

# Nakreslime si grafy

plt.rcParams["axes.grid"] = True
plt.rcParams['font.size'] = 14
plt.rcParams['axes.labelsize'] = 18
plt.figure()
plt.subplot(211)
plt.plot(t,x1)
plt.plot(t,x2)
plt.plot(t,x3)
plt.ylabel("x(t)")

plt.subplot(212)
plt.plot(t,E1,label=r"$\gamma = 0.0$")
plt.plot(t,E2,label=r"$\gamma = 0.01$")
plt.plot(t,E3,label=r"$\gamma = 0.5$")
plt.ylim(0,0.55)
plt.ylabel("E(t)")

plt.xlabel("Čas")
plt.legend(loc="center right")

plt.tight_layout()
```

A čo ak bude oscilátor aj tlmenný?

```python
def force(f0,t,w,T):
          return f0*cos(w*t)*exp(-t**2/T**2)

F1 = zeros(N)
F2 = zeros(N)
F3 = zeros(N)
for i in range(N-1):
    F1[i] = force(1.0,t[i] - 20.0,1.0,10.0)
    F2[i] = force(1.0,t[i] - 20.0,0.9,10.0)
    F3[i] = force(1.0,t[i] - 20.0,0.8,10.0)
```

```python
x1,v1,E1 = integrate(F1,0.0,0.0,0.0)
x2,v2,E2 = integrate(F1,0.0,0.0,0.01)
x3,v3,E3 = integrate(F1,0.0,0.0,0.1)

plt.figure()
plt.subplot(211)
plt.plot(t,x1)
plt.plot(t,x2)
plt.plot(t,x3)
plt.ylabel("x(t)")

plt.subplot(212)
plt.plot(t,E1,label=r"$\gamma = 0$")
plt.plot(t,E2,label=r"$\gamma = 0.01$")
plt.plot(t,E3,label=r"$\gamma = 0.1$")
pt.ylabel("E(t)")

plt.xlabel("Time")
plt.rcParams['legend.fontsize'] = 14.0
plt.legend(loc="upper left")

plt.show()
```

```python

```


# The $\chi^2$ Distribution

## $\chi^2$ Test Statistic

If we make $n$ ranom samples (observations) from Gaussian (Normal) distributions with known means, $\mu_i$, and known variances, $\sigma_i^2$, it is seen that the total squared deviation,

$$
\chi^2 = \sum_{i=1}^{n} \left(\frac{x_i - \mu_i}{\sigma_i}\right)^2\,,
$$

follows a $\chi^2$ distribution with $n$ degrees of freedom.

## Probability Distribution Function

The $\chi^2$ probability distribution function for $k$ degrees of freedom (the number of parameters that are allowed to vary) is given by

$$
f\left(\chi^2\,;k\right) = \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, \chi^{k-2}\,e^{-\chi^2/2}\,,
$$

where if there are no constrained variables the number of degrees of freedom, $k$, is equal to the number of observations, $k=n$. The p.d.f. is often abbreviated in notation from $f\left(\chi^2\,;k\right)$ to $\chi^2_k$.

A reminder that for integer values of $k$, the Gamma function is $\Gamma\left(k\right) = \left(k-1\right)!$, and that $\Gamma\left(x+1\right) = x\Gamma\left(x\right)$, and $\Gamma\left(1/2\right) = \sqrt{\pi}$.

## Mean

Letting $\chi^2=z$, and noting that the form of the Gamma function is

$$
\Gamma\left(z\right) = \int\limits_{0}^{\infty} x^{z-1}\,e^{-x}\,dx,
$$

it is seen that the mean of the $\chi^2$ distribution $f\left(\chi^2 ; k\right)$ is

$$
\begin{align}
\mu &= \textrm{E}\left[z\right] = \displaystyle\int\limits_{0}^{\infty} z\, \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z\,/2}\,dz \\
    &= \displaystyle \frac{\displaystyle 1}{\displaystyle \Gamma\left(k\,/2\right)} \int\limits_{0}^{\infty} \left(\frac{z}{2}\right)^{k/2}\,e^{-z\,/2}\,dz = \displaystyle \frac{\displaystyle 1}{\displaystyle \Gamma\left(k\,/2\right)} \int\limits_{0}^{\infty} x^{k/2}\,e^{-x}\,2 \,dx \\
    &= \displaystyle \frac{\displaystyle 2 \,\Gamma\left(k\,/2 + 1\right)}{\displaystyle \Gamma\left(k\,/2\right)} \\
    &= \displaystyle 2 \frac{k}{2} \frac{\displaystyle \Gamma\left(k\,/2\right)}{\displaystyle \Gamma\left(k\,/2\right)} \\
    &= k.
\end{align}
$$

## Variance

Likewise, the variance is

$$
\begin{align}
\textrm{Var}\left[z\right] &= \textrm{E}\left[\left(z-\textrm{E}\left[z\right]\right)^2\right] = \displaystyle\int\limits_{0}^{\infty} \left(z - k\right)^2\, \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z\,/2}\,dz \\
    &= \displaystyle\int\limits_{0}^{\infty} z^2\, f\left(z \,; k\right)\,dz - 2k\int\limits_{0}^{\infty} z\,\,f\left(z \,; k\right)\,dz + k^2\int\limits_{0}^{\infty} f\left(z \,; k\right)\,dz \\
    &= \displaystyle\int\limits_{0}^{\infty} z^2 \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z\,/2}\,dz - 2k^2 + k^2\\
    &= \displaystyle\int\limits_{0}^{\infty} \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2+1}\,e^{-z\,/2}\,dz - k^2\\
    &= \frac{\displaystyle 2}{\displaystyle \Gamma\left(k\,/2\right)} \displaystyle\int\limits_{0}^{\infty} \left(\frac{z}{2}\right)^{k/2+1}\,e^{-z\,/2}\,dz - k^2 = \frac{\displaystyle 2}{\displaystyle \Gamma\left(k\,/2\right)} \displaystyle\int\limits_{0}^{\infty} x^{k/2+1}\,e^{-x}\,2\,dx - k^2 \\
    &= \displaystyle \frac{\displaystyle 4 \,\Gamma\left(k\,/2 + 2\right)}{\displaystyle \Gamma\left(k\,/2\right)} - k^2 \\
    &= \displaystyle 4 \left(\frac{k}{2} + 1\right) \frac{\displaystyle \Gamma\left(k\,/2 + 1\right)}{\displaystyle \Gamma\left(k\,/2\right)} - k^2 \\
    &= \displaystyle 4 \left(\frac{k}{2} + 1\right) \frac{k}{2} - k^2 \\
    &= k^2 + 2k - k^2 \\
    &= 2k,
\end{align}
$$

such that the standard deviation is

$$
\sigma = \sqrt{2k}\,.
$$

Given this information we now plot the $\chi^2$ p.d.f. with various numbers of degrees of freedom to visualize how the distribution's behaviour

```python
import numpy as np
import scipy.stats as stats

import matplotlib.pyplot as plt
```

```python
# Plot the chi^2 distribution
x = np.linspace(0., 10., num=1000)

[plt.plot(x, stats.chi2.pdf(x, df=ndf), label=r'$k = ${}'.format(ndf))
 for ndf in range(1, 7)]

plt.ylim(-0.01, 0.5)

plt.xlabel(r'$x=\chi^2$')
plt.ylabel(r'$f\left(x;k\right)$')
plt.title(r'$\chi^2$ distribution for various degrees of freedom')

plt.legend(loc='best')

plt.show();
```

## Cumulative Distribution Function

The cumulative distribution function (CDF) for the $\chi^2$ distribution is (letting $z=\chi^2$)

$$
\begin{split}
F_{\chi^2}\left(x\,; k\right) &= \int\limits_{0}^{x} f_{\chi^2}\left(z\,; k\right) \,dz \\
    &= \int\limits_{0}^{x} \frac{\displaystyle 1}{\displaystyle 2^{k/2} \,\Gamma\left(k\,/2\right)}\, z^{k/2-1}\,e^{-z/2} \,dz \\
    &= \int\limits_{0}^{x} \frac{\displaystyle 1}{\displaystyle 2 \,\Gamma\left(k\,/2\right)}\, \left(\frac{z}{2}\right)^{k/2-1}\,e^{-z/2} \,dz = \frac{1}{\displaystyle 2 \,\Gamma\left(k\,/2\right)}\int\limits_{0}^{x/2} t^{k/2-1}\,e^{-t} \,2\,dt \\
    &= \frac{1}{\displaystyle \Gamma\left(k\,/2\right)}\int\limits_{0}^{x/2} t^{k/2-1}\,e^{-t} \,dt
\end{split}
$$

Noting the form of the [lower incomplete gamma function](https://en.wikipedia.org/wiki/Incomplete_gamma_function) is

$$
\gamma\left(s,x\right) = \int\limits_{0}^{x} t^{s-1}\,e^{-t} \,dt\,,
$$

and the form of the [regularized Gamma function](https://en.wikipedia.org/wiki/Incomplete_gamma_function#Regularized_Gamma_functions_and_Poisson_random_variables) is

$$
P\left(s,x\right) = \frac{\gamma\left(s,x\right)}{\Gamma\left(s\right)}\,,
$$

it is seen that

$$
\begin{split}
F_{\chi^2}\left(x\,; k\right) &= \frac{1}{\displaystyle \Gamma\left(k\,/2\right)}\int\limits_{0}^{x/2} t^{k/2-1}\,e^{-t} \,dt \\
    &= \frac{\displaystyle \gamma\left(\frac{k}{2},\frac{x}{2}\right)}{\displaystyle \Gamma\left(\frac{k}{2}\right)} \\
    &= P\left(\frac{k}{2},\frac{x}{2}\right)\,.
\end{split}
$$

Thus, it is seen that the compliment to the CDF (the complementary cumulative distribution function (CCDF)),

$$
\bar{F}_{\chi^2}\left(x\,; k\right) = 1-F_{\chi^2}\left(x\,; k\right),
$$

represents a one-sided (one-tailed) $p$-value for observing a $\chi^2$ given a model — that is, the probability to observe a $\chi^2$ value greater than or equal to that which was observed.

```python
def chi2_ccdf(x, df):
    """The complementary cumulative distribution function

    Args:
        x: the value of chi^2
        df: the number of degrees of freedom

    Returns:
        1 - the cumulative distribution function
    """
    return 1. - stats.chi2.cdf(x=x, df=df)
```

```python
x = np.linspace(0., 10., num=1000)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(14, 4.5))

for ndf in range(1,7):
    axes[0].plot(x, stats.chi2.cdf(x, df=ndf),
                   label=r'$k = ${}'.format(ndf))
    axes[1].plot(x, chi2_ccdf(x, df=ndf),
                   label=r'$k = ${}'.format(ndf))

axes[0].set_xlabel(r'$x=\chi^2$')
axes[0].set_ylabel(r'$F\left(x;k\right)$')
axes[0].set_title(r'$\chi^2$ CDF for various degrees of freedom')

axes[0].legend(loc='best')

axes[1].set_xlabel(r'$x=\chi^2$')
axes[1].set_ylabel(r'$\bar{F}\left(x;k\right) = p$-value')
axes[1].set_title(r'$\chi^2$ CCDF ($p$-value) for various degrees of freedom')

axes[1].legend(loc='best')

plt.show();
```

## Binned $\chi^2$ per Degree of Freedom

TODO

## References

- \[1\] G. Cowan, _Statistical Data Analysis_, Oxford University Press, 1998
- \[2\] G. Cowan, "Goodness of fit and Wilk's theorem", Notes, 2013


# Heat Equation

Solve the differential equation

\begin{align}
    \partial_t u(t,x) - \partial_x^2 u(t,x) &= 0, && t \in (0,T), \, x \in (x_\mathrm{min}, x_\mathrm{max}),
    \\
    u(0,x) &= u_0(x), && x \in [x_\mathrm{min}, x_\mathrm{max}]
\end{align}

with appropriate boundary conditions.

## Constant Dirichlet Boundary Conditions

Solve the heat equation with constant Dirichlet boundary conditions

\begin{align}
    u(t,x_\mathrm{min}) &= u_0(x_\mathrm{min}), && t \in (0,T),
    \\
    u(t,x_\mathrm{max}) &= u_0(x_\mathrm{max}), && t \in (0,T).
\end{align}

### Using `SummationByPartsOperators.jl`

```julia
using SummationByPartsOperators, OrdinaryDiffEq
using Plots, LaTeXStrings

xmin, xmax = -π, π
N = 512
acc_order = 4
tspan = (0., 10.)
# source of coefficients
source = MattssonSvärdShoeybi2008()
ode_alg = Tsit5()

u₀(x) = -(x - 0.5)^2 + 1/12

D = derivative_operator(source, 2, acc_order, xmin, xmax, N)
x = D.grid
u0 = u₀.(x)

function rhs!(du, u, p, t)
    mul!(du, D, u)
    @inbounds du[1] -= (u[1] - u₀(xmin)) / D.coefficients.left_weights[1]
    @inbounds du[end] -= (u[end] - u₀(xmax)) / D.coefficients.right_weights[1]
end

ode = ODEProblem(rhs!, u0, tspan)
sol = solve(ode, ode_alg, save_everystep=false, saveat=0:10)
@time sol = solve(ode, ode_alg, save_everystep=false, saveat=0:10)

# try to plot the solution at different time points using
plot(x, [sol(i) for i in 0:1:10])
```

### Using `DiffEqOperators.jl`

```julia
using DiffEqOperators, OrdinaryDiffEq
using Plots, LaTeXStrings

xmin, xmax = -π, π
N = 512
acc_order = 4
tspan = (0., 10.)
ode_alg = Tsit5()

u₀(x) = -(x - 0.5)^2 + 1/12

x = range(xmin, stop=xmax, length=N)
u0 = u₀.(x)
L = DiffEqOperators.DerivativeOperator{Float64}(2, acc_order, 2π/511, 512, :Dirichlet, :Dirichlet; BC=(u0[1],u0[end]))

ode = ODEProblem(L, u0, tspan)
sol = solve(ode, ode_alg, save_everystep=false, saveat=0:10)
@time sol = solve(ode, ode_alg, save_everystep=false, saveat=0:10)

# try to plot the solution at different time points using
plot(x, [sol(i) for i in 0:1:10])
```

```julia

```


# Session 10 - Unsupervised Learning

## Contents

- [Principal Components Analysis](#Principal-Components-Analysis)
- [Clustering Methods](#Clustering-Methods)

```python
# Import
import pandas as pd
import numpy as np
import seaborn as sns
import time

from sklearn.preprocessing import scale
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from scipy.cluster import hierarchy
from scipy.cluster.hierarchy import linkage, dendrogram, cut_tree
```

```python
# Import matplotlib for graphs
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from IPython.display import clear_output

# Set global parameters
%matplotlib inline
plt.style.use('seaborn-white')
plt.rcParams['lines.linewidth'] = 3
plt.rcParams['figure.figsize'] = (10,6)
plt.rcParams['figure.titlesize'] = 20
plt.rcParams['axes.titlesize'] = 18
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['legend.fontsize'] = 14
```

The difference between *supervised learning* and *unsupervised learning* is that in the first case we have a variable $y$ which we want to predict, given a set of variables $\{ X_1, . . . , X_p \}$.

In *unsupervised learning* are not interested in prediction, because we do not have an associated response variable $y$. Rather, the goal is to discover interesting things about the measurements on $\{ X_1, . . . , X_p \}$. Question that we can answer are:

- Is there an informative way to visualize the data?
- Can we discover subgroups among the variables or among the observations?

We are going to answe these questions with two main methods, respectively:

- Principal Components Analysis
- Clustering

## Principal Component Analysis

Suppose that we wish to visualize $n$ observations with measurements on a set of $p$ features, $\{X_1, . . . , X_p\}$, as part of an exploratory data analysis.

We could do this by examining two-dimensional scatterplots of the data, each of which contains the n observations’ measurements on two of the features. However, there are $p(p−1)/2$ such scatterplots; for example,
with $p = 10$ there are $45$ plots!

PCA provides a tool to do just this. It finds a low-dimensional represen- tation of a data set that contains as much as possible of the variation.

The **first principal component** of a set of features $\{X_1, . . . , X_p\}$ is the normalized linear combination of the features

$$
Z_1 = \phi_{11} X_1 + \phi_{21} X_2 + ... + \phi_{p1} X_p
$$

that has the largest variance. By normalized, we mean that $\sum_{i=1}^p \phi^2_{i1} = 1$.

In other words, the first principal component loading vector solves the optimization problem

$$
\underset{\phi_{11}, \ldots, \phi_{p 1}}{\operatorname{max}} \ \left\{\frac{1}{n} \sum_{i=1}^{n}\left(\sum_{j=1}^{p} \phi_{j 1} x_{i j}\right)^{2}\right\} \quad \text { subject to } \quad \sum_{j=1}^{p} \phi_{j 1}^{2}=1
$$

The objective that we are maximizing is just the sample variance of the $n$ values of $z_{i1}$.

After the first principal component $Z_1$ of the features has been determined, we can find the second principal component $Z_2$. The **second principal component** is the linear combination of $\{X_1, . . . , X_p\}$ that has maximal variance out of all linear combinations that are *uncorrelated* with $Z_1$.

We illustrate the use of PCA on the `USArrests` data set. For each of the 50 states in the United States, the data set contains the number of arrests per $100,000$ residents for each of three crimes: `Assault`, `Murder`, and `Rape.` We also record the percent of the population in each state living in urban areas, `UrbanPop`.

```python
# Load crime data
df = pd.read_csv('data/USArrests.csv', index_col=0)
df.head()
```

      Murder
      Assault
      UrbanPop
      Rape

      State

      Alabama
      13.2
      236
      58
      21.2

      Alaska
      10.0
      263
      48
      44.5

      Arizona
      8.1
      294
      80
      31.0

      Arkansas
      8.8
      190
      50
      19.5

      California
      9.0
      276
      91
      40.6

```python
# Scale data
X_scaled = pd.DataFrame(scale(df), index=df.index, columns=df.columns).values
```

```python
# Fit PCA with 2 components
pca2 = PCA(n_components=2).fit(X_scaled)
```

```python
# Get weights
weights = pca2.components_.T
df_weights = pd.DataFrame(weights, index=df.columns, columns=['PC1', 'PC2'])
df_weights
```

      PC1
      PC2

      Murder
      0.535899
      0.418181

      Assault
      0.583184
      0.187986

      UrbanPop
      0.278191
      -0.872806

      Rape
      0.543432
      -0.167319

```python
# Transform X to get the principal components
X_dim2 = pca2.transform(X_scaled)
df_dim2 = pd.DataFrame(X_dim2, columns=['PC1', 'PC2'], index=df.index)
df_dim2.head()
```

      PC1
      PC2

      State

      Alabama
      0.985566
      1.133392

      Alaska
      1.950138
      1.073213

      Arizona
      1.763164
      -0.745957

      Arkansas
      -0.141420
      1.119797

      California
      2.523980
      -1.542934

```python
fig, ax1 = plt.subplots(figsize=(10,10))
ax1.set_title('Figure 10.1');

# Plot Principal Components 1 and 2
for i in df_dim2.index:
    ax1.annotate(i, (df_dim2.PC1.loc[i], -df_dim2.PC2.loc[i]), ha='center', fontsize=14)

# Plot reference lines
m = np.max(np.abs(df_dim2.values))*1.2
ax1.hlines(0,-m,m, linestyles='dotted', colors='grey')
ax1.vlines(0,-m,m, linestyles='dotted', colors='grey')
ax1.set_xlabel('First Principal Component')
ax1.set_ylabel('Second Principal Component')
ax1.set_xlim(-m,m); ax1.set_ylim(-m,m)

# Plot Principal Component loading vectors, using a second y-axis.
ax1b = ax1.twinx().twiny()
ax1b.set_ylim(-1,1); ax1b.set_xlim(-1,1)
for i in df_weights[['PC1', 'PC2']].index:
    ax1b.annotate(i, (df_weights.PC1.loc[i]*1.05, -df_weights.PC2.loc[i]*1.05), color='orange', fontsize=16)
    ax1b.arrow(0,0,df_weights.PC1[i], -df_weights.PC2[i], color='orange', lw=2)
```

### PCA and spectral analysis

In case you haven't noticed, calculating principal components, is equivalent to calculating the eigenvectors of the design matrix $X'X$, i.e. the variance-covariance matrix of $X$. Indeed what we performed above is a decomposition of the variance of $X$ into orthogonal components.

The constrained maximization problem above can be re-written in matrix notation as

$$
\max \ \phi' X'X \phi \quad \text{ s. t. } \quad \phi'\phi = 1
$$

Which has the following dual representation

$$
\mathcal L (\phi, \lambda) = \phi' X'X \phi - \lambda (\phi'\phi - 1)
$$

Which gives first order conditions

$$
\begin{align}
& \frac{\partial \mathcal L}{\partial \lambda} = \phi'\phi - 1 \\
& \frac{\partial \mathcal L}{\partial \phi} = 2 X'X \phi - 2 \lambda \phi
\end{align}
$$

Setting the derivatives to zero at the optimum, we get

$$
\begin{align}
& \phi'\phi = 1 \\
& X'X \phi = \lambda \phi
\end{align}
$$

Thus, $\phi$ is an **eigenvector** of the covariance matrix $X'X$, and the maximizing vector will be the one associated with the largest **eigenvalue** $\lambda$.

We can now double-check it using `numpy` linear algebra package.

```python
eigenval, eigenvec = np.linalg.eig(X_scaled.T @ X_scaled)
data = np.concatenate((eigenvec,eigenval.reshape(1,-1)))
idx = list(df.columns) + ['Eigenvalue']
df_eigen = pd.DataFrame(data, index=idx, columns=['PC1', 'PC2','PC3','PC4'])

df_eigen
```

      PC1
      PC2
      PC3
      PC4

      Murder
      0.535899
      0.418181
      0.649228
      -0.341233

      Assault
      0.583184
      0.187986
      -0.743407
      -0.268148

      UrbanPop
      0.278191
      -0.872806
      0.133878
      -0.378016

      Rape
      0.543432
      -0.167319
      0.089024
      0.817778

      Eigenvalue
      124.012079
      49.488258
      8.671504
      17.828159

The spectral decomposition of the variance of $X$ generates a set of orthogonal vectors (eigenvectors) with different magnitudes (eigenvalues). The eigenvalues tell us the amount of variance of the data in that direction.

If we combine the eigenvectors together, we form a projection matrix $P$ that we can use to transform the original variables: $\tilde X = P X$

```python
X_transformed = X_scaled @ eigenvec
df_transformed = pd.DataFrame(X_transformed, index=df.index, columns=['PC1', 'PC2','PC3','PC4'])

df_transformed.head()
```

      PC1
      PC2
      PC3
      PC4

      State

      Alabama
      0.985566
      1.133392
      0.156267
      -0.444269

      Alaska
      1.950138
      1.073213
      -0.438583
      2.040003

      Arizona
      1.763164
      -0.745957
      -0.834653
      0.054781

      Arkansas
      -0.141420
      1.119797
      -0.182811
      0.114574

      California
      2.523980
      -1.542934
      -0.341996
      0.598557

This is exactly the dataset that we obtained before.

### More on PCA

#### Scaling the Variables

The results obtained when we perform PCA will also depend on whether the variables have been individually scaled. In fact, the variance of a variable depends on its magnitude.

```python
# Variables variance
df.var(axis=0)
```

    Murder        18.970465
    Assault     6945.165714
    UrbanPop     209.518776
    Rape          87.729159
    dtype: float64

Consequently, if we perform PCA on the unscaled variables, then the first principal component loading vector will have a very large loading for `Assault`, since that variable has by far the highest variance.

```python
# Fit PCA with unscaled varaibles
X = df.values
pca2_u = PCA(n_components=2).fit(X)
```

```python
# Get weights
weights_u = pca2_u.components_.T
df_weights_u = pd.DataFrame(weights_u, index=df.columns, columns=['PC1', 'PC2'])
df_weights_u
```

      PC1
      PC2

      Murder
      0.041704
      0.044822

      Assault
      0.995221
      0.058760

      UrbanPop
      0.046336
      -0.976857

      Rape
      0.075156
      -0.200718

```python
# Transform X to get the principal components
X_dim2_u = pca2_u.transform(X)
df_dim2_u = pd.DataFrame(X_dim2_u, columns=['PC1', 'PC2'], index=df.index)
df_dim2_u.head()
```

      PC1
      PC2

      State

      Alabama
      64.802164
      11.448007

      Alaska
      92.827450
      17.982943

      Arizona
      124.068216
      -8.830403

      Arkansas
      18.340035
      16.703911

      California
      107.422953
      -22.520070

```python
fig, (ax1,ax2) = plt.subplots(1,2,figsize=(18,9))

# Scaled PCA
for i in df_dim2.index:
    ax1.annotate(i, (df_dim2.PC1.loc[i], -df_dim2.PC2.loc[i]), ha='center', fontsize=14)
ax1b = ax1.twinx().twiny()
ax1b.set_ylim(-1,1); ax1b.set_xlim(-1,1)
for i in df_weights[['PC1', 'PC2']].index:
    ax1b.annotate(i, (df_weights.PC1.loc[i]*1.05, -df_weights.PC2.loc[i]*1.05), color='orange', fontsize=16)
    ax1b.arrow(0,0,df_weights.PC1[i], -df_weights.PC2[i], color='orange', lw=2)
ax1.set_title('Scaled')

# Unscaled PCA
for i in df_dim2_u.index:
    ax2.annotate(i, (df_dim2_u.PC1.loc[i], -df_dim2_u.PC2.loc[i]), ha='center', fontsize=14)
ax2b = ax2.twinx().twiny()
ax2b.set_ylim(-1,1); ax2b.set_xlim(-1,1)
for i in df_weights_u[['PC1', 'PC2']].index:
    ax2b.annotate(i, (df_weights_u.PC1.loc[i]*1.05, -df_weights_u.PC2.loc[i]*1.05), color='orange', fontsize=16)
    ax2b.arrow(0,0,df_weights_u.PC1[i], -df_weights_u.PC2[i], color='orange', lw=2)
ax2.set_title('Unscaled')

# Plot reference lines
for ax,df in zip((ax1,ax2), (df_dim2,df_dim2_u)):
    m = np.max(np.abs(df.values))*1.2
    ax.hlines(0,-m,m, linestyles='dotted', colors='grey')
    ax.vlines(0,-m,m, linestyles='dotted', colors='grey')
    ax.set_xlabel('First Principal Component')
    ax.set_ylabel('Second Principal Component')
    ax.set_xlim(-m,m); ax.set_ylim(-m,m)
```

As predicted, the first principal component loading vector places almost all of its weight on `Assault`, while the second principal component loading vector places almost all of its weight on `UrpanPop`. Comparing this to the left-hand plot, we see that scaling does indeed have a substantial effect on the results obtained. However, this result is simply a consequence of the scales on which the variables were measured.

#### The Proportion of Variance Explained

We can now ask a natural question: how much of the information in a given data set is lost by projecting the observations onto the first few principal components? That is, how much of the variance in the data is not contained in the first few principal components? More generally, we are interested in knowing the **proportion of variance explained (PVE)** by each principal component.

```python
# Four components
pca4 = PCA(n_components=4).fit(X_scaled)
```

```python
# Variance of the four principal components
pca4.explained_variance_
```

    array([2.53085875, 1.00996444, 0.36383998, 0.17696948])

```python
# As a percentage of the total variance
pca4.explained_variance_ratio_
```

    array([0.62006039, 0.24744129, 0.0891408 , 0.04335752])

In the `Arrest` dataset, the first principal component explains $62.0\%$ of the variance in the data, and the next principal component explains $24.7\%$ of the variance. Together, the first two principal components explain almost $87\%$ of the variance in the data, and the last two principal components explain only $13\%$ of the variance.

```python
fig, (ax1, ax2) = plt.subplots(1,2, figsize=(12,5))
fig.suptitle('Figure 10.2');

# Relative
ax1.plot([1,2,3,4], pca4.explained_variance_ratio_)
ax1.set_ylabel('Prop. Variance Explained')
ax1.set_xlabel('Principal Component');

# Cumulative
ax2.plot([1,2,3,4], np.cumsum(pca4.explained_variance_ratio_))
ax2.set_ylabel('Cumulative Variance Explained');
ax2.set_xlabel('Principal Component');
```

#### Deciding How Many Principal Components to Use

In general, a $n \times p$ data matrix $X$ has $\min\{n − 1, p\}$ distinct principal components. However, we usually are not interested in all of them; rather, we would like to use just the first few principal components in order to visualize or interpret the data.

We typically decide on the number of principal components required to visualize the data by examining a *scree plot*.

However, there is no well-accepted objective way to decide how many principal com- ponents are enough.

## Clustering

Clustering refers to a very broad set of techniques for finding subgroups, or clusters, in a data set. When we cluster the observations of a data set, we seek to partition them into distinct groups so that the observations within each group are quite similar to each other, while observations in different groups are quite different from each other.

In this section we focus on perhaps the two best-known clustering approaches:

1. **K-means clustering**: we seek to partition the observations into a pre-specified clustering number of clusters
2. **Hierarchical clustering**: we do not know in advance how many clusters we want; in fact, we end up with a tree-like visual representation of the observations, called a dendrogram, that allows us to view at once the clusterings obtained for each possible number of clusters, from 1 to n.

### K-Means Clustering

The idea behind K-means clustering is that a good clustering is one for which the within-cluster variation is as small as possible. Hence we want to solve the problem

$$
\underset{C_{1}, \ldots, C_{K}}{\operatorname{minimize}}\left\{\sum_{k=1}^{K} W\left(C_{k}\right)\right\}
$$

where $C_k$ is a cluster and $ W(C_k)$ is a measure of the amount by which the observations within a cluster differ from each other.

There are many possible ways to define this concept, but by far the most common choice involves **squared Euclidean distance**. That is, we define

$$
W\left(C_{k}\right)=\frac{1}{\left|C_{k}\right|} \sum_{i, i^{\prime} \in C_{k}} \sum_{j=1}^{p}\left(x_{i j}-x_{i^{\prime} j}\right)^2
$$

where $|C_k|$ denotes the number of observations in the $k^{th}$ cluster.

### Algorithm

1. Randomly assign a number, from $1$ to $K$, to each of the observations. These serve as initial cluster assignments for the observations.

2. Iterate until the cluster assignments stop changing:

    a) For each of the $K$ clusters, compute the cluster centroid. The kth cluster centroid is the vector of the $p$ feature means for the observations in the $k^{th}$ cluster.

    b) Assign each observation to the cluster whose centroid is closest (where closest is defined using Euclidean distance).

```python
np.random.seed(123)

# Simulate data
X = np.random.randn(50,2)
X[0:25, 0] = X[0:25, 0] + 3
X[0:25, 1] = X[0:25, 1] - 4
```

```python
fig, ax = plt.subplots(figsize=(6, 5))
fig.suptitle("Baseline")

# Plot
ax.scatter(X[:,0], X[:,1], s=50, alpha=0.5, c='k')
ax.set_xlabel('X0'); ax.set_ylabel('X1');
```

```python
np.random.seed(1)

# Init clusters
K = 2
clusters0 = np.random.randint(K,size=(np.size(X,0)))
```

```python
fig, ax = plt.subplots(figsize=(6, 5))
fig.suptitle("Random assignment")

# Plot
ax.scatter(X[clusters0==0,0], X[clusters0==0,1], s=50, alpha=0.5)
ax.scatter(X[clusters0==1,0], X[clusters0==1,1], s=50, alpha=0.5)
ax.set_xlabel('X0'); ax.set_ylabel('X1');
```

```python
# Compute new centroids
def compute_new_centroids(X, clusters):
    K = len(np.unique(clusters))
    centroids = np.zeros((K,np.size(X,1)))
    for k in range(K):
        if sum(clusters==k)>0:
            centroids[k,:] = np.mean(X[clusters==k,:], axis=0)
        else:
            centroids[k,:] = np.mean(X, axis=0)
    return centroids
```

```python
# Print
centroids0 = compute_new_centroids(X, clusters0)
print(centroids0)
```

    [[ 1.35725989 -2.15281035]
     [ 1.84654757 -1.99437838]]

```python
# Plot assignment
def plot_assignment(X, centroids, clusters, d, i):
    clear_output(wait=True)
    fig, ax = plt.subplots(figsize=(6, 5))
    fig.suptitle("Iteration %.0f: inertia=%.1f" % (i,d))

    # Plot
    ax.clear()
    colors = plt.rcParams['axes.prop_cycle'].by_key()['color'];
    K = np.size(centroids,0)
    for k in range(K):
        ax.scatter(X[clusters==k,0], X[clusters==k,1], s=50, c=colors[k], alpha=0.5)
        ax.scatter(centroids[k,0], centroids[k,1], marker = '*', s=300, color=colors[k])
        ax.set_xlabel('X0'); ax.set_ylabel('X1');

    # Show
    plt.show();
```

```python
# Plot
plot_assignment(X, centroids0, clusters0, 0, 0)
```

```python
# Assign X to clusters
def assign_to_cluster(X, centroids):
    K = np.size(centroids,0)
    dist = np.zeros((np.size(X,0),K))
    for k in range(K):
        dist[:,k] = np.mean((X - centroids[k,:])**2, axis=1)
    clusters = np.argmin(dist, axis=1)

    # Compute inertia
    inertia = 0
    for k in range(K):
        if sum(clusters==k)>0:
            inertia += np.sum((X[clusters==k,:] - centroids[k,:])**2)
    return clusters, inertia
```

```python
# Get cluster assignment
[clusters1,d] = assign_to_cluster(X, centroids0)
```

```python
# Plot
plot_assignment(X, centroids0, clusters1, d, 1)
```

We now have all the components to proceed iteratively.

```python
def kmeans_manual(X, K):

    # Init
    i = 0
    d0 = 1e4
    d1 = 1e5
    clusters = np.random.randint(K,size=(np.size(X,0)))

    # Iterate until convergence
    while np.abs(d0-d1) > 1e-10:
        d1 = d0
        centroids = compute_new_centroids(X, clusters)
        [clusters, d0] = assign_to_cluster(X, centroids)
        plot_assignment(X, centroids, clusters, d0, i)
        i+=1
```

```python
# Test
kmeans_manual(X, K)
```

Here the observations can be easily plotted because they are two-dimensional.
If there were more than two variables then we could instead perform PCA
and plot the first two principal components score vectors.

In this example, we knew that there really were two clusters because
we generated the data. However, for real data, in general we do not know
the true number of clusters. We could instead have performed K-means
clustering on this example with `K  =  3`. If we do this, K-means clustering will split up the two "real" clusters, since it has no information about them:

```python
# K=3
kmeans_manual(X, 3)
```

The automated function in `sklearn` to persorm $K$-means clustering is `KMeans`.

```python
# SKlearn algorithm
km1 = KMeans(n_clusters=3, n_init=1, random_state=1)
km1.fit(X)
```

    KMeans(n_clusters=3, n_init=1, random_state=1)

```python
# Plot
plot_assignment(X, km1.cluster_centers_, km1.labels_, km1.inertia_, km1.n_iter_)
```

As we can see, the results are different in the two algorithms? Why? $K$-means is susceptible to the initial values. One way to solve this problem is to run the algorithm multiple times and report only the best results

To run the `Kmeans()` function in python with multiple initial cluster assignments, we use the `n_init` argument (default: 10). If a value of `n_init` greater than one is used, then K-means clustering will be performed using multiple random assignments, and the `Kmeans()` function will report only the best results.

```python
# 30 runs
km_30run = KMeans(n_clusters=3, n_init=30, random_state=1).fit(X)
plot_assignment(X, km_30run.cluster_centers_, km_30run.labels_, km_30run.inertia_, km_30run.n_iter_)
```

It is generally recommended to always run K-means clustering with a large value of `n_init`, such as 20 or 50 to avoid getting stuck in an undesirable local optimum.

When performing K-means clustering, in addition to using multiple initial cluster assignments, it is also important to set a random seed using the `random_state` parameter. This way, the initial cluster assignments can be replicated, and the K-means output will be fully reproducible.

### Hierarchical Clustering

One potential disadvantage of K-means clustering is that it requires us to pre-specify the number of clusters $K$. Hierarchical clustering is an alternative approach which does not require that we commit to a particular choice of $K$

#### The Dendogram

Hierarchical clustering has an added advantage over K-means clustering in that it results in an attractive tree-based representation of the observations, called a **dendrogram**.

```python
plt.figure(figsize=(25, 10))
plt.title('Hierarchical Clustering Dendrogram')

# calculate full dendrogram
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    linkage(X, "complete"),
    leaf_rotation=90.,  # rotates the x axis labels
    leaf_font_size=8.,  # font size for the x axis labels
)
plt.show()
```

Each leaf of the *dendrogram* represents one observation.

As we move up the tree, some leaves begin to fuse into branches. These correspond to observations that are similar to each other. As we move higher up the tree, branches themselves fuse, either with leaves or other branches. The earlier (lower in the tree) fusions occur, the more similar the groups of observations are to each other.

We can use de *dendogram* to understand how similar two observations are: we can look for the point in the tree where branches containing those two obse rvations are first fused. The height of this fusion, as measured on the vertical axis, indicates how different the two observations are. Thus, observations that fuse at the very bottom of the tree are quite similar to each other, whereas observations that fuse close to the top of the tree will tend to be quite different.

The term **hierarchical** refers to the fact that clusters obtained by cutting the dendrogram at a given height are necessarily nested within the clusters obtained by cutting the dendrogram at any greater height.

#### The Hierarchical Clustering Algorithm

1. Begin with $n$ observations and a measure (such as Euclidean distance) of all the $n(n − 1)/2$ pairwise dissimilarities. Treat each 2 observation as its own cluster.

2. For $i=n,n−1,...,2$

    a) Examine all pairwise inter-cluster dissimilarities among the $i$ clusters and identify the **pair of clusters that are least dissimilar** (that is, most similar). Fuse these two clusters. The dissimilarity between these two clusters indicates the height in the dendrogram at which the fusion should be placed.

    b) Compute the new pairwise inter-cluster dissimilarities among the $i−1$ remaining clusters.

#### The linkage function

We have a concept of the dissimilarity between pairs of observations, but how do we define the dissimilarity between two clusters if one or both of the clusters contains multiple observations?

The concept of dissimilarity between a pair of observations needs to be extended to a pair of groups of observations. This extension is achieved by developing the notion of **linkage**, which defines the dissimilarity between two groups of observations.

The four most common types of linkage are:

1. **Complete**: Maximal intercluster dissimilarity. Compute all pairwise dissimilarities between the observations in cluster A and the observations in cluster B, and record the largest of these dissimilarities.
2. **Single**: Minimal intercluster dissimilarity. Compute all pairwise dissimilarities between the observations in cluster A and the observations in cluster B, and record the smallest of these dissimilarities.
3. **Average**: Mean intercluster dissimilarity. Compute all pairwise dissimilarities between the observations in cluster A and the observations in cluster B, and record the average of these dissimilarities.
4. **Centroid**: Dissimilarity between the centroid for cluster A (a mean vector of length p) and the centroid for cluster B. Centroid linkage can result in undesirable inversions.

Average, complete, and single linkage are most popular among statisticians. Average and complete linkage are generally preferred over single linkage, as they tend to yield more balanced dendrograms. Centroid linkage is often used in genomics, but suffers from a major drawback in that an inversion can occur, whereby two clusters are fused at a height below either of the individual clusters in the dendrogram. This can lead to difficulties in visualization as well as in interpretation of the dendrogram.

```python
# Init
linkages = [hierarchy.complete(X), hierarchy.average(X), hierarchy.single(X)]
titles = ['Complete Linkage', 'Average Linkage', 'Single Linkage']
```

```python
fig, (ax1,ax2,ax3) = plt.subplots(1,3, figsize=(18,10))

# Plot
for linkage, t, ax in zip(linkages, titles, (ax1,ax2,ax3)):
    dendrogram(linkage, ax=ax)
    ax.set_title(t)
```

For this data, complete and average linkage generally separates the observations into their correct groups.


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.5.1 (Python 3.6.10-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.5.1/

```python
#init_printing?
```

### Parameters and two Gaussians

```python
a, b, c, a1, a2 = symbols('a b c a1 a2', positive=True, real=True)
```

```python
g1=x*exp(-a1*x**2)
g2=x*exp(-a2*x**2)
g1, g2
```

### Normalization constant

```python
N=integrate(g1*g1, (x, -oo, oo))
N
```

```python
1/sqrt(N)
```

```python
printing.sstrrepr(1/sqrt(N))
```

    '2*2**(1/4)*a1**(3/4)/pi**(1/4)'

### Overlap integral S

```python
S=integrate(g1*g2, (x, -oo, oo))
S
```

```python
S.simplify()
```

```python
printing.sstrrepr(S.simplify())
```

    'sqrt(pi)/(2*(a1 + a2)**(3/2))'

### Kinetic energy $T = -\frac{\hbar^2}{2m} \frac{d^2}{dx^2} = \frac{1}{2m}\left(\frac{\hbar}{i}\frac{d}{dx} \right)^2$

```python
d1=diff(g1,x)
d2=diff(g2,x)
d1, d2
```

```python
T = 1/2 * integrate(d1*d2, (x, -oo, oo))
#T=T.simplify()
#T=T.factor()
T.factor()
```

```python
printing.sstrrepr(T.factor())
```

    '1.5*sqrt(pi)*a1*a2/(a1 + a2)**(5/2)'

### Potential $V(x) = (ax^2 - b)e^{-cx^2}$

```python
v=(a*x**2-b)*exp(-c*x**2)
v
```

```python
V = integrate(g1*v*g2, (x, -oo, oo))
V
```

```python
V.factor()
```

```python
printing.sstrrepr(V.factor())
```

    'sqrt(pi)*(3*a - 2*a1*b - 2*a2*b - 2*b*c)/(4*(a1 + a2 + c)**(5/2))'

```python

```


# Matrix Factorization implementation

Iván Vallés Pérez - 2018

In a [recommender system](https://en.wikipedia.org/wiki/Recommender_system) we have two kind of entities: users and items. We want to predict for an arbitrary user which items the user prefers. Sometimes users give explicit ratings, for example 4 or 5 star reviews. Let's start with this case and give an accurate mathematical formulation of the problem.

We will call $r_{ij}$ the rating that user $i$ gives to item $j$ and we want to build a model to predict these ratings. Let's call the predictions of the model $\hat{r}_{ij}$. It's important to realize that since each user will probably give ratings to just a handful of items and in most cases there are thousands or even millions of items we don't have ratings for the vast majority of user-item possible interactions. Let's call $I$ the set of known interactions and let's suppose that we want to minimize the squared loss, that is, we want to minimize:

\begin{equation}
L = \sum_{(i,j) \in I} (r_{ij} - \hat{r}_{ij})^2
\end{equation}

A matrix-factorization based recommender will solve the above problem by supposing that:

- We represent user $i$ with an unknown user bias $u_i^b$ and an unknown vector of length $K$ that we will call $u_i^e$, which is usually called the user embedding.
- We represent item $j$ with an unknown item bias $v_j^b$ and an unknown vector of length $K$ that we will call $v_j^e$, which is usually called the item embedding.
- The predicted rating of item $j$ by user $i$ is the biases plus the dot product of the two embeddings:

\begin{equation}
\hat{r}_{ij} = u_i^b + v_j^b + u_i^e \cdot v_j^e = u_i^b + v_j^b + \sum_{k=1}^K u_{ik}^ev_{jk}^e
\end{equation}

The above vectors are the parameters of our problem and $K$ is a hyperparameter. If we have $N$ users and $M$ items this means that we have $(K + 1)(N + M)$ parameters. Substituting inside the loss function we have:

\begin{equation}
L = \sum_{(i,j) \in I} (r_{ij} - u_i^b - v_j^b - \sum_{k=1}^K  u_{ik}^ev_{jk}^e)^2
\end{equation}

To improve the generalization capabilities of the model regularization is added and finally we have:

\begin{equation}
L = \sum_{(i,j) \in I} (r_{ij} -  u_i^b - v_j^b - \sum_{k=1}^Ku_{ik}^ev_{jk}^e)^2 + \lambda_u\sum_{i,k} (u_{ik}^e)^2 + \lambda_v\sum_{j, k} (v_{jk}^e)^2
\end{equation}

$\lambda_u$ and $\lambda_v$ are two additional hyperparameters.

```python
%cd ..
```

```python
from src.matrix_factorization import MatrixFactorization
from src.deep_factorization import DeepFactorization
```

    C:\Users\Ivan Valles Perez\AppData\Local\Continuum\Anaconda3\lib\site-packages\h5py\__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import clear_output

# Load the data and calculate the users set and items set cardinalities
df = pd.read_csv("./data/interactions.csv", names=["U", "I", "Q"])
U_cardinality = df.U.nunique()
I_cardinality = df.I.nunique()
```

```python
# Generates and shuffles the data set to remove possible tendencies
np.random.seed(655321)
mat = df.values
np.random.shuffle(mat)
```

```python
# Divide the data set into train-dev-test
train_mat = mat[:85000]
dev_mat = mat[85000:90000]
test_mat = mat[90000:]
```

```python
# Separate features from target
x_train=(train_mat[:,:2]-1)
y_train=(train_mat[:,2:])
x_dev=(dev_mat[:,:2]-1)
y_dev=(dev_mat[:,2:])
x_test=(test_mat[:,:2]-1)
y_test=(test_mat[:,2:])
```

## Matrix Factorization

```python
# Initialize the model and the performance accumulators for reporting purposes
# The metaparameters have been tuned to improve the algorithm performance
MF = MatrixFactorization(n_users=U_cardinality, n_items=I_cardinality, emb_size=5, lr=0.002, _lambda=0.1)
losses_train = [MF.evaluate(x_train, y_train, batch_size=1000)] # Add the initial loss (w. random weights)
losses_dev = [MF.evaluate(x_dev, y_dev, batch_size=1000)] # Add the initial loss (w. random weights)
```

```python
for i in range(50): # Run for 50 epochs

    MF.fit(x_train, y_train, batch_size=128) # Compute an epoch using SGD
    losses_train.append(MF.evaluate(x_train, y_train, batch_size=128)) # Compute the train performance
    losses_dev.append(MF.evaluate(x_dev, y_dev, batch_size=128)) # Compute the dev. performance

    # Plot train and dev errors over time
    clear_output(True)
    plt.figure(figsize=[10, 3])
    plt.plot(losses_train)
    plt.plot(losses_dev)
    axes = plt.gca()
    plt.legend(["Train MSE", "Dev MSE"])
    plt.title("Training errors over time (Mean Squared Error)")
    plt.ylabel("Log-MSE")
    plt.xlabel("Epochs")
    plt.ylim([0.1, axes.get_ylim()[1]])
    plt.yscale('log')
    plt.grid(True, which="both",ls="-")
    plt.show()
    print("[EPOCH {0}]   Train error = {1:.4f} | Dev error = {2:.4f}".format(i+1, losses_train[-1], losses_dev[-1]))

print("Test MSE achieved = {0:.4f}".format(MF.evaluate(x_test, y_test, batch_size=128)))

```

## Extra: solution using deep learning

```python
DF = DeepFactorization(n_users=U_cardinality, n_items=I_cardinality, emb_size=5, lr=0.002, _lambda=0.1)
losses_train = [DF.evaluate(x_train, y_train, batch_size=1000)] # Add the initial loss (w. random weights)
losses_dev = [DF.evaluate(x_dev, y_dev, batch_size=1000)] # Add the initial loss (w. random weights)
```

```python
for i in range(50): # Run for 50 epochs

    DF.fit(x_train, y_train, batch_size=128) # Compute an epoch using SGD
    losses_train.append(DF.evaluate(x_train, y_train, batch_size=128)) # Compute the train performance
    losses_dev.append(DF.evaluate(x_dev, y_dev, batch_size=128)) # Compute the dev. performance

    # Plot train and dev errors over time
    clear_output(True)
    plt.figure(figsize=[10, 3])
    plt.plot(losses_train)
    plt.plot(losses_dev)
    axes = plt.gca()
    plt.legend(["Train MSE", "Dev MSE"])
    plt.title("Training errors over time (Mean Squared Error)")
    plt.ylabel("Log-MSE")
    plt.xlabel("Epochs")
    plt.ylim([0.1, axes.get_ylim()[1]])
    plt.yscale('log')
    plt.grid(True, which="both",ls="-")
    plt.show()
    print("[EPOCH {0}]   Train error = {1:.4f} | Dev error = {2:.4f}".format(i+1, losses_train[-1], losses_dev[-1]))

print("Test MSE achieved = {0:.4f}".format(DF.evaluate(x_test, y_test, batch_size=128)))

```


Euler Problem 41
================

We shall say that an n-digit number is pandigital if it makes use of all the digits 1 to n exactly once. For example, 2143 is a 4-digit pandigital and is also prime.

What is the largest n-digit pandigital prime that exists?

```python
from itertools import permutations
from sympy import isprime

for v in permutations(range(7, 0, -1)):
    p = int(''.join(map(str, v)))
    if isprime(p):
        print(p)
        break
```

    7652413

**Explanation:** Every pandigital number $N$ with 8 or 9 digits is divisible by 9, since the sum of the digits of $N$ is $1 + 2 + 3 + \cdots + 8 = 36$ or $1 + 2 + 3 + \cdots + 9 = 45$, respectively. Therefore, pandigital primes have at most 7 digits.

We use `itertools.permutations` to iterate through all permutations of the digits 1-7 in reverse order until we find a permutation that forms a prime number.


```python
import numpy as np
import matplotlib.pyplot as plt
import emcee
import scipy.spatial.distance
from scipy.stats import multivariate_normal
```

# Linear regression: uncertainty in uncertainty

Let's say we have a data in x and y axes as below:

```python
x = np.array([0.0596779, 0.34317802, 0.39211752, 0.42310646, 0.43857224, 0.4809319, 0.68482974,
              0.71946897, 0.72904971, 0.9807642])
y = np.array([-1.39284066, 0.02093466, -0.48427289, -0.36730135, -0.33372661, 0.49791066,
              0.89920648, 0.63361326, 0.47788066, 1.07935026])
plt.plot(x, y, '.')
plt.show()
```

With the data given above, we would like to fit it with a linear regression **y = a + bx**. That is, we would like to determine the coefficients and its error in different cases:

1. Assuming the model is correct and the error is known
2. Assuming the model is correct and the error is unknown
3. The model has an assumed inadequacy and the error is unknown

## Known correct model & known error

In this first case, we will determine the uncertainty of the coefficients if we know the standard deviation of the data.
The way we invert it is using Bayesian inference with the help from `emcee` sampler.

The probability density function of the coefficients can be written as

$$\begin{equation}
P(a,b | \mathcal{D}) \propto P(\mathcal{D} | a, b) P(a, b)
\end{equation}$$

Assuming flat prior of \\(a, b\\), we can write the probability of the coefficients as

$$\begin{align}
P(a,b | \mathcal{D}) &\propto P(\mathcal{D} | a, b) \\
& \propto \exp\left[-\sum_i \frac{(a+bx_i-y_i)^2}{2\sigma^2}\right]
\end{align}$$

From the expression above, we can draw samples of \\(a,b\\) using the `emcee` sampler.

```python
def lnprob1(param, x, y):
    param = np.array(param)
    a = param[0]
    b = param[1]
    sigma = 0.28297849805199204
    return -np.sum((a + b * x - y)**2) / (2 * sigma**2)
```

```python
ndim, nwalkers = 2, 100
ivar = 1. / np.random.rand(ndim)
p0 = [np.random.rand(ndim) for i in range(nwalkers)]

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob1, args=[x, y])
pos, prob, state = sampler.run_mcmc(p0, 10000)
```

```python
a_post1 = sampler.flatchain[10000:,0]
b_post1 = sampler.flatchain[10000:,1]
plt.figure(figsize=(12,4))
plt.subplot(1,2,1)
plt.hist(a_post1, 50, normed=True)
plt.title("a")
plt.subplot(1,2,2)
plt.hist(b_post1, 50, normed=True)
plt.title("b")
plt.show()

print("a: %f +- %f (true: %f)" % (np.mean(a_post1), np.std(a_post1), -1.0856306033005612))
print("b: %f +- %f (true: %f)" % (np.mean(b_post1), np.std(b_post1), 1.9946908931671716))
```

## Known correct model and unknown error

In this case, we don't know quite sure what the error was, so we put a prior uncertainty of the \\(\sigma\\) and get the posterior belief on that as well as the other coefficients.

The probability density function of the coefficients can be written as

$$\begin{equation}
P(a,b,\sigma | \mathcal{D}) \propto P(\mathcal{D} | a, b, \sigma) P(a, b) P(\sigma)
\end{equation}$$

As before, the prior of \\(a, b\\) is assumed flat. However, we assumed the \\(\sigma\\) to have the flat log prior. Therefore, we can write the probability of the coefficients as

$$\begin{align}
P(a,b,\sigma | \mathcal{D}) &\propto P(\mathcal{D} | a, b, \sigma) \frac{1}{\sigma} \\
& \propto \exp\left[-\sum_i \frac{(a+bx_i-y_i)^2}{2\sigma^2}\right] \frac{1}{\sigma^{N+1}}
\end{align}$$
where \\(N\\) is the number of data samples.

From the expression above, we can draw samples of \\(a,b, \sigma\\) using the `emcee` sampler.

```python
def lnprob2(param, x, y):
    a, b, sigma = param
    if sigma < 0: return -np.inf

    ymodel = a + b * x
    N = len(x)
    return - np.sum((ymodel - y)**2) / (2*sigma**2) - (N+1) * np.log(sigma)
```

```python
ndim, nwalkers = 3, 100
p0 = np.random.random((nwalkers, ndim))

sampler2 = emcee.EnsembleSampler(nwalkers, ndim, lnprob2, args=[x, y])
pos, prob, state = sampler2.run_mcmc(p0, 10000)
```

```python
a_post2 = sampler2.flatchain[10000:,0]
b_post2 = sampler2.flatchain[10000:,1]
s_post2 = sampler2.flatchain[10000:,2]
plt.figure(figsize=(12,4))
plt.subplot(1,3,1)
plt.hist(a_post2, 50, normed=True)
plt.title("a")
plt.subplot(1,3,2)
plt.hist(b_post2, 50, normed=True)
plt.title("b")
plt.subplot(1,3,3)
plt.hist(s_post2, 50, normed=True)
plt.title("sigma")
plt.show()

print("a: %f +- %f (true: %f)" % (np.mean(a_post2), np.std(a_post2), -1.0856306033005612))
print("b: %f +- %f (true: %f)" % (np.mean(b_post2), np.std(b_post2), 1.9946908931671716))
print("sigma: %f +- %f (true: %f)" % (np.mean(s_post2), np.std(s_post2), 0.28297849805199204))
```

## Unknown correct model and unknown error

This is similar to the previous case, except that we are not sure that the linear model is the correct one. One way to encode our uncertainty is to express the observation as

$$\begin{equation}
\hat{y}(x) = a + bx + \varepsilon + \eta(x)
\end{equation}$$

where \\(\varepsilon \sim \mathcal{N}(0, \sigma^2)\\) is the Gaussian noise and \\(\eta(x)\\) is the model inadequacy. Let's assume the model inadequacy is a Gaussian process with mean zero and squared exponential kernel:

$$\begin{align}
\eta(x) & \sim \mathcal{GP}\left[0, c(\cdot, \cdot)\right] \\
c(x_1, x_2) & = m^2 \exp\left[-\frac{(x_1 - x_2)^2}{2 d^2}\right]
\end{align}$$

To encode the Gaussian noise and the Gaussian process in one expression, we can write it as

$$\begin{align}
(\eta(x) + \varepsilon) & \sim \mathcal{GP}\left[0, c_2(\cdot, \cdot)\right] \\
c_2(x_1, x_2) & = m^2 \exp\left[-\frac{(x_1 - x_2)^2}{2 d^2}\right] + \delta(x_1-x_2) \sigma^2
\end{align}$$

where \\(\delta\\) term is one when the argument is zero and zero otherwise.

The posterior distribution is given by

$$\begin{equation}
P(a,b,\sigma,m,d | \mathcal{D}) \propto P(\mathcal{D} | a, b, \sigma, m, d) P(a, b) P(\sigma) P(m) P(d)
\end{equation}$$

As before, the prior of \\(a, b\\) is assumed flat and \\(\sigma\\) to have the flat log prior. Here we also assume \\(m,d\\) to have the flat log prior. Thus, we can write the probability of the coefficients as

$$\begin{align}
P(a,b,\sigma,m,d | \mathcal{D}) &\propto P(\mathcal{D} | a, b, \sigma, m, d) \frac{1}{\sigma m d} \\
& \propto \mathcal{GP}\left[\hat{y}; a + bx, c_2(\cdot, \cdot)\right] \frac{1}{\sigma m d}
\end{align}$$

From the expression above, we can draw samples of \\(a,b, \sigma, m, d\\) using the `emcee` sampler.

```python
def lnprob3(param, x, y, dist):
    a, b, sigma, m, d = param
    if sigma < 0 or m < 0 or d < 0: return -np.inf
    if d > 1: return -np.inf

    # calculate the covariance matrix
    cov = m*m * np.exp(-dist**2/(2*d*d)) + np.eye(len(x)) * (sigma*sigma)
    # cov = np.eye(len(x)) * (sigma * sigma)

    ymodel = a + b * x
    obs = ymodel - y
    return multivariate_normal.logpdf(obs, mean=np.zeros(obs.shape[0]), cov=cov) - np.log(sigma * m * d)
```

```python
dist = scipy.spatial.distance.squareform(scipy.spatial.distance.pdist(np.expand_dims(x, 1)))
```

```python
ndim, nwalkers = 5, 100
p0 = np.random.random((nwalkers, ndim))

sampler3 = emcee.EnsembleSampler(nwalkers, ndim, lnprob3, args=[x, y, dist])
pos, prob, state = sampler3.run_mcmc(p0, 100000)
```

```python
nburn = 10000
a_post3 = sampler3.flatchain[nburn:,0]
b_post3 = sampler3.flatchain[nburn:,1]
s_post3 = sampler3.flatchain[nburn:,2]
m_post3 = sampler3.flatchain[nburn:,3]
d_post3 = sampler3.flatchain[nburn:,4]
plt.figure(figsize=(20,4))
plt.subplot(1,5,1)
plt.hist(a_post3, 50, normed=True)
plt.title("a")
plt.subplot(1,5,2)
plt.hist(b_post3, 50, normed=True)
plt.title("b")
plt.subplot(1,5,3)
plt.hist(s_post3, 50, normed=True)
plt.title("sigma")
plt.subplot(1,5,4)
plt.hist(m_post3, 50, normed=True)
plt.title("m")
plt.subplot(1,5,5)
plt.hist(d_post3, 50, normed=True)
plt.title("d")
plt.show()

print("a: %f +- %f (true: %f)" % (np.mean(a_post3), np.std(a_post3), -1.0856306033005612))
print("b: %f +- %f (true: %f)" % (np.mean(b_post3), np.std(b_post3), 1.9946908931671716))
print("sigma: %f +- %f (true: %f)" % (np.mean(s_post3), np.std(s_post3), 0.28297849805199204))
print("m: %f +- %f" % (np.mean(m_post3), np.std(m_post3)))
print("d: %f +- %f" % (np.mean(d_post3), np.std(d_post3)))
```

## Conclusions

We have compared the retrieved coefficients with different degrees of uncertainty: (1) known model and known error, (2) known model and unknown error, and (3) unknown model and unknown error. Here are the comparisons of the retrieved coefficients for those three cases.

```python
plt.figure(figsize=(10, 12))
a_xlim = (-2.5, 1)
b_xlim = (0, 5)

plt.subplot(3,2,1)
plt.hist(a_post1, 50, normed=True, range=a_xlim)
plt.title("a")
plt.ylabel("Known model and known error")
plt.subplot(3,2,2)
plt.hist(b_post1, 50, normed=True, range=b_xlim)
plt.title("b")

plt.subplot(3,2,3)
plt.hist(a_post2, 50, normed=True, range=a_xlim)
plt.ylabel("Known model and unknown error")
plt.subplot(3,2,4)
plt.hist(b_post2, 50, normed=True, range=b_xlim)

plt.subplot(3,2,5)
plt.hist(a_post3, 50, normed=True, range=a_xlim)
plt.ylabel("Unknown model and unknown error")
plt.subplot(3,2,6)
plt.hist(b_post3, 50, normed=True, range=b_xlim)

plt.show()
```

### Data generator

```python
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(123)
size = (10,)
a = np.random.randn()
b = np.random.randn() * 2
sigma = np.abs(np.random.randn() * 1)
x = np.sort(np.random.random(size))
y = a + b * x + np.random.randn(*size) * sigma

plt.errorbar(x, y, yerr=sigma, fmt='.')
plt.show()

print(x)
print(y)
print(a, b, sigma)
```

```python

```


# Linear Algebra: Eigenvectors and Eigenvalues

## Eigenvalues and Eigenvectors

Assume, we have two interest bearing accounts. The first gives an interest rate of 5%, the second a 3% interest, with annual compound.

Assume that after $t$ years the amounts in the two accounts are represented by a 2-vector:

$x^{(t)} = \begin{bmatrix}
 amount in Account 1 \\[0.3em]
 amount in Account 2
\end{bmatrix}$

The growth of the amounts in one year can be described in a matrix:

$x^{(t+1)} = \begin{bmatrix}
 a_{11} & a_{12} \\[0.3em]
 a_{21} & a_{22}
\end{bmatrix} x^{(t)}$

Given the specification of the interest rate above, this simple case gives us:

$x^{(t+1)} = \begin{bmatrix}
 1.05 & 0    \\[0.3em]
 0    & 1.03
\end{bmatrix} x^{(t)}$

Let $A$ denote the matrix: $\begin{bmatrix}
 1.05 & 0    \\[0.3em]
 0    & 1.03
\end{bmatrix}$

$A$ is a diagonal.

If we initially put \$ 100 on each account, we can compute the result of the accummulated interest after a year as:

```python
import numpy as np
x = np.array([[100],
              [100]])
A = np.array([[1.05, 0],
              [0,    1.03]])
A.dot(x)
```

    array([[105.],
           [103.]])

After two years the accounts would be:

```python
A.dot(A.dot(x))
```

    array([[110.25],
           [106.09]])

If we might want to know how $x^{(100)}$ compares to $x^{(0)}$, we could iterate over:

$\begin{align}
x^{(100)} & = A x^{(99)} \\
          & = A(Ax^{(98)}) \\
          & = A(A(Ax^{(97)})) \\
          & \vdots \\
          & = \underbrace{A \cdot A \dots A}_\text{100 times} \ x^{(0)}
\end{align}$

We can also write the product as $A^{100}$.

Note that $A$ is a diagonal, thus the entries of $A^{100}$ are $1.05^{100}$ and $1.03^{100}$:

$A^{100} = \begin{bmatrix}
 131.50125784630401 & 0    \\[0.3em]
 0    & 19.218631980856298
\end{bmatrix}$

What we can see is that account 1 dominates account 2, account 2 becoming less and less relevant over time.

Now consider the definition below:

**Basic definition:**

Given $A \in \mathbb{R}^{n\times n}$, $\lambda$ is the **eigenvalue** of $A$, if there is a non-zero vector $x$, the corresponding **eigenvector**, if the following is true:

$Ax = \lambda x, x \neq 0$

Formally, given a square matrix $A \in \mathbb{R}^{n\times n}$, we say that $\lambda \in \mathbb{C}$ is an **eigenvalue** of $A$ and $x \in \mathbb{C}^n$ is the corresponding **eigenvector**.

Intuitively, this definition means that multiplying $A$ by the vector $x$ results in a new vector that points in the same direction as $x$, but is scaled by a factor $\lambda$.

Also note that for any **eigenvector** $x \in \mathbb{C}^n$, and scalar $t \in \mathbb{C}, A(cx) = cAx = c\lambda x = \lambda(cx)$, so $cx$ is also an **eigenvector**. For this reason when we talk about **“the” eigenvector** associated with $\lambda$, we usually assume that the **eigenvector** is normalized to have length $1$ (this still creates some ambiguity, since $x$ and $−x$ will both be **eigenvectors**, but we will have to live with this).

For any $\lambda$ an eigenvalue of $A$ there is a vector space, the eigenspace, that corresponds to $\lambda$:

$\{ x : A x = \lambda x \}$

Any non-zero vector in this space is an eigenvector. One convenient requirement is that the eigenvector has norm $1$.

Coming back to the account example, the eigenvalues of the matrix $\begin{bmatrix}
 1.05 & 0    \\[0.3em]
 0    & 1.03
\end{bmatrix}$ are $1.05$ and $1.03$.

The eigenvector for the eigenvalue $1.05$ is $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$.  The eigenvector for the eigenvalue $1.03$ is $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$.

We can rewrite the equation above to state that $(\lambda, x)$ is an eigenvalue-eigenvector pair of $A$ if:

$(\lambda I − A)x = 0, x \neq 0$

But $(\lambda I − A)x = 0$ has a non-zero solution to $x$ if and only if $(\lambda I − A)$ has a non-empty nullspace, which is only the case if $(\lambda I − A)$ is singular:

$|(\lambda I − A)| = 0$

We can now use the previous definition of the determinant to expand this expression
into a (very large) polynomial in $\lambda$, where $\lambda$ will have maximum degree $n$.

We then find the $n$ (possibly complex) roots of this polynomial to find the $n$ eigenvalues $\lambda_1, \dots{}, \lambda_n$. To find the eigenvector corresponding to the eigenvalue $\lambda_i$, we simply solve the linear equation $(\lambda_iI − A)x = 0$. It should be noted that this is not the method which is actually used in practice to numerically compute the eigenvalues and eigenvectors (remember that the complete expansion of the determinant has $n!$ terms); it is rather a mathematical argument.

The following are properties of eigenvalues and eigenvectors (in all cases assume $A \in \mathbb{R}^{n\times n}$ has eigenvalues $\lambda_i, \dots{}, \lambda_n$ and associated eigenvectors $x_1, \dots{}, x_n$):

- The trace of a $A$ is equal to the sum of its eigenvalues,

$\mathrm{tr}A = \sum_{i=1}^n \lambda_i$
- The determinant of $A$ is equal to the product of its eigenvalues,

$|A| = \prod_{i=1}^n \lambda_i$
- The rank of $A$ is equal to the number of non-zero eigenvalues of $A$
- If $A$ is non-singular then $1/\lambda_i$ is an eigenvalue of $A^{−1}$ with associated eigenvector $x_i$, i.e., $A^{−1}x_i = (1/\lambda_i)x_i$. (To prove this, take the eigenvector equation, $Ax_i = \lambda_i x_i$ and left-multiply each side by $A^{−1}$.)
- The eigenvalues of a diagonal matrix $D = \mathrm{diag}(d_1, \dots{}, d_n)$ are just the diagonal entries $d_1, \dots{}, d_n$.

We can write all the eigenvector equations simultaneously as:

$A X = X \Lambda$

where the columns of $X \in \mathbb{R}^{n\times n}$ are the eigenvectors of $A$ and $\Lambda$ is a diagonal matrix whose entries are the eigenvalues of $A$:

$X \in \mathbb{R}^{n\times n} = \begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 x_1 & x_2 & \cdots & x_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} , \Lambda = \mathrm{diag}(\lambda_1, \dots{}, \lambda_n)$

If the eigenvectors of $A$ are linearly independent, then the matrix $X$ will be invertible, so $A = X \Lambda X^{−1}$. A matrix that can be written in this form is called **diagonalizable**.

We can compute the eigenvalues and eigenvectors in numpy in the following way:

```python
A = np.array([[ 2., 1. ],
              [ 1., 2. ]])
A
```

    array([[ 2.,  1.],
           [ 1.,  2.]])

```python
from numpy import linalg as lg
l = lg.eigvals(A)
print(l)
```

    [ 3.  1.]

We could now compute the eigenvector for each eigenvalue. See for an example the [HMC Mathematics Online Tutorial on Eigenvalues and Eigenvestors](https://www.math.hmc.edu/calculus/tutorials/eigenstuff/).

## Eigenvalues and Eigenvectors of Symmetric Matrices

Two remarkable properties come about when we look at the eigenvalues and eigenvectors
of a symmetric matrix $A \in \mathbb{S}^n$.

1. it can be shown that all the eigenvalues of $A$ are real
2. the eigenvectors of $A$ are orthonormal, i.e., the matrix $X$ defined above is an orthogonal matrix (for this reason, we denote the matrix of eigenvectors as $U$ in this case).

We can therefore represent $A$ as $A = U \Lambda U^T$, remembering from above that the inverse of an orthogonal matrix is just its transpose.

Using this, we can show that the definiteness of a matrix depends entirely on the sign of
its eigenvalues. Suppose $A \in \mathbb{S}^n = U \Lambda U^T$. Then:

$x^T A x = x^T U \Lambda U^T x = y^T \Lambda y = \sum_{i=1}^n \lambda_i y^2_i$

where $y = U^T x$ (and since $U$ is full rank, any vector $y \in \mathbb{R}^n$ can be represented in this form).

Because $y^2_i$ is always positive, the sign of this expression depends entirely on the $\lambda_i$'s. If all $\lambda_i > 0$, then the matrix is positive definite; if all $\lambda_i \leq 0$, it is positive semidefinite. Likewise, if all $\lambda_i < 0$ or $\lambda_i \leq 0$, then $A$ is negative definite or negative semidefinite respectively. Finally, if $A$ has both positive and negative eigenvalues, it is indefinite.

An application where eigenvalues and eigenvectors come up frequently is in maximizing
some function of a matrix. In particular, for a matrix $A \in \mathbb{S}^n$, consider the following maximization problem,

$\mathrm{max}_{x\in \mathbb{R}^n} x^T A x \mbox{ subject to } \|x\|^2_2 = 1$

i.e., we want to find the vector (of norm $1$) which maximizes the quadratic form. Assuming
the eigenvalues are ordered as $\lambda_1 \geq \lambda_2 \geq \dots{} \geq \lambda_n$, the optimal $x$ for this optimization problem is $x_1$, the eigenvector corresponding to $\lambda_1$. In this case the maximal value of the quadratic form is $\lambda_1$. Similarly, the optimal solution to the minimization problem,

$\mathrm{min}_{x\in \mathbb{R}^n} x^T A x \mbox{ subject to } \|x\|^2_2 = 1$

is $x_n$, the eigenvector corresponding to $\lambda_n$, and the minimal value is $\lambda_n$. This can be proved by appealing to the eigenvector-eigenvalue form of $A$ and the properties of orthogonal matrices. However, in the next section we will see a way of showing it directly using matrix calculus.

Mathematics/Linear Algebra/Linear Algebra Curated/Linear Algebra - Eigenvalues and Eigenvectors.ipynb


# Robust Data-Driven Portfolio Diversification
### Francisco A. Ibanez

1. RPCA on the sample
2. Singular Value Hard Thresholding (SVHT)
3. Truncated SVD
4. Maximize portfolio effective bets - regualization, s.t.:
    - Positivity constraint
    - Leverage 1x

The combination of (1), (2), and (3) should limit the possible permutations of the J vector when doing the spectral risk parity.

## Methodology
The goal of the overall methodology is to arrive to a portfolio weights vector which provides a well-balanced portfolio exposure to each one of the spectral risk factors present in an given investable universe.

We start with the data set $X_{T \times N}$ which containst the historical excess returns for each one of the assets that span the investable universe of the portfolio. Before performing the eigendecomposition on $X$, we need to clean the set from noisy trading observations and outliers. We apply Robust Principal Components (RPCA) on $X$ to achieve this, which seeks to decompose $X$ into a structured low-rank matrix $R$ and a sparse matrix $C$ containing outliers and corrupt data:

\begin{aligned}
X=R_0+C_0
\end{aligned}

The principal components of $R$ are robust to outliers and corrupt data in $C$. Mathematically, the goal is to find $R$ and $C$ that satisfy the following:

\begin{aligned}
\min_{R,C} ||R||_{*} + \lambda ||C||_{1} \\
\text{subject to} \\ R + C = X
\end{aligned}

```python
import pandas as pd
import numpy as np
from rpca import RobustPCA
import matplotlib.pyplot as plt
from scipy.linalg import svd
from optht import optht

raw = pd.read_pickle('etf_er.pkl')
sample = raw.dropna() # Working with even panel for now

# Outlier detection & cleaning
X = (sample - sample.mean()).div(sample.std()).values  # Normalization
t, n = X.shape
lmb = 4 / np.sqrt(max(t, n))  # Hyper-parameter
rob = RobustPCA(lmb=lmb, max_iter=int(1E6))
R, C = rob.fit(X)  # Robust, Corrupted

# Low-rank representation (compression) through hard thresholding Truncated-SVD
U, S, Vh = svd(R, full_matrices=False, compute_uv=True, lapack_driver='gesdd')
S = np.diag(S)
k = optht(X, sv=np.diag(S), sigma=None)

V = Vh.T
Vhat = V.copy()
Vhat[:, k:] = 0
Shat = S.copy()
Shat[k:, k:] = 0

cum_energy = np.cumsum(np.diag(S)) / np.sum(np.diag(S))
print(f'SVHT: {k}, {round(cum_energy[k] * 100, 2)}% of energy explained')
```

    SVHT: 8, 58.43% of energy explained

```python
D = np.diag(sample.std().values)
t, n = X.shape
w = np.array([1 / n] * n).reshape(-1, 1)
eigen_wts = V.T @ D @ w
p = np.divide(np.diag(eigen_wts.flatten()) @ S.T @ S @ eigen_wts, w.T @ D @ R.T @ R @ D @ w)
p = p.flatten()
eta_p = np.exp(-np.sum(np.multiply(p, np.log(p))))
eta_p
```

    1.4535327279694732

```python
def effective_bets(weights, singular_values_matrix, eigen_vector_matrix, volatilities, k=None):
    w = weights.reshape(-1, 1)
    eigen_wts = eigen_vector_matrix.T @ np.diag(volatilities) @ w
    p = (np.diag(eigen_wts.flatten()) @ singular_values_matrix.T @ singular_values_matrix @ eigen_wts).flatten()
    if k != None:
        p = p[:k]
    p_norm = np.divide(p, p.sum())
    eta = np.exp(-np.sum(np.multiply(p_norm, np.log(p_norm))))
    return eta

effective_bets(np.array([1 / n] * n), S, V, sample.std().values)

```

    1.4535327279694745

```python
def objfunc(weights, singular_values_matrix, eigen_vector_matrix, volatilities, k=None):
    return -effective_bets(weights, singular_values_matrix, eigen_vector_matrix, volatilities, k)

# Testing if minimizing p.T @ p yields the same results as maximizing the effective numbers of bets
def objfunc2(weights, singular_values_matrix, eigen_vector_matrix, volatilities, k=None):
    w = weights.reshape(-1, 1)
    eigen_wts = eigen_vector_matrix.T @ np.diag(volatilities) @ w
    p = np.diag(eigen_wts.flatten()) @ singular_values_matrix.T @ singular_values_matrix @ eigen_wts
    if k != None:
        p = p[:k]
    n = p.shape[0]
    p_norm = np.divide(p, p.sum())
    c = np.divide(np.ones((n, 1)), n)
    return ((p_norm - c).T @ (p_norm - c)).item()
```

```python
# POSITIVE ONLY
from scipy.optimize import minimize

cons = (
    {'type': 'ineq', 'fun': lambda x: x},
    {'type': 'ineq', 'fun': lambda x: np.sum(x) - 1}
)

opti = minimize(
    fun=objfunc,
    x0=np.array([1 / n] * n),
    args=(S, V, sample.std().values),
    constraints=cons,
    method='SLSQP',
    tol=1E-12,
    options={'maxiter': 1E9}
)

w_star = opti.x
w_star /= w_star.sum()
pd.Series(w_star, index=sample.columns).plot.bar()
print(-opti.fun)
```

```python
# UNCONSTRAINED
from scipy.optimize import minimize

cons = (
    {'type': 'ineq', 'fun': lambda x: x},
    {'type': 'ineq', 'fun': lambda x: np.sum(x) - 1}
)

opti = minimize(
    fun=objfunc,
    x0=np.array([1 / n] * n),
    args=(S, V, sample.std().values),
#    constraints=cons,
    method='SLSQP',
    tol=1E-12,
    options={'maxiter': 1E9}
)

w_star = opti.x
w_star /= w_star.sum()
pd.Series(w_star, index=sample.columns).plot.bar()
print(-opti.fun)
```

```python
eigen_wts = V.T @ np.diag(sample.std().values) @ w_star.reshape(-1, 1)
p = (np.diag(eigen_wts.flatten()) @ S.T @ S @ eigen_wts).flatten()
p = np.divide(p, p.sum())
pd.Series(p).plot.bar()
```

```python
# RC.T @ RC... different?
cons = (
    {'type': 'ineq', 'fun': lambda x: x},
    {'type': 'ineq', 'fun': lambda x: np.sum(x) - 1}
)

opti = minimize(
    fun=objfunc2,
    x0=np.array([1 / n] * n),
    args=(S, V, sample.std().values),
    constraints=cons,
    method='SLSQP',
    tol=1E-12,
    options={'maxiter': 1E9}
)

w_star = opti.x
w_star /= w_star.sum()
#pd.Series(w_star, index=sample.columns).plot.bar()
print(effective_bets(w_star, S, V, sample.std().values))
S, V, sample.std().values

eigen_wts = V.T @ np.diag(sample.std().values) @ w_star.reshape(-1, 1)
p = (np.diag(eigen_wts.flatten()) @ S.T @ S @ eigen_wts).flatten()
p = np.divide(p, p.sum())
pd.Series(p).plot.bar()
```

```python
np.array([1, 2, 3]).shape
```

    (3,)

```python

```

```python
D = np.diag(sample.std().values)
n = sample.shape[0]
Sigma = 1 / (n - 1) * D @ X.T @ X @ D
Sigma_b = 1 / (n - 1) * D @ (R + C).T @ (R + C) @ D

pd.DataFrame(R.T @ C)
pd.DataFrame(R.T @ C) + pd.DataFrame(C.T @ R)
pd.DataFrame(R.T @ R) + pd.DataFrame(C.T @ C)
```

\begin{aligned}
X &= R + C \\
R &= USV^{T}
\end{aligned}

using the Singular Value Hard Thresholding (SVHT) obtained above we can approximate $R$:
\begin{aligned}
R &\approx \tilde{U}\tilde{S}\tilde{V}^{T}
\end{aligned}

Check the algebra so everything add up and the first matrix $X$ can be recovered from this point.

\begin{aligned}
\Sigma &= \frac{1}{(n - 1)}DX^{T}XD \\
\Sigma &= \frac{1}{(n - 1)}D(R + C)^{T}(R + C))D
\end{aligned}

then, portfolio risk will be given by:
\begin{aligned}
w^{T}\Sigma w &= \frac{1}{(n - 1)}w^{T}D(R + C)^{T}(R + C))D w \\
w^{T}\Sigma w &= \frac{1}{(n - 1)}w^{T}D(R^{T}R + R^{T}C + C^{T}R + C^{T}C ) D w \\
\end{aligned}

\begin{aligned}
w^{T}\Sigma w &= \frac{1}{(n - 1)} \lbrack w^{T}D(R^{T}R)Dw + w^{T} D(R^{T}C + C^{T}R + C^{T}C ) D w \rbrack
\end{aligned}

Taking the Singular Value Decomposition of R

\begin{aligned}
R &= USV^{T} \\
\end{aligned}

we can express R in terms of its singular values and eigenvectors:

\begin{aligned}
w^{T}\Sigma w &= (n - 1)^{-1} \lbrack w^{T}D(VSU^{T}USV^{T})Dw + w^{T} D(R^{T}C + C^{T}R + C^{T}C) D w \rbrack \\
w^{T}\Sigma w &= (n - 1)^{-1} \lbrack w^{T}D(V S^{2} V^{T})Dw + w^{T} D(R^{T}C + C^{T}R + C^{T}C) D w \rbrack
\end{aligned}

where $S^{2}$ contains the eigenvalues of $R$ in its diagonal entries

\begin{aligned}
w^{T}\Sigma w &= (n - 1)^{-1} \lbrack \underbrace{w^{T}DV S^{2} V^{T}Dw}_\text{Robust Component}
+ \underbrace{w^{T} D(R^{T}C + C^{T}R + C^{T}C) D w}_\text{Noisy Component} \rbrack
\end{aligned}

TO DO: There has to be a way of reducing the noisy component, or at least interpret/explain it.

The portfolio risk contribution is then given by

\begin{aligned}
diag(w)\Sigma w &= (n - 1)^{-1} \lbrack \underbrace{\theta}_\text{Robust Component}
+ \underbrace{\gamma}_\text{Noisy Component} \rbrack \\

\theta &= diag(V^{T}Dw)S^{2} V^{T}Dw

\end{aligned}

\begin{align}
\eta (w) & \equiv \exp \left( -\sum^{N}_{n=1} p_{n} \ln{(p_{n})} \right)
\end{align}

Now we look for:
\begin{align}
\arg \max_{w} \eta(w)
\end{align}


# 符号计算

符号计算又称计算机代数,通俗地说就是用计算机推导数学公式,如对表达式进行因式分解,化简,微分,积分,解代数方程,求解常微分方程等.

符号计算主要是操作数学对象与表达式.这些数学对象与表达式可以直接表现自己,它们不是估计/近似值.表达式/对象是未经估计的变量只是以符号形式呈现.

## 使用SymPy进行符号计算

[SymPy](https://www.sympy.org/zh/index.html)是python环境下的符号计算库,他可以用于:

+ 简化数学表达式
+ 计算微分,积分与极限
+ 求方程的解
+ 矩阵运算以及各种数学函数.

所有这些功能都通过数学符号完成.
下面是使用SymPy做符号计算与一般计算的对比:

> 一般的计算

```python
import math
math.sqrt(3)
```

    1.7320508075688772

```python
math.sqrt(27)
```

    5.196152422706632

> 使用SymPy进行符号计算

```python
import sympy
sympy.sqrt(3)
```

$\displaystyle \sqrt{3}$

```python
sympy.sqrt(27)
```

$\displaystyle 3 \sqrt{3}$

SymPy程序库由若干核心能力与大量的可选模块构成.SymPy的主要功能:

+ 包括基本算术与公式简化,以及模式匹配函数,如三角函数/双曲函数/指数函数与对数函数等(核心能力)

+ 支持多项式运算,例如基本算术/因式分解以及各种其他运算(核心能力)

+ 微积分功能,包括极限/微分与积分等(核心能力)

+ 各种类型方程式的求解,例如多项式求解/方程组求解/微分方程求解(核心能力)

+ 离散数学(核心能力)

+ 矩阵表示与运算功能(核心能力)

+ 几何函数(核心能力)

+ 借助pyglet外部模块画图

+ 物理学支持

+ 统计学运算，包括概率与分布函数

+ 各种打印功能

+ LaTeX代码生成功能

## 使用SymPy的工作流

使用SymPy做符号计算不同于一般计算,它的流程是:

+ 在构建算式前申明符号,然后利用声明的符号构建算式
+ 利用算式进行推导,计算等符号运算操作
+ 输出结果

下面是一个简单的例子,就当作SymPy的helloworld吧

```python
import sympy as sp
x, y = sp.symbols('x y') #声明符号x,y
expr = x + 2*y # 构造算式
expr
```

$\displaystyle x + 2 y$

```python
expr + 1 # 在算式之上构建新算式
```

$\displaystyle x + 2 y + 1$

```python
expr + x # 新构建的算式可以明显的化简就会自动化简
```

$\displaystyle 2 x + 2 y$

```python
x*(expr) # 新算式不能明显的化简,比如这个例子,就不会自动化简
```

$\displaystyle x \left(x + 2 y\right)$

```python
expand_expr = sp.expand(x*(expr)) # 手动化简新算式
expand_expr
```

$\displaystyle x^{2} + 2 x y$

```python
sp.factor(expand_expr) #  将化简的式子做因式分解
```

$\displaystyle x \left(x + 2 y\right)$

```python
sp.latex(expand_expr) # 输出符号的latex代码
```

    'x^{2} + 2 x y'