# Μια **πρακτική**  εισαγωγή στην Αστροστατιστική
---
δηλαδή εφαρμοσμένη στατιστική με ...*εφαρμογές* στην Αστρονομία

## Που μας χρειάζεται η στατιστικη
(obvious)

### Βιβλιογραφία
1.Statistics, data mining, and machine learning in astronomy: a practical Python guide for the analysis of survey data. (Princeton University Press, 2014).

2.Wall, J. V. & Jenkins, C. R. Practical statistics for astronomers. (Cambridge University Press, 2003).

3.Chattopadhyay, A. K. & Chattopadhyay, T. Statistical Methods for Astronomical Data Analysis. 3, (Springer New York, 2014).

## Πιθανότητες και βασικές κατανομές
Παρακάτω θα εξετάσουμε κάποιες βασικές κατανομές, με μερικές πρακτικές εφαρμογές.
### Κατανομές στη Python μέσω της Scipy
Η `scipy` μας παρέχει όλες τις κατανομές μέσω του module `scipy.stats`.
Ετσι για παράδειγμα μπορούμε να δούμε διάφορες ιδιότητες/μεθόδους της κατανομής Poisson μέσω της instance `scipy.stats.poisson`.
Προσοχή, αυτό που θεωρούμε σαν κατανομή είναι στη πραγματικότητα αυτο που λέμε συνάρτηση πυκνότητας της κατανομής. Δηλαδή η συνάρτηση πυκνότητας της κατανομής Poisson μέσω της παραπάνω instance βρίσκεται από `scipy.stats.poisson.pmf()`.

Για λόγους συντομίας φορτώνουμε όλοκληρο το module `stats` ενώ ταυτόχρονα φορτώνουμε τις standard βιβλιοθήκες `numpy` και `matplotlib`

```python
from scipy import stats
from scipy import integrate
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import rc,rcParams
rc('text', usetex=True)
rcParams['figure.figsize'] = (9, 5.5)
rcParams['ytick.labelsize'],rcParams['xtick.labelsize'] = 17.,17.
rcParams['axes.labelsize']=19.
rcParams['legend.fontsize']=17.
rcParams['axes.titlesize']=20.
import seaborn as sb
sb.despine()
sb.set_style('white', {'xes.linewidth': 0.5, 'axes.edgecolor':'black'})
sb.despine(left=True)
```

    <Figure size 648x396 with 0 Axes>

### Ομοιογενής (uniform) Κατανομή

### Διωνυμική (binomial) Κατανομή
H διωνυμική κατανομή μας δείχνει τον αριθμό των επιτυχιών $k$ ενός πειράματος με δύο δυνατά αποτελέσματα (Bernouli trials) δεδομένου της πιθανότητας $p$ επιτυχίας ενός πειράματος και του συνολικού αριθμού $n$ των πειραμάτων.

Η πυκνότητα πιθανότητας δίνεται από την:
$$
Pr(k;n,p)=\binom{n}{k}p^k (1-p)^{n-k}
$$

Η πιθανότερη τιμή της κατανομής είναι:
$$
E[k]=np
$$
και η διακύμανση
$$
var[k]=np(1-p) =\sigma ^2
$$

#### Παράδειγμα 1
Ποιά είναι η κατανομή πιθανοτήτων των φορών που θα έχουμε γράμματα αν ρίξουμε ένα νόμισμα 10 φορές?

```python
kk=np.linspace(0,10,10,dtype=int)
pr=stats.binom.pmf(kk,p=0.5,n=10)
plt.bar(kk,pr)
```

#### Παράδειγμα 2
Υποθέστε ότι γνωρίζουμε από ένα δείγμα 100 γαλαξιακών σμηνών ότι σε 10 από αυτά περιέχεται ένας κυρίαρχος κεντρικός γαλαξίας. Σκοπέυουμε να ελέγξουμε ένα δεύτερο διαφορετικό δείγμα από 30 σμήνη, τα οποία έχουνε επιλεχθεί για παράδειγμα από έναν κατάλογο ακτίνων Χ.

* Σε πόσους από αυτους περιμένουμε να υπάρχει το αντίστοιχο φαινόμενο.

* Ποια η πιθανότητα να παρατήρησουμε παραπάνω από 8?

---

Υποθέτουμε ότι $p=0.1$ άρα
$$
Pr(k)=\binom{30}{k}0.1^k 0.9^{30-k}
$$
(Η κατανομή φαίνεται από το διαγραμμα παρακάτω.)

* Η πιθανότερη τιμή είναι $\mu= 3$ με τυπική απόκλιση $\sigma = 1.6$

* Η πιθανότητα να δούμε περισσότερους από 8 είναι:
$$
Pr(k>8)=\sum _{i=8}^{30} Pr(i) = 0.02
$$

Τι θα προτείνατε σε περίπτωση που παρατήρησουμε πχ 10 γαλαξίες?

```python
kk=np.linspace(0,30,30,dtype=int)
pr=stats.binom.pmf(kk,p=0.1,n=30)
plt.bar(kk,pr)
print('Pr(k>8)={:.3f}'.format(pr[kk>8].sum()))
print('Pr(k=10)={:.5f}'.format(pr[10]))
```

### Κατανομή Poisson
Η κατανομή Poisson είναι μια διακριτή κατανομή του αριθμού γεγονότων σε ένα συγκεκριμένο χρονικό διάστημα δεδομένου του μέσου αριθμού γεγονότων $\mu$ για αυτό το διάστημα. Η συνάρτηση πυκνότητας πιθανότητας είναι:
$$
Pr(x;\mu)=\frac{\mu ^x e^{-\mu}}{x!}
$$

Η πιθανότερη τιμή της κατανομής καί η διακύμανση είναι:
\begin{align}
E[x]=\mu && var[x]=\sigma ^2=\mu
\end{align}

#### Παράδειγμα
Σε έναν αγώνα ποδοσφαίρου μπαίνουν κατα μέσο όρο $2.5$ γκόλ. Ποιά είναι η πιθανότητα να μπούν $x$ γκόλ?

```python
xx=np.linspace(0,8,9,dtype=int)
#pr=stats.poisson.pmf(xx,mu=2.5)
#plt.bar(xx,pr)
for mu in np.linspace(0.5,2.5,4):
    pr=stats.poisson(mu).pmf(xx)
    plt.plot(xx,pr,label='$\mu = {:.2f}$'.format(mu))
plt.legend()
```

#### Παράδειγμα
Ο αριθμός των φωτονίων που φτάνει σε συγκεκριμένο χρονικό διάστημα σε έναν ανιχνευτή είναι σε πολύ καλή προσέγγιση (υποθέτωντας οτι η πιθανότητα εκπομπής διαδοχικών φωτονίων είναι ανεξάρτητη) είναι μια διαδικασία Poisson.

Αν υποθέσουμε $t$ το χρόνο ολοκλήρωσης του ανιχνευτή, και το ρυθμό εκπομπής φωτονίων $\lambda$ τότε η μέση τιμή των φωτονίων -και το σφάλμα- που μετράει ο ανιχνευτής είναι
\begin{align}
\mu = \lambda t && // && \sigma = \sqrt{\mu} = \sqrt{\lambda t}
\end{align}
άρα το σήμα που πέρνουμε (οι μέσες τιμες) σε σχέση με το θόρυβο θα είναι
$$
S/N \propto \sqrt{t}
$$
> Ο παραπάνω υπολογισμός δεν παίρνει υπόψιν του το θόρυβο του ανιχνευτή (readout noise). Το σφάλμα οφείλεται εξ ολοκλήρου στα φωτόνια.

The rate of a supernovae in the Milky Way is 1 every 50 yrs [citation needed]. The observed supernovae are

### Κανονική Κατανομή (Gaussian)
Η διωνυμική (για μεγάλο $N$) και η κατανομή poisson (για μεγάλο $\mu$) τείνουν στη κανονική κατανομή (ή κατανομή Gauss). Η Gauss σε σχέση με τις προηγούμενες που είδαμε είναι μια συνεχής κατανομή με δύο ανεξάρτητες παραμέτρους, τη μέση τιμή $\mu$ και τη τυπική απόκλιση $\sigma$ και έχει συνάρτηση πυκνότητας πιθανότητας:

$$
Pr(x;\mu,\sigma)=\frac{1}{\sigma \sqrt{2}}\exp \Big( -\frac{(x-\mu)^2}{2\sigma^2} \Big)
$$

```python
xx=np.linspace(-4,4,100)
for s in np.linspace(0.5,2.5,4):
    pr=stats.norm(0,s).pdf(xx)
    plt.plot(xx,pr,label='$\sigma = {:.2f}$'.format(s))
plt.legend()
```

Η Gauss είναι από ίσως η πιο σημαντική κατανομή καθώς σύμφωνα με το **Κεντρικό Οριακό Θεώρημα** οι μέσες τιμές πολλών τύχαιων μεταβλητών -ανεξαρτήτως των κατανομών που τις παρήγαγαν- κατανέμονται σαν τέτοια!

Αυτό σημαίνει οτι περιμένουμε η πλειονότητα των μετρήσεων μας να κατανέμονται σαν κανονικές κατανομές.

### Η κατανομή $x^2$
Αν τραβήξουμε ένα πλήθος $\{x_i\} ^N$ απο μια κανονική κατανομή $N(\mu,\sigma)$ τότε ορίζουμε τη ποσότητα
$$
Q = \sum _{i=1} ^N \frac{x_i-\mu}{\sigma}
$$

H Q αποδυκνύεται οτι ακολουθεί τη λεγόμενη κατανομή $x^2$ η οποία έχει σαν μοναδική παράμετρο τους βαθμούς ελευθερίας $N$.
$$
Pr(Q;N)=\frac{Q^{\frac{N}{2}-1}}{2^{N/2} \Gamma (N/2) } \exp \Big( \frac{-Q}{2} \Big)
$$

```python
xx=np.linspace(0,10)
for K in range(1,5):
    pr=stats.chi2(K).pdf(xx)
    plt.plot(xx,pr,label='K = {}'.format(K))
plt.legend()
```

### Η κατανομή t (Student's t)
Ομοίως με πριν αν τραβήξουμε  ένα πλήθος $\{x_i\} ^N$ απο μια κανονική κατανομή $N(\mu,\sigma)$ η ποσότητα
$$
t =  \frac{\bar{x}-\mu}{s/\sqrt{N}}
$$
όπου
\begin{align}
\bar{x} =\frac{1}{N}\sum _{i=1} ^N x_i && \text{και} && s=\sqrt{\frac{1}{N-1}\sum _{i=1} ^N (x_i-\bar{x})^2}
\end{align}
δηλαδή η μέση τιμή και η τυπική απόκλιση όπως τη λαμβάνουμε από τα δεδομένα.

Η κατανομή της $t$ εξαρτάται πάλι μόνο από τους βαθμούς ελευθερίας $N$:
$$
Pr(t;N)= \frac{\Gamma (\frac{N+1}{2})}{\sqrt{\pi N}\Gamma (\frac{N}{2})} \Big( 1+\frac{t^2}{N} \Big)^{-\frac{N+1}{2}}
$$

```python
xx=np.linspace(-4,4,100)
for K in range(1,5):
    pr=stats.t(K).pdf(xx)
    plt.plot(xx,pr,label='K = {}'.format(K))
plt.legend()
```

#### Παράδειγμα 3
Στο facebook γκρουπ του Μαθηματικού έγινε μια ψηφοφορία για τη κατανομή των ζωδίων των μελών του γκρουπ. Με βάση τα δεδομένα (που δίνονται παρακάτω) μπορούμε να βγάλουμε συμπεράσματα για την κατανομή γεννήσεων μέσα στο έτος?

```python
data=np.array([37,33,29,42,35,57,41,31,65,49,42,38])
data=np.array([40,35,35,46,39,60,47,34,71,54,45,42])
names=['Aqr','Psc','Ari','Tau','Gem','Cnc',
      'Leo','Vir','Lib','Sco','Sgr','Cap']
plt.bar(names,data,label='Data')
```

Κάνουμε την υπόθεση (null hypothesis) οτί η κατανομή των γεννήσεων είναι ισοπίθανη μέσα στο έτος. Άρα θα περιμένουμε ότι ο αριθμός ανθρώπων ανα ζώδιο προέρχεται από ομοιογενή κατανομή.

Η πιθανότητα να παρατηρήσουμε έναν συγκεκριμένο αριθμό ατόμων δίνεται από μια διωνυμική κατανομή με πιθανότητα $p=1/12$. Ο αριθμος των ατομων ειναι $n=548$ (27/3).
Άρα η τυπική τιμή είναι $np=45.67$ με τυπική απόκλιση $\sigma = 6.5$

```python
N=data.sum()
p=1/12
mu=N*p
var=((p)*(1-p))*N
sd=np.sqrt(var)
print('E[X] = {:.2f} / var[X] = {:.2f} (sigma = {:.2f}) for {} observations'.format(mu,var,sd,N))

plt.bar(names,data,width=0.65,label='Data')
plt.hlines(mu,0,11,linestyles='--',label='Expected Value',alpha=0.7)
plt.fill_between(range(12),(mu-sd)*np.ones(12),(mu+sd)*np.ones(12),alpha=0.65,label='Standard Deviation')
plt.legend(loc=2)
```

Γεια να εξέτασουμε αν ισχύει η υπόθεση θα χρησιμοποιήσουμε το λεγόμενο $x^2$ test του Pearson.
Τα δεδομένα μας αποτελούνται από $K=12$ διακριτές τιμές $D_i$. Θεωρώντας τα "υποτιθέμενα" δεδομένα $D_{h_i}$ τα οποία στη περίπτωση μας για ομοιογενή κατανομή είναι η τυπική τιμή $np$

Όριζουμε τη μετρική $x^2$ -για την οποία θα μιλήσουμε κι αργότερα-:
$$
x^2=\sum_{i=1} ^{K} \frac{(D_i-D_{h_i})^2}{D_{h_i}} = \sum_{i=1} ^{12} \frac{(D_i-np)^2}{np} = 30.06
$$

Στη συνέχεια συγκρίνουμε τη τιμή που μετρήσαμε με τη θεωρητική κατανομή της $x^2$ για $K-1$ βαθμούς ελευθερίας (1 βαθμός ελευθερίας λιγότερος λόγω του μοντέλου που επιλέξαμε).

Η κατανομή της $x^2$ (θα την ονομάσουμε $\text{chi2}(x)$) αντιπροσωπέυει τη πιθανότητα οι διακυμάνσεις των παρατηρήσεων από τα υποτιθέμενα δεδομένα να είναι τυχαίες. Ο υπολογισμός της πιθανότητας λοιπόν οι παραπάων παρατηρήσεις να είναι τυχαίες δίνονται από τη σχέση:
$$
P=\int _{x^2} ^{\infty} \text{chi2}(x) dx = 0.0016
$$

Άρα η πιθανότητα η κατανομή των γεννήσεων ανα μήνα να είναι ομοιογενής στο έτος σύμφωνα με τα παραπάνω δεδομένα είναι 0.16%. Επιλέγοντας μια κρίσιμη τιμή που συνήθως είναι το 5% βγάζουμε το συμπερασμα ότι η υπόθεση μας **απορρίπτεται**.

```python
x2=np.sum((N*p-data)**2/(N*p))
print('x^2 = {:.2f} / x^2 (reduced) = {:.2f}'.format(x2,x2/11))

def chi2d(x): return stats.chi2.pdf(x,11)

xx=np.linspace(0,1.5*x2,100)
plt.plot(xx,chi2d(xx))
plt.vlines(x2,0,chi2d(xx).max()/4,linestyles='--',alpha=0.4)
plt.fill_between(xx[xx>x2],0,chi2d(xx)[xx>x2],alpha=0.25)
plt.xlabel('$x^2$')

P=integrate.quad(chi2d,x2,100)[0]
Pc=0.05
print('P value = {:.4f}'.format(P))
NH = 'Accepted' if P>Pc else 'Rejected'
print('Null Hypothesis is {}'.format(NH))
```

### Παράδειγμα
Εστω οτι παρατηρούμε εναν αστέρα στον ουρανό και μετράμε τη ροή φωτονίων. Θεωρώντας ότι η ροή είναι σταθερή με το χρόνο ίση με $F_{\mathtt{true}}$.

Παίρνουμε $N$ παρατηρήσεις, μετρώντας τη ροή $F_i$ και το σφάλμα $e_i$.

Η ανίχνευση ενός φωτονίου είναι ενα ανεξάρτητο γεγονός που ακολουθεί μια τυχαία κατανομή Poisson. Από τη διακύμανση της κατανομής Poisson υπολογίζουμε το σφάλμα $e_i=\sqrt{F_i}$

```python
N=200
F_true=1000.
F=np.random.poisson(F_true*np.ones(N))
e=np.sqrt(F)

plt.errorbar(np.arange(N),F,yerr=e, fmt='ok', ecolor='gray', alpha=0.5)
plt.hlines(np.mean(F),0,N,linestyles='--')
plt.hlines(F_true,0,N)
print('Mean = {:.2f} (diff {:.2f}) // std = {:.2f}'.format(np.mean(F),np.mean(F)-F_true,np.std(F)))
```

```python
ax=sb.distplot(F)
xx=np.linspace(F.min(),F.max())
mu=F.mean()
s=F.std()
gaus=np.exp(-0.5*((xx-mu)/s)**2)/np.sqrt(2.*np.pi*s**2)
ax.plot(xx,gaus)
```

Στο παραπάνω διάγραμμα παρατηρούμε οτι οι τιμές παρότι υπακοούν σε στατιστική poisson η κατανόμη τους προσεγγίζει μια κανονική κατανομη, λόγω του Κεντρικού Οριακού Θεωρήματος.

## Εκτίμηση της ροής των φωτονίων μέσω της μέγιστης Πιθανοφάνειας (Maximum Likehood Approach)

Αυτή τη φορά αναζητούμε τις παραμέτρους του μοντέλου. Για να το κανουμε αυτό θα πρέπει να δούμε που μεγιστοποιείται η συνάρτηση πιθανοφάνειας. Δηλαδή υποθέτωντας ένα μοντέλο το οποίο κρύβεται πίσω από τα δεδομένα να υπολογίσουμε ποιά είναι η πιθανότητα να εμφανιστούν αυτά ακριβώς τα δεδομένα. Η πληροφορία αυτή βρίσκεται στη συνάρτηση πιθανοφάνειας (Likehood).

Το μοντέλο μας στη περίπτωση του αστέρα είναι ότι έχει μια σταθερή ροή $\mu$ η οποία είναι και η μοναδική παράμετρος που προσπαθούμε να υπολογίσουμε. Η πιθανοφάνεια, δηλαδή τη πιθανότητα να παρατηρηθεί η μέτρηση $D_i=(F_i,e_i)$ δεδομένου του μοντέλου σταθερής ροής είναι:
$$
P(D_i|\mu)=\frac{1}{\sqrt{2\pi e_i^2}} \exp \Big( -\frac{(F_i-\mu)^2}{2e_i^2} \Big)
$$

Ορίζουμε τη συνάρτηση πιθανοφάνειας σαν τη συνολική πιθανότητα να παρατηρηθούν σαν σύνολό οι συγκεκριμένες μετρήσεις.
$$
L(D|\mu)=\prod _{i=1}^N P(D_i|\mu) = \prod _{i=1}^N \frac{1}{\sqrt{2\pi e_i^2}} \exp \Big( -\frac{(F_i-\mu)^2}{2e_i^2} \Big)
$$

Επειδή η τιμή της συνάρτησης πιθανοφάνειας μπορεί να γίνει πολύ μικρή, είναι πιο έυκολο να χρησιμοποιήσουμε το λογάριθμο της
$$
\log L = -\frac{1}{2} \sum _{i=0}^N \big[ \log(2\pi e_i^2) + \frac{(F_i-\mu)^2}{e_i^2} \big]
$$

Αναζητούμε τώρα που αυτή μεγιστοποιείται. Άρα,
\begin{align}
\frac{d }{d\mu} \big(\log L\big) = 0 \rightarrow \mu= \frac{\sum w_i F_i}{\sum w_i} && \text{όπου} && w_i=\frac{1}{e_i^2}
\end{align}
Στη περίπτωση όπου όλα τα σφάλματα είναι ίδια (ομοσκεδαστικά(**TODO**) σφάλματα) έχουμε το αναμενόμενο αποτέλεσμα $\mu = \frac{1}{N}\sum F_i$ δηλαδή η μέση τιμή των παρατηρήσεων.

### Εκτίμηση Σφάλματος
Για τον υπολογισμό του σφάλματος κατασκευάζουμε τον Covariance Matrix (**?**) ο οποίος είναι ορίζεται από τους όρους δεύτερης τάξης της συνθήκης μεγίστου. Γενικά λοιπόν

$$
\sigma _{\jk} = \Big(-\frac{d^2 \ln L}{d\theta _j d\theta _k} \Big) ^{-\frac{1}{2}}
$$

Στη δική μας περίπτωση με μια μόνο παράμετρο δεν έχουμε πίνακα, αλλά τη τιμή:
$$
\sigma _{\mu} = \Big( \sum w_i \Big) ^{-\frac{1}{2}}
$$

---
\* Παρότι μια κατανομή Gauss ορίζεται από δύο παραμέτρους $(\mu,\sigma)$ η παράμετρος $\sigma$ αποτελεί ταυτόχρονα και το σφάλμα της παραμέτρου $\mu$. Άρα οι βαθμοί ελευθερίας του προβλήματος είναι $N-1$ (**TODO**: ανεπτυξε το)

```python
print('Ροή αστέρα: {:.1f} +/- {:.2f}'.format(np.sum(F/e**2)/np.sum(1/e**2),np.sum(1/e**2)**(-0.5)))
```

    Ροή αστέρα: 1000.4 +/- 2.24

## Τυχαίες μεταβλητές
Μια τυχαία (ή στοχαστική) μεταβλητή είναι μια μεταβλήτη η οποία προέρχεται από μια ποσότητα που υπόκεινται σε τυχαίες διακυμάνσης. Προέρχεται δηλαδή από μια στατιστική κατανομή. Υπαρχουν δύο ειδών τυχαίες μεταβλήτες, οι διακριτές (που μπορεί να προέρχονται για παράδειγμα από μια διωνυμική ή poisson κατανομή) και οι συνεχείς (gaussian αντιστοιχα)

Δύο τυχαίες μεταβλητές $x,y$ είναι ανεξάρτητες να και μόνο αν:
$$p(x,y)=p(x)p(y)$$
Στη περίπτωση όπου δύο τυχαίες μεταβλητές δεν είναι ανεξάρτητες τότε:
$$ p(x,y)=p(x|y)p(y)=p(y|x)p(x) $$
Η marginal πιθανότητα δίνεται από
$$
p(x)= \int p(x,y)dy = \int p(x|y)p(y) dy
$$
Στο παρακάτω σχήμα γίνεται εμφανές η πραγματική κατανομή δυο μή ανεξαρτήτων μεταβλητών. Από τη συνδυασμένη κατανομή παίρνουμε slices τα οποία φαίνονται στα δεξιά.

Δεν αρκεί όμως απλά να πάρουμε το slice της κατανομής. Η $p(x|1.5)$ είναι και αυτή μια κατανομή, άρα θα πρέπει συνολικά να έχει πιθανότητα 1. Άρα πρέπει να κανονικοποιήσουμε τη κατανομή, δηλαδή να διαιρέσουμε με τη marginal κατανομή της $y$ στο συγκεκριμένο slice, $$p(x|1.5)=\frac{p(x,y=1.5)}{p(y=1.5)}$$

Ο κανόνας του Bayes χτίζεται γενικεύοντας τις παραπαπάνω σχέσεις:
$$
p(y|x)=\frac{p(x|y)p(y)}{p(x)}=\frac{p(x|y)p(y)}{\int p(x|y)p(y)dy}
$$

Ο απλός αυτός κανόνας, μια απλη αγλεβρική αναδιάταξη δηλαδή, μας ανοίξε νέους ορίζοντες στο να κάνουμε στατιστική. Ας δούμε ένα απλό παράδειγμα.

Έστω ότι παρατηρούμε κάποιον -τον φώτη- να φοράει κασκόλ της αεκ, ποια η πιθανότητα να είναι οπαδός της αεκ?
Τα ενδεχόμενα είναι δύο
* ενδεχόμενο Α ο φώτης είναι αεκ
* ενδεχόμενο Α' ο φώτης δεν είναι αεκ

Αναζητούμε τη πιθανότητα δεδομένου ο φώτης να φοράει κασκολ της αεκ, να είναι και αεκ. Δηλαδή τη πιθανότητα $P(\text{αεκ}|\text{κασκολ αεκ})$.

Χρησιμοποιώντας το κανόνα του Bayes έχουμε:

$$
P(\text{αεκ}|\text{κασκολ αεκ})=
\frac{P(\text{αεκ}) P(\text{κασκολ αεκ}|\text{αεκ})}{P(\text{αεκ}) P(\text{κασκολ αεκ}|\text{αεκ})+
P(\text{οχι αεκ}) P(\text{κασκολ αεκ}|\text{οχι αεκ})}
$$

Ας μελετήσουμε τη σχέση αυτή βήμα - βήμα:

Ο όρος $P(\text{αεκ})$ αποτελεί τη πιθανότητα να είναι κάποιος ΑΕΚ γενικά. Αυτη η πιθανότητα ονομάζεται **prior** και είναι η καινοτομία (καλή ή και κακή) που μας φέρνει η bayesian προσέγγιση. Θα πρέπει λοιπόν να έχουμε μια ένδειξη της υπόθεσης μας. Θεωρώντας ότι η ΑΕΚ είναι μια γνωστή ομάδα, οπού κρατάει ένα ποσοστό κοντά αλλα χαμηλότερα στο $1/3$ στην αθήνα θεωρούμε μια τιμή $1/5$.

Ο όρος $P(\text{κασκολ ΑΕΚ}|\text{ΑΕΚ})$ αποτελεί τη πιθανότητα του να μην είναι τύχαιο κάποιος να φοράει ένα κασκολ της ΑΕΚ. Πρόκειται για τη σχέση πιθανοφάνειας ή **likehood**. Θεωρώντας ότι είναι αρκετά σπάνιο φαινόμενο αυτό δίνουμε μια τιμή $0.9$. Σημειώστε ότι χωρίς τη χρήση της bayesian προσέγγισης μάλλον η εκτίμηση μας θα ήταν αντίστοιχη αυτού του αριθμού.

Ο όρος στον παρονομάστη είναι ουσιάστικα ένας όρος κανονικοποιήσης. Στα δικά μας δεδομένα είναι έυκολο να υπολογιστεί καθώς έχουμε μόνο δύο ενδεχώμενα. Σε πραγματικά δεδομένα -όπου θέλουμε σαν αποτέλεσμα μια κατανομή- όμως είναι πολύπλοκο να υπολογιστεί. Αυτό δεν μας πειράζει όμως καθώς μπορούμε έυκολα να κάνουμε τη κανονικοποίηση χωρίς να τον υπολογίσουμε.

\begin{align}
P(\text{αεκ}) = P_0 = 0.2 \\
P(\text{όχι αεκ}) = 1-P_0 = 0.8 \\
P(\text{κασκολ αεκ}|\text{αεκ}) = 0.9 \\
P(\text{κασκολ αεκ}|\text{οχι αεκ}) = 0.1 \\
\end{align}

Η τελική πιθανότητα, ονομάζεται **posterior** υπολογίζεται λοιπόν περίπου $70\%$.

```python
0.2*0.9/(0.2*0.9+0.8*0.1)
```

    0.6923076923076924

##  Υπολογισμός Ροής μέσω της προσέγγισης Bayes
Θα δείξουμε στη συνέχεια πως μπορεί να υπολογιστεί η ροή του αστέρα του προηγούμενου παραδείγματος σύμφωνα με τη bayesian approach. Ουσιαστικά θέλουμε να υπολογίσουμε τη κατανομή πιθανοτήτων της παραμέτρου $\mu$ δεδομένου των δεδομένων $D$, ή αλλιώς $P(\mu|D)$.
Από το νόμο του bayes:

$$
P(\mu|D)=\frac{P(D|\mu)P(\mu)}{P(D)}
$$
Ας εξετάσουμε έναν έναν τους όρους
* $P(\mu|D)$: Η **posterior** κατανομή των τιμών των παραμέτρων (στη συγκεκριμένη περίπτωση έχουμε μόνο μια παράμετρο). Ο όρος που προσπαθούμε να υπολογίσουμε.
* $P(D|\mu)$: Η πιθανοφάνεια, **likehood**, ο όρος $L(D|\mu)=\prod _{i=1}^N P(D_i|\mu) = \prod _{i=1}^N \frac{1}{\sqrt{2\pi e_i^2}} \exp \Big( -\frac{(F_i-\mu)^2}{2e_i^2} \Big)$ που αναφερθήκαμε και στην κλασική προσέγγιση.
* $P(\mu)$: H **prior** κατανομή των τιμών των παραμέτρων. Εδώ χρησιμοποιείται οποιαδήποτε πρωθύστερη γνώση για τις τιμές των παραμέτρων που θέλουμε να υπολογίσουμε.
* $P(D)$: Η πιθανότητα των δεδομένων, όρος ο οποίο στη πράξη λειτουργεί σαν όρος κανονικοποίησης

```python
def log_prior(theta):
    return 1  # flat prior

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2)
                         + (F - theta[0]) ** 2 / e ** 2)

def log_posterior(theta, F, e):
    return log_prior(theta) + log_likelihood(theta, F, e)
```

```python
ndim = 1  # number of parameters in the model
nwalkers = 50  # number of MCMC walkers
nburn = 1000  # "burn-in" period to let chains stabilize
nsteps = 2000  # number of MCMC steps to take

# we'll start at random locations between 0 and 2000
starting_guesses = 2000 * np.random.rand(nwalkers, ndim)

import emcee
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F, e])
sampler.run_mcmc(starting_guesses, nsteps)

sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
sample = sampler.chain[:, nburn:, :].ravel()  # discard burn-in points
```

```python
# plot a histogram of the sample
plt.hist(sample, bins=50, histtype="stepfilled", alpha=0.3, normed=True)

# plot a best-fit Gaussian
F_fit = np.linspace(975, 1025)
pdf = stats.norm(np.mean(sample), np.std(sample)).pdf(F_fit)

plt.plot(F_fit, pdf, '-k')
plt.xlabel("F"); plt.ylabel("P(F)")
```

```python
print("""
      F_true = {0}
      F_est  = {1:.1f} +/- {2:.2f} (based on {3} measurements)
      """.format(F_true, np.mean(sample), np.std(sample), N))
```

          F_true = 1000.0
          F_est  = 1000.4 +/- 2.24 (based on 200 measurements)

**Σχόλιο:** Η μέθοδος MCMC μας δίνει σαν αποτέλεσμα μια **αριθμητική κατανομή** -όχι αναλυτική-. Επομένως οι παραπάνω τιμές είναι -λιγο- λάθος. Δεν μας είπε κανείς ότι η τελική κατανομή είναι αναγκαστικά μια κατανομή gauss. Ενα μεγάλο ζήτημα λοιπόν όταν παίρνουμε το τελικό αποτέλεσμα από τον MCMC αλγόριθμο είναι η κατανόηση και εκτίμηση κάποιον μετρικών για την εκάστοτε παράμετρο, όπως πιθανότερη τιμή και σφάλμα.

## Περίπτωση ενός αστέρα με στοχαστική ροή.
Ας εξετάσουμε τώρα τη περίτπωση όπου η ροή ενός αστέρα είναι τυχαία εκ φύσεως σαν μια κανονική κατανομή. Δηλαδή
$$
F_{\text{true}}=\frac{1}{\sqrt{2\pi \sigma^2}} \exp \Big( -\frac{(F-\mu)^2}{2\sigma^2} \Big)
$$

```python
np.random.seed(42)  # for reproducibility
N = 100  # we'll use more samples for the more complicated model
mu_true, sigma_true = 1000, 15  # stochastic flux model

F_true = stats.norm(mu_true, sigma_true).rvs(N)  # (unknown) true flux
F = stats.poisson(F_true).rvs()  # observed flux: true flux plus Poisson errors.
e = np.sqrt(F)  # root-N error, as above

plt.errorbar(np.arange(N),F,yerr=e, fmt='ok', ecolor='gray', alpha=0.5)
plt.hlines(mu_true,0,N,linestyles='--')
plt.hlines(mu_true+sigma_true,0,N,linestyles='--',alpha=0.6)
plt.hlines(mu_true-sigma_true,0,N,linestyles='--',alpha=0.6)
```

Η συνάρτηση πιθανοφάνειας βρίσκεται από τη συσχέτιση της κατανομής των σφαλμάτων με τη κατανομή της πηγής των μετρήσεων
$$
L(D|\mu,\sigma)=\prod _{i=1}^N \frac{1}{\sqrt{2\pi (e_i^2+\sigma^2)}} \exp \Big( -\frac{(F_i-\mu)^2}{2(e_i^2+\sigma ^2)} \Big)
$$

Αντίστοιχα με πριν υπολογίζουμε:
\begin{align}
\mu=\frac{\sum w_i F_i}{\sum w_i} && \text{όπου} &&  w_i=\frac{1}{\sigma ^2+e_i ^2}
\end{align}
Και εδώ υπάρχει πρόβλημα. Καθώς η τιμή της παραμέτρου $\mu$ εξαρτάται από τη παράμετρο $\sigma$.

Εφόσον δεν υπάρχει αναλυτική λύση θα καταφύγουμε σε έναν αριθμητικό υπολογισμό της μέγιστης τιμής της $L$

```python
def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * (theta[1] ** 2 + e ** 2))
                         + (F - theta[0]) ** 2 / (theta[1] ** 2 + e ** 2))

# maximize likelihood <--> minimize negative likelihood
def neg_log_likelihood(theta, F, e):
    return -log_likelihood(theta, F, e)

from scipy import optimize
theta_guess = [900, 5]
theta_est = optimize.fmin(neg_log_likelihood, theta_guess, args=(F, e))
print("""
      Maximum likelihood estimate for {0} data points:
          mu={theta[0]:.0f}, sigma={theta[1]:.0f}
      """.format(N, theta=theta_est))
```

    Optimization terminated successfully.
             Current function value: 502.839505
             Iterations: 58
             Function evaluations: 114

          Maximum likelihood estimate for 100 data points:
              mu=999, sigma=19

Όλα καλά εως εδώ. Ετσι κι αλλιώς σε πολύ λίγες περιπτώσεις μπορούμε να έχουμε αναλυτική επίλυση, άρα δεν κάναμε και κάποιο έγκλημα!

Όμως αυτό που δεν είναι προφανές πως θα υπολογίσουμε είναι το σφάλμα των τιμών $\mu,\sigma$. Υπάρχουν τρόποι που μπορούμε να το καταφέρουμε αυτό, είτε μέσω κάποιου τέστ $x^2$, είτε να αναζητήσουμε μια κανονική κατανομή στη μέγιστη τιμή της $L$. Και οι δύο προσεγγίσεις βασίζονται στην εκτίμηση όμως οτι η τελική κατανομή των τιμών των παραμέτρων είναι κανονική.

```python
def log_prior(theta):
    # sigma needs to be positive.
    if theta[1] <= 0:
        return -np.inf
    else:
        return 0

def log_posterior(theta, F, e):
    return log_prior(theta) + log_likelihood(theta, F, e)

# same setup as above:
ndim, nwalkers = 2, 150
nsteps, nburn = 2000, 1000

starting_guesses = np.random.rand(nwalkers, ndim)
starting_guesses[:, 0] *= 2000  # start mu between 0 and 2000
starting_guesses[:, 1] *= 20    # start sigma between 0 and 20

sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F, e])
sampler.run_mcmc(starting_guesses, nsteps)

#sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
trace = sampler.chain[:, nburn:, :].reshape(-1, ndim)
```

```python
import corner
```

```python
rcParams['figure.figsize'] = (15, 15)
co=corner.corner(sampler.flatchain, labels=['mu','sigma'],truths=[mu_true,sigma_true],smooth=0.25,bins=50,)
```

```python

```


[Table of Contents](http://nbviewer.ipython.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/table_of_contents.ipynb)

# Kalman Filter Math

```python
#format the book
%matplotlib inline
from __future__ import division, print_function
from book_format import load_style
load_style()
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Lora');

//@import url('http://fonts.googleapis.com/css?family=Open+Sans');
//@import url('http://fonts.googleapis.com/css?family=Vollkorn');
//@import url('http://fonts.googleapis.com/css?family=Karla');
//@import url('http://fonts.googleapis.com/css?family=Poppins');
//@import url('http://fonts.googleapis.com/css?family=Arimo');
//@import url('http://fonts.googleapis.com/css?family=Roboto');
//@import url('http://fonts.googleapis.com/css?family=Lato');
//@import url('http://fonts.googleapis.com/css?family=Domine');
//@import url('http://fonts.googleapis.com/css?family=Chivo');
//@import url('http://fonts.googleapis.com/css?family=Cardo');
//@import url('http://fonts.googleapis.com/css?family=Arvo');
//@import url('http://fonts.googleapis.com/css?family=Crimson+Text');
//@import url('http://fonts.googleapis.com/css?family=Ubuntu');
//@import url('http://fonts.googleapis.com/css?family=Fontin');
//@import url('http://fonts.googleapis.com/css?family=Raleway');
//@import url('http://fonts.googleapis.com/css?family=Merriweather');

.CodeMirror pre {
    font-family: 'Source Code Pro', Consolas, monocco, monospace;
}
    div.cell{
        //width: 950px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell_render{
        font-family: 'Lora';
        //font-family: 'Open Sans';
        //font-family: 'Karla',verdana,arial,sans-serif;
        //font-family: 'Roboto',verdana,arial,sans-serif;
        //font-family: 'Lato',verdana,arial,sans-serif;
        //font-family: 'Domine',verdana,arial,sans-serif;
        //font-family: 'Chivo',verdana,arial,sans-serif;
        //font-family: 'Cardo',verdana,arial,sans-serif;
        //font-family: 'Arvo',verdana,arial,sans-serif;
        //font-family: 'Poppins',verdana,arial,sans-serif;
        //font-family: 'Ubuntu',verdana,arial,sans-serif;
        //font-family: 'Fontin',verdana,arial,sans-serif;
        //font-family: 'Raleway',verdana,arial,sans-serif;
        //font-family: 'Merriweather',verdana,arial,sans-serif;
        //font-family: 'Crimson Text', verdana,arial,sans-serif;
        //font-family: verdana,arial,sans-serif;
        //font-family: arial,sans-serif;
        line-height: 125%;
        font-size: 130%;
        text-align: justify;
        text-justify:inter-word;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 400;
        font-size: 12pt;
        //font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
   }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }

    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.output_subarea.output_text.output_pyout {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_subarea.output_stream.output_stdout.output_text {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
}

    code{
        font-size: 6pt;

    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       color:#447adb;
    }
    a:visited{
       color: #1d3b84;
    }
    a:hover{
       color: #1d3b84;
    }
    a:focus{
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

If you've gotten this far I hope that you are thinking that the Kalman filter's fearsome reputation is somewhat undeserved. Sure, I hand waved some equations away, but I hope implementation has been fairly straightforward for you. The underlying concept is quite straightforward - take two measurements, or a measurement and a prediction, and choose the output to be somewhere between the two. If you believe the measurement more your guess will be closer to the measurement, and if you believe the prediction is more accurate your guess will lie closer it it. That's not rocket science (little joke - it is exactly this math that got Apollo to the moon and back!).

To be honest I have been choosing my problems carefully. For an arbitrary problem designing the Kalman filter matrices can be extremely difficult. I haven't been *too tricky*, though. Equations like Newton's equations of motion can be trivially computed for Kalman filter applications, and they make up the bulk of the kind of problems that we want to solve.

I have illustrated the concepts with code and reasoning, not math. But there are topics that do require more mathematics than I have used so far. This chapter presents the math that you will need for the rest of the book.

## Modeling a Dynamic System

A *dynamic system* is a physical system whose state (position, temperature, etc) evolves over time. Calculus is the math of changing values, so we use differential equations to model dynamic systems. Some systems cannot be modeled with differential equations, but we will not encounter those in this book.

Modeling dynamic systems is properly the topic of several college courses. To an extent there is no substitute for a few semesters of ordinary and partial differential equations followed by a graduate course in control system theory. If you are a hobbyist, or trying to solve one very specific filtering problem at work you probably do not have the time and/or inclination to devote a year or more to that education.

Fortunately, I can present enough of the theory to allow us to create the system equations for many different Kalman filters. My goal is to get you to the stage where you can read a publication and understand it well enough to implement the algorithms. The background math is deep, but in practice we end up using a few simple techniques.

This is the longest section of pure math in this book. You will need to master everything in this section to understand the Extended Kalman filter (EKF), the most common nonlinear filter. I do cover more modern filters that do not require as much of this math. You can choose to skim now, and come back to this if you decide to learn the EKF.

We need to start by understanding the underlying equations and assumptions that the Kalman filter uses. We are trying to model real world phenomena, so what do we have to consider?

Each physical system has a process. For example, a car traveling at a certain velocity goes so far in a fixed amount of time, and its velocity varies as a function of its acceleration. We describe that behavior with the well known Newtonian equations that we learned in high school.

$$
\begin{aligned}
v&=at\\
x &= \frac{1}{2}at^2 + v_0t + x_0
\end{aligned}
$$

Once we learned calculus we saw them in this form:

$$ \mathbf v = \frac{d \mathbf x}{d t},
\quad \mathbf a = \frac{d \mathbf v}{d t} = \frac{d^2 \mathbf x}{d t^2}
$$

A typical automobile tracking problem would have you compute the distance traveled given a constant velocity or acceleration, as we did in previous chapters. But, of course we know this is not all that is happening. No car travels on a perfect road. There are bumps, wind drag, and hills that raise and lower the speed. The suspension is a mechanical system with friction and imperfect springs.

Perfectly modeling a system is impossible except for the most trivial problems. We are forced to make a simplification. At any time $t$ we say that the true state (such as the position of our car) is the predicted value from the imperfect model plus some unknown *process noise*:

$$
x(t) = x_{pred}(t) + noise(t)
$$

This is not meant to imply that $noise(t)$ is a function that we can derive analytically. It is merely a statement of fact - we can always describe the true value as the predicted value  plus the process noise. "Noise" does not imply random events. If we are tracking a thrown ball in the atmosphere, and our model assumes the ball is in a vacuum, then the effect of air drag is process noise in this context.

In the next section we will learn techniques to convert a set of higher order differential equations into a set of first-order differential equations.  After the conversion the model of the system without noise is:

$$ \dot{\mathbf x} = \mathbf{Ax}$$

$\mathbf A$ is known as the *systems dynamics matrix* as it describes the dynamics of the system. Now we need to model the noise. We will call that $\mathbf w$, and add it to the equation.

$$ \dot{\mathbf x} = \mathbf{Ax} + \mathbf w$$

$\mathbf w$ may strike you as a poor choice for the name, but you will soon see that the Kalman filter assumes *white* noise.

Finally, we need to consider any inputs into the system. We assume an input $\mathbf u$, and that there exists a linear model that defines how that input changes the system. For example, pressing the accelerator in your car makes it accelerate, and gravity causes balls to fall. Both are contol inputs. We will need a matrix $\mathbf B$ to convert $u$ into the effect on the system. We add that into our equation:

$$ \dot{\mathbf x} = \mathbf{Ax} + \mathbf{Bu} + \mathbf{w}$$

And that's it. That is one of the equations that Dr. Kalman set out to solve, and he found an optimal estimator if we assume certain properties of $\mathbf w$.

## State-Space Representation of Dynamic Systems

We've derived the equation

$$ \dot{\mathbf x} = \mathbf{Ax}+ \mathbf{Bu} + \mathbf{w}$$

However, we are not interested in the derivative of $\mathbf x$, but in $\mathbf x$ itself. Ignoring the noise for a moment, we want an equation that recusively finds the value of $\mathbf x$ at time $t_k$ in terms of $\mathbf x$ at time $t_{k-1}$:

$$\mathbf x(t_k) = \mathbf F(\Delta t)\mathbf x(t_{k-1}) + \mathbf B(t_k) + \mathbf u (t_k)$$

Convention allows us to write $\mathbf x(t_k)$ as $\mathbf x_k$, which means the
the value of $\mathbf x$ at the k$^{th}$ value of $t$.

$$\mathbf x_k = \mathbf{Fx}_{k-1} + \mathbf B_k\mathbf u_k$$

$\mathbf F$ is the familiar *state transition matrix*, named due to its ability to transition the state's value between discrete time steps. It is very similar to the system dynamics matrix $\mathbf A$. The difference is that $\mathbf A$ models a set of linear differential equations, and is continuous. $\mathbf F$ is discrete, and represents a set of linear equations (not differential equations) which transitions $\mathbf x_{k-1}$ to $\mathbf x_k$ over a discrete time step $\Delta t$.

Finding this matrix is often quite difficult. The equation $\dot x = v$ is the simplest possible differential equation and we trivially integrate it as:

$$ \int\limits_{x_{k-1}}^{x_k}  \mathrm{d}x = \int\limits_{0}^{\Delta t} v\, \mathrm{d}t $$
$$x_k-x_0 = v \Delta t$$
$$x_k = v \Delta t + x_0$$

This equation is *recursive*: we compute the value of $x$ at time $t$ based on its value at time $t-1$. This recursive form enables us to represent the system (process model) in the form required by the Kalman filter:

$$\begin{aligned}
\mathbf x_k &= \mathbf{Fx}_{k-1}  \\
&= \begin{bmatrix} 1 & \Delta t \\ 0 & 1\end{bmatrix}
\begin{bmatrix}x_{k-1} \\ \dot x_{k-1}\end{bmatrix}
\end{aligned}$$

We can do that only because $\dot x = v$ is simplest differential equation possible. Almost all other in physical systems result in more complicated differential equation which do not yield to this approach.

*State-space* methods became popular around the time of the Apollo missions, largely due to the work of Dr. Kalman. The idea is simple. Model a system with a set of  $n^{th}$-order differential equations. Convert them into an equivalent set of first-order differential equations. Put them into the vector-matrix form used in the previous section: $\dot{\mathbf x} = \mathbf{Ax} + \mathbf{Bu}$. Once in this form we use of of several techniques to convert these linear differential equations into the recursive equation:

$$ \mathbf x_k = \mathbf{Fx}_{k-1} + \mathbf B_k\mathbf u_k$$

Some books call the state transition matrix the *fundamental matrix*. Many use $\mathbf \Phi$ instead of $\mathbf F$. Sources based heavily on control theory tend to use these forms.

These are called *state-space* methods because we are expressing the solution of the differential equations in terms of the system state.

### Forming First Order Equations from Higher Order Equations

Many models of physical systems require second or higher order differential equations with control input $u$:

$$a_n \frac{d^ny}{dt^n} + a_{n-1} \frac{d^{n-1}y}{dt^{n-1}} +  \dots + a_2 \frac{d^2y}{dt^2} + a_1 \frac{dy}{dt} + a_0 = u$$

State-space methods require first-order equations. Any higher order system of equations can be reduced to first-order by defining extra variables for the derivatives and then solving.

Let's do an example. Given the system $\ddot{x} - 6\dot x + 9x = u$ find the equivalent first order equations. I've used the dot notation for the time derivatives for clarity.

The first step is to isolate the highest order term onto one side of the equation.

$$\ddot{x} = 6\dot x - 9x + u$$

We define two new variables:

$$\begin{aligned} x_1(u) &= x \\
x_2(u) &= \dot x
\end{aligned}$$

Now we will substitute these into the original equation and solve. The solution yields a set of first-order equations in terms of these new variables. It is conventional to drop the $(u)$ for notational convenience.

We know that $\dot x_1 = x_2$ and that $\dot x_2 = \ddot{x}$. Therefore

$$\begin{aligned}
\dot x_2 &= \ddot{x} \\
         &= 6\dot x - 9x + t\\
         &= 6x_2-9x_1 + t
\end{aligned}$$

Therefore our first-order system of equations is

$$\begin{aligned}\dot x_1 &= x_2 \\
\dot x_2 &= 6x_2-9x_1 + t\end{aligned}$$

If you practice this a bit you will become adept at it. Isolate the highest term, define a new variable and its derivatives, and then substitute.

### First Order Differential Equations In State-Space Form

Substituting the newly defined variables from the previous section:

$$\frac{dx_1}{dt} = x_2,\,
\frac{dx_2}{dt} = x_3, \, ..., \,
\frac{dx_{n-1}}{dt} = x_n$$

into the first order equations yields:

$$\frac{dx_n}{dt} = \frac{1}{a_n}\sum\limits_{i=0}^{n-1}a_ix_{i+1} + \frac{1}{a_n}u
$$

Using vector-matrix notation we have:

$$\begin{bmatrix}\frac{dx_1}{dt} \\ \frac{dx_2}{dt} \\ \vdots \\ \frac{dx_n}{dt}\end{bmatrix} =
\begin{bmatrix}\dot x_1 \\ \dot x_2 \\ \vdots \\ \dot x_n\end{bmatrix}=
\begin{bmatrix}0 & 1 & 0 &\cdots & 0 \\
0 & 0 & 1 & \cdots & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
-\frac{a_0}{a_n} & -\frac{a_1}{a_n} & -\frac{a_2}{a_n} & \cdots & -\frac{a_{n-1}}{a_n}\end{bmatrix}
\begin{bmatrix}x_1 \\ x_2 \\ \vdots \\ x_n\end{bmatrix} +
\begin{bmatrix}0 \\ 0 \\ \vdots \\ \frac{1}{a_n}\end{bmatrix}u$$

which we then write as $\dot{\mathbf x} = \mathbf{Ax} + \mathbf{B}u$.

### Finding the Fundamental Matrix for Time Invariant Systems

We express the system equations in state-space form with

$$ \dot{\mathbf x} = \mathbf{Ax}$$

where $\mathbf A$ is the system dynamics matrix, and want to find the *fundamental matrix* $\mathbf F$ that propagates the state $\mathbf x$ over the interval $\Delta t$ with the equation

$$\begin{aligned}
\mathbf x(t_k) = \mathbf F(\Delta t)\mathbf x(t_{k-1})\end{aligned}$$

In other words, $\mathbf A$ is a set of continuous differential equations, and we need $\mathbf F$ to be a set of discrete linear equations that computes the change in $\mathbf A$ over a discrete time step.

It is conventional to drop the $t_k$ and $(\Delta t)$ and use the notation

$$\mathbf x_k = \mathbf {Fx}_{k-1}$$

Broadly speaking there are three common ways to find this matrix for Kalman filters. The technique most often used is the matrix exponential. Linear Time Invariant Theory, also known as LTI System Theory, is a second technique. Finally, there are numerical techniques. You may know of others, but these three are what you will most likely encounter in the Kalman filter literature and praxis.

### The Matrix Exponential

The solution to the equation $\frac{dx}{dt} = kx$ can be found by:

$$\begin{gathered}\frac{dx}{dt} = kx \\
\frac{dx}{x} = k\, dt \\
\int \frac{1}{x}\, dx = \int k\, dt \\
\log x = kt + c \\
x = e^{kt+c} \\
x = e^ce^{kt} \\
x = c_0e^{kt}\end{gathered}$$

Using similar math, the solution to the first-order equation

$$\dot{\mathbf x} = \mathbf{Ax} ,\, \, \, \mathbf x(0) = \mathbf x_0$$

where $\mathbf A$ is a constant matrix, is

$$\mathbf x = e^{\mathbf At}\mathbf x_0$$

Substituting $F = e^{\mathbf At}$, we can write

$$\mathbf x_k = \mathbf F\mathbf x_{k-1}$$

which is the form we are looking for! We have reduced the problem of finding the fundamental matrix to one of finding the value for $e^{\mathbf At}$.

$e^{\mathbf At}$ is known as the [matrix exponential](https://en.wikipedia.org/wiki/Matrix_exponential). It can be computed with this power series:

$$e^{\mathbf At} = \mathbf{I} + \mathbf{A}t  + \frac{(\mathbf{A}t)^2}{2!} + \frac{(\mathbf{A}t)^3}{3!} + ... $$

That series is found by doing a Taylor series expansion of $e^{\mathbf At}$, which I will not cover here.

Let's use this to find the solution to Newton's equations. Using $v$ as an substitution for $\dot x$, and assuming constant velocity we get the linear matrix-vector form

$$\begin{bmatrix}\dot x \\ \dot v\end{bmatrix} =\begin{bmatrix}0&1\\0&0\end{bmatrix} \begin{bmatrix}x \\ v\end{bmatrix}$$

This is a first order differential equation, so we can set $\mathbf{A}=\begin{bmatrix}0&1\\0&0\end{bmatrix}$ and solve the following equation. I have substituted the interval $\Delta t$ for $t$ to emphasize that the fundamental matrix is discrete:

$$\mathbf F = e^{\mathbf A\Delta t} = \mathbf{I} + \mathbf A\Delta t  + \frac{(\mathbf A\Delta t)^2}{2!} + \frac{(\mathbf A\Delta t)^3}{3!} + ... $$

If you perform the multiplication you will find that $\mathbf{A}^2=\begin{bmatrix}0&0\\0&0\end{bmatrix}$, which means that all higher powers of $\mathbf{A}$ are also $\mathbf{0}$. Thus we get an exact answer without an infinite number of terms:

$$
\begin{aligned}
\mathbf F &=\mathbf{I} + \mathbf A \Delta t + \mathbf{0} \\
&= \begin{bmatrix}1&0\\0&1\end{bmatrix} + \begin{bmatrix}0&1\\0&0\end{bmatrix}\Delta t\\
&= \begin{bmatrix}1&\Delta t\\0&1\end{bmatrix}
\end{aligned}$$

We plug this into $\mathbf x_k= \mathbf{Fx}_{k-1}$ to get

$$
\begin{aligned}
x_k &=\begin{bmatrix}1&\Delta t\\0&1\end{bmatrix}x_{k-1}
\end{aligned}$$

You will recognize this as the matrix we derived analytically for the constant velocity Kalman filter in the **Multivariate Kalman Filter** chapter.

SciPy's linalg module includes a routine `expm()` to compute the matrix exponential. It does not use the Taylor series method, but the [Padé Approximation](https://en.wikipedia.org/wiki/Pad%C3%A9_approximant). There are many (at least 19) methods to computed the matrix exponential, and all suffer from numerical difficulties[1]. But you should be aware of the problems, especially when $\mathbf A$ is large. If you search for "pade approximation matrix exponential" you will find many publications devoted to this problem.

In practice this may not be of concern to you as for the Kalman filter we normally just take the first two terms of the Taylor series. But don't assume my treatment of the problem is complete and run off and try to use this technique for other problem without doing a numerical analysis of the performance of this technique. Interestingly, one of the favored ways of solving $e^{\mathbf At}$ is to use a generalized ode solver. In other words, they do the opposite of what we do - turn $\mathbf A$ into a set of differential equations, and then solve that set using numerical techniques!

Here is an example of using `expm()` to solve $e^{\mathbf At}$.

```python
import numpy as np
from scipy.linalg import expm

dt = 0.1
A = np.array([[0, 1],
              [0, 0]])
expm(A*dt)
```

    array([[ 1.0,  0.1],
           [ 0.0,  1.0]])

### Time Invariance

If the behavior of the system depends on time we can say that a dynamic system is described by the first-order differential equation

$$ g(t) = \dot x$$

However, if the system is *time invariant* the equation is of the form:

$$ f(x) = \dot x$$

What does *time invariant* mean? Consider a home stereo. If you input a signal $x$ into it at time $t$, it will output some signal $f(x)$. If you instead perform the input at time $t + \Delta t$ the output signal will be the same $f(x)$, shifted in time.

A counter-example is $x(t) = \sin(t)$, with the system  $f(x) = t\,  x(t) = t \sin(t)$. This is not time invariant; the value will be different at different times due to the multiplication by t.  An aircraft is not time invariant. If you make a control input to the aircraft at a later time its behavior will be different because it will have burned fuel and thus lost weight. Lower weight results in different behavior.

We can solve these equations by integrating each side. I demonstrated integrating the time invariant system $v = \dot x$ above. However, integrating the time invariant equation $\dot x = f(x)$ is not so straightforward. Using the *separation of variables* techniques we divide by $f(x)$ and move the $dt$ term to the right so we can integrate each side:

$$\begin{gathered}
\frac{dx}{dt} = f(x) \\
\int^x_{x_0} \frac{1}{f(x)} dx = \int^t_{t_0} dt
\end{gathered}$$

If we let $F(x) = \int \frac{1}{f(x)} dx$ we get

$$F(x) - F(x_0) = t-t_0$$

We then solve for x with

$$\begin{gathered}
F(x) = t - t_0 + F(x_0) \\
x = F^{-1}[t-t_0 + F(x_0)]
\end{gathered}$$

In other words, we need to find the inverse of $F$. This is not trivial, and a significant amount of coursework in a STEM education is devoted to finding tricky, analytic solutions to this problem.

However, they are tricks, and many simple forms of $f(x)$ either have no closed form solution or pose extreme difficulties. Instead, the practicing engineer turns to state-space methods to find approximate solutions.

The advantage of the matrix exponential is that we can use it for any arbitrary set of differential equations which are *time invariant*. However, we often use this technique even when the equations are not time invariant. As an aircraft flies it burns fuel and loses weight. However, the weight loss over one second is negligible, and so the system is nearly linear over that time step. Our answers will still be reasonably accurate so long as the time step is short.

#### Example: Mass-Spring-Damper Model

Suppose we wanted to track the motion of a weight on a spring and connected to a damper, such as an automobile's suspension. The equation for the motion with $m$ being the mass, $k$ the spring constant, and $c$ the damping force, under some input $u$ is

$$m\frac{d^2x}{dt^2} + c\frac{dx}{dt} +kx = u$$

For notational convenience I will write that as

$$m\ddot x + c\dot x + kx = u$$

I can turn this into a system of first order equations by setting $x_1(t)=x(t)$, and then substituting as follows:

$$\begin{aligned}
x_1 &= x \\
x_2 &= \dot x_1 \\
\dot x_2 &= \dot x_1 = \ddot x
\end{aligned}$$

As is common I dropped the $(t)$ for notational convenience. This gives the equation

$$m\dot x_2 + c x_2 +kx_1 = u$$

Solving for $\dot x_2$ we get a first order equation:

$$\dot x_2 = -\frac{c}{m}x_2 - \frac{k}{m}x_1 + \frac{1}{m}u$$

We put this into matrix form:

$$\begin{bmatrix} \dot x_1 \\ \dot x_2 \end{bmatrix} =
\begin{bmatrix}0 & 1 \\ -k/m & -c/m \end{bmatrix}
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} +
\begin{bmatrix} 0 \\ 1/m \end{bmatrix}u$$

Now we use the matrix exponential to find the state transition matrix:

$$\Phi(t) = e^{\mathbf At} = \mathbf{I} + \mathbf At  + \frac{(\mathbf At)^2}{2!} + \frac{(\mathbf At)^3}{3!} + ... $$

The first two terms give us

$$\mathbf F = \begin{bmatrix}1 & t \\ -(k/m) t & 1-(c/m) t \end{bmatrix}$$

This may or may not give you enough precision. You can easily check this by computing $\frac{(\mathbf At)^2}{2!}$ for your constants and seeing how much this matrix contributes to the results.

### Linear Time Invariant Theory

[*Linear Time Invariant Theory*](https://en.wikipedia.org/wiki/LTI_system_theory), also known as LTI System Theory, gives us a way to find $\Phi$ using the inverse Laplace transform. You are either nodding your head now, or completely lost.  I will not be using the Laplace transform in this book. LTI system theory tells us that

$$ \Phi(t) = \mathcal{L}^{-1}[(s\mathbf{I} - \mathbf{F})^{-1}]$$

I have no intention of going into this other than to say that the Laplace transform $\mathcal{L}$ converts a signal into a space $s$ that excludes time, but finding a solution to the equation above is non-trivial. If you are interested, the Wikipedia article on LTI system theory provides an introduction. I mention LTI because you will find some literature using it to design the Kalman filter matrices for difficult problems.

### Numerical Solutions

Finally, there are numerical techniques to find $\mathbf F$. As filters get larger finding analytical solutions becomes very tedious (though packages like SymPy make it easier). C. F. van Loan [2] has developed a technique that finds both $\Phi$ and $\mathbf Q$ numerically. Given the continuous model

$$ \dot x = Ax + Gw$$

where $w$ is the unity white noise, van Loan's method computes both $\mathbf F_k$ and $\mathbf Q_k$.

I have implemented van Loan's method in `FilterPy`. You may use it as follows:

```python
from filterpy.common import van_loan_discretization

A = np.array([[0., 1.], [-1., 0.]])
G = np.array([[0.], [2.]]) # white noise scaling
F, Q = van_loan_discretization(A, G, dt=0.1)
```

In the section *Numeric Integration of Differential Equations* I present alternative methods which are very commonly used in Kalman filtering.

## Design of the Process Noise Matrix

In general the design of the $\mathbf Q$ matrix is among the most difficult aspects of Kalman filter design. This is due to several factors. First, the math requires a good foundation in signal theory. Second, we are trying to model the noise in something for which we have little information. Consider trying to model the process noise for a thrown baseball. We can model it as a sphere moving through the air, but that leave many unknown factors - the wind, ball rotation and spin decay, the coefficient of drag of a ball with stitches, the effects of wind and air density, and so on. We develop the equations for an exact mathematical solution for a given process model, but since the process model is incomplete the result for $\mathbf Q$ will also be incomplete. This has a lot of ramifications for the behavior of the Kalman filter. If $\mathbf Q$ is too small then the filter will be overconfident in its prediction model and will diverge from the actual solution. If $\mathbf Q$ is too large than the filter will be unduly influenced by the noise in the measurements and perform sub-optimally. In practice we spend a lot of time running simulations and evaluating collected data to try to select an appropriate value for $\mathbf Q$. But let's start by looking at the math.

Let's assume a kinematic system - some system that can be modeled using Newton's equations of motion. We can make a few different assumptions about this process.

We have been using a process model of

$$ \dot{\mathbf x} = \mathbf{Ax} + \mathbf{Bu} + \mathbf{w}$$

where $\mathbf{w}$ is the process noise. Kinematic systems are *continuous* - their inputs and outputs can vary at any arbitrary point in time. However, our Kalman filters are *discrete* (there are continuous forms for Kalman filters, but we do not cover them in this book). We sample the system at regular intervals. Therefore we must find the discrete representation for the noise term in the equation above. This depends on what assumptions we make about the behavior of the noise. We will consider two different models for the noise.

### Continuous White Noise Model

We model kinematic systems using Newton's equations. We have either used position and velocity, or position, velocity, and acceleration as the models for our systems. There is nothing stopping us from going further - we can model jerk, jounce, snap, and so on. We don't do that normally because adding terms beyond the dynamics of the real system degrades the estimate.

Let's say that we need to model the position, velocity, and acceleration. We can then assume that acceleration is constant for each discrete time step. Of course, there is process noise in the system and so the acceleration is not actually constant. The tracked object will alter the acceleration over time due to external, unmodeled forces. In this section we will assume that the acceleration changes by a continuous time zero-mean white noise $w(t)$. In other words, we are assuming that the small changes in velocity average to 0 over time (zero-mean).

Since the noise is changing continuously we will need to integrate to get the discrete noise for the discretization interval that we have chosen. We will not prove it here, but the equation for the discretization of the noise is

$$\mathbf Q = \int_0^{\Delta t} \mathbf F(t)\mathbf{Q_c}\mathbf F^\mathsf{T}(t) dt$$

where $\mathbf{Q_c}$ is the continuous noise. This gives us

$$\Phi = \begin{bmatrix}1 & \Delta t & {\Delta t}^2/2 \\ 0 & 1 & \Delta t\\ 0& 0& 1\end{bmatrix}$$

for the fundamental matrix, and

$$\mathbf{Q_c} = \begin{bmatrix}0&0&0\\0&0&0\\0&0&1\end{bmatrix} \Phi_s$$

for the continuous process noise matrix, where $\Phi_s$ is the spectral density of the white noise.

We could carry out these computations ourselves, but I prefer using SymPy to solve the equation.

$$\mathbf{Q_c} = \begin{bmatrix}0&0&0\\0&0&0\\0&0&1\end{bmatrix} \Phi_s$$

```python
import sympy
from sympy import (init_printing, Matrix,MatMul,
                   integrate, symbols)

init_printing(use_latex='mathjax')
dt, phi = symbols('\Delta{t} \Phi_s')
F_k = Matrix([[1, dt, dt**2/2],
              [0,  1,      dt],
              [0,  0,       1]])
Q_c = Matrix([[0, 0, 0],
              [0, 0, 0],
              [0, 0, 1]])*phi

Q=sympy.integrate(F_k * Q_c * F_k.T, (dt, 0, dt))

# factor phi out of the matrix to make it more readable
Q = Q / phi
sympy.MatMul(Q, phi)
```

$$\left[\begin{matrix}\frac{\Delta{t}^{5}}{20} & \frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{6}\\\frac{\Delta{t}^{4}}{8} & \frac{\Delta{t}^{3}}{3} & \frac{\Delta{t}^{2}}{2}\\\frac{\Delta{t}^{3}}{6} & \frac{\Delta{t}^{2}}{2} & \Delta{t}\end{matrix}\right] \Phi_{s}$$

For completeness, let us compute the equations for the 0th order and 1st order equations.

```python
F_k = sympy.Matrix([[1]])
Q_c = sympy.Matrix([[phi]])

print('0th order discrete process noise')
sympy.integrate(F_k*Q_c*F_k.T,(dt, 0, dt))
```

    0th order discrete process noise

$$\left[\begin{matrix}\Delta{t} \Phi_{s}\end{matrix}\right]$$

```python
F_k = sympy.Matrix([[1, dt],
                    [0, 1]])
Q_c = sympy.Matrix([[0, 0],
                    [0, 1]])*phi

Q = sympy.integrate(F_k * Q_c * F_k.T, (dt, 0, dt))

print('1st order discrete process noise')
# factor phi out of the matrix to make it more readable
Q = Q / phi
sympy.MatMul(Q, phi)
```

    1st order discrete process noise

$$\left[\begin{matrix}\frac{\Delta{t}^{3}}{3} & \frac{\Delta{t}^{2}}{2}\\\frac{\Delta{t}^{2}}{2} & \Delta{t}\end{matrix}\right] \Phi_{s}$$

### Piecewise White Noise Model

Another model for the noise assumes that the that highest order term (say, acceleration) is constant for the duration of each time period, but differs for each time period, and each of these is uncorrelated between time periods. In other words there is a discontinuous jump in acceleration at each time step. This is subtly different than the model above, where we assumed that the last term had a continuously varying noisy signal applied to it.

We will model this as

$$f(x)=Fx+\Gamma w$$

where $\Gamma$ is the *noise gain* of the system, and $w$ is the constant piecewise acceleration (or velocity, or jerk, etc).

Lets start by looking at a first order system. In this case we have the state transition function

$$\mathbf{F} = \begin{bmatrix}1&\Delta t \\ 0& 1\end{bmatrix}$$

In one time period, the change in velocity will be $w(t)\Delta t$, and the change in position will be $w(t)\Delta t^2/2$, giving us

$$\Gamma = \begin{bmatrix}\frac{1}{2}\Delta t^2 \\ \Delta t\end{bmatrix}$$

The covariance of the process noise is then

$$Q = \mathbb E[\Gamma w(t) w(t) \Gamma^\mathsf{T}] = \Gamma\sigma^2_v\Gamma^\mathsf{T}$$.

We can compute that with SymPy as follows

```python
var=symbols('sigma^2_v')
v = Matrix([[dt**2 / 2], [dt]])

Q = v * var * v.T

# factor variance out of the matrix to make it more readable
Q = Q / var
sympy.MatMul(Q, var)
```

$$\left[\begin{matrix}\frac{\Delta{t}^{4}}{4} & \frac{\Delta{t}^{3}}{2}\\\frac{\Delta{t}^{3}}{2} & \Delta{t}^{2}\end{matrix}\right] \sigma^{2}_{v}$$

The second order system proceeds with the same math.

$$\mathbf{F} = \begin{bmatrix}1 & \Delta t & {\Delta t}^2/2 \\ 0 & 1 & \Delta t\\ 0& 0& 1\end{bmatrix}$$

Here we will assume that the white noise is a discrete time Wiener process. This gives us

$$\Gamma = \begin{bmatrix}\frac{1}{2}\Delta t^2 \\ \Delta t\\ 1\end{bmatrix}$$

There is no 'truth' to this model, it is just convenient and provides good results. For example, we could assume that the noise is applied to the jerk at the cost of a more complicated equation.

The covariance of the process noise is then

$$Q = \mathbb E[\Gamma w(t) w(t) \Gamma^\mathsf{T}] = \Gamma\sigma^2_v\Gamma^\mathsf{T}$$.

We can compute that with SymPy as follows

```python
var=symbols('sigma^2_v')
v = Matrix([[dt**2 / 2], [dt], [1]])

Q = v * var * v.T

# factor variance out of the matrix to make it more readable
Q = Q / var
sympy.MatMul(Q, var)
```

$$\left[\begin{matrix}\frac{\Delta{t}^{4}}{4} & \frac{\Delta{t}^{3}}{2} & \frac{\Delta{t}^{2}}{2}\\\frac{\Delta{t}^{3}}{2} & \Delta{t}^{2} & \Delta{t}\\\frac{\Delta{t}^{2}}{2} & \Delta{t} & 1\end{matrix}\right] \sigma^{2}_{v}$$

We cannot say that this model is more or less correct than the continuous model - both are approximations to what is happening to the actual object. Only experience and experiments can guide you to the appropriate model. In practice you will usually find that either model provides reasonable results, but typically one will perform better than the other.

The advantage of the second model is that we can model the noise in terms of $\sigma^2$ which we can describe in terms of the motion and the amount of error we expect. The first model requires us to specify the spectral density, which is not very intuitive, but it handles varying time samples much more easily since the noise is integrated across the time period. However, these are not fixed rules - use whichever model (or a model of your own devising) based on testing how the filter performs and/or your knowledge of the behavior of the physical model.

A good rule of thumb is to set $\sigma$ somewhere from $\frac{1}{2}\Delta a$ to $\Delta a$, where $\Delta a$ is the maximum amount that the acceleration will change between sample periods. In practice we pick a number, run simulations on data, and choose a value that works well.

### Using FilterPy to Compute Q

FilterPy offers several routines to compute the $\mathbf Q$ matrix. The function `Q_continuous_white_noise()` computes $\mathbf Q$ for a given value for $\Delta t$ and the spectral density.

```python
from filterpy.common import Q_continuous_white_noise
from filterpy.common import Q_discrete_white_noise

Q = Q_continuous_white_noise(dim=2, dt=1, spectral_density=1)
print(Q)
```

    [[ 0.333  0.5]
     [ 0.5  1.0]]

```python
Q = Q_continuous_white_noise(dim=3, dt=1, spectral_density=1)
print(Q)
```

    [[ 0.05  0.125  0.167]
     [ 0.125  0.333  0.5]
     [ 0.167  0.5  1.0]]

The function `Q_discrete_white_noise()` computes $\mathbf Q$ assuming a piecewise model for the noise.

```python
Q = Q_discrete_white_noise(2, var=1.)
print(Q)
```

    [[ 0.25  0.5]
     [ 0.5  1.0]]

```python
Q = Q_discrete_white_noise(3, var=1.)
print(Q)
```

    [[ 0.25  0.5  0.5]
     [ 0.5  1.0  1.0]
     [ 0.5  1.0  1.0]]

### Simplification of Q

Many treatments use a much simpler form for $\mathbf Q$, setting it to zero except for a noise term in the lower rightmost element. Is this justified? Well, consider the value of $\mathbf Q$ for a small $\Delta t$

```python
import numpy as np

np.set_printoptions(precision=8)
Q = Q_continuous_white_noise(
    dim=3, dt=0.05, spectral_density=1)
print(Q)
np.set_printoptions(precision=3)
```

    [[ 0.00000002  0.00000078  0.00002083]
     [ 0.00000078  0.00004167  0.00125   ]
     [ 0.00002083  0.00125     0.05      ]]

We can see that most of the terms are very small. Recall that the only equation using this matrix is

$$ \mathbf P=\mathbf{FPF}^\mathsf{T} + \mathbf Q$$

If the values for $\mathbf Q$ are small relative to $\mathbf P$
than it will be contributing almost nothing to the computation of $\mathbf P$. Setting $\mathbf Q$ to the zero matrix except for the lower right term

$$\mathbf Q=\begin{bmatrix}0&0&0\\0&0&0\\0&0&\sigma^2\end{bmatrix}$$

while not correct, is often a useful approximation. If you do this you will have to perform quite a few studies to guarantee that your filter works in a variety of situations.

If you do this, 'lower right term' means the most rapidly changing term for each variable. If the state is $x=\begin{bmatrix}x & \dot x & \ddot{x} & y & \dot{y} & \ddot{y}\end{bmatrix}^\mathsf{T}$ Then Q will be 6x6; the elements for both $\ddot{x}$ and $\ddot{y}$ will have to be set to non-zero in $\mathbf Q$.

## Numeric Integration of Differential Equations

We've been exposed to several numerical techniques to solve linear differential equations. These include state-space methods, the Laplace transform, and van Loan's method.

These work well for linear ordinary differential equations (ODEs), but do not work well for nonlinear equations. For example, consider trying to predict the position of a rapidly turning car. Cars maneuver by turning the front wheels. This makes them pivot around their rear axle as it moves forward. Therefore the path will be continuously varying and a linear prediction will necessarily produce an incorrect value. If the change in the system is small enough relative to $\Delta t$ this can often produce adequate results, but that will rarely be the case with the nonlinear Kalman filters we will be studying in subsequent chapters.

For these reasons we need to know how to numerically integrate ODEs. This can be a vast topic that requires several books. If you need to explore this topic in depth *Computational Physics in Python* by Dr. Eric Ayars is excellent, and available for free here:

http://phys.csuchico.edu/ayars/312/Handouts/comp-phys-python.pdf

However, I will cover a few simple techniques which will work for a majority of the problems you encounter.

### Euler's Method

Let's say we have the initial condition problem of

$$\begin{gathered}
y' = y, \\ y(0) = 1
\end{gathered}$$

We happen to know the exact answer is $y=e^t$ because we solved it earlier, but for an arbitrary ODE we will not know the exact solution. In general all we know is the derivative of the equation, which is equal to the slope. We also know the initial value: at $t=0$, $y=1$. If we know these two pieces of information we can predict the value at $y(t=1)$ using the slope at $t=0$ and the value of $y(0)$. I've plotted this below.

```python
import matplotlib.pyplot as plt
t = np.linspace(-1, 1, 10)
plt.plot(t, np.exp(t))
t = np.linspace(-1, 1, 2)
plt.plot(t,t+1, ls='--', c='k');
```

You can see that the slope is very close to the curve at $t=0.1$, but far from it
at $t=1$. But let's continue with a step size of 1 for a moment. We can see that at $t=1$ the estimated value of $y$ is 2. Now we can compute the value at $t=2$ by taking the slope of the curve at $t=1$ and adding it to our initial estimate. The slope is computed with $y'=y$, so the slope is 2.

```python
import code.book_plots as book_plots

t = np.linspace(-1, 2, 20)
plt.plot(t, np.exp(t))
t = np.linspace(0, 1, 2)
plt.plot([1, 2, 4], ls='--', c='k')
book_plots.set_labels(x='x', y='y');
```

Here we see the next estimate for y is 4. The errors are getting large quickly, and you might be unimpressed. But 1 is a very large step size. Let's put this algorithm in code, and verify that it works by using a small step size.

```python
def euler(t, tmax, y, dx, step=1.):
    ys = []
    while t < tmax:
        y = y + step*dx(t, y)
        ys.append(y)
        t +=step
    return ys
```

```python
def dx(t, y): return y

print(euler(0, 1, 1, dx, step=1.)[-1])
print(euler(0, 2, 1, dx, step=1.)[-1])
```

    2.0
    4.0

This looks correct. So now lets plot the result of a much smaller step size.

```python
ys = euler(0, 4, 1, dx, step=0.00001)
plt.subplot(1,2,1)
plt.title('Computed')
plt.plot(np.linspace(0, 4, len(ys)),ys)
plt.subplot(1,2,2)
t = np.linspace(0, 4, 20)
plt.title('Exact')
plt.plot(t, np.exp(t));
```

```python
print('exact answer=', np.exp(4))
print('euler answer=', ys[-1])
print('difference =', np.exp(4) - ys[-1])
print('iterations =', len(ys))
```

    exact answer= 54.5981500331
    euler answer= 54.59705808834125
    difference = 0.00109194480299
    iterations = 400000

Here we see that the error is reasonably small, but it took a very large number of iterations to get three digits of precision. In practice Euler's method is too slow for most problems, and we use more sophisticated methods.

Before we go on, let's formally derive Euler's method, as it is the basis for the more advanced Runge Kutta methods used in the next section. In fact, Euler's method is the simplest form of Runge Kutta.

Here are the first 3 terms of the Euler expansion of $y$. An infinite expansion would give an exact answer, so $O(h^4)$ denotes the error due to the finite expansion.

$$y(t_0 + h) = y(t_0) + h y'(t_0) + \frac{1}{2!}h^2 y''(t_0) + \frac{1}{3!}h^3 y'''(t_0) +  O(h^4)$$

Here we can see that Euler's method is using the first two terms of the Taylor expansion. Each subsequent term is smaller than the previous terms, so we are assured that the estimate will not be too far off from the correct value.

### Runge Kutta Methods

Runge Kutta is the workhorse of numerical integration. There are a vast number of methods in the literature. In practice, using the Runge Kutta algorithm that I present here will solve most any problem you will face. It offers a very good balance of speed, precision, and stability, and it is the 'go to' numerical integration method unless you have a very good reason to choose something different.

Let's dive in. We start with some differential equation

$$\ddot{y} = \frac{d}{dt}\dot{y}$$.

We can substitute the derivative of y with a function f, like so

$$\ddot{y} = \frac{d}{dt}f(y,t)$$.

Deriving these equations is outside the scope of this book, but the Runge Kutta RK4 method is defined with these equations.

$$y(t+\Delta t) = y(t) + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4) + O(\Delta t^4)$$

$$\begin{aligned}
k_1 &= f(y,t)\Delta t \\
k_2 &= f(y+\frac{1}{2}k_1, t+\frac{1}{2}\Delta t)\Delta t \\
k_3 &= f(y+\frac{1}{2}k_2, t+\frac{1}{2}\Delta t)\Delta t \\
k_4 &= f(y+k_3, t+\Delta t)\Delta t
\end{aligned}
$$

Here is the corresponding code:

```python
def runge_kutta4(y, x, dx, f):
    """computes 4th order Runge-Kutta for dy/dx.
    y is the initial value for y
    x is the initial value for x
    dx is the difference in x (e.g. the time step)
    f is a callable function (y, x) that you supply
    to compute dy/dx for the specified values.
    """

    k1 = dx * f(y, x)
    k2 = dx * f(y + 0.5*k1, x + 0.5*dx)
    k3 = dx * f(y + 0.5*k2, x + 0.5*dx)
    k4 = dx * f(y + k3, x + dx)

    return y + (k1 + 2*k2 + 2*k3 + k4) / 6.
```

Let's use this for a simple example. Let

$$\dot{y} = t\sqrt{y(t)}$$

with the initial values

$$\begin{aligned}t_0 &= 0\\y_0 &= y(t_0) = 1\end{aligned}$$

```python
import math
import numpy as np
t = 0.
y = 1.
dt = .1

ys, ts = [], []

def func(y,t):
    return t*math.sqrt(y)

while t <= 10:
    y = runge_kutta4(y, t, dt, func)
    t += dt
    ys.append(y)
    ts.append(t)

exact = [(t**2 + 4)**2 / 16. for t in ts]
plt.plot(ts, ys)
plt.plot(ts, exact)

error = np.array(exact) - np.array(ys)
print("max error {}".format(max(error)))
```

## Bayesian Filtering

Starting in the Discrete Bayes chapter I used a Bayesian formulation for filtering. Suppose we are tracking an object. We define its *state* at a specific time as its position, velocity, and so on. For example, we might write the state at time $t$ as $\mathbf x_t = \begin{bmatrix}x_t &\dot x_t \end{bmatrix}^\mathsf T$.

When we take a measurement of the object we are measuring the state or part of it. Sensors are noisy, so the measurement is corrupted with noise. Clearly though, the measurement is determined by the state. That is, a change in state may change the measurement, but a change in measurement will not change the state.

In filtering our goal is to compute an optimal estimate for a set of states $\mathbf x_{0:t}$ from time 0 to time $t$. If we knew $\mathbf x_{0:t}$ then it would be trivial to compute a set of measurements $\mathbf z_{0:t}$ corresponding to those states. However, we receive a set of measurements $\mathbf z_{0:t}$, and want to compute the corresponding states $\mathbf x_{0:t}$. This is called *statistical inversion* because we are trying to compute the input from the output.

Inversion is a difficult problem because there is typically no unique solution. For a given set of states $\mathbf x_{0:t}$ there is only one possible set of measurements (plus noise), but for a given set of measurements there are many different sets of states that could have led to those measurements.

Recall Bayes Theorem:

$$P(x \mid z) = \frac{P(z \mid x)P(x)}{P(z)}$$

where $P(z \mid x)$ is the *likelihood* of the measurement $z$, $P(x)$ is the *prior* based on our process model, and $P(z)$ is a normalization constant. $P(x \mid z)$ is the *posterior*, or the distribution after incorporating the measurement $z$, also called the *evidence*.

This is a *statistical inversion* as it goes from $P(z \mid x)$ to $P(x \mid z)$. The solution to our filtering problem can be expressed as:

$$P(\mathbf x_{0:t} \mid \mathbf z_{0:t}) = \frac{P(\mathbf z_{0:t} \mid \mathbf x_{0:t})P(\mathbf x_{0:t})}{P(\mathbf z_{0:t})}$$

That is all well and good until the next measurement $\mathbf z_{t+1}$ comes in, at which point we need to recompute the entire expression for the range $0:t+1$.

In practice this is intractable because we are trying to compute the posterior distribution $P(\mathbf x_{0:t} \mid \mathbf z_{0:t})$ for the state over the full range of time steps. But do we really care about the probability distribution at the third step (say) when we just received the tenth measurement? Not usually. So we relax our requirements and only compute the distributions for the current time step.

The first simplification is we describe our process (e.g., the motion model for a moving object) as a *Markov chain*. That is, we say that the current state is solely dependent on the previous state and a transition probability $P(\mathbf x_k \mid \mathbf x_{k-1})$, which is just the probability of going from the last state to the current one. We write:

$$\mathbf x_k \sim P(\mathbf x_k \mid \mathbf x_{k-1})$$

The next simplification we make is do define the *measurement model* as depending on the current state $\mathbf x_k$ with the conditional probability of the measurement given the current state: $P(\mathbf z_t \mid \mathbf x_x)$. We write:

$$\mathbf z_k \sim P(\mathbf z_t \mid \mathbf x_x)$$

We have a recurrance now, so we need an initial condition to terminate it. Therefore we say that the initial distribution is the probablity of the state $\mathbf x_0$:

$$\mathbf x_0 \sim P(\mathbf x_0)$$

These terms are plugged into Bayes equation. If we have the state $\mathbf x_0$ and the first measurement we can estimate $P(\mathbf x_1 | \mathbf z_1)$. The motion model creates the prior $P(\mathbf x_2 \mid \mathbf x_1)$. We feed this back into Bayes theorem to compute $P(\mathbf x_2 | \mathbf z_2)$. We continue this predictor-corrector algorithm, recursively computing the state and distribution at time $t$ based solely on the state and distribution at time $t-1$ and the measurement at time $t$.

The details of the mathematics for this computation varies based on the problem. The **Discrete Bayes** and **Univariate Kalman Filter** chapters gave two different formulations which you should have been able to reason through. The univariate Kalman filter assumes that for a scalar state both the noise and process are linear model are affected by zero-mean, uncorrelated Gaussian noise.

The Multivariate Kalman filter make the same assumption but for states and measurements that are vectors, not scalars. Dr. Kalman was able to prove that if these assumptions hold true then the Kalman filter is *optimal* in a least squares sense. Colloquially this means there is no way to derive more information from the noise. In the remainder of the book I will present filters that relax the constraints on linearity and Gaussian noise.

Before I go on, a few more words about statistical inversion. As Calvetti and Somersalo write in *Introduction to Bayesian Scientific Computing*, "we adopt the Bayesian point of view: *randomness simply means lack of information*."[3] Our state parametize physical phenomena that we could in principle measure or compute: velocity, air drag, and so on. We lack enough information to compute or measure their value, so we opt to consider them as random variables. Strictly speaking they are not random, thus this is a subjective position.

They devote a full chapter to this topic. I can spare a paragraph. Bayesian filters are possible because we ascribe statistical properties to unknown parameters. In the case of the Kalman filter we have closed-form solutions to find an optimal estimate. Other filters, such as the discrete Bayes filter or the particle filter which we cover in a later chapter, model the probability in a more ad-hoc, non-optimal manner. The power of our technique comes from treating lack of information as a random variable, describing that random variable as a probability distribution, and then using Bayes Theorem to solve the statistical inference problem.

## Converting Kalman Filter to a g-h Filter

I've stated that the Kalman filter is a form of the g-h filter. It just takes some algebra to prove it. It's more straightforward to do with the one dimensional case, so I will do that. Recall

$$
\mu_{x}=\frac{\sigma_1^2 \mu_2 + \sigma_2^2 \mu_1} {\sigma_1^2 + \sigma_2^2}
$$

which I will make more friendly for our eyes as:

$$
\mu_{x}=\frac{ya + xb} {a+b}
$$

We can easily put this into the g-h form with the following algebra

$$
\begin{aligned}
\mu_{x}&=(x-x) + \frac{ya + xb} {a+b} \\
\mu_{x}&=x-\frac{a+b}{a+b}x  + \frac{ya + xb} {a+b} \\
\mu_{x}&=x +\frac{-x(a+b) + xb+ya}{a+b} \\
\mu_{x}&=x+ \frac{-xa+ya}{a+b}  \\
\mu_{x}&=x+ \frac{a}{a+b}(y-x)\\
\end{aligned}
$$

We are almost done, but recall that the variance of estimate is given by

$$\begin{aligned}
\sigma_{x}^2 &= \frac{1}{\frac{1}{\sigma_1^2} +  \frac{1}{\sigma_2^2}} \\
&= \frac{1}{\frac{1}{a} +  \frac{1}{b}}
\end{aligned}$$

We can incorporate that term into our equation above by observing that

$$
\begin{aligned}
\frac{a}{a+b} &= \frac{a/a}{(a+b)/a} = \frac{1}{(a+b)/a}  \\
 &= \frac{1}{1 + \frac{b}{a}} = \frac{1}{\frac{b}{b} + \frac{b}{a}}  \\
 &= \frac{1}{b}\frac{1}{\frac{1}{b} + \frac{1}{a}} \\
 &= \frac{\sigma^2_{x'}}{b}
 \end{aligned}
$$

We can tie all of this together with

$$
\begin{aligned}
\mu_{x}&=x+ \frac{a}{a+b}(y-x) \\
&= x + \frac{\sigma^2_{x'}}{b}(y-x) \\
&= x + g_n(y-x)
\end{aligned}
$$

where

$$g_n = \frac{\sigma^2_{x}}{\sigma^2_{y}}$$

The end result is multiplying the residual of the two measurements by a constant and adding to our previous value, which is the $g$ equation for the g-h filter. $g$ is the variance of the new estimate divided by the variance of the measurement. Of course in this case $g$ is not a constant as it varies with each time step as the variance changes. We can also derive the formula for $h$ in the same way. It is not a particularly illuminating derivation and I will skip it. The end result is

$$h_n = \frac{COV (x,\dot x)}{\sigma^2_{y}}$$

The takeaway point is that $g$ and $h$ are specified fully by the variance and covariances of the measurement and predictions at time $n$. In other words, we are picking a point between the measurement and prediction by a scale factor determined by the quality of each of those two inputs.

## References

 * [1] C.B. Molwer and C.F. Van Loan "Nineteen Dubious Ways to Compute the Exponential of a Matrix, Twenty-Five Years Later,", *SIAM Review 45, 3-49*. 2003.

 * [2] C.F. van Loan, "Computing Integrals Involving the Matrix Exponential," IEEE *Transactions Automatic Control*, June 1978.

 * [3] Calvetti, D and Somersalo E, "Introduction to Bayesian Scientific Computing: Ten Lectures on Subjective Computing,", *Springer*, 2007.


*Sebastian Raschka*
last modified: 03/31/2014

I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note
via Twitter: [@rasbt](https://twitter.com/rasbt)
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)

### Problem Category
- Statistical Pattern Recognition
- Supervised Learning
- Parametric Learning
- Bayes Decision Theory
- Multivariate data (2-dimensional)
- 2-class problem
- different variances
- equal prior probabilities
- Gaussian model (2 parameters)
- with conditional Risk (1-0 loss functions)

# Sections

• Given information

• Deriving the decision boundary

• Classifying some random example data

• Calculating the empirical error rate

## Given information:

[back to top]

####model: continuous univariate normal (Gaussian) model for the class-conditional densities

$ p(\pmb{x} | \omega_j) \sim N(\pmb{\mu}|\Sigma) $

$ p(\pmb{x} | \omega_j) \sim \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \exp{ \bigg[-\frac{1}{2} (\pmb{x}-\pmb{\mu})^t \Sigma^{-1}(\pmb{x}-\pmb{\mu}) \bigg] } $

####Prior probabilities:

$ P(\omega_1) = P(\omega_2) = 0.5 $

####Loss functions:

where

$ \lambda(\alpha_i|\omega_j) = \lambda_{ij}$,

the loss occured if $action_i$ is taken if the actual true class is $ \omega_j$ (assuming that $action_i$ classifies sample as $\omega_i$)

\begin{equation}
\lambda = \begin{pmatrix}
\lambda_{11} \quad \lambda_{12} \\
\lambda_{21} \quad \lambda_{22}
\end{pmatrix}
= \begin{pmatrix}
0 \quad 1 \\
1 \quad 0 \\
\end{pmatrix}\\
\end{equation}

The samples are of 2-dimensional feature vectors:

$ \pmb{x} = \bigg[
\begin{array}{c}
x_1 \\
x_2 \\
\end{array} \bigg] $

#### Means of the sample distributions for 2-dimensional features:

$ \pmb{\mu}_{\,1} = \bigg[
\begin{array}{c}
0 \\
0 \\
\end{array} \bigg] $,
$ \; \pmb{\mu}_{\,2} = \bigg[
\begin{array}{c}
1 \\
2 \\
\end{array} \bigg] $

#### Covariance matrices for the statistically independend and identically distributed ('i.i.d') features:

$ \Sigma_i = \bigg[
\begin{array}{cc}
\sigma_{11}^2 & \sigma_{12}^2\\
\sigma_{21}^2 & \sigma_{22}^2 \\
\end{array} \bigg], \;
\Sigma_1 = \bigg[
\begin{array}{cc}
1 & 0\\
0 & 1 \\
\end{array} \bigg], \;
\Sigma_1 = \bigg[
\begin{array}{cc}
2 & 0\\
0 & 2 \\
\end{array} \bigg] $

## Deriving the decision boundary
[back to top]

### Bayes' Rule:

$ P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)}$

### Risk Functions:

$ R(\alpha_1|\;\pmb{x}) = \lambda_{11}P(\omega_1|\;\pmb{x}) + \lambda_{12}P(\omega_2|\;\pmb{x}) $

$ R(\alpha_2|\;\pmb{x}) = \lambda_{21}P(\omega_1|\;\pmb{x}) + \lambda_{22}P(\omega_2|\;\pmb{x}) $

with 1-0 loss function:

$ R(\alpha_1|\; \pmb{x}) = P(\omega_2|\;\pmb{x}) = 1 -  P(\omega_1|\;\pmb{x})  $

$ R(\alpha_2|\;\pmb{x}) = P(\omega_1|\;\pmb{x}) = 1 -  P(\omega_2|\;\pmb{x}) $

### Discriminant Functions:

The **objective function** is to maximize the discriminant function, which we define as the posterior probability here to perform a **minimum-error classification** (Bayes classifier).

$ g_1(\pmb{x}) = P(\omega_1 | \; \pmb{x}), \quad  g_2(\pmb{x}) = P(\omega_2 | \; \pmb{x})$

$ \Rightarrow g_1(\pmb{x}) = P(\pmb{x}\;|\;\omega_1) \;\cdot\; P(\omega_1) \quad | \; ln \\
\quad g_2(\pmb{x}) = P(\pmb{x}\;|\;\omega_2) \;\cdot\; P(\omega_2) \quad | \; ln $

We can drop the prior probabilities (since we have equal priors in this case):

$ \Rightarrow g_1(\pmb{x}) = ln(P(\pmb{x}\;|\;\omega_1))  \\
\quad g_2(\pmb{x}) = ln(P(\pmb{x}\;|\;\omega_2))  $

The equations for the general multivariate normal case with arbitrary covariance matrices (i.e., different covariance matrices for each case):

$ \Rightarrow g_1(\pmb{x}) = \pmb{x}^{\,t} - \frac{1}{2} \Sigma_1^{-1} \pmb{x} + \bigg( \Sigma_1^{-1} \pmb{\mu}_{\,1}\bigg)^t + \bigg( -\frac{1}{2} \pmb{\mu}_{\,1}^{\,t} \Sigma_{1}^{-1} \pmb{\mu}_{\,1} -\frac{1}{2} ln(|\Sigma_1|)\bigg)   \\
\quad g_2(\pmb{x}) = \pmb{x}^{\,t} - \frac{1}{2} \Sigma_2^{-1} \pmb{x} + \bigg( \Sigma_2^{-1} \pmb{\mu}_{\,2}\bigg)^t + \bigg( -\frac{1}{2} \pmb{\mu}_{\,2}^{\,t} \Sigma_{2}^{-1} \pmb{\mu}_{\,2} -\frac{1}{2} ln(|\Sigma_2|)\bigg)  $

**Let:**

$\pmb{W}_{\,i} = - \frac{1}{2} \Sigma_i^{-1}\\
\pmb{w}{\,i} =   \Sigma_i^{-1} \pmb{\mu}_{\,i} \\
\omega_{i0} = \bigg( -\frac{1}{2} \pmb{\mu}_{\,i}^{\,t} \Sigma_{i}^{-1} \pmb{\mu}_{\,i} -\frac{1}{2} ln(|\Sigma_i|)\bigg)$

$ \pmb{W}_{\,1} = \bigg[
\begin{array}{cc}
(1/4) & 0\\
0 & (1/4) \\
\end{array} \bigg] $

$ \pmb{w}_{\,1} = \bigg[
\begin{array}{cc}
(1/4) & 0\\
0 & (1/4) \\
\end{array} \bigg] \cdot \bigg[
\begin{array}{c}
0 \\
0 \\
\end{array} \bigg] = \bigg[
\begin{array}{c}
0 \\
0 \\
\end{array} \bigg]$

$ \omega_{10} = -\frac{1}{2} [0 \quad 0 ] \bigg[
\begin{array}{cc}
(1/4) & 0\\
0 & (1/4) \\
\end{array} \bigg] \cdot \bigg[
\begin{array}{c}
0 \\
0 \\
\end{array} \bigg]
- ln(2) = -ln(2)$

$ \pmb{W}_{\,2} = \bigg[
\begin{array}{cc}
(-1/2) & 0\\
0 & (-1/2) \\
\end{array} \bigg] $

$ \pmb{w}_{\,2} = \bigg[
\begin{array}{c}
1 \\
2 \\
\end{array} \bigg]$

$ \omega_{20} = -\frac{1}{2} [1 \quad 2 ] \bigg[
\begin{array}{cc}
(1/4) & 0\\
0 & (1/4) \\
\end{array} \bigg] \cdot \bigg[
\begin{array}{c}
1 \\
2 \\
\end{array} \bigg]
- \frac{1}{2}ln(1) = -2.5$

$ \Rightarrow g_1(\pmb{x}) = \pmb{x}^{\,t} \bigg[
\begin{array}{cc}
(1/4) & 0\\
0 & (1/4) \\
\end{array} \bigg]
\pmb{x} + \bigg[
\begin{array}{c}
0 \\
0 \\
\end{array} \bigg]^t \pmb x- ln(2) = \pmb{x}^{\,t} - \frac{1}{4} \pmb{x} - ln(2)$

$ \Rightarrow g_2(\pmb{x}) = \pmb{x}^{\,t} \bigg[
\begin{array}{cc}
(-1/2) & 0\\
0 & (-1/2) \\
\end{array} \bigg]
\pmb{x} + \bigg[
\begin{array}{c}
1 \\
2 \\
\end{array} \bigg]^t \pmb{x} -2.5 = \pmb{x}^{\,t} - \frac{1}{2} \pmb{x} + [1 \quad 2] \; \pmb{x} - 2.5$

### Decision Boundary

$g_1(\pmb{x}) = g_2(\pmb{x}) $

$ \Rightarrow \pmb{x}^{\,t} \cdot (- \frac{1}{4} \pmb{x}) - ln(2) = \pmb{x}^{\,t} - \frac{1}{2} \pmb{x} + [1 \quad 2] \; \pmb{x} - 2.5 \quad \bigg| \; \cdot \; 4$

$ \Rightarrow \pmb{x}^{\,t} \cdot (-  \pmb{x}) - 4ln(2) = \pmb{x}^{\,t} - 2 \pmb{x}+ 4 \;( [1 \quad 2] \; \pmb{x}) - 10 $

$ \Rightarrow [x_1 \quad x_2] \cdot \bigg(-\bigg[ \begin{array}{c}
x_1 \\
x_2 \\
\end{array}\bigg] \bigg) -4ln(2) = [x_1 \quad x_2] \cdot \bigg(-\bigg[ \begin{array}{c}
2x_1 \\
2x_2 \\
\end{array}\bigg] \bigg) + [4 \quad 8] \cdot
\bigg[ \begin{array}{c}
x_1 \\
x_2 \\
\end{array}\bigg] -10$

$ \Rightarrow -x_1^2 - x_2^2 - 4ln(2) = -2x_1^2 - 2x_2^2 + 4x_1 + 8x_2 -10 $

$ \Rightarrow x_1^2 + x_2^2 -4x_1 - 8x_2 - 4ln(2) + 10 = 0 $

## Classifying some random example data

[back to top]

```
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def decision_boundary(x_1):
    """ Calculates the x_2 value for plotting the decision boundary."""
    return 4 - np.sqrt(-x_1**2 + 4*x_1 + 6 + np.log(16))

def decision_rule(x_vec)
x_1^2 + x_2^2 - -4x_1 - 8x_2 - 4ln(2) + 10 = 0

# Generate 100 random patterns for class1
mu_vec1 = np.array([0,0])
cov_mat1 = np.array([[2,0],[0,2]])
x1_samples = np.random.multivariate_normal(mu_vec1, cov_mat1, 100)
mu_vec1 = mu_vec1.reshape(1,2).T # to 1-col vector

# Generate 100 random patterns for class2
mu_vec2 = np.array([1,2])
cov_mat2 = np.array([[1,0],[0,1]])
x2_samples = np.random.multivariate_normal(mu_vec2, cov_mat2, 100)
mu_vec2 = mu_vec2.reshape(1,2).T # to 1-col vector

# Scatter plot
f, ax = plt.subplots(figsize=(7, 7))
ax.scatter(x1_samples[:,0], x1_samples[:,1], marker='o', color='green', s=40, alpha=0.5)
ax.scatter(x2_samples[:,0], x2_samples[:,1], marker='^', color='blue', s=40, alpha=0.5)
plt.legend(['Class1 (w1)', 'Class2 (w2)'], loc='upper right')
plt.title('Densities of 2 classes with 100 bivariate random patterns each')
plt.ylabel('x2')
plt.xlabel('x1')
ftext = 'p(x|w1) ~ N(mu1=(0,0)^t, cov1=2*I)\np(x|w2) ~ N(mu2=(1,2)^t, cov2=I)'
plt.figtext(.15,.8, ftext, fontsize=11, ha='left')

# Plot decision boundary
x_1 = np.arange(-5, 5, 0.1)
bound = decision_boundary(x_1)
plt.plot(x_1, bound, 'r--', lw=4)

x_vec = np.linspace(*ax.get_xlim())
x_1 = np.arange(0, 100, 0.05)

plt.show()
```

## Calculating the empirical error rate

[back to top]

```
def decision_rule(x_vec):
    """ Returns value for the decision rule of 2-d row vectors """
    x_1 = x_vec[0]
    x_2 = x_vec[1]
    return x_1**2 + x_2**2 -4*x_1 - 8*x_2 - 4*np.log(2) + 10

w1_as_w2, w2_as_w1 = 0, 0

for x in x1_samples:
    if decision_rule(x) < 0:
        w1_as_w2 += 1
for x in x2_samples:
    if decision_rule(x) > 0:
        w2_as_w1 += 1

emp_err = (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))

print('Empirical Error: {}%'.format(emp_err * 100))
```

    Empirical Error: 19.5%