# Statistical Rethinking Second Edition

Ini adalah kesimpulan yang didapatkan dari buku *Statistical Rethinking* tulisan Professor Richard McElreath. Buku ini berisi tentang pendekatan statistik untuk penelitian dengan cara Bayesian. Dalam buku ini, dapat diketemukan beberapa contoh kasus dan penerapan-nya dalam kode R yang akan kami terjemahkan ke dalam python.

```python
import numpy as np
import itertools
```

## Chapter 2: Small Worlds and Large Worlds

Pada bab ini, penulis berargumen bahwa dalam setiap penelitian selalu ada dua buah dunia. Pertama adalah dunia nyata dimana kasus terjadi dan yang kedua adalah dunia tempat penelitian dijalankan. Beberapa penelitian dalam buku ini adalah membentuk model yang juga menjadi representasi dari *small word*. Dalam upaya mengambil beberapa bagian dari dunia nyata dan memodelkannya dalam ruang isolasi, dibutuhkan asumsi-asumsi yang berkaitan dengan upaya tersebut. Asumsi ini ada karena *small world* tidak mungkin mereplikasi apa yang sebenernya terjadi--entah karena memang tidak ada model yang cocok atau membutuhkan usaha komputasi yang besar--dan memudahkan perhitungan untuk beberapa kasus.

**Soal 1** Ada empat buah kelereng bewarna biru dan putih dan sebuah kantung. Tidak diketahui berapa jumlah kelerang yang biru dan yang putih. Estimasi jumlah kelereng berwarna biru dan putih berdasarkan hasil yang ditarik (dan dikembalikan lagi) dari kantung.

```python
collection = itertools.combinations_with_replacement("WB", 4)
for num, val in enumerate(collection):
    print(num, val)
```

    0 ('W', 'W', 'W', 'W')
    1 ('W', 'W', 'W', 'B')
    2 ('W', 'W', 'B', 'B')
    3 ('W', 'B', 'B', 'B')
    4 ('B', 'B', 'B', 'B')

Setidaknya ada 5 kemungkinan bagaimana komposisi kelereng dalam kantung. Dalam buku ini, penulis mengatakan ini adalah dugaan dari komposisi dalam kantung. Tugas kami adalah mencari mana yang paling masuk akal. Asumsilah bahwa penarikan kelereng menghasilkan **Biru**, **Putih**, dan **Biru**.

Mari kami telaah satu dari lima dugaan yaitu nomor 1. Dalam dugaan nomor 1, ada 4 kemungkinan yang terjadi saat pengambilan kelereng. dimana ada 3 warna putih dan satu warna biru. Pada pengambilan kelereng yang kedua, masing-masing kemungkinan akan memiliki 4 kemungkinan juga sehingga total kemungkinan menjadi $4^2=16$. Pada pengambilan ketiga masing masing dari 16 kemungikinan ini akan memiliki 4 kemungkinan lagi sehingga total kemungkinan adalah $4^3=64$

```python
collection1 = itertools.product('WWWB', repeat=1)
print("First draw")
for num, val in enumerate(collection1):
    print(num, val)

collection2 = itertools.product('WWWB', repeat=2)
print("Second draw")
for num, val in enumerate(collection2):
    print(num, val)

collection3 = itertools.product('WWWB', repeat=3)
print("Third draw")
for num, val in enumerate(collection3):
    print(num, val)
```

    First draw
    0 ('W',)
    1 ('W',)
    2 ('W',)
    3 ('B',)
    Second draw
    0 ('W', 'W')
    1 ('W', 'W')
    2 ('W', 'W')
    3 ('W', 'B')
    4 ('W', 'W')
    5 ('W', 'W')
    6 ('W', 'W')
    7 ('W', 'B')
    8 ('W', 'W')
    9 ('W', 'W')
    10 ('W', 'W')
    11 ('W', 'B')
    12 ('B', 'W')
    13 ('B', 'W')
    14 ('B', 'W')
    15 ('B', 'B')
    Third draw
    0 ('W', 'W', 'W')
    1 ('W', 'W', 'W')
    2 ('W', 'W', 'W')
    3 ('W', 'W', 'B')
    4 ('W', 'W', 'W')
    5 ('W', 'W', 'W')
    6 ('W', 'W', 'W')
    7 ('W', 'W', 'B')
    8 ('W', 'W', 'W')
    9 ('W', 'W', 'W')
    10 ('W', 'W', 'W')
    11 ('W', 'W', 'B')
    12 ('W', 'B', 'W')
    13 ('W', 'B', 'W')
    14 ('W', 'B', 'W')
    15 ('W', 'B', 'B')
    16 ('W', 'W', 'W')
    17 ('W', 'W', 'W')
    18 ('W', 'W', 'W')
    19 ('W', 'W', 'B')
    20 ('W', 'W', 'W')
    21 ('W', 'W', 'W')
    22 ('W', 'W', 'W')
    23 ('W', 'W', 'B')
    24 ('W', 'W', 'W')
    25 ('W', 'W', 'W')
    26 ('W', 'W', 'W')
    27 ('W', 'W', 'B')
    28 ('W', 'B', 'W')
    29 ('W', 'B', 'W')
    30 ('W', 'B', 'W')
    31 ('W', 'B', 'B')
    32 ('W', 'W', 'W')
    33 ('W', 'W', 'W')
    34 ('W', 'W', 'W')
    35 ('W', 'W', 'B')
    36 ('W', 'W', 'W')
    37 ('W', 'W', 'W')
    38 ('W', 'W', 'W')
    39 ('W', 'W', 'B')
    40 ('W', 'W', 'W')
    41 ('W', 'W', 'W')
    42 ('W', 'W', 'W')
    43 ('W', 'W', 'B')
    44 ('W', 'B', 'W')
    45 ('W', 'B', 'W')
    46 ('W', 'B', 'W')
    47 ('W', 'B', 'B')
    48 ('B', 'W', 'W')
    49 ('B', 'W', 'W')
    50 ('B', 'W', 'W')
    51 ('B', 'W', 'B')
    52 ('B', 'W', 'W')
    53 ('B', 'W', 'W')
    54 ('B', 'W', 'W')
    55 ('B', 'W', 'B')
    56 ('B', 'W', 'W')
    57 ('B', 'W', 'W')
    58 ('B', 'W', 'W')
    59 ('B', 'W', 'B')
    60 ('B', 'B', 'W')
    61 ('B', 'B', 'W')
    62 ('B', 'B', 'W')
    63 ('B', 'B', 'B')

Tentu saja tidak semua kemungkinan diatas didukung oleh data yang kami miliki. Oleh karena itu, hasil yang terakhir yang berjumlah 64 akan disaring dan akan dikumpulkan mana yang cocok dengan data yang dimiliki. Muncul 3 buah kemungkinan yang didukung oleh data. Ini hanya satu dari banyak lima dugaan yang ada. Mari kami periksa bagaimana dugaan-dugaan yang kami miliki.

```python
def conjecture_fitting(conjecture, data):
    collection = itertools.product(conjecture, repeat=len(data))
    arr_buff = []
    for num, val in enumerate(collection):
        arr = list(val)
        if arr == data:
            arr_buff.append(1)
    return len(arr_buff)

print('Conjecture WWWW ',conjecture_fitting('WWWW', ['B', 'W', 'B']))
print('Conjecture WWWB ',conjecture_fitting('WWWB', ['B', 'W', 'B']))
print('Conjecture WWBB ',conjecture_fitting('WWBB', ['B', 'W', 'B']))
print('Conjecture WBBB ',conjecture_fitting('WBBB', ['B', 'W', 'B']))
print('Conjecture BBBB ',conjecture_fitting('BBBB', ['B', 'W', 'B']))
```

    Conjecture WWWW  0
    Conjecture WWWB  3
    Conjecture WWBB  8
    Conjecture WBBB  9
    Conjecture BBBB  0

Menurut perhitungan yang kami buat, dugaan yang paling sesuai dengan data adalah dugaan ke 3 karena memiliki jumlah kemungkinan kombinasi yang sama dengan data paling tinggi dibandingkan dugaan lain. Oleh karena itu, memilih komposisi kelereng dugaan ke 3 adalah hal yang paling masuk akal. Asumsilah kami mengambil satu buah kelereng dan warananya adalah biru. Kita dapat menghitung ulang menggunakan cara yang sama dengan total kemungkinan adalah $4^4= 256$. Kita akan membandingkan dua cara, yaitu mengulang ulang perhitungan untuk 4 buah penarikan dan mengalikan kemungkinan sebelumnya (saat hanya memiliki data 3 buah penarikan) dengan kemungkinan yang baru (3 + 1 penarikan).

```python
print("Recounting")
data = ['B', 'W', 'B', 'B']
print('Conjecture WWWW ',conjecture_fitting('WWWW', data))
print('Conjecture WWWB ',conjecture_fitting('WWWB', data))
print('Conjecture WWBB ',conjecture_fitting('WWBB', data))
print('Conjecture WBBB ',conjecture_fitting('WBBB', data))
print('Conjecture BBBB ',conjecture_fitting('BBBB', data))

print("\nUpdate Counting  P N T")
prev_data = ['B', 'W', 'B']
next_data = ['B']
def update_conjecture(conjecture, prev_data, next_data):
    prev_con = conjecture_fitting(conjecture, prev_data)
    next_con = conjecture_fitting(conjecture, next_data)
    return prev_con, next_con

prev_con, next_con = update_conjecture('WWWW', prev_data, next_data)
print('Conjecture WWWW ', prev_con, next_con, prev_con * next_con)

prev_con, next_con = update_conjecture('WWWB', prev_data, next_data)
print('Conjecture WWWB ', prev_con, next_con, prev_con * next_con)

prev_con, next_con = update_conjecture('WWBB', prev_data, next_data)
print('Conjecture WWBB ', prev_con, next_con, prev_con * next_con)

prev_con, next_con = update_conjecture('WBBB', prev_data, next_data)
print('Conjecture WBBB ', prev_con, next_con, prev_con * next_con)

prev_con, next_con = update_conjecture('BBBB', prev_data, next_data)
print('Conjecture BBBB ', prev_con, next_con, prev_con * next_con)
```

    Recounting
    Conjecture WWWW  0
    Conjecture WWWB  3
    Conjecture WWBB  16
    Conjecture WBBB  27
    Conjecture BBBB  0

    Update Counting  P N T
    Conjecture WWWW  0 0 0
    Conjecture WWWB  3 1 3
    Conjecture WWBB  8 2 16
    Conjecture WBBB  9 3 27
    Conjecture BBBB  0 4 0

Ternyata hasil dari menggunakan perhitungan ulang dan memperbarui perhitungan menggunakan perhitungan sebelumnya menghasilkan hasil yang sama. Secara komputasi, menggukanan metode memperbarui perhitungan lebih unggul karena tidak terbentur kompleksitas $4^n$ yang dimiliki oleh perhitungan ulang.

Ditanyakan proporsi antara air dan tanah pada suatu bola bumi. Informasi yang diberikan adalah hasil dari melempar bola bumi tersebut dan menangkapnya. Diambil letak jari manis pada tangan kanan untuk menentukan apakah hasil lemparan ini mendarat pada tanah atau air. Ini sebenernya mirip dengan kasus pelemparan koin, hanya saja untuk pelemparan koin khalayak umum biasanya mengasumsi koin jujur--kemungkinan muncul angka dan gambar adalah sama--sedangkan untuk bola bumi khalayak umum hanya memiliki asumsi bahwa air lebih banyak daripada tanah.

Kita akan coba memodelkan pelemparan ini sambil menerjemahkan kode penulis dari R menjadi Python.

Jumlah pelemparan yang menghasilkan air $W$ dilambangkan dengan proporsi $p$ sedangkan yang menghasilkan tanah $L$ dilambangkan dengan $1-p$. Dikarenakan ini adalah contoh kasus binomial, maka akan diterapkan distribusi binomial yang dapat dijabarkan sebagai

$$Pr(W, L|p) = \frac{(W+L)!}{W!L!}p^W(1-p)^L$$

Dibawah ini adalah class dari pelemparan yang sudah ada metode metode untuk membantu memudahkan pengolahan.

```python
import numpy as np
from scipy.stats import binom, norm, beta
from sklearn.preprocessing import MinMaxScaler

from scipy.special import comb
import matplotlib.pyplot as plt

class globe_toss:
    def __init__(self):
        self.toss_history = []
        self.probability_land = 0.5
        self.probability_water = 0.5
        self.prior_function = None

    def toss(self, number_toss):
        result = np.random.choice(["W", "L"], number_toss, p=[self.probability_water, self.probability_land])
        self.toss_history = np.append(self.toss_history, result)
        return

    def reset_history(self):
        self.toss_history = []
        return

    def get_water_total(self):
        return np.count_nonzero(self.toss_history == "W")

    def get_land_total(self):
        return np.count_nonzero(self.toss_history == "L")

    def get_binom_toss(self):
        k = self.get_water_total()
        n = len(self.toss_history)
        p = self.probability_water
        return binom.pmf(k, n, p)

    def get_binom_grid(self, x_grid):
        k = self.get_water_total()
        n = len(self.toss_history)
        return [binom.pmf(k, n, p) for p in x_grid]

    def analytical_solution(self, n_grid=100):
        a = self.get_water_total()
        b = self.get_land_total()
        return [beta.pdf(i, a + 1, b + 1) for i in np.linspace(0,1,n_grid)]

    def grid_calculate_posterior(self, n_grid=100):
        x_grid = np.linspace(0, 1, n_grid)
        if self.prior_function is None:
            y_prior = np.ones(n_grid)
        else:
            y_prior = np.array([self.prior_function(i) for i in x_grid])
        likelihood = self.get_binom_grid(x_grid)
        posterior = np.multiply(y_prior, likelihood) / np.sum(np.multiply(y_prior, likelihood))
        return x_grid, self.normalize_array(posterior)

    def normalize_array(self, array):
        array = np.array(array)
        arr_max = np.max(array)
        arr_min = np.min(array)
        interval = arr_max - arr_min

        result = (array - arr_min) / interval
        return result
    def register_prior_function(self, prior_function):
        self.prior_function = prior_function

    def grid_approx_plot(self):
        plt.figure(1, figsize = (11,6))
        plt.title("Posterior Distribution (Grid Approximation)")
        axes = plt.gca()
        axes.set_xlim([-0.1, 1.1])
        axes.set_xlabel("Probability of Water")
        axes.set_ylabel("Posterior")
        axes.grid()

        x_grid, posterior = self.grid_calculate_posterior(5)
        plt.plot(x_grid, posterior, label='5 Approx Normalized')

        x_grid, posterior = self.grid_calculate_posterior(10)
        plt.plot(x_grid, posterior, label='10 Approx Normalized')

        x_grid, posterior = self.grid_calculate_posterior(25)
        plt.plot(x_grid, posterior, label='25 Approx Normalized')

        x_grid, posterior = self.grid_calculate_posterior(100)
        plt.plot(x_grid, posterior, label='100 Approx Normalized')

        x_grid = np.linspace(0,1,100)
        plt.plot(
            x_grid,
            self.normalize_array(np.array(self.analytical_solution(100))),
            label='Analytical Normalized'
        )
        plt.legend()

    def quadratic_approximation_plot(self):
        prior = 1

        plt.figure(1, figsize = (11,6))
        plt.title("Posterior Distribution (Quadratic Approximation)")
        axes = plt.gca()
        axes.set_xlim([-0.1, 1.1])
        axes.set_xlabel("Probability of Water")
        axes.set_ylabel("Posterior")
        axes.grid()

        plt.plot(x_grid, posterior)

```

```python
toss1 = globe_toss()
# Override the history so it match with book's example
toss1.toss_history = np.array(['W', 'L', 'W', 'W', 'W', 'L', 'W', 'L', 'W'])
toss1.get_binom_toss()
```

    0.16406250000000006

Dalam ilmu probabilistik kemungkinan terjadinya dua kejadian pada saat yang sama dapat dituliskan sebagai

$$Pr(a, b) = Pr(a|b) Pr(b)$$
> Kemungkinan terjadinya kejadian $a$ dan $b$ adalah hasil perkalian dari kemungkinan kejadian $a$ apabila $b$ terjadi dan kemungkinan $b$ terjadi.

Dalam kasus yang kami miliki $a$ sama dengan $W, L$ atau proporsi dari darat dan laut--dapat digabungkan karena apabila $W$ diketahui maka $L$ juga dapat diketahui--dan $b$ adalah kemungkinan mendaratkan air $p$. Yang kami cari disini adalah probablitas dari pendaratan di air $p$ apabila jumlah $W$ dan $L$ diketahui. Sehingga dapat ditulis

$$
\begin{align}
    &Pr(W, L, p)  = Pr(W, L|p) Pr(p) \\
    &Pr(W, L, p)  = Pr(p | W, L) Pr(W, L) \\
    &Pr(W, L|p) Pr(p)  = Pr(p | W, L) Pr(W, L) \\
    &Pr(p|W, L)  = \frac{Pr(W, L| p) Pr(p)}{Pr(W, L)} \\
\end{align}
$$
Ada empat komponen dalam persamaan diatas

- $Pr(W, L|p)$ adalah apa yang disebut dengan **likelihood** data yang dimiliki untuk saat ini
- $Pr(p)$ adalah apa yang disebut dengan **prior** yaitu probabilitas yang awal sebelum diperbarui
- $Pr(W, L)$ adalah **averaging** agar probabilitas tidak lebih dari satu
- $Pr(p|W, L)$ adalah **posterior** probabilitas yang didapatkan setelah memperbarui dengan data yang tersedia

Berbekal pengetahuan ini kita dapat mengestimasi probabilitas dari pendaratan di air. Ada tiga metode numerik yang dapat diterapkan dalam estimasi ini yaitu *Grid Approximation*, *Quadratic Approximation*, dan *Markov Chain Monte Carlo*. Dimulai dengan Grid Approximation dahulu karena cukup intuitif.

```python
toss1.grid_approx_plot()
```

Grid approximation ini adalah memberikan sebuah vector dengan ukuran $n$ dan memiliki batas $[0, 1]$ dan memasukan masing-masing vector ke persamaan. Semakin tinggi bentuk vectorisasi-nya, maka akan semakin akurat juga.

> Ini kenapa kalo ngga di normalized semakin gede aproksimasinya semakin merunduk ya grafiknya?

Quadratic approximation adalah menggunakan distribusi normal untuk mencari bentuk distribusi normal mana yang paling mirip dengan posterior. Ini dapat dikerjakan dengan mencari rerata dan standar deviasi yang paling mewakili. Kami mencari rerata dengan mencari puncak dari estimasi. Puncak tersebut akan menjadi rerata dari distribusi normal.

### Chapter 2 Practice

**2M1**

```python
toss2 = globe_toss()
toss2.toss_history = np.array(['W', 'W', 'W'])
toss2.grid_approx_plot()
```

```python
toss3 = globe_toss()
toss3.toss_history = np.array(['W', 'W', 'W', 'L'])
toss3.grid_approx_plot()
```

```python
toss4 = globe_toss()
toss4.toss_history = np.array(['L', 'W', 'W', 'L', 'W', 'W', 'L'])
toss4.grid_approx_plot()
```

**Note** Mengapa nilai grid approksimasi dengan nominal kecil seperti 5 memiliki puncak berbeda dengan aproksimasi yang memiliki nominal lebih besar? Ini terjadi karena adanya normalisasi sehingga setiap puncak harus berada pada titik 1.

**2M2**

```python
def step_function(x):
    if x < 0.5:
        return 0
    elif x >= 0.5:
        return 1

toss5 = globe_toss()
toss5.register_prior_function(step_function)

toss5.toss_history = np.array(['W', 'W', 'W', 'L'])
toss5.grid_approx_plot()
```

Seperti yang dapat dilihat pada hasil aproksimasi, ada nilai cutoff pada 0.5 dikarenakan prior dengan nilai yang lebih rendah dari 0.5 nilainya 0 dan menghasilkan plot yang flat 0 pada nilai sebelum 0.5

**2M3**.

Berdasarkan deskripsi diatas dapat dipetakan menjadi dua probabilitas, yang pertama adalah probabilitas dari mana yang akan dilemparkan, bumi atau mars, yang sama sama bernilai 0.5. Selanjutnya adalah kejadian tempat mendaratnya, apakah mendarat pada tanah atau air. Untuk bumi kemungkinan mendarat pada tanah adalah 0.3 dan pada air adalah 0.7. Untuk mars, mendarat pada air adalah 0 dan mendarat pada darat adalah 1.

$Pr(Earth|land)$ dapat dihasilkan dari $\frac{Pr(Earth, land)}{Pr(land)}$. $Pr(land)$ dapat dihasilkan dari menjumlahkan kemungkinan land dan dibagi dengan asalnya

$$Pr_{land} = \frac{Pr_{land}^{mars} + Pr_{land}^{earth}}{2} = \frac{1 + 0.3}{2} = 0.65$$

$$Pr(Earth, land) = Pr(Earth) Pr_{land}^{earth} = 0.15 $$

$$Pr(Earth|land) = \frac{0.15}{0.65} = 0.23$$

**2M3**

Ada 6 kemungkinan yang bisa didapatkan dari penjumlahan kartu. Ini hasil dari perkalian jumlah kartu yang dimiliki dan sisi yang dimiliki setiap kartu. Yang ditanyakan adalah berapa kemungkinan sisi bawah kartu berwarna hitam apabila sisi atas kartu berwarna hitam.

Kemungkinan mendapatkan kartu hitam adalah $0.5$. Tapi dikarenakan yang ditanyakan adalah kemungkinan kartu sisi bawah adalah hitam menjadikan sampel kartu hanya 2--yang keduanya hitam dan yang memiliki sisi hitam dan putih.

$$
\begin{align}
Pr(Black, Black) &= Pr(Black | Black) Pr(Black) \\
\frac{1}{3} &= Pr(Black | Black) \frac{1}{2} \\
Pr(Black | Black) &=\frac{1}{3}2 \\
Pr(Black | Black) &=\frac{2}{3} \\
\end{align}
$$

```python
import itertools, pprint
import numpy as np
pp = pprint.PrettyPrinter(indent=4)

possible_draw = list(itertools.product('BW', repeat=2))
first_draw_possibilities = [ np.count_nonzero(np.array(i)[0] == 'B') for i in possible_draw]
second_draw_possibilities = [ np.count_nonzero(np.array(i)[1] == 'B') for i in possible_draw]
for num, i in enumerate(possible_draw):
    print(num, i, first_draw_possibilities[num], second_draw_possibilities[num])
```

**2H1**

Mencari tahu probabilitas kejadian apabila suatu kejadian terjadi dapat dikerjakan dengan distribusi geometeri. Penjabaran distribusi geometeri adalah sebagai berikut, berapa kejadian yang terjadi dibutuhkan untuk terjadinya suatu kejadian. Semisal pada suatu kasus pelemparan koin yang sudah diketahui biasnya--semisal kemungkinan keluarnya kepala adalah 0.3--pada pelemparan keberapa akan muncul kepala.

Ini dapat dituliskan dengan

$$Pr(X = h) = (1 - p)^kp $$

semakin tinggi nilai $k$--jumlah percobaan ke $k$ yang menghasilkan kepala pertama kali-- semakin rendah pula probabilitasnya. Ini terjadi karena semakin berjalanya percobaan semakin kecil kemungkinan pelemparan koin selalu menghasilkan buntut.

Apabila diterapkan dengan ditanyakan oleh soal, ada dua spesies yang memiliki dua perbedaan melahirkan kembar. Diantara dua spesies tersebut, harus dibandingkan mana yang lebih mungkin melahirkan bayi kembar.
Spesies A memiliki kemungkinan $10%$ dan spesies B memiliki kemungkinan $20$.

Untuk spesies A kita mendapat
$$Pr(a) = (1-p)^kp = 0.9^1 \times 0.1 = 0.09$$

Untuk spesies B kita mendapat
$$Pr(b) = (1-p)^kp = 0.8^1 \times 0.2 = 0.16$$

Untuk mengetahui kemungkinan terlepas dari spesies dapat dituliskan dengan $Pr(a) + Pr(b) = 0.09 + 0.16 = 0.25$

Selanjutnya ditanyakan berapa kemungkinan kelahiran yang kedua juga kembar. Dibuat peta kemungkinan dari kejadian kelahiran

| Stage 0 | Stage 1         | Stage 2  | Stage 3  |
|---------|-----------------|----------|----------|
| Start   | Spesies A (0.5) | SB (0.9) | SB (0.9) |
|    -    |       -         |    -     | TB (0.1) |
|    -    |       -         | TB (0.1) | SB (0.9) |
|    -    |       -         |    -     | TB (0.1) |
|    -    | Spesies B (0.5) | SB (0.8) | SB (0.8) |
|    -    |       -         |    -     | TB (0.2) |
|    -    |       -         | TB (0.2) | SB (0.8) |
|    -    |       -         |    -     | TB (0.2) |

Ada dua jalur yang menghasilkakn kelahiran kembar (TB) yaitu `Spesies A-TB-TB` dan `Spesies B-TB-TB` dan kemungkinan terjadinya masing-masing adalah $0.5 \times 0.1 \times 0.1 = 0.005$ dan $0.5 \times 0.2 \times 0.2 = 0.02$ dan djumlahkan akan menghasilkan $0.025$. Angka yang didapatkan adalah probalitas mendapatkan dua kali kelahiran kembar **bukan** kelahiran kembar apabila sebelumnya adalah kembar.

$$
\begin{align}
    P(\text{TB} | \text{TB}) &= \frac{P(TB, TB)}{P(TB)} \\
    P(\text{TB}, \text{TB}) &= P(A, (\text{TB}, \text{TB})) + P(B, (\text{TB}, \text{TB})) = 0.005 + 0.02 = 0.025 \\
    P(TB) &= P(TB, B) + P(TB, A) = 0.5 \times 0.1 + 0.5 \times 0.2 = 0.15 \\
    P(\text{TB} | \text{TB}) &= \frac{0.025}{0.15} = \frac{1}{6} \\
\end{align}
$$

**2H2**
Menghitung kemungkinan bahwa yang melahirkan kembar untuk kedua kali adalah spesies A dapat ditulis sebagai $P(A|\text{TB})$. Diketahui bahwa $P(A, \text{TB}))$ adalah $0.5 \times 0.1 = 0.05$
$$
\begin{align}
    P(A|\text{TB}) &= \frac{P(A, \text{TB})}{P(A)} = \frac{0.05}{0.5} = 0.1 \\
    P(B|\text{TB}) &= \frac{P(B, \text{TB})}{P(B)} = \frac{0.1}{0.5} = 0.2 \\
    \\
    P(A|\text{SB}) &= \frac{P(A, \text{SB})}{P(A)} = \frac{0.45}{0.5} = 0.9 \\
    P(B|\text{SB}) &= \frac{P(B, \text{SB})}{P(B)} = \frac{0.4}{0.5} = 0.8 \\
\end{align}
$$

Dikarenakan $\text{TB}$ sudah menjadi ruang kemungkinan sendiri maka hasil dari $P(A|\text{TB})$ dan $P(B|\text{TB})$ dapat dinormalisasi dengan ketentuan $P(A|\text{TB}) + P(B|\text{TB}) = 1$. Setelah normalisasi, $P(A|\text{TB}) = \frac{1}{3}$ dan $P(B|\text{TB}) = \frac{2}{3}$

**2H3**
Berdasarkan informasi yang sudah dikerjakan diatas, diketahui bahwa kelahiran kedua adalah tidak kembar. Menggunakan baeysian, dapat ditulis

$$P_{posterior}(A|(\text{TB}, \text{SB})) = \frac{P((\text{TB},\text{SB}|A)\times P_{prior}(A|\text{TB})}{P(\text{TB},\text{SB})}$$

Masing-masing bisa didapatkan dari peta kemungkinan dan soal sebelumnya

$$
\begin{align}
    P_{prior}(A|\text{TB}) &= \frac{1}{3} \\
    P(\text{TB},\text{SB}) &= (0.1 \times 0.9) + (0.2 \times 0.8) = 0.25 \\
    P((\text{TB},\text{SB})|A) &= 0.1 \times 0.9 = 0.09 \\
    P_{posterior}(A|(\text{TB}, \text{SB})) &= \frac{0.09 \times \frac{1}{3}}{0.25} = 0.12
\end{align}
$$

Untuk posterior apabila melahirkan kembar dua kali

$$
\begin{align}
    P_{prior}(A|\text{TB}) &= \frac{1}{3} \\
    P(\text{TB},\text{TB}) &= (0.1 \times 0.1) + (0.2 \times 0.2) = 0.05 \\
    P((\text{TB},\text{TB})|A) &= 0.1 \times 0.1 = 0.01 \\
    P_{posterior}(A|(\text{TB}, \text{TB})) &= \frac{0.01 \times \frac{1}{3}}{0.05} = 0.066
\end{align}
$$

**2H4**
Ada tes genetik yang dapat menentukan bahwa spesies yang diteliti adalah A atau B. Tes tersebut memiliki probabilitas berhasil 0.8 untuk spesies A dan 0.65 untuk spesies B. Tes mengeluarkan hasil bahwasanya spesies yang dieteliti adalah spesies A.

$$
\begin{align}
    P(A|\text{GT}_A) &= \frac{P(A, \text{GT}_A)}{P(A)} = \frac{0.5 \times 0.8}{0.5} = 0.8
\end{align}
$$

Menggunakan informasi sebelumnya bahwa spesies tersebut sudah melahirkan kembar sekali dan non kembar sekali maka dapat ditulis

$$P_{posterior}(A|(\text{TB}, \text{SB}, \text{GT}_A)) = \frac{P((\text{TB},\text{SB}, \text{GT}_A)|A)\times P_{prior}(A|(\text{TB},\text{SB}))}{P(\text{TB},\text{SB}, \text{GT}_A)}$$

$$
\begin{align}
    P_{prior}(A|(\text{TB},\text{SB})) &= 0.12 \\
    P(\text{TB},\text{SB}, \text{GT}_A) &= (0.1 \times 0.9 \times 0.8) + (0.2 \times 0.8 \times 0.2) = 0.104 \\
    P((\text{TB},\text{SB}, \text{GT}_A)|A) &= 0.1 \times 0.9 \times 0.8 = 0.072 \\
    P_{posterior}(A|(\text{TB}, \text{SB})) &= \frac{0.072 \times 0.12}{0.104} = 0.083
\end{align}
$$

Apabila dilihat dari sebelumnya maka posterior lebih besar dari sebelum adanya $GT_A$ ini menandakan hasil dari tes genetik menambah kepercayaan dari $0.066$ menjadi $0.083$


# Use PY in Linear Algebrea

## Chapter Zero 乘法和庫

### 矩陣和向量的乘法
1. 點乘（dot product）：也被稱為「數量積」「內積」「點積」。結果的絕對值是一個向量在另一個向量方向上的投影的長度誠意另一個向量的長度，是一個標量。
2. 叉乘（cross product）：也被稱為「向量積」「外積」。對於兩個 $n$ 緯的向量叉乘，結果是一個和 $n-1$ 個向量都垂直的向量，方向符合右手定則。
3. 矩陣乘法：代表了線性變換的過程。

### python 的 sympy 库 和 numpy 库
说明：sympy 用於模擬和理解，numpy 用於实践。

#### sympy
`sympy`使用`*`代表矩陣乘法；

`sympy`使用`dot`表示點乘；

`sympy`使用`cross`表示叉乘

`det()`表示行列式；`inv()`表示擬矩陣；`adjugate()`伴隨矩陣

#### numpy
`numpy`使用`*`表示`array`乘法默認是按照位置的乘法，矩陣`matrix`默認的是矩陣乘法，混合起來的話默認是矩陣乘法；

`numpy`使用`dot`表示矩陣`matrix`和`array`的矩陣乘法；

`numpy`使用`cross`表示叉乘；

`numpy`使用`det()`表示行列式；

`.T` 返回自身的轉置；

`.H` 返回自身的共軛轉置；

`.J` 返回自身的逆矩陣；

`.A` 返回自身數據的2維數組的一個視圖（沒有做任何的拷貝）

## Chapter One Matrix

數學上，一個$m\times n$的矩陣是由一個$m$行(row)$n$列(column)元素排列成的矩形陣列，矩陣裡的元素可以是數字、符號或者數學式。在圖像處理、人工智能燈領域，使用矩陣來表示和處理數據非常常見。例：

$\mathbf{A}_{2 \times 3}=\begin{bmatrix} 5 & 2 & 7 \\ 1 & 3 & 4  \end{bmatrix}$

其中，矩陣$A$的下標$2\times 3$表示$A$是一個2行3列的矩陣。類似的，另一個示例：

$\mathbf{ B }_{ 4 \times 4 }=\begin{bmatrix} 5 & 2 & 7 & 6 \\ 1 & 3 & 4 & 2 \\ 7 & -1 & 9 & 0 \\ 8  & 2 & -2 & 3 \end{bmatrix}$

特別的，如果我們要表示矩陣$A$的第二行第二個元素：3，可以使用$A[2,2]$或者$a_{2,2}$。

在`python`中的`numpy`庫中提供了`ndarrary`類用於儲存高維數組及普通的數組運算，另外提供了`matrix`類用來支持矩陣運算（矩陣乘法）。下面來舉個例子：

```python
#first method to store the matrix
import numpy as np

a = np.matrix('5 2 7;1 3 4')
b = np.matrix('15 2 7 6;1 3 4 2;7 -1 9 0;8 2 -2 3')
print(a)#print matrix a
print(b)#print matrix b
```

    [[5 2 7]
     [1 3 4]]
    [[15  2  7  6]
     [ 1  3  4  2]
     [ 7 -1  9  0]
     [ 8  2 -2  3]]
    [[5 2 7]
     [1 3 4]]
    [[15  2  7  6]
     [ 1  3  4  2]
     [ 7 -1  9  0]
     [ 8  2 -2  3]]

```python
#second method to store the matrix
import numpy as np

a = np.matrix([[5,2,7],[1,3,4]])
b = np.matrix([[5,2,7,6],[1,3,4,2],[8,2,-2,3]])
print(a)#print matrix a
print(b)#print matrix b
```

    [[5 2 7]
     [1 3 4]]
    [[ 5  2  7  6]
     [ 1  3  4  2]
     [ 8  2 -2  3]]
    [[5 2 7]
     [1 3 4]]
    [[ 5  2  7  6]
     [ 1  3  4  2]
     [ 8  2 -2  3]]

這兩種形式完全等效，但是第一種更加直觀，不容易犯錯，所以推薦第一種方式（也很類似`LaTeX`中構成矩陣的寫法）；

要把一個`matrix`的對象轉換為`ndarray`對象，可以直接用`getA()`的方法。如果吧`ndarray`對象轉換成`matrx`對象可以用`asmatrix()`方法。

```python
b = a.getA()#turn a into ndarray
print(b)
print(type(b))# the type of b
c = np.asmatrix(b)
print(c)
print(type(c))# the type of c
```

    [[5 2 7]
     [1 3 4]]
    <class 'numpy.ndarray'>
    [[5 2 7]
     [1 3 4]]
    <class 'numpy.matrixlib.defmatrix.matrix'>
    [[5 2 7]
     [1 3 4]]
    <class 'numpy.ndarray'>
    [[5 2 7]
     [1 3 4]]
    <class 'numpy.matrixlib.defmatrix.matrix'>

如果你要取出矩陣中的某個值，可以使用類似於數組的下標運算符。但是要注意，計算機的計數方式是從0開始的。例如，要取出$A[2,2]$，應該使用：

```python
a[1,1]
```

    3

    3

### 運算

#### 加
矩陣加法的定義非常符合直覺。假設有$\mathbf{ A }_{ 3 \times 3 }=\begin{bmatrix} 1 & 0 & 1 \\ 1 & 2 & 1 \\ 2 & 1 & 1 \end{bmatrix}$，$\mathbf{ B }_{ 3 \times 3 }=\begin{bmatrix} 2 & 1 & -1 \\ 0 & -1 & 2 \\ 2 & -1 & 0 \end{bmatrix}$，
 則：
$\mathbf{A}+\mathbf{B} = \begin{bmatrix} 1 & 0 & 1 \\ 1 & 2 & 1 \\ 2 & 1 & 1 \end{bmatrix} + \begin{bmatrix} 2 & 1 & -1 \\ 0 & -1 & 2 \\ 2 & -1 & 0 \end{bmatrix} = \begin{bmatrix} 1+2 & 0+1 & 1+(-1) \\ 1+ 0 & 2+(-1) & 1+2 \\ 2+2 & 1+(-1) & 1+0 \end{bmatrix} = \begin{bmatrix} 3 & 1 & 0 \\ 1 & 1 & 3 \\ 4 & 0 & 1 \end{bmatrix}$

要注意，兩個矩陣的行數和列數必須相同，否則無定義。下面是`python`示例：

```python
a = np.matrix('1 0 1;1 2 1;2 1 1')
b = np.matrix('2 1 -1;0 -1 2;2 -1 0')
print(a+b)# print a+b
```

    [[3 1 0]
     [1 1 3]
     [4 0 1]]
    [[3 1 0]
     [1 1 3]
     [4 0 1]]

很容易看出，矩陣的加法滿足交換律和結合律，即$A+B=B+A$，$(A+B)+C=A+(B+C)$。

#### 減
矩陣減法也和加法一樣簡單。對於上面給出的$A$和$B$，有：

$\mathbf{A}-\mathbf{B}=\begin{bmatrix} 1 & 0 & 1 \\ 1 & 2 & 1 \\ 2 & 1 & 1 \end{bmatrix}-\begin{bmatrix} 2 & 1 & -1 \\ 0 & -1 & 2 \\ 2 & -1 & 0 \end{bmatrix}=\begin{bmatrix} 1-2 & 0-1 & 1-(-1) \\ 1-0 & 2-(-1) & 1-2 \\ 2-2 & 1-(-1) & 1-0 \end{bmatrix}=\begin{bmatrix} -1 & -1 & 2 \\ 1 & 3 & -1 \\ 0 & 2 & 1 \end{bmatrix}$

同樣，相減的兩個矩陣行數和列數必須完全相同，否則無定義。下面是`python`示例：

```python
print(a-b)
```

    [[-1 -1  2]
     [ 1  3 -1]
     [ 0  2  1]]
    [[-1 -1  2]
     [ 1  3 -1]
     [ 0  2  1]]

#### 乘
矩陣乘法的定義是$A_{i\times j}$矩陣的每一行的元素分別與$B_{j\times k}$矩陣的每一列的元素兩兩相乘並詳加，從而，得到新的矩陣$C_{j\times k$。兩個矩陣能相乘的充份必要條件是第一個矩陣的列數等於第二個矩陣的行數相等，否則，無定義。$\begin {aligned}
\mathbf{A} \times \mathbf{B} &=\begin{bmatrix} 1 & 0 & 1 \\ 1 & 2 & 1 \\ 2 & 1 & 1 \end{bmatrix}\times \begin{bmatrix} 2 & 1 & -1 \\ 0 & -1 & 2 \\ 2 & -1 & 0 \end{bmatrix} \\\ &=\begin{bmatrix} 1\cdot 2+0\cdot 0+1\cdot 2 & 1\cdot 1+0\cdot (-1)+1\cdot (-1) & 1\cdot (-1)+0\cdot 2+1\cdot 0 \\ 1\cdot 2+2\cdot 0+1\cdot 2 & 1\cdot 1+2\cdot (-1)+1\cdot (-1) & 1\cdot (-1)+2\cdot 2+1\cdot 0 \\ 2\cdot 2+1\cdot 0+1\cdot 2 & 2\cdot 1+1\cdot (-1)+1\cdot (-1) & 2\cdot (-1)+1\cdot 2+1\cdot 0 \end{bmatrix}\\\ &=\begin{bmatrix} 4 & 0 & -1 \\ 4 & -2 & 3 \\ 6 & 0 & 0 \end{bmatrix}
\end {aligned}$

再舉一個行列數不同的例子，假設有$\mathbf{C}_{2 \times 3} = \begin{bmatrix} 5 & 7 & 2 \\ 4 & 3 & 1 \end{bmatrix}$和$\mathbf{D}_{3 \times 1} = \begin{bmatrix} 1 \\ 5 \\ 6 \end{bmatrix}$，那麼，我們可以得出：$\mathbf{C}\times \mathbf{D} = \begin{bmatrix} 5 & 7 & 2 \\ 4 & 3 & 1 \end{bmatrix}\times \begin{bmatrix} 1 \\ 5 \\ 6 \end{bmatrix}
 =\begin{bmatrix} 5 \cdot 1+ 7 \cdot 5+ 2\cdot 6 \\ 4\cdot 1+3\cdot 5+1\cdot 6  \end{bmatrix} =\begin{bmatrix} 52 \\ 25 \end{bmatrix}$

與初等代數中學的乘法不同，矩陣的乘法並不滿足交換律，即，$A \times B \neq B \times A$，但是，它滿足分配律：$(A\times B)\times C = A\times (B\times C)$。

再介紹兩個特殊的矩陣：

1. 單位矩陣（Identity Matrix）他的特點是與行數和列數相等，對角線上的值都是1，其他值都是0。另一個特點是：它與任意矩陣$A$相乘結果等於$A$。

$\mathbf{I}_{3 \times 3} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$這是一個 $3\times 3$的單位矩陣

2. 零矩陣（Null Matrix）顧名思義就是全部元素都是0的矩陣。零矩陣乘以任何矩陣都是零矩陣，與任何矩陣$A$都等於$A$。

```python
#python示例
print("this is a times b:")
print(a*b)
print("this is b times a:")
print(b*a)
```

    this is a times b:
    [[ 4  0 -1]
     [ 4 -2  3]
     [ 6  0  0]]
    this is b times a:
    [[ 1  1  2]
     [ 3  0  1]
     [ 1 -2  1]]
    this is a times b:
    [[ 4  0 -1]
     [ 4 -2  3]
     [ 6  0  0]]
    this is b times a:
    [[ 1  1  2]
     [ 3  0  1]
     [ 1 -2  1]]

```python
c = np.matrix('5 7 2;4 3 1')
d = np.matrix('1;5;6')
print("this is c times d:")
print(c*d)
print("this is a times b times d:")
print(a*b*d)
print("first,we user b times d, then we use a to times the b*d:")
print(a*(b*d))
I = np.eye(3)
#we defined I as a 3 by 3 indentity matrix;eye() equals to'I = np.matrix('1 0 0;0 1 0;0 0 1')'
print("This is a times I:")
print(a*I)
print("This is I times a:")
print(I*a)
z = np.zeros([3,3])
#we define z as a 3 by 3 null matrix; np.zeros() equals to'z = np.matrix('0 0 0;0 0 0;0 0 0')'
print("This is a times z:")
print(a*z)
print("This is z times a:")
print(z*a)
```

    this is c times d:
    [[52]
     [25]]
    this is a times b times d:
    [[-2]
     [12]
     [ 6]]
    first,we user b times d, then we use a to times the b*d:
    [[-2]
     [12]
     [ 6]]
    This is a times I:
    [[ 1.  0.  1.]
     [ 1.  2.  1.]
     [ 2.  1.  1.]]
    This is I times a:
    [[ 1.  0.  1.]
     [ 1.  2.  1.]
     [ 2.  1.  1.]]
    This is a times z:
    [[ 0.  0.  0.]
     [ 0.  0.  0.]
     [ 0.  0.  0.]]
    This is z times a:
    [[ 0.  0.  0.]
     [ 0.  0.  0.]
     [ 0.  0.  0.]]
    this is c times d:
    [[52]
     [25]]
    this is a times b times d:
    [[-2]
     [12]
     [ 6]]
    first,we user b times d, then we use a to times the b*d:
    [[-2]
     [12]
     [ 6]]
    This is a times I:
    [[ 1.  0.  1.]
     [ 1.  2.  1.]
     [ 2.  1.  1.]]
    This is I times a:
    [[ 1.  0.  1.]
     [ 1.  2.  1.]
     [ 2.  1.  1.]]
    This is a times z:
    [[ 0.  0.  0.]
     [ 0.  0.  0.]
     [ 0.  0.  0.]]
    This is z times a:
    [[ 0.  0.  0.]
     [ 0.  0.  0.]
     [ 0.  0.  0.]]

#### ~~除~~（求逆）

矩陣中沒有除法這種操作，但是有一個相似的運算，叫做求逆。在線性代數中，給定一個$n$階的方陣$A$ ，若存在一個$n$階矩陣$B$，使得$AB = BA = I_n$，其中$I_n$為$n$階的單位矩陣，則稱$A$是**可逆**的，而$B$則是$A$的逆矩陣，記作：$A^{-1}$。

##### 伴隨陣法：
$\mathbf{A}^{-1}=\frac{1}{|\mathbf{A}|}\begin{bmatrix}A_{11} & A_{21} & \ldots & A_{n1} \\ A_{12} & A_{22} & \ldots & A_{n2} \\ \ldots & \ldots & \ldots & \ldots \\ A_{1n} & A_{2n} & \ldots & A_{nn} \end{bmatrix}$，其中$A_{ij}$是$|A|$中元素$a_{ij}$的代數余子式。

矩陣$\begin{bmatrix}A_{11} & A_{21} & \ldots & A_{n1} \\ A_{12} & A_{22} & \ldots & A_{n2} \\ \ldots & \ldots & \ldots & \ldots \\ A_{1n} & A_{2n} & \ldots & A_{nn} \end{bmatrix}$稱為矩陣$A$ 的伴隨矩陣，記作$A*$

##### 初等變換法：
如果矩陣$A$和$B$互逆，則$AB=BA=I$。由$AB=BA$可以得出：兩個矩陣都是方陣；由$AB=I$以及定理“兩個矩陣的乘積的行列式等於兩個矩陣的行列式的乘積”可得：兩個矩陣的行列式不等於零。現在假設三階矩陣${ \mathbf{A} }_{ 3 \times 3 }=\begin{bmatrix} 1 & 0 & 1 \\ 1 & 2 & 1 \\ 2 & 1 & 1 \end{bmatrix}$，則：

$\begin{aligned}
\begin{bmatrix}\mathbf{A} \mathbf{I}\end{bmatrix}
& \rightarrow
\begin{bmatrix}
1 & 0 & 1 & 1 & 0 & 0 \\
1 & 2 & 1 & 0 & 1 & 0 \\
2 & 1 & 1 & 0 & 0 & 1
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 0 & 1 & 1 & 0 & 0 \\
0 & 2 & 0 & -1 & 1 & 0 \\
2 & 1 & 1 & 0 & 0 & 1
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 0 & 1 & 1 & 0 & 0 \\
0 & 1 & 0 & -0.5 & 0.5 & 0 \\
2 & 1 & 1 & 0 & 0 & 1
\end{bmatrix}\\
& \rightarrow
\begin{bmatrix}
1 & 0 & 1 & 1 & 0 & 0 \\
0 & 1 & 0 & -0.5 & 0.5 & 0 \\
1 & 1 & 0 & -1 & 0 & 1
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 0 & 1 & 1 & 0 & 0 \\
0 & 1 & 0 & -0.5 & 0.5 & 0 \\
1 & 0 & 0 & -0.5 & -0.5 & 1
\end{bmatrix}
\rightarrow
\begin{bmatrix}
0 & 0 & 1 & 1.5 & 0.5 & -1 \\
0 & 1 & 0 & 0 & 0.5 & 0 \\
1 & 0 & 0 & -0.5 & -0.5 & 1
\end{bmatrix}\\
&\rightarrow
\begin{bmatrix}
1 & 0 & 0 & -0.5 & -0.5 & 1 \\
0 & 1 & 0 & -0.5 & 0.5 & 0 \\
0 & 0 & 1 & 1.5 & 0.5 & -1
\end{bmatrix}
\end{aligned}$

所以：$\mathbf{A}^{-1}=\begin{bmatrix}-0.5 & -0.5 & 1 \\ -0.5 & 0.5 & 0 \\ 1.5 & 0.5 & -1\end{bmatrix}$

### 奇異矩陣（Singular Matrix）
如果一個方陣$A$滿足條件$|A|(det(A))\neq 0$，則稱$A$為非奇異矩陣（non-singular matrix），否則則稱為奇異矩陣。簡單的說**沒有逆矩陣的矩陣就是奇異矩陣，有逆矩陣的矩陣就是非奇異矩陣**

`python`求逆示例：

```python
a = np.matrix('1 0 1; 1 2 1; 2 1 1')
print(a.I) #the inverse of matrix a
print(a*a.I) #a times a inverse equals to indentity matrix
f = np.matrix('0 1;0 0')
print(f.I)#because f has no inverse, f is a singular matrix
```

### 矩陣的轉置（Transport）
在線性代數中，矩陣$A$的轉置是另一個矩陣$A^T$，如下所示：

* 把$A$的橫行寫成$A^T$的縱列；
* 把$A$的縱列寫成$A^T$的橫行；
形式上說就是，$m\times n$的矩陣$A$的轉置是$n\times m$矩陣。

矩陣$\underset{m\times n}{\mathbf{A}} = \begin{bmatrix}a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & \ldots & a_{2n} \\ \ldots \\ a_{m1} & a_{m2} & \ldots & a_{mn}\end{bmatrix}$的轉置定義為$\underset{n\times n}{A^{T}} = \begin{bmatrix}a_{11} & a_{21} & \ldots & a_{m1} \\ a_{12} & a_{22} & \ldots & a_{m2} \\ \ldots \\ a_{1n} & a_{2n} & \ldots & a_{mn}\end{bmatrix}$

```python
#python 示例：
a = np.matrix('2 4;1 3')
print(a.T)#the transport of a
b = np.matrix('1 2 3;4 5 6')
print(b.T)#the transport of b
```

    [[2 1]
     [4 3]]
    [[1 4]
     [2 5]
     [3 6]]
    [[2 1]
     [4 3]]
    [[1 4]
     [2 5]
     [3 6]]

**特別的，矩陣的轉置有一個性質：矩陣的轉置等於矩陣調換後分別做轉置的乘積：$（A\bullet B)^T = B^T\bullet A^T$**

```python
a = np.matrix('2 4;1 3')
b = np.matrix('1 6;2 5')
c = a*b
print(c.T)
print(b.T*a.T)
```

    [[10  7]
     [32 21]]
    [[10  7]
     [32 21]]
    [[10  7]
     [32 21]]
    [[10  7]
     [32 21]]

```python
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

xlim(-3, 3)
ylim(0, 3.5)
plt.plot([0,2.5],[3.5,0],'blue',[-1,3],[0,3.5],'red')
```

    [<matplotlib.lines.Line2D at 0x7605c410>,
     <matplotlib.lines.Line2D at 0x6f831830>]

    [<matplotlib.lines.Line2D at 0x7605c410>,
     <matplotlib.lines.Line2D at 0x6f831830>]

可以用矩陣表示成：$\begin{bmatrix}
3 & 2 \\
-1 & 1
\end{bmatrix}
\begin{bmatrix}
x \\
y
\end{bmatrix}
=
\begin{bmatrix}
7\\
1
\end{bmatrix}$

設$\begin{bmatrix}3 & 2 \\-1 & 1\end{bmatrix}$為矩陣$A$，將等式兩邊左乘一個$A$的逆，就可以得到：

$\begin{aligned}
A^{-1}A
\begin{bmatrix}
x \\ y
\end{bmatrix}
&=
A^{-1}
\begin{bmatrix}
7\\
1
\end{bmatrix}\\\
&=
\frac{1}{|A|}\begin{bmatrix}1 & -2 \\ 1 & 3\end{bmatrix}
\begin{bmatrix}
7\\
1
\end{bmatrix}\\\
&=
\frac{1}{5}\begin{bmatrix}1 & -2 \\ 1 & 3\end{bmatrix}
\begin{bmatrix}
7\\
1
\end{bmatrix}\\\
&=
\frac{1}{5}\begin{bmatrix}5 \\ 10\end{bmatrix}
\end{aligned}$

因此，$\begin{bmatrix}x \\ y\end{bmatrix}=\begin{bmatrix}1 \\ 2\end{bmatrix}$

雖然這個方法在二元一次方程這看起來還沒有初中學的校園法解得快，但使用矩陣的好處在於對於更高維的數據，比如成白上千的未知數，這個解法依然有效。

在`python`中，我們可以使用`numpy`的線性代數算法庫`linalg`提供的`solve`方法來求解方程組。例：

```python
a = np.matrix('3 2; -1 1')
b = np.matrix('7; 1')
print(np.linalg.solve(a, b))
```

    [[ 1.]
     [ 2.]]
    [[ 1.]
     [ 2.]]

### 求向量組合
假設有向量$\vec { a }  =  \begin{bmatrix} 3 \\ -1 \end{bmatrix}$，$\vec { b }  =  \begin{bmatrix} 2 \\ 1 \end{bmatrix}$，求兩者如何組成新向量$\vec { c }  =  \begin{bmatrix} 7 \\ 1 \end{bmatrix}$？

```python
import numpy as np
import matplotlib.pyplot as plt

soa = np.array([[0, 0, 3, -1], [0, 0, 2, 1], [0, 0, 7, 1]])
X, Y, U, V = zip(*soa)
plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([-1, 10])
ax.set_ylim([-1, 10])
plt.draw()
plt.show()
```

如果用$x$和$y$分別表示兩個向量的背書，這個問題就可以用矩陣表示成：$\begin{bmatrix}3 \\-1 \end{bmatrix}x + \begin{bmatrix}2 \\1\end{bmatrix}y=\begin{bmatrix}7\\1\end{bmatrix}$

這樣就把這個問題轉換成和上一個問題同構了，使用同樣的方法可以得出：$\begin{bmatrix}x \\ y\end{bmatrix}=\begin{bmatrix}1 \\ 2\end{bmatrix}$

```python
import numpy as np
import matplotlib.pyplot as plt

soa = np.array([[0, 0, 3, -1], [0, 0, 4, 2], [0, 0, 7, 1],[4,2,3,-1]])
X, Y, U, V = zip(*soa)
plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([-1, 10])
ax.set_ylim([-1, 10])
plt.draw()
plt.show()
```

## Chapter Two  Determinant

### 什麼是行列式

**行列式**（Dterminant），在顯示代數中，行列式是一個值，一個數值可以告訴我們一個矩陣是否是奇異的。對於每一個$n\times n$的矩陣$A$，據對應一個標量$det(A)$，它的值將告訴我們矩陣是否為非奇異的。

1. $2\times 2$ 矩陣的行列式值

$det(\
\left[\begin{array}{cc}
    a & b\\
    c & d\\
\end{array}\right]
\  ) = ad - bc$

我們也可以用`sympy`進行核對：

```python
from sympy import *

a,b,c,d = symbols("a,b,c,d")
M1=Matrix([[a, b], [c, d]])
print(M1.det())# print the determinant of matrix M1
```

    a*d - b*c
    a*d - b*c

### 行列式的幾何意義：二維和三維歐式空間中的例子

行列式的一個自然的怨氣就是$n$維平行體的體積。行列式的定義和$n$維平行體的體積有著本質上的關聯。

在二維空間中，行列式如上面例子所示；比如說：$det(A,B) =\
\left|\begin{array}{cc}
    2 & 3\\
    1 & 4\\
\end{array}\right|
\ = 2\bullet 4 - 3\bullet 1 = 5$，我們經過計算可以知道，當係數是實數的時候，行列式表示的就是向量$\vec A$和$\vec B$ 形成的平行四邊形的**有向面積**

```python
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

plt.plot([0,2],[0,1],'blue',[3,5],[4,5],'blue',[0,3],[0,4],'red',[2,5],[1,5],'red')
```

```python
import numpy as np

a = np.matrix([[2,1],[3,4]])
print(np.linalg.det(a))# print the determinant of matrix M1
```

    5.0
    5.0

我們通過計算，可以得到這個矩陣的行列式等於$5$，可以看出，上圖由紅線和藍線形成的是一個平行四邊形，我們可以通過計算得出，它的面積等於$5$

*這裡只是簡單介紹二維行列式，具體請查閱課本**

那如果是三位空間呢？在三維的有向空間中，三個三維向量的行列式是$\left|\begin{array}{ccc}
    x & x' & x''\\
    y & y' & z''\\
    z & z' & z''
\end{array}\right|
\ = xy'z'' + x'y''z + x''yz' - xy''z' - xyz'' - x''y'z $

比如說，三個向量$(2,1,5),(6,0,8)$ 和 $(3,2,4)$ 的行列式為：
$\left|\begin{array}{ccc}
    2 & 6 & 3\\
    1 & 0 & 2\\
    5 & 8 & 4
\end{array}\right| = 28
$

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

p0 = [2, 1,5]
p1 = [6,0,8]
p2 = [3,2,4]

origin = [0,0,0]
X, Y, Z = zip([0,0,0],origin,origin)
U, V, W = zip(p0,p1,p2)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.quiver(X,Y,Z,U,V,W,arrow_length_ratio=0.001)
ax.set_xlim([-1, 5])
ax.set_ylim([-1, 5])
ax.set_zlim([-1, 8])

plt.show()
```

類比一下二維向量的行列式，我們可以猜測一下三維向量組的行列式的幾何意義，就是這三個向量行程的平行六面體的**有向體積**
；我們可以用`python`來算它的行列式：

```python
a = np.matrix([[2,1,5],[6,0,8],[3,2,4]])
print(np.linalg.det(a))
```

    28.0
    28.0

***特別的，如果向量組中，所有向量共線或者共面（線性相關），面積或體積為零**，什麼是線性相關，我們會在後面解釋*

## Chapter Three Vector

### What is Vector

在初中（或高中）物理，我們就已經接觸過向量了，那向量到底是什麼？它是指一個同事具有大小和方向，且滿足平行四邊形法則的集合對象。一般地，同時滿足具有大小和方向兩個性質的集合對象就可以稱作為向量（物理中的電流：有方向，有大小，但是它不是向量，這是特殊情況，我們在線性代數中不討論）。與向量相對的叫做**標量**（scalar），一般標量和向量最大的區別就是是否有向。

在$R^n$空間中定義$\vec V$，可以用一個包含$n$個實數的有序集來表示：$\vec V = \begin{bmatrix} v_1 \\ v_2 \\ \dots \\ v_n\end{bmatrix}$，這個有序集中的每一個元素被稱為向量的分量（在某一維度的量）。例如，一個在$R^2$空間的向量$\begin{bmatrix} 2 \\ 1 \end{bmatrix}$,我們也可以用$(2,1)$或$<2,1>$來表示：

```python
import numpy as np
import matplotlib.pyplot as plt

soa = np.array([[0, 0, 2, 1]])
X, Y, U, V = zip(*soa)
plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([0, 5])
ax.set_ylim([0, 5])
plt.draw()
plt.show()
```

向量的長度被定義為$\parallel v \parallel = \sqrt{v_1^2 + v_2^2 +\dots + v_n^2}$；當然的，當$\parallel v \parallel = 1$，我們稱這個向量是**單位向量**

### 基本運算

#### 加法

向量$a$和向量$b$的加法定義為：$\mathbf{a} + \mathbf{b} = \begin{bmatrix} a_1 + b_2 \\ a_2 + b_2 \\ \dots \\a_n + b_n \end{bmatrix}$

下面通過畫圖來示意向量$\mathbf{a} = \begin{bmatrix} -1 \\ 2 \end{bmatrix}$與$\mathbf{b} = \begin{bmatrix}3 \\ 1\end{bmatrix}$相加，等於$\begin{bmatrix} 2 \\ 3 \end{bmatrix}$

```python
import numpy as np
import matplotlib.pyplot as plt

soa = np.array([[0, 0, -1, 2], [0, 0, 3, 1], [0, 0, 2, 3],[3,1,-1,2]])
X, Y, U, V = zip(*soa)
plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([-2, 5])
ax.set_ylim([-1, 5])
plt.draw()
plt.show()
```

可以看出，向量的加法是遵從平行四邊形法則的，在`python`中，我們可以直接用`numpy`的`ndarray`來表示向量：

```python
import numpy as np
a = np.array([-1,2])
b = np.array([3,1])
print(a+b)
```

    [2 3]
    [2 3]

#### 減法

了解了加法了，減法就相當於加了一個反方向的向量，$\mathbf{a} - \mathbf{b} = \begin{bmatrix} a_1 - b_1 \\ a_2 - b_2 \\ \ldots \\ a_n - b_n \end{bmatrix}$，下面通過畫圖來示意向量$\mathbf{a} = \begin{bmatrix} -1 \\ 2 \end{bmatrix}$與$\mathbf{b} = \begin{bmatrix}3 \\ 1\end{bmatrix}$相減，等於$\begin{bmatrix} -4 \\ 1 \end{bmatrix}$

```python
import numpy as np
import matplotlib.pyplot as plt

soa = np.array([[0, 0, -1, 2], [0, 0, -3, -1], [0, 0, -4, 1],[-3,-1,-1,2]])
X, Y, U, V = zip(*soa)
plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([-5, 3])
ax.set_ylim([-3, 3])

plt.show()
```

```python
import numpy as np
a = np.array([-1, 2])
b = np.array([3, 1])
print(a - b)

```

    [-4  1]
    [-4  1]

#### 乘法

##### 標量乘以向量

標量$c$乘以向量$\mathbf{a}$定義為：$c \cdot \mathbf{a} = \begin{bmatrix} c \cdot a_1 \\ c \cdot a_2 \\ \ldots \\ c \cdot a_n \end{bmatrix} = \begin{bmatrix} a_1 \cdot c \\ a_2 \cdot c \\ \ldots \\ a_n \cdot c \end{bmatrix}$

用圖來示意$\mathbf{a} \begin{bmatrix} -1 \\2 \end{bmatrix}$乘以一個標量$3$得到$\begin{bmatrix}-3 \\ 6 \end{bmatrix}$

```python
import numpy as np
a = np.array([-1,2])
print(3 * a)
```

    [-3  6]
    [-3  6]

##### 向量點積

定義如下：$\vec{\mathbf{a}}\cdot \vec{\mathbf{b}} = \begin{bmatrix} a_1 \\ a_2 \\ \ldots \\ a_n\end{bmatrix} \cdot \begin{bmatrix} b_1 \\ b_2 \\ \ldots \\ b_n \end{bmatrix} = a_{1}b_{1} + a_{2}b_{2} + \ldots + a_{n}b_{n}$ 可以看出，向量點積得到的是一個標量；

例如: $\begin{bmatrix} 3 \\ 5 \\ 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 4 \\ 7 \end{bmatrix} = 3 \cdot 1 + 5 \cdot 4 + 2 \cdot 7 = 37$

```python
import numpy as np

vec_1 = np.array([3,5,2])
vec_2 = np.array([1,4,7])
print(np.dot(vec_1, vec_2)) # dot product of vectors
print(np.cross(vec_1, vec_2)) # cross multipication of vectors
```

    37
    [ 27 -19   7]
    37
    [ 27 -19   7]

```python
import numpy as np
import matplotlib.pyplot as plt

soa = np.array([[0, 0, -1, 2], [0, 0, -3, 6]])
X, Y, U, V = zip(*soa)
plt.figure()
ax = plt.gca()
ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1)
ax.set_xlim([-3, 3])
ax.set_ylim([-3, 8])
plt.draw()
plt.show()
```

可以看出，第一個答案是點乘出來的結果，在`python-numpy`中，我們用`np.dot(vector, vector)`來計算兩個或多個向量的點積。而`(vector * vector)`則是計算向量的叉乘結果，下面來介紹一下叉乘（向量積）：

它是向量和向量的乘積，不過結果是一個向量，它的集合意義是所得的向量和被乘向量所在的平面垂直，方向由右手定則規定，大小是兩個被乘向量張成的平行四邊形的面積。所以，向量積不滿足交換律。下面主要介紹在$\mathbf{R}^2$和$\mathbf{R}^3$的情況下的向量積：

$\mathbf{R}^2$的向量積：

$\begin{bmatrix} a_1 \\ a_2\end{bmatrix} \times \begin{bmatrix} b_1 \\ b_2 \end{bmatrix} = \begin{bmatrix} a_1 \cdot b_2 - a_2 \cdot b_1\end{bmatrix}$

例如：$\begin{bmatrix} 1 \\ 2 \end{bmatrix} \times \begin{bmatrix} 3 \\4 \end{bmatrix}
=\begin{bmatrix} 1 \cdot 4 - 3 \cdot 2 \end{bmatrix}= \begin{bmatrix}-2\end{bmatrix}$

$\mathbf{R}^3$的向量積：

$\begin{bmatrix} a_1 \\ a_2 \\ a_3\end{bmatrix} \times \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix} = \begin{bmatrix} a_2 \cdot b_3 - a_3 \cdot b_2 \\ a_3 \cdot b_1 - a_1 \cdot b_3 \\ a_1 \cdot b_2 - a_2 \cdot b_1\end{bmatrix}$

例如：$\begin{bmatrix} 3 \\ 5 \\ 2 \end{bmatrix} \times \begin{bmatrix} 1 \\ 4 \\ 7 \end{bmatrix} =\begin{bmatrix} 5 \cdot 7 - 2 \cdot 4 \\ 2 \cdot 1 - 3 \cdot 7 \\ 3 \cdot 4 - 5 \cdot 1\end{bmatrix}= \begin{bmatrix} 27 \\ -19 \\ 7\end{bmatrix}$

可以看出，向量積的結果就是一個新的向量。

```python
import numpy as np
a = np.array([3, 5, 2])
b = np.array([1, 4, 7])
print(np.cross(a, b)) # cross multipication of vec a and vec b
```

    [ 27 -19   7]
    [ 27 -19   7]

之前我們所講的向量長度的定義式：$\left\| \vec{\mathbf{v}} \right\| = \sqrt{v_{1}^{2} + v_{2}^{2} + \ldots + v_{n}^{2}}$，如果我們連立向量點積的定義就可以得出$\parallel \mathbf{v} \parallel = \sqrt{\vec{v}\cdot \vec{v}}$

關於點積，還有一個重要的重要的性質，就是**柯西不等式**：
**對於兩個非零向量，$\vec{x}, \vec{y} \in \mathbf{R}^n, |\vec{x}\cdot \vec{y}| \leq \parallel \mathbf{x}\parallel \parallel\mathbf{y} \parallel$，當且僅當$\vec{\mathbf{x}} = c\vec{\mathbf{y}}$時，等號成立**。證明如下：

$\begin{align}
\left\|\vec{\mathbf{x}} + \vec{\mathbf{y}}\right\|^2 & = (\vec{\mathbf{x}} + \vec{\mathbf{y}})\cdot (\vec{\mathbf{x}} + \vec{\mathbf{y}}) \\\
 & = \left\|\vec{\mathbf{x}}\right\|^2 + 2\vec{\mathbf{x}}\vec{\mathbf{y}} + \left\|\vec{\mathbf{y}}\right\|^2 \\\
& \le
\left\|\vec{\mathbf{x}}\right\|^2 + 2\left\|\vec{\mathbf{x}}\right\|\left\|\vec{\mathbf{y}}\right\| + \left\|\vec{\mathbf{y}}\right\|^2
\end{align}$

所以：

$\left\|\vec{\mathbf{x}} + \vec{\mathbf{y}}\right\|^2 \le (\left\|\vec{\mathbf{x}}\right\| + \left\|\vec{\mathbf{y}}\right\|)^2$

兩邊開平方後：

$\left\|\vec{\mathbf{x}} + \vec{\mathbf{y}}\right\| \le
\left\|\vec{\mathbf{x}}\right\| + \left\|\vec{\mathbf{y}}\right\|$

這也就是我們高中三角函數裡學過的**三角不等式**，從幾何的角度來說，向量的點積和向量的夾角$\theta$的餘弦值時有關的：$\vec{\mathbf{a}}\cdot\vec{\mathbf{b}} = \left\|\vec{\mathbf{a}}\right\|\left\|\vec{\mathbf{b}}\right\|cos\theta$

從另一個角度來看，這其實就是一個向量在另一個向量上的投影: $Proj_L({\vec{\mathbf{x}}}) =  c\vec{\mathbf{v}} = (\frac{\vec{\mathbf{x}}\cdot \vec{\mathbf{v}}}{\vec{\mathbf{v}}\cdot \vec{\mathbf{v}}})\vec{\mathbf{v}}$

```python
import numpy as np
#the first way to define the function
def get_projection(a, b):
    return a.dot(b)*1.0*b/b.dot(b)

a = np.array([1, 2])
b = np.array([2, 2])
print(get_projection(a, b))
```

    [ 1.5  1.5]

```python
import numpy as np
#the second way to define the function
find_projection = lambda a, b : a.dot(b) * 1.0 * b/b.dot(b)

a = np.array([1, 2])
b = np.array([2, 2])

print(find_projection(a,b))
```

    [ 1.5  1.5]

這時候再考慮向量積，從集合的角度，向量積和向量間的夾角$\theta$的正弦值有關：$\left\|\vec{\mathbf{a}}\times\vec{\mathbf{b}}\right\| = \left\|\vec{\mathbf{a}}\right\|\left\|\vec{\mathbf{b}}\right\|sin\theta$，這就意味著，向量的向量積反應了兩個向量的正交程度（正交：向量積等於零），$sin\theta = 0$ 正交程度最小，$sin\theta = 1$，正交程度最大。

##### 矩陣的向量積

*矩陣乘法*
我們可以將向量理解成一個$1 \times n$的矩陣，當矩陣$A$的列數與向量$\vec x$的分量數目想等式，矩陣和向量的積有定義：

$\underset{m\times n}{A}\vec{\mathbf{x}}=\begin{bmatrix}a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & \ldots & a_{2n} \\ \ldots \\ a_{m1} & a_{m2} & \ldots & a_{mn}\end{bmatrix}\begin{bmatrix}x_1 \\ x_2 \\ \ldots \\ x_n \end{bmatrix} = \begin{bmatrix}a_{11}x_1 + a_{12}x_2 + \ldots + a_{1n}x_n \\ a_{21}x_1 + a_{22}x_2 + \ldots + a_{2n}x_n \\ \ldots \\ a_{m1}x_1 + a_{m2}x_2 + \ldots + a_{mn}x_n \\ \end{bmatrix}$

例：$\begin{bmatrix}4 & 3 & 1 \\ 1 & 2 & 5\end{bmatrix}  \begin{bmatrix}5 \\ 2 \\ 7\end{bmatrix} = \begin{bmatrix}4\cdot 5 + 3\cdot 2 + 1\cdot 7 \\ 1 \cdot 5 + 2 \cdot 2 + 5 \cdot 7\end{bmatrix} = \begin{bmatrix}33 \\ 44\end{bmatrix}$

矩陣的向量積，可以當作時矩陣的所有列向量的線性組合：

$\underset { m\times n }{ \mathbf{A} } \vec { \mathbf{x} } =\begin{bmatrix} \underbrace { \begin{bmatrix} a_{ 11 } \\ a_{ 21 } \\ \ldots \\ a_{ m1 } \end{bmatrix} }_{ \vec { \mathbf{ V }_{ 1 } }  }  & \underbrace { \begin{bmatrix} a_{ 12 } \\ a_{ 22 } \\\ldots  \\ a_{ m2 } \end{bmatrix} }_{ \vec { \mathbf{ V_{ 2 } } }  } & \ldots & \underbrace { \begin{bmatrix} a_{ 1n } \\ a_{ 2n } \\ \ldots \\ a_{ mn } \end{bmatrix} }_{ \vec { \mathbf{ V_{ n } } }  }  \end{bmatrix}\begin{bmatrix} x_{ 1 } \\ x_{ 2 } \\ \ldots \\ x_{ n } \end{bmatrix}=x_1\vec{\mathbf{V}_1}+x_2\vec{\mathbf{V}_2}+\ldots+x_n\vec{\mathbf{V}_n}$

而向量$\vec{x}$的每一個份量可以看成$\mathbf{A}$的每一個列向量的加權。**一個矩陣其實就是一個線性變換，一個矩陣乘以一個向量後得到的向量，其實就相當於將這個向量進行了線性變換。**

### 向量的轉置

向量$ \vec{\mathbf{V}} = \underbrace{\begin{bmatrix}v_1 \\ v_2 \\ \ldots \\ v_n \end{bmatrix}}_{n\times 1}$ 的轉置定義為$\mathbf{V}^T = \underbrace{\begin{bmatrix}v_1 & v_2 & \ldots & v_n \end{bmatrix}}_{1 \times n}$

```python
# we use V = (2,4) to give a example
import numpy as np

vec_3 = np.array([[2,4]]) # if we use double bracket, we are writing a two dimentional vecotr
print(vec_3.T) # print the transport of vec_3
```

    [[2]
     [4]]

向量的轉置有一個性質：一個向量$\vec {v}$ 點乘另一個向量$\vec{w}$，其結果和向量$\vec{v}$轉置後和向量$\vec{w}$做矩陣乘法相同。即：$\vec{\mathbf{v}} \cdot \vec{\mathbf{w}} = \vec{\mathbf{v}}^T \vec{\mathbf{w}}$

## Chapter Four Vector Space

### 張成空間（span）

一組向量的張成空間，說白了就是指這些向量隨便線性組合後能夠表示多少個向量，也被稱為線性生成空間。記做$span(S)$；如果$S = \{v_1, v_2, \dots, v_n\}$是 $V$的有限子集，則生成空間為：$span(S) = span(v_1, v_2, \dots, v_n) = \{\lambda_1 v_1, \lambda_2 v_2, \dots, \lambda_n v_n|\lambda_1, \lambda_2, \dots, \lambda_n \in K\}$

例如，對於$\mathbf{R}^2$空間中兩個不平行的非零向量$\vec{vec_4} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$和向量$\vec{vec_5} = \begin{bmatrix} 0 \\ 3 \end{bmatrix}$，不難發現，這兩個向量可以表示二維空間$\mathbf{R}^2$中的任意一個向量，即，$span(\vec{vec_4}, \vec{vec_5}) = \mathbf{R}^2$，證明如下：

對於$\mathbf{R}^2$中任意向量$\begin{bmatrix} x \\ y \end{bmatrix}$，假設可以用$vec_4$和$vec_5$線性組合而成，

那麼有：$c_1 \begin{bmatrix}2 \\ 1\end{bmatrix} + c_2 \begin{bmatrix} 0 \\ 3 \end{bmatrix} = \begin{bmatrix} x \\ y \end{bmatrix}$

即：$\left\{
\begin{align}
c_1 \cdot 2 & + c_2 \cdot 0 &= x\\\
c_1 \cdot 1 & + c_2 \cdot 3 &= y
\end{align}
\right.$

求解得：
$\left\{
\begin{align}
c_1 &= \frac{x}{2}\\
c_2 &= \frac{y}{3} - \frac{x}{6}
\end{align}
\right.$

由於$x, y$的值已經確定，所以$c_1，c_2$的值也必然是唯一的。

### 線性相關和線性無關（Linearly Dependent and Linearly Independent）

在線性代數裡，向量空間的一組元素中，若沒有向量可用有限個其他向量的線性組合所表示，则稱為線性無關或線性獨立（linearly independent），反之稱為線性相關（linearly dependent）。例如在三維歐幾里得空間$\mathbf{R}^3$的三個向量$(1, 0, 0)，(0, 1, 0)$ 和$(0, 0, 1)$線性無關。但$(2, −1, 1)，(1, 0, 1)$和 $(3, −1, 2)$線性相關，因為第三個是前兩個的和。

*相信各位都已知道那八條還是十條相關性，這裡不做贅述*

上面那個例子例如，$\vec{vec_4} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$和向量$\vec{vec_5} = \begin{bmatrix} 0 \\ 3 \end{bmatrix}$，如果給他們再加一個向量$\vec{vec_6} = \begin{bmatrix} 5 \\ 2 \end{bmatrix}$，由於$\vec{vec_5}$可以由其他兩個向量線性組合而成，由三個向量共同張成的空間並沒有發生變化，仍然是$\mathbf{R}^3$，因此稱集合$\{\vec{vec_3}, \vec{vec_4}, \vec{vec_5}\}$線性相關。

### 如何判斷是否線性相關

一個向量集合$s = v_1, v_2, \dots, v_n$ 線性相關的充分必要是存在一部分非零係數使得：$c_1v_1 + c_2v_2 + \dots + c_nv_n = \begin{bmatrix} 0 \\ 0 \\ \dots \\ 0 \end{bmatrix}$。

例如：向量：$\begin{bmatrix}2 \\ 1\end{bmatrix}$ 和 $\begin{bmatrix}3 \\ 2\end{bmatrix}$，則先寫出如下等式：
$c_1 \begin{bmatrix}2 \\ 1\end{bmatrix} + c_2 \begin{bmatrix}3 \\ 2\end{bmatrix} = \begin{bmatrix}0 \\ 0\end{bmatrix}$,

容易求解得：$\begin{bmatrix}c_1 \\ c_2\end{bmatrix} = \begin{bmatrix}0 \\ 0\end{bmatrix}$，說明這兩個向量線性無關。

類似的，對於三個$\mathbf{R}^3$中的向量$\begin{bmatrix}2 \\ 0 \\ 0\end{bmatrix}, \begin{bmatrix} 0 \\ 1 \\ 0\end{bmatrix}$和$\begin{bmatrix} 0 \\ 0 \\ 7\end{bmatrix}$，不難判斷出它們是線性無關的，它們共同張成了$\mathbf{R}^3$空間。

而對於向量集合：$\left\{\begin{bmatrix}2 \\ 1\end{bmatrix}, \begin{bmatrix}3 \\ 2\end{bmatrix}, \begin{bmatrix}1 \\ 2 \end{bmatrix}\right\}$，不難算出存在非零係數：$\begin{bmatrix}c_1 \\ c_2 \\ c_3\end{bmatrix} = \begin{bmatrix}-4 \\ 3 \\ -1\end{bmatrix}$ 使得 $c1 \begin{bmatrix}2 \\ 1\end{bmatrix} + c_2 \begin{bmatrix}3 \\ 2\end{bmatrix} + c_3 \begin{bmatrix}1 \\ 2 \end{bmatrix} = \begin{bmatrix}0 \\ 0\end{bmatrix}$。因此，此集合線性相關。

## Chapter Five Linear Algebra Advanced Text

在前面幾張，已經簡單介紹了線性代數中最基本的數據表達方式：**矩陣**和**向量**。有了這兩個數學工具作為基礎，我們可以進一步討論下面的內容：

1. 如何理解線性代數的基。

2. 向量中的子空間、另空間、列空間、行空間、零空間都是什麼？我們應該如何求解？

3. 如何用線性代數的知識來擬合數據？

4. 機器學習、圖形處理中常見的“特徵向量”究竟是什麼？它和變換矩陣有什麼關係？

### 階梯形矩陣（Echelon Matrix）

階梯形矩陣是一類非常實用的工具，可以幫助我們求解線性空間的基，這就是能用在諸如計算解不是唯一的方程組之類的問題上。

#### 階梯形矩陣

若矩陣$\mathbf{A}$滿足如下條件：

1. 若有零行（元素全為零的行），則零行應在最下方；

2. 非零首元（即非零行的第一個不為零的元素）的列標號隨行標號的增加而嚴格遞增。

則這個矩陣稱為階梯形矩陣。

示例：

$\begin{bmatrix}
2 & 0 & 2 & 1 \\
0 & 5 & 2 & -2 \\
0 & 0 & 3 & 2 \\
0 & 0 & 0 & 0
\end{bmatrix}$

#### 行簡化階梯形矩陣（Row Echelon Matrix）

若矩陣$\mathbf{A}$滿足如下條件：

1. 它是階梯形矩陣;

2. 非零首元所在列出了非零首元外，其餘元素都為零。

則稱此矩陣為行簡化階梯形矩陣。

示例：

$\begin{bmatrix}
2 & 0 & 2 & 1 \\
0 & 5 & 2 & -2 \\
0 & 0 & 3 & 2 \\
0 & 0 & 0 & 0
\end{bmatrix}$

#### 行最簡階梯形矩陣（reduced row echelon form）

若矩陣$\mathbf{A}$滿足如下條件：

1. 它是行簡化階梯形矩陣；

2. 非零首元都為一。

則稱它為行最簡階梯形矩陣。下面我們來講講如何將矩陣化簡為最簡階梯形：

對於這個矩陣$\mathbf{A} = \begin{bmatrix}
1 & 2 & 1 & 1 & 7\\
1 & 2 & 2 & -1 & 12\\
2 & 4 & 0 & 6 & 4
\end{bmatrix}$，使用**初等變換（Elementary Operations of Linear System）**就可以將矩陣轉換成如下形式：
$\begin{bmatrix}
1 & 2 & 1 & 1 & 7\\
1 & 2 & 2 & -1 & 12\\
2 & 4 & 0 & 6 & 4
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 2 & 1 & 1 & 7\\
0 & 0 & 1 & -2 & 5\\
2 & 4 & 0 & 6 & 4
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 2 & 1 & 1 & 7\\
0 & 0 & 1 & -2 & 5\\
0 & 0 & -2 & 4 & -10
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 2 & 1 & 1 & 7\\
0 & 0 & 1 & -2 & 5\\
0 & 0 & 0 & 0 & 0
\end{bmatrix}
\rightarrow
\begin{bmatrix}
1 & 2 & 0 & 3 & 2\\
0 & 0 & 1 & -2 & 5\\
0 & 0 & 0 & 0 & 0
\end{bmatrix}$

**行最簡形**非常實用，可以用來解方程（滑稽；例如下面這個方程組：

$\left\{
\begin{eqnarray}
x_1 + 2x_2 + x_3 + x_4 &=& 7 \\\
x_1 + 2x_2 + 2x_3 - x_4 &=& 12 \\\
2x_1 + 4x_2 + 6x_4 &=& 4
\end{eqnarray}
\right.$

按照常理，只有三個方程，肯定無法求解出四個未知數，此時，如果我們使用`numpy.linalg.solve`，求解時就會觸發`LinAlgError`，但是通過化成最簡形，我們可以進一步找出變量的憲制關係。首先，將方程組表達成增廣矩陣形式：

$\begin{bmatrix}
1 & 2 & 1 & 1 & 7\\
1 & 2 & 2 & -1 & 12\\
2 & 4 & 0 & 6 & 4
\end{bmatrix}$

然後將其行簡化，得：

$\begin{bmatrix}
1 & 2 & 0 & 3 & 2\\
0 & 0 & 1 & -2 & 5\\
0 & 0 & 0 & 0 & 0
\end{bmatrix}$

從結果可以看出這個矩陣的主列有兩個，而且是線性無關的。所以矩陣$A$的秩是$2$，，即，$rank\mathbf{(A)} = 2$.

**矩陣的秩的一個特性：矩陣$A$等於矩陣$A$的轉置地秩序，即，**$rank(\mathbf{A})=rank(\mathbf{A^T})$

在`python`中，可以使用`numpy`包中的`linalg.matrix_rank`方法計算矩陣的秩：

```python
import numpy as np
a = np.matrix('1 1 1 1;1 2 3 4;4 3 2 1')
print(np.linalg.matrix_rank(a))
```

    2

**注意**⚠，在`numpy`中的秩和線性代數裡的秩不是同一個概念。在`numpy`中維度（dimensions）叫做軸（axes），軸的個數叫做秩。

```python
import numpy as np
a = np.matrix('1 1 1 1;1 2 3 4; 0 0 1 0')
print (a.ndim)  # 2（维度）
print (np.rank(a))  # 2（a.ndim 的别名，已经过时）
print (np.linalg.matrix_rank(a))  # 3（秩）
```

    2
    2
    3

    /usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:4: VisibleDeprecationWarning: `rank` is deprecated; use the `ndim` attribute or function instead. To find the rank of a matrix see `numpy.linalg.matrix_rank`.
      after removing the cwd from sys.path.

#### 行空間

有了列空間的定義，行空間顧名思義其實就是矩陣每一行轉置得到的向量張成的自空間，也就是矩陣的轉置的列空間，記做：$R(\mathbf{A}) = C(\mathbf{A}^T)$

例如：矩陣$\mathbf{A} = \begin{bmatrix}1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\4 & 3 & 2 & 1\end{bmatrix}$的行空間就等於：$R(\mathbf{A}) = C(\mathbf{A}^T) = span\left(\begin{bmatrix}1 \\ 1 \\ 1 \\ 1\end{bmatrix}\begin{bmatrix}1 \\ 2 \\ 3 \\ 4\end{bmatrix}\begin{bmatrix}4 \\ 3 \\ 2 \\ 1\end{bmatrix}\right)$

#### 零空間

上面已經講過了零空間，現在這裡的零空間是指矩陣$A^T$的零空間，也就是**左零空間**。即：$N(\mathbf{A}^T) = \left\{ \vec{\mathbf{x}} | \mathbf{A}^{T} \vec{\mathbf{x}} = \vec{\mathbf{0}} \right\} = \left\{ \vec{\mathbf{x}} | \vec{\mathbf{x}}^{T} \mathbf{A} = \vec{\mathbf{0}}^{T} \right\}$

例如，矩陣$\mathbf{B} = \begin{bmatrix}1 & 1 & 4 \\ 1 & 2 & 3 \\1 & 4 & 2\\ 1 & 3 & 1\end{bmatrix}$的轉置是矩陣$\mathbf{A} = \mathbf{A} = \begin{bmatrix}1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\4 & 3 & 2 & 1\end{bmatrix}$，因此左零空間等於：$N(\mathbf{B^T}) = N(\mathbf{A}) = span\left(\begin{bmatrix} 1 \\ -2 \\ 1 \\ 0 \end{bmatrix} \begin{bmatrix} 2 \\ -3 \\ 0 \\ 1 \end{bmatrix}\right)$；由於轉置是對稱的，所以矩陣$A$的轉置的左零空間也是矩陣$A$的零空間。

#### 子空間的正交補

假設$V$是$R^n$的一個子空間，那麼$V$的正交補$V^\perp$也是一個子空間，定義為$\left\{\vec{\mathbf{x}} | \vec{\mathbf{x}} \vec{\mathbf{v}}=0\right\}$，也就是說$R^n$中所有正交於$V$的向量所組成的子空間。

由於正交是對稱的，所以正交補也是對稱的。一個子空間的正交補的正交補依然等於這個子空間。

矩陣的零空間是行空間的正交補：$N(\mathbf{A}) = R(\mathbf{A})^{\bot}$；反過來，矩陣的左零空間是列空間的正交補：$N(\mathbf{B}^T) = C(\mathbf{B})^{\bot}$

### 最小二乘逼近

最小二乘法是一個使用的數學工具，利用它在方程誤解的情況下給出近似解。在機器學習中，最小二乘逼近是一個重要的擬合方法。

假設有一個方程：$\underset{n\times k}{\mathbf{A}}\vec{\mathbf{x}} = \vec{\mathbf{b}}$

無解。把上式寫成：$\vec{a_1}\vec{\mathbf{x}} + \vec{a_2}\vec{\mathbf{x}} + \ldots + \vec{a_k}\vec{\mathbf{x}} = \vec{\mathbf{b}}$

無解，就意味著$\mathbf{A}$的所有列向量的張成空間不包括向量$\vec{b}$。即：$\vec{\mathbf{b}} \notin span(C(\mathbf{A}))$

我們可以通過最小二乘法，求解出近似解。即，要找出一些$\vec{x*}$使得$\left\|\vec{\mathbf{b}}-\mathbf{A}\vec{\mathbf{x}^*}\right\|$最小。用向量$\vec{\mathbf{V}}$代表$\mathbf{A}\vec{\mathbf{x*}}$，有：

$\left\|
\begin{bmatrix}
\vec{b_1}-\vec{v_1}\\
\vec{b_2}-\vec{v_2}\\
\ldots\\
\vec{b_n}-\vec{v_n}\\
\end{bmatrix}
\right\|^2
=
(b_1-v_1)^2 + (b_2-v_2)^2 + \ldots + (b_n-v_n)^2$

把這個值最小化的過程叫做**最小二乘逼近**。

如何求出$\mathbf{A}\vec{\mathbf{x*}}$這個近似值呢？從集合上考慮，列空間可以看成空間中張成的一個平面，而平面$\vec{b}$並不落在這個平面上。但我們知道，在這個平面上與向量$\vec{b}$最接近的向量就是它的投影！所以，$\mathbf{A}\vec{\mathbf{x}^*} = Proj_{C(\mathbf{A})}\vec{\mathbf{b}}$，直接計算$Proj_{C(\mathbf{A})}\vec{\mathbf{b}}$並不簡單。不過，$\vec{\mathbf{b}}-\mathbf{A}\vec{\mathbf{x}}$其實就是$\mathbf{A \vec{x}}$的正交補，所以一個簡單的求解方法是將原來無解的方程做成一個$\mathbf{A}$的轉置再求解：$\mathbf{A}^T\mathbf{A}\vec{\mathbf{x}^*} = \mathbf{A}^T\vec{\mathbf{b}}$得出的解就是原方程的近似解。

#### 示例 解下列方程組
$\left\{
\begin{eqnarray}
x + y &=& 3 \\\
x - y &=& -2 \\\
y &=& 1
\end{eqnarray}
\right.$

將三個方程表示的直線畫出來，可以看出這三條直線並沒有交點：

```python
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

plt.xlim(-3,3)
plt.ylim(0,6)
plt.plot([-2,3],[0,5],'blue',[-3,3],[6,0],'yellow',[-3,3],[1,1],'red')
```

那麼，我們應該如何找出一個與三條直線距離最近的一個點呢？這時候就可以使用最小二乘逼近了。

先將方程寫成矩陣和向量的形式：

$\underbrace{
\begin{bmatrix}
1 & 1 \\
1 & -1 \\
0 & 1
\end{bmatrix}
}_{\mathbf{A}}
\underbrace{
\begin{bmatrix}
x \\
y
\end{bmatrix}
}_{\vec{\mathbf{x}}}
=
\underbrace{
\begin{bmatrix}
3 \\
-2 \\
1
\end{bmatrix}
}_{\vec{\mathbf{b}}}$

這個等式的最小二乘逼近就是：

$\begin{align}
\begin{bmatrix}
1 & 1 & 0 \\
1 & -1 & 1\\
\end{bmatrix}
\begin{bmatrix}
1 & 1 \\
1 & -1 \\
0 & 1
\end{bmatrix}
\begin{bmatrix}
x^* \\
y^*
\end{bmatrix}
& =
\begin{bmatrix}
1 & 1 & 0 \\
1 & -1 & 1\\
\end{bmatrix}
\begin{bmatrix}
3 \\
-2 \\
1
\end{bmatrix}
\\\
\begin{bmatrix}
2 & 0 \\
0 & 3
\end{bmatrix}
\begin{bmatrix}
x^* \\
y^*
\end{bmatrix}
& =
\begin{bmatrix}
1 \\
6
\end{bmatrix}
\end{align}$

由於是二解方程，我們可以很容易求出矩陣$\begin{bmatrix}2 & 0 \\ 0 & 3\end{bmatrix}$的逆是：$\begin{bmatrix}\frac{1}{2} & 0 \\ 0 & \frac{1}{3}\end{bmatrix}$

所以：

$\begin{bmatrix}
x^* \\
y^*
\end{bmatrix}
=
\begin{bmatrix}\frac{1}{2} & 0 \\ 0 & \frac{1}{3}\end{bmatrix}
\begin{bmatrix}
1 \\
6
\end{bmatrix}
=
\begin{bmatrix}
\frac{1}{2} \\
2
\end{bmatrix}$

在`python`中，可以使用`numpy.linalg.lstsq`方法來求解最小二乘逼近。

```python
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

x = [0.5]
y = [2]
plt.xlim(-3,3)
plt.ylim(0,6)
plt.plot([-2,3],[0,5],'blue',[-3,3],[6,0],'yellow',[-3,3],[1,1],'red',xData,yData1,'black')
plt.scatter(x,y,color = 'black')
```

```python
a = np.array([[1, 1], [1, -1], [0, 1]])
b = np.array([3, -2, 1])
x = np.linalg.lstsq(a,b)
print(x)
```

    (array([ 0.5,  2. ]), array([ 1.5]), 2, array([ 1.73205081,  1.41421356]))

`numpy.inalg.lstsq`的返回值包括了四個部分：

1. 最小二乘逼近，如果`b`是二維的，那麼這個逼近的結果有多個列，每一列是一個逼近解，逼近解就是$\begin{bmatrix}0,5 \\ 2 \end{bmatrix}$.

2. 殘差。即，每一個`b - a*x`的長度的和。對於上例，殘差是$1.5$.

3. 矩陣$a$的秩。對於上例，矩陣$a$的秩為$2$。

4. 矩陣$a$的歧異值。對於上例，矩陣$a$的奇異值為：$\begin{bmatrix}1.73205081 \\  1.41421356\end{bmatrix}$

#### 线性回归

問題：給定$4$個坐標點$(-1,0),(0,1),(1,),(2,1)$，求一條經過這些點的直線$y = mx + b$。

將四個點畫圖如下：

```python
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

x = [-1,0,1,2]
y =[0,1,2,1]

plt.scatter(x,y,color = 'black')
```

顯然這樣的直線並不存在。然而我們能夠使用最小二乘逼近，找到一條盡可能靠近這些點的直線。將四個點表示成方程組的形式：

$\left\{
\begin{eqnarray}
f(-1) &= -m + b = 0\\\
f(0) &= 0 + b  = 1\\\
f(1) &= m + b = 2\\\
f(2) &= 2m + b = 1
\end{eqnarray}
\right.$

然後方程組用矩陣和向量的形式：

    $\underbrace{
\begin{bmatrix}
-1 & 1 \\
0 & 1 \\
1 & 1 \\
2 & 1
\end{bmatrix}
}_{\mathbf{A}}
\underbrace{
\begin{bmatrix}
m\\
b
\end{bmatrix}
}_{\vec{\mathbf{x}}}
=
\underbrace{
\begin{bmatrix}
0\\
1\\
2\\
1
\end{bmatrix}
}_{\vec{\mathbf{b}}}$

這兩個等式的最小二乘逼近就是：

    $\begin{align}
\begin{bmatrix}
-1 & 0 & 1 & 2 \\
1 & 1 & 1 & 1
\end{bmatrix}
\begin{bmatrix}
-1 & 1 \\
0 & 1 \\
1 & 1 \\
2 & 1
\end{bmatrix}
\begin{bmatrix}
m^*\\
b^*
\end{bmatrix}
&=
\begin{bmatrix}
-1 & 0 & 1 & 2 \\
1 & 1 & 1 & 1
\end{bmatrix}
\begin{bmatrix}
0\\
1\\
2\\
1
\end{bmatrix}\\\
\begin{bmatrix}
6 & 2 \\
2 & 4
\end{bmatrix}
\begin{bmatrix}
m^*\\
b^*
\end{bmatrix}
&=
\begin{bmatrix}
4\\
4
\end{bmatrix}
\end{align}$

容易求得$\begin{bmatrix}6 & 2\\2 & 4\end{bmatrix}$的逆$\frac{1}{20}\begin{bmatrix}4 & -2\\-2 & 6\end{bmatrix}$，因此

    $\begin{bmatrix}m^*\\b^*\end{bmatrix} = \frac{1}{20}\begin{bmatrix}4 & -2\\-2 & 6\end{bmatrix}\begin{bmatrix}4 \\ 4\end{bmatrix} = \frac{1}{20}\begin{bmatrix}8 \\ 16\end{bmatrix} = \begin{bmatrix}\frac{2}{5} \\ \frac{4}{5}\end{bmatrix}$

將直線$y = \frac{2}{5}x + \frac{4}{5}$繪圖如下所示：

```python
import numpy as np
import matplotlib.pyplot as plt
from pylab import *

x = [-1,0,1,2]
y =[0,1,2,1]
x1 = np.linspace(-3,3)
y1 = lambda x: 2*x/5 + 4/5
plt.scatter(x,y,color = 'black')
plt.plot(x1,y1(x1),'r')
```

這就是所求的直線的近似解。

```python
a = np.matrix('-1 1;0 1;1 1;2 1')
b = np.array([0, 1, 2, 1])
x = np.linalg.lstsq(a, b)
print(x)
```

    (array([ 0.4,  0.8]), array([ 1.2]), 2, array([ 2.68999405,  1.66250775]))

Notebooks/.ipynb_checkpoints/Use PY in Linear Algebra-checkpoint.ipynb


# The Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Properties

The Fourier transform has a number of specific properties. They can be concluded from its definition. The most important ones in the context of signals and systems are reviewed in the following.

### Invertibility

According to the [Fourier inversion theorem](https://en.wikipedia.org/wiki/Fourier_inversion_theorem), for many types of signals it is possible to recover the signal $x(t)$ from its Fourier transformation $X(j \omega) = \mathcal{F} \{ x(t) \}$

\begin{equation}
x(t) = \mathcal{F}^{-1} \left\{ \mathcal{F} \{ x(t) \} \right\}
\end{equation}

A sufficient condition for the theorem to hold is that both the signal $x(t)$ and its Fourier transformation are absolutely integrable and $x(t)$ is continuous at the considered time $t$. For this type of signals, above relation can be proven by applying the definition of the inverse Fourier transform to the Fourier transform and rearranging terms

\begin{equation}
\begin{split}
x(t) &= \frac{1}{2 \pi} \int_{-\infty}^{\infty} \underbrace{\int_{-\infty}^{\infty} x(\tau) e^{-j \omega \tau} d\tau}_{X(j \omega)} \; e^{j \omega t} d\omega \\
&= \int_{-\infty}^{\infty} x(\tau) \left( \frac{1}{2 \pi} \int_{-\infty}^{\infty} e^{-j \omega \tau} e^{j \omega t}  d\omega \right) d\tau \\
&= \int_{-\infty}^{\infty} x(\tau) \delta(t - \tau) d\tau = x(t)
\end{split}
\end{equation}

The solution $\delta(t - \tau)$ of the inverse Fourier transform $\mathcal{F}^{-1} \{ e^{-j \omega \tau} \}$ in the brackets can be deduced from the [Fourier transform of the Dirac impulse](definition.ipynb#Transformation-of-the-Dirac-Impulse). The invertibility of the Fourier transformation holds also for more general signals $x(t)$, composed for instance from Dirac delta distributions.

**Example**

The invertibility of the Fourier transform is illustrated at the example of the [rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal) $x(t) = \text{rect}(t)$. The inverse of [its Fourier transform](definition.ipynb#Transformation-of-the-Rectangular-Signal) $X(j \omega) = \text{sinc} \left( \frac{\omega}{2} \right)$ is computed to show that the rectangular signal, although it has discontinuities, can be recovered by inverse Fourier transformation.

```python
import sympy as sym
sym.init_printing()

def fourier_transform(x):
    return sym.integrals.transforms._fourier_transform(x, t, w, 1, -1, 'Fourier')

def inverse_fourier_transform(X):
    return sym.integrals.transforms._fourier_transform(X, w, t, 1/(2*sym.pi), 1, 'Inverse Fourier')
```

```python
t, w = sym.symbols('t omega')

X = sym.sinc(w/2)
x = inverse_fourier_transform(X)
x
```

```python
sym.plot(x, (t, -1, 1), ylabel=r'$x(t)$');
```

### Duality

Comparing the [definition of the Fourier transform](definition.ipynb) with its inverse

\begin{align}
X(j \omega) &= \int_{-\infty}^{\infty} x(t) \, e^{-j \omega t} \; dt \\
x(t) &= \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t} \; d\omega
\end{align}

reveals that both are very similar in their structure. They differ only with respect to the normalization factor $2 \pi$ and the sign of the exponential function. The duality principle of the Fourier transform can be deduced from this observation. Let's assume that we know the Fourier transformation $x_2(j \omega)$ of a signal $x_1(t)$

\begin{equation}
x_2(j \omega) = \mathcal{F} \{ x_1(t) \}
\end{equation}

It follows that the Fourier transformation of the signal

\begin{equation}
x_2(j t) = x_2(j \omega) \big\vert_{\omega=t}
\end{equation}

is given as

\begin{equation}
\mathcal{F} \{ x_2(j t) \} = 2 \pi \cdot x_1(- \omega)
\end{equation}

The duality principle of the Fourier transformation allows to carry over results from the time-domain to the spectral-domain and vice-versa. It can be used to derive new transforms from known transforms. This is illustrated at an example. Note, that the Laplace transformation shows no duality. This is due to the mapping of a complex signal $x(t)$ with real valued independent variable $t \in \mathbb{R}$ to its complex transform $X(s) \in \mathbb{C}$ with complex valued independent variable $s \in \mathbb{C}$.

#### Transformation of the exponential signal

The Fourier transform of a shifted Dirac impulse $\delta(t - \tau)$ is derived by introducing it into the definition of the Fourier transform and exploiting the sifting property of the Dirac delta function

\begin{equation}
\mathcal{F} \{ \delta(t - \tau) \} = \int_{-\infty}^{\infty} \delta(t - \tau) \, e^{-j \omega t} \; dt =  e^{-j \omega \tau}
\end{equation}

Using the duality principle, the Fourier transform of $e^{-j \omega_0 t}$ can be derived from this result by

1. substituting $\omega$ with $t$ and $\tau$ with $\omega_0$ on the right-hand side to yield the time-domain signal $e^{-j \omega_0 t}$
2. substituting $t$ by $- \omega$, $\tau$ with $\omega_0$ and multiplying the result by $2 \pi$ on the left-hand side

\begin{equation}
\mathcal{F} \{ e^{-j \omega_0 t} \} = 2 \pi \cdot \delta(\omega + \omega_0)
\end{equation}

### Linearity

The Fourier transform is a linear operation. For two signals $x_1(t)$ and $x_2(t)$ with Fourier transforms $X_1(j \omega) = \mathcal{F} \{ x_1(t) \}$ and $X_2(j \omega) = \mathcal{F} \{ x_2(t) \}$ the following holds

\begin{equation}
\mathcal{F} \{ A \cdot x_1(t) + B \cdot x_2(t) \} = A \cdot X_1(j \omega) + B \cdot X_2(j \omega)
\end{equation}

with $A, B \in \mathbb{C}$. The Fourier transform of a weighted superposition of signals is equal to the weighted superposition of the individual Fourier transforms. This property is useful to derive the Fourier transform of signals that can be expressed as superposition of other signals for which the Fourier transforms are known or can be calculated easier. Note, linearity holds also for the inverse Fourier transform.

#### Transformation of the cosine and sine signal

The Fourier transform of $\cos(\omega_0 t)$ and $\sin(\omega_0 t)$ is derived by expressing both as harmonic exponential signals using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula)

\begin{align}
\cos(\omega_0 t) &= \frac{1}{2} \left( e^{j \omega_0 t} + e^{-j \omega_0 t} \right) \\
\sin(\omega_0 t) &= \frac{1}{2j} \left( e^{j \omega_0 t} - e^{-j \omega_0 t} \right)
\end{align}

together with the Fourier transform $\mathcal{F} \{ e^{-j \omega_0 t} \} = 2 \pi \cdot \delta(\omega - \omega_0)$ from above yields

\begin{align}
\mathcal{F} \{ \cos(\omega_0 t) \} &= \pi \left( \delta(\omega + \omega_0) + \delta(\omega - \omega_0)  \right) \\
\mathcal{F} \{ \sin(\omega_0 t) \} &= j \pi \left( \delta(\omega + \omega_0) - \delta(\omega - \omega_0) \right)
\end{align}

### Symmetries

In order to investigate the symmetries of the Fourier transform $X(j \omega) = \mathcal{F} \{ x(t) \}$ of a signal $x(t)$ with specific properties, first the case of a real valued signal $x(t) \in \mathbb{R}$ is considered. The results are then generalized to complex signals $x(t) \in \mathbb{C}$.

#### Real valued signals

Decomposing a real valued signal $x(t) \in \mathbb{R}$ into its even and odd part $x(t) = x_\text{e}(t) + x_\text{o}(t)$ and introducing these into the definition of the Fourier transform yields

\begin{align}
X(j \omega) &= \int_{-\infty}^{\infty} \left[ x_\text{e}(t) + x_\text{o}(t) \right] e^{-j \omega t} \; dt \\
&= \int_{-\infty}^{\infty} \left[ x_\text{e}(t) + x_\text{o}(t) \right] \cdot \left[ \cos(\omega t) - j \sin(\omega t) \right] \; dt \\
&= \underbrace{\int_{-\infty}^{\infty} x_\text{e}(t) \cos(\omega t) \; dt}_{X_\text{e}(j \omega)} +
j \underbrace{\int_{-\infty}^{\infty} - x_\text{o}(t) \sin(\omega t) \; dt}_{X_\text{o}(j \omega)}
\end{align}

For the last equality the fact was exploited that an integral with symmetric limits is zero for odd functions. Note that the multiplication of an odd function with an even/odd function results in an even/odd function. In order to conclude on the symmetry of $X(j \omega)$ its behavior for a reverse of the sign of $\omega$ has to be investigated. Due to the symmetry properties of $\cos(\omega t)$ and $\sin(\omega t)$, it follows that the Fourier transform of the

* even part $x_\text{e}(t)$ is real valued with even symmetry $X_\text{e}(j \omega) = X_\text{e}(-j \omega)$
* odd part $x_\text{o}(t)$ is imaginary valued with odd symmetry $X_\text{o}(j \omega) = - X_\text{o}(-j \omega)$

Combining this, it can be concluded that the Fourier transform $X(j \omega)$ of a real-valued signal $x(t) \in \mathbb{R}$ shows complex conjugate symmetry

\begin{equation}
X(j \omega) = X^*(- j \omega)
\end{equation}

It follows that the magnitude spectrum $|X(j \omega)|$ of a real-valued signal shows even symmetry

\begin{equation}
|X(j \omega)| = |X(- j \omega)|
\end{equation}

and the phase $\varphi(j \omega) = \arg \{ X(j \omega) \}$ odd symmetry

\begin{equation}
\varphi(j \omega) = - \varphi(- j \omega)
\end{equation}

Due to these symmetries, both are often plotted only for positive frequencies $\omega \geq 0$. However, without the information that the signal is real-valued it is not possible to conclude on the magnitude spectrum and phase for the negative frequencies $\omega < 0$.

#### Complex Signals

By following the same procedure as above for an imaginary signal, the symmetries of the Fourier transform of the even and odd part of an imaginary signal can be derived. The results can be combined, by decomposing a complex signal $x(t) \in \mathbb{C}$ and its Fourier transform into its even and odd part for both the real and imaginary part. This results in the following symmetry relations of the Fourier transform

**Example**

The Fourier transform $X(j \omega)$ of the signal $x(t) = \text{sgn}(t) \cdot \text{rect}(t)$ is computed. The signal is real valued with odd symmetry due to the sign function. It follows from the symmetry relations of the Fourier transform, that $X(j \omega)$ is imaginary with odd symmetry.

```python
x = - sym.Heaviside(t + 1/2) + 2 * sym.Heaviside(t) - sym.Heaviside(t - 1/2)
sym.plot(x, (t, -2, 2), xlabel=r'$t$', ylabel=r'$x(t)$');
```

```python
X = fourier_transform(x)
X = X.rewrite(sym.cos).simplify()
X
```

```python
sym.plot(sym.im(X), (w, -30, 30), xlabel=r'$\omega$',
         ylabel=r'$\Im \{ X(j \omega) \}$');
```

**Exercise**

* What symmetry do you expect for the Fourier transform of the signal $x(t) = j \cdot \text{sgn}(t) \cdot \text{rect}(t)$? Check your results by modifying above example.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
from sympy import *
x1, x2, x3, y1, y2, y3, answer = symbols('x1 x2 x3 y1 y2 y3 answer')

A = Point(0, 0)
B = Point(x2, y2)
C = Point(x3, y3)
eqs = [A.distance(B) + A.distance(C) + B.distance(C) - 5,
 A.distance(B) - 2,
 B.distance(C) - 1,
 -answer + A.distance(C)]

solve(eqs, [answer])

```

```python
from sympy import *
A, B, C, ans = symbols('A B C ans', real=True)
a, b, c = symbols('a b c', real=True)
C = C + I*c
equations = [
    Abs(A - B) + Abs(A - C) + Abs(B - C) - 5,
    Abs(B - C) - 1,
    Abs(A - B) - 2,
    Abs(A - C) - ans
]
solve(equations, [im(A), im(B), im(C), im(ans)])
```

    [(0, -sqrt((-B + C + 1)*(B - C + 1))), (0, sqrt((-B + C + 1)*(B - C + 1)))]

```python
Symbol('x', real=False).is_real
```

    False

```python
a, b, c = symbols('a b c')
solve([
    (a - b) - 2,
    (a - b) - c
], [c])

```

    {c: a - b}

```python
solve([
    (a - b) - 2
], [a - b])
```

    {a - b: 2}

```python

```


# Modelado de Robots

Recordando la práctica anterior, tenemos que la ecuación diferencial que caracteriza a un sistema masa-resorte-amoritguador es:

$$
m \ddot{x} + c \dot{x} + k x = F
$$

y revisamos 3 maneras de obtener el comportamiento de ese sistema, sin embargo nos interesa saber el comportamiento de un sistema mas complejo, un robot; empezaremos con un pendulo simple, el cual tiene la siguiente ecuación de movimiento:

$$
m l^2  \ddot{q} + m g l \cos{q} = \tau
$$

Como podemos ver, son similares en el sentido de que involucran una sola variable, sin embargo, en la segunda ecuación, nuestra variable esta involucrada adentro de una función no lineal ($\cos{q}$), por lo que nuestra ecuación diferencial es no lineal, y por lo tanto _no_ podemos usar el formalismo de función de transferencia para resolverla; tenemos que usar la función ```odeint``` para poder resolverla.

Como es de segundo grado, tenemos que dividir nuestra ecuación diferencial en dos mas simples, por lo tanto usaremos el siguiente truco:

$$
\frac{d}{dt} q = \dot{q}
$$

entonces, tenemos dos ecuaciones diferenciales, por lo que podemos resolver dos incognitas $q$ y $\dot{q}$.

Utilizando nuestros conocimientos de algebra lineal, podemos acomodar nuestro sistema de ecuaciones en una matriz, de tal manera que si antes teniamos que:

$$
\begin{align}
\frac{d}{dt} q &= \dot{q} \\
\frac{d}{dt} \dot{q} &= \ddot{q} = \frac{\tau  - m g l \cos{q}}{ml^2}
\end{align}
$$

Por lo que podemos ver que nuestro sistema de ecuaciones tiene un estado mas grande que antes; la ecuación diferencial que teniamos como no lineal, de segundo orden, podemos escribirla como no lineal, de primer orden siempre y cuando nuestro estado sea mas grande.

Definamos a lo que nos referimos con estado:

$$
x =
\begin{pmatrix}
q \\
\dot{q}
\end{pmatrix}
$$

con esta definición de estado, podemos escribir el sistema de ecuaciónes de arriba como:

$$
\frac{d}{dt} x = \dot{x} = \frac{d}{dt}
\begin{pmatrix}
q \\
\dot{q}
\end{pmatrix} =
\begin{pmatrix}
\dot{q} \\
\frac{\tau  - m g l \cos{q}}{ml^2}
\end{pmatrix}
$$

o bien $\dot{x} = f(x)$, en donde $f(x)$ es una función vectorial, o bien, un vector de funciones:

$$
f(x) =
\begin{pmatrix}
\dot{q} \\
\frac{\tau  - m g l \cos{q}}{ml^2}
\end{pmatrix}
$$

Por lo que ya estamos listos para simular este sistema mecánico, con la ayuda de ```odeint()```; empecemos importando laas librerias necesarias:

```python
from scipy.integrate import odeint
```

```python
from numpy import linspace
```

y definiendo una función que devuelva un arreglo con los valores de $f(x)$

```python
def f(x, t):
    from numpy import cos
    q, q̇ = x
    τ = 0
    m = 1
    g = 9.81
    l = 1
    return [q̇, τ - m*g*l*cos(q)/(m*l**2)]
```

Vamos a simular desde el tiempo $0$, hasta $10$, y las condiciones iniciales del pendulo son $q=0$ y $\dot{q} = 0$.

```python
ts = linspace(0, 10, 100)
x0 = [0, 0]
```

Utilizamos la función ```odeint``` para simular el comportamiento del pendulo, dandole la función que programamos con la dinámica de $f(x)$ y sacamos los valores de $q$ y $\dot{q}$ que nos devolvió ```odeint``` envueltos en el estado $x$

```python
xs = odeint(func = f, y0 = x0, t = ts)
qs, q̇s = list(zip(*xs.tolist()))
```

En este punto ya tenemos nuestros datos de la simulación, tan solo queda graficarlos para interpretar los resultados:

```python
%matplotlib inline
from matplotlib.pyplot import style, plot, figure
style.use("ggplot")
```

```python
fig1 = figure(figsize = (8, 8))

ax1 = fig1.gca()

ax1.plot(xs);
```

```python
fig2 = figure(figsize = (8, 8))

ax2 = fig2.gca()

ax2.plot(qs)
ax2.plot(q̇s);
```

Pero las gráficas de trayectoria son aburridas, recordemos que podemos hacer una animación con matplotlib:

```python
from matplotlib import animation
from numpy import sin, cos, arange
```

```python
# Se define el tamaño de la figura
fig = figure(figsize=(8, 8))

# Se define una sola grafica en la figura y se dan los limites de los ejes x y y
axi = fig.add_subplot(111, autoscale_on=False, xlim=(-1.5, 1.5), ylim=(-2, 1))

# Se utilizan graficas de linea para el eslabon del pendulo
linea, = axi.plot([], [], "-o", lw=2, color='gray')

def init():
    # Esta funcion se ejecuta una sola vez y sirve para inicializar el sistema
    linea.set_data([], [])
    return linea

def animate(i):
    # Esta funcion se ejecuta para cada cuadro del GIF

    # Se obtienen las coordenadas x y y para el eslabon
    xs, ys = [[0, cos(qs[i])], [0, sin(qs[i])]]
    linea.set_data(xs, ys)

    return linea

# Se hace la animacion dandole el nombre de la figura definida al principio, la funcion que
# se debe ejecutar para cada cuadro, el numero de cuadros que se debe de hacer, el periodo
# de cada cuadro y la funcion inicial
ani = animation.FuncAnimation(fig, animate, arange(1, len(qs)), interval=25,
                              blit=True, init_func=init)

# Se guarda el GIF en el archivo indicado
ani.save('./imagenes/pendulo-simple.gif', writer='imagemagick');
```

# Problemas

1. Realiza una gráfica de trayectoria y una animación de un pendulo doble.

Practicas/.ipynb_checkpoints/Practica 5 - Modelado de Robots-checkpoint.ipynb


# Neuromatch Academy: Week 1, Day 5, Tutorial 2
# Dimensionality Reduction: Principal component analysis

---

In this notebook we'll learn how to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

Steps:
 1. Calculate the eigenvectors of the sample covariance matrix.
 2. Perform PCA by projecting data onto the eigenvectors of the covariance matrix.
 3. Plot the eigenvalues.

---

```python
#@title Video: PCA
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="Ze0uK1ipEn0", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=Ze0uK1ipEn0

# Setup
Run these cells to get the tutorial started.

```python
#library imports
import time                        # import time
import numpy as np                 # import numpy
import scipy as sp                 # import scipy
import math                        # import basic math functions
import random                      # import basic random number generator functions

import matplotlib.pyplot as plt    # import matplotlib
from IPython import display
```

```python
#@title Figure Settings
%matplotlib inline
fig_w, fig_h = (8, 8)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
plt.style.use('ggplot')
%config InlineBackend.figure_format = 'retina'
```

```python
#@title Helper functions
# new helper functions
def plot_eigenvalues(evals):
  """
  Plots eigenvalues.

  Args:
     (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """
  plt.plot(np.arange(1,len(evals)+1),evals,'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
  plt.xticks(np.arange(1,len(evals)+1))
  plt.ylim([0,2.5])

def sort_evals_descending(evals,evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats):   Vector of eigenvalues
    evectors (numpy array of floats):   Corresponding matrix of eigenvectors
                                each column corresponds to a different eigenvalue

  Returns:
    (numpy array of floats) : Vector of eigenvalues after sorting
    (numpy array of floats) : Matrix of eigenvectors after sorting
  """
  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:,index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:,0], 1./np.sqrt(2)*np.array([1,1]))) > np.pi/2.:
      evectors[:,0] = -evectors[:,0]
    if np.arccos(np.matmul(evectors[:,1], 1./np.sqrt(2)*np.array([-1,1]))) > np.pi/2.:
      evectors[:,1] = -evectors[:,1]
  return evals, evectors

  # Old helper functions from Tutorial 1

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  scatter plot of their joint activity. The title indicates the sample correlation
  calculated from the data.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(X[:,0],color='k')
  plt.ylabel('Neuron 1')
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(X[:,1],color='k')
  plt.xlabel('Sample Number (sorted)')
  plt.ylabel('Neuron 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:,0],X[:,1],'.',markerfacecolor=[.5,.5,.5], markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:,0],X[:,1])[0,1]))

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random variable.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    cov_matrix (numpy array of floats): desired covariance matrix

  Returns:
    (numpy array of floats) : samples from the bivariate Gaussian, with
                              each column corresponding to a different random variable
  """
  mean = np.array([0,0])
  X = np.random.multivariate_normal(mean,cov_matrix,size = 1000)
  indices_for_sorting = np.argsort(X[:,0])
  X = X[indices_for_sorting,:]
  return X

def calculate_cov_matrix(var_1,var_2,corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation coefficient.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    corr_coef (scalar):      correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1,cov],[cov,var_2]])
  return cov_matrix

def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats):    arbitrary 2D vector used for new basis

  Returns:
    (numpy array of floats) : new orthonormal basis
                                columns correspond to basis vectors
  """
  u = u / np.sqrt(u[0]**2 + u[1]**2)
  w = np.array([-u[1],u[0]])
  W = np.column_stack((u,w))
  return W

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data but
  with colors corresponding to projections onto basis 1 (red) and basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random variable.

  Args:
    Y (numpy array of floats):   Data matrix in new basis
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(Y[:,0],'r')
  plt.ylabel('Projection \n basis vector 1')
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(Y[:,1],'b')
  plt.xlabel('Projection \n basis vector 1')
  plt.ylabel('Projection \n basis vector 2')
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:,0],Y[:,1],'.',color=[.5,.5,.5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:,0],Y[:,1])[0,1]))

def change_of_basis(X,W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix
                                each column corresponding to a different random variable
    W (numpy array of floats):  new orthonormal basis
                                columns correspond to basis vectors

  Returns:
    (numpy array of floats) : Data matrix expressed in new basis
  """
  Y = np.matmul(X,W)
  return Y

def plot_basis_vectors(X,W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable
    W (numpy array of floats):   Square matrix representing new orthonormal basis
                                  each column represents a basis vector

  Returns:
    Nothing.
  """
  plt.figure(figsize=[4,4])
  plt.plot(X[:,0],X[:,1],'.',color=[.5,.5,.5],label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0,W[0,0]],[0,W[1,0]],color='r',linewidth=3,label = 'Basis vector 1')
  plt.plot([0,W[0,1]],[0,W[1,1]],color='b',linewidth=3,label = 'Basis vector 2')
  plt.legend()

```

# Calculate the eigenvectors of the the sample covariance matrix.

PCA represents data in a new basis defined by the eigenvectors of the covariance matrix. In the previous tutorial, we generated bivariate normal data with a specified covariance matrix $\bf \Sigma$, whose $(i,j)$th element is:
\begin{equation}
\Sigma_{ij} = E[ x_i x_j ] - E[ x_i] E[ x_j ] .
\end{equation}
However, in real life we don't have access to the ground-truth covariance matrix. To get around this, we can use the sample covariance matrix, $\bf\hat\Sigma$, calculated directly from the data. The $(i,j)$th element of the sample covariance matrix is:
\begin{equation}
 \hat \Sigma_{ij} =  \frac{1}{N_\text{samples}}{\bf x}_i^T {\bf x}_j - \bar {\bf x}_i \bar{\bf x}_j ,
\end{equation}
where ${\bf x}_i = [ x_i(1), x_i(2), \dots,x_i(N_\text{samples})]^T$ is a column vector representing all measurements of neuron i, and  $\bar {\bf x}_i$ is a scalar representing its sample mean:
\begin{equation}
\bar {\bf x}_i = \frac{1}{N_\text{samples}} \sum_{k=1}^{N_\text{samples}} x_i(k).
\end{equation}
If we assume that the data has already been mean-subtracted, then we can write:
\begin{align}
{\bf \hat \Sigma}
&= \frac{1}{N_\text{samples}} {\bf X}^T {\bf X}.
\end{align}

---

#### Exercise

Before we move on to calculating the eigenvectors, we'll first calculate the sample covariance matrix from the data. Your job is to finish the function `get_sample_cov_matrix`. Then you can use the provided code to generate bivariate normal data with `get_data`, and calculate the sample covariance matrix with your finished `get_sample_cov_matrix`. Then you can compare this estimate to the true covariate matrix using `calculate_cov_matrix`,

```python
help(get_data)
help(calculate_cov_matrix)
```

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

        Note that samples are sorted in ascending order for the first random variable.

        Args:
          var_1 (scalar):    variance of the first random variable
          var_2 (scalar):    variance of the second random variable
          cov_matrix (numpy array of floats): desired covariance matrix

        Returns:
          (numpy array of floats) : samples from the bivariate Gaussian, with
                                    each column corresponding to a different random variable

    Help on function calculate_cov_matrix in module __main__:

    calculate_cov_matrix(var_1, var_2, corr_coef)
        Calculates the covariance matrix based on the variances and correlation coefficient.

        Args:
          var_1 (scalar):    variance of the first random variable
          var_2 (scalar):    variance of the second random variable
          corr_coef (scalar):      correlation coefficient

        Returns:
          (numpy array of floats) : covariance matrix

```python
def get_sample_cov_matrix(X):
  """
    Returns the sample covariance matrix of data X

    Args:
      X (numpy array of floats):   Data matrix
                                   each column corresponds to a different random variable

    Returns:
      (numpy array of floats) : Covariance matrix
  """
  ###################################################################
  ## Insert your code here to:
  ##                Subtract the mean of X
  ##                Calculate the covariance matrix (hint: use np.matmul)

  #X = ...
  #cov_matrix = ...

  #uncomment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  ###################################################################
  return cov_matrix

## Uncomment below code to:
##                generate bivariate Gaussian data with variances of 1
##                and a correlation coefficient of 0.8
##                compare the true and sample covariance matrices
###################################################################
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

#cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
#print(cov_matrix)

#X = get_data(cov_matrix)
#sample_cov_matrix = get_sample_cov_matrix(X)
#print(sample_cov_matrix)

```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial2_Solution_279f87fc.py)

#### Exercise
Next we will compare the eigenvectors of the covariance matrix to the geometrical view of the data.

**Suggestions:**
* Calculate the eigenvalues and eigenvectors of the sample covariance matrix. (**Hint:** use `np.linalg.eig`).
* Use the provided code to sort the eigenvalues in descending order.
* Plot the eigenvectors on a scatter plot of the data, using the function `plot_basis_vectors`.

```python
help(sort_evals_descending)
help(plot_basis_vectors)
```

    Help on function sort_evals_descending in module __main__:

    sort_evals_descending(evals, evectors)
        Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
        eigenvectors to be in first two quadrants (if 2D).

        Args:
          evals (numpy array of floats):   Vector of eigenvalues
          evectors (numpy array of floats):   Corresponding matrix of eigenvectors
                                      each column corresponds to a different eigenvalue

        Returns:
          (numpy array of floats) : Vector of eigenvalues after sorting
          (numpy array of floats) : Matrix of eigenvectors after sorting

    Help on function plot_basis_vectors in module __main__:

    plot_basis_vectors(X, W)
        Plots bivariate data as well as new basis vectors.

        Args:
          X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable
          W (numpy array of floats):   Square matrix representing new orthonormal basis
                                        each column represents a basis vector

        Returns:
          Nothing.

```python
###################################################################
## Insert your code here to:
##                Calculate the eigenvalues and eigenvectors
##                Sort the eigenvalues in descending order
##                plot the eigenvectors against the data
###################################################################

# evals, evectors = ...
# evals, evectors = ..
# plot_basis_vectors(X,evectors)

```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial2_Solution_5dcb5b4d.py)

*Example output:*

# Perform PCA by projecting data onto the eigenvectors.

To perform PCA, we will project the data onto the eigenvectors of the covariance matrix, i.e.:
\begin{equation}
\bf S = X W
\end{equation}
where $\bf S$ is an $N_\text{samples} \times N$ matrix representing the projected data (also called *scores*), and $W$ is an $N\times N$ orthogonal matrix, each of whose columns represents the eigenvectors of the covariance matrix (also called *weights* or *loadings*).

#### Exercise
We will now perform PCA on the data using the intuition we have built. Fill in the function below to carry out the steps to perform PCA by projecting the data onto the eigenvectors of its covariance matrix.

**Suggestions**
* First subtract the mean.
* Then calculate the covariance.
* Then find the eigenvalues and order them in descending order.
* Finally project the mean-centered data onto the eigenvectors.

```python
help(change_of_basis)
help(plot_data_new_basis)
```

    Help on function change_of_basis in module __main__:

    change_of_basis(X, W)
        Projects data onto a new basis.

        Args:
          X (numpy array of floats) : Data matrix
                                      each column corresponding to a different random variable
          W (numpy array of floats):  new orthonormal basis
                                      columns correspond to basis vectors

        Returns:
          (numpy array of floats) : Data matrix expressed in new basis

    Help on function plot_data_new_basis in module __main__:

    plot_data_new_basis(Y)
        Plots bivariate data after transformation to new bases. Similar to plot_data but
        with colors corresponding to projections onto basis 1 (red) and basis 2 (blue).
        The title indicates the sample correlation calculated from the data.

        Note that samples are re-sorted in ascending order for the first random variable.

        Args:
          Y (numpy array of floats):   Data matrix in new basis
                                        each column corresponds to a different random variable

        Returns:
          Nothing.

```python
def pca(X):
  """
  Sorts eigenvalues and eigenvectors in decreasing order.

  Args:
     X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    (numpy array of floats) : Data projected onto the new basis
    (numpy array of floats) : Vector of eigenvalues
    (numpy array of floats) : Corresponding matrix of eigenvectors

  """
  ###################################################################
  ## Insert your code here to ...
  ##                Subtract the mean of X
  ##                Calculate the sample covariance matrix
  ##                Calculate the eigenvalues and eigenvectors
  ##                Sort the eigenvalues in descending order
  ##                Project the data onto the new eigenvector basis
  ###################################################################
  return score, evectors, evals

###################################################################
## Insert your code here to:
##                perform PCA on the data matrix X
##                Plot the data projected into the new basis
###################################################################

# score, evectors, evals = ...
# plot_data_new_basis(score)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5_DimensionalityReduction/solutions/W1D5_Tutorial2_Solution_612334f2.py)

*Example output:*

# Plot and explore the eigenvalues.

```python
#@title Video: Properties of PCA
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="uPcrB8B9S7A", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=uPcrB8B9S7A

Finally, we will examine the eigenvalues of the covariance matrix, which represent the variance of the projected data. Run the code below to plot the eigenvalues (sometimes called the "scree plot"). Which eigenvalue is larger?

```python
 plot_eigenvalues(evals)
```

Now, run the following cell and use the slider to change the correlation coefficient in the data. You should see the scree plot and the plot of basis vectors update.

**Questions:**
* What happens to the eigenvalues as you change the correlation coefficient?
* Can you find a value when both eigenvalues are equal?
* Can you find a value when only one eigenvalue is nonzero?

```python
###### MAKE SURE TO RUN THIS CELL VIA THE PLAY BUTTON TO ENABLE SLIDERS ########

import ipywidgets as widgets

def refresh(corr_coef = .8):
  cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
  X = get_data(cov_matrix)
  score, evectors, evals = pca(X)
  plot_eigenvalues(evals)
  plot_basis_vectors(X,evectors)

_ = widgets.interact(refresh,
  corr_coef = (-1, 1, .1))

```