```python
%%HTML
<style>
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}
</style>
```

# Metody Numeryczne

## Elementy analizy numerycznej

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Informacje ogólne
- Katedra Automatyki i Robotyki, C3, p. 214
- Konsultacje
    - Czwartki 11:00-12:00
    (o ile nie ma Kolegium Wydziałowego lub seminarium)
- jb@agh.edu.pl
- wykłady dostępne tutaj: https://github.com/KAIR-ISZ/public_lectures

# Reprezentacja liczb

## Kod binarny

- Zapis liczby z wykorzystaniem dwóch symboli **1** i **0**
- Podstawa współczesnego sposobu reprezentacji informacji

## Zamierzchła historia

- Pingala, Chandaḥśāstra i Prozodia
    - Ok. 4 wiek pne
    - Wykorzystanie zapisu w formie zer i jedynek do opisu metrum
- Chiny, hexagramy, Shao Yong, I-Ching
- Leibniz

## Algebra Boole'a

$$
\begin{align}
x \land y & = xy & \mathsf{Koniunkcja}\\
x \lor y & = x+y-xy & \mathsf{Alternatywa}\\
\neg x & =1-x & \mathsf{Negacja}\\
x \rightarrow y & = (\neg x\lor y) & \mathsf{Implikacja}\\
x \oplus y & = (x \lor y)\land\neg(x\land y) & \mathsf{EXOR}\\
x = y & = \neg(x\oplus y) & \mathsf{Równoważność}\\
\end{align}
$$

## Nieco mniej zamierzchła historia
- 1937 Shannon – przekaźnikowa realizacja operacji binarnych i algebry Boole’a
- 1937 Stibitz – Pierwszy komputer przekaźnikowy (dodawanie)

## Kod binarny
| **0**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =2^5+2^3+2^1+2^0=32+8+2+1=43$

## Liczby naturalne
- Ogólnie zakres od 0 do 2n-1
- 8 bit – zakres od 0 do 255
- 16 bit – zakres od 0 do 65,535 (short, int)
- 32 bit – zakres od 0 do 4,294,967,295 (long)

W Pythonie i matlabie za bardzo nie przejmujemy się typami, chyba że je wymusimy

## Operacje na liczbach binarnych

- Dodawanie
    - 0+0=0
    - 0+1=1
    - 1+0=1
    - 1+1=0, przenieś 1
- Jak w dodawaniu pisemnym

`` 1 1 1 1 1   ``(cyfry przenoszone)
``   0 1 1 0 1 ``(1310)
``+  1 0 1 1 1 ``(2310)
``------------ ``
``=1 0 0 1 0 0 `` (3610)

## Operacje na liczbach binarnych

- Odejmowanie
    - 0-0=0
    - 0-1=1, pożyczka 1
    - 1-0=1
    - 1-1=0,
- Analogicznie

``    *   * * *  ``(pożyczki)
``  1 1 0 1 1 1 0``(11010)
``-     1 0 1 1 1``(2310)
``--------------- ``
``= 1 0 1 0 1 1 1`` (8710)

## Co z liczbami ujemnymi?

Uzupełniamy zapis o tzw. bit znaku

| **1**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| S       | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =(-1)^1(2^3+2^1+2^0)=-(8+2+1)=-11$

Zmieniają się zakresy:
- 8 bit (-128 do 127)
- 16 bit (−32,768 do 32,767)
- itd

## Problemy

- Niepraktyczny zapis
- Trzeba przekodowywać wyniki operacji
- Potencjalnie podatniejsze na błędy

## Kod uzupełnienia do 2 (U2)
| **1**   | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $-2^{7}$| $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =-2^7+2^6+2^5+2^4+2^3+2^1+2^0$

$=-128+64+32+16+8+2+1=-5$

## Bardzo łatwa konwersja
- Liczby dodatnie są takie same jak były
- Aby zamienić liczbę na jej przeciwną wystarczy zanegować wszystkie bity i do wyniku dodać 1 (*w obie strony*)

| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | oryginał  |
|----------|---------|---------|---------|---------|---------|---------|---------|-----------------|-----------|
| 1    | 1   | 1   | 1   | 1   | 0   | 1   | 0   |                 | negacja   |
| **1**    | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   | -510 | dodanie 1 |
| 0    | 0   | 0   | 0   | 0   | 1   | 0   | 0   |                 | negacja   |
| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | dodanie 1 |
| -$2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |                 |           |

## Jaka z tego korzyść?
- Odejmowanie staje się dodawaniem (prawie)
$$ A - B = A + \neg B + 1$$
- Przykład 13 – 7 (na 8 bitach)

``  1 1 1 1     1  ``(cyfry przenoszone)
``  0 0 0 0 1 1 0 1``(1310)
``  1 1 1 1 1 0 0 0``(zanegowane 710)
``+               1``(jedynka)
``-----------------``
``= 0 0 0 0 0 1 1 0`` (610)

## Operacje na liczbach binarnych
Mnożenie również przypomina mnożenie pisemne

``          1 0 1 1`` 1110
``        * 1 0 1 0`` 1010
``      -----------``
``          0 0 0 0``
``  +     1 0 1 1   ``
``  +   0 0 0 0``
``  + 1 0 1 1``
``  ---------------``
``  = 1 1 0 1 1 1 0`` 11010

# Metody Numeryczne

## Reprezentacja liczb wymiernych

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## A co z ułamkami?
Są dwa sposoby zapisu liczb niecałkowitych
- Stałoprzecinkowy (stałopozycyjny)
- Zmiennoprzecinkowy (zmiennopozycyjny)

## Zapis stałoprzecinkowy
| **1**   | **0**   | **1**   | **1**   | **1**   | **0**   | **0**   | **0**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ |

$$
2^1+2^{-1}+2^{-2}+2^{-3}=2+\frac{1}{2}+\frac{1}{4}+\frac{1}{8}=2.875
$$

## Zalety zapisu stałoprzecinkowego
- Nie ma różnicy w kodowaniu
- Mamy stale określoną dokładność, którą możemy w miarę dokładnie kształtować
- Stosunkowa prostota
- Małe wymagania sprzętowe

## Wady zapisu stałoprzecinkowego
Problemy z dokładnością, np. nie da się dokładnie przedstawić liczby 0.1
- Na 3 bitach części ułamkowej różnica wynosi 0.025
- Na 7 bitach części ułamkowej różnica wynosi ok. 0.001

## Jak wykonujemy działania?
- Działania wykonujemy traktując zapis liczby stałoprzecinkowej jako normalną binarną
- Kod U2 dalej działa
- Należy pamiętać, że wtedy liczba jest pomnożona przez 2n gdzie n to ilość bitów części ułamkowej
- W liczbach poddanych działaniu liczba bitów części całkowitej i ułamkowej musi być równa

## Działania stałoprzecinkowe
- Dodawanie wykonujemy identycznie
- W przypadku mnożenia wynik musimy podzielić przez 2n
- Mnożenie liczb stałoprzecinkowych przez potęgę 2 polega tylko na przesuwaniu bitów (bardzo proste w realizacji)

| 1             | 0             | 1   | 1   | 1   | 0   | 0   | 0   |   |
|---------------|---------------|-----|-----|-----|-----|-----|-----|---|
| 0             | 0             | 1   | 0   | 1   | 1   | 1   | 0   | Podzielenie przez $2^{2}$  |
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ ||

## Format zmiennoprzecinkowy
- Bardziej zaawansowany sposób przedstawiania liczb
- Ustandaryzowany normą IEEE
- Dający pod pewnymi względami większą dokładność

## Format zmiennoprzecinkowy

Reprezentacja liczby

$$
x=S\cdot M\cdot B^E
$$

- S – znak (*sign*)
- M – mantysa (*mantissa*, także *fraction*)
- B – podstawa (*base*, zazwyczaj 2, rzadziej 10)
- E - wykładnik (*exponent*)

## Mantysa
- Liczba odpowiadająca za ułamkową część zapisu
- Format stałoprzecinkowy, zazwyczaj liczba z przedziału [1,2)

## Podstawa i wykładnik

- Pozwalają na określenie szerokiego zakresu
- Ze względu na kodowanie, zazwyczaj podstawa to 2
- Wykładnik może być ujemny lub dodatni.
- Wykładnik koduje się w U2, lub też wprowadza się przesunięcie

## Działania na liczbach zmiennoprzecinkowych
Dodawanie i odejmowanie

$$
x_1\pm x_2=\left(M_1\pm M_2\cdot B^{E_2-E_1}\right)\cdot B^{E_1}
$$

Mnożenie i dzielenie

$$
x_1\cdot x_2=(S_1\cdot S_2)\cdot (M_1\cdot M_2)\cdot B^{E_1+E_2}
$$

$$
x_1 / x_2=(S_1\cdot S_2)\cdot (M_1/ M_2)\cdot B^{E_1-E_2}
$$

## Dzielenie
- Mając możliwość zapisu liczby ulamkowej można sformułować operację dzielenia.
- Istnieje wiele algorytmów np.
    - *restoring division*
    - *non-restoring division*
    - SRT
    - algorytm Newtona-Raphsona
    - algorytm Goldschmidta
- Są one już zaimplementowane, jedno dzielenie zazwyczaj wymaga przeprowadzenia 3-4 mnożeń

## Ważne formaty – IEEE Single precision

- 8 bitów wykładnika, wykładnik przesunięty o 127 (zamiana z -126 do 127 na 1 do 244)
- 24 bity mantysy, ale zawsze koduje się tylko 23 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float32``

## Ważne formaty – IEEE Double precision

- 11 bitów wykładnika, wykładnik przesunięty o 1023 (zamiana z -1022 do 1023 na 1 do 2046)
- 53 bity mantysy, ale zawsz koduje się tylko 52 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float64``, ale w zasadzie każda liczba w Pythonie i Matlabie to double, chyba że wymusimy inaczej

## Wyświetlanie liczb
- Normalnie
- Notacja inżynierska
    - $3700=3.7\cdot10^3$, $0.12=120\cdot10^{-3}$
- Notacja naukowa
    - ``3700=3.7E3``, ``0.12=1.2E-1``

# Metody Numeryczne

## Błedy numeryczne

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Podstawowe definicje
Wartość dokładna
$$y=\tilde{y}+\varepsilon$$
- $\tilde{y}$ - wartość przybliżona
- $\varepsilon$ - błąd

## Błąd bezwzględny
Wartość bezwzględna różnicy między rozwiązaniem dokładnym i przybliżonym
$$ \varepsilon=|y-\tilde{y}|$$

## Błąd względny
Stosunek błędu bezwzględnego do wartości bezwzględnej rozwiązania
$$\eta=\frac{|y-\tilde{y}|}{|y|}=\left|\frac{y-\tilde{y}}{y}\right|=\left|1-\frac{\tilde{y}}{y}\right|$$
Czasami błąd względny wyrażamy w procentach

## Przykłady
Pierwiastek kwadratowy ze 122

$$
\begin{align}
y{}&=\sqrt{122}\approx 11.04536\\
\tilde{y}{}&=11\\
\varepsilon{}&=|y-\tilde{y}|=0.04536\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00411
\end{align}
$$

## Przykłady
Liczba obywateli Polski (stan na ostatni spis powszechny z 2011)

$$
\begin{align}
y{}&=38\ 538\ 447\\
\tilde{y}{}&=38\ 500\ 000\\
\varepsilon{}&=|y-\tilde{y}|=38\ 447\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=9.97627\cdot10^{-4}\approx 0.001
\end{align}
$$

## Przykłady
Obliczanie stałej grawitacji
$$
\begin{align}
y{}&=6.673841\cdot10^{-11}\\
\tilde{y}{}&=6.7\cdot10^{-11}\\
\varepsilon{}&=|y-\tilde{y}|=2.6159\cdot10^{-13}\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00391
\end{align}
$$

## Źródła błędów
Błędy powstające przy formułowaniu zagadnienia
- Błędy pomiaru
- Błędy wynikające z przyjęcia określonych przybliżeń opisu zjawisk fizycznych

Błędy powstające przy obliczeniach
- Błędy grube (pomyłki)
- Błędy metody (obcięcia)
- Błędy zaokrągleń

## Błędy grube
- Błąd przy wpisywaniu wzoru do komputera
np. ``x=A/b`` zamiast ``x=A\b``
- Zła implementacja algorytmu
- Niewłaściwa kolejność wykonywania działań

## Błędy metody (obcięcia)
- Błędy obcięcia są nieodłącznym elementem obliczeń numerycznych.
- Błąd obcięcia jest to błąd wynikający z tego, że do uzyskania dokładnego rozwiązania potrzebujemy wykonać nieskończenie wiele obliczeń

## Przykłady błędów metody
Można wykazać, że
$$
\begin{align}
\sin x={}&x-\frac{x^3}{3!}+\frac{x^5}{5!}-\frac{x^7}{7!}+\ldots=\\
={}&\sum\limits_{n=0}^\infty(-1)^n\frac{x^{2n+1}}{(2n+1)!}
\end{align}
$$
Błędem odcięcia będzie
$$
\sin x\approx x-\frac{x^3}{3!}+\frac{x^5}{5!}
$$

## Przykłady błędów metody

Metoda bisekcji

```python
def bisection(f,a,b,N):
    a_n = a
    b_n = b
    for n in range(1,N+1):
        m_n = (a_n + b_n)/2
        f_m_n = f(m_n)
        if f(a_n)*f_m_n < 0:
            a_n = a_n
            b_n = m_n
        elif f(b_n)*f_m_n < 0:
            a_n = m_n
            b_n = b_n
    return (a_n + b_n)/2
```

Szukamy pierwiastka wielomianu $x^2-2$, w przedziale $[1,2]$. Rozwiązanie to $\sqrt{2}$.

```python
f = lambda x: x**2 - 2 # definicja funkcji
bisection(f,1,2,5) # 5 kroków
```

    1.421875

```python
bisection(f,1,2,10) # 10 kroków
```

    1.41455078125

```python
bisection(f,1,2,15) # 15 kroków
```

    1.4141998291015625

```python
import numpy as np
np.sqrt(2)
```

    1.4142135623730951

## Błąd metody - podsumowanie
- Praktycznie wszystkie metody numeryczne mają jakiś błąd metody
- Dobre algorytmy podają jednak jego oszacowanie, w ten sposób wiemy jak daleko jesteśmy od rozwiązania nawet jak przerwiemy obliczenia

# Metody Numeryczne

## Błędy zaokrągleń

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Błędy zaokrągleń
Kolejne nieusuwalne w pełni źródło błędów, nad którym mamy mniejszą kontrolę niż nad błędem metody

## Zaokrąglenie i cyfry znaczące
Liczba $\tilde{y}=\mathrm{rd}(y)$ jest poprawnie zaokrąglona do *d* miejsc po przecinku, jeżeli

$$
\varepsilon=|y-\tilde{y}|\leq\frac{1}{2}\cdot10^{-d}
$$
*k*-tą cyfrę dziesiętną liczby $\tilde{y}$ nazwiemy znaczącą gdy
$$|y-\tilde{y}|\leq\frac{1}{2}\cdot10^{-k}$$
oraz
$$|\tilde{y}|\geq10^{-k}
$$

## Rzeczywiste obliczenia zmiennoprzecinkowe
$$
\begin{align}
\mathrm{fl}(x+y)={}&\mathrm{rd}(x+y)\\
\mathrm{fl}(x-y)={}&\mathrm{rd}(x-y)\\
\mathrm{fl}(x\cdot y)={}&\mathrm{rd}(x\cdot y)\\
\mathrm{fl}(x/y)={}&\mathrm{rd}(x/y)\\
\end{align}
$$

## Liczby maszynowe
- Liczba maszynowa, to taka liczba jaką można przedstawić w komputerze. Zbiór tych liczb oznaczamy A
- Dokładność maszynową (epsilon maszynowy) – eps, $\varepsilon_m$,  definiujemy:
$$
\mathrm{eps}=\min\{x\in{A}\colon \mathrm{fl}(1+x)>1,\ x>0\}
$$
Innymi słowy, jest to najmniejsza liczba, którą możemy dodać do 1, aby uzyskać coś większego od 1.

## Epsilon maszynowy w różnych formatach

Zależy on od liczby bitów na część ułamkową
- Single precision $\varepsilon_m=2^{-24}\approx 5.96\cdot10^{-8}$
- Double precision $\varepsilon_m=2^{-52}\approx 1.11\cdot10^{-16}$

### Przykład

```python
a=10**(-15)
b=10**(-17)
1+a>1,1+b>1
```

    (True, False)

## Maksymalny błąd reprezentacji
Dla każdej liczby rzeczywistej $x$ istnieje taka liczba $\varepsilon$, taka że $|\varepsilon|<\varepsilon_m$, że
$\mathrm{fl}(x)=x(1+\varepsilon)$

Oznacza to, że **błąd względny między liczbą rzeczywistą, a jej najbliższą reprezentacją zmiennoprzecinkową jest zawsze mniejszy od $\varepsilon_m$**

## Lemat Wilkinsona
Błedy zaokrągleń powstałe podczas wykonywania działań zmiennoprzecinkowych są równoważne zastępczemu zaburzeniu liczb, na których wykonujemy działania

$$
\begin{align}
\mathrm{fl}(x+y)={}&(x+y)(1+\varepsilon_1)\\
\mathrm{fl}(x-y)={}&(x-y)(1+\varepsilon_2)\\
\mathrm{fl}(x\cdot y)={}&(x\cdot y)(1+\varepsilon_3)\\
\mathrm{fl}(x/y)={}&(x/y)(1+\varepsilon_4)\\
|\varepsilon_i|<{}&\varepsilon_m
\end{align}
$$
(dla każdej pary liczb $x,\ y$ zaburzenia zastępcze $\varepsilon_i$ są inne)

## Konsekwencja lematu Wilkinsona
Prawa łączności i rozdzielności operacji matematycznych są ogólnie nieprawdziwe dla obliczeń zmiennoprzecinkowych

### Przykład

```python
a=np.float32(0.23371258*10**(-4))
b=np.float32(0.33678429*10**(2))
c=np.float32(-0.33677811*10**(2))
print([a,b,c])
```

    [2.3371258e-05, 33.67843, -33.67781]

Chcemy obliczyć ``a+b+c``

## Obliczenia

```python
## Podejście 1
d=b+c
wynik_1=a+d
print(wynik_1)
```

    0.0006413522

```python
## Podejście 2
e=a+b
wynik_2=e+c
print(wynik_2)
```

    0.00064086914

## Co tu się porobiło?

## Konsekwencje obliczen zmiennoprzecinkowych

```python
m_a, e_a = np.frexp(a)
print(m_a,e_a)
m_b,e_b = np.frexp(b)
print(m_b,e_b)
m_c,e_c = np.frexp(c)
print(m_c,e_c)
```

    0.7658294 -15
    0.52622545 6
    -0.5262158 6

Wykładnik ``a`` od wykładników ``b`` i ``c`` różni się o 21. Oznacza to, że z 23 bitów mantysy liczby ``a`` po sprowadzeniu do wspólnego wykładnika z ``b`` zostaną nam tylko 2 najbardziej znaczące.

## Konsekwencje cd..
Jeżeli dodajemy małą liczbę do dużej, zawsze musimy się liczyć z zaokrągleniem i to normalne. W tym przypadku jednak dwie duże liczby ``b`` i ``c`` są przeciwnych znaków i bliskie co do wartości bezwzględnej. Wynik tego działania:

```python
m_d,e_d = np.frexp(d)
print(m_d,e_d)
print(wynik_2)
```

    0.6328125 -10
    0.00064086914

W konsekwencji dodając ``a`` do ``d`` na zaokrągleniu stracimy jedynie 5 bitów mantysy ``a``.

## O ile się pomyliliśmy (w stosunku do dokładniejszych obliczeń)

```python
a_dbl=(0.23371258*10**(-4))
b_dbl=(0.33678429*10**(2))
c_dbl=(-0.33677811*10**(2))
d_dbl=b_dbl+c_dbl
wynik_dbl=a_dbl+d_dbl
epsilon_1=np.abs((wynik_1)-wynik_dbl)
eta_1=epsilon_1/np.abs(wynik_dbl)
print("Metoda 1: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_1,eta_1))
epsilon_2=np.abs((wynik_2)-wynik_dbl)
eta_2=epsilon_2/np.abs(wynik_dbl)
print("Metoda 2: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_2,eta_2))

```

    Metoda 1: Błąd bezwzględny   1.91e-08, Błąd względny   2.97e-05
    Metoda 2: Błąd bezwzględny   5.02e-07, Błąd względny   7.83e-04

# Przenoszenie się błędów zaokrągleń
Korzystając z rachunku różniczkowego (różniczkowa analiza błędów) możemy podać wzór na przenoszenie się błędów.

Niech $y=\varphi(x_1,\ x_2,,\ldots\ x_n)$ będzie wielkością, którą chcemy obliczyć a $x_i$ są zaokrąglone z błędem $\varepsilon_{x_i}$. Błąd względny wyliczania $y$ wynosi w przybliżeniu:

$$
\varepsilon_y = \sum_{i=0}^n \frac{x_i}{\varphi(\mathbf{x})}
\cdot \frac{\partial\varphi(\mathbf{x})}{\partial x_i}\cdot\varepsilon_{x_i}
$$

# Nieunikniony błąd obliczeń
Ze względu na zaokrąglenia pewnych błędów nigdy nie unikniemy. Nieunikniony błąd wartości składa się z błędu wyliczenia wartości (przeniesienia błędów) oraz samego błędu zaokrąglenia:

$$
\frac{\Delta y}{y} = \epsilon_y + \mathrm{eps}
$$

## Przykład
Wyliczanie pierwiastka równania kwadratowego $y^2+2py-q=0$ o mniejszej wartości bezwzględnej:
$$ y=-p+\sqrt{p^2+q} $$
można policzyć, że
$$
\varepsilon_y=-\frac{p}{\sqrt{p^2-q}}\varepsilon_p+\frac{p+\sqrt{p^2-q}}{2\sqrt{p^2-q}}\varepsilon_q
$$

## Analiza błędu nieuniknionego
Ponieważ dla $q>0$ mamy

$$
\left|\frac{p}{\sqrt{p^2-q}}\right|\leq1,\quad \left|\frac{p+\sqrt{p^2-q}}{2\sqrt{p^2-q}}\right|\leq1
$$
to wtedy mamy (przyjmując, że nie zachodzi $p^2\approx q$)
$$
\mathrm{eps}\leq\left|\frac{\Delta y}{y}\right| = |\epsilon_y + \mathrm{eps}|\leq 3 \mathrm{eps}
$$

## Porównanie algorytmów
Rozpartrzmy dwa sposoby wyliczania $y$ dla $p$ i $q$ mniejszych od zera

$$
\begin{aligned}
s:={}&p^2\\
t:={}&s+q\\
u:={}&\sqrt{t}\\
y:={}&-p+q
\end{aligned}
\quad \quad \quad \quad
\begin{aligned}
s:={}&p^2\\
t:={}&s+q\\
u:={}&\sqrt{t}\\
v:={}&p+u\\
y:={}&q/v
\end{aligned}
$$

## Algorytm 1
Podstawowym źródłem błędu będzie wzmocnienie błędu zaokrąglenia wyliczania pierwiastka z $t$ poprzez odejmowanie dwóch liczb przy wyliczaniu $y$
$$\varepsilon_y=\frac{p\sqrt{p^2+q}+p^2+q}{q}\varepsilon=\kappa\varepsilon$$
$\kappa$ można oszacować z dołu, przez
$$
\kappa>\frac{2 p^2}{q} >0
$$
co oznacza, że dla małych $q$ błąd obliczeń będzie dużo większy niż błąd nieunikniony.

## Algorytm 2
W tym algorytmie zakokrąglenie przez odejmowanie nie wystąpi

$$
\varepsilon_y = -\frac{\sqrt{p^2+q}}{p+\sqrt{p^2+q}}\varepsilon = \kappa\varepsilon
$$
w tym przypadku zawsze $|\kappa|<1$.

```python
def algorytm_1(p,q):
    s=p**2
    t=s+q
    u=np.sqrt(t)
    return u-p

def algorytm_2(p,q):
    s=p**2
    t=s+q
    u=np.sqrt(t)
    v=p+u
    return q/v
```

# Porównanie obliczeń

```python
p=1000
q=0.018000000081
exact_sol=np.max(np.roots([1,2*p,-q]))
```

```python
epsilon_1=np.abs((algorytm_1(p,q))-exact_sol)
eta_1=epsilon_1/np.abs(exact_sol)
epsilon_2=np.abs((algorytm_2(p,q))-exact_sol)
eta_2=epsilon_2/np.abs(exact_sol)
```

```python
print('Algorytm 1')
print(algorytm_1(p,q))
print('Algorytm 2')
print(algorytm_2(p,q))
print('Rozwiązanie dokładne')
print(exact_sol)
print("Algorytm 1: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_1,eta_1))
print("Algorytm 2: Błąd bezwzględny %10.2e, Błąd względny %10.2e"%(epsilon_2,eta_2))
```

    Algorytm 1
    8.999999977277184e-06
    Algorytm 2
    9e-06
    Rozwiązanie dokładne
    9e-06
    Algorytm 1: Błąd bezwzględny   2.27e-14, Błąd względny   2.52e-09
    Algorytm 2: Błąd bezwzględny   0.00e+00, Błąd względny   0.00e+00

# Metody Numeryczne

## Ocena algorytmów numerycznych

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Notacja O duże
- Mówimy, że dla wielkości zależnej od parametru np. $F(n)$ zachodzi
$$
F(n)=O(G(n))
$$
jeżeli istnieje taka stała $C$, że przy $n$ zmierzającym do nieskończoności (odpowiednio dużym), mamy
$$F(n)≤C G(n)$$
- Jeżeli interesuje nas $O(c)$, gdzie $c$ jest stałą, zależność ta ma zachodzić niezależnie od wielkości parametru.
- Mówimy potocznie, gdy błąd jest równy $O(n^2)$, że błąd jest rzędu $n^2$

## Ocena algorytmu
- Naszym celem jest obliczenie pewnej wielkości $f(x)$, zależnej od danych wejściowych $x$
- W przypadku obliczeń komputerowych zawsze mamy do czynienia z obliczaniem przybliżonym stąd algorytm obliczania $f(x)$ będziemy oznaczać jako $f^*(x)$
- Dane w komputerze również są reprezentowane w sposób zaokrąglony, więc będziemy je oznaczać jako $x^*$

## Uwarunkowanie problemu

- Mówimy, że problem $f(x)$ jest dobrze uwarunkowany, jeżeli mała zmiana $x$ powoduje małą zmianę w $f(x)$
- Problem jest źle uwarunkowany, jeżeli mała zmiana $x$ powoduje dużą zmianę w $f(x)$
- Miarą uwarunkowania jest stała $\kappa$ (kappa), która (nieformalnie) określa największy iloraz zaburzeń $f(x)$  wywołanych przez najmniejsze zaburzenia $x$.
- Stałą $\kappa$ można wyliczyć tylko w niektórych probemach

## Dokładność algorytmu
- Algorytm jest dokładny, jeżeli
$$
\frac{\Vert f^*(x)-f(x) \Vert}{\Vert f(x)\Vert}=O(\varepsilon_m)
$$
- Zagwarantowanie, że algorytm jest dokładny wg tej definicji jest niezwykle trudne, zwłaszcza dla źle uwarunkowanych problemów

## Stabilność algorytmu

Mówimy, że algorytm jest stabilny, gdy dla każdego $x$, zachodzi
$$
\frac{\Vert f^*(x)-f(x^*) \Vert}{\Vert f(x^*)\Vert}=O(\varepsilon_m)
$$
dla takich $x^*$, że
$$\frac{\Vert x-x^* \Vert}{\Vert x\Vert}=O(\varepsilon_m)$$
Innymi słowy
**Stabilny algorytm daje prawie dobrą odpowiedź na prawie dobre pytanie**

## Stabilność wsteczna algorytmu
Algorytm jest stabilny wstecznie, jeżeli dla każdego $x$, zachodzi
$$f^*(x)=f(x^*)$$
dla takich $x^*$, że
$$\frac{\Vert x-x^* \Vert}{\Vert x\Vert}=O(\varepsilon_m)$$
Innymi słowy
**Stabilny wstecznie algorytm daje prawidłową odpowiedź na prawie dobre pytanie**

## Dokładność algorytmów stabilnych wstecznie przy złym uwarunkowaniu
Jeśli algorytm jest stabilny wstecznie, to jego błąd względny pogarsza się proporcjonalnie do stałej uwarunkowania tj. $O(\kappa\varepsilon_m)$

# Metody Numeryczne

## Problemy techniczne

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

Metody Numeryczne 2019/Lecture 1 (errors and stuff)/Lecture  1.ipynb


```python
%matplotlib inline
```

```python
#environment setup with watermark
%load_ext watermark
%watermark -a 'Gopala KR' -u -d -v -p watermark,numpy,pandas,matplotlib,nltk,sklearn,tensorflow,theano,mxnet,chainer
```

    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

    Gopala KR
    last updated: 2018-01-30

    CPython 3.6.3
    IPython 6.2.1

    watermark 1.6.0
    numpy 1.13.1
    pandas 0.20.3
    matplotlib 2.0.2
    nltk 3.2.5
    sklearn 0.19.0
    tensorflow 1.3.0
    theano 1.0.1
    mxnet 1.0.0
    chainer 3.3.0

# The Johnson-Lindenstrauss bound for embedding with random projections

The `Johnson-Lindenstrauss lemma`_ states that any high dimensional
dataset can be randomly projected into a lower dimensional Euclidean
space while controlling the distortion in the pairwise distances.

Theoretical bounds
==================

The distortion introduced by a random projection `p` is asserted by
the fact that `p` is defining an eps-embedding with good probability
as defined by:

\begin{align}(1 - eps) \|u - v\|^2 < \|p(u) - p(v)\|^2 < (1 + eps) \|u - v\|^2\end{align}

Where u and v are any rows taken from a dataset of shape [n_samples,
n_features] and p is a projection by a random Gaussian N(0, 1) matrix
with shape [n_components, n_features] (or a sparse Achlioptas matrix).

The minimum number of components to guarantees the eps-embedding is
given by:

\begin{align}n\_components >= 4 log(n\_samples) / (eps^2 / 2 - eps^3 / 3)\end{align}

The first plot shows that with an increasing number of samples ``n_samples``,
the minimal number of dimensions ``n_components`` increased logarithmically
in order to guarantee an ``eps``-embedding.

The second plot shows that an increase of the admissible
distortion ``eps`` allows to reduce drastically the minimal number of
dimensions ``n_components`` for a given number of samples ``n_samples``

Empirical validation
====================

We validate the above bounds on the digits dataset or on the 20 newsgroups
text document (TF-IDF word frequencies) dataset:

- for the digits dataset, some 8x8 gray level pixels data for 500
  handwritten digits pictures are randomly projected to spaces for various
  larger number of dimensions ``n_components``.

- for the 20 newsgroups dataset some 500 documents with 100k
  features in total are projected using a sparse random matrix to smaller
  euclidean spaces with various values for the target number of dimensions
  ``n_components``.

The default dataset is the digits dataset. To run the example on the twenty
newsgroups dataset, pass the --twenty-newsgroups command line argument to this
script.

For each value of ``n_components``, we plot:

- 2D distribution of sample pairs with pairwise distances in original
  and projected spaces as x and y axis respectively.

- 1D histogram of the ratio of those distances (projected / original).

We can see that for low values of ``n_components`` the distribution is wide
with many distorted pairs and a skewed distribution (due to the hard
limit of zero ratio on the left as distances are always positives)
while for larger values of n_components the distortion is controlled
and the distances are well preserved by the random projection.

Remarks
=======

According to the JL lemma, projecting 500 samples without too much distortion
will require at least several thousands dimensions, irrespective of the
number of features of the original dataset.

Hence using random projections on the digits dataset which only has 64 features
in the input space does not make sense: it does not allow for dimensionality
reduction in this case.

On the twenty newsgroups on the other hand the dimensionality can be decreased
from 56436 down to 10000 while reasonably preserving pairwise distances.

```python
print(__doc__)

import sys
from time import time
import numpy as np
import matplotlib.pyplot as plt
from sklearn.random_projection import johnson_lindenstrauss_min_dim
from sklearn.random_projection import SparseRandomProjection
from sklearn.datasets import fetch_20newsgroups_vectorized
from sklearn.datasets import load_digits
from sklearn.metrics.pairwise import euclidean_distances

# Part 1: plot the theoretical dependency between n_components_min and
# n_samples

# range of admissible distortions
eps_range = np.linspace(0.1, 0.99, 5)
colors = plt.cm.Blues(np.linspace(0.3, 1.0, len(eps_range)))

# range of number of samples (observation) to embed
n_samples_range = np.logspace(1, 9, 9)

plt.figure()
for eps, color in zip(eps_range, colors):
    min_n_components = johnson_lindenstrauss_min_dim(n_samples_range, eps=eps)
    plt.loglog(n_samples_range, min_n_components, color=color)

plt.legend(["eps = %0.1f" % eps for eps in eps_range], loc="lower right")
plt.xlabel("Number of observations to eps-embed")
plt.ylabel("Minimum number of dimensions")
plt.title("Johnson-Lindenstrauss bounds:\nn_samples vs n_components")

# range of admissible distortions
eps_range = np.linspace(0.01, 0.99, 100)

# range of number of samples (observation) to embed
n_samples_range = np.logspace(2, 6, 5)
colors = plt.cm.Blues(np.linspace(0.3, 1.0, len(n_samples_range)))

plt.figure()
for n_samples, color in zip(n_samples_range, colors):
    min_n_components = johnson_lindenstrauss_min_dim(n_samples, eps=eps_range)
    plt.semilogy(eps_range, min_n_components, color=color)

plt.legend(["n_samples = %d" % n for n in n_samples_range], loc="upper right")
plt.xlabel("Distortion eps")
plt.ylabel("Minimum number of dimensions")
plt.title("Johnson-Lindenstrauss bounds:\nn_components vs eps")

# Part 2: perform sparse random projection of some digits images which are
# quite low dimensional and dense or documents of the 20 newsgroups dataset
# which is both high dimensional and sparse

if '--twenty-newsgroups' in sys.argv:
    # Need an internet connection hence not enabled by default
    data = fetch_20newsgroups_vectorized().data[:500]
else:
    data = load_digits().data[:500]

n_samples, n_features = data.shape
print("Embedding %d samples with dim %d using various random projections"
      % (n_samples, n_features))

n_components_range = np.array([300, 1000, 10000])
dists = euclidean_distances(data, squared=True).ravel()

# select only non-identical samples pairs
nonzero = dists != 0
dists = dists[nonzero]

for n_components in n_components_range:
    t0 = time()
    rp = SparseRandomProjection(n_components=n_components)
    projected_data = rp.fit_transform(data)
    print("Projected %d samples from %d to %d in %0.3fs"
          % (n_samples, n_features, n_components, time() - t0))
    if hasattr(rp, 'components_'):
        n_bytes = rp.components_.data.nbytes
        n_bytes += rp.components_.indices.nbytes
        print("Random matrix with size: %0.3fMB" % (n_bytes / 1e6))

    projected_dists = euclidean_distances(
        projected_data, squared=True).ravel()[nonzero]

    plt.figure()
    plt.hexbin(dists, projected_dists, gridsize=100, cmap=plt.cm.PuBu)
    plt.xlabel("Pairwise squared distances in original space")
    plt.ylabel("Pairwise squared distances in projected space")
    plt.title("Pairwise distances distribution for n_components=%d" %
              n_components)
    cb = plt.colorbar()
    cb.set_label('Sample pairs counts')

    rates = projected_dists / dists
    print("Mean distances rate: %0.2f (%0.2f)"
          % (np.mean(rates), np.std(rates)))

    plt.figure()
    plt.hist(rates, bins=50, normed=True, range=(0., 2.), edgecolor='k')
    plt.xlabel("Squared distances rate: projected / original")
    plt.ylabel("Distribution of samples pairs")
    plt.title("Histogram of pairwise distance rates for n_components=%d" %
              n_components)

    # TODO: compute the expected value of eps and add them to the previous plot
    # as vertical lines / region

plt.show()
```

```python

```

```python
test complete; Gopal
```


# Machine learning compilation of quantum circuits
> Optimal compiling of unitaries reaching the theoretical lower bound

- toc: true
- badges: true
- comments: true
- categories: [machine learning, compilation, qiskit, paper review]
- image: images/grovercirc.png

# Introduction

I am going to review a recent [preprint](http://arxiv.org/abs/2106.05649) by Liam Madden and
Andrea Simonetto that uses techniques from machine learning to tackle the problem of quantum circuits compilation. I find the approach suggested in the paper very interesting and the preliminary results quite promising.

## What is compilation?
> Note that a variety of terms are floating around the literature and used more or less interchangibly. Among those are **synthesis**, **compilation**, **transpilation** and **decomposition** of quantum circuits. I will not make a distinction and try to stick to **compilation**.

But first things first, what is a compilation of a quantum circuit? The best motivation and illustration for the problem is the following. Say you need to run a textbook quantum circuit on a real hardware. The real hardware usually allows only for a few basic one and two qubit gates. In contrast, your typical textbook quantum circuit may feature (1) complex many-qubit gates, for example multi-controlled gates and (2) one and two qubit gates which are not supported by the hardware. As a simple example take this 3-qubit Grover's circuit (from [qiskit textbook](https://qiskit.org/textbook/ch-algorithms/grover.html)):

```python
# collapse
#initialization
import matplotlib.pyplot as plt
import numpy as np

# importing Qiskit
from qiskit import IBMQ, Aer, assemble, transpile
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister
from qiskit.providers.ibmq import least_busy

# import basic plot tools
from qiskit.visualization import plot_histogram

def initialize_s(qc, qubits):
    """Apply a H-gate to 'qubits' in qc"""
    for q in qubits:
        qc.h(q)
    return qc

def diffuser(nqubits):
    qc = QuantumCircuit(nqubits)
    # Apply transformation |s> -> |00..0> (H-gates)
    for qubit in range(nqubits):
        qc.h(qubit)
    # Apply transformation |00..0> -> |11..1> (X-gates)
    for qubit in range(nqubits):
        qc.x(qubit)
    # Do multi-controlled-Z gate
    qc.h(nqubits-1)
    qc.mct(list(range(nqubits-1)), nqubits-1)  # multi-controlled-toffoli
    qc.h(nqubits-1)
    # Apply transformation |11..1> -> |00..0>
    for qubit in range(nqubits):
        qc.x(qubit)
    # Apply transformation |00..0> -> |s>
    for qubit in range(nqubits):
        qc.h(qubit)
    # We will return the diffuser as a gate
    U_s = qc.to_gate()
    U_s.name = "U$_s$"
    return U_s

qc = QuantumCircuit(3)
qc.cz(0, 2)
qc.cz(1, 2)
oracle_ex3 = qc.to_gate()
oracle_ex3.name = "U$_\omega$"

n = 3
grover_circuit = QuantumCircuit(n)
grover_circuit = initialize_s(grover_circuit, [0,1,2])
grover_circuit.append(oracle_ex3, [0,1,2])
grover_circuit.append(diffuser(n), [0,1,2])
grover_circuit = grover_circuit.decompose()
grover_circuit.draw(output='mpl')
```

The three qubit gates like Toffoli are not generally available on a hardware and one and two qubit gates my be different from those in the textbook algorithm. For example ion quantum computers are good with [Mølmer–Sørensen gates](https://en.wikipedia.org/wiki/M%C3%B8lmer%E2%80%93S%C3%B8rensen_gate) and may need several native one qubit gates to implement the Hadamard gate.

Additional important problem is to take into account qubit connectivity. Usually textbook algorithms assume full connectivity, meaning that two-qubit gates can act on any pair of qubits. On most hardware platforms however a qubit can only interact with its neighbors. Assuming that one and two qubits gates available on the hardware can implement a SWAP gate between adjacent qubits, to solve the connectivity problem one can insert as many SWAPs as necessary to connect topologically disjoint qubits. Using SWAPs however leads to a huge overhead in the number of total gates in the compiled circuit, and it is of much importance use them as economically as possible. In fact, the problem of optimal SWAPping alone in generic situation is [NP-complete](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=on+the+complexity+of+quantum+circuit+compilation&btnG=).

## Simplified problem
When compiling a quantum circuit one has to decide which resulting circuits are considered to be efficient. Ideally, one should optimize for the total fidelity of the circuit. Let us imagine running the algorithm on a real device. Probably my theorist's image of a real device is still way too platonic, but I will try my best. Many details need to be taken into account. For example, gates acting on different qubits or pairs of qubits may have different fidelities. Decoherence of qubits with time can make circuits where many operations can be executed in parallel more favorable. Cross-talk (unwanted interactions) between neighboring qubits may lead to exotic patterns for optimal circuits. A simple proxy for the resulting fidelity that is often adopted is the number of two-qubit gates (which are generically much less accurate than a single-qubit gates). So the problem that is often studied, and that is addressed in the preprint we are going to discuss, is the problem of optimal compilation into a gate set consisting of arbitrary single-qubit gates and CNOTs, the only two qubits gate. The compiled circuit must

1. Respect hardware connectivity.
1. Have as few CNOTs as possible.
1. Exceed a given fidelity threshold.

Last item here means that we also allow for an approximate compilation. By increasing the number of CNOTs one can always achieve an exact compilation, but since in reality each additional CNOT comes with its own fidelity cost this might not be a good trade-off. Note also that a specific choice for two-qubit gate is made, a CNOT gate. Any two-qubit gate can be decomposed into at most 3 CNOTs [see e.g. here](https://arxiv.org/pdf/quant-ph/0308006.pdf), so in terms of computational complexity this is of course inconsequential. However in the following discussion we will care a lot about constant factors and may wish to revisit this choice at the end.

## Existing results

Since finding the exact optimal solution to the compilation problem is intractable, as with many things in life one needs to resort to heuristic methods. A combination of many heuristic methods, in fact. As an example one can check out the [transpilation workflow](https://qiskit.org/documentation/apidoc/transpiler.html) in `qiskit`. Among others, there is a step that compiles >2 qubit gates into one and two qubit gates; the one that tries to find a good initial placement of the logical qubits onto physical hardware; the one that 'routes' the desired circuit to match a given topology being as greedy on SWAPs as possible. Each of these steps can use several different heuristic optimization algorithms, which are continuously refined and extended (for example this [recent preprint](https://arxiv.org/abs/2106.06446) improves on the default rounting procedure in `qiskit`). In my opinion it would be waay better to have one unified heuristic for all steps of the process, especially taking into account that they are not completely independent. Although this might be too much to ask for, some advances are definitely possible and machine learning tools might prove very useful. The paper we are going to discuss is an excellent demonstration.

## Theoretical lower bound and quantum Shannon decomposition
There is a couple of very nice theoretical results about the compilation problem that I need to mention. But first, let us agree that we will compile unitaries, not circuits.  What is the difference? Of course, any quantum circuit (without measurements and neglecting losses) corresponds to a unitary matrix. However, to compute that unitary matrix for a large quantum circuit explicitly is generally an intractable problem, precisely for the same reasons that quantum computation is assumed to be more powerful than classical. Still, taking as the input a unitary matrix (which is in general hard to compute from the circuit) is very useful both theoretically and practically. I will discuss pros and cons of this approach later on.

OK, now the fun fact. Generically, one needs at least this many CNOTs

\begin{align}
 L:=\# \text{CNOTs} \geq \frac14\left(4^n-3n-1\right) \label{TLB}
\end{align}

to exactly compile an $n$-qubit unitary. 'Generically' means that the set of $n$-qubit unitaries that can be compiled exactly with smaller amount of CNOTs has measure zero. Keep in mind though, that there are important unitaries in this class like multi-controlled gates or qubit permutations. We will discuss compilation of some gates from the 'measure-zero' later on.

The authors of the preprint (I hope you and me still remember that there is some actual results to discuss, not just my overly long introduction to read) refer to \eqref{TLB} as the theoretical lower bound or TLB for short. The proof of this [fact](https://dl.acm.org/doi/10.5555/968879.969163) is actually rather simple and I will sketch it. A general $d\times d$ unitary has $d^2$ real parameters. For $n$ qubits $d=2^n$. Single one-qubit gate has 3 real parameters. Any sequence of one-qubit gates applied to the same qubit can be reduced to a single one-qubit gate and hence can have no more than 3 parameters. That means, that without CNOTs we can only have 3n parameters in our circuit, 3 for each one-qubit gate. This is definitely not enough to describe an arbitrary unitary on $n$ qubits which has $d^2=4^n$ parameters.

Now, adding a single CNOT allows to insert two more 1-qubit unitaries after it, like that

```python
#collapse
from qiskit.circuit import Parameter

a1, a2, a3 = [Parameter(a) for a in ['a1', 'a2', 'a3']]
b1, b2, b3 = [Parameter(b) for b in ['b1', 'b2', 'b3']]

qc = QuantumCircuit(2)
qc.cx(0, 1)
qc.u(a1, a2, a3, 0)
qc.u(b1, b2, b3, 1)

qc.draw(output='mpl')
```

At the first glance this allows to add 6 more parameters. However, each single-qubit unitary can be represented via the Euler angles as a product of only $R_z$ and $R_x$ rotations either as $U=R_z R_x R_z$ or $U=R_x R_y R_z$ (I do not specify angles). Now, CNOT can be represented as $CNOT=|0\rangle\langle 0|\otimes I+|1\rangle\langle 1|\otimes X$. It follows that $R_z$ commutes with the control of CNOT and $R_x$ commutes with the target of CNOT, hence they can be dragged to the left and joined with preceding one-qubit gates. So in fact each new CNOT gate allows to add only 4 real parameters:

```python
#collapse
a1, a2 = [Parameter(a) for a in ['a1', 'a2']]
b1, b2 = [Parameter(b) for b in ['b1', 'b2']]

qc = QuantumCircuit(2)
qc.cx(0, 1)
qc.rx(a1, 0)
qc.rz(a2, 0)
qc.rz(b1, 1)
qc.rx(b2, 1)

qc.draw(output='mpl')
```

 That's it, there are no more caveats. Thus, the total number of parameters we can get with $L$ CNOTs is $3n+4L$ and we need to describe a $d\times d$ unitary which has $4^n$ parameters. In fact, the global phase of the unitary is irrelevant so we only need $3n+4L \geq 4^n-1$. Solving for $L$ gives the TLB \eqref{TLB}. That's pretty cool, isn't it?

Now there is an algorithm, called *quantum Shannon decomposition* (see [ref](https://arxiv.org/abs/quant-ph/0406176)), which gives an exact compilation of any unitary with the number of CNOTs twice as much as the TLB requires. In complexity-theoretic terms an overall factor of two is of course inessential, but for current NISQ devices we want to get as efficient as possible. Moreover, to my understanding the quantum Shannon decomposition is not easily extendable to restricted topology while inefficient generalizations lead to a much bigger overhead (roughly an order of magnitude).

# What's in the preprint?
## Templates
I've already wrote an introduction way longer than intended so from now on I will try to be brief and to the point. The authors of the preprint propose two templates inspired by the quantum Shannon decomposition. The building block for each template is a 'CNOT unit'

```python
#collapse
a1, a2 = [Parameter(a) for a in ['a1', 'a2']]
b1, b2 = [Parameter(b) for b in ['b1', 'b2']]

qc = QuantumCircuit(2)
qc.cx(0, 1)
qc.ry(a1, 0)
qc.rz(a2, 0)
qc.ry(b1, 1)
qc.rx(b2, 1)

qc.draw(output='mpl')
```

First template is called **sequ** in the paper and is obtained as follows. There are $n(n-1)/2$ different CNOTs on $n$-qubit gates. We enumerate them somehow and simply stack sequentially. Here is a 3-qubut example with two layers (I use `qiskit` gates `cz` instead of our 'CNOT units' for the ease of graphical representation)

```python
#collapse
qc = QuantumCircuit(3)
for _ in range(2):
    qc.cz(0, 1)
    qc.cz(0, 2)
    qc.cz(1, 2)
    qc.barrier()
qc.draw(output='mpl')
```

The second template is called **spin** and for 4 qubits looks as follows

```python
#collapse
qc = QuantumCircuit(4)
for _ in range(2):
    qc.cz(0, 1)
    qc.cz(1, 2)
    qc.cz(2, 3)
    qc.barrier()
qc.draw(output='mpl')
```

I'm sure you get the idea. That's it! The templates fix the pattern of CNOTs while angles of single-qubit gates are adjustable parameters which are collectively denoted by $\theta$.

The idea now is simple. Try to optimize these parameters to achieve the highest possible fidelity for a given target unitary to compile. I am not at all an expert on the optimization methods, so I might miss many subtleties, but on the surface the problem looks rather straightforward. You can choose your favorite flavor of the gradient descent and hope for convergence. The problem appears to be non-convex but the gradient descent seems to work well in practice. One technical point that I do not fully understand is that the authors choose to work with fidelity defined by the Frobenius norm $||U-V||_F^2$ which is sensitive to the global phase of each unitary. To my understanding they often find that local minima of this fidelity coincides with the global minimum up to a global phase. OK, so in the rest of the post I refer to the 'gradient descent' as the magic numerical method which does good job of finding physically sound minimums.

## Results
### Compiling random unitaries
OK, finally, for the surprising results. The authors find experimentally that both **sequ** and **spin** perform surprisingly well on random unitaries always coming very close to the TLB \eqref{TLB} with good fidelity. More precisely, the tests proceed as follows. First, one generates a random unitary. Next, for each number $L$ of CNOTs below the TLB one runs the gradient descent to see how much fidelity can be achieved with this amount of CNOTs. Finally, one plots the fidelity as a function of $L$. Impressively, on the sample of hundred unitaries the fidelity always approaches 100% when the number of CNOTs reaches the TLB. For the $n=3$ qubits TLB is $L=14$, for $n=5$ $L=252$ (these are the two cases studied). So, in all cases studied, the gradient descent lead by the provided templates seems to always find the optimal compilation circuit! Recall that this is two times better than quantum Shannon decomposition. Please see the original paper for nice plots that I do not reproduce here.

### Compiling on restricted topology
These tests were performed on the fully connected circuits. The next remarkable discovery is that restricting the connectivity does not to seem to harm the performance of the compilation! More precisely, the authors considered two restricted topologies in the paper, 'star' where all qubits are connected to single central one and 'line' where well, they are connected by links on a line. The **spin** template can not be applied to star topology, but it can be applied to line topology. The **sequ** template can be generalized to any topology by simply omitting CNOTs that are not allowed. Again, as examining a hundred of random unitaries on $n=3$ and $n=5$ qubits shows, the fidelity nearing 100% can be achieved right at the TLB in all cases, which hints that topology restriction may not be a problem in this approach at all! To appreciate the achievement, imagine decomposing each unitary via the quantum Shannon decomposition and then routing on restricted topology with swarms of SWAPs, a terrifying picture indeed. It would be interesting to compare the results against the performance of `qiskit` transpiler which is unfortunately not done in the paper to my understanding.

### Compiling specific 'measure zero' gates
Some important multi-qubit gates fall into the 'measure zero' set which can be compiled with a smaller amount of CNOTs than is implied by the TLB \eqref{TLB}. For example, 4-qubit Toffoli gate can be compiled with 14 CNOTs while the TLB requires 61 gates. Numerical tests show that the plain version of the algorithm presented above does not generically obtain the optimal compilation for special gates. However, with some tweaking and increasing the amount of attempts the authors were able to find optimal decompositions for a number of known gates such as 3- and 4-qubit Toffoli, 3-qubit Fredkin and 1-bit full adder on 4 qubits. The tweaking included randomly changing the orientation of some CNOTs (note that in both **sequ** and **spin** the control qubit is always at the top) and running many optimization cycles with random initial conditions. The best performing method appeared to be **sequ** with random flips of CNOTs. The whole strategy might look a bit fishy, but I would argue that it is not. My argument is simple: you only need to find a good compilation of the 4-qubit Toffoli *once*. After that you pat yourself on the back and use the result in all your algorithms. So it does not really matter how hard it was to find the compilation as long as you did not forget to write it down.

### Compressing the quantum Shannon decomposition
Finally, as a new twist on the plot the authors propose a method to compress the standard quantum Shannon decomposition (which is twice the TLB, remember?). The idea seems simple and works surprisingly well. The algorithm works as follows.
1. Compile a unitary exactly using the quantum Shannon decomposition.
1. Promote parameters in single-qubit gates variables (they have fixed values in quantum Shannon decomposition).
2. Add [LASSO](https://en.wikipedia.org/wiki/Lasso_(statistics)-type regularization term, which forces one-qubit gates to have small parameters, ideally zero (which makes the corresponding gates into identities).
3. Run a gradient descent on the regularized cost function (fidelity+LASSO term). Some one-qubit gates will become identity after that (one might need to tune the regularization parameter here).
4. After eliminating identity one-qubit gates one can end up in the situation where there is a bunch of CNOTs with no single-qubit gates in between. There are efficient algorithms for reducing the amount of CNOTs in this case.
5. Recall that the fidelity was compromised by adding regularization terms. Run the gradient descent once more, this time without regularization, to squeeze out these last percents of fidelity.

From the description of this algorithm it does not appear obvious that the required cancellations (elimination of single-qubit gates and cancellations in resulting CNOT clusters) is bound to happen, but the experimental tests show that they do. Again, from a bunch of random unitaries it seems that the $\times 2$ reduction to the TLB is almost sure to happen! Please see the preprint for plots.

## Weak spots
Although I find results of the paper largely impressive, a couple of weak spots deserve a mention.
### Limited scope of experiments
The numerical experiments were only carried out for $n=3$ and $n=5$ qubits which of course is not much. To see if the method keeps working as the number of qubits is scaled is sure very important. There may be two promblems. First, the templates can fail to be expressive enough for larger circuits. The authors hope to attack this problem from the theoretical side and show that the templates do fill the space of unitaries. Well, best of luck with that! Another potential problem is that although the templates work fine for higher $n$, the learning part might become way more challenging. Well, I guess we should wait and see.
### Unitary as the input
As I discussed somewhere way above, for a realistic quantum computation we can not know the unitary matrix that we need to compile. If we did, there would no need in the quantum computer in the first place. I can make two objects here. First, we are still in the NISQ era and pushing the existing quantum computers to their edge is a very important task. Even if an algorithm can be simulated classically, running it on a real device might be invaluable. Second, even quantum circuits on 1000 qubits do not usually feature  100-qubit unitaries. So it could be possible to separate a realistic quantum circuit into pieces, each containing only a few qubits, and compile them separately.

# Final remarks
To me, the algorithms presented in the preprint seem to be refreshingly efficient and universal. At some level it appears to be irrelevant which exact template do we use. Near the theoretical lower bound they all perform similarly well, even on restricted topology. This might be a justification for choosing CNOT as the two-qubit gate, as this probably does not matter in the end! I'm really cheering for a universal algorithm like that to win the compilation challenge over a complicated web of isolated heuristics, which are currently state of the art.

_notebooks/2021-07-22-Machine learning compilation of quantum circuits.ipynb


## Introduction to Python 2016

University of Melbourne

School of Earth Sciences

Louis Moresi

[louis.moresi@unimelb.edu.au](mailto:louis.moresi@unimelb.edu.au)

[www.moresi.info](http://www.moresi.info)

### Quick links

 - [Browse](/notebooks/Content/Notebooks/) all files that make up the course, start a new notebook or access the terminal.
 - Run the [Mapping Notebooks](/notebooks/Content/Notebooks/Mapping)
 - If you are running in the Docker environment, you can drill through to [local files](/notebooks/Content/Notebooks/external) if they have been correctly mounted as a volume within the container.

### What is this ?

We will be working in the iPython / Jupyter notebook system. I like these because they are a form of literate programming in which we can mix textbook instruction and explanations with code that can also be run and edited.
The text and mathematics in the notebooks requires a little preliminary learning.

The notebook system also includes a [file browser](/tree) which also allows you to add your own notebook, add a text file or start a terminal on the machine running this notebook.

### Markdown

You can document your iPython notebooks by making some cells into **Markdown** cells. Markdown is a way of formatting text that is supposed to be almost as readable un-rendered as when it is tidied up. You might argue that it looks equally bad either way, but that's tough because the notebooks use it and that's how I want you to produce nice looking output to hand in as an assignment !

If you look at the **Markdown** cells as source code (by double-clicking on them) you will see how the raw text looks. To get back to the pretty version of the text, hit shift-enter.

### Maths

In a browser, you can render beautiful equations using a javascript tool called **Mathjax** which is build into the iPython notebooks.

You can build in symbols to your text such as $\pi$ and $\epsilon$ if you use the \$ signs to indicate where your equations begin and end, and you know enough $\LaTeX$ [try it here !](http://www.codecogs.com/latex/eqneditor.php) to get by.

Equations in 'display' mode are written like this (again look at the source for this cell to see what is used)

\\[ e^{i\pi} + 1 = 0 \\]

or even like this

\begin{equation}
%%
    \nabla^4 \psi = \frac{\partial T}{\partial x}
%%
\end{equation}

Go back to the rendered form of the cell by 'running' it.

### Links

[Markdown Website](http://daringfireball.net/projects/markdown/)

[Mathjax Website](http://docs.mathjax.org)

[Jupyter Notebooks](http://www.jupyter.org)

```python

```


# Introduction to Jupyter

Jupyter is a web based development environment.

The three moons in the symbol are _Julia_, _R_ and _Python_

In these examples, we will use only python2 or 3 (depending on yout taste).
Here's some shortcut commands for the Notebook:

```bash
    Alt + Return/Enter:   Evaluates current cell and creates a new cell
    Shift + Return/Enter: Evaluates current cell ang goes to next cell
    Ctrl + Return/Enter:  Evaluates current cell stays in current cell
    Tab:                  Gives you auto completion
    Shift + Tab:          Gives you Help
    Enter:                Goes into Edit mode (notice the color change in the cell from blue to green)
    Esc:                  Goes into Command mode (notice the color change in the cell from green to blue)
    Command Mode Actions:
        a:  Creates Cell Above
        b:  Creates Cell below
        s:  Saves notebook
        y:  Changes Cell type to CODE
        m:  Changes Cell type to Markdown
        d:  Deletes Cell
        x:  Cuts Cell
        And many more... go to Edit -> Keyboard Shortcuts
        Shift + Ctrl + P: Command Palette
```

Each **cell** accepts a series of commands that are run at the time of evaluation.

```python
a=1
b=2
a+b
```

You can embbed plots:

```python
import matplotlib.pyplot as plt
import numpy as np
```

```python
x = np.arange(-5,5,0.1)
y = np.sin(x)
```

```python
plt.plot(x,y,label='sin')
plt.legend()
plt.show()
```

Or you can change the way figures are displayed and retain some interactivity:

```python
%matplotlib notebook
#inline
import matplotlib.pyplot as plt
import numpy as np
x = np.arange(-5,5,0.1)
y = np.sin(x)
plt.plot(x,y,label='sin')
plt.show()
#remember to close figure
```

    <IPython.core.display.Javascript object>

You can set up packages with your functions if you are going to use them repetitively. If you want to share them later, documment them thouroughly.

```python
%matplotlib inline
import matplotlib.pyplot as plt
from my_utils.utils import testf

x, y = testf()
plt.plot(x,y)
plt.show()
```

There are many formatting options:

# Heading 1
# Heading 2
## Heading 2.1
### Heading 2.1.1
#### Heading 2.1.1.1 all the way to 6 (1-6 while in Command mode)

1. Make a list
2. With the entries
    3. Organized
    4. As you
        5. Want
        6. Oops

* Well
* Numbers
    * Are not exclusive
    * To the lists

Go to [Jupyter Notebook](http://jupyter.org/) for more info!

$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$

\begin{equation}
c = a \times b
\end{equation}

```python
print "Hello World"
```

| This | is   |    | Try it out! |
|------|------|----|-------------|
|   a  |table |    |   Huzzah    |

Adds an image from the local notebook directory.

Thanks to F. da Silva
<video width="800" height="500" controls src="Files/ims_V-band_50GHz.m4v" />

```python
from IPython.display import Image
#from IPython.core.display import HTML
Image(url= "http://www.ipp.mpg.de/36468/zoom-1369313316.jpg")
```

```python

```

```python
from IPython.display import YouTubeVideo
YouTubeVideo("QCK51vqWunU")
```

And so many __many__ other _functionalities_, like *__Magics__*:

```python
%lsmagic
```

```python
ll #or probably %cd for windows users...
```

And you can __Cell Magics__, that allow to virtually run wahtever you want (next example for unix users only):

```bash
%%bash

grep *.ipynb -e "pandas "
```

###### You can get user input
Notice the cell keeps running until you provide input

```python
name = raw_input("Username: ")
print "Hello, " + name
```

You can even produce shareable GUIs, thanks to the widgets library:

```python
%matplotlib inline
import ipywidgets as wd
import numpy as np
import matplotlib.pyplot as plt

t = np.arange(0.1, 1.0, 0.01)

def pltsin(f, A, offset):
    plt.plot(t, A*np.sin(2.0*np.pi*f*t) + offset)
    plt.ylim(-5,5)
    plt.show()

wd.interact(pltsin, f=(1, 10, 0.1), A=(0.0, 5.0, 0.1), offset=(-5,5,0.1))
```

    <function __main__.pltsin>

## How to share notebooks

```python
import webbrowser
#Either in Github or hosting an html
webbrowser.open('https://nbviewer.jupyter.org/')
#Running a private server
```

01 - Introduction to the Jupyter Notebook.ipynb


```python
%matplotlib inline
```

DCGAN Tutorial
==============

**Author**: `Nathan Inkawhich <https://github.com/inkawhich>`__

Introduction
------------

This tutorial will give an introduction to DCGANs through an example. We
will train a generative adversarial network (GAN) to generate new
celebrities after showing it pictures of many real celebrities. Most of
the code here is from the dcgan implementation in
`pytorch/examples <https://github.com/pytorch/examples>`__, and this
document will give a thorough explanation of the implementation and shed
light on how and why this model works. But don’t worry, no prior
knowledge of GANs is required, but it may require a first-timer to spend
some time reasoning about what is actually happening under the hood.
Also, for the sake of time it will help to have a GPU, or two. Lets
start from the beginning.

Generative Adversarial Networks
-------------------------------

What is a GAN?
~~~~~~~~~~~~~~

GANs are a framework for teaching a DL model to capture the training
data’s distribution so we can generate new data from that same
distribution. GANs were invented by Ian Goodfellow in 2014 and first
described in the paper `Generative Adversarial
Nets <https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf>`__.
They are made of two distinct models, a *generator* and a
*discriminator*. The job of the generator is to spawn ‘fake’ images that
look like the training images. The job of the discriminator is to look
at an image and output whether or not it is a real training image or a
fake image from the generator. During training, the generator is
constantly trying to outsmart the discriminator by generating better and
better fakes, while the discriminator is working to become a better
detective and correctly classify the real and fake images. The
equilibrium of this game is when the generator is generating perfect
fakes that look as if they came directly from the training data, and the
discriminator is left to always guess at 50% confidence that the
generator output is real or fake.

Now, lets define some notation to be used throughout tutorial starting
with the discriminator. Let $x$ be data representing an image.
$D(x)$ is the discriminator network which outputs the (scalar)
probability that $x$ came from training data rather than the
generator. Here, since we are dealing with images, the input to
$D(x)$ is an image of CHW size 3x64x64. Intuitively, $D(x)$
should be HIGH when $x$ comes from training data and LOW when
$x$ comes from the generator. $D(x)$ can also be thought of
as a traditional binary classifier.

For the generator’s notation, let $z$ be a latent space vector
sampled from a standard normal distribution. $G(z)$ represents the
generator function which maps the latent vector $z$ to data-space.
The goal of $G$ is to estimate the distribution that the training
data comes from ($p_{data}$) so it can generate fake samples from
that estimated distribution ($p_g$).

So, $D(G(z))$ is the probability (scalar) that the output of the
generator $G$ is a real image. As described in `Goodfellow’s
paper <https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf>`__,
$D$ and $G$ play a minimax game in which $D$ tries to
maximize the probability it correctly classifies reals and fakes
($logD(x)$), and $G$ tries to minimize the probability that
$D$ will predict its outputs are fake ($log(1-D(G(z)))$).
From the paper, the GAN loss function is

\begin{align}\underset{G}{\text{min}} \underset{D}{\text{max}}V(D,G) = \mathbb{E}_{x\sim p_{data}(x)}\big[logD(x)\big] + \mathbb{E}_{z\sim p_{z}(z)}\big[log(1-D(G(z)))\big]\end{align}

In theory, the solution to this minimax game is where
$p_g = p_{data}$, and the discriminator guesses randomly if the
inputs are real or fake. However, the convergence theory of GANs is
still being actively researched and in reality models do not always
train to this point.

What is a DCGAN?
~~~~~~~~~~~~~~~~

A DCGAN is a direct extension of the GAN described above, except that it
explicitly uses convolutional and convolutional-transpose layers in the
discriminator and generator, respectively. It was first described by
Radford et. al. in the paper `Unsupervised Representation Learning With
Deep Convolutional Generative Adversarial
Networks <https://arxiv.org/pdf/1511.06434.pdf>`__. The discriminator
is made up of strided
`convolution <https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d>`__
layers, `batch
norm <https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm2d>`__
layers, and
`LeakyReLU <https://pytorch.org/docs/stable/nn.html#torch.nn.LeakyReLU>`__
activations. The input is a 3x64x64 input image and the output is a
scalar probability that the input is from the real data distribution.
The generator is comprised of
`convolutional-transpose <https://pytorch.org/docs/stable/nn.html#torch.nn.ConvTranspose2d>`__
layers, batch norm layers, and
`ReLU <https://pytorch.org/docs/stable/nn.html#relu>`__ activations. The
input is a latent vector, $z$, that is drawn from a standard
normal distribution and the output is a 3x64x64 RGB image. The strided
conv-transpose layers allow the latent vector to be transformed into a
volume with the same shape as an image. In the paper, the authors also
give some tips about how to setup the optimizers, how to calculate the
loss functions, and how to initialize the model weights, all of which
will be explained in the coming sections.

```python
from __future__ import print_function
#%matplotlib inline
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

# Set random seed for reproducibility
manualSeed = 999
#manualSeed = random.randint(1, 10000) # use if you want new results
print("Random Seed: ", manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)
```

Inputs
------

Let’s define some inputs for the run:

-  **dataroot** - the path to the root of the dataset folder. We will
   talk more about the dataset in the next section
-  **workers** - the number of worker threads for loading the data with
   the DataLoader
-  **batch_size** - the batch size used in training. The DCGAN paper
   uses a batch size of 128
-  **image_size** - the spatial size of the images used for training.
   This implementation defaults to 64x64. If another size is desired,
   the structures of D and G must be changed. See
   `here <https://github.com/pytorch/examples/issues/70>`__ for more
   details
-  **nc** - number of color channels in the input images. For color
   images this is 3
-  **nz** - length of latent vector
-  **ngf** - relates to the depth of feature maps carried through the
   generator
-  **ndf** - sets the depth of feature maps propagated through the
   discriminator
-  **num_epochs** - number of training epochs to run. Training for
   longer will probably lead to better results but will also take much
   longer
-  **lr** - learning rate for training. As described in the DCGAN paper,
   this number should be 0.0002
-  **beta1** - beta1 hyperparameter for Adam optimizers. As described in
   paper, this number should be 0.5
-  **ngpu** - number of GPUs available. If this is 0, code will run in
   CPU mode. If this number is greater than 0 it will run on that number
   of GPUs

```python
# Root directory for dataset
dataroot = "data/celeba"

# Number of workers for dataloader
workers = 2

# Batch size during training
batch_size = 128

# Spatial size of training images. All images will be resized to this
#   size using a transformer.
image_size = 64

# Number of channels in the training images. For color images this is 3
nc = 3

# Size of z latent vector (i.e. size of generator input)
nz = 100

# Size of feature maps in generator
ngf = 64

# Size of feature maps in discriminator
ndf = 64

# Number of training epochs
num_epochs = 5

# Learning rate for optimizers
lr = 0.0002

# Beta1 hyperparam for Adam optimizers
beta1 = 0.5

# Number of GPUs available. Use 0 for CPU mode.
ngpu = 1
```

Data
----

In this tutorial we will use the `Celeb-A Faces
dataset <http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html>`__ which can
be downloaded at the linked site, or in `Google
Drive <https://drive.google.com/drive/folders/0B7EVK8r0v71pTUZsaXdaSnZBZzg>`__.
The dataset will download as a file named *img_align_celeba.zip*. Once
downloaded, create a directory named *celeba* and extract the zip file
into that directory. Then, set the *dataroot* input for this notebook to
the *celeba* directory you just created. The resulting directory
structure should be:

::

   /path/to/celeba
       -> img_align_celeba
           -> 188242.jpg
           -> 173822.jpg
           -> 284702.jpg
           -> 537394.jpg
              ...

This is an important step because we will be using the ImageFolder
dataset class, which requires there to be subdirectories in the
dataset’s root folder. Now, we can create the dataset, create the
dataloader, set the device to run on, and finally visualize some of the
training data.

```python
# We can use an image folder dataset the way we have it setup.
# Create the dataset
dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
# Create the dataloader
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)

# Decide which device we want to run on
device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

# Plot some training images
real_batch = next(iter(dataloader))
plt.figure(figsize=(8,8))
plt.axis("off")
plt.title("Training Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(),(1,2,0)))
```

Implementation
--------------

With our input parameters set and the dataset prepared, we can now get
into the implementation. We will start with the weight initialization
strategy, then talk about the generator, discriminator, loss functions,
and training loop in detail.

Weight Initialization
~~~~~~~~~~~~~~~~~~~~~

From the DCGAN paper, the authors specify that all model weights shall
be randomly initialized from a Normal distribution with mean=0,
stdev=0.02. The ``weights_init`` function takes an initialized model as
input and reinitializes all convolutional, convolutional-transpose, and
batch normalization layers to meet this criteria. This function is
applied to the models immediately after initialization.

```python
# custom weights initialization called on netG and netD
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)
```

Generator
~~~~~~~~~

The generator, $G$, is designed to map the latent space vector
($z$) to data-space. Since our data are images, converting
$z$ to data-space means ultimately creating a RGB image with the
same size as the training images (i.e. 3x64x64). In practice, this is
accomplished through a series of strided two dimensional convolutional
transpose layers, each paired with a 2d batch norm layer and a relu
activation. The output of the generator is fed through a tanh function
to return it to the input data range of $[-1,1]$. It is worth
noting the existence of the batch norm functions after the
conv-transpose layers, as this is a critical contribution of the DCGAN
paper. These layers help with the flow of gradients during training. An
image of the generator from the DCGAN paper is shown below.

.. figure:: /_static/img/dcgan_generator.png
   :alt: dcgan_generator

Notice, the how the inputs we set in the input section (*nz*, *ngf*, and
*nc*) influence the generator architecture in code. *nz* is the length
of the z input vector, *ngf* relates to the size of the feature maps
that are propagated through the generator, and *nc* is the number of
channels in the output image (set to 3 for RGB images). Below is the
code for the generator.

```python
# Generator Code

class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)
```

Now, we can instantiate the generator and apply the ``weights_init``
function. Check out the printed model to see how the generator object is
structured.

```python
# Create the generator
netG = Generator(ngpu).to(device)

# Handle multi-gpu if desired
if (device.type == 'cuda') and (ngpu > 1):
    netG = nn.DataParallel(netG, list(range(ngpu)))

# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.02.
netG.apply(weights_init)

# Print the model
print(netG)
```

Discriminator
~~~~~~~~~~~~~

As mentioned, the discriminator, $D$, is a binary classification
network that takes an image as input and outputs a scalar probability
that the input image is real (as opposed to fake). Here, $D$ takes
a 3x64x64 input image, processes it through a series of Conv2d,
BatchNorm2d, and LeakyReLU layers, and outputs the final probability
through a Sigmoid activation function. This architecture can be extended
with more layers if necessary for the problem, but there is significance
to the use of the strided convolution, BatchNorm, and LeakyReLUs. The
DCGAN paper mentions it is a good practice to use strided convolution
rather than pooling to downsample because it lets the network learn its
own pooling function. Also batch norm and leaky relu functions promote
healthy gradient flow which is critical for the learning process of both
$G$ and $D$.

Discriminator Code

```python
class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 4 x 4
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.main(input)
```

Now, as with the generator, we can create the discriminator, apply the
``weights_init`` function, and print the model’s structure.

```python
# Create the Discriminator
netD = Discriminator(ngpu).to(device)

# Handle multi-gpu if desired
if (device.type == 'cuda') and (ngpu > 1):
    netD = nn.DataParallel(netD, list(range(ngpu)))

# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.2.
netD.apply(weights_init)

# Print the model
print(netD)
```

Loss Functions and Optimizers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

With $D$ and $G$ setup, we can specify how they learn
through the loss functions and optimizers. We will use the Binary Cross
Entropy loss
(`BCELoss <https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss>`__)
function which is defined in PyTorch as:

\begin{align}\ell(x, y) = L = \{l_1,\dots,l_N\}^\top, \quad l_n = - \left[ y_n \cdot \log x_n + (1 - y_n) \cdot \log (1 - x_n) \right]\end{align}

Notice how this function provides the calculation of both log components
in the objective function (i.e. $log(D(x))$ and
$log(1-D(G(z)))$). We can specify what part of the BCE equation to
use with the $y$ input. This is accomplished in the training loop
which is coming up soon, but it is important to understand how we can
choose which component we wish to calculate just by changing $y$
(i.e. GT labels).

Next, we define our real label as 1 and the fake label as 0. These
labels will be used when calculating the losses of $D$ and
$G$, and this is also the convention used in the original GAN
paper. Finally, we set up two separate optimizers, one for $D$ and
one for $G$. As specified in the DCGAN paper, both are Adam
optimizers with learning rate 0.0002 and Beta1 = 0.5. For keeping track
of the generator’s learning progression, we will generate a fixed batch
of latent vectors that are drawn from a Gaussian distribution
(i.e. fixed_noise) . In the training loop, we will periodically input
this fixed_noise into $G$, and over the iterations we will see
images form out of the noise.

```python
# Initialize BCELoss function
criterion = nn.BCELoss()

# Create batch of latent vectors that we will use to visualize
#  the progression of the generator
fixed_noise = torch.randn(64, nz, 1, 1, device=device)

# Establish convention for real and fake labels during training
real_label = 1.
fake_label = 0.

# Setup Adam optimizers for both G and D
optimizerD = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))
```

Training
~~~~~~~~

Finally, now that we have all of the parts of the GAN framework defined,
we can train it. Be mindful that training GANs is somewhat of an art
form, as incorrect hyperparameter settings lead to mode collapse with
little explanation of what went wrong. Here, we will closely follow
Algorithm 1 from Goodfellow’s paper, while abiding by some of the best
practices shown in `ganhacks <https://github.com/soumith/ganhacks>`__.
Namely, we will “construct different mini-batches for real and fake”
images, and also adjust G’s objective function to maximize
$logD(G(z))$. Training is split up into two main parts. Part 1
updates the Discriminator and Part 2 updates the Generator.

**Part 1 - Train the Discriminator**

Recall, the goal of training the discriminator is to maximize the
probability of correctly classifying a given input as real or fake. In
terms of Goodfellow, we wish to “update the discriminator by ascending
its stochastic gradient”. Practically, we want to maximize
$log(D(x)) + log(1-D(G(z)))$. Due to the separate mini-batch
suggestion from ganhacks, we will calculate this in two steps. First, we
will construct a batch of real samples from the training set, forward
pass through $D$, calculate the loss ($log(D(x))$), then
calculate the gradients in a backward pass. Secondly, we will construct
a batch of fake samples with the current generator, forward pass this
batch through $D$, calculate the loss ($log(1-D(G(z)))$),
and *accumulate* the gradients with a backward pass. Now, with the
gradients accumulated from both the all-real and all-fake batches, we
call a step of the Discriminator’s optimizer.

**Part 2 - Train the Generator**

As stated in the original paper, we want to train the Generator by
minimizing $log(1-D(G(z)))$ in an effort to generate better fakes.
As mentioned, this was shown by Goodfellow to not provide sufficient
gradients, especially early in the learning process. As a fix, we
instead wish to maximize $log(D(G(z)))$. In the code we accomplish
this by: classifying the Generator output from Part 1 with the
Discriminator, computing G’s loss *using real labels as GT*, computing
G’s gradients in a backward pass, and finally updating G’s parameters
with an optimizer step. It may seem counter-intuitive to use the real
labels as GT labels for the loss function, but this allows us to use the
$log(x)$ part of the BCELoss (rather than the $log(1-x)$
part) which is exactly what we want.

Finally, we will do some statistic reporting and at the end of each
epoch we will push our fixed_noise batch through the generator to
visually track the progress of G’s training. The training statistics
reported are:

-  **Loss_D** - discriminator loss calculated as the sum of losses for
   the all real and all fake batches ($log(D(x)) + log(1 - D(G(z)))$).
-  **Loss_G** - generator loss calculated as $log(D(G(z)))$
-  **D(x)** - the average output (across the batch) of the discriminator
   for the all real batch. This should start close to 1 then
   theoretically converge to 0.5 when G gets better. Think about why
   this is.
-  **D(G(z))** - average discriminator outputs for the all fake batch.
   The first number is before D is updated and the second number is
   after D is updated. These numbers should start near 0 and converge to
   0.5 as G gets better. Think about why this is.

**Note:** This step might take a while, depending on how many epochs you
run and if you removed some data from the dataset.

```python
# Training Loop

# Lists to keep track of progress
img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
# For each epoch
for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch, accumulated (summed) with previous gradients
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Compute error of D as sum over the fake and the real batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()

        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()

        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1
```

Results
-------

Finally, lets check out how we did. Here, we will look at three
different results. First, we will see how D and G’s losses changed
during training. Second, we will visualize G’s output on the fixed_noise
batch for every epoch. And third, we will look at a batch of real data
next to a batch of fake data from G.

**Loss versus training iteration**

Below is a plot of D & G’s losses versus training iterations.

```python
plt.figure(figsize=(10,5))
plt.title("Generator and Discriminator Loss During Training")
plt.plot(G_losses,label="G")
plt.plot(D_losses,label="D")
plt.xlabel("iterations")
plt.ylabel("Loss")
plt.legend()
plt.show()
```

**Visualization of G’s progression**

Remember how we saved the generator’s output on the fixed_noise batch
after every epoch of training. Now, we can visualize the training
progression of G with an animation. Press the play button to start the
animation.

```python
#%%capture
fig = plt.figure(figsize=(8,8))
plt.axis("off")
ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated=True)] for i in img_list]
ani = animation.ArtistAnimation(fig, ims, interval=1000, repeat_delay=1000, blit=True)

HTML(ani.to_jshtml())
```

**Real Images vs. Fake Images**

Finally, lets take a look at some real images and fake images side by
side.

```python
# Grab a batch of real images from the dataloader
real_batch = next(iter(dataloader))

# Plot the real images
plt.figure(figsize=(15,15))
plt.subplot(1,2,1)
plt.axis("off")
plt.title("Real Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))

# Plot the fake images from the last epoch
plt.subplot(1,2,2)
plt.axis("off")
plt.title("Fake Images")
plt.imshow(np.transpose(img_list[-1],(1,2,0)))
plt.show()
```

Where to Go Next
----------------

We have reached the end of our journey, but there are several places you
could go from here. You could:

-  Train for longer to see how good the results get
-  Modify this model to take a different dataset and possibly change the
   size of the images and the model architecture
-  Check out some other cool GAN projects
   `here <https://github.com/nashory/gans-awesome-applications>`__
-  Create GANs that generate
   `music <https://deepmind.com/blog/wavenet-generative-model-raw-audio/>`__