## TensorFlow로 Logistic Classification 구현하기(new)

### Logistic Regression
\begin{align}
H(X) = \frac{1}{1 + e^{-W^{T}X}} \\
\end{align}

\begin{align}
cost(W) = -\frac{1}{m}\sum ylog(H(x)) + (1 - y)log(1 - H(x)) \\
\end{align}

\begin{align}
W := W - \alpha \frac{\sigma}{\sigma W}cost(W) \\
\end{align}

```python
import tensorflow as tf
```

    /anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

### Training Data
`y`의 값은 **0 또는 1**의 Binary 값

```python
x_data = [
    [1, 2], [2, 3], [3, 1],
    [4, 3], [5, 3], [6, 2],
]
y_data = [
    [0], [0], [0],
    [1], [1], [1],
]

# placeholders for a tensor that will be always fed.
X = tf.placeholder(tf.float32, shape=[None, 2])
Y = tf.placeholder(tf.float32, shape=[None, 1])
```

### Hypothesis
\begin{align}
H(X) = \frac{1}{1 + e^{-W^{T}X}} \\
\end{align}

`W`의 `shape`은 **[들어오는 데이터 개수, 나가는 데이터 개수]**

`b`의 `shape`은 **나가는 데이터 개수**

```python
W = tf.Variable(tf.random_normal([2, 1]), name="weight")
b = tf.Variable(tf.random_normal([1]), name="bias")

# Hypothesis using sigmoid : tf.div(1., 1. + tf.exp(tf.matmul(X, W) + b))
hypothesis = tf.sigmoid(tf.matmul(X, W) + b)
```

### Cost Function
\begin{align}
cost(W) = -\frac{1}{m}\sum ylog(H(x)) + (1 - y)log(1 - H(x)) \\
\end{align}

```python
# Cost/Loss function
cost = -tf.reduce_mean(Y * tf.log(hypothesis)
                       + (1 - Y) * tf.log(1 - hypothesis))
```

### Optimizing with Gradient Descent
\begin{align}
W := W - \alpha \frac{\sigma}{\sigma W}cost(W) \\
\end{align}

```python
train = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(cost)
```

### Accuuracy computtation

```python
# True if hypothesis > 0.5 else False
predicted = tf.cast(hypothesis > 0.5, dtype=tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype=tf.float32))
```

### Train the model

```python
# Launch graph
with tf.Session() as sess:
    # Initialize Tenserflow variables
    sess.run(tf.global_variables_initializer())

    for step in range(10001):
        cost_val, _ = sess.run([cost, train],
                              feed_dict={X: x_data, Y: y_data})

        if step % 1000 == 0:
            print(step, cost_val)

    # Accuracy report
    h, c, a = sess.run([hypothesis, predicted, accuracy],
                      feed_dict={X: x_data, Y: y_data})
    print("\nHypothesis : ", h,
         "\nCorrect (Y) : ", c,
         "\nAccuracy : ", a)
```

    0 0.8831861
    1000 0.30194607
    2000 0.2640162
    3000 0.2339095
    4000 0.20963901
    5000 0.18977194
    6000 0.17327213
    7000 0.15938494
    8000 0.14755455
    9000 0.13736634
    10000 0.12850672

    Hypothesis :  [[0.02243203]
     [0.14577791]
     [0.26223317]
     [0.80160093]
     [0.95159554]
     [0.98427284]]
    Correct (Y) :  [[0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]]
    Accuracy :  1.0

### Classifying diabetes

```python
import numpy as np

xy = np.loadtxt('data-03-diabetes.csv',
               delimiter=',',
               dtype=np.float32)
x_data = xy[:, 0:-1]
y_data = xy[:, [-1]]
```

```python
# placeholders for a tensor that will be always fed.
X = tf.placeholder(tf.float32, shape=[None, 8])
Y = tf.placeholder(tf.float32, shape=[None, 1])

W = tf.Variable(tf.random_normal([8, 1]), name="weight")
b = tf.Variable(tf.random_normal([1]), name="bias")

# Hypothesis using sigmoid : tf.div(1., 1. + tf.exp(tf.matmul(X, W)))
hypothesis = tf.sigmoid(tf.matmul(X, W) + b)

# Cost/Loss function
cost = -tf.reduce_mean(Y * tf.log(hypothesis)
                      + (1 - Y) * tf.log(1 - hypothesis))

# Optimizing with Gradient Descent
train = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(cost)

# Accuracy computation, True if hypothesis > 0.5 else False
predicted = tf.cast(hypothesis > 0.5, dtype=tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype=tf.float32))

# Launch Graph
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    feed = {X: x_data, Y: y_data}

    for step in range(10001):
        sess.run(train, feed_dict=feed)

        if step % 1000 == 0:
            print(step, sess.run(cost, feed_dict=feed))

    # Accuracy report
    h, c, a = sess.run([hypothesis, predicted, accuracy], feed_dict=feed)
    print("\nHypothesis : ", h,
         "\nCorrect (Y) : ", c,
         "\nAccuracy : ", a)
```

    0 0.899115
    1000 0.5735633
    2000 0.5399573
    3000 0.5193476
    4000 0.5059997
    5000 0.49698162
    6000 0.49068317
    7000 0.48616815
    8000 0.48286337
    9000 0.48040378
    10000 0.4785474

    Hypothesis :  [[0.38358444]
     [0.92869014]
     [0.22825797]
     [0.93518037]
     [0.10852826]
     [0.787232  ]
     [0.9218224 ]
     [0.5370475 ]
     [0.22235288]
     [0.58532554]
     [0.7563662 ]
     [0.16214497]
     [0.28535488]
     [0.22292668]
     [0.715882  ]
     [0.4566469 ]
     [0.74745923]
     [0.7087745 ]
     [0.8117335 ]
     [0.64360595]
     [0.6781297 ]
     [0.11629537]
     [0.70068216]
     [0.6537068 ]
     [0.33199382]
     [0.9449027 ]
     [0.5891875 ]
     [0.70406467]
     [0.6685961 ]
     [0.47938004]
     [0.95094585]
     [0.92311436]
     [0.6121467 ]
     [0.8260642 ]
     [0.37224534]
     [0.646797  ]
     [0.81102914]
     [0.51438195]
     [0.42212456]
     [0.38334653]
     [0.88059723]
     [0.19786389]
     [0.37716255]
     [0.05471164]
     [0.5385413 ]
     [0.94200104]
     [0.6673482 ]
     [0.68329626]
     [0.96004236]
     [0.9231035 ]
     [0.9355266 ]
     [0.25386885]
     [0.31512013]
     [0.9532639 ]
     [0.17026478]
     [0.5018234 ]
     [0.13977583]
     [0.65864277]
     [0.83452433]
     [0.48667055]
     [0.9481998 ]
     [0.70706546]
     [0.62005216]
     [0.8639901 ]
     [0.66317   ]
     [0.60235244]
     [0.9642639 ]
     [0.7535737 ]
     [0.8546952 ]
     [0.6633986 ]
     [0.2769991 ]
     [0.7679933 ]
     [0.9299653 ]
     [0.9135071 ]
     [0.88641137]
     [0.7847674 ]
     [0.33811414]
     [0.8781557 ]
     [0.87426686]
     [0.9090308 ]
     [0.88624156]
     [0.83624446]
     [0.43712312]
     [0.8326604 ]
     [0.48518074]
     [0.8588396 ]
     [0.35864374]
     [0.9078545 ]
     [0.932034  ]
     [0.7972444 ]
     [0.773362  ]
     [0.65706444]
     [0.7649187 ]
     [0.5573284 ]
     [0.89538395]
     [0.974031  ]
     [0.86720425]
     [0.60665303]
     [0.24617603]
     [0.64418596]
     [0.7195454 ]
     [0.96596944]
     [0.7508909 ]
     [0.71327007]
     [0.9582323 ]
     [0.63900334]
     [0.90623814]
     [0.8434402 ]
     [0.4995852 ]
     [0.29049766]
     [0.9417074 ]
     [0.85972506]
     [0.35603616]
     [0.49255237]
     [0.62461555]
     [0.7877343 ]
     [0.86387163]
     [0.9309887 ]
     [0.11604828]
     [0.70303154]
     [0.86532795]
     [0.62632674]
     [0.6536208 ]
     [0.6061622 ]
     [0.66193706]
     [0.8142311 ]
     [0.8445073 ]
     [0.6493651 ]
     [0.5092369 ]
     [0.35851786]
     [0.39206305]
     [0.75958556]
     [0.94690526]
     [0.79780835]
     [0.771738  ]
     [0.84408265]
     [0.5057738 ]
     [0.7897179 ]
     [0.7703196 ]
     [0.7317643 ]
     [0.8516123 ]
     [0.5916011 ]
     [0.537085  ]
     [0.6881422 ]
     [0.9135119 ]
     [0.7498713 ]
     [0.42113176]
     [0.93955517]
     [0.64037216]
     [0.7928342 ]
     [0.30435547]
     [0.42243373]
     [0.08742994]
     [0.22916287]
     [0.8898132 ]
     [0.87501043]
     [0.9509773 ]
     [0.08886664]
     [0.6041262 ]
     [0.7482316 ]
     [0.59354657]
     [0.84721893]
     [0.49927032]
     [0.8142368 ]
     [0.60762745]
     [0.6484976 ]
     [0.71865565]
     [0.9079216 ]
     [0.79706436]
     [0.6102605 ]
     [0.8869913 ]
     [0.8467155 ]
     [0.9478291 ]
     [0.22149368]
     [0.8296477 ]
     [0.2557624 ]
     [0.3374676 ]
     [0.41993418]
     [0.90932643]
     [0.6453444 ]
     [0.9252844 ]
     [0.9045975 ]
     [0.6144223 ]
     [0.12228866]
     [0.16505796]
     [0.72903234]
     [0.76904243]
     [0.67800164]
     [0.81878966]
     [0.59202397]
     [0.3385018 ]
     [0.13053364]
     [0.8797435 ]
     [0.40903696]
     [0.84522754]
     [0.9082498 ]
     [0.7045461 ]
     [0.6039686 ]
     [0.64174235]
     [0.58747447]
     [0.7232668 ]
     [0.95661044]
     [0.7225852 ]
     [0.8426126 ]
     [0.11361366]
     [0.3871605 ]
     [0.88346976]
     [0.22192602]
     [0.9318903 ]
     [0.292458  ]
     [0.2859427 ]
     [0.40722886]
     [0.7248754 ]
     [0.17400154]
     [0.7231237 ]
     [0.71808714]
     [0.83675617]
     [0.6143943 ]
     [0.11475439]
     [0.39208904]
     [0.68992007]
     [0.4504621 ]
     [0.93425184]
     [0.944193  ]
     [0.70267725]
     [0.29773793]
     [0.04137846]
     [0.63602203]
     [0.3898668 ]
     [0.4048831 ]
     [0.96501935]
     [0.62167454]
     [0.95183396]
     [0.20411196]
     [0.15985925]
     [0.3315646 ]
     [0.81985027]
     [0.90031475]
     [0.88629854]
     [0.63003635]
     [0.6242349 ]
     [0.5907765 ]
     [0.17632507]
     [0.52242297]
     [0.13350667]
     [0.5469541 ]
     [0.9031147 ]
     [0.5836815 ]
     [0.7707274 ]
     [0.96387446]
     [0.7971679 ]
     [0.74378055]
     [0.7226816 ]
     [0.73948747]
     [0.86569303]
     [0.35577413]
     [0.39786932]
     [0.50820833]
     [0.8281822 ]
     [0.6415036 ]
     [0.64939183]
     [0.8130901 ]
     [0.28968212]
     [0.4647438 ]
     [0.5497614 ]
     [0.6155614 ]
     [0.4244815 ]
     [0.89031714]
     [0.8049671 ]
     [0.9300796 ]
     [0.49883235]
     [0.7638981 ]
     [0.8035795 ]
     [0.8219076 ]
     [0.68962693]
     [0.87371486]
     [0.32560325]
     [0.5654856 ]
     [0.7267813 ]
     [0.39760894]
     [0.8313555 ]
     [0.31627622]
     [0.63412035]
     [0.93630517]
     [0.77209014]
     [0.8591702 ]
     [0.6550491 ]
     [0.49656102]
     [0.5505635 ]
     [0.3133578 ]
     [0.419552  ]
     [0.66538423]
     [0.6887774 ]
     [0.6215462 ]
     [0.62904257]
     [0.18703406]
     [0.64939237]
     [0.9132772 ]
     [0.52297413]
     [0.684328  ]
     [0.7585538 ]
     [0.45062017]
     [0.7009901 ]
     [0.52640545]
     [0.71014017]
     [0.89537716]
     [0.6436055 ]
     [0.73423815]
     [0.83183444]
     [0.57117295]
     [0.851155  ]
     [0.9598591 ]
     [0.32634813]
     [0.757582  ]
     [0.26061553]
     [0.7597451 ]
     [0.82698536]
     [0.69915843]
     [0.42889503]
     [0.77002996]
     [0.7703182 ]
     [0.72849274]
     [0.1617552 ]
     [0.82702786]
     [0.8430812 ]
     [0.61185277]
     [0.9318309 ]
     [0.189637  ]
     [0.7298491 ]
     [0.9508516 ]
     [0.18748707]
     [0.41735935]
     [0.6934426 ]
     [0.34852085]
     [0.17845592]
     [0.8667196 ]
     [0.9298109 ]
     [0.86477363]
     [0.6652992 ]
     [0.61939263]
     [0.58353186]
     [0.72281116]
     [0.8220983 ]
     [0.938206  ]
     [0.7347169 ]
     [0.76720524]
     [0.6439237 ]
     [0.9400085 ]
     [0.9425537 ]
     [0.70657086]
     [0.30745777]
     [0.63426465]
     [0.2686333 ]
     [0.72978246]
     [0.21973273]
     [0.23753798]
     [0.38102332]
     [0.75168777]
     [0.33533517]
     [0.57203305]
     [0.8027488 ]
     [0.67653024]
     [0.84326744]
     [0.96392274]
     [0.82449627]
     [0.12120374]
     [0.43176952]
     [0.7972516 ]
     [0.83308125]
     [0.61315393]
     [0.25664648]
     [0.9113332 ]
     [0.88139886]
     [0.256256  ]
     [0.68566614]
     [0.8616177 ]
     [0.86975026]
     [0.854755  ]
     [0.9144662 ]
     [0.89128494]
     [0.9154641 ]
     [0.69511503]
     [0.6556278 ]
     [0.56007963]
     [0.84427446]
     [0.866325  ]
     [0.213172  ]
     [0.80044496]
     [0.8916546 ]
     [0.36534813]
     [0.6304005 ]
     [0.85538477]
     [0.52837366]
     [0.9308946 ]
     [0.26669717]
     [0.8197333 ]
     [0.6532666 ]
     [0.88392293]
     [0.31488928]
     [0.5936445 ]
     [0.72237927]
     [0.8004662 ]
     [0.11547218]
     [0.20287035]
     [0.74461806]
     [0.82905227]
     [0.50877666]
     [0.793081  ]
     [0.40516073]
     [0.41162694]
     [0.8712263 ]
     [0.47417462]
     [0.9394584 ]
     [0.8167123 ]
     [0.6984427 ]
     [0.916346  ]
     [0.57659566]
     [0.7689996 ]
     [0.3121647 ]
     [0.28678358]
     [0.72142   ]
     [0.38839293]
     [0.5227999 ]
     [0.90421295]
     [0.9037276 ]
     [0.9170964 ]
     [0.9581307 ]
     [0.7322389 ]
     [0.90620685]
     [0.29048604]
     [0.36845684]
     [0.51541346]
     [0.9473159 ]
     [0.6561935 ]
     [0.23820554]
     [0.93025374]
     [0.8003683 ]
     [0.5915066 ]
     [0.7900606 ]
     [0.0184541 ]
     [0.9338559 ]
     [0.78582954]
     [0.73882365]
     [0.77685446]
     [0.9694569 ]
     [0.6676639 ]
     [0.7474692 ]
     [0.7099186 ]
     [0.81676024]
     [0.15007444]
     [0.55627155]
     [0.9117084 ]
     [0.56511766]
     [0.76684713]
     [0.94969153]
     [0.83362895]
     [0.8980183 ]
     [0.6616757 ]
     [0.7462703 ]
     [0.92941016]
     [0.7202547 ]
     [0.5987894 ]
     [0.29739735]
     [0.4967917 ]
     [0.5258038 ]
     [0.57703656]
     [0.5785218 ]
     [0.77639693]
     [0.5954364 ]
     [0.81289905]
     [0.8567499 ]
     [0.7500248 ]
     [0.68101394]
     [0.4699417 ]
     [0.6353964 ]
     [0.9284204 ]
     [0.8549291 ]
     [0.19647479]
     [0.39261264]
     [0.41699085]
     [0.0873605 ]
     [0.9020166 ]
     [0.15188427]
     [0.89375114]
     [0.90035474]
     [0.8238246 ]
     [0.69117326]
     [0.8791041 ]
     [0.33970386]
     [0.788192  ]
     [0.94604796]
     [0.28747395]
     [0.45900217]
     [0.9039191 ]
     [0.8705921 ]
     [0.6162381 ]
     [0.78579426]
     [0.81214714]
     [0.819016  ]
     [0.30889592]
     [0.7285797 ]
     [0.86410505]
     [0.62943417]
     [0.77875507]
     [0.6763221 ]
     [0.79466283]
     [0.87196624]
     [0.9228989 ]
     [0.60461724]
     [0.45846954]
     [0.718733  ]
     [0.8151281 ]
     [0.9717164 ]
     [0.7952107 ]
     [0.66224056]
     [0.38260308]
     [0.6859242 ]
     [0.91641927]
     [0.9508483 ]
     [0.90837383]
     [0.69767344]
     [0.6730251 ]
     [0.8042912 ]
     [0.45807278]
     [0.7971515 ]
     [0.78399247]
     [0.8866215 ]
     [0.588403  ]
     [0.7511089 ]
     [0.8772425 ]
     [0.52906245]
     [0.6179532 ]
     [0.6312149 ]
     [0.7421923 ]
     [0.67092866]
     [0.92058206]
     [0.9339238 ]
     [0.24425109]
     [0.11854582]
     [0.74922556]
     [0.5412127 ]
     [0.37996605]
     [0.83507556]
     [0.90722895]
     [0.7234883 ]
     [0.9347857 ]
     [0.9061849 ]
     [0.76524854]
     [0.82875824]
     [0.68826485]
     [0.46296638]
     [0.76978564]
     [0.5831379 ]
     [0.09870809]
     [0.89825726]
     [0.8714632 ]
     [0.7527517 ]
     [0.91331583]
     [0.8428286 ]
     [0.8533452 ]
     [0.5840169 ]
     [0.65526676]
     [0.8810204 ]
     [0.80954665]
     [0.8452553 ]
     [0.89560115]
     [0.68619055]
     [0.7300278 ]
     [0.7852422 ]
     [0.5288223 ]
     [0.51920885]
     [0.08239   ]
     [0.26165548]
     [0.8252926 ]
     [0.62899053]
     [0.6507457 ]
     [0.5645968 ]
     [0.93729556]
     [0.38737082]
     [0.82055014]
     [0.31288937]
     [0.9011081 ]
     [0.3214235 ]
     [0.7852942 ]
     [0.59080106]
     [0.8557567 ]
     [0.5888349 ]
     [0.28144592]
     [0.74702007]
     [0.87797666]
     [0.36643296]
     [0.89359045]
     [0.9120931 ]
     [0.84984934]
     [0.83074284]
     [0.40477473]
     [0.29221603]
     [0.62750995]
     [0.21096067]
     [0.95624024]
     [0.34753793]
     [0.9278199 ]
     [0.85792667]
     [0.36835852]
     [0.23152813]
     [0.74183136]
     [0.39906916]
     [0.8554612 ]
     [0.7899913 ]
     [0.98137164]
     [0.59682804]
     [0.5978143 ]
     [0.8114339 ]
     [0.8573951 ]
     [0.09404772]
     [0.6784412 ]
     [0.77475804]
     [0.8692159 ]
     [0.6164473 ]
     [0.46066296]
     [0.60526216]
     [0.8948087 ]
     [0.63127905]
     [0.79138565]
     [0.8176325 ]
     [0.89226776]
     [0.78856075]
     [0.5600209 ]
     [0.79038894]
     [0.9158394 ]
     [0.6974972 ]
     [0.9666435 ]
     [0.8433426 ]
     [0.6226161 ]
     [0.504053  ]
     [0.8251609 ]
     [0.86935425]
     [0.46509883]
     [0.701803  ]
     [0.17538516]
     [0.5998416 ]
     [0.7998604 ]
     [0.9428865 ]
     [0.81477207]
     [0.7342592 ]
     [0.7349205 ]
     [0.88854223]
     [0.4126339 ]
     [0.93202716]
     [0.59678125]
     [0.879602  ]
     [0.3333787 ]
     [0.09619869]
     [0.35827997]
     [0.34921813]
     [0.6423691 ]
     [0.8496715 ]
     [0.5726471 ]
     [0.73201334]
     [0.7601192 ]
     [0.497501  ]
     [0.33966666]
     [0.88593066]
     [0.9215844 ]
     [0.44887346]
     [0.65370625]
     [0.18553129]
     [0.437857  ]
     [0.70273453]
     [0.63762975]
     [0.90324396]
     [0.97665954]
     [0.16625205]
     [0.66616285]
     [0.6581229 ]
     [0.43553898]
     [0.74571335]
     [0.7547589 ]
     [0.8680974 ]
     [0.7820558 ]
     [0.46601865]
     [0.7172281 ]
     [0.2173857 ]
     [0.6802295 ]
     [0.47634494]
     [0.9114828 ]
     [0.622409  ]
     [0.58653283]
     [0.79168177]
     [0.7756396 ]
     [0.4448875 ]
     [0.76821584]
     [0.66892487]
     [0.37090668]
     [0.5378888 ]
     [0.89806485]
     [0.8433975 ]
     [0.5391709 ]
     [0.7175815 ]
     [0.28513443]
     [0.83353496]
     [0.583816  ]
     [0.774162  ]
     [0.3329831 ]
     [0.5917075 ]
     [0.8473725 ]
     [0.1276211 ]
     [0.33256516]
     [0.8070793 ]
     [0.79440725]
     [0.7772855 ]
     [0.9276509 ]
     [0.7405661 ]
     [0.688033  ]
     [0.7385463 ]
     [0.82124895]
     [0.68927526]
     [0.8142723 ]
     [0.54975504]
     [0.569551  ]
     [0.86365515]
     [0.82062167]
     [0.7166321 ]
     [0.3211108 ]
     [0.8683431 ]
     [0.8347557 ]
     [0.8062445 ]
     [0.7057985 ]
     [0.88100314]
     [0.8233751 ]
     [0.75604784]
     [0.38913724]
     [0.8534568 ]
     [0.90455055]
     [0.40011317]
     [0.17607643]
     [0.74703175]
     [0.4406432 ]
     [0.79000634]
     [0.3033449 ]
     [0.41360813]
     [0.47358236]
     [0.78020144]
     [0.8695359 ]
     [0.1537763 ]
     [0.3795958 ]
     [0.6702843 ]
     [0.5631636 ]
     [0.46744844]
     [0.79770285]
     [0.16813281]
     [0.9151287 ]
     [0.15277003]
     [0.845123  ]
     [0.68586665]
     [0.7215009 ]
     [0.84273595]
     [0.6738128 ]
     [0.90443367]]
    Correct (Y) :  [[0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [0.]
     [0.]
     [1.]
     [1.]
     [0.]
     [0.]
     [1.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [0.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]
     [1.]]
    Accuracy :  0.77470356


# Kentel Python3 - Grafikoù

## Dispenn an izotopoù radioaktivel

Gant ar fonksionoù ankivil [aléatoires] (pe kentoc'h damankivil [pseudo-aléatoire]) hag ar fed gallout ober kalz a jedadennoù buan tre eo an urzhiataerioù ostilhoù dispar evit ober darvanoù [simulations] luziet.

Klasket e vo darvanañ digresk eksponantel ur c'hementad a rannigoù radioaktivel e kerz an amzer.

Kroget e vo da tresañ grafik fonksion matematikel an digresk eksponantel evit gwelet he stumm ha pleustriñ gant ar grafikoù.

Kendalc'het e vo gant un darvan eeunek ha difetiz a-walc'h (met reiz memes tra).

Echuet e vo gant un darvan tostoc'h d'ar gwirion (ar pezh a fell deomp).

Gouzout a reomp un digresk eksponantel a zo eus ar stumm

$\begin{equation}
N(t) = N_0 \times e^{-\lambda t}
\end{equation}$

Gant :

$N_0$ Niver a rannigoù radioaktivel da gentañ (en amzer $t_0$)

$\lambda$ Ar c'honstantenn digresk

$t$ an amzer

## Grafikoù gant ar module `matplotlib`

Tu vefe termeniñ ur fonksion `N(t)` gant Python met `matplotlib` ne oar ket tresañ fonksionoù evel-se. Implijet e vez **listennoù roadennoù niverel** nemetken evit tresañ grafikoù.

```python
# Enframmet e vez ar module evit tresañ grafikoù
import matplotlib.pyplot as plt
# Ar ger 'as' a servij d'ober un alias etre matplotlib.pyplot ha plt
# Berroc'h ha kevatal e vo skrivañ 'plt' eget 'matplotlib.pyplot'
```

```python
# Ur skouer simpl

roadennou = [1, 2.2, 4, 8.9, 6, 5, 3, 1, 1.5, 0.2]

plt.plot(roadennou)  # Tresañ ar grafik (e memor an urzhiataer)
plt.show()           # Diskouez ar grafik war ar skramm
```

```python
# Tresomp an digresk eksponantel bremañ

# Kavet e vez ar fonksion eksponantel e-barzh ar module 'math'
from math import exp

kde = 0.1  # Konstantenn digresk eksponantel
n0 = 1000

roadennou = []

# Jedet e vez 100 talvoud da-heul
for t in range(100):
    roadennou.append(n0*exp(-kde*t))

plt.plot(roadennou)
plt.show()
```

## Un darvan difetiz

Pep rannig radioaktivel eus ar memes sort en eus ar memes probablentez bezañ dispennet e pad ur c'houlzad bennak.

Ma vez lavaret, da skouer, ez eo 1% ar probablentez evit ur rannig bezañ dispennet e-kerzh ur c'houlzad e dalvez ivez e vo dispennet 1% eus kementad ar rannigoù e-kerzh ar c'houlzad.

```python
prob_dispenn = 0.1
n0 = 1000

# Lakaet e vez talvoud n0 el listenn
roadennou = [n0]

for i in range(100):
    # Kemeret e vez an talvoud diwezhañ el listenn
    # Lemet e vez an niver a rannigoù dispennet e-kerzh ar c'houlzad
    # ouzhpennet e vez an talvoud nevez e fin al listenn
    roadennou.append(roadennou[i] - roadennou[i]*prob_dispenn)

plt.plot(roadennou)
plt.show()
```

## Un darvan tostoc'h eus ar gwirionnez

Ar skouerioù a-us a heulie formulennoù matematikel anavezet en araok. Ar formulennoù matematikel a zo dedennus evit diskriv ar fenomenoù naturel a-bell, a geidenn hag en un doare parfed met ar fenomenoù naturel ne heuliont ket formulennoù matematikel dre ret pa vezont sellet a-dost. Setup perak ne oant ket darvanoù da vat.

Evit kaout un darvan gwir e vez roet perzhioù ar sistem nemetken hag an diskoulm a vez dianavezet en araok. Pep gwech ma vez peurgaset an darvan e c'heller kavout un diskoulm disheñvel.

```python
# Enframmañ ar module evit implij fonksionoù ankivil
import random
```

```python
# Cheñchit talvoud ar roadennoù-se hervez o c'hoantoù
prob_dispenn = 0.1
rannigou = 1000

# El listenn 'mañ e vo miret ar roadennou net jedet
roadennou = [rannigou]

while rannigou>0:
    # Evit pep rannig a chom e vez tennet un niverenn d'ar sort
    for i in range(rannigou):
        # Ma 'vez tennet un niverenn bihanoc'h eget ar probablentez dibabet
        # e vez dispennet ur rannig
        if random.random() < prob_dispenn:
            rannigou -= 1

    # E fin ar roll e vez stoked a niver a rannigoù a chom en omp listennad data
    roadennou.append(rannigou)
```

Bremañ ne chom nemet da dresañ an data bet jedet war ur grafik brav gant ar stadamantoù da-heul.

```python
# Cheñch ment ar grafik
plt.rcParams["figure.figsize"] = [12, 8]

# Tresañ ar grafik
plt.plot(roadennou)

# Dibab un titl (dirediet)
plt.title("Dispenn an izotopoù radioaktivel")

# Dibab anvioù evit an axoù (dirediet)
plt.xlabel("Red an amzer (koulzadoù)")
plt.ylabel("Niver a rannigoù")

# Diskouez ar grafik
plt.show()
```


```python
from sympy import *
import import_ipynb
import Linear_ODE_real_evs as real_nb
import Linear_ODE_complex_evs as complex_nb
# Use pretty printing for SymPy `expr`essions
init_printing()
```

    importing Jupyter notebook from Linear_ODE_real_evs.ipynb
    importing Jupyter notebook from Linear_ODE_complex_evs.ipynb

# Linear ODEs: solve with Python

In this lecture I will show you how to make [Python] and [SymPy] solve linear ODEs in a few lines of code. I will reuse example ODEs from the other two videos.

[Python]: https://www.python.org "Python programming language"
[SymPy]: https://www.sympy.org "Symboic math in Python"

## Computation with matrix exponential

[SymPy] can compute matrix exponential, so any linear ODE $\dot x=Ax$ can be solved in one line: `x = exp(A * t) * c`, where `c` is the initial condition: `x(0)=c`.

[SymPy]: https://www.sympy.org "Symboic math in Python"

### Distinct real eigenvalues
Consider the ODE $\dot x=Ax$, where $A=\begin{bmatrix}1&2\\2&1\end{bmatrix}$.

```python
t, c, A = real_nb.t, real_nb.c, real_nb.A
x = exp(A * t) * c
assert x.diff(t).expand() == (A * x).expand(), "Not a solution of $\dot x=Ax$"
assert x.subs(t, 0) == c, "Does not satisfy the initial condition"
assert x.expand() == real_nb.x.expand(), "Solutions obtained by different methods are not equal to each other"
A, x
```

### Complex eigenvalues

Now let us use matrix exponential to get to the same answer. First, we ask `Python` to compute $\exp(At)$ and verify that it provides us with a solution of $\dot x=Ax$.

```python
t, c, A = complex_nb.t, complex_nb.c, complex_nb.A
display(A)
x = (exp(A * t) * c).expand()
assert x.diff(t).expand() == (A * x).expand(), "Not a solution of $\dot x=Ax$"
assert x.subs(t, 0) == c, "Does not satisfy the initial condition"
if x.expand() != complex_nb.x.expand():
    print("Solutions obtained by different methods are not equal to each other")
display(x)
```

The first two `assert`s tell us that `x` is a correct answer. Why is different from the one we found in another file?

```python
complex_nb.x
```

Have we found another solution of the same ODE? This would contradict the general theory. But `Python` says that these expressions are different! Why?

Actually, `x` and `complex_nb.x` are two different expressions for the same function, and `Python`s `==` test is not clever enough to figure out that these functions are actually equivalent. In order to make sure that these functions are actually equal, we can, e.g., rewrite all trig functions in `x` in terms of complex exponential. In `Python` this can be done using `x.rewrite(exp)`.

```python
assert complex_nb.x.rewrite(exp).expand() == x.expand()
complex_nb.x.rewrite(exp)
```

The `assert` line above ensures that the two formulas are equivalent. But it rewrites both formulas in terms of complex exponentials. How can we make `Python` rewrite `x` in terms of trig functions instead?

```python
x = x.rewrite(cos)
x
```

This `rewrite` replaced each $\exp$ with its expression in terms of $\cos$, $\sin$, $\cosh$, and $\sinh$. While it did what we wanted for $e^{\pm it}$, it replaced $e^{-2t}$ with $\cosh(2t)-\sinh(2t)$. I don't know how to prevent this unwanted rewriting, so we'll revert it with another `rewrite`.

```python
x = x.rewrite([cosh, sinh], exp)
x
```

Finally, expanding this expression we give the desired result.

```python
x = expand(x)
assert x == complex_nb.x.expand()
x
```

Here is the whole computation condensed into one line.

```python
(exp(A * t) * c).expand().rewrite(cos).rewrite([cosh, sinh], exp).expand()
```

## Computation with `dsolve`

We can also a generic ODE solver `dsolve` to solve the original ODE. Actually, the solver will figure out that our ODE is a linear ODE with constant coefficients and compute $\exp(At)$ but all this will be done for us behind the scene.

**NB**: most nonlinear differential equations can't be solved, and there are many nonlinear ODEs which can be solved but not by `dsolve`.

```python
var('x1 x2', cls=Function)
x = Matrix([x1(t), x2(t)])
A, c = real_nb.A, real_nb.c
eqs = [Eq(x.diff(t)[k,0], (A * x)[k,0]) for k in range(2)]
display(eqs)
res = dsolve(eqs, ics = {x1(0): c[0], x2(0): c[1]})
res = [expand(e) for e in res]
display(*res, expand(real_nb.x))
```

```python
var('x1 x2', cls=Function)
x = Matrix([x1(t), x2(t)])
A, c = complex_nb.A, complex_nb.c
eqs = [Eq(x.diff(t)[k,0], (A * x)[k,0]) for k in range(2)]
display(eqs)
res = dsolve(eqs, ics = {x1(0): c[0], x2(0): c[1]})
res = [expand(e) for e in res]
display(*res, expand(complex_nb.x))
```


[Table of Contents](./table-of-contents.ipynb)

## Statistical measures from a geometric perspective
In this section, we are going to look at the statistical measures from the perspective of linear algebra and geometry. Our goal is to investigate random variables and their relationships through correlation and variance analysis. We will build our content on the vector spaces and the dot product. We will put a special emphasis on considering various definitions of the concepts because they will shape our intuition.
### Introducing our star: Dot Product
Let’s start with the dot product. What is the dot product? It is the element-wise multiplication and then the addition of the elements of the vectors.

\begin{equation*}
   \overrightarrow{a}.\overrightarrow{b} = a_1.b_1 + \dots +  a_n.b_n
\end{equation*}

But what is the intuition behind this? Let’s play around algebraically first and consider Euclidean distance between these two vectors $\overrightarrow{a}$ and $\overrightarrow{b}$.

\begin{aligned}
\|\overrightarrow{a} -\overrightarrow{b}\|^2 = \sum_i (a_i - b_i)^2 &= \sum (a_i^2 + b_i^2 - 2.a_i.b_i) \\
&= \sum_i a_i^2 + \sum_i b_i^2 - 2\sum_i a_i.b_i \\
&=  \|\overrightarrow{a}\|^2 + \|\overrightarrow{b}\|^2 - 2.\overrightarrow{a}.\overrightarrow{b}
\end{aligned}

\begin{equation*}
\Longrightarrow \overrightarrow{a}.\overrightarrow{b} = \frac{1}{2} (\|\overrightarrow{a}\|^2 + \|\overrightarrow{b}\|^2 - \|\overrightarrow{a} -\overrightarrow{b}\|^2)
\end{equation*}

In this case, we can observe that as the distance between the vectors decreases the value of the dot product increases and vice versa. So, we can interpret the dot product as an operation that rewards the similarity which increases when the two vectors get closer.

Let’s continue to investigate this “closeness” in a geometric perspective and consider the effect of their direction.

     Figure 1.1. a and b vectors in x − y coordinates.

Let’s derive the formula in terms of vector components.
Let $\overrightarrow{i}$,$\overrightarrow{j}$ be the basis vectors  in  $\mathbb{R}^2 $ with length one and point parallel and in positive direction to the $x-$axis and  $y-$axis and let $\overrightarrow{a}=(a_x,a_y)$ and $\overrightarrow{b}=(b_x,b_y)$ be vectors in this space. So, we have:
\begin{aligned}
    \overrightarrow{a}.\overrightarrow{b} &= (a_x \overrightarrow{i} + a_y \overrightarrow{j}).(b_x \overrightarrow{i} + b_y \overrightarrow{j})  \\
    &= a_x b_x\overrightarrow{i}.\overrightarrow{i} + a_x b_y\overrightarrow{i}.\overrightarrow{j} +a_y b_x\overrightarrow{j}.\overrightarrow{i} + a_y b_y \overrightarrow{j}.\overrightarrow{j} \\
    &=  a_x b_x +  a_y b_y
\end{aligned}

where $\overrightarrow{i}.\overrightarrow{i}=\overrightarrow{j}.\overrightarrow{j}=1$ and $\overrightarrow{i}.\overrightarrow{j}=\overrightarrow{j}.\overrightarrow{i}=0.$

So, we can interpret that when we dot product two vectors, we reward their similarity with respect to their components in $x-y$ direction.
But there are infinitely many bases for this space, and $i-j$ pair is just one of them. Observe that if we rotate the $x-y$ axis, their basis vectors $\overrightarrow{i}, \overrightarrow{j}$ won't change as they point toward the positive direction of their respective axis. So, for example, we can choose a basis  whose unit vectors on the direction of $\overrightarrow{b}$ and calculate the dot product as follows:
\begin{equation}
    \overrightarrow{a}.\overrightarrow{b} = a_a . b_a + a_{a_{\perp}} . b_{a_{\perp}} = \left | \overrightarrow{b} \right |.\cos{\theta}  . \left | \overrightarrow{a} \right |
\end{equation}

We can also choose the unit vector in the direction of the vector $\overrightarrow{a}$ as $\overrightarrow{u} = \frac{\overrightarrow{a}}{||\overrightarrow{a}||}$  and the unit vector perpendicular to that  $\overrightarrow{a_\perp}$ as a basis. In this case our interpretation of their similarity shifted to:

\begin{equation} \label{eq:projection_a} \tag{1}
\overrightarrow{a}.\overrightarrow{b} = b_a.a_a + b_{a_\perp}.a_{a_\perp} = b_a.a_a = |\overrightarrow{b}|\cos{\theta}.|\overrightarrow{a}|\cos{0} = |\overrightarrow{a}| .|\overrightarrow{b}|.\cos{\theta}
\end{equation}

In this case, we reward the similarity between $\overrightarrow{a}$ and $\overrightarrow{b}$ with respect to $\overrightarrow{a}$ and its perpendicular vector $\overrightarrow{a_\perp}$.
The underlying reason is the same when we dot product we are rewarding the similarity, it is just the matter of the rotation of the axes. When we rotate two vectors together, their lengths and the angle between them stays the same.

Let's consider a daily example. Suppose you are swimming in the sea and let's think about these three different directional cases:

1. *Swimming against the waves:* You have to put extra effort to keep your speed at constant, waves will have a negative effect on your speed
2. *Swimming in the direction of the waves:* You will gain extra speed boost if you swim this direction
3. *Swimming in the perpendicular direction to the waves:* It will only change your direction but won't have an effect on your speed.

So, the waves will boost your speed by the projection on the shared direction and we can infer that  the effect of the waves on your speed is proportional to the dot product between the speed of the waves and yours.

\begin{equation} \label{eq:projection_b} \tag{2}
\overrightarrow{a}.\overrightarrow{b} = a_b.b_b + a_{b_\perp}.b_{b_\perp} = a_b.b_b = \left | \overrightarrow{a} \right | . \left | \overrightarrow{b} \right |.\cos{\theta}
\end{equation}

So by (\ref{eq:projection_a}) and (\ref{eq:projection_b}) we have:

\begin{aligned}
 a_b  =  \left | \overrightarrow{a} \right | . \cos\theta \quad \text{and} \quad  b_a  =  \left| \overrightarrow{b} \right| . \cos\theta
\end{aligned}

We can combine our derivations in the following formula:

\begin{aligned}
\overrightarrow{a}.\overrightarrow{b} =a_x.b_x + a_y.b_y = a_b .\left| \overrightarrow{b}   \right| =  b_a . \left| \overrightarrow{a}  \right| = \left| \overrightarrow{a}  \right|.\left| \overrightarrow{b}  \right|.\cos{\theta}
\end{aligned}

### Cosine, tell us how similar we are!
It is no surprise to see the cosine function when we are talking about similarities. Cosine is directly related to the similarity of two vectors and used as a similarity measure in the inner product spaces. Note that cosine similarity is a judgment of the orientation and not the magnitude.

1. When two vectors are in the same direction, the angle between them is $0^{\circ}$
     and $\cos(0^{\circ}) = +1$
2. When they are perpendicular to each other and do not share any single component, the angle between them is $90^{\circ}$ and $\cos(90^{\circ}) = 0$.
3. When two vectors are in the opposite direction, the angle between them is $180^{\circ}$ and $\cos(180^{\circ}) = -1$.

### Correlation is just a spicy Cosine
Now we will investigate the relationship between two parameters which we will call $X$ and $Y$. Relationship in this context is not a cause-and-effect type of relationship, but how a change in one parameter is accompanied by the other one. It is also important to note that the correlation we consider here is Pearson’s Correlation and it only captures the linear relationship between the parameters. That’s why it is possible to have a zero Pearson correlation between two non-linearly correlated parameters.

Suppose we are running a supermarket and know that apple sales will be higher in the spring. To make storage and sales planning, we want to know if there is any relationship between lemon and apple sales so that for example, if  there is a positive correlation, we can infer lemon sales will increase in the spring too and we have to increase our storage capacity.

You can think $X$ as the number of apples and $Y$ as the number of lemons sold in the supermarket. $X$ and $Y$ will consist of several measurements corresponding to different sales days. So, if the data is collected for ten  days, $X$ and $Y$ will consist of ten values, each corresponding to a sales of a different day. We want to know if there is a relation between apple and lemon sales.

Let's think $X$ and $Y$ as vectors, $X= [x_1, x_2, \dots, x_n ]^T$ and $Y = [y_1, y_2, \dots, y_n ]^T$, and each components of $X$ and $Y$ are corresponding to the consecutive measurements of two different things that we would like to analyze their relationship.

We can expect the values of $X$ and $Y$ to have different scales. In order to compare $X$ and $Y$, we need to scale them by subtracting their mean, element-wise, so that they will be zero-centered. From a geometric point of view, in this mean-centering procedure, we re-position the coordinate system to make the origins coincide with the average point.

This scaling will provide us the ability to  compare the parameters with respect to their direction in each dimension. In other words, we can expect components of $X$ and $Y$ to share the same signs when they behave similarly. These values and how they relate to the relationship between parameters can remind us the cosine function as we are investigating the degree of similarity between two vectors.

Hmm, we are investigating the relationship between two vectors. Isn’t the dot product boost the similar components of the vectors? It can be helpful in our case.

From the dot product we know that:
\begin{equation*}
\overrightarrow{X}.\overrightarrow{Y} =\left | \overrightarrow{X}\right |. \left |\overrightarrow{Y} \right|.\cos{\alpha}
\end{equation*}

Element-wise subtracting their means from the parameters will give us:

\begin{equation*}
(\overrightarrow{X} - \overrightarrow{\mu_X}).(\overrightarrow{Y} - \overrightarrow{\mu_Y}) = \left | (\overrightarrow{X} - \overrightarrow{\mu_X}) \right |.  \left | (\overrightarrow{Y} - \overrightarrow{\mu_Y}) \right |.\cos{\theta}
\end{equation*}

Finally, we have the cosine of the angle between $X_C$ and $Y_C$ as correlation

\begin{equation*}
Corr(X_C,Y_C) =\cos{\theta} = \frac{(\overrightarrow{X} - \overrightarrow{\mu_X}).(\overrightarrow{Y} - \overrightarrow{\mu_Y})}{\left | (\overrightarrow{X} - \overrightarrow{\mu_X}) \right |. \left | (\overrightarrow{Y} - \overrightarrow{\mu_Y}) \right |}
\end{equation*}

That’s it! Correlation is just the cosine of the angle between the mean-centered random variables that we are investigating.
If the random variables are perpendicular to each other, they have nothing in common and the correlation between them would be
$\cos(90^{\circ}) = 0$. If they are in the same direction, they are highly correlated and it will give us $\cos(0^{\circ}) = + 1$. If they point to the opposite directions, they are highly and negatively correlated as $\cos(180^{\circ}) = -1.$


>>> Work in Progress (Following are the lecture notes of Prof Fei-Fei Li/Prof Justin Johnson/Prof Serena Yeung - CS231n - Stanford. This is my interpretation of their excellent teaching and I take full responsibility of any misinterpretation or misinformation provided herein.)

### Lecture 3: Loss Function and Optimization

#### Outline:
- Loss functions
  - A loss function quantifies the unhappiness with the scores across the training data
    - a function that takes in a W and tells us how bad quantitatively is that W
    - minimize the loss on training example
    - different types of loss
- Optimization
  - Come up with a way of efficient procedure to calculate W
    - efficiently come up with the procedure of searching through the space of all possible Ws and come up with what is the correct value of W that is the least bad

#### Loss function
- Given a dataset $\{(x_{i}, y_{i})\}_{i=1}^{N}$, where $x_{i}$ is image and $y_{i}$ is (integer) label
- Loss over the dataset is sum of loss over examples:
> $L(W) = \frac{1}{N}\sum\limits_{i}L_{i}(f(x_{i},W),y_{i}) + \lambda R(W)$
  - where 1st term is the data loss
  - 2nd term is the regularization correction - making the model simple
- binary SVM - has 2 classes - each example will be classified as positive or negative example
- multinomial SVM - handle multiple classes

#### Multiclass SVM loss
- Given a dataset $\{(x_{i}, y_{i})\}_{i=1}^{N}$, where $x_{i}$ is image and $y_{i}$ is (integer) label
- and scores vector $s = f(x_{i}, W)$
  - predicted scores that are coming from the classifier
  - $y_{i}$ is the ground truth label
  - $s_{y_{i}}$ denotes score of the true class for the ith example in training set
  - $s_{1}$ and $s_{2}$ will be cat and dog score respectively

- SVM loss has the form - **Hinge Loss**:

> \begin{equation}\\
\begin{aligned}\\
  L_{i} &= \sum\limits_{j \neq y_{i}}
    \begin{cases}
      0 & \text{if $s_{y_{i}} \geq s_{j} + 1$}\\
      s_{j} - s_{y_{i}} + 1 & \text{otherwise}\\
    \end{cases}\\
    &= \sum\limits_{j \neq y_{i}} max(0, s_{j} - s_{y_{i}} + 1)
\end{aligned}\\
\end{equation}\\

$\tiny{\text{YouTube-Stanford-CS231n-Justin Johnson}}$

- If the true score is high, that is good. Otherwise, we will have to incur some loss and that would be bad.

$\tiny{\text{YouTube-Stanford-CS231n-Justin Johnson}}$

$\tiny{\text{YouTube-Stanford-CS231n-Justin Johnson}}$

$\tiny{\text{YouTube-Stanford-CS231n-Justin Johnson}}$

- Why +1?
  - We care about the relative scores

#### Regularization - 2nd term
> $L(W) = \frac{1}{N}\sum\limits_{i}L_{i}(f(x_{i},W),y_{i}) + \lambda R(W)$
> where $\lambda$ is the regularization strength (hyperparameter)

- Types of regularization:
  - L2 regularization - weight decay - Euclidean norm or squared norm - penalize the Euclidean norm of this weight vector
    > $R(W) = \sum_{k}\sum_{l}W^{2}_{k,l}$
  - L1 regularization - nice property of encouraging sparsity in matrix W
    > $R(W) = \sum_{k}\sum_{l}|W_{k,l}|$
  - Elastic net (L1 + L2) regularization - combination of L1 and L2
    > $R(W) = \sum_{k}\sum_{l}\beta W^{2}_{k,l} + |W_{k,l}|$
  - Max norm regularization - penalizes the max norm rather than L1 and L2 norm
  - Dropout regularization - specific to deep learning
  - Fancier regularization: Batch normalization, stochastic depth

- Goal of regularization term is that it penalizes the complexity of the model rather than explicitly trying to fit the training data

#### Softmax Classifier (Multinomial Logistic Regression)

- Multiclass SVM
  - there was no interpretation of loss function
  - the model f spits out scores for the classes, which didn't actually had much interpretation
  - all we cared about was the score of correct class must be greater than score of incorrect class
- Multinomial Logistic Regression
  - in this case, the scores will have meaning
  > Softmax function $P(Y=k|X=x_{i}) = \frac{e^{s}k}{\sum_{j}e^{s_{j}}}$
  > where scores $s = f(x_{i}; W)$ = unnormalized log probabilities of the classes
  - the probability of softmax function sum to 1
  - To maximize the log likelihood, or (for a loss function) to minimize the negative log likelihood of the correct class:
  > $L_{i} = -$log $P(Y=y_{i}|X=x_{i}) $
  - more weight (i.e., probability of 1) should be on the cat and 0 probability for all other classes
  - computed probability distribution coming out of the softmax function should match this target probability distribution that has all the mass on the correct class
    - use KL divergence
    - maximum likelihood estimate
  - Goal is the probability of true class is high and as close to 1
- Loss function will be the -log of the probability of true class
  > $L_{i} = -$log $\frac{e^{s_{y_{i}}}}{\sum_{j}e^{s_{j}}} $
- Calculation steps:
  - Calculate unnormalized log probabilities, as above
  - calculate exponent of it(unnormalized probabilities)
  - calculate normalized value (probabilities)
  - calculate negative log (softmax loss function value) (or multinomial logistic regression)

#### Optimization

- find bottom of the valley
- use types of iterative method
- types
  - random search
    - depends on luck
  - follow the slope.
    - use local geometry, which way will take me little bit down
    - gradient is the vector of (partial derivatives) along each dimension
    - slope in any direction is the dot product of the direction with the gradient
    - direction of steepest descent is negative gradient
    - use finite differences
      - adv
        - easy to write
      - disadv
        - approximate
        - can be very slow if size is large
        - in practice, it is never used
    - instead compute analytic gradient
      - calculate dW in one step instead of looping over iteratively
        - adv
          - exact, fast
        - disadv
          - error prone
    - in practice, always use analytic gradient, but check implementation with numerical gradient - gradient check

$\tiny{\text{YouTube-Stanford-CS231n-Justin Johnson}}$

$\tiny{\text{YouTube-Stanford-CS231n-Justin Johnson}}$

  - gradient descent
    - most used
    - initialize W random
    - compute loss and gradient
    - update the weights in opposite of the gradient direction
      - gradient points to the direction of greatest increase
      - minus gradient points in the direction of greatest decrease
      - take small step in the direction of minus gradient
      - repeat till it converges
    - step size or learning rate is a hyperparameter
      - tells us how far we step in the direction of gradient
      - step size is the first hyperparameter we check
      - model size and regularization can be dealt later, but step size should be the primary focus

#### Stochastic Gradient Descent (SGD)
> $L(W) = \frac{1}{N}\sum\limits_{i}L_{i}(f(x_{i},y_{i},W) + \lambda R(W)$
> $\nabla_{W}L(W) = \frac{1}{N}\sum\limits_{i}\nabla_{W} L_{i}(f(x_{i},y_{i},W) + \lambda\nabla_{W} R(W)$
- Vanilla Minibatch Gradient Descent
  - minibatch of size 32/64/128

#### Image features
- Instead of feeding raw pixels into linear classifiers doesnot work too well
- Prior to deep neural network popularity, two stage approach was used
- first, take your image, compute various feature representations
- then concatenate these feature vectors to give some feature representation of image
- trick is to use right feature transform for the problem statement
- example
  - color histogram
    - count how many pixels fall into each bucket
    - tells us what type of color exist in image
  - histogram of oriented gradients (HoG)
    - dominant edge direction of each pixel
    - compute histogram over these different edge orientation in bucket
    - tells us what type of edge exist in image
    - was used for object recognition in the past
  - bag of words (comes from NLP)
    - in NLP, number of words in a paragraph are counted
    - apply same concept in images
    - no straightforward analogy of words and images
    - create your own version of vocabulary of visual words
    - get sample

- ConvNets

```python

```


##  3D Mobius transformation of  meshes. 4D Rotations

### 3D Mobius transformations

$\newcommand{\R}{\mathbb{R}}$
$\newcommand{\C}{\mathbb{C}}$
$\newcommand{\th}{\theta}$
$\newcommand{\lam}{\lambda}$
$\newcommand{\ovl}{\overline}$
$\newcommand{\ds}{\displaystyle}$
$\newcommand{\barr}{\begin{array}}$
$\newcommand{\earr}{\end{array}}$
$\newcommand{\bea}{\begin{eqnarray}}$
$\newcommand{\eea}{\end{eqnarray}}$
$\newcommand{\beq}{\begin{equation}}$
$\newcommand{\eeq}{\end{equation}}$

The classical [Mobius transformation](https://en.wikipedia.org/wiki/M%C3%B6bius_transformation) is  known as acting on the extended complex plane, identified to the extended space, $\hat{\mathbb{R}}^2=\mathbb{R}^2\cup \{\infty\}$. But is there a Mobius transformation that acts on the extended space $\hat{\mathbb{R}}^n$, $n>2$? The answer is positive, and the definition and construction of such a transformation is presented in detail in the excellent book, by Alan F. Beardon -  *The geometry of discrete groups*, Springer, 1995 (chapter 3, Mobius transformation on $\mathbb{R}^n$).
The presentation from this book, and an article from Notices of AMS, [https://www.ams.org/notices/200810/tx081001226p.pdf](https://www.ams.org/notices/200810/tx081001226p.pdf), suggested how we can apply a Mobius transformation to a 3d mesh.

By these references   a  Möbius transformation  on the 3d extended space can be defined as the transformation that maps this  space through  the inverse of the stereographic projection onto an admissible sphere $S(a, R) =\{(x,y, z, u)\in\mathbb{R}^4\:|\: (x-a_1)^2+(y-a_2)^2+(z-a_3)^2+(u-a_u)^2=R^2\}$  in the space $\mathbb{R^4}$ (i.e. a sphere for which the  north pole $N(a_1, a_2, a_3, a_4+R)$, lies in the upper half-space $H=\{(x, y, z, u)\in\mathbb{R}^4\:|\: u>0\}$,
 followed by a rigid motion (rotation+translation) of the sphere in $\mathbb{R}^4$,  which maps it to another admissible sphere, and finally mapping the last sphere, back to the space $\mathbb{R}^3$,  by the stereographic projection.
 Following Chapter 3 from Beardon's book and this method of construction, valid for Mobius transformation in any dimension,  we can illustrate visually, how a Mobius transformation defined on the 3d space acts on a 3d mesh.

In Computer Graphics a 3d mesh is a numerical representation of a 3d model (surface or volume). It consists in two arrays,
the array of vertices and the array of polygons (usually quadrangles and triangles).
The array of vertices (of shape (m, 3)) contains on rows their x, y, z coordinates, while the array of quadrangles/triangles is
and array of integers, of shape (n, 4) or (n, 3), where each row,  contains the indices in the array of vertices, of the points that define the corresponding polygon.

This repository is dedicated to  illustrating how a 3d mesh is deformed  by a Mobius transformation that acts as follows:
  - map the mesh vertices to the 3-dimesnional unit sphere, $S^3$,  in the 4d space, by the inverse of the stereographic projection.
  - rotate   the spherical mesh vertices, by a 4d rotation, which is known as  leaving the sphere invariant (maps points on sphere onto points on sphere);
  - eventually translate the initial unit sphere to an admissible unit (of radius =1) sphere in $\R^4$;
  -  map back to the 3d space the rotated and translated sphere vertices, through the stereographic projection from the corresponding sphere  to $\hat{\mathbb{R}}^3$.

  The projected points as vertices and the initial mesh polygons define a Mobius transformed 3d mesh.

A Mobius transformation is a conformal transformation (i.e. a transformation that preserves the angles, but not the lengths), because it is a composition of conformal mappings (the stereographic projection and its inverse) to a rigid transformation.

The stereographic projection from an admissible  unit sphere, of center $C(c_1, c_2, c_3, c_4)$, $S^3=\{(x, y, z, u)\in\mathbb{R}^4\:|\: (x-c_1)^2+(y-c_2)^2+(z-c_3)^2+(u-c_4)^2=1\}$,
to the euclidean space $\mathbb{R}^3$, is defined similarly to the stereographic projection from the sphere $S^2$ to $\mathbb{R}^2$.
Namely, it associates to each point, $P(x,y,z, u)$, on the sphere, the point of intersection of the line through the north pole,
$N(c_1, c_2, c_3, c_4+1)$, and P, to the space $\mathbb{R}^3$, identified with  the subspace of $\mathbb{R}^4$, defined by $\{(x,y,z,u)\:|\: u=0\}$. Similarly one defines the inverse of  the stereographic projection.

The functions `stereo_aS3_R3()`, `inv_stereo_aS3()`, from `mobius3d`, implement the stereographic projection and its inverse.

An important step in defining a 3d Mobius transformation is the 4d rotation. Below we present in details the definition and
the orthogonal decomposition of a 4d rotation, as a key point in defining  interesting Mobius 3d transformations of  3d meshes.

### 4D rotation - orthogonal decomposition

A 4d rotation, $Rot:\R^4\to\R^4$, is represented by an orientation preserving orthogonal matrix, Q, i.e. $Q^TQ=QQ^T = Id_4$, and determinant(Q)=1.
From orthogonality property it follows that the columns of such a matrix are real-orthonormal vectors.

But the orthogonal matrices are particular normal matrices. That's why we are  pointing out, below, the basic properties of normal matrices.

Let $\C^n$  be the n-dimensional complex space, with the hermitian inner product defined by:
$$<v, w> =\sum_{k=1}^n \ovl{z}_kz'_k = v^*w, \forall\:\: v=(z_1, z_2,  \ldots, z_n)^T, w=(z'_1, z'_2, \ldots, z'_n)^T \in \C^n$$
($v^*$ denotes the conjugate transpose of v). This inner product induces the standard inner product on the space $\R^n$.

A complex $n\times n$ -  matrix, $A$,  is a normal matrix if $A^*A=AA^*$, where $A^*=\ovl{A}^T$, denotes the conjugate transpose of $A$.
A unitary matrix, $U$, i.e. a matrix with the property $U^*U=UU^*=Id$, and in particular an  orthogonal matrix, $Q$, is a normal matrix.

Spectral properties of normal matrices:

 1. The algebraic multiplicity of an eigenvalue coincides with its geometric multiplicity.
 2. To distinct eigenvalues, $\lam\neq \mu$,  correspond  orthogonal eigenvectors, $v, w$, i.e.  $v^*w=0$.

These two properties ensure that for any normal matrix, $A$, there is an orthonormal basis in $\C^n$, consisting in  eigenvectors of $A$, and the matrix $A$ can be decomposed as $A= V D V^*$, where  $D$ is the diagonal matrix of its eigenvalues, and $V$ is the unitary matrix with the corresponding orthonormal eigenvectors on its columns. Hence a normal matrix is diagonalizable by a unitary matrix, and $A=V D V^*$ is called unitary decomposition.

 Being an orientation preserving orthogonal matrix, a rotation  is a rigid transformation, i.e. $||Qv||=||v||$ (it preserves lengths and angles). In particular if $v$ is an eigenvector, $Qv=\lam v$, we get that  $||v||=||Qv|=||\lam v||= |\lam|||v||$, i.e. $|\lam|=1$. Hence the eigenvalues of a rotation matrix lie on the unit circle (they are of the form $\lam, \ovl{\lam}=a\pm ib$, $a^2+b^2=1$ or equivalently, $\lam, \ovl{\lam}=e^{\pm it}=\cos(t)+i\sin(t)$, for some $t\in (-\pi, \pi]$.
If $t=0$ we get  the eigenvalue 1 which can be   a double eigenvalue or an eigenvalue multiple of order 4, while for $t=\pi$, -1 is a double eigenvalue or  multiple of order 4.

From the spectral properties of normal matrices we get that for a 4d rotation, $Q$,  we have the  unitary decomposition:

$$Q= V D V^*$$
where $D$ is the diagonal matrix of its eigenvalues $\lam_1, \lam_2\lam_3\lam_4\in\C$, and $V$ has as columns the orthonormal   eigenvectors, with respect to the hermitian inner product in $\C^4$), corresponding to these eigenvalues.

The numpy function `np.linalg.eig` identifies orthogonal matrices and returns V, as a unitary matrix.

A method to generate an  example of 4d rotation is to get the $QR$ decomposition of a random  real $4\times 4$-matrix:

```python
import numpy as np

def get_rotation(A, det_tol=1.e-10):
    #returns the Q matrix from the QR-decomposition of the given matrix, A, and its eigenvals and eigenvect
    if A.ndim !=2 or A.shape[0] != A.shape[1] != 4:
        raise ValueError('A must have the shape (4,4)')
    Q, _ = np.linalg.qr(A)
    if abs(np.linalg.det(Q)+1) < det_tol: # if det(Q) is close to  -1 revert two columns in Q
        Q = np.array([Q[:, 1], Q[:, 0], Q[:, 2], Q[:, 3]]).T
    eigvalues, V = np.linalg.eig(Q)
    return Q, eigvalues, V
```

```python
A = np.array([[ 2, -1,  3, -1],
              [-3,  0,  2,  0],
              [ 1,  0,  0,  2],
              [ 0,  1,  3, -1]])
Q, lam, V = get_rotation(A)
```

Check orthogonality:

```python
np.linalg.norm(Q.T @ Q-np.eye(4)) #Frobenius norm of the matrix difference Q^TQ-Id_4
```

    5.177377601847236e-16

```python
np.linalg.det(Q) > 0
```

    True

```python
lam  #eiegenvalues of the matrix Q
```

    array([-0.84420342+0.53602294j, -0.84420342-0.53602294j,
            0.9288035 +0.37057261j,  0.9288035 -0.37057261j])

Hence our othogonal matrix has two distinct pairs of complex conjugate eigenvalues. The corresponding orthonormal vectors
are the columns of the unitary matrix V:

Check if  $V$ is a unitary matrix:

```python
P = np.matrix(V).getH()  @ V  #  np.matrix(V) converts V to a numpy.matrix W, W.getH() is the conjugate transpose of W
np.linalg.norm (P- np.eye(4))  #  distance to Id_4
```

    2.9249508704071502e-15

In order to point out how a 4d rotation,  $Q$, acts on points from  ℝ4, in particular on the sphere from  $\R^4$, we are converting the unitary decomposition of  𝑄  into an orthogonal decomposition,  $Q=UR_kU^T$ , where  $U$  is an orthogonal matrix,
and  $R_k$, $k=1, 2$  is a particular rotation matrix derived from the complex eigenvalues the matrix $Q$ .

The diagonal matrix $D$, from the unitary decomposition $Q=V D V^*$, is  the representative matrix of the rotation with respect to the basis $B'=(v_1, v_2, v, v_4)$, consisting in the columns of $V$

**a)** If $\lam_1, \lam_2=1, and \lam_3, \lam_4 = e^{\pm it}$, for some $t\in (-\pi, \pi]\setminus\{0\})$,
then from the basis $\mathcal{B}'$ we extract an orthonormal basis in $\R^4$, $\mathcal{B}''=(u_1, u_2, u_3, u_4)$, as follows:
$$u_1=v_1, u_2=v_2, u_3 =\ds\frac{1}{\sqrt{2}} (v_3 +v_4), u_4 =\ds\frac{1}{i\sqrt{2}} (v_3 -v_4)$$                              where $v_4=\ovl{v}_3$ (to complex conjugate eigenvalues correspond complex conjugate eigenvectors).

Note that although the complex conjugate vectors, $v_3, v_4$,  are unit vectors with respect to the norm in $\C^4$, their real part $(v_3+v_4)/2$ and imaginary part, $(v_3-v_4)/2i$, do not have unit norm. The factors $\ds\frac{1}{\sqrt{2}}$, $\ds\frac{1}{i\sqrt{2}}$ ensure that the corresponding real vectors are unit vectors.

It is a simple excercise to show that  $\mathcal{B}''$ is an orthonormal basis (the orthogonality of the first two vectors is obvious, while for example, $<u_1, u_3> = <v_1, \ds\frac{1}{\sqrt{2}} (v_3 +v_4)>=0$).

The representative matrix, $R_1$,  of the rotation  with respect to the basis $\mathcal{B}''$ is derived from the matrix $D$ that represents the rotation with    respect to the basis $\mathcal{B}'$, and the unitary matrix
$$W = \left(\barr{cccc} 1&0&0&0\\
                        0&1&0&0\\
                        0&0& \frac{1}{\sqrt{2}}&\frac{1}{i\sqrt{2}}\\
                        0&0& \frac{1}{\sqrt{2}}&-\frac{1}{i\sqrt{2}}\earr\right)$$
whose columns are the  coordinates of the vectors $u_1, u_2, u_3, u_4$ with respect to the basis $\mathcal{B}'$:
$$R_1  = W \left(\barr{cccc} 1&0&0&0\\
0&1&0&0\\
0&0&\cos(t)+i\sin(t)&0\\
0&0&0&\cos(t)-i\sin(t)\earr\right) W^*= \left(\barr{rrrr}1&0&0&0\\0&1&0&0\\
                   0&0&\cos(t)&\sin(t)\\0&0&-\sin(t)&\cos(t)\earr\right)$$

 As a consequence
 the orthogonal decomposition of the rotation matrix $Q$,  is in this case,  $Q = U R_1 U^T$,
where $U$ has on its columns the coordinates of the vectors $u_1, u_2, u_3, u_4$ with respect to the standard basis in $\R^4$.

**b)**  Analogously we get  the matrix $R_{-1}$ corresponding to the case  $\lam_1, \lam_2 = -1$ and$\lam_4=\ovl{\lam}_3$, by replacing  the diagonal block $I_2$ in $R_1$ by  $-I_2$.

**c)** If the eigenvalues of the matrix $Q$ are $\lam_2=\ovl{\lam}_1$, $\lam_4=\ovl{\lam}_3$, i.e. $\lam_1,\lam_2=\cos(s)\pm i\sin(s)$, $\lam_3, \lam_4 = \cos(t)\pm i \sin(t)$, $s, t \in (-\pi, \pi)\setminus\{0\}$, and the corresponding orthonormal basis of eigenvectors is  $\mathcal{B}'=(v_1, \ovl{v}_1, v_2, \ovl{v}_2)$, then define the orthonormal real vectors:
$$u_1 = \ds\frac{1}{\sqrt{2}} (v_1 +\ovl{v}_1), u_2 = \ds\frac{1}{i\sqrt{2}} (v_1 -\ovl{v}_1),
u_3=\ds\frac{1}{\sqrt{2}} (v_2 +\ovl{v}_2), u_4 = \ds\frac{1}{i\sqrt{2}} (v_2 -\ovl{v}_2)$$
that define a basis $\mathcal{B}''$ in $\R^4\subset\C^4$.

The matrix of the initial rotation with respect to this basis, deduced similarly as in the previous case, is of the form:
$$R_2=\left(\barr{rrrr}\cos(s)&\sin(s)&0&0\\-\sin(s)&\cos(s)&0&0\\0&0&\cos(t)&\sin(t)\\0&0&-\sin(t)&\cos(t)\earr\right)$$
and the corresponding orthogonal decomposition of the rotation $Q$ is:

$$Q = U R_2 U^T,\:\:    U=[u_1|u_2|u_3|u_4]$$

Hence the derivation of the orthogonal decomposition from the unitary decomposition of a rotation matrix exploited the property that the $\R^4$-vectors of the pairs  $(real(v_i), real(v_j$, $(real(v_i), imag(v_j))$ associated to  complex eigenvectors  $v_i, v_j$, respectively
$(real(v_i), imag(v_i)$, $i, j  =\ovl{1,4,  i\neq j$, are orthogonal vectors.

**d)** If in **a)** and **b)**, t=0, we get  $Q =  V Id_4 V^*=Id_4$, respectively  $Q=-Id_4$.

Let us point out how a 4D rotation that contains $R_1$, respectively $R_2$, as the central matrix in its orthogonal decomposition acts on different subspaces of $\R^4$.

- A 4d rotation matrix having the orthogonal decomposition $Q=UR_1U^T$ has the property that $Q(\alpha_1 u_1+\alpha_2u_2)=\alpha_1 u_1+\alpha_2u_2$, $\alpha_1, \alpha_2\in\R$,
i.e.  the vectors of the subspace of $\R^4$ generated by the first two columns in the matrix $U$, are fixed by the rotation $Q$.

On the other hand the vectors in the subspace generated by the last two columns (i.e. the vectors in the orthogonal complement of the former subspace) are rotated by an angle of $t$ radians.  Such a rotation matrix defines a rotation about the plane
defined by the origin $O$ and the vectors $u_1, u_2$. The fixed plane and the invariant plane (the rotated plane) are orthogonal.
Hence, unlike the 3D rotation that occurs about an axis, in 4D we have rotations about a plane.

- A 4d rotation of the form $Q=UR_2U^T$ is called double rotation, because it rotates both the vectors in the subspace generated by the first two columns in $U$
and the vectors in the subspace generated by the last two columns of $U$.  If  the angles of rotation are equal, i.e. s=t, then Q is called  isoclinic rotation.

Notice that from an  orthogonal decomposition of the form $Q=U R_2U^T$, we can recover any particular orthogonal decomposition, by setting suitable values for $s$ and $t$.

- for $s=t=0$ we get $Q=Id_4$;
- for $s=t=\pi$, $Q=-I_4$;
- for $s=0$, $t\in (-\pi, \pi]\setminus\{0\}$, $Q=U R_1U^T$;
- for $s=\pi$, $t\in (-\pi, \pi]\setminus\{0\}$, $Q=U R_{-1}U^T$, where  $R_{-1}$ is gotten from $R_1$, replacing the diagonal block $Id_2$, by $-Id_2$;
- for $s, t \in (-\pi, \pi]\setminus\{0\}$, $Q=UR_2U^T$;

That's why in `mobius3d` we defined a function `setup_rotation()`, which from particular values for s and t, and an orthogonal matrix $U$, returns the corresponding rotation matrix $Q=U R_2(s,t)U^T$.

```python

```