```python
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei')
plt.rcParams['axes.grid'] = False

from IPython.display import SVG

def show_image(filename, figsize=None):
    if figsize:
        plt.figure(figsize=figsize)

    plt.imshow(plt.imread(filename))
```

评价函数模块 `_criterion.*` 详解
=================================

### 0. 大纲

本文会按照类的继承关系，由上到下介绍各个类的成员及实现细节，尽量说得详细点。

```python
SVG("./res/uml/Model___criterion_1.svg")
```

### 1. Criterion, 接口类
Criterion 作为接口之用，相当于 Java 中的抽象类。其接口定义在 `_criterion.pxd` 文件，实现在 `_criterion.pyx` 中。

我们对 Criterion 分为两部份来说明：内部成员，和函数方法。

##### 1.0 内部成员
为了计算和存储效率，sklearn 做了些针对性的数据设计。当然，其实思路挺直觉的，我们带着具体的成员变量来说明。

+ 数据分条 (stride)

  二维数组会用一维数组（其实是个连续的缓存）加上偏移量来实现，我以前在数字图像处理中常看到。

  ```Python
     26     # Internal structures
     27     cdef DOUBLE_t* y                     # Values of y
     28     cdef SIZE_t y_stride                 # Stride in y (since n_outputs >= 1)
     29     cdef DOUBLE_t* sample_weight         # Sample weights
  ```

  分类的标签或者回归的值（$y$）可能是个向量，所以需要对 $y$ 值分条，每条的长度自然是 `y_stride`。
  `sample_weight` 是各个样本的权重值，不用多讲。

+ 使用索引

  决策树分割节点时会涉及到移动样本，因为 $x$ 通常维数较多，开销较大。对于这类问题，一个常见的做法就是交换索引，而不变动值。具体到数组操组，这个索引就是下标值。

  ```Python
  31     cdef SIZE_t* samples                 # Sample indices in X, y
  32     cdef SIZE_t start                    # samples[start:pos] are the samples in the left node
  33     cdef SIZE_t pos                      # samples[pos:end] are the samples in the right node
  34     cdef SIZE_t end
  ```

  这里样本值的索引区间其实是前闭后开，即 samples[start:pos) 和 samples[pos:end)。

+ 加权处理

  加权的样本数目统计
  ```Python
  36     cdef SIZE_t n_outputs                # Number of outputs
  37     cdef SIZE_t n_node_samples           # Number of samples in the node (end-start)
  38     cdef double weighted_n_samples       # Weighted number of samples (in total)
  39     cdef double weighted_n_node_samples  # Weighted number of samples in the node
  40     cdef double weighted_n_left          # Weighted number of samples in the left node
  41     cdef double weighted_n_right         # Weighted number of samples in the right node
  ```

  加权的 $y$ 值累加和（若是分类，则是对各个类别单独做累加）
  ```Python
  43     cdef double* sum_total          # For classification criteria, the sum of the
  44                                     # weighted count of each label. For regression,
  45                                     # the sum of w*y. sum_total[k] is equal to
  46                                     # sum_{i=start}^{end-1} w[samples[i]]*y[samples[i], k],
  47                                     # where k is output index.
  48     cdef double* sum_left           # Same as above, but for the left side of the split
  49     cdef double* sum_right          # same as above, but for the right side of the split
  ```

##### 1.1 函数方法
+ 纯接口

```Python
 54     # Methods
 55     cdef void init(self, DOUBLE_t* y, SIZE_t y_stride, DOUBLE_t* sample_weight,
 56                    double weighted_n_samples, SIZE_t* samples, SIZE_t start,
 57                    SIZE_t end) nogil
 58     cdef void reset(self) nogil
 59     cdef void reverse_reset(self) nogil
 60     cdef void update(self, SIZE_t new_pos) nogil
 61     cdef double node_impurity(self) nogil
 62     cdef void children_impurity(self, double* impurity_left,
 63                                 double* impurity_right) nogil
 64     cdef void node_value(self, double* dest) nogil
```

+ 计算不纯度提升量

```Python
 65     cdef double impurity_improvement(self, double impurity) nogil
 66     cdef double proxy_impurity_improvement(self) nogil
```

`proxy_impurity_improvement`：

\begin{align}
    & \frac{N_t}{N} \left ( \text{ impurity } - \frac{N_{t,R}}{N_t} \text{ right_impurity } - \frac{N_{t,L}}{N_t} \text{ left_impurity } \right ) \\
    &= \frac{N_t}{N} \text{ impurity } - \frac{N_{t,R}}{N} \text{ right_impurity } - \frac{N_{t,L}}{N} \text{ left_impurity } \\
    &= \frac{N_t}{N} \text{ impurity } - \frac{1}{N} \underbrace{ \left ( N_{t,R} \text{ right_impurity } - N_{t,L} \text{ left_impurity } \right ) }_{\text{impurity_improvement}} \\
\end{align}

对一个节点，去除公式中的常量，弄了个 `impurity_improvement` 来减少计算量。

### 2. 分类评价函数

#### 2.0 ClassificationCriterion
ClassificationCriterion 类是分类评估函数的父类，主要𫐄实现初始化和一些预处理函数。我们从上到下依序看代码。

##### 2.0.0 变量

```Python
 207 cdef class ClassificationCriterion(Criterion):
 208     """Abstract criterion for classification."""
 209
 210     cdef SIZE_t* n_classes
 211     cdef SIZE_t sum_stride
```

这里分类为了支持多输出（target）问题，用 `n_classes` 来统计每个输出的类别个数，用 `sum_stride` 来对齐不同输出的结果。用文字表述比较困难，下面的图说明更清晰点。

```python
show_image("./res/ClassificationCriterion.jpg", figsize=(8,15))
```

##### 2.0.1 `__cinit__`
接下来，我们看构造函数：

```Python
 213     def __cinit__(self, SIZE_t n_outputs,
 214                   np.ndarray[SIZE_t, ndim=1] n_classes):
 215         """Initialize attributes for this criterion.
 216 +--  7 lines: Parameters---------------------------------------------------------
 223         """
 224
 225 #+-- 20 lines: self.y = NULL------------------------------------------------------
 245
 246         safe_realloc(&self.n_classes, n_outputs)
 247 #+--  6 lines: cdef SIZE_t k = 0--------------------------------------------------
 253         for k in range(n_outputs):
 254             self.n_classes[k] = n_classes[k]
 255
 256             if n_classes[k] > sum_stride:
 257                 sum_stride = n_classes[k]
 258
 259         self.sum_stride = sum_stride
 260
 261         cdef SIZE_t n_elements = n_outputs * sum_stride
 262         self.sum_total = <double*> calloc(n_elements, sizeof(double))
 263 #+--  7 lines: self.sum_left = <double*> calloc(n_elements, sizeof(double))-------
```

主要是三个工作：

+ 初始化变量，设置默认值。
+ 复制 `n_classes`，同时记录最大值 `sum_stride` 用于对齐。
+ 为 `sum_*` 统计各类别信息的变量申请空间。

##### 2.0.2 `init`
应该是为了重复使用，这里单独写了初始化函数。

```Python
 282     cdef void init(self, DOUBLE_t* y, SIZE_t y_stride,
 283                    DOUBLE_t* sample_weight, double weighted_n_samples,
 284                    SIZE_t* samples, SIZE_t start, SIZE_t end) nogil:
 285         """Initialize the criterion at node samples[start:end] and
 286 +-- 19 lines: children samples[start:start] and samples[start:end].--------------
 305         """
 306
 307 #+-- 19 lines: self.y = y---------------------------------------------------------
 326
 327         for k in range(self.n_outputs):
 328             memset(sum_total + offset, 0, n_classes[k] * sizeof(double))
 329             offset += self.sum_stride
 330
 331         for p in range(start, end):
 332             i = samples[p]
 333 #+--  4 lines: w is originally set to be 1.0, meaning that if no sample weights---
 337                 w = sample_weight[i]
 338
 339             # Count weighted class frequency for each target
 340             for k in range(self.n_outputs):
 341                 c = <SIZE_t> y[i * y_stride + k]
 342                 sum_total[k * self.sum_stride + c] += w
 343
 344             self.weighted_n_node_samples += w
 345
 346         # Reset to pos=start
 347         self.reset()
```

主要流程是：

+ 307L-326L, 复制参数和设置默认值；
+ 327L-330L, 清零 `sum_total`。
  注意 328L 的优化， `memset` 最后一个参数是各输出的实际类别数，所以只会清实际用到的空间。
+ 331L-344L, 统计各类别的权重值 `sum_total` 和总权重值 `weighted_n_node_samples`。
  其中，340L 取到输出 $k$, 341L 取到类别 $c$，累加相应的总值 $\text{sum_total}[k, c]$。

##### 2.0.3 `reset`
reset 方法正如其名，做了重置工作，将节点位置 `pos` 恢复到最左边 `start` 位置，并更新统计信息。

```Python
 349     cdef void reset(self) nogil:                                                  350         """Reset the criterion at pos=start."""
 351
 352         self.pos = self.start
 353
 354         self.weighted_n_left = 0.0
 355         self.weighted_n_right = self.weighted_n_node_samples
 356
 357 #+--  7 lines: cdef double* sum_total = self.sum_total----------------------------
 364         for k in range(self.n_outputs):
 365             memset(sum_left, 0, n_classes[k] * sizeof(double))
 366             memcpy(sum_right, sum_total, n_classes[k] * sizeof(double))
 367
 368             sum_total += self.sum_stride
 369 #+---  2 lines: sum_left += self.sum_stride---------------------------------------
```

主要工作，清掉左节点的信息，覆盖右节点。

`reverse_reset` 和 `reset` 操作互反，不赘述。

##### 2.0.4 `update`
update 函数将节点移到指定位置，并更新统计信息。

```Python
 394     cdef void update(self, SIZE_t new_pos) nogil:
 395         """Updated statistics by moving samples[pos:new_pos] to the left child.
 396 +--  6 lines: Parameters---------------------------------------------------------
 402         """
 403         cdef DOUBLE_t* y = self.y
 404 #+-- 17 lines: cdef SIZE_t pos = self.pos-----------------------------------------
 421
 422 #+--  8 lines: Update statistics up to new_pos------------------------------------
 430         if (new_pos - pos) <= (end - new_pos):
 431             for p in range(pos, new_pos):
 432 #+--  8 lines: i = samples[p]-----------------------------------------------------
 440                     sum_left[label_index] += w
 441
 442                 self.weighted_n_left += w
 443
 444         else:
 445             self.reverse_reset()
 446
 447             for p in range(end - 1, new_pos - 1, -1):
 448 #+--  8 lines: i = samples[p]-----------------------------------------------------
 456                     sum_left[label_index] -= w
 457
 458                 self.weighted_n_left -= w
 459
 460         # Update right part statistics
 461         self.weighted_n_right = self.weighted_n_node_samples - self.weighted_n_le     ft
 462         for k in range(self.n_outputs):
 463             for c in range(n_classes[k]):
 464                 sum_right[c] = sum_total[c] - sum_left[c]
 465 #+--  4 lines: sum_right += self.sum_stride---------------------------------------
 469
 470         self.pos = new_pos
```

主要流程如下：

+ 初始化参数；
+ 422L-458L, 更新左节点的统计信息。
  这里的优化，是判断指定位置、当前节点和最右边三者的距离，遍历较小的距离。
+ 460L-465L, 更新右节点的统计信息：$\text{sum_right } = \text{ sum_total } - \text{ sum_left }$。

##### 2.0.5 `node_value`
作用是将 `sum_total` 的数据拷贝出来，不多说。

```Python
 479     cdef void node_value(self, double* dest) nogil:
 480         """Compute the node value of samples[start:end] and save it into dest.
 481 +--  5 lines: Parameters---------------------------------------------------------
 486         """
 487 #+--  4 lines: cdef double* sum_total = self.sum_total----------------------------
 491
 492         for k in range(self.n_outputs):
 493             memcpy(dest, sum_total, n_classes[k] * sizeof(double))
 494 #+--  2 lines: dest += self.sum_stride--------------------------------------------
```

##### 2.0.6 小结
我们可以看出，`ClassificationCriterion` 的主要工作是在更新 `sum_*` 变量。这三个变量的数据是相应节点中各个类的权重值，用这三个变量可以算出各类 $k$ 在节点 $i$ 的权重占比 $p_i^k$。而我们知道，常用的 gini、entropy 和 misclassification_rate 都是基于 $p_i^k$ 来表达。所以，将更新权重信息函数汇总，便于复用，正是此父类的存在意义。

有了 $p_k^i$ 权重占比，再下面的子类就纯粹是数学计算了，加上这几个式子都简单，代码非常清晰。需要注意，对多输出的问题，评价值是统一加合取平均。

#### 2.1 Entropy

 $\text{cross-entropy  } = -\sum_{k=0}^{K-1} \text{ count_k } \log( \text{ count_k }) $

```Python
 498 cdef class Entropy(ClassificationCriterion):
 499     """Cross Entropy impurity criterion.
 500 +-- 11 lines: This handles cases where the target is a classification taking valu
 511         cross-entropy = -\sum_{k=0}^{K-1} count_k log(count_k)
 512     """
 513
 514     cdef double node_impurity(self) nogil:
 515 #+-- 10 lines: """Evaluate the impurity of the current node, i.e. the impurity of-
 525         for k in range(self.n_outputs):
 526             for c in range(n_classes[k]):
 527                 count_k = sum_total[c]
 528                 if count_k > 0.0:
 529                     count_k /= self.weighted_n_node_samples
 530                     entropy -= count_k * log(count_k)
 531 #+--  3 lines: sum_total += self.sum_stride---------------------------------------
 534         return entropy / self.n_outputs
 535
 536     cdef void children_impurity(self, double* impurity_left,
 537                                 double* impurity_right) nogil:
 538 #+-- 22 lines: """Evaluate the impurity in children nodes-------------------------
 560         for k in range(self.n_outputs):
 561             for c in range(n_classes[k]):
 562                 count_k = sum_left[c]
 563                 if count_k > 0.0:
 564                     count_k /= self.weighted_n_left
 565                     entropy_left -= count_k * log(count_k)
 566
 567                 count_k = sum_right[c]
 568                 if count_k > 0.0:
 569                     count_k /= self.weighted_n_right
 570                     entropy_right -= count_k * log(count_k)
 571 #+--  3 lines: sum_left += self.sum_stride----------------------------------------
 574
 575         impurity_left[0] = entropy_left / self.n_outputs
 576         impurity_right[0] = entropy_right / self.n_outputs
```

#### 2.2 Gini
$ \text{ index } = 1 - \sum_{k=0}^{K-1} \text{ count_k}^2 $

```Python
 579 cdef class Gini(ClassificationCriterion):
 580     """Gini Index impurity criterion.
 581 +-- 11 lines: This handles cases where the target is a classification taking valu
 592         index = \sum_{k=0}^{K-1} count_k (1 - count_k)
 593               = 1 - \sum_{k=0}^{K-1} count_k ** 2
 594     """
 595
 596     cdef double node_impurity(self) nogil:
 597 #+-- 12 lines: """Evaluate the impurity of the current node, i.e. the impurity of-
 609         for k in range(self.n_outputs):
 610             sq_count = 0.0
 611
 612             for c in range(n_classes[k]):
 613                 count_k = sum_total[c]
 614                 sq_count += count_k * count_k
 615
 616             gini += 1.0 - sq_count / (self.weighted_n_node_samples *
 617                                       self.weighted_n_node_samples)
 618
 619             sum_total += self.sum_stride
 620
 621         return gini / self.n_outputs
 622
 623     cdef void children_impurity(self, double* impurity_left,
 624                                 double* impurity_right) nogil:
 625         """Evaluate the impurity in children nodes
 626 #+-- 45 lines: i.e. the impurity of the left child (samples[start:pos]) and the---
```

### 3. 回归评价函数

#### 3.0 RegressionCriterion
RegressionCriterion 是回归评价的父类，与分类预计算权重占比 $p_k^i$ 类似，回归会预计算节点的方差值。为了节省计算量，对公式做了变形，降低了计算次数，具体推导如下所示：

\begin{align}
    \text{var } &= \frac{1}{n} \sum_i^n (y_i - \bar{y})^2 \\
                &= E \left [ (y_i - E(y_i))^2 \right ] \\
                &= E[y_i^2] - E^2[y_i] \\
                &= \frac{1}{n} \sum_i^n (y_i^2) - \bar{y}^2 \\
                &= \frac{1}{n} \sum_i^n (y_i^2) - \left ( \frac{1}{n} \sum_i^n y_i \right )^2 \\
                & \text{ 引入权重值 } \\
                &= \sum_i^n ( w_i (y_i^2) ) / \sum_i^n w_i - \left ( \sum_i^n (w_i y_i) / \sum_i^n w_i \right )^2 \\
                &= \frac{\text{sq_sum_total}}{\text{weighted_n_node_samples}} - \left (\frac{\text{sum_total}}{\text{weighted_n_node_samples}} \right )^2
\end{align}

所以，RegressionCriterion 的方法都是在计算和更新上述式子中的三个变量（还有子节点的），因为整个代码和分类相似，不再赘述。

##### 3.1 MSE
其实这里 MSE 用的公式就是上述的方差公式。虽然实际实义中 $\text{MSE } = E [ y_i - \hat{y}_i ]$，但因为决策树预测值用的是均值，所以全部 $\hat{y}_i = \bar{y_i}$，于是退化成了计算方差的公式。

##### 3.2 FriedmanMSE
FriedmanMSE 来源于 Friedman 论文 [Greedy Function Approximation: A Gradient Boosting Machine(https://statweb.stanford.edu/~jhf/ftp/trebst.pdf) 中的公式 (35)，跟 Gradient Boosting 相关，这方面我还没看，等后继写到 GBDT 时再回来补上。

### 总结

本文较为详细地介绍了 sklearn 中评估函数模块的实现细节：一个接口类，两个辅助计算的父类，四个具体计算的子类。


# **6.1 Energy flow - supply, storage, dissipation**

[](https://moodle.rwth-aachen.de/mod/page/view.php?id=551846) part 1

        Starting point

Inelastic deformation induces energy dissipation. By distinguishing stored and lost energy within a system we can provide another physical perspective to the changes in the material structure. This perspective is particularly important for materials exhibiting softening behavior which which induces an energy dissipation within a small volume of material.

        Where are we heading

In the present notebook we will revisit the nonlinear pullout curve and show how to evaluate the stored and released energy. We start with an abstract description explaining the evaluation procedure. To provide an elementary example, we then apply the described concept to an elementary example of an elastic bar with no energy dissipation. This prepares the framework for the next example showing applying the evaluation procedure to the pullout test in the notebook [6.2](6_2_Energy_released_in_pullout_constant_bond_and_rigid_matrix.ipynb#top) and, most importantly, to the notebook [6.3]() setting up the ground for the description of fracture. Thus, our next tour goes through the following three stations: tensile test, pullout test and bending test. We will show a unified framework for the evaluation of energy dissipation in all three of them.

# **Classification of energies**

[](https://moodle.rwth-aachen.de/mod/page/view.php?id=551846) part 2

## Supplied energy

Consider a general force displacement curve obtained in an experiment. The curve represents a response of a test setup with an arbitrary geometry of the specimen, material components and their layout, and even the boundary conditions.
The only information we need from the lab is the time profile of the control displacement $w(t)$ and the profile of force $P(t)$ measured by the hydraulic cell at the point of the imposed displacement. Using these two series of data points, we can plot the force displacement curves.

This curve can be used to answer the question, how much energy must have been supplied by the testing machine to reach the point $A$ of the measured response. Let us first recall that work is defined as the force needed achieve a unit displacement, i.e.
$$
 \mathcal{W} = P w \;\; \mathrm{[Nm]}
$$
In case of a nonlinear profile obtained from the testing machine we apply this definition to evaluate the work increment needed to achieve an infinitesimal displacement
$$
 \mathrm{d}\mathcal{W} = P \; \mathrm{d}w
$$
so that the total work needed to achieve the point $A$ with the displacement $w$ is obtained as an integral
$$
 \mathcal{W} = \int_0^w P(w) \; \mathrm{d}w
$$
representing the area below the force displacement curve.

## Stored energy

We already know, that the nonlinear shape of the force-displacement response is due to irreversible changes in the material structure. These changes are connected with energy dissipation. Thus, in point $A$ some fraction of energy has already been dissipated and some is stored in the specimen. How much energy is stored in the specimen at point $A$?
In the experiment, we can answer this question by unloading the specimen to the zero force and evaluating the area remaining below the unloading curve. But since this unloading step has not been done, we have to knowledge we gained during the Tours focused on plasticity and damage type of behavior. We know that assuming damage material behavior, the specimen would unload to the origin while with purely plastic behavior, the unloading material stiffness would be equal to the initial stiffness. Therefore, we can use these two limiting assumptions to obtain two bounds on stored energy.

The two bounds on the stored energy are depicted in the figure as the stored energy corresponding to the plastic lower bound $\mathcal{U}_\pi$ below the dashed curve, which is parallel to the initial stiffness $K_0$, and damage upper bound below the unloading curve, which connects the point $A$ with the origin.

A more accurate evaluation of stored energy is possible with the help of a model that can capture the inelastic effects of plasticity and damage. Then, the stored energy can be evaluated as an integral over the product of the stress and elastic strain
$$
\mathcal{U} =
\frac{1}{2} \int_\Omega \sigma(\boldsymbol{x}) \ \varepsilon_\mathrm{el}(\boldsymbol{x}) \; \mathrm{d} \boldsymbol{x}
$$

## Released energy

The difference between supplied and stored energy delivers the dissipated energy
$$
 G = \mathcal{W} - \mathcal{U}
$$

The described procedure has been implemented in the numerical simulator used in the model applications presented during the course so that it is possible to monitor the supplied, stored and released energy for all inelastic material models shown so far. We will use this feature to develop an understanding of the energy development using the pullout test and the bending test later.

To start in a transparent way, let us first address two elementary examples that can be solved analytically: elastic bar test and a pullout test with constant bond-slip law which has been analytical solved in Tour 2.

# **Application to elastic bar**

[](https://moodle.rwth-aachen.de/mod/page/view.php?id=551846) part 3

The first exercise demonstrating the above equations presents the elastic bar for which we can readily derive the force-displacement curve $P(w)$ and the stored energy function $\mathcal{W}$.

Assuming linear elastic material behavior we can employ the equilibrium condition and kinematic relation to link the load  directly to the control displacement $w$ as
\begin{align}
P = A \sigma = A E \varepsilon = A E \frac{1}{L} w,
\end{align}
where $E$ denotes the Young’s modulus, $\sigma$ the stresses, $A$ the cross-sectional area and $\varepsilon$ the strain.

**The energy supply** is then obtained as
\begin{align}
\label{eq:elastic_energy_supply}
\mathcal{W} = \int^w_0 P(w) \; \mathrm{d}w =
A E \frac{1}{L}  \int^w_0 w \; \mathrm{d}w
=
\frac{1}{2} A E  \frac{1}{L}  w^2
=
\frac{1}{2} P w
\end{align}

**Stored elastic energy** is
represented by the area below the unloading branch integrated over the whole volume of the structure.
In case of the tensile specimen with uniform profile of stress $\sigma$ and strain $\varepsilon$, the integral reduces to
\begin{align}
\mathcal{U}
=
\frac{1}{2}
\int_\Omega
\sigma
\cdot
\varepsilon
\;
\mathrm{d}x
=
\frac{1}{2}
A L \; \sigma \cdot \varepsilon
\end{align}

Realizing that $P = A \sigma$ and $w = L \varepsilon$ we obtain the expression for stored energy

\begin{align}
\mathcal{U}
=
\frac{1}{2}
P w
\end{align}

which means that the supplied and stored energy is equivalent, i.e.
\begin{align}
\mathcal{W} = \mathcal{U} \implies G = 0
\end{align}
recovering the obvious fact that the supplied energy is stored in the elastic specimen without any loss.

      5.2 Pullout behavior governed by damage

    6.2 Frictional pullout and energy dissipation

```python

```


# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Quantization of Filter Coefficients

The finite numerical resolution of digital number representations has impact on the properties of filters, as already discussed for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-Effects). The quantization of coefficients, state variables, algebraic operations and signals plays an important role in the design of recursive filters. Compared to non-recursive filters, the impact of quantization is often more prominent due to the feedback. Severe degradations from the desired characteristics and instability are potential consequences of a finite word length in practical implementations.

A recursive filter of order $N \geq 2$ can be [decomposed into second-order sections (SOS)](../recursive_filters/cascaded_structures.ipynb). Due to the grouping of poles/zeros to filter coefficients with a limited amplitude range, a realization by cascaded SOS is favorable in practice. We therefore limit our investigation of quantization effects to SOS. The transfer function of a SOS is given as

\begin{equation}
H(z) = \frac{b_0 + b_1 z^{-1} + b_2 z^{-2}}{1 + a_1 z^{-1} + a_2 z^{-2}}
\end{equation}

This can be [split into a non-recursive part and a recursive part](../recursive_filters/introduction.ipynb#Recursive-Filters). The quantization effects of non-recursive filters have already been discussed. We therefore focus here on the recursive part given by the transfer function

\begin{equation}
H(z) = \frac{1}{1 + a_1 z^{-1} + a_2 z^{-2}}
\end{equation}

This section investigates the consequences of quantization in recursive filters. As for non-recursive filters, we first take a look at the quantization of filter coefficients. The structure used for the realization of the filter has impact on the quantization effects. We begin with the direct form followed by the coupled form, as example for an alternative structure.

### Direct Form

Above transfer function of the recursive part of a SOS can be rewritten in terms of its complex conjugate poles $z_{\infty}$ and $z_{\infty}^*$ as

\begin{equation}
H(z) = \frac{1}{(z-z_{\infty}) (z-z_{\infty}^*)} = \frac{z^{-2}}{ 1 \underbrace{- 2 r \cos(\varphi)}_{a_1} \; z^{-1} + \underbrace{r^2}_{a_2} \; z^{-2} }
\end{equation}

where $r = |z_{\infty}|$ and $\varphi = \arg \{z_{\infty}\}$ denote the absolute value and phase of the pole $z_{\infty}$, respectively. Let's assume a [linear uniform quantization](../quantization/linear_uniform_quantization_error.ipynb#Quantization-Error-of-a-Linear-Uniform-Quantizer) of the coefficients $a_1$ and $a_2$ with quantization step $Q$. Discarding clipping, the following relations for the locations of the poles can be found

\begin{align}
r_n &= \sqrt{n \cdot Q} \\
\varphi_{nm} &= \arccos \left( \sqrt{\frac{m^2 Q}{4 n}} \right)
\end{align}
for $n \in \mathbb{N}_0$ and $m \in \mathbb{Z}$. Quantization of the filter coefficients $a_1$ and $a_2$ into a finite number of amplitude values leads to a finite number of pole locations. In the $z$-plane the possible pole locations are given by the intersections of

* circles whose radii $r_n$ are given by $r_n = \sqrt{n \cdot Q}$ with
* equidistant vertical lines which intersect the horizontal axis at $\frac{1}{2} m \cdot Q$.

The finite number of pole locations may lead to deviations from a desired filter characteristic since a desired pole location is moved to the next possible pole location. The filter may even get unstable, when poles are moved outside the unit circle. For illustration, the resulting pole locations for a SOS realized in direct form are computed and plotted.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Circle
import scipy.signal as sig
import itertools

def compute_pole_locations(Q):
    a1 = np.arange(-2, 2+Q, Q)
    a2 = np.arange(0, 1+Q, Q)

    p = np.asarray([np.roots([1, n, m]) for (n,m) in itertools.product(a1, a2)])
    p = p[np.imag(p)!=0]

    return p

def plot_pole_locations(p, Q):
    ax = plt.gca()
    for n in np.arange(np.ceil(2/Q)+1):
        circle = Circle((0,0), radius=np.sqrt(n*Q), fill=False, color='black', ls='solid', alpha=0.05)
        ax.add_patch(circle)
        ax.axvline(.5*n*Q, color='0.95')
        ax.axvline(-.5*n*Q, color='0.95')

    unit_circle = Circle((0,0), radius=1, fill=False, color='red', ls='solid')
    ax.add_patch(unit_circle)

    plt.plot(np.real(p), np.imag(p), 'b.', ms = 4)
    plt.xlabel(r'Re{$z$}')
    plt.ylabel(r'Im{$z$}')
    plt.axis([-1.1, 1.1, -1.1, 1.1])

# compute and plot pole locations
for w in [5,6]:
    Q = 2/(2**(w-1))  # quantization stepsize
    plt.figure(figsize=(5, 5))
    p = compute_pole_locations(Q)
    plot_pole_locations(p, Q)
    plt.title(r'Direct form coefficient quantization to $w=%d$ bits'%w)
```

**Exercise**

* What consequences has the distribution of pole locations on the desired characteristics of a filter for e.g. low/high frequencies?

### Coupled Form

Besides the quantization step $Q$, the pole distribution depends also on the topology of the filter. In order to gain a different distribution of pole locations after quantization, one has to derive structures where the coefficients of the multipliers are given by other values than the direct form coefficients $a_1$ and $a_2$.

One of these alternative structures is the coupled form (also known as Gold & Rader structure)

where $\Re\{z_\infty\} = r \cdot \cos \varphi$ and $\Im\{z_\infty\} = r \cdot \sin \varphi$ denote the real- and imaginary part of the complex pole $z_\infty$, respectively. Analysis of the structure reveals its difference equation as

\begin{align}
w[k] &= x[k] + \Re\{z_\infty\} \, w[k-1] - \Im\{z_\infty\} \, y[k-1] \\
y[k] &= \Im\{z_\infty\} \, w[k-1] + \Re\{z_\infty\} \, y[k-1]
\end{align}

and its transfer function as

\begin{equation}
H(z) = \frac{\Im\{z_\infty\} \; z^{-1}}{ 1 - 2 \Re\{z_\infty\} \; z^{-1} + (\Re\{z_\infty\}^2 + \Im\{z_\infty\}^2) \; z^{-2} }
\end{equation}

Note that the numerator of the transfer function differs from the recursive only SOS given above. However, this can be considered in the design of the transfer function of a general SOS.

The real- and imaginary part of the pole $z_\infty$ occur directly as coefficients for the multipliers in the coupled form. Quantization of these coefficients results therefore in a Cartesian grid of possible pole locations in the $z$-plane. This is illustrated in the following.

```python
def compute_pole_locations(w):
    Q = 1/(2**(w-1))  # quantization stepsize
    a1 = np.arange(-1, 1+Q, Q)
    a2 = np.arange(-1, 1+Q, Q)

    p = np.asarray([n+1j*m for (n,m) in itertools.product(a1, a2) if n**2+m**2 <= 1])

    return p

def plot_pole_locations(p):
    ax = plt.gca()

    unit_circle = Circle((0,0), radius=1, fill=False, color='red', ls='solid')
    ax.add_patch(unit_circle)

    plt.plot(np.real(p), np.imag(p), 'b.', ms = 4)
    plt.xlabel(r'Re{$z$}')
    plt.ylabel(r'Im{$z$}')
    plt.axis([-1.1, 1.1, -1.1, 1.1])

# compute and plot pole locations
for w in [5,6]:
    plt.figure(figsize=(5, 5))
    p = compute_pole_locations(w)
    plot_pole_locations(p)
    plt.title(r'Coupled form coefficient quantization to $w=%d$ bits'%w)
```

**Excercise**

* What is the benefit of this representation in comparison to the direct from discussed in the previous section?

### Example

The following example illustrates the effects of coefficient quantization for a recursive [Butterworth filter](https://en.wikipedia.org/wiki/Butterworth_filter) realized in cascaded SOSs in transposed direct form II.

```python
w = 12  # wordlength of filter coefficients
N = 5  # order of filter

def uniform_midtread_quantizer(x, w, xmin=1):
    # quantization step
    Q = xmin/(2**(w-1))
    # limiter
    x = np.copy(x)
    idx = np.where(x <= -xmin)
    x[idx] = -1
    idx = np.where(x > xmin - Q)
    x[idx] = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

# coefficients of recursive filter
b, a = sig.butter(N, 0.2, 'low')
# decomposition into SOS
sos = sig.tf2sos(b, a, pairing='nearest')
sos = sos/np.amax(np.abs(sos))
# quantization of SOS coefficients
sosq = uniform_midtread_quantizer(sos, w, xmin=2)
# compute overall transfer function of (quantized) filter
H = np.ones(512)
Hq = np.ones(512)
for n in range(sos.shape[0]):
    Om, Hn = sig.freqz(sos[n, 0:3], sos[n, 3:6])
    H = H * Hn
    Om, Hn = sig.freqz(sosq[n, 0:3], sosq[n, 3:6])
    Hq = Hq * Hn

# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot(Om, 20 * np.log10(abs(H)), label='continuous')
plt.plot(Om, 20 * np.log10(abs(Hq)), label='quantized')
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.legend(loc=3)
plt.grid()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot(Om, np.unwrap(np.angle(H)), label='continuous')
plt.plot(Om, np.unwrap(np.angle(Hq)), label='quantized')
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.legend(loc=3)
plt.grid()
```

**Exercise**

* Decrease the word length `w` of the filter. What happens? At what word length does the filter become unstable?
* Increase the order `N` of the filter for a fixed word length `w`. What happens?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# 고유분해와 특이값 분해

정방 행렬 $A$에 대해 다음 식을 만족하는 단위 벡터 $v$, 스칼라 $\lambda$을 여러 개 찾을 수 있다.

$$ Av = \lambda v $$

* $ A \in \mathbf{R}^{M \times M} $

* $ \lambda \in \mathbf{R} $

* $ v \in \mathbf{R}^{M} $

이러한 실수 $\lambda$를 고유값(eigenvalue), 단위 벡터 $v$ 를 고유벡터(eigenvector) 라고 하며 고유값과 고유벡터를 찾는 작업을 고유분해(eigen-decomposition)라고 한다.

$ A \in \mathbf{R}^{M \times M} $ 에 대해 최대 $M$개의 고유값-고유벡터 쌍이 존재할 수 있다.

예를 들어 다음 행렬 $A$

$$
A=
\begin{bmatrix}
1 & -2 \\
2 & -3
\end{bmatrix}
$$
에 대해 다음 단위 벡터와 스칼라 값은 고유벡터-고유값이 된다.

$$\lambda = -1$$

$$
v=
\begin{bmatrix}
\dfrac{1}{\sqrt{2}}  \\
\dfrac{1}{\sqrt{2}}
\end{bmatrix}
$$

복수 개의 고유 벡터가 존재하는 경우에는 다음과 같이 고유벡터 행렬 $V$와 고유값 행렬 $\Lambda$로 표기할 수 있다.

$$
A \left[ v_1 \cdots v_M \right] =
\left[ \lambda_1 v_1 \cdots \lambda_M v_M \right] =
\left[ v_1 \cdots v_M \right]
\begin{bmatrix}
\lambda_{1} & 0 & \cdots & 0 \\
0 & \lambda_{2} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & \lambda_{M} \\
\end{bmatrix}
$$

$$ AV = V\Lambda $$

여기에서

$$
V = \left[ v_1 \cdots v_M \right]
$$

$$
\Lambda =
\begin{bmatrix}
\lambda_{1} & 0 & \cdots & 0 \\
0 & \lambda_{2} & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & \lambda_{M} \\
\end{bmatrix}
$$

numpy linalg 서브패키지에서는 고유값과 고유벡터를 구할 수 있는 `eig` 명령을 제공한다.

```python
w, V = np.linalg.eig(np.array([[1, -2], [2, -3]]))
```

```python
w
```

    array([-0.99999998, -1.00000002])

```python
V
```

    array([[ 0.70710678,  0.70710678],
           [ 0.70710678,  0.70710678]])

## 대칭 행렬의 고유 분해

행렬 $A$가 대칭(symmetric) 행렬이면 고유값 벡터 행렬 $V$는 다음과 같이 전치 행렬이 역행렬과 같아진다.

$$ V^T V = V V^T = I$$

이 때는 고유 분해가 다음과 같이 표시된다.

$$ A = V\Lambda V^T = \sum_{i=1}^{M} {\lambda_i} v_i v_i^T$$

$$ A^{-1} = V \Lambda^{-1} V^T = \sum_{i=1}^{M} \dfrac{1}{\lambda_i} v_i v_i^T$$

## 확률 변수의 좌표 변환

확률 변수의 공분산 행렬 $\Sigma$ 은 대칭 행렬이므로 위의 관계식이 성립한다.

따라서 다변수 가우시안 정규 분포의 확률 밀도 함수는 다음과 같이 표시할 수 있다.

$$
\begin{eqnarray}
\mathcal{N}(x \mid \mu, \Sigma)
&=& \dfrac{1}{(2\pi)^{D/2} |\Sigma|^{1/2}} \exp \left( -\dfrac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right) \\
&=& \dfrac{1}{(2\pi)^{D/2} |\Sigma|^{1/2}} \exp \left( -\dfrac{1}{2} (x-\mu)^T V \Lambda^{-1} V^T (x-\mu) \right) \\
&=& \dfrac{1}{(2\pi)^{D/2} |\Sigma|^{1/2}} \exp \left( -\dfrac{1}{2} (V^T(x-\mu))^T  \Lambda^{-1} (V^T (x-\mu)) \right) \\
\end{eqnarray}
$$

즉 변환 행렬 $V^T$로 좌표 변환하면 서로 독립인 성분들로 나누어진다.

```python
mu = [2, 3]
cov = [[2, 3],[3, 7]]
rv = sp.stats.multivariate_normal(mu, cov)
xx = np.linspace(0, 4, 120)
yy = np.linspace(1, 5, 150)
XX, YY = np.meshgrid(xx, yy)
plt.grid(False)
plt.contourf(XX, YY, rv.pdf(np.dstack([XX, YY])))

x1 = np.array([0, 2])
x1_mu = x1 - mu
x2 = np.array([3, 4])
x2_mu = x2 - mu
plt.plot(x1_mu[0] + mu[0], x1_mu[1] + mu[1], 'bo', ms=20)
plt.plot(x2_mu[0] + mu[0], x2_mu[1] + mu[1], 'ro', ms=20)

plt.axis("equal")
plt.show()
```

```python
w, V = np.linalg.eig(cov)
```

```python
w
```

    array([ 0.59487516,  8.40512484])

```python
V
```

    array([[-0.90558942, -0.4241554 ],
           [ 0.4241554 , -0.90558942]])

```python
rv = sp.stats.multivariate_normal(mu, w)
xx = np.linspace(0, 4, 120)
yy = np.linspace(1, 5, 150)
XX, YY = np.meshgrid(xx, yy)
plt.grid(False)
plt.contourf(XX, YY, rv.pdf(np.dstack([XX, YY])))

x1 = np.array([0, 2])
x1_mu = x1 - mu
x2 = np.array([3, 4])
x2_mu = x2 - mu

x1t_mu = V.T.dot(x1_mu)  # 좌표 변환
x2t_mu = V.T.dot(x2_mu)  # 좌표 변환

plt.plot(x1t_mu[0] + mu[0], x1t_mu[1] + mu[1], 'bo', ms=20)
plt.plot(x2t_mu[0] + mu[0], x2t_mu[1] + mu[1], 'ro', ms=20)

plt.axis("equal")
plt.show()
```

## 특이값 분해

정방 행렬이 아닌 행렬 $M$에 대해서도 고유 분해와 유사한 분해가 가능하다. 이를 특이값 분해(singular value decomposition)이라고 한다.

* $M \in \mathbf{R}^{m \times n}$

$$M = U \Sigma V^T$$

여기에서
* $U \in \mathbf{R}^{m \times m}$
* $\Sigma  \in \mathbf{R}^{m \times n}$
* $V \in \mathbf{R}^{n \times n}$

이고 행렬 $U$와 $V$는 다음 관계를 만족한다.

$$ U^T U = UU^T = I $$
$$ V^T V = VV^T = I $$

예를 들어

$$\mathbf{M} = \begin{bmatrix}
                      1 & 0 & 0 & 0 & 2 \\
                      0 & 0 & 3 & 0 & 0 \\
                      0 & 0 & 0 & 0 & 0 \\
                      0 & 2 & 0 & 0 & 0
                    \end{bmatrix}
$$
에 대한 특이값 분해 결과는 다음과 같다.

$$
\begin{align}
\mathbf{U} &= \begin{bmatrix}
               0 & 0 & 1 &  0 \\
               1 & 0 & 0 &  0 \\
               0 & 0 & 0 & -1 \\
               0 & 1 & 0 &  0 \\
             \end{bmatrix} \\
\boldsymbol{\Sigma} &= \begin{bmatrix}
                        \sqrt{5} & 0 & 0 & 0 & 0 \\
                        0 & 2 & 0 & 0 & 0 \\
                        0 & 0 & 1 & 0 & 0 \\
                        0 & 0 & 0 & 0 & 0
                      \end{bmatrix} \\
\mathbf{V}^T &= \begin{bmatrix}
                           0 & 0 & \sqrt{0.2} & 0 & \sqrt{0.8} \\
                           0 & 1 & 0 & 0 & 0 \\
                           1 & 0 & 0 & 0 & 0 \\
                           0 & 0 & -\sqrt{0.8} & 0 & \sqrt{0.2} \\
                           0 & 0 & 0 & 1 & 0 \\
               \end{bmatrix}
\end{align}$$

이는 다음과 같이 확인 할 수 있다.

$$\begin{align}
\mathbf{U} \mathbf{U^T} &=
   \begin{bmatrix}
               0 & 0 & 1 &  0 \\
               1 & 0 & 0 &  0 \\
               0 & 0 & 0 & -1 \\
               0 & 1 & 0 &  0 \\
             \end{bmatrix}
\cdot
  \begin{bmatrix}
               0 & 1 & 0 & 0 \\
               0 & 0 & 0 & 1 \\
               1 & 0 & 0 & 0 \\
               0 & 0 & -1 & 0 \\
             \end{bmatrix}
   =
  \begin{bmatrix}
    1 & 0 & 0 & 0 \\
    0 & 1 & 0 & 0 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1
  \end{bmatrix}
   = \mathbf{I}_4 \\
\mathbf{V} \mathbf{V^T} &=
  \begin{bmatrix}
                           0 & 0 & \sqrt{0.2} & 0 & \sqrt{0.8} \\
                           0 & 1 & 0 & 0 & 0 \\
                           1 & 0 & 0 & 0 & 0 \\
                           0 & 0 & -\sqrt{0.8} & 0 & \sqrt{0.2} \\
                           0 & 0 & 0 & 1 & 0 \\
               \end{bmatrix}
  \cdot
  \begin{bmatrix}
              0 & 0 & 1 & 0 &          0 \\
              0 & 1 & 0 & 0 &          0 \\
     \sqrt{0.2} & 0 & 0 & -\sqrt{0.8} & 0\\
              0 & 0 & 0 & 0 &          1 \\
     \sqrt{0.8} & 0 & 0 & \sqrt{0.2} & 0 \\
  \end{bmatrix}
  =
  \begin{bmatrix}
    1 & 0 & 0 & 0 & 0 \\
    0 & 1 & 0 & 0 & 0 \\
    0 & 0 & 1 & 0 & 0 \\
    0 & 0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 0 & 1
  \end{bmatrix}
   = \mathbf{I}_5
\end{align}$$

```python
from pprint import pprint
M = np.array([[1,0,0,0,0],[0,0,2,0,3],[0,0,0,0,0],[0,2,0,0,0]])
print("\nM:"); pprint(M)
U, S0, V0 = np.linalg.svd(M, full_matrices=True)
print("\nU:"); pprint(U)
S = np.hstack([np.diag(S0), np.zeros(M.shape[0])[:, np.newaxis]])
print("\nS:"); pprint(S)
print("\nV:"); pprint(V)
V = V0.T
print("\nU.dot(U.T):"); pprint(U.dot(U.T))
print("\nV.dot(V.T):"); pprint(V.dot(V.T))
print("\nU.dot(S).dot(V.T):"); pprint(U.dot(S).dot(V.T))
```

    M:
    array([[1, 0, 0, 0, 0],
           [0, 0, 2, 0, 3],
           [0, 0, 0, 0, 0],
           [0, 2, 0, 0, 0]])

    U:
    array([[ 0.,  0.,  1.,  0.],
           [ 1.,  0.,  0.,  0.],
           [ 0.,  0.,  0., -1.],
           [ 0.,  1.,  0.,  0.]])

    S:
    array([[ 3.60555128,  0.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  2.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  1.        ,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ]])

    V:
    array([[-0.        , -0.        ,  1.        ,  0.        ,  0.        ],
           [ 0.        ,  1.        ,  0.        ,  0.        ,  0.        ],
           [ 0.5547002 ,  0.        ,  0.        , -0.83205029,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  0.        ,  1.        ],
           [ 0.83205029,  0.        ,  0.        ,  0.5547002 ,  0.        ]])

    U.dot(U.T):
    array([[ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

    V.dot(V.T):
    array([[ 1.,  0.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.,  0.],
           [ 0.,  0.,  1.,  0.,  0.],
           [ 0.,  0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  0.,  1.]])

    U.dot(S).dot(V.T):
    array([[ 1.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  2.,  0.,  3.],
           [ 0.,  0.,  0.,  0.,  0.],
           [ 0.,  2.,  0.,  0.,  0.]])

06. 기초 선형대수/08. 고유분해와 특이값 분해.ipynb


Adrian Swartz and Alfred Cheung

4/8/2019

# Analysis Code for "Superconducting Tunneling Spectroscopy of Spin-Orbit Coupling and Orbital Depairing in Nb:SrTiO$_3$"

This is the source code for the analysis of spin-orbit coupling and orbital depairing in thin Nb-doped SrTiO$_3$ thin films as measured by electron tunneling spectroscopy. This code was used for the data analysis presented in [Swartz, et al., Physical Review Letters __121__, 167003 (2018)](https://doi.org/10.1103/PhysRevLett.121.167003), and also available on the [arxiv](https://arxiv.org/abs/1805.00047). A couple of example calculations are presented here in a Jupyter Notebook for the purpose of demonstrating the code in operation. Lasly, the numerical solutions to Maki's theory is implemented here in python 2.

Complete details on the experimental measurements and physics of spin-orbit coupling are provided in the published manuscript. Here, I briefly summarize the physics and math relevant for computing the best fit according to Maki's theory for the spin-resolved density of states (DOS) of a superconductor. The tunneling current is given by integrating the DOS over the experimental bias window ($eV$). Tuning the voltge (the energy ($E$)) between a counter electrode and the sample enables a direct measurement of the differential conductance ($di/dv$) (proportional to the sample DOS). The complete DOS is the sum of the two spin-components as given by,

$\begin{equation}
\label{Maki1}
\rho_{\uparrow,\downarrow} = \frac{\rho_0}{2} \,\textrm{Re} \! \left( \frac{u_{\pm}}{\sqrt{u_{\pm}^2 - 1}}   \right),
\end{equation}
%$
where $\rho_0$ is the normal-state DOS and $u_{\pm}$ are defined by,
$
%
\begin{equation}
\label{Maki2}
u_{\pm} = \frac{E\mp \mu_B H}{\Delta_0} +
\zeta \frac{u_{\pm}}{\sqrt{1-u_{\pm}^2}} +
b \left(    \frac{u_{\mp}-u_{\pm}}{\sqrt{1-u_{\mp}^2}}    \right).
\end{equation}
$

Here, $E$ is the experimental energy, $\mu_B$ is the Bohr magneton, $H$ is the applied magnetic field, $\Delta_0$ is the energy of the superconducting gap, $\zeta = \alpha_i + \alpha_o H_{\parallel}^2$ is the total contribution to orbital depairing including both the field-dependent orbital depairing parameter ($\alpha_o$) and the intrinsic depairing term ($\alpha_i=0.056$), and lastly $b$ is the dimensionless spin-orbit mixing rate. Note that $b$ is dimensionless, while $\alpha_o$ has units of T$^{-2}$. The two free parameters in the fit are $\alpha_o$ and $b$, while the magnetic field and temperature are fixed by the experimental environment. The size of the superconducting gap ($\Delta_0$) is deterimined at zero magnetic field.

This is the recursive formula for the spin-resolved density of states as originally given by Maki. Early experiments on elemental superconductors by Tedrow and Meservey in the 1970's employed Maki's theory conceptually and qualitatively, but did not provide detailed computational procedure. More "recently," Alexander provided a set of four equations with four complex unknowns which is equivalent to Maki's equation which was used to explore Fermi liquid effects on the superconducting DOS and tunneling spectrum. Alexender's approach simplifies the problem to solving a set of complex (real and imaginary) linear equations. This numerical approach was applied as a proof of concept to the problem of spin-split superconducting tunneling spectra by Wordledge and Geballe for Al superconducting contacts to thin LSMO films. Wordledge and Geballe work demonstrated that this numeriacal analysis method was valid in the low spin-orbit limit (for a single magnetic field).

Here, I have adopted the approach presented by Worledge and Geballe for superconducting tunneling spectra of SrTiO$_3$ thin films in the 2D superconducting limit. In this case, the spin-orbit coupling strength cannot assume to be weak. An applied magnetic field breaks the superconducting Cooper pairs apart (destroying the superconducting phase) by the combination of orbital depairing and spin-orbit mixing. In the limit of strong spin-orbit coupling, these two processes both manifest as effective broadening of the superconducting gap. The spin-orbit coupling and orbital depairing contributions cannot therefore be independently quantified when there is finite spin-orbit coupling. Therefor, much of the published paper details focuses on carefully quantifying the experimental limits which can be placed on $\alpha_o$ and $b$.

Even though the model parameters ($\alpha_o$ and $b$) have a complex and non-linear relationship with the dependent variable $E$, this is still essentially a regression problem. The solution to the set of four complex equations is calculated at each energy and the function `dosarray_filter` in the `DOSMODEL_class` ensures smoothness of the local minima of the density of states accross the entire energy range. The metric for the quality of the fit to the data is the mean squared error integrated (and normalized) over the energy range. Further details are available in the main text and supplementary materials of the [published manuscript](https://doi.org/10.1103/PhysRevLett.121.167003).

```python
#import libraries
import numpy as np
import cmath as cm
from scipy.optimize import minimize
from scipy.optimize import fsolve
import pylab as pl

import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sns
cp = sns.color_palette()
```

```python
class TOSOLVE_class:
    """ create the set of 8 non-linear equations (4 real and 4 imaginary)
    E is just a parameter (not an array)
    creates the 4 equations to be solved for a specific energy (could be positive or negative)
    E, delta, alphaorb, alphasoc, and h are all single parameters
    """

    def __init__(self,E,Delta,alphaorb,alphasoc,h):
        self.Delta = Delta
        self.alphaorb = alphaorb
        self.alphasoc = alphasoc
        self.h = h
        self.E = E

    def func(self,x):
        # returns a list of 8 equesions; first 4 are real, the final 4 are the imaginary piece
        return [\
        self.E*x[2]+x[0]+self.h**2*x[6]-self.alphaorb*(x[0]*x[2]-x[1]*x[3]-self.h**2*x[4]*x[6]+self.h**2*x[5]*x[7])/np.pi,\
        self.E*x[3]+x[1]+self.h**2*x[7]-self.alphaorb*(x[0]*x[3]+x[1]*x[2]-self.h**2*x[4]*x[7]-self.h**2*x[5]*x[6])/np.pi,\
        self.E*x[6]+x[2]-x[4]+self.alphaorb*(x[2]*x[4]-x[3]*x[5]-x[0]*x[6]+x[1]*x[7])/np.pi-self.alphasoc*(x[0]*x[6]-x[1]*x[7]+x[2]*x[4]-x[3]*x[5])/np.pi,\
        self.E*x[7]+x[3]-x[5]+self.alphaorb*(x[2]*x[5]+x[3]*x[4]-x[0]*x[7]-x[1]*x[6])/np.pi-self.alphasoc*(x[0]*x[7]+x[1]*x[6]+x[2]*x[5]+x[3]*x[4])/np.pi,\
        x[0]**2-x[1]**2-self.Delta**2*(x[2]**2-x[3]**2)+self.h**2*(x[4]**2-x[5]**2)-self.h**2*self.Delta**2*(x[6]**2-x[7]**2)+np.pi**2,\
        x[0]*x[1]-self.Delta**2*x[2]*x[3]+self.h**2*x[4]*x[5]-self.h**2*self.Delta**2*x[6]*x[7],\
        x[0]*x[4]-x[1]*x[5]+self.Delta**2*(x[2]*x[6]-x[3]*x[7]),\
        x[0]*x[5]+x[1]*x[4]+self.Delta**2*(x[2]*x[7]+x[3]*x[6])\
        ]

```

```python

class DOSMODEL_class:
    """ Cretes a DOS curve (model) for a specific energy range given
    For fitting the data, Edomain should be a list of the biases from experimental data to be fit
    if not fitting, Edomain can be any energy range and step size
    """

    def __init__(self,Edomain,Delta,alphaorb,alphasoc,h):
        self.Delta = Delta
        self.alphaorb = alphaorb
        self.alphasoc = alphasoc
        self.h = h
        self.Edomain = Edomain
        self.N = len(Edomain)
        self.Espace = self.Edomain[1]-self.Edomain[0]

    def soln(self,E,X1,X2):
        #Takes one energy from E domain to calculate the DOS at that specific energy
        #X1, X2 are arbitrary fake initial values to input into fsolve

        #create the class w/ the 8 equations for that energy
        tosolve_class = TOSOLVE_class(E,self.Delta,self.alphaorb,self.alphasoc,self.h)

        #call the tosolve class function func which returns a list of the 8 equations
        #Then use fsolve on the "list" of 8 equations
        #However, fsolve requires an input of a function and however many necessary initial conditions
        #X1, X2 here could be altered to no effect
        sol = fsolve(tosolve_class.func,[X1,X1,X2,X2,X1,X2,X1,X2])

        #returns a list of the best solutions for the 8 parameters
        #To equate with Alexander 1985, x0, x1 are real and imaginary part of y1, ...

        return sol

    def upm(self,E,X1,X2):
        """Using soln() func, generate the Alexander coefficients for u+/- (Maki)
        reconstruct Alexander complex coefficients (y1 through y4)
        """
        ys = self.soln(E,X1,X2)
        y1 = ys[0]+(1j)*ys[1]
        y2 = ys[2]+(1j)*ys[3]
        y3 = ys[4]+(1j)*ys[5]
        y4 = ys[6]+(1j)*ys[7]
        up = cm.sqrt(1.-np.pi**2/(self.Delta*(y2-self.h*y4))**2)
        um = cm.sqrt(1.-np.pi**2/(self.Delta*(y2+self.h*y4))**2)
        # up = (1./self.Delta)*(y1+self.h*y3)/(y2-self.h*y4)
        # um = (1./self.Delta)*(y1-self.h*y3)/(y2+self.h*y4)
        return up,um

    def dos(self,E,X1,X2):
        """ Now that you have Maki's u+ and u-,
        calculate the total superconducting DOS
        spin resolved can be obtained from uminu and uplus
        """
        UPM = self.upm(E,X1,X2)
        uplus = UPM[0]
        uminu = UPM[1]
        uplus = uplus/cm.sqrt(uplus**2-1.)
        uminu = uminu/cm.sqrt(uminu**2-1.)
        return uplus.real+uminu.real

    def dosarray(self):
        """Now generate an array (along the energy axis) of the DOS model
        """
        result = np.zeros(self.N)
        x1 = 1.0
        x2 = 1.0
        for i in xrange(self.N):
            result[i] = self.dos(self.Edomain[i],x1,x2)
        return result

    """dosarray() has a numerical problem: generates many "bad" points.
        Since we are solving for the DOS at each point, susceptible to solutions
        which converge locally but aren't continuous between different energies
        Need to add some contraints in order to maintain "smoothness" of DOS curve.
        """

    def dosarray_filter(self):
        """This is where all the magic happens.
        Meant to do the same thing as above but
        but now we check that the DOS result at each energy
        isn't too far away from the previous DOS result
        If it is, we change the initial conditions and increase the number of trials
        """

        result = np.zeros(self.N)

        Echange = 2.0
        Mgiveup = 10
        Mgiveup1 = 10
        skip = 0.1
        slopetol = 150

        initialx = 5.0	#Set to larger values (~5.0) for moderate fields or larger SOC, else ~0.5
        initialx1 = 0.5	#Set to larger values (~5.0) for moderate fields or larger SOC, else ~0.5; if see kink away from gap, set to 0.5
        x1 = initialx1
        x2 = initialx1
        current = self.dos(self.Edomain[0],x1,x2)
        base = current
        result[0] = current/base
        for i in xrange(1,(self.N-1)/2+1):
        #for i in xrange(1,self.N):
            energy = self.Edomain[i]
            if (np.abs(energy)<Echange):
                tol = 0.1
                x1 = initialx	#Enable reset of IC for moderate fields
                x2 = initialx
                new = self.dos(energy,x1,x2)
                chnge = new-current
                abchnge = np.abs(chnge)
                n = 0
                while ((abchnge>tol) and (n<=Mgiveup)):
                    x1 += skip
                    x2 += skip
                    n += 1
                    new = self.dos(energy,x1,x2)
                    abchnge = np.abs(new-current)
                current = new
                result[i] = current/base
                if (current/base<0.05):
                    break
            else:
                tol = 0.01
                # x1= initialx1
                # x2 = initialx1
                new = self.dos(energy,x1,x2)
                abchnge = np.abs(new-current)
                n = 0
                while (abchnge>tol) and (n<Mgiveup1):
                    x1 += skip
                    x2 += skip
                    n += 1
                    new = self.dos(energy,x1,x2)
                    abchnge = np.abs(new-current)
                current = new
                result[i] = current/base
                slope = abchnge/self.Espace

        # subresult = np.zeros(self.N)
        # for i in range(self.N):
        # subresult[i] = result[i]+result[self.N-1-i]

        #START FROM OTHER END OF DOMAIN
        x1 = initialx1
        x2 = initialx1
        current = self.dos(-self.Edomain[self.N-1],x1,x2)
        base = current
        result[self.N-1] = current/base
        for i in range(self.N-2,(self.N-1)/2,-1):
            energy = -self.Edomain[i]              #!!!!!!!!!!!!
            if (np.abs(energy)<Echange):
                tol = 0.1
                x1 = initialx
                x2 = initialx
                new = self.dos(energy,x1,x2)
                chnge = new-current
                abchnge = np.abs(chnge)
                n = 0
                while ((abchnge>tol) and (n<=Mgiveup)):
                    x1 += skip
                    x2 += skip
                    n += 1
                    new = self.dos(energy,x1,x2)
                    abchnge = np.abs(new-current)
                current = new
                result[i] = current/base
            # if (current<0.05):
            # break
            else:
                tol	= 0.01
            # x1 = initialx1
            # x2 = initialx1
                new = self.dos(energy,x1,x2)
                abchnge = np.abs(new-current)
                n = 0
                while (abchnge>tol) and (n<Mgiveup1):
                    x1 += skip
                    x2 += skip
                    n += 1
                    new = self.dos(energy,x1,x2)
                    abchnge = np.abs(new-current)
                current = new
                result[i] = current/base

        # for i in xrange(0,(self.N-1)/2+1):
        # result[self.N-1-i] = result[i]

        tosave = np.vstack([self.Edomain,result])
        np.savetxt('DOS_alphaorb_{0:.3f}_alphasoc_{1:.3f}_h_{2:.3f}_B.dat'.\
        format(self.alphaorb,self.alphasoc,self.h),tosave.transpose(), delimiter='\t')

        return result

```

```python
class DIDV_class:
    def __init__(self,Edomain,kT,Delta,alphaorb,alphasoc,h):
        self.Delta = Delta
        self.alphaorb = alphaorb
        self.alphasoc = alphasoc
        self.h = h
        self.kT = kT
        self.beta = 1./kT
        self.DeltaE = Edomain[1]-Edomain[0]
        self.cutoff = 4*int(self.kT/self.DeltaE)
        self.N = len(Edomain)
        self.Edomain = Edomain
        self.subEdomain = np.delete(Edomain,range(0,self.cutoff)+range(self.N-self.cutoff,self.N))

    def fdprime(self,E):
        return self.beta*np.exp(self.beta*(E))/(1.+np.exp(self.beta*(E)))**2

    def riemann(self,i,dosarr):	#i is the location of the eV in index form. (length-cutoff-1)>=i>=cutoff
        ans = 0.
        for j in xrange(-self.cutoff,self.cutoff+1):
            ans += dosarr[i+j]*self.fdprime(self.DeltaE*j)*self.DeltaE
        return ans

    def producedidv(self):
        subN = len(self.subEdomain)
        dosmodel_class = DOSMODEL_class(self.Edomain,self.Delta,self.alphaorb,self.alphasoc,self.h)
        dosarr = dosmodel_class.dosarray_filter()
        result = np.zeros(subN)
        didv = self.riemann(self.cutoff,dosarr)
        base = didv
        result[0] = didv/base
        for i in xrange(self.cutoff+1,self.N-self.cutoff):
            didv = self.riemann(i,dosarr)
            result[i-self.cutoff] = didv/base
        return result
```

```python
class FIT_class:
    """ FIT_class calls the DIDV, DOSMODEL, and TOSOLVE classes
    FIT_class does most of the heavy lifting including opening the file,
    passing the parameters for the model, fitting the model to the data,
    and returning the didv according to maki theory and the chi-squared error.

    In principle, a grid search can be completed by sweeping [orbital, spin-orbit] parameters
    and calling the sumofres method.

    Please note that it takes some time to calculate the Maki DOS.
    """

    def __init__(self, datfiletofit, kT, gapinvolts, h):
        self.kT = kT
        self.beta = 1./kT
        self.data = np.loadtxt(datfiletofit,unpack=True)
        self.dmn = self.data[0]/gapinvolts
        self.N = len(self.dmn)
        self.DeltaE = self.dmn[1]-self.dmn[0]
        self.cutoff = 4*int(self.kT/self.DeltaE)
        self.meas = self.data[1]
        self.submeas = np.delete(self.meas,range(0,self.cutoff)+range(self.N-self.cutoff,self.N))
        self.subdmn = np.delete(self.dmn,range(0,self.cutoff)+range(self.N-self.cutoff,self.N))
        self.bgoffset = self.meas[0]-1.
        self.h = h

    def sumofres(self, params): #params = [aorb,asoc]
        aorb = params[0]
        asoc = params[1]
        model_class = DIDV_class(self.dmn,kT,Delta,aorb,asoc,self.h)
        theory_didv = model_class.producedidv()+self.bgoffset
        tosave = np.vstack([self.subdmn,theory_didv])
        np.savetxt('DIDV_kT_{0:.3f}_alphaorb_{1:.3f}_alphasoc_{2:.3f}_h_{3:.3f}_B.dat'.\
        format(self.kT,aorb,asoc,self.h),\
        tosave.transpose(), delimiter='\t')
        residuals = (self.submeas-theory_didv)**2
        return np.sum(residuals)

    def chi_squared_result(self, params): #params = [aorb,asoc]
        aorb = params[0]
        asoc = params[1]
        model_class = DIDV_class(self.dmn,kT,Delta,aorb,asoc,self.h)
        theory_didv = model_class.producedidv()+self.bgoffset
        residuals = (self.submeas-theory_didv)**2
        return np.sum(residuals)

    def get_didv_curves(self, params): #params = [aorb,asoc]
        aorb = params[0]
        asoc = params[1]
        model_class = DIDV_class(self.dmn,kT,Delta,aorb,asoc,self.h)
        theory_didv = model_class.producedidv()+self.bgoffset
        return (self.data, [self.subdmn, theory_didv])

    def bestfit(self, initialguess):
        bestparams = minimize(self.sumofres,initialguess,method='CG',options={'maxiter':10,'disp':True},callback=self.callbackF,tol=0.5)
        return bestparams.x

    def callbackF(self, Xi):
        print('aorb={0:.6f}\tasoc={1:.6f}'.format(Xi[0], Xi[1]))

    def bestfit_custom(self, initialguess):
        ctraorb = initialguess[0]
        ctrasoc = initialguess[1]
        widthaorb = 0.01
        widthasoc = 0.50
        grid = 6
        # result = np.zeros([grid,grid])
        F0=open('zoomin.dat','w')
        for asoc in np.linspace(ctrasoc-widthasoc,ctrasoc+widthasoc,grid):
            for aorb in np.linspace(ctraorb-widthaorb,ctraorb+widthaorb,grid):
                parameters = [aorb,asoc]
                X2 = self.sumofres(parameters)
                print('asoc={}\taorb={}\tX2={}'.format(asoc,aorb,X2))
                F0.write('{0:12.10E}\t{1:12.10E}\t{2:12.10E}\n'.format(asoc,aorb,X2))
            print('\n')
            F0.write('\n')
        F0.close()

```

```python
# Global Parameters
dimensionfulDelta= 0.000047 # in eV
kT = 0.15 # ~80mK (0.14667*Delta), 90mK (0.165), 100mK (0.183)
Delta = 1.00 # dimensionless, Delta=1 corresponds to ~47 micro-eV
mu_B = 5.78838E-5
#h = 0.740 # mu times H (600 mT; Delta=1 corresponds to ~0.812 T)
```

Example of creating a single example Maki DOS using a FIT_class object for a single spectra measured at 600 mT.

```python
h = (600 * 1E-3)*5.78838E-5 / dimensionfulDelta

fit_class = FIT_class('data/600mT.txt',kT,dimensionfulDelta,h)

#alphaorb=0.1; betaSOC=1
didv_data, didv_maki = fit_class.get_didv_curves([0.1, 1])

fig, ax = plt.subplots()
plt.plot(didv_maki[0], didv_maki[1] , label='Maki_DOS', color=cp[3])
plt.plot(didv_data[0]/dimensionfulDelta, didv_data[1], label='data',color=cp[0])
plt.legend()
ax.set_xlabel('Delta/Delta_0')
```

__Plot the best-fit di/dv results ($\alpha_o = 0.117$ and $b = 6$). $b>1$ indicates a large spin-mixing rate relative to the energy scale associated with forming the superconducting condensate.__

```python
data_to_fit = [400,500,600,700] # Magnetic fields in mT
alphaorb = 0.117
betaSOC = 6

fig, axes = plt.subplots(1,len(data_to_fit), figsize=(15,4))
for i, h_i in enumerate(data_to_fit):
    path = 'data/' + str(h_i) + 'mT.txt'
    h = (h_i * 1E-3)*5.78838E-5 / dimensionfulDelta
    fit_class = FIT_class(path,kT,dimensionfulDelta,h)
    didv_data, didv_maki = fit_class.get_didv_curves([alphaorb, betaSOC])

    axes[i].plot(didv_data[0]/dimensionfulDelta, didv_data[1], label=str(h_i)+'_data', lw=2, color='black')
    axes[i].plot(didv_maki[0], didv_maki[1], label=str(h_i)+'_Maki', lw=1, color='red')
    axes[i].set_xlabel('Delta/Delta_0')
    axes[i].legend()
    axes[i].set_xlim((-10,10))

plt.tight_layout()
```

__Plot an example of a bad Maki fit ($\alpha_o = 0.1$ and $b = 0$). An example DOS if there wass no Spin orbit coupling.__

```python
data_to_fit = [400,500,600,700]
alphaorb = 0.1
betaSOC = 0

fig, axes = plt.subplots(1,len(data_to_fit), figsize=(15,4))
for i, h_i in enumerate(data_to_fit):
    path = 'data/' + str(h_i) + 'mT.txt'
    h = (h_i * 1E-3)*5.78838E-5 / dimensionfulDelta
    fit_class = FIT_class(path,kT,dimensionfulDelta,h)
    didv_data, didv_maki = fit_class.get_didv_curves([alphaorb, betaSOC])

    axes[i].plot(didv_data[0]/dimensionfulDelta, didv_data[1], label=str(h_i)+'_data', lw=2, color='black')
    axes[i].plot(didv_maki[0], didv_maki[1], label=str(h_i)+'_Maki', lw=1, color='red')
    axes[i].set_xlabel('Delta/Delta_0')
    axes[i].legend()
    axes[i].set_xlim((-10,10))

plt.tight_layout()
```

__Plot example $\chi^2$ results for a single set of coupled orbital and spin-orbit depairing parameters.__

```python
fit_class = FIT_class('data/500mT.txt',kT,dimensionfulDelta,h)

alphaorb = 0.1
betaSOC = [3,4,5,6,7,8,9,10,11,12]
chi_sq = []

#print('\n===========PRINTING ITERATION RESULTS============')

for i, b_i in enumerate(betaSOC):
    h = (500 * 1E-3)*5.78838E-5 / dimensionfulDelta
    fit_class = FIT_class('data/500mT.txt',kT,dimensionfulDelta,h)
    x2 = fit_class.chi_squared_result([alphaorb, b_i])
    chi_sq.append(x2)

#    print('betaSOC = {0:12.10E}'.format(b_i))
#    print('Chi Squared = {0:12.10E}'.format(x2))
#    print('\n')
#print('\n================DONE=================')

```

```python
fig, ax = plt.subplots(figsize=(8,6))
ax.plot(betaSOC, chi_sq, '-o')

ax.set_xlabel('betaSOC')
ax.set_ylabel('Chi squared')
ax.set_xlim((0,30))
#ax.set_ylim((0,15))
ax.legend()
```

Here I have just plotted a few examples with the main purpose of demonstrating how the FIT_class object can be used to show different Maki DOS and compare them with the experimental results. In the manuscript, we have first used a spin-independent version of Maki's theory to identify the $\alpha_i$ intercept. We then fit the data at several magnetic fields and calculate the combined MSE accross all of the fields. The results of which are shown below.

```python

```


```python
import numpy as np
import pandas as pd
from sympy import Matrix
```

```python
from IPython.display import SVG, display
```

# Types of vectors

Vectors come in three flavors: (1) geometric vectors, (2) polynomials, (3) and elements of Rn space.

## Geometric vectors
Geometric vectors are oriented segments. These are the kind of vectors you probably learned about in high-school
physics and geometry. Many linear algebra concepts come from the geometric point of view of vectors: space,
plane, distance, etc.

Fig. 2: Geometric vectors

```python
def show_svg():
    display(SVG(url='https://pabloinsente.github.io/assets/post-10/b-geometric-vectors.svg'))
show_svg()
```

## Polynomials
A polynomial is an expression like f(x)=x2+y+1. This is, a expression adding multiple “terms” (nomials).
Polynomials are vectors because they meet the definition of a vector: they can be added together to get another
polynomial, and they can be multiplied together to get another polynomial.

function addition is valid f(x)+g(x) and multiplying by a scalar is valid 5×f(x)

Fig. 3: Polynomials

```python
def show_svg():
    display(SVG(url='https://pabloinsente.github.io/assets/post-10/b-polynomials-vectors.svg'))
show_svg()
```

```python
# dot product
x, y = np.array([[-2], [2]]), np.array([[4], [-3]])
np.dot(x.T, y)
```

    array([[-14]])

# Vector space, span and subspace

## Vector Space

In its more general form, a vector space, also known as linear space, is a collection of objects that follow the rules defined for vectors in Rn. We mentioned those rules when we defined vectors: they can be added together and multiplied by scalars, and return vectors of the same type. More colloquially, a vector space is the set of proper vectors and all possible linear combinatios of the vector set. In addition, vector addition and multiplication must follow these eight rules:

- commutativity: x+y=y+x
- associativity: x+(y+x)=(y+x)+z
- unique zero vector such that: x+0=x ∀ x
- ∀ x there is a unique vector x such that x+−x=0
- identity element of scalar multiplication: 1x=x
- distributivity of scalar multiplication w.r.t vector addition: x(y+z)=xz+zy
- x(yz)=(xy)z
- (y+z)x=yx+zx

## Vector Span

Consider the vectors x and y and the scalars α and β. If we take all possible linear combinations of αx+βy we would obtain the span of such vectors. This is easier to grasp when you think about geometric vectors. If our vectors x and y point into different directions in the 2-dimensional space, we get that the span(x,y) is equal to the entire 2-dimensional plane, as shown in the middle-pane in Fig. 5. Just imagine having an unlimited number of two types of sticks: one pointing vertically, and one pointing horizontally. Now, you can reach any point in the 2-dimensional space by simply combining the necessary number of vertical and horizontal sticks (including taking fractions of sticks).

Fig. 5: Vector Span

What would happen if the vectors point in the same direction? Now, if you combine them, you just can span a line, as shown in the left-pane in Fig. 5. If you have ever heard of the term “multicollinearity”, it’s closely related to this issue: when two variables are “colinear” they are pointing in the same direction, hence they provide redundant information, so can drop one without information loss.

With three vectors pointing into different directions, we can span the entire 3-dimensional space or a hyper-plane, as in the right-pane of Fig. 5. Note that the sphere is just meant as a 3-D reference, not as a limit.

Four vectors pointing into different directions will span the 4-dimensional space, and so on. From here our geometrical intuition can’t help us. This is an example of how linear algebra can describe the behavior of vectors beyond our basics intuitions.

## Vector subspaces

A vector subspace (or linear subspace) is a vector space that lies within a larger vector space. These are also known as linear subspaces. Consider a subspace S. For a vector to be a valid subspace it has to meet three conditions:

- Contains the zero vector, 0∈S
- Closure under multiplication, ∀α∈R→α×si∈S
- Closure under addition, ∀si∈S→s1+s2∈S

Intuitively, you can think in closure as being unable to “jump out” from space into another. A pair of vectors laying flat in the 2-dimensional space, can’t, by either addition or multiplication, “jump out” into the 3-dimensional space.

Fig. 6: Vector subspaces

Consider the following questions: Is x=[1 1] a valid subspace of R2? Let’s evaluate x on the three conditions:

**Contains the zero vector**: it does. Remember that the span of a vector are all linear combinations of such a vector. Therefore, we can simply multiply by 0 to get [0 0]:

x×0=0[1 1]=[0 0]

**Closure under multiplication** implies that if take any vector belonging to x and multiply by any real scalar α, the resulting vector stays within the span of x. Algebraically is easy to see that we can multiply [1 1] by any scalar α, and the resulting vector remains in the 2-dimensional plane (i.e., the span of R2).

**Closure under addition** implies that if we add together any vectors belonging to x, the resulting vector remains within the span of R2. Again, algebraically is clear that if we add x + x, the resulting vector will remain in R2. There is no way to get to R3 or R4 or any space outside the two-dimensional plane by adding x multiple times.

## Linear dependence and independence

The left-pane shows a triplet of linearly dependent vectors, whereas the right-pane shows a triplet of linearly independent vectors.

Fig. 7: Linear dependence and independence

A set of vectors is linearly dependent if at least one vector can be obtained as a linear combination of other vectors in the set. As you can see in the left pane, we can combine vectors x and y to obtain z.

There is more rigurous (but slightly harder to grasp) definition of linear dependence. Consider a set of vectors x1,…,xk and scalars β∈R. If there is a way to get 0=∑βixi with at least one β≠0, we have linearly dependent vectors. In other words, if we can get the zero vector as a linear combination of the vectors in the set, with weights that are not all zero, we have a linearly dependent set.

A set of vectors is linearly independent if none vector can be obtained as a linear combination of other vectors in the set. As you can see in the right pane, there is no way for us to combine vectors x and y to obtain z. Again, consider a set of vectors x1,…,xk and scalars β∈R. If the only way to get 0=∑ki=1βixi requires all β1,…,βk=0, the we have linearly independent vectors. In words, the only way to get the zero vectors in by multoplying each vector in the set by 0.

## Vector Null space

Now that we know what subspaces and linear dependent vectors are, we can introduce the idea of the null space. Intuitively, the null space of a set of vectors are all linear combinations that “map” into the zero vector. Consider a set of geometric vectors w, x, y, and z as in Fig. 8. By inspection, we can see that vectors x and z are parallel to each other, hence, independent. On the contrary, vectors w and y can be obtained as linear combinations of x and z, therefore, dependent.

Fig. 8: Vector null space

As result, with this four vectors, we can form the following two combinations that will “map” into the origin of the coordinate system, this is, the zero vector (0,0):

z−y+x=0
z−x+w=0

## Vector Norms

Measuring vectors is another important operation in machine learning applications. Intuitively, we can think about the norm or the length of a vector as the distance between its “origin” and its “end”.

Norms “map” vectors to non-negative values. In this sense are functions that assign length ∥x∥∈Rn to a vector x. To be valid, a norm has to satisfy these properties (keep in mind these properties are a bit abstruse to understand):

Absolutely homogeneous: ∀α∈R,∥αx∥=|α∥∥x∥. In words: for all real-valued scalars, the norm scales proportionally with the value of the scalar.
Triangle inequality: ∥x+y∥≤∥x∥+∥y∥. In words: in geometric terms, for any triangle the sum of any two sides must be greater or equal to the lenght of the third side. This is easy to see experimentally: grab a piece of rope, form triangles of different sizes, measure all the sides, and test this property.
Positive definite: ∥x∥≥0 and ∥x∥=0⟺x=0. In words: the length of any x has to be a positive value (i.e., a vector can’t have negative length), and a length of 0 occurs only of x=0
Grasping the meaning of these three properties may be difficult at this point, but they probably become clearer as you improve your understanding of linear algebra.

Fig. 9: Vector norms

### Euclidean Norm

```python
x = np.array([[3], [4]])
np.linalg.norm(x, 2) #If you remember the first “Pythagorean triplet”, you can confirm that the norm is correct.
```

    5.0

```python
# In machine learning, unless made explicit, we can safely assume that an inner product refers to the dot product.
x, y = np.array([[-2],[2]]), np.array([[4],[-3]])
x.T @ y
```

    array([[-14]])

```python
# As with the inner product, usually, we can safely assume that distance stands for the Euclidean distance or
# L2 norm unless otherwise noted. To compute the L2 distance between a pair of vectors:

distance = np.linalg.norm(x-y, 2)
print(f'L_2 distance : {distance}')
```

    L_2 distance : 7.810249675906654

### Manhattan Norm

```python
x = np.array([[3], [-4]])
np.linalg.norm(x, 1) #easy to confirm that the sum of the absolute values of 3 and −4 is 7.
```

    7.0

## Max Norm

```python
x = np.array([[3], [-4]])
np.linalg.norm(x, np.inf)
```

    4.0

## Vector angles and orthogonality

```python
x, y = np.array([[1], [2]]), np.array([[5], [7]])
cos_theta = np.dot(x.T, y) / (np.linalg.norm(x,2) * (np.linalg.norm(y,2)))

print(f"Cos of the angle:{np.round(cos_theta, 3)} ")
```

    Cos of the angle:[[0.988]]

```python
cos_inv = np.arccos(cos_theta)
print(f"Angle in Radians: {np.round(cos_inv, 3)}")
```

    Angle in Radians: [[0.157]]

```python
degrees = cos_inv * (180/np.pi)
print(f"Angle in degrees: {np.round(degrees,3)}")
```

    Angle in degrees: [[8.973]]

**Orthogonality** is often used interchangeably with “independence” although they are mathematically different concepts. Orthogonality can be seen as a generalization of perpendicularity to vectors in any number of dimensions.

**We say that a pair of vectors x and y are orthogonal if their inner product is zero, ⟨x,y⟩=0**. The notation for a pair of orthogonal vectors is x⊥y. In the 2-dimensional plane, this equals to a pair of vectors forming a 90∘ angle.

## Solving a system of Linear Equations
x+2y=8

5x−3y=1

```python
df = pd.DataFrame({"x1": [0, 2], "y1":[8, 3], "x2": [0, 2], "y2": [0, 3]})
df

```

      x1
      y1
      x2
      y2

      0
      0
      8
      0
      0

      1
      2
      3
      2
      3

```python
df.plot(x = "x1", y = ["y1", "y2"])
```

## Matrix Inverse

```python
A = np.array([[1, 2, 1],
              [4, 4, 5],
              [6, 7, 7]])

A_inv = np.linalg.inv(A)
A_inv
```

    array([[-7., -7.,  6.],
           [ 2.,  1., -1.],
           [ 4.,  5., -4.]])

## Hadamard product
It is matrix-matrix multiplication as an element-wise operation.

aij⋅bij:=cij

```python
A = np.array([[0,2],
              [1,4]])
B = np.array([[1,3],
              [2,1]])

np.multiply(A,B)
```

    array([[0, 6],
           [2, 4]])

## Solving systems of linear equations with Matrices

```python
# Ax = y

A = np.array([[1, 3, 5],
              [2, 2, -1],
              [1, 3, 2]])
y = np.array([[-1],
              [1],
              [2]])

np.linalg.solve(A, y)
```

    array([[-2.],
           [ 2.],
           [-1.]])

## Matrix Basis and Rank

```python
A = Matrix([[1, 0, 1],
            [0, 1, 1]])

B = Matrix([[1, 2, 3, -1],
            [2, -1, -4, 8],
            [-1, 1, 3, -5],
            [-1, 2, 5, -6],
            [-1, -2, -3, 1]])

A_rref, A_pivots = A.rref()
print(f'Reduced row echelon form of A: {A_rref}')
```

    Reduced row echelon form of A: Matrix([[1, 0, 1], [0, 1, 1]])

```python
print(f'Column pivots of A: {A_pivots}')
```

    Column pivots of A: (0, 1)

```python
B_rref, B_pivots = B.rref()
print(f'Reduced row echelon form of B:{B_rref}')
```

    Reduced row echelon form of B:Matrix([[1, 0, -1, 0], [0, 1, 2, 0], [0, 0, 0, 1], [0, 0, 0, 0], [0, 0, 0, 0]])

```python
print(f'Column pivots of A: {B_pivots}')
```

    Column pivots of A: (0, 1, 3)

For A, we found that the first and second column vectors are the basis, whereas for B is the first, second, and fourth.

Now that we know about a basis and how to find it, understanding the concept of rank is simpler. The rank of a matrix A is the dimensionality of the vector space generated by its number of linearly independent column vectors. This happens to be identical to the dimensionality of the vector space generated by its row vectors. We denote the rank of matrix as rk(A) or rank(A).

For an square matrix Rm×n (i.e., m=n), we say is full rank when every column and/or row is linearly independent. For a non-square matrix with m>n (i.e., more rows than columns), we say is full rank when every row is linearly independent. When m<n (i.e., more columns than rows), we say is full rank when every column is linearly independent.

From an applied machine learning perspective, the rank of a matrix is relevant as [a measure of the information content of the matrix](https://math.stackexchange.com/questions/21100/importance-of-matrix-rank). Take matrix B from the example above. Although the original matrix has 5 columns, we know is rank 4, hence, it has less information than it appears at first glance.

## Matrix Norm

As with vectors, we can measure the size of a matrix by computing its norm. There are multiple ways to define the norm for a matrix, as long it satisfies the same properties defined for vectors norms: (1) absolutely homogeneous, (2) triangle inequality, (3) positive definite (see vector norms section).

For our purposes, I’ll cover two of the most commonly used norms in machine learning: (1) Frobenius norm, (2) max norm, (3) spectral norm.

```python
A = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])
np.linalg.norm(A, 'fro')
```

    16.881943016134134

```python
A = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])
np.linalg.norm(A, np.inf)
#In this case, is easy to see that the third row has the largest absolute value.
```

    24.0

```python
A = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])
np.linalg.norm(A, 2)
```

    16.84810335261421

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## REFERNCES

https://pabloinsente.github.io/intro-linear-algebra

Linear Algebra/.ipynb_checkpoints/Linear Algebra - Overview-checkpoint.ipynb

Linear Algebra/Linear Algebra - Overview.ipynb


```python
import numpy as np
import qiskit
import sympy
from sympy.physics.quantum import TensorProduct
```

```python
basis_1d = []
basis_1d.append(sympy.Matrix([[1],[0]]))
basis_1d.append(sympy.Matrix([[0],[1]]))
```

```python
basis_2d = []
for i in basis_1d:
    for j in basis_1d:
        basis_2d.append(TensorProduct(i,j))
```

```python

```

```python

```


```python
#Auther:杨昊鹏@Milko
```

**initialize area

```python
import matplotlib.pyplot as plt
import numpy as np
#import sympy as sp
from scipy.integrate import solve_ivp
import math
%matplotlib inline

```

**calculate area

```python
for i in range(1,200):
    theta = np.pi/2/200*i
    init=[0,350*np.cos(theta),0,350*np.sin(theta)]

    def f1(t,x):
        return x[1],-0.001*np.sqrt(x[1]**2+x[3]**2)*x[1],x[3],-0.001*np.sqrt(x[1]**2+x[3]**2)*x[3]-9.8

    def hit_ground(t,x):
        return x[2]

    res=solve_ivp(f1,(0,30),init,t_eval=np.arange(0,30,1e-3),method='RK45',dense_output=True,events=hit_ground)

    plt.xlabel('θ')
    plt.ylabel('hit point distance')
    print(res.y_events[0][1][0])
    print(theta)
    plt.scatter(theta,res.y_events[0][1][0])
plt.show()

```

Physics_Projectile with air resistance_Milko.ipynb