# Lab03: Machine learning

- MSSV:
- Họ và tên:

## Yêu cầu bài tập

**Cách làm bài**

Bạn sẽ làm trực tiếp trên file notebook này; trong file, từ `TODO` để cho biết những phần mà bạn cần phải làm.

Bạn có thể thảo luận ý tưởng cũng như tham khảo các tài liệu, nhưng *code và bài làm phải là của bạn*.

Nếu vi phạm thì sẽ bị 0 điểm cho bài tập này.

**Cách nộp bài**

Trước khi nộp bài, rerun lại notebook (`Kernel` -> `Restart & Run All`).

Sau đó, tạo thư mục có tên `MSSV` của bạn (vd, nếu bạn có MSSV là 1234567 thì bạn đặt tên thư mục là `1234567`) Chép file `Lab03-MachineLearning.ipynb` vào, rồi nén thư mục `MSSV` này lại và nộp ở link trên moodle.

**Nội dung bài tập**

Bài tập 3 là bài tập cá nhân. Trong bài này, bạn sẽ cài đặt 2 thuật toán học máy:
1. Cây quyết định (Decision tree)
2. Gaussian Naive Bayes (Lớp CNTN sẽ làm thêm phần 2)

### Import library

```python
import matplotlib.pyplot as plt
from sklearn import datasets
import pandas as pd
import numpy as np
from sklearn.metrics import accuracy_score
```

### Load Iris dataset

```python
from sklearn.model_selection import train_test_split
iris=datasets.load_iris()

X=iris.data
y=iris.target

#split dataset into training data and testing data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.33, random_state=42)
```

## 1. Cây quyết định: Iterative Dichotomiser 3 (ID3)

### 1.1 Information Gain

Thông tin kỳ vọng (entropy):

$$Entropy=-\sum_{i}^{n}p_ilog_{2}(p_i)$$

Hàm entropy đạt giá trị nhỏ nhất nếu có một giá trị $p_i=1$, đạt giá trị lớn nhất nếu tất cả các $p_i$ bằng nhau. Những tính chất này của hàm entropy khiến nó được sử dụng trong việc đo độ hỗn loạn của một phép phân chia của ID3.

```python
def entropy(counts, n_samples):
    """
    Parameters:
    -----------
    counts: shape (n_classes): list number of samples in each class
    n_samples: number of data samples

    -----------
    return entropy
    """
    #TODO

```

```python
def entropy_of_one_division(division):
    """
    Returns entropy of a divided group of data
    Data may have multiple classes
    """
    n_samples = len(division)
    n_classes = set(division)

    counts=[]
    #count samples in each class then store it to list counts
    #TODO:

    return entropy(counts,n_samples),n_samples

def get_entropy(y_predict, y):
    """
    Returns entropy of a split
    y_predict is the split decision by cutoff, True/Fasle
    """
    n = len(y)
    entropy_true, n_true = entropy_of_one_division(y[y_predict]) # left hand side entropy
    entropy_false, n_false = entropy_of_one_division(y[~y_predict]) # right hand side entropy
    # overall entropy
    #TODO s=?
    return s
```

Độ lợi thông tin phân lớp tập D theo thuộc tính A:
$$ Gain(A)=Entrophy(D)-Entrophy_{A}(D)$$

Trong ID3, tại mỗi node, thuộc tính được chọn được xác định dựa trên là thuộc tính khiến cho information gain đạt giá trị lớn nhất.

Các thuộc tính của tập Iris đều có giá trị liên tục. Do đó ta cần rời rạc hóa cho từng thuộc tính. Cách đơn giản là sử dụng một ngưỡng `cutoff` chia giá trị của dữ liệu trên mỗi thuộc tính sẽ làm 2 phần: `<cutoff` và `>=cutoff`.

Để tìm ngưỡng `cutoff` tốt nhất cho mỗi thuộc tính ta lần lượt thay `cutoff` bằng các giá trị của thuộc tính sau đó tính entropy, `cutoff` tốt nhất khi entropy bé nhất $ \left(\arg\min Entrophy_{A}(D)\right)$.

### 1.2 Decision tree

```python
class DecisionTreeClassifier:
    def __init__(self, tree=None, depth=0):
        '''Parameters:
        -----------------
        tree: decision tree
        depth: depth of decision tree after training'''

        self.depth = depth
        self.tree=tree
    def fit(self, X, y, node={}, depth=0):
        '''Parameter:
        -----------------
        X: training data
        y: label of training data
        ------------------
        return: node

        node: each node represented by cutoff value and column index, value and children.
         - cutoff value is thresold where you divide your attribute
         - column index is your data attribute index
         - value of node is mean value of label indexes,
           if a node is leaf all data samples will have same label

        Note that: we divide each attribute into 2 part => each node will have 2 children: left, right.
        '''

        #Stop conditions

        #if all value of y are the same
        if np.all(y==y[0]):
            return {'val':y[0]}

        else:
            col_idx, cutoff, entropy = self.find_best_split_of_all(X, y)    # find one split given an information gain
            y_left = y[X[:, col_idx] < cutoff]
            y_right = y[X[:, col_idx] >= cutoff]
            node = {'index_col':col_idx,
                        'cutoff':cutoff,
                   'val':np.mean(y)}
            node['left'] = self.fit(X[X[:, col_idx] < cutoff], y_left, {}, depth+1)
            node['right'] = self.fit(X[X[:, col_idx] >= cutoff], y_right, {}, depth+1)
            self.depth += 1
            self.tree = node
            return node

    def find_best_split_of_all(self, X, y):
        col_idx = None
        min_entropy = 1
        cutoff = None
        for i, col_data in enumerate(X.T):
            entropy, cur_cutoff = self.find_best_split(col_data, y)
            if entropy == 0:                   #best entropy
                return i, cur_cutoff, entropy
            elif entropy <= min_entropy:
                min_entropy = entropy
                col_idx = i
                cutoff = cur_cutoff

        return col_idx, cutoff, min_entropy

    def find_best_split(self, col_data, y):
        ''' Parameters:
        -------------
        col_data: data samples in column'''

        min_entropy = 10

        #Loop through col_data find cutoff where entropy is minimum

        for value in set(col_data):
            y_predict = col_data < value
            my_entropy = get_entropy(y_predict, y)
            #TODO
            #min entropy=?, cutoff=?

        return min_entropy, cutoff

    def predict(self, X):
        tree = self.tree
        pred = np.zeros(shape=len(X))
        for i, c in enumerate(X):
            pred[i] = self._predict(c)
        return pred

    def _predict(self, row):
        cur_layer = self.tree
        while cur_layer.get('cutoff'):
            if row[cur_layer['index_col']] < cur_layer['cutoff']:
                cur_layer = cur_layer['left']
            else:
                cur_layer = cur_layer['right']
        else:
            return cur_layer.get('val')
```

### 1.3 Classification on Iris Dataset

```python
model = DecisionTreeClassifier()
tree = model.fit(X_train, y_train)
pred=model.predict(X_train)
print('Accuracy of your decision tree model on training data:', accuracy_score(y_train,pred))
pred=model.predict(X_test)
print('Accuracy of your decision tree model:', accuracy_score(y_test,pred))

```

    Accuracy of your decision tree model on training data: 1.0
    Accuracy of your decision tree model: 0.96

## 2. Định lý Bayes

Định lý Bayes được phát biểu dưới dạng toán học như sau:
$$\begin{equation}
P\left(A|B\right)= \dfrac{P\left(B|A\right)P\left(A\right)}{P\left(B\right)}
\end{equation}$$

Nếu ta coi $B$ là dữ liệu $\mathcal{D}$, các thông số cần ước tính $A$ là $w$, ta có:

$$ \begin{align}
    \underbrace{P(w|\mathcal{D})}_{Posterior}= \dfrac{1}{\underbrace{P(\mathcal{D})}_{Normalization}} \overbrace{P(\mathcal{D}|w)}^{\text{Likelihood}} \overbrace{P(w)}^{Prior}
    \end{align}
$$

#### Naive Bayes
Để giúp cho việc tính toán được đơn giản, người ta thường giả sử một cách đơn giản nhất rằng các thành phần của biến ngẫu nhiên $D$ (hay các thuộc tính của dữ liệu $D$) là độc lập với nhau, nếu biết $w$. Tức là:
$$P(\mathcal{D}|w)=\prod _{i=1}^{d}P(x_i|w)$$

$d$: số lượng thuộc tính

### 2.1. Probability Density Function

```python
class pdf:
    def __init__(self,hist=None):
        '''
        A probability density function represented by a histogram

        hist: shape (n,1), n: number of hypotheses
        hypo: hypothesis (simply understand as label)
        ------------------
        hist[hypo]=P(hypo)
        '''
        self.hist = hist

    #virtual function
    def likelihood(self, data, hypo):
        '''Paramters:
        data: new data record
        hypo: hypothesis (simply understand as label)
        ---------
        return P(data/hypo)
        '''
        raise Exception()

    #update histogram for new data
    def update(self, data):
        '''
        P(hypo/data)=P(data/hypo)*P(hypo)*(1/P(data))
        '''

        #Likelihood * Prior
        #TODO
        for hypo in self.hist.keys():
            #self.hist[hypo]=?

        #Normalization

        #TODO: s=P(data)
        #s=?

        for hypo in self.hist.keys():
            self.hist[hypo] = self.hist[hypo]/s

    def plot_pdf(self):
        #plot Histogram
        #TODO

    def maxHypo(self):
        #find the hypothesis with maximum probability from hist
        #TODO

```

### 2.2 Classification on Iris Dataset

#### Gaussian Naive Bayes

- Naive Bayes có thể được mở rộng cho dữ liệu với các thuộc tính có giá trị là số thực, phổ biến nhất bằng cách sử dụng phân phối chuẩn (Gaussian distribution).

- Phần mở rộng này được gọi là Gaussian Naive Bayes. Các hàm khác có thể được sử dụng để ước tính phân phối dữ liệu, nhưng Gauss (hoặc phân phối chuẩn) là dễ nhất để làm việc vì chỉ cần ước tính giá trị trung bình và độ lệch chuẩn từ dữ liệu huấn luyện.

#### Define Gauss function

$$ f\left(x;\mu,\sigma \right)= \dfrac{1}{\sigma \sqrt{2\pi}}
\exp \left({-\dfrac{\left(x-\mu\right)^2}{2 \sigma^2}}\right) $$

```python
def Gauss(std,mean,x):
    #Compute the Gaussian probability distribution function for x
    #TODO

```

```python
class NBGaussian(pdf):
    def __init__(self, hist=None, std=None, mean=None):
        '''Parameters:

        '''
        pdf.__init__(self, hist)
        self.std=std
        self.mean=mean
    def likelihood(self,data, hypo):
        '''
        Returns: res=P(data/hypo)
        -----------------
        Naive bayes:
            Atributes are assumed to be conditionally independent given the class value.
        '''

        std=self.std[hypo]
        mean=self.mean[hypo]
        res=1
        #TODO
        #res=res*P(x1/hypo)*P(x2/hypo)...

        return res
    def fit(self, X,y):
        """Parameters:
        X: training data
        y: labels of training data
        """
        n=len(X)
        #number of iris species
        #TODO
        #n_species=???

        hist={}
        mean={}
        std={}

        #separate  dataset into rows by class
        for hypo in range(0,n_species):
            #rows have hypo label
            #TODO rows=

            #histogram for each hypo
            #TODO probability=?

            hist[hypo]=propability

            #Each hypothesis represented by its mean and standard derivation
            '''mean and standard derivation should be calculated for each column (or each attribute)'''
            #TODO mean[hypo]=?, std[hypo]=?

        self.mean=mean
        self.std=std
        self.hist=hist

    def _predict(self, data, plot=False):
        """
        Predict label for only 1 data sample
        ------------
        Parameters:
        data: data sample
        plot: True: draw histogram after update new data sample
        -----------
        return: label of data
        """
        model=NBGaussian(hist=self.hist.copy(),std=self.std.copy(), mean=self.mean.copy())
        model.update(data)
        if (plot): model.plot_pdf()
        return model.maxHypo()

    def predict(self, data):
        """Parameters:
        Data: test data
        ----------
        return labels of test data"""

        pred=[]
        for x in data:
            pred.append(self._predict(x))
        return pred
```

#### Show histogram of training data

```python
model_1=NBGaussian()
model_1.fit(X_train, y_train)
model_1.plot_pdf()
```

#### Test wih 1 data record

```python
#label of y_test[10]
print('Label of X_test[10]: ', y_test[10])
#update model and show histogram with X_test[10]:

print('Our histogram after update X_test[10]: ')
model_1._predict(X_test[10],plot=True)

```

#### Evaluate your Gaussian Naive Bayes model

```python
pred=model_1.predict(X_test)
print('Accuracy of your Gaussian Naive Bayes model:', accuracy_score(y_test,pred))
```

    Accuracy of your Gaussian Naive Bayes model: 0.96


# Vector Fitting
This is a brief introduction to Vector Fitting. The concept and its algorithm was proposed in 1999 by Bjørn Gustavsen and Adam Semlyen [[1](#link_ref1)]. See the Vector Fitting website for more information [[2](#link_ref2)].
The main application of Vector Fitting is to model the original sampled frequency responses of an active or passive device in a circuit simulator.

## Mathematical Description
The idea of Vector Fitting is to fit a set of rational model functions to a set of sampled frequency responses $\mathbf{\underline{H}}_\mathrm{sampled}$, for example from a [S](https://en.wikipedia.org/wiki/Scattering_parameters), [Y](https://en.wikipedia.org/wiki/Admittance_parameters) or [Z](https://en.wikipedia.org/wiki/Impedance_parameters) matrix. The model function $\mathbf{\underline{H}}(\mathrm{\underline{s}})$ is defined in the Laplace domain with $\mathrm{\underline{s}} = \sigma + \mathrm{j} \omega$:
\begin{equation}
\mathbf{\underline{H}}(\mathrm{\underline{s}}) = \mathbf{d} + \mathrm{\underline{s}} \mathbf{e} + \sum_{k=1}^K \frac{\underline{\mathbf{c}}_{k}}{\mathrm{\underline{s}}-\underline{p}_k}
\end{equation}

For the desired fit, that model function shall match the given frequency responses at each of its frequency samples $\omega_n$:
\begin{equation}
\mathbf{\underline{H}}(\mathrm{\underline{s}} = \mathrm{j} \omega_n) \overset{!}{=} \mathbf{\underline{H}}_\mathrm{sampled}(\omega_n)
\end{equation}

Generally, $\mathbf{\underline{H}}(\mathrm{\underline{s}})$ is a vector holding the individual complex frequency responses $\underline{H}_1(\mathrm{\underline{s}})$, $\underline{H}_2(\mathrm{\underline{s}})$, ..., $\underline{H}_M(\mathrm{\underline{s}})$ of the model. All elements in $\mathbf{\underline{H}}(\mathrm{\underline{s}})$ share a common set of complex poles $\underline{p}_k$, but have individual sets of complex residues $\mathbf{\underline{c}}_k$, real constants $\mathbf{d}$  and real proportional coefficients $\mathbf{e}$, which are therefore vectors as well:
\begin{equation}
\mathbf{\underline{p}} = \begin{pmatrix} \underline{p}_1 & \underline{p}_2 & \underline{p}_3 & \cdots & \underline{p}_K \end{pmatrix}
\end{equation}

\begin{equation}
\mathbf{\underline{c}} = \begin{pmatrix}
\underline{c}_{1,1} & \underline{c}_{2,1} & \underline{c}_{3,1} & \cdots & \underline{c}_{K,1} \\
\underline{c}_{1,2} & \underline{c}_{2,2} & \underline{c}_{3,2} & \cdots & \underline{c}_{K,2} \\
\vdots\\
\underline{c}_{1,M} & \underline{c}_{2,M} & \underline{c}_{3,M} & \cdots & \underline{c}_{K,M} \\
\end{pmatrix}
\end{equation}

\begin{equation}
\mathbf{d} = \begin{pmatrix} d_1 \\ d_2 \\ \vdots \\ d_M \end{pmatrix}
\end{equation}

\begin{equation}
\mathbf{e} = \begin{pmatrix} e_1 \\ e_2 \\ \vdots \\ e_M \end{pmatrix}
\end{equation}

The number of poles, $K$, required for a good fit of $\mathbf{\underline{H}}(\mathrm{\underline{s}})$ to $\mathbf{\underline{H}}_\mathrm{sampled}$ depends on the shape of the responses.

As an example, the goal could be to fit the rational model function to the S matrix of a 2-port ($M = 4$) sampled at $N$ frequencies $\omega_n$:

\begin{equation}
\begin{pmatrix}
\underline{S}_{11}(\omega_1) \\
\underline{S}_{12}(\omega_1) \\
\underline{S}_{21}(\omega_1) \\
\underline{S}_{22}(\omega_1) \\
\vdots \\
\underline{S}_{11}(\omega_\mathrm{N}) \\
\underline{S}_{12}(\omega_\mathrm{N}) \\
\underline{S}_{21}(\omega_\mathrm{N}) \\
\underline{S}_{22}(\omega_\mathrm{N})
\end{pmatrix}
\overset{!}{=}
\begin{pmatrix}
d_{11} + \mathrm{j} \omega_1 e_{11} + \sum_{k=1}^K \frac{\underline{c}_{k,11}}{\mathrm{j} \omega_1 - \underline{p}_k}
\\
d_{12} + \mathrm{j} \omega_1 e_{12} + \sum_{k=1}^K \frac{\underline{c}_{k,12}}{\mathrm{j} \omega_1 - \underline{p}_k}
\\
d_{21} + \mathrm{j} \omega_1 e_{21} + \sum_{k=1}^K \frac{\underline{c}_{k,21}}{\mathrm{j} \omega_1 - \underline{p}_k}
\\
d_{22} + \mathrm{j} \omega_1 e_{22} + \sum_{k=1}^K \frac{\underline{c}_{k,22}}{\mathrm{j} \omega_1 - \underline{p}_k}
\\
\vdots
\\
d_{11} + \mathrm{j} \omega_\mathrm{N} e_{11} + \sum_{k=1}^K \frac{\underline{c}_{k,11}}{\mathrm{j} \omega_\mathrm{N} - \underline{p}_k}
\\
d_{12} + \mathrm{j} \omega_\mathrm{N} e_{12} + \sum_{k=1}^K \frac{\underline{c}_{k,12}}{\mathrm{j} \omega_\mathrm{N} - \underline{p}_k}
\\
d_{21} + \mathrm{j} \omega_\mathrm{N} e_{21} + \sum_{k=1}^K \frac{\underline{c}_{k,21}}{\mathrm{j} \omega_\mathrm{N} - \underline{p}_k}
\\
d_{22} + \mathrm{j} \omega_\mathrm{N} e_{22} + \sum_{k=1}^K \frac{\underline{c}_{k,22}}{\mathrm{j} \omega_\mathrm{N} - \underline{p}_k}
\end{pmatrix}
\end{equation}

During the Vector Fitting process, the model parameters $\mathbf{\underline{p}}$, $\mathbf{\underline{c}}$, $\mathbf{d}$ and $\mathbf{e}$ will be optimized in an iterative manner until a good fit is achieved.

## Equivalent Electrical Circuits
The benefit of Vector Fitting the sampled frequency responses is the easy representation of the rational basis function with equivalent electrical circuits. A detailed derivation can be found in [[3](#link_ref3)].

### Constant and Proportional Term
The constant and proportional terms $\mathbf{d} + \mathrm{\underline{s}} \mathbf{e}$ can be represented in an electrical circuit by equivalent impedances $\underline{Z}_\mathrm{RL}(\mathrm{\underline{s}})$ or equivalent admittances $\underline{Y}_\mathrm{RC}(\mathrm{\underline{s}})$ build out of a series RL or parallel RC circuits.

Target response of the constant and proportional term:
\begin{equation}
\underline{H}_\mathrm{target}(\mathrm{\underline{s}}) = d_i + \mathrm{\underline{s}} e_i
\end{equation}

#### Impedance of a series RL circuit:
\begin{equation}
\underline{Z}_\mathrm{RL}(\mathrm{\underline{s}}) = R + \mathrm{\underline{s}} L
\end{equation}
This impedance matches the target response if $R = d_i$ and $L = e_i$.

#### Admittance of a parallel RC circuit:
\begin{equation}
\underline{Y}_\mathrm{RC}(\mathrm{\underline{s}}) = \frac{1}{R} + \mathrm{\underline{s}} C
\end{equation}
This admittance matches the target response if $R = \frac{1}{d_i}$ and $C = e_i$.

### Real Poles and Zeros
The individual terms $\frac{c_{k,i}}{\mathrm{\underline{s}} - p_{k,i}}$ of the fit with a real pole $p_{k,i}$ and a real residue $c_{k,i}$ can be represented in an electrical circuit by equivalent impedances $\underline{Z}_\mathrm{RC}(\mathrm{\underline{s}})$ or equivalent admittances $\underline{Y}_\mathrm{RL}(\mathrm{\underline{s}})$ build out of a parallel RC or series RL circuits.

Target response of the real pole-residue term:
\begin{equation}
\underline{H}_\mathrm{target}(\mathrm{\underline{s}}) = \frac{c_{k,i}}{\mathrm{\underline{s}} - p_{k,i}}
\end{equation}

#### Impedance of a parallel RC circuit:
The parallel RC circuit is the same as above for the constant and proportional term, but this time its impedance $\underline{Z}_\mathrm{RC}(\mathrm{\underline{s}})$ is used instead of its admittance:
\begin{equation}
\underline{Z}_\mathrm{RC}(\mathrm{\underline{s}}) = \frac{\frac{1}{C}}{\mathrm{\underline{s}} + \frac{1}{RC}}
\end{equation}
This impedance matches the target response if $C = \frac{1}{c_{k,i}}$ and $R = \frac{c_{k,i}}{-p_{k,i}}$.

#### Admittance of a series RL circuit:
The series RL circuit is the same as above for the constant and proportional term, but this time its admittance $\underline{Y}_\mathrm{RL}(\mathrm{\underline{s}})$ is used instead of its impedance:
\begin{equation}
\underline{Y}_\mathrm{RL}(\mathrm{\underline{s}}) = \frac{\frac{1}{L}}{\mathrm{\underline{s}} + \frac{R}{L}}
\end{equation}
This admittance matches the target response if $L = \frac{1}{c_{k,i}}$ and $R = \frac{-p_{k,i}}{c_{k,i}}$.

### Complex Conjugate Poles and Zeros
In Vector Fitting, complex poles $\underline{p}_k = p'_k + \mathrm{j} p''_k$ and residues $\underline{c}_k = c'_k + \mathrm{j} c''_k$ always come in complex conjugate pairs $(\underline{p}_k, \underline{p}_k^*)$ and $(\underline{c}_k, \underline{c}_k^*)$. The target response for an equivalent electrical circuit of such a complex conjugate pair is therefore:

\begin{equation}
\underline{H}_\mathrm{target}(\mathrm{\underline{s}}) = \frac{\underline{c}_k}{\mathrm{\underline{s}} - \underline{p}_k} + \frac{\underline{c}^*_k}{\mathrm{\underline{s}} - \underline{p}^*_k} = \frac{2 c'_k \mathrm{\underline{s}} - 2 (c'_k p'_k + c''_k p''_k)}{\mathrm{\underline{s}}^2 - 2 p'_k \mathrm{\underline{s}} + |\underline{p}_k|^2}
\end{equation}

There are different ways to build an equivalent circuit out of passive or active components, which can be dimensioned to match this target frequency response. Four such circuits are presented and analyzed in [[3](#link_ref3)], for example a series RLC circuit combined with a parallel current source controlled by the voltage over the capacitor.

This circuit is providing an admittance $\underline{Y}_\mathrm{RCL,I}(\mathrm{\underline{s}})$ that can be matched to the target response:

\begin{equation}
\underline{Y}_\mathrm{RCL,I}(\mathrm{\underline{s}}) = \frac{1/L \mathrm{\underline{s}} + b}{\mathrm{\underline{s}}^2 + R/L \mathrm{\underline{s}} + 1 / (LC)}
\end{equation}

This admittance matches the target response if $L = \frac{1}{2 c'_k}$, $R = \frac{-p'_k}{ c'_k}$, $C = \frac{2 c'_k}{|\underline{p}_k|^2}$ and $b = -2 (c'_k p'_k + c''_k p''_k)$ with $g_\mathrm{m} = bLC = \frac{b}{|\underline{p}_k|^2}$.

## Equivalent Circuit of a Vector Fitted $N$-Port
### Case 1: Vector Fitted S Parameters
The equivalent circuit of a $N$-port with a Vector Fitted S matrix consists of an interface network and a transfer network for each port. The figure below shows the structure of the interface and transfer networks of one such port $i$ with the external port voltage $V_i$ and port current $I_i$. The individual frequency responses $\underline{S}_{i,j}$ of the Vector Fit are reproduced with the equivalent admittances $\underline{Y}_{\mathrm{S},i,j}$ based on the fitting parameters $\mathbf{\underline{p}}$, $\mathbf{\underline{c}}$, $\mathbf{d}$ and $\mathbf{e}$ as described above.

### Case 2: Vector Fitted Y Parameters
Not implemented. See [Y Parameters](https://en.wikipedia.org/wiki/Admittance_parameters) for a general equivalent circuit.

### Case 3: Vector Fitted Z Parameters
Not implemented. See [Z Parameters](https://en.wikipedia.org/wiki/Impedance_parameters) for a general equivalent circuit.

## References

[1] B. Gustavsen and A. Semlyen, "Rational approximation of frequency domain responses by vector fitting," in IEEE Transactions on Power Delivery, vol. 14, no. 3, pp. 1052-1061, July 1999, DOI: [10.1109/61.772353](https://doi.org/10.1109/61.772353).

[2] https://www.sintef.no/projectweb/vectorfitting/

[3] G. Antonini, "SPICE equivalent circuits of frequency-domain responses," in IEEE Transactions on Electromagnetic Compatibility, vol. 45, no. 3, pp. 502-512, Aug. 2003, DOI: [10.1109/TEMC.2003.815528](https://doi.org/10.1109/TEMC.2003.815528).


# Differentiable Convex Optimization Layers: JAX Tutorial

```python
import cvxpy as cp
import numpy as np

import jax
import jax.numpy as jnp
from cvxpylayers.jax import CvxpyLayer

import matplotlib.pyplot as plt
plt.style.use('bmh')

np.set_printoptions(precision=2, suppress=True)
```

This notebook introduces the [JAX](https://github.com/google/jax)
library accompanying our
[NeurIPS paper](https://web.stanford.edu/~boyd/papers/pdf/diff_cvxpy.pdf)
on differentiable convex optimization layers with [CVXPY](https://www.cvxpy.org/).
A convex optimization layer solves a parametrized convex optimization problem
in the forward pass to produce a solution.
It computes the derivative of the solution with respect to the
parameters in the backward pass.
For more details, see our main repository at
[cvxgrp/cvxpylayers](https://github.com/cvxgrp/cvxpylayers)
and our
[blog post](https://locuslab.github.io/2019-10-28-cvxpylayers/).

## Parametrized convex optimization
Express problems of the form
$$
\begin{array}{ll} \mbox{minimize} & f_0(x;\theta)\\
\mbox{subject to} & f_i(x;\theta) \leq 0, \quad i=1, \ldots, m\\
& A(\theta)x=b(\theta),
\end{array}
$$
with variable $x \in \mathbf{R}^n$ and parameters $\theta\in\Theta\subseteq\mathbf{R}^p$

* objective and inequality constraints $f_0, \ldots, f_m$ are *convex* in $x$ for each $\theta$, *i.e.*, their graphs curve upward
* equality constraints are linear
* for a given value of $\theta$, find a value for $x$ that minimizes objective, while satisfying constraints
* we can efficiently solve these globally with near-total reliability

## Solution map
* Solution $x^\star$ is an implicit function of $\theta$
* When unique, define solution map as function
$x^\star = \mathcal S(\theta)$
* Need to call numerical solver to evaluate
* This function is often differentiable
* We show how to analytically differentiate this function, using the implicit function theorem
* Benefits of analytical differentiation: works with nonsmooth objective/constraints, low memory usage, don't compound errors

## CVXPY
* High level domain-specific language (DSL) for convex optimization
* Define variables, parameters, objective, and constraints
* Synthesize into problem object, then call solve method
* We've added derivatives to CVXPY (forward and backward)
* More information available [here](https://www.cvxpy.org/)

## CVXPYlayers

* Convert CVXPY problems into callable, differentiable Pytorch and Tensorflow modules in one line

## Applications of differentiable convex optimization
* [learning convex optimization models (structured prediction)](https://stanford.edu/~boyd/papers/learning_copt_models.html)
* learning decision-making policies (reinforcement learning)
  * [differentiable model-predictive control](https://arxiv.org/abs/1810.13400)
  * [convex optimization control policies](https://stanford.edu/~boyd/papers/learning_cocps.html)
* [machine learning hyper-parameter tuning and feature engineering](https://stanford.edu/~boyd/papers/lsat.html)
* [repairing infeasible or unbounded optimization problems](https://stanford.edu/~boyd/papers/auto_repair_cvx.html)
* [as protection layers in neural networks](http://physbam.stanford.edu/~fedkiw/papers/stanford2019-10.pdf)
* [learning constraints and rules](https://arxiv.org/abs/1703.00443)
* [custom neural network layers (sparsemax, csoftmax, csparsemax, LML)]( https://locuslab.github.io/2019-10-28-cvxpylayers/)
* [meta-learning through differentiable solvers](https://arxiv.org/abs/1904.03758)
* [and many more... ](https://github.com/bamos/thesis)

# ReLU, sigmoid, and softmax examples

This example follows the setup in
[our blog post](https://locuslab.github.io/2019-10-28-cvxpylayers/)
and shows how to implement standard activation functions
as differentiable convex optimization layers.
These are presented as an exercise to illustrate the
representational and modeling power of differentiable convex
optimization, and for these examples, provides no implementation
benefit over the original forms of these layers.

## The ReLU

As shown in [Section 2.4 here](https://github.com/bamos/thesis).
we can interpret the ReLU as projecting
a point $x\in\mathbb{R}^n$ onto the non-negative orthant as

$$
  \hat y = {\rm argmin}_y \;\; \frac{1}{2}||y-x||_2^2 \;\; {\rm s.t.} \;\; y\geq 0.
$$

The usual explicit solution can be obtained by taking the
first-order optimality conditions.

Using `cvxpylayers`, we are able to easily implement
this optimization problem as a JAX layer.
This is powerful since it does not require that
our optimization problem has an explicit closed-form
solution (even though the ReLU does) and this
also shows how easy we can now take this optimization
problem and tweak it if we wanted to do so, without
having to re-derive the appropriate solution and
backwards pass.
Since this is computing the same function as the ReLU,
we would expect that the derivative looks the same.
This is indeed true and we have made it so our optimization layer
can be differentiated through just like any other JAX layer
using automatic differentiation.

```python
# Implement the problem as usual with CVXPY:
n = 100
_x = cp.Parameter(n)
_y = cp.Variable(n)
obj = cp.Minimize(cp.sum_squares(_y-_x))
cons = [_y >= 0]
prob = cp.Problem(obj, cons)

# Create the JAX layer
relu_layer = CvxpyLayer(prob, parameters=[_x], variables=[_y])

# Differentiate through the JAX layer
def relu_layer_sum(x):
    return jnp.sum(relu_layer(x)[0])

drelu_layer = jax.grad(relu_layer_sum)

nrow, ncol = 1, 2
fig, axs = plt.subplots(nrow, ncol, figsize=(6*ncol, 4*nrow))
fig.suptitle('The ReLU')

x = jnp.linspace(-5, 5, num=n)

ax = axs[0]
ax.plot(x, relu_layer(x)[0])
ax.set_xlabel('$x$')
ax.set_ylabel('$f(x)$')

ax = axs[1]
ax.plot(x, drelu_layer(x))
ax.set_xlabel('$x$')
ax.set_ylabel(r'$\nabla f(x)$')
```

## The sigmoid

Similarly, **the sigmoid or logistic** activation
function $f(x) = (1+e^{-x})^{-1}$ can be
seen from a convex optimization perspective as discussed in
[Section 2.4 here](https://github.com/bamos/thesis).
The sigmoid
projects a point $x\in\mathbb{R}^n$ onto
the interior of the unit hypercube as

$$
  f(x) = {\rm argmin}_{0\lt y\lt 1} \; -x^\top y -H_b(y),
$$

where $H_b(y) = - \left(\sum_i y_i\log y_i + (1-y_i)\log (1-y_i)\right)$ is the
binary entropy function.
This can be proved by looking at the KKT conditions.

```python
n = 100

_x = cp.Parameter(n)
_y = cp.Variable(n)
obj = cp.Minimize(-_x.T@ _y - cp.sum(cp.entr(_y) + cp.entr(1.-_y)))
prob = cp.Problem(obj)

layer = CvxpyLayer(prob, parameters=[_x], variables=[_y])

# Differentiate through the JAX layer
def layer_sum(x):
    return jnp.sum(layer(x)[0])

dlayer = jax.grad(layer_sum)

nrow, ncol = 1, 2
fig, axs = plt.subplots(nrow, ncol, figsize=(6*ncol, 4*nrow))
fig.suptitle('The Sigmoid')

x = jnp.linspace(-5, 5, num=n)

ax = axs[0]
ax.plot(x, layer(x)[0])
ax.set_xlabel('$x$')
ax.set_ylabel('$f(x)$')

ax = axs[1]
ax.plot(x, dlayer(x))
ax.set_xlabel('$x$')
ax.set_ylabel(r'$\nabla f(x)$')
```

## The softmax
Lastly the **softmax** activation function
$f(x)\_j = e^{x\_j} / \sum\_i e^{x\_i}$ can
be implemented in JAX as follows.
This time we'll consider a single vector
instead of range of values since the softmax is
only interesting in higher dimensions.

We can again interpret the softmax as projecting a point
$x\in\mathbb{R}^n$ onto
the interior of the $(n-1)$-simplex
$$\Delta_{n-1}=\{p\in\mathbb{R}^n\; \vert\; 1^\top p = 1 \; \; {\rm and} \;\; p \geq 0 \}$$
as

$$
  f(x) = {\rm argmin}_{0\lt y\lt 1} \;\; -x^\top y - H(y) \;\; {\rm s.t.}\;\; 1^\top y = 1
$$

where $H(y) = -\sum\_i y\_i \log y\_i$ is the entropy function.
This is also proved in [Section 2.4 here](https://github.com/bamos/thesis)
by using the KKT conditions.
We can implement the variational form of the softmax with:

```python
d = 5
_x = cp.Parameter(d)
_y = cp.Variable(d)
obj = cp.Minimize(-_x.T @ _y - cp.sum(cp.entr(_y)))
cons = [np.ones(d).T @ _y == 1.]
prob = cp.Problem(obj, cons)
layer = CvxpyLayer(prob, parameters=[_x], variables=[_y])

key = jax.random.PRNGKey(0)
key, k1 = jax.random.split(key, 2)
x = jax.random.normal(k1, shape=(d,))
y, = layer(x)

dsoftmax = jax.grad(lambda x: jax.nn.softmax(x)[0])
dlayer = jax.grad(lambda x: layer(x)[0][0])

print('=== Softmax forward pass')
print('jax.nn.softmax: ', jax.nn.softmax(x))
print('Convex optimization layer: ', layer(x)[0])

print('\n=== Softmax backward pass')
print('jax.nn.softmax: ', dsoftmax(x))
print('Convex optimization layer: ', dlayer(x))
```

    === Softmax forward pass
    jax.nn.softmax:  [0.02 0.01 0.66 0.27 0.03]
    Convex optimization layer:  [0.02 0.01 0.66 0.27 0.03]

    === Softmax backward pass
    jax.nn.softmax:  [ 0.02 -0.   -0.01 -0.01 -0.  ]
    Convex optimization layer:  [ 0.02 -0.   -0.01 -0.01 -0.  ]

## Elastic-net regression example
This example has a similar setup to Section 6.1
of [our NeurIPS paper](https://arxiv.org/abs/1910.12430).
We are given training data $(x_i, y_i)_{i=1}^{N}$,
where $x_i\in\mathbf{R}$ are inputs and $y_i\in\mathbf{R}$ are outputs.
Suppose we fit a model for this regression problem by solving the elastic-net problem
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \frac{1}{N}\sum_{i=1}^N (ax_i + b - y_i)^2 + \lambda |a| + \alpha a^2,
\end{array}
\label{eq:trainlinear}
\end{equation}
where $\lambda,\alpha>0$ are hyper-parameters.

We hope that the test loss $\mathcal{L}^{\mathrm{test}}(a,b) =
\frac{1}{M}\sum_{i=1}^M (a\tilde x_i + b - \tilde y_i)^2$ is small, where
$(\tilde x_i, \tilde y_i)_{i=1}^{M}$ is our test set.

First, we set up our problem, where $\{x_i, y_i\}_{i=1}^N$, $\lambda$, and $\alpha$ are our parameters.

```python
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split

np.random.seed(0)
n = 2
N = 60
X, y = make_blobs(N, n, centers=np.array([[2, 2], [-2, -2]]), cluster_std=3)
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=.5)

Xtrain, Xtest, ytrain, ytest = map(
    jnp.array, [Xtrain, Xtest, ytrain, ytest])
m = Xtrain.shape[0]

a = cp.Variable((n, 1))
b = cp.Variable((1, 1))
X = cp.Parameter((m, n))
Y = ytrain[:, np.newaxis]

log_likelihood = (1. / m) * cp.sum(
    cp.multiply(Y, X @ a + b) - cp.logistic(X @ a + b)
)
regularization = - 0.1 * cp.norm(a, 1) - 0.1 * cp.sum_squares(a)
prob = cp.Problem(cp.Maximize(log_likelihood + regularization))
fit_logreg = CvxpyLayer(prob, [X], [a, b])

np.random.seed(0)
n = 1
N = 60
X = np.random.randn(N, n)
theta = np.random.randn(n)
y = X @ theta + .5 * np.random.randn(N)
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=.5)
Xtrain, Xtest, ytrain, ytest = map(
    jnp.array, [Xtrain, Xtest, ytrain, ytest])

m = Xtrain.shape[0]

# set up variables and parameters
a = cp.Variable(n)
b = cp.Variable()
X = cp.Parameter((m, n))
Y = cp.Parameter(m)
lam = cp.Parameter(nonneg=True)
alpha = cp.Parameter(nonneg=True)

# set up objective
loss = (1/m)*cp.sum(cp.square(X @ a + b - Y))
reg = lam * cp.norm1(a) + alpha * cp.sum_squares(a)
objective = loss + reg

# set up constraints
constraints = []

prob = cp.Problem(cp.Minimize(objective), constraints)
```

```python
# convert into pytorch layer in one line
fit_lr = CvxpyLayer(prob, [X, Y, lam, alpha], [a, b])
# this object is now callable with JAX arrays
fit_lr(Xtrain, ytrain, jnp.zeros(1), jnp.zeros(1))
```

    (DeviceArray([[-0.66]], dtype=float32), DeviceArray([0.14], dtype=float32))

```python
# sweep over values of alpha, holding lambda=0, evaluating the gradient along the way
alphas = jnp.logspace(-3, 2, 200)
test_losses = []
grads = []

def loss(alpha):
    a_tch, b_tch = fit_lr(Xtrain, ytrain, jnp.zeros(1), alpha)
    test_loss = jnp.mean((Xtest @ a_tch.flatten() + b_tch - ytest)**2)
    return test_loss

# derivative of the loss with respect to alpha
loss_grad = jax.grad(loss)

for alpha in alphas:
    test_losses.append(loss(alpha))
    grads.append(loss_grad(alpha))
```

```python
plt.semilogx()
plt.plot(alphas, test_losses, label='test loss')
plt.plot(alphas, grads, label='analytical gradient')
plt.plot(alphas[:-1], np.diff(test_losses) / np.diff(alphas), label='numerical gradient', linestyle='--')
plt.legend()
plt.xlabel("$\\alpha$")
plt.show()
```

```python
# sweep over values of lambda, holding alpha=0, evaluating the gradient along the way
lams = jnp.logspace(-3, 2, 200)
test_losses = []
grads = []

def loss(lam):
    a_tch, b_tch = fit_lr(Xtrain, ytrain, lam, jnp.zeros(1))
    test_loss = jnp.mean((Xtest @ a_tch.flatten() + b_tch - ytest)**2)
    return test_loss

# derivative of the loss with respect to lambda
loss_grad = jax.grad(loss)

for lam in lams:
    test_losses.append(loss(lam))
    grads.append(loss_grad(lam))
```

```python
plt.semilogx()
plt.plot(lams, test_losses, label='test loss')
plt.plot(lams, grads, label='analytical gradient')
plt.plot(lams[:-1], np.diff(test_losses) / np.diff(lams), label='numerical gradient', linestyle='--')
plt.legend()
plt.xlabel("$\\lambda$")
plt.show()
```

```python
# compute the gradient of the test loss wrt all the training data points, and plot
plt.figure(figsize=(10, 6))

def loss(X):
    a_tch, b_tch = fit_lr(
        X, ytrain, jnp.array([.05]), jnp.array([.05]), solver_args={"eps": 1e-8})
    test_loss = jnp.mean((Xtest @ a_tch.flatten() + b_tch - ytest)**2)
    return test_loss, a_tch, b_tch

# derivative of the loss with respect to lambda
loss_grad = jax.grad(lambda X: loss(X)[0], argnums=0)

test_loss, a_tch, b_tch = loss(Xtrain)
Xtrain_grad = loss_grad(Xtrain)

a_tch_test, b_tch_test = fit_lr(
    Xtest, ytest, jnp.array([0.]), jnp.array([0.]), solver_args={"eps": 1e-8})
plt.scatter(Xtrain, ytrain, s=20)
plt.plot([-5, 5], [-3*a_tch.item() + b_tch.item(),3*a_tch.item() + b_tch.item()], label='train')
plt.plot([-5, 5], [-3*a_tch_test.item() + b_tch_test.item(), 3*a_tch_test.item() + b_tch_test.item()], label='test')

for i in range(Xtrain.shape[0]):
    plt.arrow(Xtrain[i].item(), ytrain[i],
              -30.*Xtrain_grad[i][0], 0., color='k')
plt.legend()
plt.show()
```

```python
# move the training data points in the direction of their gradients, and see the train line get closer to the test line
plt.figure(figsize=(10, 6))
Xtrain_new = Xtrain - 30. * Xtrain_grad
a_tch, b_tch = fit_lr(
    Xtrain_new, ytrain, jnp.array([.05]), jnp.array([.05]), solver_args={"eps": 1e-8})
plt.scatter(Xtrain_new, ytrain, s=20)
plt.plot([-5, 5], [-3*a_tch.item() + b_tch.item(),3*a_tch.item() + b_tch.item()], label='train')
plt.plot([-5, 5], [-3*a_tch_test.item() + b_tch_test.item(), 3*a_tch_test.item() + b_tch_test.item()], label='test')
plt.legend()
plt.show()
```


# Quantization of Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing.

## Characteristic of a Linear Uniform Quantizer

The characteristics of a quantizer depend on the mapping functions $f(\cdot)$, $g(\cdot)$ and the rounding operation $\lfloor \cdot \rfloor$ introduced in the [previous section](introduction.ipynb). A linear quantizer bases on linear mapping functions $f(\cdot)$ and $g(\cdot)$. A uniform quantizer splits the mapped input signal into quantization steps of equal size. Quantizers can be described by their nonlinear in-/output characteristic $x_Q[k] = \mathcal{Q} \{ x[k] \}$, where $\mathcal{Q} \{ \cdot \}$ denotes the quantization process. For linear uniform quantization it is common to differentiate between two characteristic curves, the so called mid-tread and mid-rise. Both are introduced in the following.

### Mid-Tread Characteristic Curve

The in-/output relation of the mid-tread quantizer is given as

\begin{equation}
x_Q[k] = Q \cdot \underbrace{\left\lfloor \frac{x[k]}{Q} + \frac{1}{2} \right\rfloor}_{index}
\end{equation}

where $Q$ denotes the constant quantization step size and $\lfloor \cdot \rfloor$ the [floor function](https://en.wikipedia.org/wiki/Floor_and_ceiling_functions) which maps a real number to the largest integer not greater than its argument. Without restricting $x[k]$ in amplitude, the resulting quantization indexes are [countable infinite](https://en.wikipedia.org/wiki/Countable_set). For a finite number of quantization indexes, the input signal has to be restricted to a minimal/maximal amplitude $x_\text{min} < x[k] < x_\text{max}$ before quantization. The resulting quantization characteristic of a linear uniform mid-tread quantizer is shown below

The term mid-tread is due to the fact that small values $|x[k]| < \frac{Q}{2}$ are mapped to zero.

#### Example - Mid-tread quantization of a sine signal

The quantization of one period of a sine signal $x[k] = A \cdot \sin[\Omega_0\,k]$ by a mid-tread quantizer is simulated. $A$ denotes the amplitude of the signal, $x_\text{min} = -1$ and $x_\text{max} = 1$ are the smallest and largest output values of the quantizer, respectively.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

A = 1.2  # amplitude of signal
Q = 1/10  # quantization stepsize
N = 2000  # number of samples

def uniform_midtread_quantizer(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(np.abs(x) >= 1)
    x[idx] = np.sign(x[idx])
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

def plot_signals(x, xQ):
    e = xQ - x
    plt.figure(figsize=(10,6))
    plt.plot(x, label=r'signal $x[k]$')
    plt.plot(xQ, label=r'quantized signal $x_Q[k]$')
    plt.plot(e, label=r'quantization error $e[k]$')
    plt.xlabel(r'$k$')
    plt.axis([0, N, -1.1*A, 1.1*A])
    plt.legend()
    plt.grid()

# generate signal
x = A * np.sin(2*np.pi/N * np.arange(N))
# quantize signal
xQ = uniform_midtread_quantizer(x, Q)
# plot signals
plot_signals(x, xQ)
```

**Exercise**

* Change the quantization stepsize `Q` and the amplitude `A` of the signal. Which effect does this have on the quantization error?

Solution: The smaller the quantization step size, the smaller the quantization error is for $|x[k]| < 1$. Note, the quantization error is not bounded for $|x[k]| > 1$ due to the clipping of the signal $x[k]$.

### Mid-Rise Characteristic Curve

The in-/output relation of the mid-rise quantizer is given as

\begin{equation}
x_Q[k] = Q \cdot \Big( \underbrace{\left\lfloor\frac{ x[k] }{Q}\right\rfloor}_{index}  + \frac{1}{2} \Big)
\end{equation}

where $\lfloor \cdot \rfloor$ denotes the floor function. The quantization characteristic of a linear uniform mid-rise quantizer is illustrated below

The term mid-rise copes for the fact that $x[k] = 0$ is not mapped to zero. Small positive/negative values around zero are mapped to $\pm \frac{Q}{2}$.

#### Example - Mid-rise quantization of a sine signal

The previous example is now reevaluated using the mid-rise characteristic

```python
A = 1.2  # amplitude of signal
Q = 1/10  # quantization stepsize
N = 2000  # number of samples

def uniform_midrise_quantizer(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(np.abs(x) >= 1)
    x[idx] = np.sign(x[idx])
    # linear uniform quantization
    xQ = Q * (np.floor(x/Q) + .5)

    return xQ

# generate signal
x = A * np.sin(2*np.pi/N * np.arange(N))
# quantize signal
xQ = uniform_midrise_quantizer(x, Q)
# plot signals
plot_signals(x, xQ)
```

**Exercise**

* What are the differences between the mid-tread and the mid-rise characteristic curves for the given example?

Solution: The mid-tread and the mid-rise quantization of the sine signal differ for signal values smaller than half of the quantization interval. Mid-tread has a representation of $x[k] = 0$ while this is not the case for the mid-rise quantization.


 INF285/ILI285 Computación Científica

 COP-R

 Ecuación Cuadrática Matricial

# Librerías

```python
import numpy as np
import scipy.sparse.linalg as spla
```

# Pregunta

La tradicional ecuación cuadrática $\lambda\,x^2+\theta\,x+\gamma=0$ se estudió profundamente al comienzo del curso en el tema de errores de cancelación, esta ecuación se obtiene haciendo una manipulación algebraica conveniente, particularmente la solución es la siguiente:

\begin{equation}
x_{\pm}=\dfrac{-\theta\pm\sqrt{\theta^2-4\,\lambda\,\gamma}}{2\,\lambda}
\end{equation}

Uno podría extender la ecuación cuadrática de la siguiente forma:

\begin{equation}
\Lambda\,X^2+\Theta\,X+\Gamma=\underline{0}
\end{equation}

donde $\Lambda, \Theta, \Gamma$ y $X \in \mathbb{R}^{n\times n}$, $\underline{0}$ corresponde a la matriz nula de $n \times n$ y $X^2=X\,X$. Una posible simplificación puede ser:

\begin{equation}
X^2+\Lambda^{-1}\,\Theta\,X+\Lambda^{-1}\,\Gamma=\underline{0}
\end{equation}

Este caso en general es un poco más complejo de lo esperado, por lo cual estudiaremos la siguiente ecuación:

\begin{equation}
(X+B)^2=C+CX+X^2+D
\end{equation}

Los archivos de datos se obtienen desde el siguiente link: https://github.com/sct-utfsm/INF-285/tree/master/cop/r/data/

1. Considerando en este caso que $B=D=\underline{0}$ y $C=$C1.npy, obtenga la norma infinito de $X$, es decir $\|X\|_\infty$.
2. Considerando en este caso que $C=\underline{0}$, $B=$B2.npy y $D=B^2$, obtenga la norma infinito de $X$, es decir $\|X\|_\infty$.
3. Considerando en este caso que $B=$B3.npy, $C=$C3.npy y $D=\underline{0}$, obtenga la norma infinito de $X$, es decir $\|X\|_\infty$.

# Desarrollo

```python
# Loading data
C1 = np.load('data/C1.npy')
B2 = np.load('data/B2.npy')
B3 = np.load('data/B3.npy')
C3 = np.load('data/C3.npy')
n = C1.shape[0]
```

```python
# 1. (X+B)^2=C+C X+X^2 + D
#    X^2 = C+C X+X^2
#    0 = C+C X
#    -C = C X
#    -C^{-1}C=X
#    -I=X
X = -np.eye(n)
print(np.linalg.norm(X,np.inf))

# 2. (X+B)^2=C+C X+X^2 + D
#    (X+B)^2=X^2 + B^2
#    X^2 + B X + X B + B^2 = X^2 + B^2
#    B X + X B = 0
X = np.zeros((n,n))
print(np.linalg.norm(X,np.inf))

# 3. (X+B)^2=C+C X+X^2 + D
#    (X+B)^2=C+C X+X^2
#    X^2+B X +X B + B^2=C+C X+X^2
#    B X +X B + B^2=C+C X
#    (B - C) X +X B = C - B^2
def compute_matrix_vector_product(x,B,C,n):
    X = np.reshape(x,(n,n))
    out = np.dot(B-C,X)+np.dot(X,B)
    return out.flatten()
afun = spla.LinearOperator((n**2, n**2), matvec = lambda x: compute_matrix_vector_product(x,B3,C3,n))
x, exitCode = spla.gmres(afun, (C3-np.dot(B3,B3)).flatten(), tol=1e-10)
X_GMRes = np.reshape(x,(n,n))
print(np.linalg.norm(X_GMRes,np.inf))
```

    1.0
    0.0
    29.40492924190162


# **Finite difference method on ODE**

## **Exercise:**

Consider ODE

\begin{equation}
    - \epsilon u^{''} + u = x, \, \forall x \in (0, 1) \, u(0) = u(1) = 0,
\end{equation}

with $\epsilon = 10^{-10}$. This example is taken from Example $5.2$ of

- Qingshuo Song, George Yin, Zhimin Zhang, An epsilon-uniform finite element method for singularly perturbed boundary value problems.

Instead of FEM, we are going to discuss CFD solution of the above ODE. Answer the following questions:

(i) Prove that
\begin{equation}
    u(x) = x - \frac{\exp (\frac{x-1}{\sqrt{\epsilon}}) - \exp (- \frac{x+1}{\sqrt{\epsilon}})}{1 - \exp(- \frac{2}{\sqrt{\epsilon}})}
\end{equation}
is the unique solution.

(ii) Using CFD on above ODE, find out the matrix $L^{h}$ and vector $R^{h}f$, such that the numerical solution satisfies $L^{h}u^{h} = R^{h}f$.

(iii) Prove the consistency and stability of $L^{h}$.

(iv) Compute CFD solution $u^{h}$ with $h = \frac{1}{5}$. Compare with the FEM solution of the paper, which one is better?

## **Solution:**

For the question (iv):

When $ h = \frac{1}{5}$ and $\epsilon = 10^{-10}$, we can get the exact value of $s = 1 + 50 \times 10^{-10}$ and $r = 25 \times 10^{-10}$. And we have $R^{h} f = (0, \frac{1}{5}, \frac{2}{5}, \frac{3}{5}, \frac{4}{5}, 0)$ and

$$ A =
 \begin{bmatrix}
 1  &  0  &  0  &  0  &  0  & 0  \\
 -r &  s  & -r  &  0  &  0  & 0  \\
 0  & -r  & s   &  -r &  0  & 0  \\
 0  &  0  & -r  &  s  & -r  & 0   \\
 0  &  0  & 0   & -r  &  s  & -r  \\
 0  &  0  & 0   &  0  &  0  &  1
\end{bmatrix}
$$

Then by the $L^{h} u^{h} = R^{h}f$, we can get the numerical solution of the above ODE.

```
# Get the exact value of s and r

e = 10**(-10)
s = 1 + 50 * e
r = 25 * e

print("the value of e is", e)
print("the value of s is", s)
print("the value of r is", r)
```

    the value of e is 1e-10
    the value of s is 1.000000005
    the value of r is 2.5e-09

```
import numpy as np
from scipy.linalg import solve

a = np.array([[1, 0, 0 ,0, 0, 0], [-r, s, -r ,0, 0, 0],\
              [0, -r, s, -r, 0, 0], [0, 0, -r, s, -r, 0],\
              [0, 0, 0, -r, s, -r], [0, 0, 0 ,0, 0, 1]])
b = np.array([0, 1/5, 2/5, 3/5, 4/5, 0])
numerical_solution = solve(a, b)
print(numerical_solution)
```

    [0.  0.2 0.4 0.6 0.8 0. ]

```
# Get the exact value of the ODE

def exact_value(x, e):
  u = x - (np.exp((x - 1)/np.sqrt(e)) - np.exp(- (x+1)/np.sqrt(e))) / (1 - np.exp(- 2 / np.sqrt(e)))
  return u

exact_value_list = []
e = 10**(-10)
exact_value_list = [exact_value(i/5, e) for i in range(6)]
print(exact_value_list)

```

    [0.0, 0.2, 0.4, 0.6, 0.8, 0.0]

```
# Plot the exact value with the numerical solution of the ODE

import matplotlib.pyplot as plt
from pylab import plt
plt.style.use('seaborn')
%matplotlib inline

x = np.array([0, 1/5, 2/5, 3/5, 4/5, 1])
plt.plot(x, numerical_solution, 'o', label='Numerical solution of the ODE')
plt.plot(x, exact_value_list, label = 'Exact value of the ODE')

plt.ylabel('u(x)')
plt.xlabel('x')
plt.legend();
```


# In this demo

This demo presents the design and functionality of `Signal`s, which are the core objects for representing coefficients in models. `Signal`s are structured to represent the mathematical formula:
\begin{equation}
    s(t) = Re[f(t)e^{i(2 \pi \nu t + \phi)}],
\end{equation}
where
- $f(t)$ is a complex-valued envelope,
- $\nu$ is the carrier frequency, and
- $\phi$ is the phase.

Using this formula as a basis for the representation of `Signal`s is motivated by:
- This is a very common representation when modelling control sequences in quantum systems; in particular, for representing signals whose Fourier spectrum is localized about a particular frequency.
- It is general; even when the previous point is not immediately applicable, the full generality of the envelope $f(t)$ allows for specification of arbitrary real-valued functions.

Taking the above as a given, the design for `Signal`s is further motivated to facilitate different modes of operation:
- Completely general envelopes: The user specifies $f(t)$ as python-callable function.
    - This allows for arbitrary signal definitions, including continuous functions.
- Piecewise constant, or *discrete* envelopes, in which $f(t)$ is a piecewise constant function of time.
    - This allows for sample-based construction and computation on `Signals`.
    - Signal processing and evaluation can be performed purely as collective array-based operations, which is more efficient than the function-based products required in the fully general case.

```python
import numpy as np
from qiskit_dynamics.signals import Signal, DiscreteSignal, SignalList, DiscreteSignalSum
from matplotlib import pyplot as plt

%matplotlib inline
```

# 1. `Signal` and its subclasses

## 1.1 `Signal`

The most general signal object takes the envelope as a python callable function.

```python
sig = Signal(envelope=lambda t: t**2, carrier_freq=2.)
```

We can evaluate the signal by directly calling the object, or we can evaluate its complex envelope or full complex value.

```python
print(sig(1.1))
print(sig.envelope(1.1))
print(sig.complex_value(1.1))
```

    0.373910563193685
    1.2100000000000002
    (0.373910563193685+1.1507783847171364j)

The evaluation functions are all setup for vectorized evaluation, and we assume that the user-specified envelope is vectorized. If not, it can be vectorized using `numpy.vectorize`.

```python
t_vals = np.array([0.1, 1.1])
print(sig(t_vals))
print(sig.envelope(t_vals))
print(sig.complex_value(t_vals))
```

    [0.00309017 0.37391056]
    [0.01 1.21]
    [0.00309017+0.00951057j 0.37391056+1.15077838j]

We can also plot the evaluation functions.

```python
# plot the signal
sig.draw(0, 5, 100)
```

```python
# plot the complex value
sig.draw(0, 5, 100, function='complex_value')
```

```python
# plot just the envelope
sig.draw(0, 5, 100, function='envelope')
```

## 1.2 Constant `Signal`

This is not a subclass, but an alternate mode of behaviour for `Signal`. Generally a user shouldn't need to do this, but internally, constant values are converted to a `Signal` with constant envelope and zero carrier frequency/phase.

This can be instantiated by passing a numeric value to the constructor of `Signal`.

```python
constant = Signal(2.)
constant.draw(0, 5, 100)
```

```python
constant.is_constant
```

    True

## 1.3 `DiscreteSignal(Signal)`

A `DiscreteSignal` represents a `Signal` whose envelope is piecewise constant, and is specified as an array of samples.

```python
discrete_signal = DiscreteSignal(dt=0.5, samples=[1., 2., 3.])
discrete_signal.draw(0, 1.5, 100)
```

A `DiscreteSignal` can still have an analog carrier.

```python
discrete_signal = DiscreteSignal(dt=0.5, samples=[1., 2., 3.], carrier_freq=1.)
discrete_signal.draw(0, 1.5, 100)
```

# 2. Algebraic operations on `Signal`s

It is natural to add and multiply signals, as these mathematical operations correspond to the behaviour of physical circuit elements. To facilitate this, we introduce the `SignalSum` object, which represents a sum of `Signal`s. The purpose of this class is to allow higher level algebraic operations on `Signal`s while preserving frequency information.

Algebraic operations have been implemented using python dunder methods, so that a user can add and multiply signals using standard python syntax.

## 2.1 Adding general `Signal`s

```python
sig1 = Signal(lambda t: t, carrier_freq=10.)
sig2 = Signal(lambda t: 10 * t, carrier_freq=20.)

# add using standard python syntax
sig_sum = sig1 + sig2
```

`sig_sum` is now a `SignalSum` object:

```python
print(type(sig_sum))
str(sig_sum)
```

    <class 'qiskit_dynamics.signals.signals.SignalSum'>

    'Signal(carrier_freq=10.0, phase=0.0) + Signal(carrier_freq=20.0, phase=0.0)'

This is a container class storing the original signals in the sum in the `components` attribute:

```python
sig_sum.components
```

    [<qiskit_dynamics.signals.signals.Signal at 0x1494616d0>,
     <qiskit_dynamics.signals.signals.Signal at 0x1495b8f40>]

We can interact with this object in the same way as a regular `Signal`, e.g. we can draw it:

```python
sig_sum.draw(0, 1, 100)
```

We can produce the same plot by evaluating the individual signals in the sum separately:

```python
t_vals = np.linspace(0, 1, 100)
plt.plot(t_vals, sig1(t_vals) + sig2(t_vals))
```

## 2.2 Multiplying signals

Multiplication of signals is implemented via the formula:

\begin{equation}
    Re[f(t)e^{i(2 \pi \nu t + \phi)}] \times Re[g(t)e^{i(2 \pi \omega t + \psi)}]
         = Re[\frac{1}{2} f(t)g(t)e^{i(2\pi (\omega + \nu)t + (\phi + \psi))} ]
          + Re[\frac{1}{2} f(t)\overline{g(t)}e^{i(2\pi (\omega - \nu)t + (\phi - \psi))} ].
\end{equation}

When two signals are multiplied, a `SignalSum` representing the right hand side of the above equation is constructed.

```python
sig_prod = sig1 * sig2
str(sig_prod)
```

    'Signal(carrier_freq=30.0, phase=0.0) + Signal(carrier_freq=-10.0, phase=0.0)'

We see that the above product consists of a sum of signals centred at frequencies `4.0` and `0.0`.

```python
sig_prod(2.) - sig1(2.) * sig2(2.)
```

    0.0

## 2.3 `DiscreteSignalSum` and operations on `DiscreteSignal`s

The previous sections show how addition/multiplication work for general `Signal`s. In the general case, as the envelope is a user-specified function, we can only multiply envelopes as functions. For `DiscreteSignal`s with compatible sample strcuture, we can multiply envelopes as arrays.

```python
discrete_sig1 = DiscreteSignal(dt=1., samples=[1., 2., 3.], carrier_freq=2.)
discrete_sig2 = DiscreteSignal(dt=1., samples=[4j, 5j, 6j], carrier_freq=2.)
discrete_prod = discrete_sig1 * discrete_sig2
```

```python
print(type(discrete_prod))
str(discrete_prod)
```

    <class 'qiskit_dynamics.signals.signals.DiscreteSignalSum'>

    'DiscreteSignal(dt=1.0, carrier_freq=4.0, phase=0.0) + DiscreteSignal(dt=1.0, carrier_freq=0.0, phase=0.0)'

A `DiscreteSignalSum` stores the samples of its constituent components in a single array.

```python
discrete_prod.samples
```

    Array([[0.+2.j, 0.-2.j],
           [0.+5.j, 0.-5.j],
           [0.+9.j, 0.-9.j]], backend='numpy')

## 2.4 Frequency preservation

One of the benefits of `SignalSum` is that it "preserves frequency information", in the sense that, in applications where each `Signal` has a well-defined central frequency, the algebraic operations will result in `SignalSum`s whose components have well-defined central frequencies. This is useful for operations that require analyzing the frequency components of signals, e.g. the rotating wave approximation.

We can visualize this by examining the FFT of the product of signals above. The product of signals results in a component whose central frequency is at the sum of the frequencies in the product, as well as a component at the difference:

```python
N = 1000
T = 10.
t_vals = np.linspace(0., T, N)
fft_freqs = np.fft.fftfreq(N, T / N)
sig_prod_fft = np.fft.fft(sig_prod(t_vals))
plt.plot(fft_freqs, np.real(sig_prod_fft), fft_freqs, np.imag(sig_prod_fft))
```

We see components at `30.` and `10.`, as expected. Now, we can look at the FFT of each component of the product:

```python
N = 1000
T = 10.
t_vals = np.linspace(0., T, N)
fft_freqs = np.fft.fftfreq(N, T / N)
sig_prod0_fft = np.fft.fft(sig_prod[0](t_vals))
plt.plot(fft_freqs, np.real(sig_prod0_fft), fft_freqs, np.imag(sig_prod0_fft))
```

We see the first component in the sum is the term whose frequency is at the sum of the product term frequencies.

```python
N = 1000
T = 10.
t_vals = np.linspace(0., T, N)
fft_freqs = np.fft.fftfreq(N, T / N)
sig_prod1_fft = np.fft.fft(sig_prod[1](t_vals))
plt.plot(fft_freqs, np.real(sig_prod1_fft), fft_freqs, np.imag(sig_prod1_fft))
```

The second component in the sum is at the difference.

# 3. IQ mixer via algebraic operations

```python
# define some envelope function
def env(t):
    return t * (1 - t)

w_d = 2.
w_lo = 1.9
w_if = 0.1

I = Signal(env, carrier_freq=w_if, name='I') # envelope times cosine at w_if
Q = Signal(env, carrier_freq=w_if, phase=-np.pi/2, name='Q') # envelope times sine at w_if

slo_I = Signal(1., carrier_freq=w_lo, name='slo_I') # local oscillator for I
slo_Q = Signal(1., carrier_freq=w_lo, phase=-np.pi/2, name='slo_Q') # local oscillator for Q
```

The IQ output is a sum of two products. Hence, it produces a `SignalSum` with 4 outputs.

```python
IQ_output = (I * slo_I) - (Q * slo_Q)
str(IQ_output)
```

    'Signal(carrier_freq=2.0, phase=0.0) + Signal(carrier_freq=-1.7999999999999998, phase=0.0) + Signal(carrier_freq=2.0, phase=-3.141592653589793) + Signal(carrier_freq=-1.7999999999999998, phase=0.0)'

```python
IQ_output.draw(0, 1, 100, title='IQ output')
```

The $0^{th}$ and $2^{nd}$ terms in the sum are at frequency `2.0`, which we can plot:

```python
IQ_output[[0, 2]].draw(0, 1, 100, title='freq==2.0 terms')
```

The $1^{st}$ and $3^{rd}$ terms are at frequency `1.8`:

```python
IQ_output[[1, 3]].draw(0, 1, 100, title='freq==1.8 terms')
```

This demonstrates the cancellation at the lower sideband, with the remaining signal at the upper sideband. When modelling imperfect operation of an IQ mixer, this cancellation may not occur perfectly.

# 4. `SignalList`

A model of a Hamiltonian contains a list of `Signal`s representing coefficients of operators. Each coefficient may be any subclass of `Signal` (including `SignalSum` and `DiscreteSignalSum`). The `SignalList` object is similar to `SignalSum`, in that it stores a list of `Signal`s, but the evaluation functions return an array of each component evaluated separately.

```python
sig_list = SignalList([2. + Signal(lambda t: t, carrier_freq=1.),
                       1.,
                       DiscreteSignal(dt=1., samples=[1., 2., 3.])])
```

```python
sig_list(2.)
```

    array([4., 1., 3.])

`SignalList` also contains a helper `drift` property for evaluating the constant part:

```python
sig_list.drift
```

    Array([2., 1., 0.], backend='numpy')

# 5. Transfer functions

```python
from qiskit_dynamics.signals.transfer_functions import Convolution
```

```python
def gaus(t):
    sigma = 10
    dt = 0.1
    return 2.*dt/np.sqrt(2.*np.pi*sigma**2)*np.exp(-t**2/(2*sigma**2))
```

```python
ts = np.linspace(0, 100, 1000)
np.sum([gaus(_) for _ in ts])
```

    1.0029894228040142

```python
plt.plot(ts, gaus(ts))
```

```python
convolution = Convolution(gaus)
```

```python

samples = [0. if t<20. or t > 80. else 1 for t in ts]
piecewise_const = DiscreteSignal(dt=ts[1]-ts[0], samples=samples, carrier_freq=0.0, start_time=0)
```

```python
piecewise_const.draw(0, 100, 200)
piecewise_const.carrier_freq
```

```python
convolved = convolution(piecewise_const)
convolved.draw(0, 150, 200)
convolved.carrier_freq
```

```python
convolved.carrier_freq = 0.3
convolved.draw(0, 200, 500)
convolved.carrier_freq
```

# 6. Fancy indexing

`Signal` subclasses representing collections of signals can be subscripted in the same manner as a 1d numpy array.

```python
sig_sum = 1. + Signal(lambda t: t) + DiscreteSignal(dt=0.5, samples=[1, 2, 3, 4, 5])
str(sig_sum)
```

    'Signal(carrier_freq=0.0, phase=0.0) + Constant(1.0) + DiscreteSignal(dt=0.5, carrier_freq=0.0, phase=0.0)'

We can access the first two components of the sum by subscripting with the list `[0, 1]`.

```python
str(sig_sum[[0, 1]])
```

    'Signal(carrier_freq=0.0, phase=0.0) + Constant(1.0)'

This works for both `DiscreteSignalSum` and `SignalList`. Subscripting these results in an object of the appropriate type.

# 7. Sampling

We can generate `DiscreteSignal`s and `DiscreteSignalSum`s by sampling, respectively, general `Signal`s and `SignalSum`s. When performing this discretization, we can choose to sample the carrier as well, or keep it analog.

```python
sig = Signal(lambda t: t, carrier_freq=2.)

disc1 = DiscreteSignal.from_Signal(sig, dt=0.1, start_time=0., n_samples=10)
disc2 = DiscreteSignal.from_Signal(sig, dt=0.1, start_time=0., n_samples=10, sample_carrier=True)
```

```python
# plot the signal and the discretized version when keeping the carrier analog
sig.draw(0, 1, 100)
disc1.draw(0, 1, 100)
```

```python
# plot the signal and the discretized version when also sampling the carrier
sig.draw(0, 1, 100)
disc2.draw(0, 1, 100)
```

The same can be done with `DiscreteSignalSum`.

```python
sig1 = Signal(lambda t: t, carrier_freq=2.)
sig2 = Signal(lambda t: 1j * t**2, carrier_freq=3.)
sig_sum = sig1 + sig2

disc1 = DiscreteSignalSum.from_SignalSum(sig_sum, dt=0.1, start_time=0., n_samples=10)
disc2 = DiscreteSignalSum.from_SignalSum(sig_sum, dt=0.1, start_time=0., n_samples=10, sample_carrier=True)
```

```python
sig_sum.draw(0, 1, 100)
disc1.draw(0, 1, 100)
```

```python
sig_sum.draw(0, 1, 100)
disc2.draw(0, 1, 100)
```


# PS2, part 1: Welfare effects of capital injection

Import packages, classes and settings:

```python
import sys
import numpy as np
import math
import itertools
from scipy import optimize
import scipy.stats as stats
import PS2 as func
# For plots:
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
```

The exercise consists of the core setup:
* Two periods, $t=0,1$,
* Two types of agents: Consumers and entrepreneurs.
* **Consumers:**
    * Linear utility $u_c = c_0^c+c_1^c$.
    * Large endowment of consumption goods in each period: $e$.
    * Labor endowment of $e_l=1$. Inelastically supplied in a competitive labor market at the price $w_1$.
* **Entrepreneurs:**
    * Linear utility $u_e = c_0^e+c_1^e$.
    * Small endowment $n_0$ in period 0.
    * Borrow $b_1$, invest in capital $k_1$.
    * Period 1 they hire labor $l_1$ at wage $w_1$.
    * Entrepreneurs utilize capital and labor in $t=1$ with Cobb-Douglas production function
    $$ \begin{align}
    y_1 = k_1^{\alpha}l_1^{1-\alpha} \notag
    \end{align} $$
    * **Entrepreneurs are borrowing constrained:**
    $$\begin{align}
    b_1\leq \lambda \left(y_1-w_1l_1\right), && \lambda\in (0,1) \notag
    \end{align} $$
* Assume that endowment of consumption goods $(e)$ is *sufficiently* large, for the interest rate to be 1 in equilibrium. (We will show what *sufficiently* means here).

## Q1: Three parts. (i) State the entrepreneur's problem. (ii) Argue that entrepreneurs will choose to maximize profits in period 1, a linear function of $k_1$, and (3) restate entrepreneur's problem as a simple linear problem.

### (i) Entrepreneur problem:

Go through the steps:

1. For the entrepreneur write up the maximization problem. This inludes:
    * Objective function, what is maximized? Constraints on the problem are
    * The budget for $t=0$, (with "=")
    * the budget for $t=1$, (with "=")
    * the production function, (with "=")
    * the credit constraint (with "$\leq$").
    Let us refer ot this as **(Problem 1)**.

2. Next, substitute for budgets $t=0,t=1$ into the utility function.
3. Substitute for the production function in both the function from step 2., and the credit constraint. We end up with what we refer to as **problem 2**, that should look like something like this:

$$\begin{align}
    \max_{k_1,l_1,b_1} n_0+k_1^{\alpha}l_1^{1-\alpha}-k_1-w_1l_1, && s.t. b_1\leq \lambda \left(k_1^{\alpha}l_1^{1-\alpha}-w_1l_1\right). \tag{Problem 2}
\end{align}$$

### (ii) Optimal choice of labor:

Note that profits in $t=1$ is given by:

$$\begin{align}
    \pi_1 = k_1^{\alpha}l_1^{1-\alpha}-w_1l_1.
\end{align}$$

Find the first order conditions in **(Problem 2)** defined above, and show that this implies:

$$\begin{align}
    w_1 = (1-\alpha)k_1^{\alpha}l_1^{-\alpha}
\end{align}$$

and thus also

$$\begin{align}
w_1l_1^* = (1-\alpha)k_1^{\alpha}(l_1^*)^{1-\alpha}, \tag{Labor Share}
\end{align}$$

and

$$\begin{align}
    l_1^* = \dfrac{(1-\alpha)^{1/\alpha}k_1}{w_1^{1/\alpha}}. \tag{$l^*$}
\end{align}$$

Use this to write the **profit function** as:

$$\begin{align}
    \pi_1(l=l^*) &= \underbrace{\dfrac{\alpha (1-\alpha)^{(1-\alpha)/\alpha}}{w_1^{(1-\alpha)/\alpha}}}_{\equiv R(w_1)}k_1 \tag{Profit}
\end{align}$$

### (iii) Restate as linear problem.

Use the results above. Refer to this linear problem as **(Problem 3)**.

## Q2: Derive entrepreneur's demand for capital, and argue how the solution depends on $\lambda$ and $R(w_1)$.

Problem $3$ is linear in $k_1$. Thus capital demand depends on the range of the interest rate. Explain the three potential solutions:

* If $R(w_1)<1$?
* If $\lambda R(w_1)>1$?
* If $\lambda R(w_1)<1<R(w_1)$?
* If $R(w_1)=1$?

In the figures below you can see how the demand changes with parameter values of e.g. $\lambda$. (The other line in the graph is $(\alpha k^{\alpha-1})$ which also has to hold in equilibrium)

.

.

.

.

.

.

.

Hint: Capital demand in the different ranges are given by (not necessarily correct order compared to bullets above)

$$\begin{align}
     k_1^* &= \dfrac{n_0}{1-\lambda R(w_1)} \\
     k_1^* &= 0 \\
     k_1^* &= \infty \\
      k_1^*&\in\left[0,\mbox{ }\dfrac{n_0}{1-\lambda}\right] \\
\end{align}$$

**Plot demand:**

```python
Model = func.TwoAgent_GE()
Model.plot_interactive_ds() # plots the demand
```

    interactive(children=(SelectionSlider(description='Endowment, $n$', options=(0.0, 0.02, 0.04, 0.06, 0.08, 0.1,…

**Plot equilibrium:**

```python
Model.plot_interactive_cequi()
```

    interactive(children=(SelectionSlider(description='Borrowing constr., $\\lambda$', options=(0.0, 0.04, 0.08, 0…

## Q3: Show the cutoff $n^*$, where $n_0 \geq n^*$$ implies that the first-best level of capital can be financed, and the borrowing constraint is not binding.

You can think of the following:

* Impose $l_1=1$ in equilibrium (endowment of labor from consumers, supplied inelastically).
* Define the first-best level of capital, as the one that **maximizes net ouptut** $(y_1-k_1)$.
* In equilibrium we further use that the market for capital is competitive, implying that the rate of return on capital has to follow:

$$\begin{align}
    R(w_1) = \alpha k_1^{\alpha-1}, && \Rightarrow && k_1 = \left(\dfrac{\alpha}{R(w_1)}\right)^{1/(1-\alpha)}
\end{align}$$

Given this and the capital demand function from Q2, when is the first-best level feasible?

## Q4: Show that if n<n* there is an equilibrium where the entrepreneurs are constrained and $k_1$ becomes an increasing function of $n_0$.

It follows from Q2-Q3 that the threshold exists, and that the entrepreneur is credit-constrained in this region. Now:

* Use the relevant capital demand from Q2.
* Plug in the expression for equilibrium $R(w_1)$.
* Solve for $k$ in this expression.

**Note:** When the entrepreneur is credit-constrained, $k_1$ is not only an increasing function of $n_0$, the effect of increased equity is **more** than 1-to-1. Why is this the case?

## Q5: Introduce a lump-sum tax on consumers in period 0 that transfers wealth to entrepreneurs. Derive expected utility as a function of the lump sum tax rate.

For Q5-Q7 it might be useful to consider the following plot of utility in equilibrium with varying tax rate $\tau$.

```python
Model.plot_u_ce_tau()
```

Hints:

* Split up the consideration into the region where E is credit-constrained, and where E is not.
* You can use all of the solutions from above and simply replace $n_0$ with $n_0+\tau$ for the entrepreneur.

You do not need to arrive at exactly this expression, but show that the effect of an increase in $\tau$ has the following effect on utility of entrepreneurs:

$$\begin{align}
    \dfrac{\partial U_e}{\partial \tau} = \left\lbrace \begin{array}{ll}  1 & n_0+\tau\geq n^* \\
    \dfrac{\alpha R(w_1)(1-\lambda)}{1-\alpha \lambda R(w_1)} & n_0+\tau<n^* \end{array}\right.
\end{align}$$

## Q6: Show that the relationship between consumer utility and the lump-sum tax is non-monotone. Show that for sufficiently small $n_0$ the tax can be pareto-improving.

Entrepreneurs always benefit from the tax.

For consumers the tax includes three potential effects. Show that the effect on utility is given by:

$$\begin{align}
    \dfrac{\partial U_c}{\partial \tau} = \left\lbrace \begin{array}{ll} \color{red}{-1}, & n_0+\tau\geq n^* \\ \color{red}{-1} +\color{blue}{(1-\alpha)R(w_1)}\color{blueviolet}{\dfrac{\partial k_1}{\partial \tau}}, & n_0+\tau<n^*\end{array}\right.
\end{align}$$

Explain the three effects (here marked with different colors).

## Q7: Discuss how this justifies government's capital injection during the financial crisis. What features might the model be missing?

## Q8: How does key findings in Q6 depend on the type of borrowing constraint?


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## notation for differentation
We'll mostly use Lagrange's notation, the first three deriviatives of a function $f$ are denoted $f'$, $f''$ and $f'''$. After that we'll use $f^{(4)}, f^{(5)}, \ldots, f^{(n)}$.

## limits
Below is a function $f$ with two cases.

$
f(x) = \begin{cases}
x + 1, & \text{if $x \gt 0$} \\
-x + 2, & \text{if $x \lt 0$}
\end{cases}
$

Notice that for this function, the value of $f$ for when $x = 0$ is undefined. Also note that this is a *discontinuous* function.

```python
c1 = lambda x: x + 1
c2 = lambda x: -x + 2
x1 = np.linspace(0.01, 2, 10)
x2 = np.linspace(-2, -0.01, 10)
plt.plot(x1, c1(x1), label=r"$y = x + 1$")
plt.plot(x2, c2(x2), label=r"$y = -x + 2$")
plt.plot(0, 2, 'wo', markersize=7)
plt.plot(0, 1, 'wo', markersize=7)
ax = plt.axes()
ax.set_ylim(0, 4)
plt.legend(loc=3)
```

Even though the value of $y$ for when $x = 0$ is undefined we can say something about the *limits* of this function.

#### right-hand limit
If we approach from a position where $x \gt 0$ we can say that as $x$ approaches 0 the limit of the function is $x + 1$.

$$\lim_{x^+\to 0}f(x) = \lim_{x\to 0}x + 1 = 1$$

#### left-hand limit
On the other hand, if we approach from a position where $x \lt 0$ then we see that as $x$ approaches 0 the limit of the function is $-x + 2$.

$$\lim_{x^-\to 0}f(x) = \lim_{x\to 0}-x + 2 = 2$$

And now that we have defined limits we can also define what it means for a function to be *continuous* at a certain value. A function $f$ is continuous at $x_0$ when $\lim_{x\to x_0}f(x) = f(x_0)$

## discontinuous functions
### jump discontinuity
The limits of a funcion $f(x)$ at $x_0$ exist but are not equal. This is basically the example from above.

$$\lim_{x^+\to x_0}f(x) \neq \lim_{x^-\to x_0}f(x)$$

### removable discontinuity
Let $g(x) = \frac{\sin x}{x}$ and $h(x) = \frac{1 - \cos x}{x}$

```python
g = lambda x: np.sin(x) / x
h = lambda x: (1 - np.cos(x)) / x
x = np.linspace(-3 * np.pi, 3 * np.pi, 100)
ax = plt.axes()
ax.set_xlim(-3 * np.pi, 3 * np.pi)
ax.set_ylim(-1, 1.25)
plt.plot(x, g(x), label=r"$y = g(x) = \frac{\sin x}{x}$")
plt.plot(x, h(x), label=R"$y = h(x) = \frac{1 - \cos x}{x}$")
plt.plot(0, 1, 'wo', markersize=7)
plt.plot(0, 0, 'wo', markersize=7)
plt.legend(loc=4)
```

Note that dividing by zero is an undefined operation, both of these functions are undefined for when $x = 0$ so we'll have two little circles in the plot. However we can see that $\lim_{x^+\to 0}g(x) = 1$ and that $\lim_{x^-\to 0}g(x) = 1$ so generally we can say that $\lim_{x\to 0}g(x) = 1$. We can also see that $\lim_{x^+\to 0}h(x) = 0$ and $\lim_{x^-\to 0}h(x) = 0$ so $\lim_{x\to 0}h(x) = 0$.

Because for both functions $\lim_{x^+\to 0} = \lim_{x^-\to 0}$ we can say that these functions have a removable discontinuity at $x = 0$

### infinite discontinuity
This time we'll use $y = f(x) = \frac{1}{x}$

```python
f = lambda x: 1/x
x1 = np.linspace(-0.5, -0.01, 1000)
x2 = np.linspace(0.01, 0.5, 1000)
ax = plt.axes()
#ax.spines['left'].set_position(('data', 0))
#ax.spines['bottom'].set_position(('data', 0))
ax.set_xlim(-0.1, 0.1)
plt.plot(x1, f(x1), 'b')
plt.plot(x2, f(x2), 'b')
```

Now we see that $\lim_{x^+\to 0}\frac{1}{x} = \infty$ and $\lim_{x^-\to 0}\frac{1}{x} = -\infty$ and even though some people might say that these limits are undefined they are going in a definite direction so if able we should specify what they are.

However we **cannot** say that $\lim_{x\to 0}\frac{1}{x} = \infty$ even though this is sometimes done it's usually because people are sloppy and only considering $y = \frac{1}{x}$ for when $x \gt 0$.

There's an interesting thing we can observe when we plot the derivative of this function.

```python
f0 = lambda x: 1/x
f1 = lambda x: -1/x**2
x1 = np.linspace(-0.5, -0.01, 1000)
x2 = np.linspace(0.01, 0.5, 1000)

p1 = plt.subplot(211)
p1.set_xlim(-0.1, 0.1)
plt.plot(x1, f0(x1), 'b', label=r"$y = 1/x$")
plt.plot(x2, f0(x2), 'b')
plt.legend(loc=4)

p2 = plt.subplot(212)
p2.set_xlim(-0.1, 0.1)
p2.set_ylim(-2000, 0)
plt.plot(x1, f1(x1), 'g', label=r"$y = -1/x^2$")
plt.plot(x2, f1(x2), 'g')
plt.legend(loc=4)
```

If we take the derivative of an odd function we get an even function.

### other (ugly) discontinuities
Take for example the function $y = \sin \frac{1}{x}$ as $x\to 0$

```python
f = lambda x: np.sin(1/x)
x1 = np.linspace(-0.1, -0.01, 100)
x2 = np.linspace(0.01, 0.1, 100)
ax = plt.axes()
ax.set_xlim(-0.1, 0.1)
ax.set_ylim(-1.2, 1.2)
plt.plot(x1, f(x1))
plt.plot(x2, f(x2))
```

As we approach $x = 0$ it will oscilate into infinity. There is no left or right limit in this case.

## rate of change
Let's start with the question of what is a derivative? We'll look at a few different aspects:

* Geometric interpretation
* Physical interpretation
* Importance to measurements

We'll start with the geometric interpretation.

### geometric interpretation
Find the tangent line to the graph of some function $y = f(x)$ at some point $P = (x_0, y_0)$. We also know this line can be written as the equation $y - y_0 = m(x - x_0)$. In order to figure out this equation we need to know two things, point $P$ which is $(x_0, y_0)$ where $y_0 = f(x_0)$ and the value of $m$ which is the slope of the line. In calculus we also call this the derivative or $f'(x)$.

```python
f = lambda x: x**2

fig, ax = plt.subplots()

ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)

ax.spines['bottom'].set_position(('data', 0))
ax.spines['left'].set_position(('data', 0))

ax.xaxis.set_ticks_position('bottom')
ax.yaxis.set_ticks_position('left')

ax.xaxis.set_ticklabels(['$x_0$', '$x$'])
ax.yaxis.set_ticklabels(['$y_0$', '$y$'])

ax.xaxis.set_ticks([1, 1.5])
ax.yaxis.set_ticks([1, f(1.5)])

ax.set_xlim(-1, 2)
ax.set_ylim(-1, 3)

x = np.linspace(-1, 2, 100)

plt.plot(x, f(x))
plt.plot(1, f(1), 'ko')
plt.plot(1.5, f(1.5), 'ko')
plt.plot([1, 1.5], [f(1), f(1)], 'k--')
plt.plot([1.5, 1.5], [f(1), f(1.5)], 'k--')
plt.plot([1, 1.5], [f(1), f(1.5)], 'k--')

plt.annotate('$P$', (0.8, 1))
plt.annotate('$Q$', (1.3, f(1.5)))
plt.annotate('$\Delta{x}$', (1.25, 0.75))
plt.annotate('$\Delta{f}$', (1.55, 1.5))
```

We can now define, $f'(x_0)$ (the derivative) of $f$ at $x_0$ is the slope of the tangent line to $y = f(x)$ at the point $P$. The tangent line is equal to the limit of *secant lines* $PQ$ as $Q\to P$ where $P$ is fixed. In the picture above we can see that the slope of our our secant line $PQ$ is simply defined as $\frac{\Delta{f}}{\Delta{x}}$. However we can now define the slope $m$ of our tangent line as

$$m = \lim_{\Delta{x}\to 0}\frac{\Delta{f}}{\Delta{x}}$$

The next thing we want to do is to write $\Delta{f}$ more explicitly. We already have $P = (x_0, f(x_0))$ and $Q = (x_0 + \Delta{x}, f(x_0 + \Delta{x})$. With this information information we can write down:

$$f'(x_0) = m = \lim_{\Delta{x}\to 0}\frac{f(x_0 + \Delta{x}) - f(x_0)}{\Delta{x}}$$

### recital
Let $f(x) = \frac{1}{1 + x^2}$. Graph $y = f(x)$ and compute $f'(x)$.

```python
f = lambda x: 1 / (1 + x**2)
x = np.linspace(-2, 2, 100)
y = f(x)
ax = plt.axes()
ax.set_ylim(0, 1.25)
plt.plot(x, y, label=r"$y = \frac{1}{1 + x^2}$")
plt.legend()
```

Now let's compute $f'(x)$.

$$
\begin{align}
f'(x) & = \lim_{\Delta{x}\to 0}\frac{f(x + \Delta{x}) - f(x)}{\Delta{x}} \\
& = \lim_{\Delta{x}\to 0}\frac{\frac{1}{1 + (x + \Delta{x})^2} - \frac{1}{1 + x^2}}{\Delta{x}} \\
& = \lim_{\Delta{x}\to 0}\frac{1}{\Delta{x}}\frac{1 + x^2 - (1 + (x + \Delta{x})^2)}{(1 + (x + \Delta{x})^2)(1 + x^2)} \\
& = \lim_{\Delta{x}\to 0}\frac{1}{\Delta{x}}\frac{1 + x^2 -1 - x^2 - 2x\Delta{x} - \Delta{x}^2}{(1 + (x + \Delta{x})^2)(1 + x^2)} \\
& = \lim_{\Delta{x}\to 0}\frac{-2x - \Delta{x}}{(1 + (x + \Delta{x}^2)(1 + x^2)} \\
& = \frac{-2x}{(1 + x^2)^2}
\end{align}
$$

```python
f_acc = lambda x: (-2 * x) / ((1 + x**2)**2)
x = np.linspace(-2, 2, 100)
plt.plot(x, f_acc(x))
```


Computational Seismology

Finite Element Method - Static Elasticity

Seismo-Live: http://seismo-live.org

##### Authors:
* David Vargas ([@dvargas](https://github.com/davofis))
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))

## Basic Equations

Static elasticity can be thought as the particular case derived from the elastic wave equation when the displacement does not depend on time, i.e $\partial_t^2 u(x,t) = 0$. Under this assumption and departing from the 1D elastic wave equation, the differential equation turns into the Poisson equation

\begin{equation}
-\mu \partial_x^2 u = f,
\end{equation}

where $\mu$ is the shear modulus for a homogeneous media, $u$ is the displacement field, and $f$ is the external force. The solution for this problems is found after bringing this equation into its weak form, applying the free boundary condition, and using the Galerkin principle with a suitable basis. Then, the displacement defined in a discrete set of points $x_i$ is given as the solution of a system of N equations, with

\begin{equation}
\mathbf{u} = (\mathbf{K}^{T})^{-1} \mathbf{f}
\end{equation}

where $\mathbf{K}$ is the stiffness matrix. For an elastic physical system with constant shear modulus $\mu$ and uniform element size $h$, it is given as

\begin{equation}
K_{ij} = \frac{\mu}{h}
 \begin{pmatrix}
   1 & -1 &    &    & \\
  -1 &  2 & -1 &    & \\
     &    & \ddots  &  &  \\
     &    & -1 &  2 & -1   \\
     &    &    & -1 &  1
 \end{pmatrix}
\end{equation}

The purpose of this notebook is to illustrate how the problem of static elasticity is solved with the finite- element method. We also compare the solution using finite - differences, the so call relaxation method which solution is:

\begin{equation}
u_{i}^{k+1} = \dfrac{u_{i}^{k+1} + u_{i}^{k+1}}{2} + \dfrac{h^2}{2 \mu}f_i
\end{equation}

```python
# Import all necessary libraries, this is a configuration step for the exercise.
# Please run it before the simulation code!
import numpy as np
import matplotlib.pyplot as plt

# Show the plots in the Notebook.
plt.switch_backend("nbagg")
```

### Finite - Element solution

```python
# ---------------------------------------------------------------
# Initialization of setup
# ---------------------------------------------------------------
nx = 20              # Number of boundary points
u  = np.zeros(nx)    # Solution vector
f  = np.zeros(nx)    # Source vector
mu = 1               # Constant shear modulus

# Element boundary points
x = np.linspace(0, 1, nx)  # x in [0,1]
h = x[2] - x[1]            # Constant element size

# ---------------------------------------------------------------
# Assemble stiffness matrix K_ij (Eq 6.30)
# ---------------------------------------------------------------
K = np.zeros((nx, nx))
for i in range(1, nx-1):
    for j in range(1, nx-1):
        if i == j:
            K[i, j] = 2*mu/h
        elif i == j + 1:
            K[i, j] = -mu/h
        elif i + 1 == j:
            K[i, j] = -mu/h
        else:
            K[i, j] = 0

# ---------------------------------------------------------------
# Souce term is a spike at i = 3*nx/4
f[int(3*nx/4)] = 1

# Boundary condition at x = 0
u[0] = 0.15 ; f[1] = u[0]/h

# Boundary condition at x = 1
u[nx-1] = 0.05 ; f[nx-2] = u[nx-1]/h

# ---------------------------------------------------------------
# Finite element solution. (Eq 6.19)
# ---------------------------------------------------------------
u[1:nx-1] = np.linalg.inv(K[1:nx-1, 1:nx-1]) @ np.transpose(f[1:nx-1])

# ---------------------------------------------------------------
# Plotting section
# ---------------------------------------------------------------
xfe = u
plt.plot(x, xfe, color='r', lw=2, label='Finite elements')
plt.title('Static Elasticity', size=16)
plt.ylabel('Displacement $u(x)$', size=16)
plt.xlabel('Position $x$', size=16)
plt.axis([0, 1, 0.04, .28])
plt.legend()
plt.grid(True)
plt.show()
```

### Finite - Difference solution

```python
# Poisson's equation with relaxation method
# ---------------------------------------------------------------
nt = 500     # Number of time steps
iplot = 20   # Snapshot frequency

# non-zero boundary conditions
u  = np.zeros(nx)   # set u to zero
du = np.zeros(nx)   # du/dx
f  = np.zeros(nx)   # forcing

f[int(3*nx/4)] = 1./h

xfd = np.arange(0, nx)*h

# ---------------------------------------------------------------
# Initialize animated plot
# ---------------------------------------------------------------
plt.figure(figsize=(8,6))

line1 = plt.plot(x, xfe, color='r', lw=2, label='FE')
line2 = plt.plot(xfd, u, color='k', ls='-.', label='FD relaxation')
plt.title('Static Elasticity with relaxation method', size=16)
plt.ylabel('Displacement, $u$', size=16)
plt.xlabel('Position, $x$', size=16)
plt.legend(loc=4)
plt.grid(True)

plt.ion()   # set interective mode
plt.show()
# ---------------------------------------------------------------
for it in range(nt):
    # Calculate the average of u (omit boundaries)
    for i in range(1, nx-1):
        du[i] =u [i+1] + u[i-1]
    u = 0.5*( f*h**2/mu + du )
    u[0] = 0.15    # Boundary condition at x=0
    u[nx-1] = 0.05 # Boundary condition at x=1
    fd = u

    # --------------------------------------
    # Animation plot. Display both solutions
    if not it % iplot:
        for l in line2:
            l.remove()
            del l
        line1 = plt.plot(x, xfe, color='r', lw=2)
        line2 = plt.plot(xfd, fd, color='k', ls='-.')
        plt.gcf().canvas.draw()
```

```python

```