# Artificial Neurons and Single-Layer Neural Networks

**This article offers a brief glimpse of the history and basic concepts of machine learning. We will take a look at the first algorithmically described neural network and the gradient descent algorithm in context of adaptive linear neurons, which will not only introduce the principles of machine learning but also serve as the basis for modern multilayer neural networks.**

**This articled was adapted from the work done [here](https://github.com/rasbt/pattern_classification#neural-network).**

## Sections

- [Introduction](#Introduction)
- [Artificial Neurons and the McCulloch-Pitts Model](#Artificial-Neurons-and-the-McCulloch-Pitts-Model)
- [Frank Rosenblatt's Perceptron](#Frank-Rosenblatt's-Perceptron)
    - [The Unit Step Function](#The-Unit-Step-Function)
    - [The Perceptron Learning Rule](#The-Perceptron-Learning-Rule)
    - [Implementing the Perceptron Rule in Python](#Implementing-the-Perceptron-Rule-in-Python)
    - [Problems with Perceptrons](#Problems-with-Perceptrons)
- [Adaptive Linear Neurons and the Delta Rule](#Adaptive-Linear-Neurons-and-the-Delta-Rule)
    - [Gradient Descent](#Gradient-Descent)
    - [The Gradient Descent Rule in Action](#The-Gradient-Descent-Rule-in-Action)
    - [Online Learning via Stochastic Gradient Descent](#Online-Learning-via-Stochastic-Gradient-Descent)
- [What's Next?](#What's-Next?)
- [References](#References)

# Introduction

[[back to top](#Sections)]

Machine learning is one of the hottest and most exciting fields in the modern age of technology. Thanks to machine learning, we enjoy robust email spam filters, convenient text and voice recognition, reliable web search engines, challenging chess players, and, hopefully soon, safe and efficient self-driving cars.

Without any doubt, machine learning has become a big and popular field, and sometimes it may be challenging to see the (random) forest for the (decision) trees. Thus, I thought that it might be worthwhile to explore different machine learning algorithms in more detail by not only discussing the theory but also by implementing them step by step.

To briefly summarize what machine learning is all about: "[Machine learning is the] field of study that gives computers the ability to learn without being explicitly programmed" (Arthur Samuel, 1959). Machine learning is about the development and use of algorithms that can recognize patterns in data in order to make decisions based on statistics, probability theory, combinatorics, and optimization.

The first article in this series will introduce perceptrons and the adaline (ADAptive LINear NEuron), which fall into the category of single-layer neural networks. The perceptron is not only the first algorithmically described learning algorithm [[1](#References)], but it is also very intuitive, easy to implement, and a good entry point to the (re-discovered) modern state-of-the-art machine learning algorithms: Artificial neural networks (or "deep learning" if you like). As we will see later, the adaline is a consequent improvement of the perceptron algorithm and offers a good opportunity to learn about a popular optimization algorithm in machine learning: gradient descent.

# Artificial Neurons and the McCulloch-Pitts Model

[[back to top](#Sections)]

The initial idea of the perceptron dates back to the work of Warren McCulloch and Walter Pitts in 1943 [[2](#References)], who drew an analogy between biological neurons and simple logic gates with binary outputs. In more intuitive terms, neurons can be understood as the subunits of a neural network in a biological brain. Here, the signals of variable magnitudes arrive at the dendrites. Those input signals are then accumulated in the cell body of the neuron, and if the accumulated signal exceeds a certain threshold, a output signal is generated that which will be passed on by the axon.

# Frank Rosenblatt's Perceptron

[[back to top](#Sections)]

To continue with the story, a few years after McCulloch and Walter Pitt, Frank Rosenblatt published the first concept of the Perceptron learning rule [[1](#References)]. The main idea was to define an algorithm in order to learn the values of the weights $w$ that are then multiplied with the input features in order to make a decision whether a neuron fires or not. In context of pattern classification, such an algorithm could be useful to determine if a sample belongs to one class or the other.

To put the perceptron algorithm into the broader context of machine learning: The perceptron belongs to the category of supervised learning algorithms, single-layer binary linear classifiers to be more specific. In brief, the task is to predict to which of two possible categories a certain data point belongs based on a set of input variables. In this article, I don't want to discuss the concept of predictive modeling and classification in too much detail, but if you prefer more background information, please see my previous article "[Introduction to supervised learning](http://sebastianraschka.com/Articles/2014_intro_supervised_learning.html)".

## The Unit Step Function

[[back to top](#Sections)]

Before we dive deeper into the algorithm(s) for learning the weights of the artificial neuron, let us take a brief look at the basic notation. In the following sections, we will label the *positive* and *negative* class in our binary classification setting as "1" and "-1", respectively. Next, we define an activation function $g(\mathbf{z})$ that takes a linear combination of the input values $\mathbf{x}$ and weights $\mathbf{w}$ as input ($\mathbf{z} = w_1x_{1} + \dots + w_mx_{m}$), and if $g(\mathbf{z})$ is greater than a defined threshold $\theta$ we predict 1 and -1 otherwise; in this case, this activation function $g$ is an alternative form of a simple "unit step function," which is sometimes also called "Heaviside step function."

*(Please note that the *unit step* is classically defined as being equal to 0 if $ z < 0$ and 1 for $z \ge 0$; nonetheless, we will refer to the following piece-wise linear function with -1 if $z < \theta$ and 1 for $z \ge \theta$ as *unit step function* for simplicity).*

$$
 g(\mathbf{z}) =\begin{cases}
    1 & \text{if $\mathbf{z} \ge \theta$}\\
    -1 & \text{otherwise}.
  \end{cases}
$$

where

$$\mathbf{z} =  w_1x_{1} + \dots + w_mx_{m} = \sum_{j=1}^{m} x_{j}w_{j} \\ = \mathbf{w}^T\mathbf{x}$$

$\mathbf{w}$ is the feature vector, and $\mathbf{x}$ is an $m$-dimensional sample from the training dataset:

$$
\mathbf{w} = \begin{bmatrix}
    w_{1}  \\
    \vdots \\
    w_{m}
\end{bmatrix}
\quad  \mathbf{x} = \begin{bmatrix}
    x_{1}  \\
    \vdots \\
    x_{m}
\end{bmatrix}$$

In order to simplify the notation, we bring $\theta$ to the left side of the equation and define $w_0 = -\theta  \text{ and } x_0=1$

so that

$$\begin{equation}
 g({\mathbf{z}}) =\begin{cases}
    1 & \text{if $\mathbf{z} \ge 0$}\\
    -1 & \text{otherwise}.
  \end{cases}
\end{equation}$$

and

$$\mathbf{z} = w_0x_{0} + w_1x_{1} + \dots + w_mx_{m} = \sum_{j=0}^{m} x_{j}w_{j} \\ = \mathbf{w}^T\mathbf{x}.$$

## The Perceptron Learning Rule

[[back to top](#Sections)]

It might sound like extreme case of a reductionist approach, but the idea behind this "thresholded" perceptron was to mimic how a single neuron in the brain works: It either "fires" or not. To summarize the main points from the previous section: A perceptron receives multiple input signals, and if the sum of the input signals exceed a certain threshold it either returns a signal or  remains "silent" otherwise. What made this a "machine learning" algorithm was Frank Rosenblatt's idea of the perceptron learning rule: The perceptron algorithm is about learning the weights for the input signals in order to draw linear decision boundary that allows us to discriminate between the two linearly separable classes +1 and -1.

Rosenblatt's initial perceptron rule is fairly simple and can be summarized by the following steps:

1. Initialize the weights to 0 or small random numbers.
2. For each training sample $\mathbf{x^{(i)}}$:
    2. Calculate the *output* value.
    2. Update the weights.

The output value is the class label predicted by the unit step function that we defined earlier (output $=g(\mathbf{z})$) and the weight update can be written more formally as  $w_j := w_j + \Delta w_j$.

The value for updating the weights at each increment is calculated by the learning rule

$\Delta w_j = \eta \; (\text{target}^{(i)} - \text{output}^{(i)})\;x^{(i)}_{j}$

where $\eta$ is the learning rate (a constant between 0.0 and 1.0), "target" is the true class label, and the "output" is the predicted class label.

It is important to note that all weights in the weight vector are being updated simultaneously. Concretely, for a 2-dimensional dataset, we would write the update as:

$\Delta w_0 = \eta(\text{target}^{(i)} - \text{output}^{(i)})$
$\Delta w_1 = \eta(\text{target}^{(i)} - \text{output}^{(i)})\;x^{(i)}_{1}$
$\Delta w_2 = \eta(\text{target}^{(i)} - \text{output}^{(i)})\;x^{(i)}_{2}$

Before we implement the perceptron rule in Python, let us make a simple thought experiment to illustrate how beautifully simple this learning rule really is. In the two scenarios where the perceptron predicts the class label correctly, the weights remain unchanged:

- $\Delta w_j = \eta(-1^{(i)} - -1^{(i)})\;x^{(i)}_{j} = 0$
- $\Delta w_j = \eta(1^{(i)} - 1^{(i)})\;x^{(i)}_{j} = 0$

However, in case of a wrong prediction, the weights are being "pushed" towards the direction of the positive or negative target class, respectively:

- $\Delta w_j = \eta(1^{(i)} - -1^{(i)})\;x^{(i)}_{j} = \eta(2)\;x^{(i)}_{j}$
- $\Delta w_j = \eta(-1^{(i)} - 1^{(i)})\;x^{(i)}_{j} = \eta(-2)\;x^{(i)}_{j}$

It is important to note that the convergence of the perceptron is only guaranteed if the two classes are linearly separable. If the two classes can't be separated by a linear decision boundary, we can set a maximum number of passes over the training dataset ("epochs") and/or a threshold for the number of tolerated misclassifications.

## Implementing the Perceptron Rule in Python

[[back to top](#Sections)]

In this section, we will implement the simple perceptron learning rule in Python to classify flowers in the Iris dataset.
Please note that I omitted some "safety checks" for clarity, for a more "robust" version please see the following [code on GitHub](https://github.com/rasbt/mlxtend/blob/master/mlxtend/classifier/perceptron.py).

```python
import numpy as np

class Perceptron(object):

    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs

    def train(self, X, y):

        self.w_ = np.zeros(1 + X.shape[1])
        self.errors_ = []

        for _ in range(self.epochs):
            errors = 0
            for xi, target in zip(X, y):
                update = self.eta * (target - self.predict(xi))
                self.w_[1:] +=  update * xi
                self.w_[0] +=  update
                errors += int(update != 0.0)
            self.errors_.append(errors)
        return self

    def net_input(self, X):
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def predict(self, X):
        return np.where(self.net_input(X) >= 0.0, 1, -1)
```

For the following example, we will load the Iris data set from the [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/) and only focus on the two flower species *Setosa* and *Versicolor*. Furthermore, we will only use the two features *sepal length* and *petal length* for visualization purposes.

```python
import pandas as pd
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data', header=None)

# setosa and versicolor
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', -1, 1)

# sepal length and petal length
X = df.iloc[0:100, [0,2]].values
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
from mlxtend.plotting import plot_decision_regions

ppn = Perceptron(epochs=10, eta=0.1)

ppn.train(X, y)
print('Weights: %s' % ppn.w_)
plot_decision_regions(X, y, clf=ppn)
plt.title('Perceptron')
plt.xlabel('sepal length [cm]')
plt.ylabel('petal length [cm]')
plt.show()

plt.plot(range(1, len(ppn.errors_)+1), ppn.errors_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Misclassifications')
plt.show()
```

As we can see, the perceptron converges after the 6th iteration and separates the two flower classes perfectly.

## Problems with Perceptrons

[[back to top](#Sections)]

Although the perceptron classified the two Iris flower classes perfectly, convergence is one of the biggest problems of the perceptron. Frank Rosenblatt proofed mathematically that the perceptron learning rule converges if the two classes can be separated by linear hyperplane, but problems arise if the classes cannot be separated perfectly by a linear classifier. To demonstrate this issue, we will use two different classes and features from the Iris dataset.

```python
# versicolor and virginica
y2 = df.iloc[50:150, 4].values
y2 = np.where(y2 == 'Iris-virginica', -1, 1)

# sepal width and petal width
X2 = df.iloc[50:150, [1,3]].values

ppn = Perceptron(epochs=25, eta=0.01)
ppn.train(X2, y2)

plot_decision_regions(X2, y2, clf=ppn)
plt.show()

plt.plot(range(1, len(ppn.errors_)+1), ppn.errors_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Misclassifications')
plt.show()
```

```python
print('Total number of misclassifications: %d of 100' % (y2 != ppn.predict(X2)).sum())
```

    Total number of misclassifications: 43 of 100

Even at a lower training rate, the perceptron failed to find a good decision boundary since one or more samples will always be misclassified in every epoch so that the learning rule never stops updating the weights.

It may seem paradoxical in this context that another shortcoming of the perceptron algorithm is that it stops updating the weights as soon as all samples are classified correctly. Our intuition tells us that a decision boundary with a large margin between the classes (as indicated by the dashed line in the figure below) likely has a better generalization error than the decision boundary of the perceptron. But large-margin classifiers such as Support Vector Machines are a topic for another time.

# Adaptive Linear Neurons and the Delta Rule

[[back to top](#Sections)]

The perceptron surely was very popular at the time of its discovery, however, it only took a few years until Bernard Widrow and his doctoral student Tedd Hoff proposed the idea of the Adaptive Linear Neuron (adaline) [[3](#References)].

In contrast to the perceptron rule, the delta rule of the adaline (also known as Widrow-Hoff" rule or Adaline rule) updates the weights based on a linear activation function rather than a unit step function; here, this linear activation function $g(\mathbf{z})$ is just the identity function of the net input $g(\mathbf{w}^T\mathbf{x}) = \mathbf{w}^T\mathbf{x}$. In the next section, we will see why this linear activation is an improvement over the perceptron update and where the name "delta rule" comes from.

## Gradient Descent

[[back to top](#Sections)]

Being a continuous function, one of the biggest advantages of the linear activation function over the unit step function is that it is differentiable. This property allows us to define a cost function $J(\mathbf{w})$ that we can minimize in order to update our weights. In the case of the linear activation function, we can define the cost function $J(\mathbf{w})$ as the *sum of squared errors* (SSE), which is similar to the cost function that is minimized in ordinary least squares (OLS) linear regression.

$$J(\mathbf{w})  = \frac{1}{2} \sum_{i} (\text{target}^{(i)} - \text{output}^{(i)})^2 \quad \quad \text{output}^{(i)} \in \mathbb{R}$$

(The fraction $\frac{1}{2}$ is just used for convenience to derive the gradient as we will see in the next paragraphs.)

In order to minimize the SSE cost function, we will use gradient descent, a simple yet useful optimization algorithm that is often used in machine learning to find the local minimum of linear systems.

Before we get to the fun part (calculus), let us consider a convex cost function for one single weight. As illustrated in the figure below, we can describe the principle behind gradient descent as "climbing down a hill" until a local or global minimum is reached. At each step, we take a step into the opposite direction of the gradient, and the step size is determined by the value of the learning rate as well as the slope of the gradient.

Now, as promised, onto the fun part -- deriving the Adaline learning rule.
As mentioned above, each update is updated by taking a step into the opposite direction of the gradient $\Delta \mathbf{w} = - \eta \nabla J(\mathbf{w})$, thus, we have to compute the partial derivative of the cost function for each weight in the weight vector:  $\Delta w_j = - \eta \frac{\partial J}{\partial w_j}$.

The partial derivative of the SSE cost function for a particular weight can be calculated as follows:

$$\begin{equation}
 \frac{\partial J}{\partial w_j} = \frac{\partial }{\partial w_j} \frac{1}{2} \sum_i  (t^{(i)} - o^{(i)})^2 \\
= \frac{1}{2} \sum_i \frac{\partial}{\partial w_j} (t^{(i)} - o^{(i)})^2 \\
= \frac{1}{2} \sum_i 2 (t^{(i)} - o^{(i)}) \frac{\partial}{\partial w_j} (t^{(i)} - o^{(i)}) \\
=  \sum_i (t^{(i)} - o^{(i)}) \frac{\partial}{\partial w_j} \bigg(t^{(i)} - \sum_j w_j x^{(i)}_{j}\bigg) \\
= \sum_i  (t^{(i)} - o^{(i)})(-x^{(i)}_{j})
\end{equation}$$

(t = target, o = output)

And if we plug the results back into the learning rule, we get

$\Delta w_j = - \eta \frac{\partial J}{\partial w_j} = - \eta \sum_i  (t^{(i)} - o^{(i)})(- x^{(i)}_{j}) = \eta \sum_i (t^{(i)} - o^{(i)})x^{(i)}_{j}$,

Eventually, we can apply a simultaneous weight update similar to the perceptron rule:

$\mathbf{w} := \mathbf{w} + \Delta \mathbf{w}$.

**Although, the learning rule above looks identical to the perceptron rule, we shall note the two main differences:**

1. Here, the output "o" is a real number and not a class label as in the perceptron learning rule.
2. The weight update is calculated based on all samples in the training set (instead of updating the weights incrementally after each sample), which is why this approach is also called "batch" gradient descent.

## The Gradient Descent Rule in Action

[[back to top](#Sections)]

Now, it's time to implement the gradient descent rule in Python.

```python
import numpy as np

class AdalineGD(object):

    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs

    def train(self, X, y):

        self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.epochs):
            output = self.net_input(X)
            errors = (y - output)
            self.w_[1:] += self.eta * X.T.dot(errors)
            self.w_[0] += self.eta * errors.sum()
            cost = (errors**2).sum() / 2.0
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        return self.net_input(X)

    def predict(self, X):
        return np.where(self.activation(X) >= 0.0, 1, -1)
```

In practice, it often requires some experimentation to find a good learning rate for optimal convergence, thus, we will start by plotting the cost for two different learning rates.

```python
ada = AdalineGD(epochs=10, eta=0.01).train(X, y)
plt.plot(range(1, len(ada.cost_)+1), np.log10(ada.cost_), marker='o')
plt.xlabel('Iterations')
plt.ylabel('log(Sum-squared-error)')
plt.title('Adaline - Learning rate 0.01')
plt.show()

ada = AdalineGD(epochs=10, eta=0.0001).train(X, y)
plt.plot(range(1, len(ada.cost_)+1), ada.cost_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Sum-squared-error')
plt.title('Adaline - Learning rate 0.0001')
plt.show()
```

The two plots above nicely emphasize the importance of plotting learning curves by illustrating two most common problems with gradient descent:

1. If the learning rate is too large, gradient descent will overshoot the minima and diverge.
2. If the learning rate is too small, the algorithm will require too many epochs to converge and can become trapped in local minima more easily.

Gradient descent is also a good example why feature scaling is important for many machine learning algorithms.
It is not only easier to find an appropriate learning rate if the features are on the same scale, but it also often leads to faster convergence and can prevent the weights from becoming too small (numerical stability).

A common way of feature scaling is standardization

$$\mathbf{x}_{j, std} = \frac{\mathbf{x}_j - \mathbf{\mu}_j}{\mathbf{\sigma}_j}$$

where $\mathbf{\mu}_j$ is the sample mean of the feature $\mathbf{x}_{j}$ and $\mathbf{\sigma}_j$ the standard deviation, respectively. After standardization, the features will have unit variance and are centered around mean zero.

```python
# standardize features
X_std = np.copy(X)
X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()
X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()
```

```python
ada = AdalineGD(epochs=15, eta=0.01)

ada.train(X_std, y)
plot_decision_regions(X_std, y, clf=ada)
plt.title('Adaline - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.show()

plt.plot(range(1, len( ada.cost_)+1), ada.cost_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Sum-squared-error')
plt.show()
```

## Online Learning via Stochastic Gradient Descent

[[back to top](#Sections)]

The previous section was all about "batch" gradient descent learning. The "batch" updates refers to the fact that the cost function is minimized based on the complete training data set. If we think back to the perceptron rule, we remember that it performed the weight update incrementally after each individual training sample. This approach is also called "online" learning, and in fact, this is also how Adaline was first described by Bernard Widrow et al. [[3](#References)]

The process of incrementally updating the weights is also called "stochastic" gradient descent since it approximates the minimization of the cost function. Although the stochastic gradient descent approach might sound inferior to gradient descent due its "stochastic" nature and the "approximated" direction (gradient), it can have certain advantages in practice. Often, stochastic gradient descent converges much faster than gradient descent since the updates are applied immediately after each training sample; stochastic gradient descent is computationally more efficient, especially for very large datasets. Another advantage of online learning is that the classifier can be immediately updated as new training data arrives, e.g., in web applications, and old training data can be discarded if storage is an issue. In large-scale machine learning systems, it is also common practice to use so-called "mini-batches", a compromise with smoother convergence than stochastic gradient descent.

In the interests of completeness let us also implement the stochastic gradient descent Adaline and confirm that it converges on the linearly separable iris dataset.

```python
import numpy as np

class AdalineSGD(object):

    def __init__(self, eta=0.01, epochs=50):
        self.eta = eta
        self.epochs = epochs

    def train(self, X, y, reinitialize_weights=True):

        if reinitialize_weights:
            self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.epochs):
            for xi, target in zip(X, y):
                output = self.net_input(xi)
                error = (target - output)
                self.w_[1:] += self.eta * xi.dot(error)
                self.w_[0] += self.eta * error

            cost = ((y - self.activation(X))**2).sum() / 2.0
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        return self.net_input(X)

    def predict(self, X):
        return np.where(self.activation(X) >= 0.0, 1, -1)
```

One more advice before we let the adaline learn via stochastic gradient descent is to shuffle the training dataset to iterate over the training samples in random order.

We shall note that the "standard" stochastic gradient descent algorithm uses sampling "with replacement," which means that at each iteration, a training sample is chosen randomly from the entire training set. In contrast, sampling "without replacement," which means that each training sample is evaluated exactly once in every epoch, is not only easier to implement but also shows a better performance in empirical comparisons. A more detailed discussion about this topic can be found in Benjamin Recht and Christopher Re's paper *Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences* [[4](#References)].

```python
ada = AdalineSGD(epochs=15, eta=0.01)

# shuffle data
np.random.seed(123)
idx = np.random.permutation(len(y))
X_shuffled, y_shuffled =  X_std[idx], y[idx]

# train and adaline and plot decision regions
ada.train(X_shuffled, y_shuffled)
plot_decision_regions(X_shuffled, y_shuffled, clf=ada)
plt.title('Adaline - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.show()

plt.plot(range(1, len(ada.cost_)+1), ada.cost_, marker='o')
plt.xlabel('Iterations')
plt.ylabel('Sum-squared-error')
plt.show()
```

# What's Next?

[[back to top](#Sections)]

Although we covered many different topics during this article, we just scratched the surface of artificial neurons.

In later articles, we will take a look at different approaches to dynamically adjust the learning rate, the concepts of "One-vs-All" and "One-vs-One" for multi-class classification, regularization to overcome overfitting by introducing additional information, dealing with nonlinear problems and multilayer neural networks, different activation functions for artificial neurons, and related concepts such as logistic regression and support vector machines.

### References

[[back to top](#Sections)]

[1] F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.

[2] W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5(4):115–133, 1943.

[3] B. Widrow et al. Adaptive ”Adaline” neuron using chemical ”memistors”. Number Technical Report 1553-2. Stanford Electron. Labs., Stanford, CA, October 1960.

[4] B. Recht and C. R ́e. Beneath the valley of the noncommutative arithmetic-geometric mean inequality: conjectures, case-studies, and consequences. arXiv preprint arXiv:1202.4184, 2012.

http://glowingpython.blogspot.com/2011/10/perceptron.html
https://github.com/rasbt/pattern_classification#neural-networks
https://blog.dbrgn.ch/2013/3/26/perceptrons-in-python/


## Computer Science Intensive Course - MindX

# LAB 4. THUẬT TOÁN TÌM KIẾM

```python
# run this cell FIRST
%run test_cases_4.ipynb
```

## Bài 1. Đếm Từ

Cho một đoạn văn gồm nhiều từ và dấu câu, mỗi từ trong đoạn văn cách nhau bằng ít nhất một dấu cách. Đếm tần suất xuất hiện của một từ trong đoạn văn đó (không phân biệt chữ hoa và thường).

**Input**: Một đoạn văn dưới dạng *str* với độ dài < 10^6 ký tự và một từ.
**Output**: Tần suất xuất hiện của từ đã cho trong đoạn văn.
**Ví dụ**:
- Input: "The cat is chasing the rat in Netherlands. The dog is also chasing the rat.", "the"
- Output: 4
- Giải thích: Từ "the" xuất hiện 4 lần, không phân biệt chữ hoa và thường.

**Gợi ý**: Kiểm tra một ký tự có nằm trong bảng chữ cái không:

```python
ch = 'a'
ch2 = '.'
print(ch.isalpha())
print(ch2.isalpha())
```

    True
    False

```python
def count_word(inp_str, inp_word):
    pass
```

```python
# !!! DO NOT MODIFY THIS CELL
# Check result on test cases
test1(count_word)
```

    Testing on 10 cases.
    - Test 1 PASSED.
    - Test 2 PASSED.
    - Test 3 PASSED.
    - Test 4 PASSED.
    - Test 5 PASSED.
    - Test 6 PASSED.
    - Test 7 PASSED.
    - Test 8 PASSED.
    - Test 9 PASSED.
    - Test 10 PASSED.
    CONGRATULATIONS! All test cases passed!

## Bài 2. Rút Gọn Phân Số

**Input**: Một phân số a/b với 0 < a, b < 10^9 là số nguyên.
**Output**: Phân số a/b sau khi đã rút gọn.
**Ví dụ**:
Ví dụ 1:
- Input: a = 20, b = 4
- Output: a = 5, b = 1

Ví dụ 2:
- Input: a = 7, b = 20
- Output: a = 7, b = 20

```python
def reduce_fraction(a, b):
    pass
```

```python
# !!! DO NOT MODIFY THIS CELL
# Check result on test cases
test2(reduce_fraction)
```

    Testing on 10 cases.
    - Test 1 PASSED.
    - Test 2 PASSED.
    - Test 3 PASSED.
    - Test 4 PASSED.
    - Test 5 PASSED.
    - Test 6 PASSED.
    - Test 7 PASSED.
    - Test 8 PASSED.
    - Test 9 PASSED.
    - Test 10 PASSED.
    CONGRATULATIONS! All test cases passed!

## Bài 3. Fibonacci

Dãy Fibonacci có dạng 1, 1, 2, 3, 5, 8, 13,... với công thức truy hồi:
\begin{align}
    fibo_{n} &= fibo_{n-1} + fibo_{n-2} \\
    fibo_{1} &= fibo_{2} = 1
\end{align}
Sử dụng đệ quy để tìm số Fibonacci thứ *n*.

**Input**: Một số nguyên *0 < n < 1000*.
**Output**: Một số nguyên là số Fibonacci thứ *n*.
**Ví dụ**:

Ví dụ 1:
- Input: n = 3
- Output: 2

Ví dụ 2:
- Input: n = 6
- Output: 8

```python
def fibonacci(n):
    pass
```

```python
# !!! DO NOT MODIFY THIS CELL
# Check result on test cases
test3(fibonacci)
```

    Testing on 12 cases.
    - Test 1 PASSED.
    - Test 2 PASSED.
    - Test 3 PASSED.
    - Test 4 PASSED.
    - Test 5 PASSED.
    - Test 6 PASSED.
    - Test 7 PASSED.
    - Test 8 PASSED.
    - Test 9 PASSED.
    - Test 10 PASSED.
    - Test 11 PASSED.
    - Test 12 PASSED.
    CONGRATULATIONS! All test cases passed!

## Bài 4. Kiểm Tra Dãy Con

Cho hai chuỗi ký tự *str1* và *str2* chỉ bao gồm các chữ cái viết thường từ 'a' đến 'z'. Kiểm tra có thể tạo thành *str2* bằng cách chọn và sắp xếp các phần tử trong *str1* hay không.

**Input**: *str1* và *str2* có độ dài < 10^6
**Output**: *True* nếu có thể tạo thành *str2* từ *str1*, ngược lại *False*.
**Ví dụ**:

Ví dụ 1:
- Input: str1 = "acbdfe", str2 = "face"
- Output: True
- Giải thích: Ta có thể chọn các ký tự 'a', 'c', 'e', 'f' từ *str1* và sắp xếp lại để tạo thành *str2*

Ví dụ 2:
- Input: str1 = "aabccdef", str2 = "faded"
- Output: False
- Giải thích: *str1* chỉ có một ký tự 'd', còn *str2* cần hai ký tự 'd'

```python
def check_substring(str1, str2):
    pass
```

```python
# !!! DO NOT MODIFY THIS CELL
# Check result on test cases
test4(check_substring)
```

    Testing on 10 cases.
    - Test 1 PASSED.
    - Test 2 PASSED.
    - Test 3 PASSED.
    - Test 4 PASSED.
    - Test 5 PASSED.
    - Test 6 PASSED.
    - Test 7 PASSED.
    - Test 8 PASSED.
    - Test 9 PASSED.
    - Test 10 PASSED.
    CONGRATULATIONS! All test cases passed!

```python

```


# Time Series Forecasting w/ Facebook's Prophet

Adopted from https://facebook.github.io/prophet/docs/quick_start.html.

Other good resources include:
https://xang1234.github.io/prophet/.

And this textbook on Forecasting https://otexts.com/fpp2/.

By default, Prophet uses an additive model

\begin{align}
y(t) = g(t) + s(t) + h(t) + \epsilon_t \\
\end{align}

where
g(t) represents the trend, s(t) the periodic (i.e. seasonal) component,
h(t) holiday related events and ϵ_t the error.
The periodic component can be broken into monthly seasonality and weekly seasonality components.
The model can also be switched to use a multiplicative process.

If you want to look into Bayesian time series modeling or dynamic linear modeling, checkout this repo: https://github.com/wwrechard/pydlm.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import git
repo = git.Repo("./", search_parent_directories=True)
homedir = repo.working_dir
```

```python
# Facebook's Open Source Time Series Analysis Tool
# pip install pystan
# pip install fbprophet
# If above ^ command doesn't install, see below issue
# https://github.com/facebook/prophet/issues/775
# pip install fbprophet --no-cache-dir --no-binary :all:
# Also for interactive visualizations you can install Plotly
# pip install plotly
import datetime
from fbprophet import Prophet
from fbprophet.plot import add_changepoints_to_plot
from fbprophet.diagnostics import cross_validation
from fbprophet.diagnostics import performance_metrics
from fbprophet.plot import plot_cross_validation_metric
```

```python
df = pd.read_csv(f"{homedir}/data/us/flu/cases/ILI_Ages.csv")
```

```python
df.head()
```

      REGION TYPE
      REGION
      YEAR
      WEEK
      % WEIGHTED ILI
      %UNWEIGHTED ILI
      AGE 0-4
      AGE 25-49
      AGE 25-64
      AGE 5-24
      AGE 50-64
      AGE 65
      ILITOTAL
      NUM. OF PROVIDERS
      TOTAL PATIENTS

      0
      National
      X
      1997
      40
      1.10148
      1.21686
      179
      X
      157
      205
      X
      29
      570
      192
      46842

      1
      National
      X
      1997
      41
      1.20007
      1.28064
      199
      X
      151
      242
      X
      23
      615
      191
      48023

      2
      National
      X
      1997
      42
      1.37876
      1.23906
      228
      X
      153
      266
      X
      34
      681
      219
      54961

      3
      National
      X
      1997
      43
      1.19920
      1.14473
      188
      X
      193
      236
      X
      36
      653
      213
      57044

      4
      National
      X
      1997
      44
      1.65618
      1.26112
      217
      X
      162
      280
      X
      41
      700
      213
      55506

```python
# Prophet requires daily data, so we will just transform from week to day
def get_first_day_of_week(yr, wk):
    d = "%s-W%s" % (yr, wk)
    # The -1 and -%w pattern tells the parser to pick the Monday in that week
    firstdayofweek = datetime.datetime.strptime(d + '-1', "%Y-W%W-%w")
    return firstdayofweek
```

```python
# Prophet also requires dates to be stored in a column called 'ds'
# And that the values to predict will be stored in a column called 'y'
y_name = "ILITOTAL"
df["ds"] = df.apply(lambda row: get_first_day_of_week(row.YEAR, row.WEEK), axis=1)
df["y"] = df[y_name]
df.tail()
```

      REGION TYPE
      REGION
      YEAR
      WEEK
      % WEIGHTED ILI
      %UNWEIGHTED ILI
      AGE 0-4
      AGE 25-49
      AGE 25-64
      AGE 5-24
      AGE 50-64
      AGE 65
      ILITOTAL
      NUM. OF PROVIDERS
      TOTAL PATIENTS
      ds
      y

      1167
      National
      X
      2020
      7
      5.99056
      6.18745
      17176
      26193
      X
      36531
      9179
      5397
      94476
      2946
      1526897
      2020-02-17
      94476

      1168
      National
      X
      2020
      8
      5.39490
      5.60028
      15122
      22944
      X
      28164
      8280
      5116
      79626
      2950
      1421823
      2020-02-24
      79626

      1169
      National
      X
      2020
      9
      5.08334
      5.10641
      13514
      21767
      X
      24935
      8147
      5006
      73369
      2881
      1436802
      2020-03-02
      73369

      1170
      National
      X
      2020
      10
      5.17801
      5.28829
      13380
      23434
      X
      25452
      8728
      5561
      76555
      2794
      1447633
      2020-03-09
      76555

      1171
      National
      X
      2020
      11
      5.82936
      6.51981
      13236
      28704
      X
      25981
      10933
      6430
      85284
      2497
      1308074
      2020-03-16
      85284

```python
# Possible setting changes
# changepoint_prior_scale=0.5 (1 default)
# seasonality_mode='multiplicative' ('additive' default)
# Adding separate regressors to the model
m = Prophet()
m.fit(df)
```

    INFO:fbprophet:Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this.
    INFO:fbprophet:Disabling daily seasonality. Run prophet with daily_seasonality=True to override this.

    <fbprophet.forecaster.Prophet at 0x1213396d8>

```python
future = m.make_future_dataframe(periods=90)
future.tail()
```

      ds

      1257
      2020-06-10

      1258
      2020-06-11

      1259
      2020-06-12

      1260
      2020-06-13

      1261
      2020-06-14

```python
forecast = m.predict(future)
```

```python
forecast.tail()
```

      ds
      trend
      yhat_lower
      yhat_upper
      trend_lower
      trend_upper
      additive_terms
      additive_terms_lower
      additive_terms_upper
      yearly
      yearly_lower
      yearly_upper
      multiplicative_terms
      multiplicative_terms_lower
      multiplicative_terms_upper
      yhat

      1257
      2020-06-10
      37662.477521
      20521.489965
      41853.727480
      37660.323084
      37671.612578
      -6057.632839
      -6057.632839
      -6057.632839
      -6057.632839
      -6057.632839
      -6057.632839
      0.0
      0.0
      0.0
      31604.844682

      1258
      2020-06-11
      37676.564368
      20405.927010
      43110.709987
      37674.264855
      37686.333353
      -6139.844646
      -6139.844646
      -6139.844646
      -6139.844646
      -6139.844646
      -6139.844646
      0.0
      0.0
      0.0
      31536.719722

      1259
      2020-06-12
      37690.651215
      21293.790358
      42295.222673
      37687.896755
      37700.759800
      -6218.081653
      -6218.081653
      -6218.081653
      -6218.081653
      -6218.081653
      -6218.081653
      0.0
      0.0
      0.0
      31472.569562

      1260
      2020-06-13
      37704.738063
      19771.558879
      41826.208123
      37700.751998
      37715.379664
      -6291.960357
      -6291.960357
      -6291.960357
      -6291.960357
      -6291.960357
      -6291.960357
      0.0
      0.0
      0.0
      31412.777706

      1261
      2020-06-14
      37718.824910
      19848.764023
      41229.740499
      37714.414772
      37730.108956
      -6361.215884
      -6361.215884
      -6361.215884
      -6361.215884
      -6361.215884
      -6361.215884
      0.0
      0.0
      0.0
      31357.609026

```python
forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].tail()
```

      ds
      yhat
      yhat_lower
      yhat_upper

      1257
      2020-06-10
      31604.844682
      20521.489965
      41853.727480

      1258
      2020-06-11
      31536.719722
      20405.927010
      43110.709987

      1259
      2020-06-12
      31472.569562
      21293.790358
      42295.222673

      1260
      2020-06-13
      31412.777706
      19771.558879
      41826.208123

      1261
      2020-06-14
      31357.609026
      19848.764023
      41229.740499

```python
# Note that H1N1 outbreak and increased volumn of patients break the forecast
fig1 = m.plot(forecast, xlabel="Time", ylabel=y_name)
# Uncomment to see where the trend line changes
# a=add_changepoints_to_plot(fig1.gca(), m, forecast)
```

```python
fig1 = m.plot(forecast, xlabel="Time", ylabel=y_name)
# Uncomment to see where the trend line changes
a=add_changepoints_to_plot(fig1.gca(), m, forecast)
```

```python
# Plot the yearly trends and monthly seasonality component
fig2 = m.plot_components(forecast)
```

```python
# Cross Validation to measure forecast error
# Note: This takes a couple min (3-5)
%time df_cv = cross_validation(m, initial='%i days' % (20 * 365), period='30 days', horizon = '365 days')
```

    INFO:fbprophet:Making 18 forecasts with cutoffs between 2017-10-23 00:00:00 and 2019-03-17 00:00:00

    CPU times: user 28 s, sys: 118 ms, total: 28.1 s
    Wall time: 28.2 s

```python
df_p = performance_metrics(df_cv)
df_p.head()
```

      horizon
      mse
      rmse
      mae
      mape
      mdape
      coverage

      0
      36 days
      3.967410e+08
      19918.359395
      13476.337830
      0.497669
      0.386900
      0.565217

      1
      37 days
      3.744693e+08
      19351.208048
      13119.277358
      0.495094
      0.386900
      0.568841

      2
      38 days
      4.094088e+08
      20233.853040
      13723.357097
      0.505929
      0.401154
      0.554348

      3
      39 days
      3.953127e+08
      19882.471297
      13349.531616
      0.493594
      0.386900
      0.568841

      4
      40 days
      4.122768e+08
      20304.600873
      13884.929793
      0.513079
      0.401154
      0.543478

```python

```


Computational Seismology

 SBP-SAT finite difference method for the 1D elastic wave equation in first order form

This notebook is based on the paper [Dynamic earthquake rupture simulations on nonplanar faults embedded in 3D geometrically complex, heterogeneous Earth models](https://pangea.stanford.edu/~edunham/publications/Duru_Dunham_FD3d_JCP16.pdf), and on the theory of summation-by-parts (SBP) finite difference methods and weak implementation of boundary conditions using the simultaneous-approximation-term (SAT).

##### Authors:
* Kenneth Duru

---

## Basic Equations ##

We consider in this notebook a system of hyperbolic PDE, the elastic wave equation, in 1D. The source-free elastic wave equation in a heterogeneous 1D medium is

\begin{align}
\rho(x)\frac{\partial v(x,t)}{\partial t} -\frac{\partial\sigma(x,t)}{\partial x} & = 0\\
\frac{1}{\mu(x)}\frac{\partial\sigma(x,t)}{\partial t} -\frac{\partial v(x,t)}{\partial x} & = 0
\end{align}

with $\rho(x)$ the density, $\mu(x)$ the shear modulus and $x \in [0, L]$. At the boundaries $ x = 0, x = L$ we pose the general well-posed linear boundary conditions

\begin{equation}
\begin{split}
B_0(v, \sigma, Z_{s}, r_0): =\frac{Z_{s}}{2}\left({1-r_0}\right){v} -\frac{1+r_0}{2} {\sigma} = 0,  \quad \text{at} \quad x = 0, \\
 B_L(v, \sigma, Z_{s}, r_n): =\frac{Z_{s}}{2} \left({1-r_n}\right){v} + \frac{1+r_n}{2}{\sigma} = 0, \quad \text{at} \quad  x = L.
 \end{split}
\end{equation}

with the reflection coefficients $r_0$, $r_n$ being real numbers and  $|r_0|, |r_n| \le 1$.

Note that at $x = 0$,  while  $r_0 = -1$ yields a clamped wall, $r_0 = 0$  yields  an absorbing boundary, and  with $r_0 = 1$  we have a free-surface boundary condition. Similarly, at $x = L$, $r_n = -1$ yields a clamped wall, $r_n = 0$ yields an absorbing boundary, and  $r_n = 1$  gives a free-surface boundary condition.

We introduce the mechanical energy defined by
\begin{equation}
E(t) = \int_0^L{\left(\frac{\rho(x)}{2} v^2(x, t) + \frac{1}{2\mu(x)}\sigma^2(x, t)\right) dx},
\end{equation}

where $E(t)$ is the sum of the kinetic energy and the strain energy.
We have

\begin{equation}
\frac{d E(t)}{dt} = -v(0, t)\sigma(0, t) + v(L, t)\sigma(L, t) \le 0.
\end{equation}

From the boundary conditions, it is easy to check that  $v(0, t)\sigma(0, t) \ge 0$ and $v(L, t)\sigma(L, t) \le 0$, for all $|r_0|, |r_n| \le 1$. This energy loss through the boundaries is what the numerical method should emulate.

1) Discretize the spatial domain $x$ into $N$ discrete nodes with the uniform spatial step $\Delta{x} = L/(N-1)$, denote the unknown fields at the nodes: $\mathbf{v}\left(t\right) = [v_1\left(t\right), v_2\left(t\right), \cdots, v_N\left(t\right)]$, and $\boldsymbol{\sigma}\left(t\right) = [\sigma_1\left(t\right), \sigma_2\left(t\right), \cdots, \sigma_N\left(t\right)]$.

2) At the grid-point $x_j = (j-1)\Delta{x}$: Approximate the spatial derivative by a finite difference operator $\partial v/\partial x\Big|_{x = x_j} \approx \left(\mathbf{D}\mathbf{v}\right)_j $. Here $\mathbf{D}$ is a finite difference matrix satisfying the summation-by-parts property:

\begin{align}
\mathbf{H}\mathbf{D} = \mathbf{Q}, \quad \mathbf{Q} + \mathbf{Q} = \left(\boldsymbol{e}_{N}\boldsymbol{e}_{N}^T -\boldsymbol{e}_{1}\boldsymbol{e}_{1}^T\right), \quad \mathbf{H}^T = \mathbf{H} > 0,
\end{align}

where, $\boldsymbol{e}_{0} = [1, 0, \dots,  0 ]^T, \quad  \boldsymbol{e}_{L} = [ 0, 0, \dots,  1 ]^T$ and $\mathbf{H}$ defines a dicrete norm. We consider only diagonal norm SBP operators with $H_{jj} = h_j > 0$, and define the quadrature rule

\begin{equation}
 \sum_{i = 1}^{N} f(x_j)h_j \approx \int_{0}^{L}f(x) dx.
\end{equation}

The second order accurate SBP operator for the first derivative is:
\begin{align}
\left(\mathbf{D}\mathbf{v}\right)_j = \frac{v_{j+1}-v_{j-1}}{2 \Delta{x}}, \quad j = 2, 3, \cdots N-1, \quad
\left(\mathbf{D}\mathbf{v}\right)_1 = \frac{v_{2}-v_{1}}{\Delta{x}},\quad
\left(\mathbf{D}\mathbf{v}\right)_N = \frac{v_{N}-v_{N-1}}{\Delta{x}}, \quad j = N.
\end{align}

Note that the interior stencils are centered, with second order accuracy and the boundary stencils are one-sided and first order accurate.

Higher order SBP operators can be found in the book: High Order Difference Methods for Time Dependent PDE, by B. Gustafsson. In this notebook we implement SBP operators with interior accuracy 2, 4 and 6. The implementation of the spatial derivative operators can be found in the file first_derivative_sbp_operators.py

To construct a stable  semi-discrete approximation we replace the spatial derivatives by the SBP operators, and add the boundary conditions as SAT-terms with special penalty weights having:

\begin{align}
\frac{d \mathbf{v}(t)}{d t} = {\boldsymbol{\rho}}^{-1}
\left(\mathbf{D} \boldsymbol{\sigma}(t) - \underbrace{\mathbf{H}^{-1}\left(\tau_{11}\boldsymbol{e}_{1}B_0\left(v_1, \sigma_1, Z_{s}, r_0\right) + \tau_{12}\boldsymbol{e}_{N}B_L\left(v_N, \sigma_N, Z_{s}, r_n\right)\right)}_{SAT \to 0}\right),
\end{align}

\begin{align}
\frac{d \boldsymbol{\sigma}(t)}{d t} = \boldsymbol{\mu}
\left(\mathbf{D} \mathbf{v}(t) + \underbrace{\mathbf{H}^{-1}\left(\tau_{21}\frac{\boldsymbol{e}_{1}}{Z_{s}}B_0\left(v_1, \sigma_1, Z_{s}, r_0\right) - \tau_{22}\frac{\boldsymbol{e}_{N}}{Z_{s}}B_L\left(v_N, \sigma_N, Z_{s}, r_n\right)\right)}_{SAT \to 0}\right).
\end{align}
Here $\tau_{ij}$ are penalty parameters determined by requiring stability.

Approximate the mechanical energy by the above quadrature rule, having
\begin{align}
\mathcal{E}( t) = \sum_{j}^{N}\frac{1}{2}\left(\rho_jv_j^2 + \frac{1}{\mu_j}\sigma_j^2\right)h_j > 0.
\end{align}

By chosing the penalty parameters $\tau_{ij} = 1$, the semi-discrete approximation satisfies the energy estimate:
\begin{align}
\frac{d \mathcal{E}( t)}{d t} = -\frac{1}{2}\left(\left(1-r_0\right)Zv_1^2 + \frac{\left(1+r_0\right)}{Z}\sigma_1^2 +
\left(1-r_n\right)Zv_N^2 + \frac{\left(1+r_n\right)}{Z}\sigma_N^2\right) \le 0.
\end{align}

3) The discrete mechanical energy can never grow in time, and thus the semidiscrete numerical approximation is asymptotically stable.

4)  Time integration can be performed using any stable time stepping scheme. This notebook implements the fourth order accurate Runge-Kutta method.

To keep the problem simple, we use as spatial initial condition a Gauss function with half-width $\delta$

\begin{equation}
v(x,t=0)  = e^{-1/\delta^2 (x - x_{o})^2}, \quad \sigma(x,t=0) = 0
\end{equation}

**** Exercises****

```python
# Parameters initialization and plotting the simulation
# Import necessary routines
import numpy as np
import matplotlib.pyplot as plt
import time_integrator
import rate
import utils
import timeit

#plt.switch_backend("TkAgg")          # plots in external window
plt.switch_backend("nbagg")           # plots within this notebook
```

```python
# Initializations
L = 10.0         # length of the domain (km)
t = 0.0          # initial time
tend = 1.45      # final time
nx = 501        # grid points in x
dx = L/(nx-1)    # grid increment in x
cs = 3.464       # velocity (km/s) (can be an array)
iplot = 5       # snapshot frequency
rho = 2.6702     # density [g/cm^3]
mu = rho*cs**2   # shear modulus [GPa]
Zs = rho*cs      # shear impedance

order = 6        # order of accuracy

#Initialize the domain
y = np.zeros((nx, 1))

# Initial particle velocity perturbation and discretize the domain
for j in range(0, nx):
    y[j, :] = j*dx                                             # discrete domain

# Time stepping parameters
cfl = 1.0                         # CFL number
dt = (cfl/cs)*dx                  # Time step
nt = int(round(tend/dt))          # number of time steps
n = 0                             # counter

# Boundary condition reflection coefficients
r0 = 1                            # r=0:absorbing, r=1:free-surface, r=-1: clamped
r1 = 1                            # r=0:absorbing, r=1:free-surface, r=-1: clamped

# penalty parameters
tau_11 = 1
tau_12 = 1
tau_21 = 1
tau_22 = 1

# Initialize: particle velocity (v); and shear stress (s)
v = np.zeros((nx, 1))
s = np.zeros((nx, 1))

U = np.zeros((nx, 1))
V = np.zeros((nx, 1))
U_t = np.zeros((nx, 1))
V_t = np.zeros((nx, 1))
U_x = np.zeros((nx, 1))
V_x = np.zeros((nx, 1))

# Difference between analyticla and numerical solutions
EV = [0]                                 # initialize errors in V (velocity)
EU = [0]                                 # initialize errors in U (stress)
T = [0]                                  # later append every time steps to this
```

```python
# Computation and plotting

# Initialize animated plot for velocity and stress
fig1 = plt.figure(figsize=(10,10))
ax1 = fig1.add_subplot(4,1,1)
line1 = ax1.plot(y, v, 'r', y, U, 'k--')
plt.title('numerical vs exact')
plt.xlabel('x [km]')
plt.ylabel('velocity [m/s]')

ax2 = fig1.add_subplot(4,1,2)
line2 = ax2.plot(y, s, 'r', y, V, 'k--')
plt.title('numerical vs exact')
plt.xlabel('x[km]')
plt.ylabel('stress [MPa]')

# Initialize error plot (for velocity and stress)
ax3 = fig1.add_subplot(4,1,3)
line3 = ax3.plot(T, EV, 'r')
plt.title('relative error in particle velocity')
plt.xlabel('time [s]')
ax3.set_ylim([10**-5, 1])
plt.ylabel('error')

ax4 = fig1.add_subplot(4,1,4)
line4 = ax4.plot(T, EU, 'r')
plt.ylabel('error')
plt.xlabel('time[t]')
ax4.set_ylim([10**-5, 1])
plt.title('relative error in stress')

plt.tight_layout()
plt.ion()
plt.show()

t=0   # initial time

forcing = 1.0  # forcing function, forcing = 1,  and no forcing function, forcing = 0

# type of initial data: Gaussian or Sinusoidal
type_0 = 'Gaussian'
#type_0 = 'Sinusoidal'

if type_0 in ('Sinusoidal'):
            forcing = 1.0  # we must use forcing for Sinusoidal initial condition

# L2-norm normalizer
# Generate  conditions for normalization
rate.mms(v, s, U_t, V_t, U_x, V_x, y, 0.65, type_0)
A =  (np.linalg.norm(v))
B =  (np.linalg.norm(s))

# Loop through time and evolve the wave-fields using ADER time-stepping scheme of N+1 order of accuracy
start = timeit.default_timer()

# Generate initial conditions
rate.mms(v, s, U_t, V_t, U_x, V_x, y, t, type_0)

for t in utils.drange (0.0, tend+dt,dt):
    n = n+1

    # compute numerical solution
    time_integrator.elastic_RK4(v, s, v, s, rho, mu, nx, dx, order, y, t, dt, r0, r1,  tau_11,\
                    tau_21, tau_12, tau_22, type_0, forcing)

    # Analytical solution
    rate.mms(U, V, U_t, V_t, U_x, V_x, y, t+dt, type_0)

    # compute error and append to the error array
    EU.append(np.linalg.norm(U-v)/A)
    EV.append(np.linalg.norm(V-s)/B)

    T.append(t)

    # Updating plots
    if n % iplot == 0:
        for l in line1:
            l.remove()
            del l
        for l in line2:
            l.remove()
            del l
        for l in line3:
            l.remove()
            del l
        for l in line4:
            l.remove()
            del l

        # Display lines
        line1 = ax1.plot(y, v, 'r', y, U, 'k--')
        ax1.legend(iter(line1),('Numerical', 'Analytical'))
        line2 = ax2.plot(y, s, 'r', y, V, 'k--')
        ax2.legend(iter(line2),('Numerical', 'Analytical'))
        line3 = ax3.plot(T, EU, 'k--')
        ax3.set_yscale("log")#, nonposx='clip')
        line4 = ax4.plot(T, EV, 'k--')
        ax4.set_yscale("log")#, nonposx='clip')
        plt.gcf().canvas.draw()

plt.ioff()
plt.show()

# Simulation end time
stop = timeit.default_timer()
print('total simulation time = ', stop - start)                   # print the time required for simulation
print('spatial order  of accuracy = ', order)                                  # print the polynomial degree used
print('number of grid points = ', nx)                     # print the degree of freedom
print('maximum relative error in particle velocity = ', max(EU))  # max. relative error in particle velocity
print('maximum relative error in stress = ', max(EV))             # max. relative error in stress
```

```python

```


# Kozeny-Carman equation

\begin{equation}
    K = \dfrac{d_p^2}{180}\dfrac{\theta^3}{(1-\theta)^2} \dfrac{\rho g }{\mu}
\end{equation}

```python
%reset -f
```

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import root

#Globals
rho = 1000.  #kg/m3
g   = 9.81   #cm/s2
mu  = 0.001  #Ns/m2
dp  = 4.4E-4 #m

def KozenyCarman(theta):
    return dp**2 * theta**3 * rho * g / (180 * (1-theta)**2 * mu)

def KozenyCarman(theta):
    return dp**2 * theta**3 * rho * g / (180 * (1-theta)**2 * mu)

def findTheta(K_expected=1.0E-8):

    def minimizer(theta):
        K_init = KozenyCarman(theta)
        return (K_init - K_expected)**2

    solution = root(minimizer,0.1)
    print(solution.message + f" >> Porosity = {solution.x}")
    return solution.x
```

```python
porosity = np.linspace(0.001,0.5,100)
hydrCond = KozenyCarman(porosity)
```

```python
fig,ax = plt.subplots(figsize=(8,5),facecolor="white");

ax.plot(porosity,hydrCond,lw=3,c="blue",label='Kozeny-Carman')
ax.plot(porosity,840*(porosity**3.1),lw=3,c="red",label="Chen2010")
ax.set_yscale('log')
ax.set_xlabel("Porosity $\\theta$ ")
ax.set_ylabel("Hydraulic conductivity \n$K$ [m/s]")
ax.axhline(y=1.0E-8,lw=1,ls='dotted')
ax.legend()
plt.show()
```

```python
theta2 = findTheta(1.0E-7)
```

    The solution converged. >> Porosity = [0.02086702]

```python
print("{:.4E} m/s".format(KozenyCarman(0.35)))
```

    1.0707E-03 m/s

```python
from jupypft import attachmentRateCFT
```

```python
katt,_ = attachmentRateCFT.attachmentRate(dp=1.0E-7,dc=4.4E-4,
    q=0.35E-3,
    theta=0.35,
    visco=0.001,
    rho_f=1000.,
    rho_p=1050.0,
    A=1.0E-20,
    T=298.0,
    alpha=0.0043273861959162,
    debug=True)
```

    Diffusion coeff:  4.3654E-12
    Darcy velocity:   3.5000E-04
    Pore-water vel:   1.0000E-03
    ---
    Happel parameter: 5.2527E+01
    NR number:        2.2727E-04
    NPe number:       3.5277E+04
    NvW number:       2.4305E+00
    NGr number:       3.1211E-06
    ---
    etaD collector:   1.0409E-02
    etaI collector:   1.9641E-05
    etaG collector:   2.8626E-07
    eta0 collector:   1.0429E-02
    ---
    Attach rate   :   1.0000E-04

```python
"{:.6E}".format(0.0043273861959162)
```

    '4.327386E-03'

```python
1.0E-4/katt
```

    1.0000000000000002

```python

```


```python
import sympy
from sympy.physics.quantum import TensorProduct as kr
sympy.init_printing(use_latex='mathjax')

import kwant
from discretizer import Discretizer
from discretizer import momentum_operators

from types import SimpleNamespace

import numpy as np
import scipy.sparse.linalg as sla

from matplotlib import pyplot
%matplotlib inline
```

```python
params = {
    "HgTe" : SimpleNamespace(A=3.65, B=-68.6, D=-51.1, M=-0.01, C=0),
    "InAs/GaSb" : SimpleNamespace(A=0.37, B=-66.0, D=-5.8, M=-0.0075, C=0)
}
```

```python
s0 = sympy.eye(2)
sx = sympy.physics.matrices.msigma(1)
sy = sympy.physics.matrices.msigma(2)
sz = sympy.physics.matrices.msigma(3)

kx, ky, kz = momentum_operators
A, B, D, M, C = sympy.symbols('A B D M C', real=True)

k = sympy.sqrt(kx**2+ky**2)
hamiltonian = (+ C * kr(s0, s0) + M * kr(s0, sz) - D * k**2 * kr(s0, s0)
               - B * k**2 * kr(s0, sz) + A * (kx * kr(sz, sx) - ky * kr(s0, sy)))

hamiltonian
```

$$\left[\begin{matrix}- B \left(k_{x}^{2} + k_{y}^{2}\right) + C - D \left(k_{x}^{2} + k_{y}^{2}\right) + M & A \left(k_{x} + i k_{y}\right) & 0 & 0\\A \left(k_{x} - i k_{y}\right) & B \left(k_{x}^{2} + k_{y}^{2}\right) + C - D \left(k_{x}^{2} + k_{y}^{2}\right) - M & 0 & 0\\0 & 0 & - B \left(k_{x}^{2} + k_{y}^{2}\right) + C - D \left(k_{x}^{2} + k_{y}^{2}\right) + M & A \left(- k_{x} + i k_{y}\right)\\0 & 0 & A \left(- k_{x} - i k_{y}\right) & B \left(k_{x}^{2} + k_{y}^{2}\right) + C - D \left(k_{x}^{2} + k_{y}^{2}\right) - M\end{matrix}\right]$$

```python
# get tight binding Hamiltonian by creating Discretizer instance
tb = Discretizer(hamiltonian, lattice_constant=20, verbose=True)
```

    Discrete coordinates set to:  ['x', 'y']

    Function generated for (0, 1):
    def _anonymous_func(site1, site2, p):
        (x, y, ) = site2.pos
        D, B, A = p.D, p.B, p.A
        return (np.array([[B/400 + D/400, -A/40, 0, 0], [A/40, -B/400 + D/400, 0, 0], [0, 0, B/400 + D/400, -A/40], [0, 0, A/40, -B/400 + D/400]]))

    Function generated for (1, 0):
    def _anonymous_func(site1, site2, p):
        (x, y, ) = site2.pos
        D, B, A = p.D, p.B, p.A
        return (np.array([[B/400 + D/400, 1.j*A/40, 0, 0], [1.j*A/40, -B/400 + D/400, 0, 0], [0, 0, B/400 + D/400, -1.j*A/40], [0, 0, -1.j*A/40, -B/400 + D/400]]))

    Function generated for (0, 0):
    def _anonymous_func(site, p):
        (x, y, ) = site.pos
        D, M, B, C = p.D, p.M, p.B, p.C
        return (np.array([[-B/100 + C - D/100 + M, 0, 0, 0], [0, B/100 + C - D/100 - M, 0, 0], [0, 0, -B/100 + C - D/100 + M, 0], [0, 0, 0, B/100 + C - D/100 - M]]))

```python
L, W = 2000, 1000

def shape(pos):
    (x, y) = pos
    return (0 <= y < W and 0 <= x < L)

def lead_shape(pos):
    (x, y) = pos
    return (0 <= y < W)

# call helper method of Discretizer instance
# to easily create Kwant's system
sys = tb.build(shape, (0, 0))
lead = tb.build(lead_shape, (0, 0), symmetry=(-1, 0))

sys.attach_lead(lead)
sys.attach_lead(lead.reversed())

sys = sys.finalized()

kwant.plot(sys)
pyplot.show()
```

```python
material = "HgTe"
kwant.plotter.bands(sys.leads[0], args=[params[material]],
                    momenta=np.linspace(-0.3, 0.3, 100, endpoint=False), show=False)
if material == "HgTe":
    pyplot.ylim(-0.05, 0.05)
else:
    pyplot.ylim(-0.02, 0.02)
```

```python
E = 0.0
ldos = kwant.ldos(sys, energy=E, args=[params[material]])

ldos_sum = ldos[::4] + ldos[1::4] + ldos[2::4] + ldos[3::4]
kwant.plotter.map(sys, ldos_sum)
pyplot.show()
```


```python

import os
import sys
import numpy as np
import pandas as pd
from scipy.io import mmread
from scipy.linalg import hessenberg
import scipy.linalg as sl
sys.path.append("../qr")

from qr import *
import sympy as sp
```

```python
a = np.random.default_rng().random(size = (10, 10))
```

```python
%%timeit
max(a[0])
```

    1.19 µs ± 22.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

```python
%%timeit
np.max(a[0])
```

    3.42 µs ± 97.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
path = "../test_matrices"
mat_1_file = "west0381"
ext = ".mtx.gz"

mat = mmread(os.path.join(path, "".join((mat_1_file, ext))))
```

```python
m = mat.toarray()
m
```

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           ...,
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 1., 0., 0.]])

```python
import mpmath as mpm

mpm.dps = 15

def complex_matrix(n: int, a: float, b: float) -> np.ndarray:
	if a >= b:
		raise ValueError("Required: b > a")

	r = (b - a) * np.random.default_rng().random(size = (n, n)) + a
	c = (b - a) * np.random.default_rng().random(size = (n, n)) + a
	m = r + 1j * c

	return m.astype(np.complex128)

def householder_reflector(x: np.array):
	"""
	Produces the Householder
	vector based on the input
	vector x. The householder
 	vector acts as:

	|a_1|		|alpha|
	|a_2|	->	|0|
	|a_3|		|0|

	Parameters
	----------
	x:
		A numpy array who's entries
		after the 1st element needs to
		be 0ed.

	Returns
	-------
	A numpy array that acts as the
	Householder vector.
	"""
	u = x.copy()

	rho = -np.exp(1j * np.angle(u[0]), dtype = np.complex128)

	# Set the Householder vector
	# to u = u \pm alpha e_1 to
	# avoid cancellation.
	u[0] -= rho * mpm.norm(u)

	# Vector needs to have 1
	# in the 2nd dimension.
	# print(u)
	return u.reshape(-1, 1)

def hessenberg_transform_1(M: np.ndarray) -> np.ndarray:
	"""
	Converts a given matrix to
	Hessenberg form using
	Houeholder transformations.

	Parameters
	----------
	M:
 		A complex square
		numpy 2darray.

	Returns
	-------
	A tuple consisting of numpy
 	2-D arrays which are the
	hessenberg form and the
	permutation matrix.
	"""
	h = M.copy()
	n = np.array(h.tolist()).shape[0]
	u = np.eye(n, dtype = np.complex128)
	householder_vectors = list()

	# MAIN LOOP.
	for l in range(n - 2):
		# Get the Householder vector for h.
		t = householder_reflector(h[l + 1 :, l])

		# Norm**2 of the Householder vector.
		t_norm_squared = t.conj().T @ t

		# p = np.eye(h[l + 1:, l].shape[0]) - 2 * (np.outer(t, t)) / t_norm_squared

		# # Resize and refactor the Householder matrix.
		# p = np.pad(p, ((l + 1, 0), (l + 1, 0)), mode = "constant", constant_values = ((0, 0), (0, 0)))
		# for k in range(l + 1):
		# 	p[k, k] = 1

		# Perform a similarity transformation on h
		# using the Householder matrix.
		# h = p @ h @ p.

		# --- REAL --- #
		# Left multiplication by I - 2uu^{*}.
		# h_real[l + 1 :, l :] -= 2 * (t @ (t.conj().T @ h_real[l + 1 :, l :])) / t_norm_squared
		# Right multiplication by I - 2uu^{*}.
		# h_real[ :, l + 1 :] -= 2 * ((h[ :, l + 1 :] @ t) @ t.conj().T) / t_norm_squared
		# print(f"{np.array(h[l + 1 :, l :].tolist()).shape = }")
		# print(f"{np.array(t.transpose_conj().tolist()).shape = }")
		# print(f"{np.array((t.transpose_conj() * h[l + 1 :, l :]).tolist()).shape = }")
		factor = 2 / t_norm_squared

		h[l + 1 :, l :] -= factor * (t @ (t.conj().T @ h[l + 1 :, l :]))

		# --- IMAGINARY --- #
		# Left multiplication by I - 2uu^{*}.
		# h_imag[l + 1 :, l :] -= 2 * (t @ (t.conj().T @ h_imag[l + 1 :, l :])) / t_norm_squared
		# Right multiplication by I - 2uu^{*}.
		# h_imag[ :, l + 1 :] -= 2 * ((h[ :, l + 1 :] @ t) @ t.conj().T) / t_norm_squared
		h[ :, l + 1 :] -= factor * ((h[ :, l + 1 :] @ t) @ t.conj().T)

		# Force elements below main
		# subdiagonal to be 0.
		h[l + 2 :, l] = 0.0

		# Store the transformations
		# to compute u.
		householder_vectors.append(t)

	# Store the transformations.
	for k in reversed(range(n - 2)):
		t = householder_vectors[k]
		t_norm_squared = np.dot(t.conj().T, t)
		u[k + 1 :, k + 1 :] = 2 * t * (t.conj().T @ u[k + 1 :, k + 1 :]) / t_norm_squared

	# h = h_real + 1j * h_imag
	return h, u
```

```python
n = 1000
a = 10.0
b = 20.0
# m = complex_matrix(n, a, b)
# M = mpm.matrix(m.tolist())
hess_from_alg, _ = hessenberg_transform_1(m)
hess_from_scipy = hessenberg(m)
```

    ../qr/hessenberg.py:51: ComplexWarning: Casting complex values to real discards the imaginary part

```python
%%capture cap --no-stderr

pd.options.display.max_columns = 200
pd.set_option("display.width", 1000)
pd.set_option("display.max_columns", 200)
pd.set_option("display.max_rows", 1000)

# print(f" Hessenberged:\n {pd.DataFrame(hess_alg)}")
# print(f"Hessenberged (scipy):\n {pd.DataFrame(hess_from_scipy)}")
eigs = np.sort(np.linalg.eig(hess_from_alg)[0])
eigs_scipy = np.sort(np.linalg.eig(hess_from_scipy)[0])
print(f"Eigs:\n {pd.DataFrame(np.vstack([eigs, eigs_scipy]).T)}")
print(f"Equality of eigs: {np.testing.assert_allclose(eigs_scipy, eigs, rtol = 1e-6)}")
```

```python
with open("test_ipynb_output.txt", "w") as f:
    f.write(cap.stdout)
```

```python
sl.blas.daxpy([1, 2, 3], [1, 2, 3], a = 0.5)
```

    array([1.5, 3. , 4.5])

```python
def sign(z: complex) -> complex:
		if z == 0:
			return 1
		return z / abs(z)

sign(-2.0 + 2.j)
```

    (-0.7071067811865475+0.7071067811865475j)

```python
a = np.array([1.00345345, 2, 1, -1, 2])
b = np.array([1.00354, 2, 1, -1, 2])
[i for i, _ in enumerate(a) if np.isclose(_, 1.00354, 1e-3)]
```

    0

```python
np.round(1.011234, 1) == 1.0
```

    True

```python
dec = 6
a = f"{0:.{dec}f}"
print(float(a))
```

```python
np.prod([1, 2, 3, 4])
```

    24