# Music Machine Learning - Support Vector Machines (SVM)

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A recap on the [motivation](#motivation) behind Support Vector Machines (SVM)
2. A simple use case for [understanding and fitting SVMs](#fitting) with the scikit library
3. Developing our own [implementation](#implem) of linear SVMs
4. An introuction to [kernels](#improve) in SVMs and how to implement them

## Support Vector Machines (SVM)

In this tutorial, we will cover a more advanced classification algorithm through the use of *Support Vector Machines* (SVMs). The goal is to gain an intuition of how SVMs works and how to use *Gaussian kernel* with SVMs to find out the decision boundary. The implementation proposed here follows a simplified version of the *Sequential Minimal Optimization* (SMO) algorithm for training support vector machines, which is detailed in the following [paper](http://cs229.stanford.edu/materials/smo.pdf). The complete SMO algorithm can then be quite easily implemented following the [full paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf).

Once again, to simplify your work, we provide the following set of functions that you should find in the `data` folder, and in the `helper_plot` file

  |**File**|*Explanation*|
  |-------:|:------------|
  |`svm_linear.mat`|Example data for a quasi-linear problem|
  |`svm_gaussian.mat`|Example data for a non-linear but well-defined problem|
  |`plot_data`|Allows to plot the set of data points|
  |`visualize_boundary`|Plots a non-linear decision boundary from a SVM model|
  |`plot_svc_decision_function`|Plots a linear decision boundary|

```python
import numpy as np
from scipy import io as sio
from matplotlib import pyplot as plt
from helper_plot import plot_data, visualize_boundary, visualize_boundary_linear, hdr_plot_style, plot_svc_decision_function
```

## Motivation

In the previous course on neural networks, we learned how to train a simple neuron. We saw that this simple model could be interpreted as defining a line, separating the space into two different regions. This allows to consider problems of *discriminative classification*: rather than modeling each class, we simply find a line that divides the classes from each other. As an example of these _linearly separable_ problems, consider the simple case of a classification task, in which the two classes of points are well separated:

```python
from sklearn.datasets.samples_generator import make_blobs
hdr_plot_style()
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.65)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter');
plt.title("Linear classification example");
```

As we have seen previously, a single neuron classifier would attempt to draw a straight line separating the two sets of data, and thereby create a model for classification, by optimizing the parameters $\theta=\{\mathbf{w}, b\}$ of

$$
\begin{equation}
y=\sigma\left(\sum_{i = 1}^{n}w_{i}.x_{i} + b\right)
\label{eq1}
\end{equation}
$$

For two dimensional data like that shown here, this is a task we could do by hand. But the major problem that we can directly see, is that there is an infinite number of lines that can perfectly discriminate between the two classes, as exemplified in the next example.

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-w')
plt.xlim(-1, 3.5);
```

Note that these solutions are all valid, and will mostly depend on the (random) initialization of our parameters. Although these all perfectly discriminate between our samples, we can clearly see that they are more or less "optimal" to the problem at hand. So our question today is how could we formalize this problem, and if we could find a formulation to adress and solve this issue.

### Notion of _margin_

The intuition behind comparing different solutions is that rather than estimating only the line between the classes, we can also look at the *margin* of some width defined around those line. In our simple case, it would look like this

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b, d in [(1, 0.65, 0.25), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.15)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-w')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='w', color='#AAAAAA', alpha=0.6)
plt.xlim(-1, 3.5);
```

In *Support Vector Machines (SVMs)*, the goal is to find that optimal model, in which our separating line maximizes this margin. Hence, SVMs are part of a class of models called *maximum margin* estimator.

## Fitting a SVM

More formally the goal of SVMs is that given a set of input vectors $ \mathbf{x} $ separated in two classes (noted $ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $), we would like to find the *optimal boundary* (separating line) between the two classes. To do so, we want to find the separation that gives the *maximal margin* between the two sets. Hence, we can first define the following problem

$$
\begin{equation}
\begin{cases}
\mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}\geq1\\
\mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}\leq-1
\end{cases}
\label{eq1}
\end{equation}
$$

By setting the variable $ y_{i}={+}/{-}1 $, we can rewrite this problem as

$$
\begin{equation}
y_{i}\left(x_{i}\cdot\mathbf{w}+\mathbf{b}\right)-1\geq0
\label{eq2}
\end{equation}
$$

The idea is that for some of the input vectors (called *support vectors*), we will have an equality in equation (1), where $ \mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}=1 $ and $ \mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}=-1 $. By substracting those two equation, we see that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\label{eq3}
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification).

### Fitting the SVM with scikit-learn

First, we can see the result of fitting this approach to our original data. To do so, we will use the SVM model found in `scikit-learn` to train an SVM model on this data. To do so, we can use the following code

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear')
model.fit(X, y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

To visualize the results, we will use our quick convenience function that plots the SVM decision boundaries

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model);
```

This separating line is the one that maximizes the margin between our two sets of points.
Notice that some training points are right on the margin. These points are the pivotal elements of this fit, and are known as the *support vectors*, and give the algorithm its name. In `scikit-learn`, we can directly obtain the values of these points, stored in the ``support_vectors_`` attribute of the classifier:

```python
model.support_vectors_
```

    array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]])

The major advantage of SVM is that for the fit, only the position of the support vectors matter; any points further from the margin which are on the correct side do not modify the fit. Technically, this is because these points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin. We can see this, for example, if we plot the model learned from the first 50 or 150 points of this dataset

```python
# Plotting SVM trained on subsampled dataset
def plot_svm_subsample(N=10, ax=None):
    X, y = make_blobs(n_samples=200, centers=2, random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    model = SVC(kernel='linear', C=1E10)
    model.fit(X, y)
    ax = ax or plt.gca()
    ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 6)
    plot_svc_decision_function(model, ax)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
for axi, N in zip(ax, [60, 150]):
    plot_svm_subsample(N, axi)
    axi.set_title('N = {0}'.format(N))
```

In the left panel, we see the model and the support vectors for 60 training points.
In the right panel, we have doubled the number of training points, but the model has not changed: the three support vectors from the left panel are still the support vectors from the right panel.
This insensitivity to the exact behavior of distant points is one of the strengths of the SVM model.

## Implementing our own linear SVM

Here, we are going to try to implement our own linear SVM. We recall that, given a set of input vectors $ \mathbf{x} $ separated in two classes ($ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $, we would like to find the *optimal boundary* (separating line) between the two classes. This means that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\label{eq3}
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification). As seen in the slides, we know that this problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

Where the function $\phi(.)$ will be used later to introduce _kernels_, but for now, we will consider that $\phi(x)=x$, which reduces our problem to

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle x_{i},x \right\rangle +b
\end{equation}
$$

we can introduce a kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm and in the first part of this tutorial, we will only consider the *linear* kernel $K\left(x_{i},x\right)=x_{i}\cdot x$.

Now, if we look at the way the SVM model that we trained is implemented in `scikit-learn`, we have the following variables

```python
print(vars(model))
```

    {'decision_function_shape': 'ovr', '_impl': 'c_svc', 'kernel': 'linear', 'degree': 3, 'gamma': 'auto', 'coef0': 0.0, 'tol': 0.001, 'C': 1.0, 'nu': 0.0, 'epsilon': 0.0, 'shrinking': True, 'probability': False, 'cache_size': 200, 'class_weight': None, 'verbose': False, 'max_iter': -1, 'random_state': None, '_sparse': False, 'class_weight_': array([1., 1.]), 'classes_': array([0, 1]), '_gamma': 0.5, 'support_': array([18, 25,  7, 46], dtype=int32), 'support_vectors_': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'n_support_': array([2, 2], dtype=int32), 'dual_coef_': array([[-0.24571733, -0.99437155,  1.        ,  0.24008889]]), 'intercept_': array([3.45657164]), 'probA_': array([], dtype=float64), 'probB_': array([], dtype=float64), 'fit_status_': 0, 'shape_fit_': (50, 2), '_intercept_': array([-3.45657164]), '_dual_coef_': array([[ 0.24571733,  0.99437155, -1.        , -0.24008889]])}

For the moment, our variables of interest are
- `support_vectors_` : _Support vectors_ (dataset points that support classification)
- `dual_coef_` : The $\alpha_{i} * y_{i}$ (coefficients from the dual formulation) for the _support_vectors_
- `n_support_` : Number of support vectors for each class (ordered)
- `_intercept_` : Value of $b$ in our formulation (bias)

In order to begin our own implementation of the SVM, we are going to start by computing the _decision function_ of the model. We are going to slightly change the model structure, to better reflect the mathematical formulation than `scikit-learn`. Hence, our model structure and all of its corresponding parameters will be defined as

``` Python
model = {};
model["x"];            # Values of _support vectors_ in input data
model["y"];            # Classes labels of _support vectors_
model["n"];            # Number of support for each class
model["b"];            # Value of bias
model["alphas"];       # Value of alphas
model["kernel"];       # Type of kernel
model["w"];            # Weights of vectors in primal formulation
```

In the following, we show how to shift from the `scikit-learn` version to ours

```python
#%% Creating a fake (random) model
our_model = {};
# Values of _support vectors_ in input data
our_model["x"] = model.support_vectors_
# Corresponding classes of the support vectors
our_model["y"] = np.concatenate([[0] * model.n_support_[0], [1] * model.n_support_[1]])
# Number of vectors for each class
our_model["n"] = model.n_support_
# Value of threshold
our_model["b"] = model._intercept_
# Values of alphas
our_model["alphas"] = model._dual_coef_[0]
print(our_model)
```

    {'x': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'y': array([0, 0, 1, 1]), 'n': array([2, 2], dtype=int32), 'b': array([-3.45657164]), 'alphas': array([ 0.24571733,  0.99437155, -1.        , -0.24008889])}

Computing the _predictions_ of the SVM is performed by calling `model.decision_function(X)` in `scikit_learn`. We are going to implement this behavior ourselves by coding our own `decision_function` function

``` Python
def decision_function(model, X)
    """
    % Returns a vector of predictions using a trained SVM model.
    % model   : svm model returned from svmTrain.
    % X       : m x n matrix where each example is a row.
    """
    return pred
```

For the time being we are using a _linear kernel_, but we might later use a different type of kernel, so we will directly use a `kernel` function which computes our kernel between the support vectors `x_s` and a dataset `X`.

***

**Exercice**
1. Implement the linear kernel function
1. Complete the decision function based on the dual formulation

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    ######################
    # Solution
    if (type == 'linear'):
        k = [np.dot(x_s, X) for x_s in x]
    return k
    ######################

    if (type == 'linear'):
        k = ...
    return k

# This function should replicates the scikit-learn one
def decision_function(model, X):

    ######################
    # YOUR CODE GOES HERE
    ######################

    ######################
    # Solution
    # calculate the kernels
    k = kernel(model["x"], X)
    c = 0
    for p in range(len(k)):
        c += model["alphas"][p] * k[p]
    # add the intercept
    return c + model["b"]
    ######################

    return ...

# Evaluate the decision function
ax = plt.figure(figsize=(10,8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(our_model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--']);
ax.scatter(our_model["x"][:, 0], our_model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
```

## Learning phase

In order to perform learning, we will try to maximize the *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. $C$ is a chosen *regularization hyper-parameter*. To ease your work, we will implement only a simplified version of the SMO algorithm which is mainly focused on the optimization aspects. The SMO algorithm selects two $\alpha$ parameters, $\alpha_{i}$ and $\alpha_{j}$ and optimizes the objective value jointly for both these $\alpha_{i,j}$. Finally it adjusts the $b$ parameter based on the new $\alpha_{i,j}$. This process is repeated until the $\alpha_{i,j}$ converge.

### Selecting $\alpha$

The full SMO algorithm use heuristics to choose which $\alpha_{i}$ and $\alpha_{j}$ to optimize (which greatly increase the convergence speed. However, for this simplified version, we will simply iterate on all $\alpha_{i},i=1,\cdots ,m$ and select a random $\alpha_{j}$, to perform the joint optimization of $\alpha_{i}$ and $\alpha_{j}$

### Optimizing $\alpha_{i}$ and $\alpha_{j}$

To jointly optimize the value, we first find bounds $L$ and $H$ such that $L\leq \alpha_{j} \leq H$ holds to satisfy the constraint of the original problem $0\leq \alpha_{i} \leq C$

$$
\begin{array}{ccc}
if\mbox{ }y_{i}\neq y_{j}, & L=max\left(0,\alpha_{j}-\alpha_{i}\right), & H=min\left(C,C+\alpha_{j}-\alpha_{i}\right)\\
if\mbox{ }y_{i}=y_{j}, & L=max\left(0,\alpha_{i}+\alpha_{j}-C\right), & H=min\left(C,\alpha_{i}+\alpha_{j}\right)
\end{array}
$$

Now we want to find $\alpha_{j}$ so as to maximize the objective function. It can be shown (as in the original paper) that the optimal $\alpha_{j}$ is given by

$$
\alpha_{j}=\alpha_{j}-\frac{y_{j}\left(E_{i}-E_{j}\right)}{\eta}
$$

where

$$
\begin{array}{ccc}
E_{k} & = & f\left(x_{k}\right)-y_{k}\\
\eta & = & 2\left\langle x_{i},x_{j}\right\rangle -\left\langle x_{i},x_{i}\right\rangle -\left\langle x_{j},x_{j}\right\rangle
\end{array}
$$

$E_{k}$ defines the error between the prediction of the SVM on the example $k$ and its desired label. When computing $\eta$, we can also replace the inner product by a kernel function $K$. Finally, we clip the value of $\alpha_{j}$ to lie within the selected bounds.

$$
\alpha_{j}=\begin{cases}
H & if\mbox{ }\alpha_{j}>H\\
\alpha_{j} & if\mbox{ }L\leq\alpha_{j}\leq H\\
L & if\mbox{ }\alpha_{j}<L
\end{cases}
$$

Now that we have solved $\alpha_{j}$, we can update the value for $\alpha_{i}$ through

$$
\alpha_{i}^{t}=\alpha_{i}^{t-1}+y_{i}y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)
$$

### Computing threshold $b$

Finally, we can compute the threshold $b$ to satisfy the original constraints by selecting

$$
b=\begin{cases}
b_{1} & if\mbox{ }0<\alpha_{i}<C\\
b_{2} & if\mbox{ }0<\alpha_{j}<C\\
\left(b_{1}+b_{2}\right)/2 & otherwise
\end{cases}
$$

where

$$
b_{1}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{i}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{i},x_{j}\right\rangle
$$
$$
b_{2}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{j}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{j},x_{j}\right\rangle
$$

For the first part of this tutorial, we will compute the main iterations of the algorithm (minimization of the objective function), while relying on a *linear* kernel. This implies that we will only be able to perform linear discrimination. However, remember that the formulation of the SVMs provide an *optimal* and (gloriously) *convex* answer to this problem.

```Python
def svm_train(X, Y, C, kernelFunction, tol, maxIter):
    """
    % Trains an SVM classifier using a simplified SMO algorithm.
    % X       : m x n matrix of m training examples (with n-dimensional features).
    % Y       : column vector of class identifiers
    % C       : standard SVM regularization parameter
    % tol     : tolerance value used for determining equality of floating point numbers.
    % maxIter : number of iterations over the dataset before the algorithm stops.
    """
    return model
```

***

**Exercise**

  1. Update the `svm_train` code to perform the training of a SVM with linear kernel.
  2. Run your `svm_train` function on our linear dataset to obtain similar results as `scikit-learn`
  3. Update the framework to display the decision boundary at each iteration of the algorithm.

***

```python
def svm_train(X, Y_b, C, kernel_type, tol=1e-3, max_iter=5):
    """
    SVMTRAIN Trains an SVM classifier using a simplified version of the SMO
    # X       : m x n matrix of m training examples (with n-dimensional features).
    # Y       : column vector of class identifiers (0 or 1)
    # C       : standard SVM regularization parameter
    # tol     : tolerance value used for determining equality of floating point numbers.
    # maxIter : number of iterations over the dataset before the algorithm stops.
    """
    # Data parameters
    m, n = X.shape
    # Variables
    alphas = np.zeros((m, 1))
    b, passes, eta, L, H = 0, 0, 0, 0, 0
    E = np.zeros((m, 1))
    Y = Y_b.copy()
    Y[Y == 0] = -1

    # Pre-compute the kernel matrix (in practice, the optimized packages will _not_ do this)
    K = np.array(kernel(X, X.transpose(), kernel_type))

    # Train
    print('Training ...')
    while passes < max_iter:
        # Check that some alphas changed
        num_changed_alphas = 0;
        # Iterative over all alpha_i
        for i in range(m):
            ######################
            # YOUR CODE GOES HERE
            # => Perform one pass of the SMO algorithm for each alpha_i
            ######################

            ######################
            # Solution
            E[i] = (b + np.sum(alphas.transpose() * Y.transpose() * K[:, i]) - Y[i])
            if ((np.dot(Y[i], E[i]) < -tol and alphas[i] < C) or (np.dot(Y[i], E[i]) > tol and alphas[i] > 0)):
                # In practice, there are many heuristics one can use to select
                # the i and j. In this simplified code, we select them randomly.

                j = int(np.floor(m * np.random.rand()))
                while j == i:  # Make sure i \neq j
                    j = int(np.floor(m * np.random.rand()))

                # Calculate Ej = f(x(j)) - y(j) using (2).
                E[j] = b + np.sum(alphas.transpose() * Y.transpose() * K[:, j]) - Y[j]
                # Save old alphas
                alpha_i_old = alphas[i].copy()
                alpha_j_old = alphas[j].copy()
                # Compute L and H by (10) or (11).
                if (Y[i] == Y[j]):
                    L = np.max([0, alphas[j] + alphas[i] - C])
                    H = np.min([C, alphas[j] + alphas[i]])
                else:
                    L = np.max([0, alphas[j] - alphas[i]])
                    H = np.min([C, C + alphas[j] - alphas[i]])
                if (L == H):
                    # Continue to next i.
                    continue
                # Compute eta by (14).
                eta = 2 * K[i, j] - K[i, i] - K[j, j];
                if (eta >= 0):
                    # continue to next i.
                    continue;
                # Compute and clip new value for alpha j using (12) and (15).
                alphas[j] = alphas[j].copy() - (Y[j] * (E[i] - E[j])) / eta
                # Clip
                alphas[j] = np.min([H, alphas[j]])#.copy()
                alphas[j] = np.max([L, alphas[j]])#.copy()
                # Check if change in alpha is significant
                if (np.abs(alphas[j] - alpha_j_old) < tol):
                    # continue to next i and replace anyway
                    alphas[j] = alpha_j_old
                    continue
                # Determine value for alpha i using (16).
                alphas[i] = (alphas[i] + Y[i]*Y[j]*(alpha_j_old - alphas[j]))
                # Compute b1 and b2 using (17) and (18) respectively.
                b1 = b - E[i] - Y[i] * (alphas[i] - alpha_i_old) *  K[i, j].transpose() - Y[j] * (alphas[j] - alpha_j_old) *  K[i, j].transpose()
                b2 = b - E[j] - Y[i] * (alphas[i] - alpha_i_old) *  K[i, j].transpose() - Y[j] * (alphas[j] - alpha_j_old) *  K[j, j].transpose()
                # Compute b by (19).
                if (0 < alphas[i] and alphas[i] < C):
                    b = b1.copy()
                elif (0 < alphas[j] and alphas[j] < C):
                    b = b2.copy()
                else:
                    b = ((b1+b2)/2).copy();
                num_changed_alphas = num_changed_alphas + 1;

            ######################

        if (num_changed_alphas == 0):
            passes = passes + 1
        else:
            passes = 0
    print(' Done !');
    # Save the model
    idx = (alphas > 0).transpose()[0];
    model = {}
    model["x"] = X[idx, :]
    model["y"] = Y[idx]
    model["b"] = b
    model["alphas"] = alphas[idx, 0] * Y[idx]
    model["kernel"] = kernel
    return model

# Training Linear SVM
print('Training Linear SVM ...')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-9       # Tolerance value for thresholds
nb_iterations = 50      # Number of iterations
# Perform the training (try to fiddle the previous values to see effect).
model = svm_train(X, y, C, 'linear', tolerance, nb_iterations);
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

# Beyond linear problems with _kernels_

For the time being, our SVM is only able to solve _linear_ problems. However, the SVM approach becomes truly powerful is when it is combined with *kernels*. As seen in the slides, the major idea is to project our (non-linearly separable) data into a higher-dimensional space, where it becomes linearly separable (allowing to fit for nonlinear relationships with a linear classifier).

To better understand this idea, we can look at the same non linearly separable data that we discussed in the slides. Here, we try to fit a _linear_ SVM to this non-linear problem.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(clf, plot_support=False);
```

It is clear that no linear discrimination will *ever* be able to separate this data. As discussed, we need to think about how we might project the data into a higher dimension such that a linear separator *would* be sufficient. For example, one simple projection we could use would be to compute a *radial basis function* centered on the middle group of points.

```python
r = np.exp(-(X ** 2).sum(1))
```

We can visualize this extra data dimension using a three-dimensional plot

```python
from mpl_toolkits import mplot3d

def plot_3D(elev=30, azim=30, X=X, y=y):
    plt.figure(figsize=(10, 8));
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, edgecolor='w', cmap='winter')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')
plot_3D(elev=30, azim=-50, X=X, y=y);
```

We can see that with this additional dimension, the data becomes trivially linearly separable, by drawing a separating plane. Here we had to choose and carefully tune our projection. In general, the need to make such a choice is a problem, as we would like to somehow automatically find the best basis functions to use. One strategy to this end is to compute a basis function centered at *every* point in the dataset, and let the SVM algorithm find the result. This type of basis function transformation is known as a *kernel transformation*, as it is based on a similarity relationship (or kernel) between **each pair of points**.

To see this idea in action first, we will (as previously) use `scikit-learn`, where we can apply _kernelized SVM_ simply by changing our _linear kernel_ to an _RBF (radial basis function)_ kernel, using the ``kernel`` model hyperparameter:

```python
model = SVC(kernel='rbf', C=1E6)
model.fit(X, y)
```

    SVC(C=1000000.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

Once fitted, we can see that our SVM is able to find the largest margin between these two sets of data

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1], s=300, lw=1, facecolors='none');
```

Using this kernelized support vector machine, we learn a suitable nonlinear decision boundary.
This kernel transformation strategy is used often in machine learning to turn fast linear methods into fast nonlinear methods, especially for models in which the kernel trick can be used.

## Implementing the gaussian kernel

As we have seen previously, we know that the complete SVM optimization problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

where we can introduce the kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm. Consequently, the complete *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. The Gaussian kernel (also called *Radial Basis Function (RBF)* kernel) allows to perform the optimization of non-linear classification problems. It is defined as

$$
K\left(x,y\right)=exp\left(\frac{-\left\Vert x-y\right\Vert ^{2}}{\left(2\sigma^{2}\right)}\right)
$$

As seen in the slides, the underlying idea is that instead of trying to solve a non-linear separation problem in a given space, we might first transform the space into a target space in which the problem is linearly separable. Therefore, we transform the space through a kernel (which is directly expressed in the minimization problem) and solve the linear separation problem in the target space. Given the starter code and the previous exercise, you should be able to directly plug this kernel inside the `svmTrain` and `svmPredict` functions.

***

**Exercise**

  1. Update the `kernel` function code to be able to rely on a Gaussian kernel.
  2. Check that you can run the `decision_function` code to rely on this Gaussian kernel.
  2. Run the `svm_train` function on our new (non-linear) dataset to obtain similar results as `scikit-learn`
  3. Once again, display the decision boundary at each iteration of the algorithm.

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    ######################
    # Solution
    if (type == 'linear'):
        k = [np.dot(x_s, X) for x_s in x]
    elif kernel == 'gaussian':
        X2 = np.sum(np.power(X, 2), axis=1)
        K = X2 + (X2.transpose() - 2 * np.dot(X, X.transpose()))
        kernelV = np.exp(-np.sum(np.power(1,2))/(2 * np.power(0.1, 2)));
        k = np.power(kernelV, K)
    return k
    ######################

    if (type == 'linear'):
        k = ...
    elif kernel == 'gaussian':
        k = ...
    return k

plt.title('Traning Dataset')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-3       # Tolerance value for thresholds
nb_iterations = 20     # Number of iterations
sigma = 0.1            # Variance of Gaussian kernel
# Train the SVM with a Gaussian (RBF) kernel
model = svm_train(X, y, C, 'gaussian', tolerance, nb_iterations)
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

## Supplementary kernels

Now that you have mastered the concept of kernels and updated your code to include the Gaussian kernel, you can augment your classifier and optimizaton problem to include several different kernels as those proposed afterwards.

#### Polynomial kernel.

Intuitively, the polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these.

$$
K\left(x,y\right)=\left(x^{T}y+1\right)^{d}
$$

#### Sigmoid (tanh) kernel.

This kernel takes two parameters: $a$ and $r$. For $a > 0$, we can view $a$ as a scaling parameter of the input data, and $r$ as a shifting parameter that controls the threshold of mapping. For $a < 0$, the dot-product of the input data is not only scaled but reversed.

$$
K\left(x,y\right)=tanh\left(kx^{T}y+\Theta\right)
$$

***

**Exercise**

  1. Update the `kernel` function code to include these supplementary kernels.
  2. Test both the `decision_function` and `svm_train` codes for these new kernels.

***

```python

```

## Audio application

As seen in the previous tutorial, we have considered only a *binary classification* problem (where elements can only belong to one of two classes). This simplifies the work as we simply need to separate the space between two types of points. However, in real-world problems, we usually want to solve *multi-class* problems with any given number of classes. For this tutorial, we will rely on a simple trick which is to consider that a $n$-class problem can simply be restated as $n$ different binary classification problems. The underlying idea is that each class defines a binary classifier which tells us if a given element is part of this class, or belongs to *any of the other* classes.

***

**Exercise**

  1. Change the training loop to define $n$separate binary problems
  2. For each problem, train an SVM to perform classification.
  3. Evaluate the global classification accuracy of your system.

***

```python

```

```python
from helper_data import import_dataset, compute_transforms, compute_features
class_path = 'data/classification'
# 0.1 - Import the classification dataset
data_struct = import_dataset(class_path, 'classification')
# 0.2 - Pre-process the audio to obtain spectral transforms
data_struct = compute_transforms(data_struct)
# 0.3 - Compute a set of temporal and spectral features
data_struct = compute_features(data_struct)
```


# FFM232, Klassisk fysik och vektorfält - Föreläsningsanteckningar

 **[Christian Forssén](http://fy.chalmers.se/subatom/tsp/), Institutionen för fysik, Chalmers, Göteborg, Sverige**

Date: **Aug 29, 2016**

# 1. Fält och derivator

Ett fält är en fysikalisk storhet som kan beskrivas som en funktion av
positionen i rummet $\vec{r}$ och tiden $t$.  I den här kursen kommer vi
att behandla två typer av fält, skalära fält och vektorfält.

### Skalärfält

$$
\begin{equation}
\Phi = \Phi(\vec{r},t), \quad \Phi \in \mathbf{R},\quad \vec{r} \in \mathbf{R}^3
\end{equation}
$$

  Skalär  	 Symbol	                        Enhet

   tryck         	    $p$       	    Pascal (Pa), 1 Pa = 1 $\mathrm{N/m}^2$

   densitet      	    $\rho$    	    $\mathrm{kg/m}^3$

   temperatur    	    $T$       	    Kelvin (K), Celsiusgrader ($^\mathrm{o}\mathrm{C}$)

### Vektorfält

$$
\begin{equation}
\vec{F} = \vec{F}(\vec{r},t), \quad \vec{F} \in \mathbf{R}^3,\quad \vec{r} \in \mathbf{R}^3 \\
\end{equation}
$$

  Vektor  	   Symbol 	   Enhet

   hastighet     	    $\vec{v}$    	    m/s

   kraft         	    $\vec{F}$    	    N

   Magnetfält    	    $\vec{B}$    	    Tesla (T)

## Rum och koordinatsystem

En viktig egenskap hos skalärer och vektorer är att dessa är **oberoende** av koordinatsystem (transformation). Mer om detta senare när vi pratar om tensorfält.

I ett Cartesiskt koordinatsystem på $\mathbf{R}^3$ gäller att en allmän vektor $\vec{A}$ kan skrivas

$$
\begin{equation}
\vec{A} = A_x \hat{x} + A_y \hat{y} + A_z \hat{z}
\end{equation}
$$

där $\hat{x}$, $\hat{y}$ och $\hat{z}$ är ortonormerade **basvektorer**.

I detta koordinatsystem har vi följande partiella derivator $\frac{\partial}{\partial x}$, $\frac{\partial}{\partial y}$, $\frac{\partial}{\partial z}$. Vi använder ibland kortnotationen $\partial_x \equiv \frac{\partial}{\partial x}$.

Detta ger vektoroperatorn

$$
\begin{equation}
\nabla = \hat{x} \partial_x + \hat{y} \partial_y + \hat{z} \partial_z,
\end{equation}
$$

som uttalas *nabla* eller *dell*.

Ibland stöter man på notationen $\vec{\nabla}$ som tydliggör att detta är en **vektor**-operator, men vi använder samma beteckning som i kompendiet.

## Vektoralgebra

Repetera detta om ni känner er osäkra!

Exemplen nedan i Cartesiska koordinater.

### Addition

$$
\begin{equation}
\vec{A}+\vec{B} = (A_x + B_x) \hat{x} + (A_y + B_y) \hat{y} + (A_z + B_z) \hat{z}
\end{equation}
$$

och motsvarande för subtraktion.

Resultatet är en vektor.

### Skalärprodukt

$$
\begin{equation}
\vec{A} \cdot \vec{B} = |\vec{A}| |\vec{B}| \cos\theta,
\end{equation}
$$

där $|\vec{A}|$ och $|\vec{B}|$ är vektornormerna enligt nedan och $\theta$ är vinkeln mellan vektorerna.

På komponentform

$$
\begin{equation}
\vec{A} \cdot \vec{B} = A_x B_x + A_y B_y + A_z B_z,
\end{equation}
$$

eftersom $\hat{x} \cdot \hat{x} = \hat{y} \cdot \hat{y} = \hat{z} \cdot \hat{z} = 1$ och $\hat{x} \cdot \hat{y} = \hat{x} \cdot \hat{z} = \hat{y} \cdot \hat{z} = 0$.

Resultatet är en skalär.

### Vektornorm

$$
\begin{equation}
A = |\vec{A}| = (\vec{A} \cdot \vec{A})^{1/2} = (A_x^2 + A_y^2 + A_z^2)^{1/2}.
\end{equation}
$$

### Kryssprodukt

$$
\begin{equation}
\vec{A} \times \vec{B}  = \left|
\begin{array}{ccc}
\hat{x}  \hat{y}  \hat{z}
\end{equation}
$$

$$
\begin{equation}
A_x  A_y  A_z
\end{equation}
$$

$$
\begin{equation}
B_x  B_y  B_z
\end{array}
\right|
\end{equation}
$$

$$
\begin{equation}
 = (A_y B_z - A_z B_y)\hat{x} + (A_z B_x - A_x B_z)\hat{y} + (A_x B_y - A_y B_x) \hat{z}.
\end{equation}
$$

## Skalärfält

Ett skalärt fält representeras av ett tal i varje punkt.  Exempel på sådana fält är temperatur, tryck, elektrisk laddningstäthet.

### Nivåyta

Geometriskt brukar vi beskriva ett skalärt fält $\Phi(x,y,z,t)$ genom  att rita de ytor på vilka fältet är konstant.  Vi kallar sådana ytor för **nivåytor**, eller i två dimensioner **nivåkurvor**.  Vissa typer av nivåkurvor har fått speciella namn, till exempel isotermer för kurvor längs vilka temperaturen är konstant och isobarer längs vilka trycket är konstant.  Man kan konstruera nivåytorna genom att ställa upp  ekvationen $C = \Phi(x,y,z,t)$, och lösa den för olika C.

### Gradient, fältlinjer

Ett viktigt begrepp när vi arbetar med skalära fält är **gradienten**. Betrakta två punkter $(x,y,z)$ och $(x+\mbox{d}x,y+\mbox{d}y,z+\mbox{d}z)$. Skillnaden i skalären $\Phi$ mellan dessa båda punkter är

$$
\begin{equation}
  \mbox{d}\Phi = \Phi\left(x+\mbox{d}x, y+\mbox{d}y,z+\mbox{d}z\right) - \Phi\left(x,y,z\right) \nonumber
\end{equation}
$$

$$
\begin{equation}
 \approx \Phi\left(x,y,z\right) + \frac{\partial \Phi}{\partial x}\mbox{d}x
+ \frac{\partial \Phi}{\partial y}\mbox{d}y + \frac{\partial \Phi}{\partial z}
\mbox{d}z - \Phi\left(x,y,z\right) \nonumber
\end{equation}
$$

$$
\begin{equation}
 = \frac{\partial \Phi}{\partial x}\mbox{d}x
+ \frac{\partial \Phi}{\partial y}\mbox{d}y + \frac{\partial \Phi}{\partial z}
\mbox{d}z.
\end{equation}
$$

Den här skillnaden kan vi tolka som skalärprodukten mellan vektorn

$$
\begin{equation}
  \mbox{d}\vec{r} = \left(\mbox{d}x,\mbox{d}y,\mbox{d}z\right),
\end{equation}
$$

som beskriver separationen mellan de båda punkterna och vektorn

$$
\begin{equation}
  \nabla \Phi = \left(\frac{\partial \Phi}{\partial x},
\frac{\partial \Phi}{\partial y}, \frac{\partial \Phi}{\partial z}\right),
\end{equation}
$$

som vi kallar för gradienten av $\Phi$. Dvs $\mbox{d}\Phi = \mbox{d}\vec{r} \cdot \nabla \Phi$.

Lägg märke till att om $\mbox{d}\vec{r}$ har längden 1, så är $\mbox{d}\vec{r} \cdot \nabla \Phi$ riktningsderivatan för $\Phi$ i riktningen $\mbox{d}\vec{r}$, det vill säga den mäter hur snabbt $\Phi$ varierar i denna riktning.

Gradienten

$$
\begin{equation}
  \nabla = \left(\frac{\partial}{\partial x}, \frac{\partial}{\partial y},
\frac{\partial}{\partial z}\right),
\end{equation}
$$

är i sig själv inte en vektor, utan den är en vektoroperator, som tar ett skalärfält och producerar ett vektorfält.  Skillnaden mellan en vektoroperator och en vektor kommer att märkas när vi bekantar oss med räknereglerna för vektoroperatorer.

Det är enkelt att visa att $\nabla \Phi$ är ortogonal mot nivåytorna för $\Phi$.  Låt $\hat{e}_t$ vara en tangentvektor till en nivåyta genom en punkt $\vec{r}_0$. Eftersom $\Phi = \Phi_0$ (konstant) längs nivåytan gäller att

$$
\hat{e}_t \cdot \nabla \Phi = 0 \quad \Rightarrow \quad \hat{e}_t \perp \nabla \Phi.
$$

Kurvor som i varje punkt har $\nabla \Phi$ som tangentvektor kallas för **fältlinjer**.

### Exempel

Betrakta en bergstopp med en höjdyta som ges av

$$
\begin{equation}
h(\vec{r}) = \frac{h_0}{1+\frac{x^2+y^2}{R^2}},
\end{equation}
$$

där $h$ ger höjden över havet, $h_0$ är höjden på toppen av berget, och $R$ är ett mått på bergets utbredning.

```
import numpy as np
import pylab as p

# Make an x,y grid
grid = np.linspace(-10000,10000,500)
x, y = np.meshgrid(grid,grid)
# and the 2D field
R0=4000  # radius measure
h0= 2106 # peak height
h = h0 / (1 + (x**2 + y**2)/R0**2)
```

```
from mpl_toolkits.mplot3d import Axes3D
fig = p.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(x/1000, y/1000, h)
#ax.set_zlim3d(-1, 1)
ax.set_xlabel(r'$x$ (km)')
ax.set_ylabel(r'$y$ (km)')
ax.set_zlabel(r'$h$ (m)')
```

Vi kan göra en höjdkarta genom att rita upp några nivåytor, dvs kurvor som motsvarar konstanta värden på $h$. Vi ser enkelt att kurvan $x^2 + y^2 = C^2$ (konstant) ger $h_C = \frac{h_0}{1+\frac{C^2}{R^2}}$ (konstant) och att höjdkurvorna därför motsvarar cirklar med centrum i origo ($x=y=0$).

```
%matplotlib inline

from matplotlib import colors
p.figure(figsize=(6,6))
heights = np.linspace(500,2000,4)
CS = p.contour(x/1000,y/1000,h,heights,colors='k')
p.clabel(CS, inline=1, fontsize=10)
p.xlabel(r'$x$ (km)')
p.ylabel(r'$y$ (km)')
```

Eller kombinerat med höjdkurvor enligt en kontinuerlig färgskala

```
p.pcolormesh(x/1000, y/1000, h, cmap=p.get_cmap('cubehelix_r'))
```

Gradienten av detta fält kommer att motsvara höjdändringen per längdenhet i horisontalplanet. Vi finner denna genom att skriva $x^2 + y^2 = r^2$ så att $h=h(r)$. Vi får

$$
\begin{equation}
\frac{\mbox{d}h}{\mbox{d}r} = - \frac{h_0}{(1+r^2/R^2)^2} \frac{2r}{R^2}
\end{equation}
$$

Vidare är

$$
\begin{equation}
\frac{\partial r}{\partial x} = \frac{1}{2} (x^2 + y^2)^{-1/2} 2 x = \frac{x}{r},
\end{equation}
$$

och pss $\partial_y r = y/r$. Därför blir

$$
\begin{equation}
\nabla h = \hat{x} \partial_x h + \hat{y} \partial_y h
= \hat{x} \frac{\partial h}{\partial r} \frac{\partial r}{\partial x}
+ \hat{y} \frac{\partial h}{\partial r} \frac{\partial r}{\partial y}
\end{equation}
$$

$$
\begin{equation}
= - \frac{2h_0}{R^2 (1+r^2/R^2)^2} \left( x \hat{x} + y \hat{y} \right).
\end{equation}
$$

Riktningen på denna vektor ser vi är $x\hat{x} + y\hat{y} = r\hat{r}$, dvs den horisontella höjdgradienten är riktad radiellt inåt - mot större värden av $h$.

```
p.figure(figsize=(6,6))
grid_course =  np.linspace(-10000,10000,11)
x_c, y_c = np.meshgrid(grid_course,grid_course)
radius = 2000
outside = x_c**2 + y_c**2 > radius**2
gradh_x = - 2*h0 / (R0**2 * (1 + (x_c**2+y_c**2)/R0**2)**2) * x_c
gradh_y = - 2*h0 / (R0**2 * (1 + (x_c**2+y_c**2)/R0**2)**2) * y_c
p.quiver(x_c[outside]/1000,y_c[outside]/1000,gradh_x[outside],gradh_y[outside],scale=3)
p.xlabel(r'$x$ (km)')
p.ylabel(r'$y$ (km)')
```

## Vektorfält

Ett vektorfält beskriver på samma sätt en fysikalisk kvantitet som i varje punkt i rummet är en vektor.  Exempel på sådana kvantiteter är hastigheten i en strömmande fluid, den magnetiska fältstyrkan, och den elektriska strömtätheten.

### Fältlinjer

Geometriskt kan vi representera ett vektorfält genom att konstruera fältlinjer. Fältlinjerna till ett vektorfält $\vec{F}(\vec{r})$ är de kurvor som överallt har $\vec{F}(\vec{r})$ som tangentvektor.

För att förstå begreppet fältlinjer kan vi börja med att tänka oss ett vektorfält $\vec{V}(x,y,z)$ som representerar en hastighet i en fluid.  En testpartikel i fluiden följer då en bana $\vec{r}(t)$ som ges av

$$
\begin{equation}
  \frac{\mbox{d}\vec{r}}{\mbox{d}t} = \vec{V}\left(\vec{r}\right).
\end{equation}
$$

Analogt med detta definierar vi en fältlinje som den bana vi får genom att följa med ett godtyckligt vektorfält $\vec{F}$ som om det var ett hastighetsfält.  Ekvationen för ett sådant vektorfält är

$$
\begin{equation}
  \frac{\mbox{d}\vec{r}}{\mbox{d}\tau} = C \vec{F}\left(\vec{r}(\tau)\right),
\end{equation}
$$

där $C$ är en godtycklig konstant (dvs du kan välja den som du själv
vill, så länge som du inte sätter den till noll), och $\tau$ är en
parameter för att numrera punkterna längs fältlinjen.

### Exempel:

Vi vill konstruera fältlinjerna till

$$
\begin{equation}
  \vec{F}\left(\vec{r}\right) = F_0\left(\frac{x}{a} \hat{x} +
\hat{y}\right).
\end{equation}
$$

Detta ger oss differentialekvationerna

2
7

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\begin{equation}
  \frac{\mbox{d}y}{\mbox{d}\tau} = C F_0.
\end{equation}
$$

Vi väljer nu $C = a/F_0$, så att ekvationerna blir

2
9

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\begin{equation}
  \frac{\mbox{d}y}{\mbox{d}\tau} = a
\end{equation}
$$

Dessa har lösningen

3
1

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\begin{equation}
  y = a \tau + y_0
\end{equation}
$$

där $x_0$ och $y_0$ är den punkt på fältlinjen där $\tau = 0$.

### Divergens och rotation

Vi kan derivera ett vektorfält på två huvudsakliga sätt. Dels kan vi bilda en skalär genom **divergensen** $\nabla \cdot \vec{F}$ (eller $\mathrm{div} \vec{F}$), eller en vektor genom rotationen $\nabla \times \vec{F}$ (eller $\mathrm{rot} \vec{F}$). I cartesiska koordinater blir de explicita uttrycken

$$
\begin{equation}
\nabla \cdot \vec{F} = \partial_x F_x + \partial_y F_y + \partial_z F_z,
\end{equation}
$$

$$
\begin{equation}
\nabla \times \vec{F}  = \left|
\begin{array}{ccc}
\hat{x}  \hat{y}  \hat{z}
\end{equation}
$$

$$
\begin{equation}
\partial_{x}  \partial_{y}  \partial_{z}
\end{equation}
$$

$$
\begin{equation}
F_x  F_y  F_z
\end{array}
\right|
= (\partial_y F_z - \partial_z F_y)\hat{x} + (\partial_z F_x - \partial_x F_z)\hat{y} + (\partial_x F_y - \partial_y F_x) \hat{z}.
\end{equation}
$$

Divergensen ger ett mått på hur mycket fältet *"går isär"* (divergerar), medan rotationen mäter hur mycket det *"snurrar runt"*.

### Exempel

Vektorfältet $\vec{F} = x\hat{x} + y\hat{y} + z\hat{z}$ har divergensen $\nabla \cdot \vec{F} = 3$ och rotationen $\nabla \times \vec{F} = 0$ (fältet är rotationsfritt). Notera att negativa värden på divergensen betyder att fältet *"går ihop"* (konvergerar).

### Exempel

Hastighetsfältet för en roterande fluid $\vec{v} = \vec{\omega} \times \vec{r}$. Med $\vec{\omega} = \omega \hat{z}$ fås $\vec{v} = -\omega y\hat{x} + \omega sx\hat{y}$ och rotationen blir $\nabla \times \vec{v} = 2 \omega \hat{z}$ medan divergensen $\nabla \cdot \vec{v} = 0$. Detta specifika hastighetsfält är alltså **divergensfritt**.

### Laplaceoperatorn

Vi kommer också att stöta på den viktiga **Laplaceoperatorn**: $\Delta = \nabla \cdot \nabla$. Verkande på ett skalärt fält i cartesiska koordinater blir detta

$$
\begin{equation}
\Delta \Phi = \nabla \cdot \nabla \Phi = \left( \partial_x^2 + \partial_y^2 + \partial_z^2 \right) \Phi.
\end{equation}
$$

Likaså kan denna verka på ett vektorfält $\Delta \vec{F} = \nabla \cdot \nabla \vec{F} = \left( \partial_x^2 + \partial_y^2 + \partial_z^2 \right) \vec{F}$.

## Exempel: Potentialer

En potential, $\Phi$,  är ett viktigt specialfall av ett skalärt fält.  Ur en  potential kan vi beräkna ett vektorfält $\vec{F} = - \nabla \Phi$, som vi ofta kallar för en fältstyrka.

Vi kan då både konstruera nivåytorna till potentialen, vilka kallas ekvipotentialytor, och fältlinjerna till fältstyrkan.  Lägg märke till att ekvipotentialytorna och fältlinjerna överallt måste vara ortogonala mot varandra.

### Exempel:

Bestäm fältbilden till potentialen

$$
\begin{equation}
  \Phi\left(x,y\right) = \Phi_0 \frac{x^2 + y^2}{ax}.
\end{equation}
$$

Vi börjar med att bestämma ekvipotentialkurvorna genom att sätta $\Phi = 2C/\Phi_0$, så att vi får ekvationen

$$
\begin{equation}
  2C = \frac{x^2+y^2}{ax}.
\end{equation}
$$

Vi multiplicerar med $ax$ och sätter alla termerna på samma sida om likhetstecknet

$$
\begin{equation}
  0 = x^2 -2aCx +y^2.
\end{equation}
$$

Efter kvadratkomplettering får vi

$$
\begin{equation}
  0 = \left(x - aC\right)^2 -a^2 C^2 + y^2.
\end{equation}
$$

Denna ekvation kan vi skriva om som

$$
\begin{equation}
  a^2 C^2 = \left(x - aC\right)^2 + y^2,
\end{equation}
$$

vilket är ekvationen för en cirkel med centrum i $(aC,0)$ och radien $|aC|$.  Detta innebär att alla ekvipotentialytorna går genom origo.

Nu bestämmer vi fältstyrkan

$$
\begin{equation}
  \vec{F} = - \nabla \Phi = - \frac{\Phi_0}{a} \left(1 - \frac{y^2}{x^2},
\frac{2y}{x}\right).
\end{equation}
$$

Fältlinjerna är då lösningen till

4
4

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\begin{equation}
  \frac{\mbox{d}y}{\mbox{d}\tau} = -C \frac{\Phi_0}{a} \frac{2y}{x}.
\label{y_ekv} \tag{2}
\end{equation}
$$

Vi dividerar nu [(2)](#y_ekv) med [(1)](#x_ekv)

$$
\begin{equation}
\label{diffekv} \tag{3}
  \frac{\mbox{d}y}{\mbox{d}x} = \frac{2y/x}{1-y^2/x^2},
\end{equation}
$$

### Lösning 1

(Se avsnitt 7.2 i Arfken.)

Vi noterar att vi har den allmänna formen av en första ordningens differentialekvation

$$
\begin{equation}
\frac{\mbox{d}y}{\mbox{d}x} = - \frac{P(x,y)}{Q(x,y)}.
\end{equation}
$$

Hade vår ekvation varit separabel hade vi fått en väldigt snabb lösning. För en separabel diff.ekv. gäller att $P(x,y)=P(x)$ och $Q(x,y)=Q(y)$ så att

$$
\begin{equation}
Q(y) \mbox{d}y = -P(x) \mbox{d}x,
\end{equation}
$$

vilket vi enkelt kan integrera. Men tyvärr, i vårt fall kan vi inte skriva ekvationen på denna form. Istället undersöker vi om den är en exakt differential. I sådana fall existerar det en funktion $\varphi(x,y)$ så att

$$
\begin{equation}
\partial_x \varphi(x,y) = P(x,y) \quad \mathrm{och} \quad \partial_y \varphi(x,y) = Q(x,y).
\end{equation}
$$

Detta är i sin tur uppfyllt omm

$$
\begin{equation}
\partial_y P(x,y) = \partial_x Q(x,y).
\label{exaktdiff} \tag{4}
\end{equation}
$$

I vårt fall är detta inte omedelbart sant eftersom vi har $P(x,y) = - 2y/x$ och $Q(x,y) = 1-y^2/x^2$. Men strukturen ser ändå lovande ut; vi skulle dock önska att kvoterna var de omvända, dvs $x/y$. Vi multiplicerar täljare och nämnare med $-x^2/y^2$ vilket ger

$$
\begin{equation}
P(x,y) = \frac{2x}{y},
\end{equation}
$$

$$
\begin{equation}
Q(x,y) = 1 - \frac{x^2}{y^2}.
\end{equation}
$$

vilket uppfyller $\partial_y P(x,y) = \partial_x Q(x,y)$ (kolla gärna). En sådan funktion (i vårt fall $\alpha(x,y) = -x^2/y^2$) kallas för en integrerande faktor och går alltid att finna. Men tyvärr existerar det inget systematiskt sätt att konstruera denna (förutom för linjära, första ordningens ODE).

Under detta villkor har diff.ekvationen lösningen

$$
\begin{equation}
\int_{x_0}^x P(x',y) \mbox{d}x' + \int_{y_0}^y Q(x_0,y') \mbox{d}y' = \mathrm{konstant}.
\end{equation}
$$

Vi får att

$$
\begin{equation}
\mathrm{konstant} = \left[ \frac{x'^2}{y} \right]_{x_0}^x + \left[ y' + \frac{x_0^2}{y'} \right]_{y_0}^y.
\end{equation}
$$

Vi kan baka ihop alla konstanter och kalla denna för $2D$ vilket ger

$$
\begin{equation}
x^2 + y^2 - 2 D y = 0.
\end{equation}
$$

Efter kvadratkomplettering får vi

$$
\begin{equation}
  x^2 + \left(y-D\right)^2 = D^2,
\end{equation}
$$

vilket är ekvationen för en cirkel med centrum i $(0,D)$ och med radien $D$, det vill säga alla fältlinjerna går genom origo.

### Lösning 2

Diff.ekvationen ovan kan skrivas om som

$$
\begin{equation}
  \left(1 - \frac{y^2}{x^2}\right) \frac{\mbox{d}y}{\mbox{d}x} = \frac{2y}{x}.
\end{equation}
$$

Om vi flyttar om termerna så får vi

$$
\begin{equation}
  \left(1 - \frac{y^2}{x^2}\right) \frac{\mbox{d}y}{\mbox{d}x} - \frac{2y}{x} =
0.
\end{equation}
$$

Multiplicera med $-x^2/y^2$

$$
\begin{equation}
  \left(1 - \frac{x^2}{y^2}\right) \frac{\mbox{d}y}{\mbox{d}x} + \frac{2x}{y}
= 0.
\end{equation}
$$

Vi noterar nu att

$$
\begin{equation}
  \frac{\mbox{d}}{\mbox{d}x} \left(\frac{x^2}{y} + y\right) =
\frac{2x}{y} - \frac{x^2}{y^2} \frac{\mbox{d}y}{\mbox{d}x} +
\frac{\mbox{d}y}{\mbox{d}x} = \left(1 - \frac{x^2}{y^2}\right)
\frac{\mbox{d}y}{\mbox{d}x} + \frac{2x}{y},
\end{equation}
$$

det vill säga vår ekvation kan skrivas

$$
\begin{equation}
  \frac{\mbox{d}}{\mbox{d}x} \left(\frac{x^2}{y} + y\right) = 0.
\end{equation}
$$

Om vi integrerar detta uttryck får vi

$$
\begin{equation}
  \frac{x^2}{y} + y = 2D,
\end{equation}
$$

där $D$ är en integrationskonstant.  Vi multiplicerar med $y$

$$
\begin{equation}
  x^2 + y^2 = 2Dy.
\end{equation}
$$

Detta är samma lösning som ovan.


# Generating C Code for the Scalar Wave Equation in Cartesian Coordinates

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

## This module generates the C Code for the Scalarwave in Cartesian coordinates and sets up [Plane Wave Initial Data](https://en.wikipedia.org/wiki/Initial_value_problem).

### NRPy+ Source Code for this module:
* [ScalarWave/ScalarWave_RHSs.py](../edit/ScalarWave/ScalarWave_RHSs.py)
* [ScalarWave/InitialData_PlaneWave.py](../edit/ScalarWave/InitialData_PlaneWave.py)

## Introduction:
### Problem Statement

We wish to numerically solve the scalar wave equation as an [initial value problem](https://en.wikipedia.org/wiki/Initial_value_problem) in Cartesian coordinates:
$$\partial_t^2 u = c^2 \nabla^2 u \text{,}$$
where $u$ (the amplitude of the wave) is a function of time and space: $u = u(t,x,y,...)$ (spatial dimension as-yet unspecified) and $c$ is the wave speed, subject to some initial condition

$$u(0,x,y,...) = f(x,y,...)$$

and suitable spatial boundary conditions.

As described in the next section, we will find it quite useful to define
$$v(t,x,y,...) = \partial_t u(t,x,y,...).$$

In this way, the second-order PDE is reduced to a set of two coupled first-order PDEs

\begin{align}
\partial_t u &= v \\
\partial_t v &= c^2 \nabla^2 u.
\end{align}

We will use NRPy+ to generate efficient C codes capable of generating both initial data $u(0,x,y,...) = f(x,y,...)$; $v(0,x,y,...)=g(x,y,...)$, as well as finite-difference expressions for the right-hand sides of the above expressions. These expressions are needed within the *Method of Lines* to "integrate" the solution forward in time.

### The Method of Lines

Once we have initial data, we "evolve it forward in time", using the [Method of Lines](https://reference.wolfram.com/language/tutorial/NDSolveMethodOfLines.html). In short, the Method of Lines enables us to handle
1. the **spatial derivatives** of an initial value problem PDE using **standard finite difference approaches**, and
2. the **temporal derivatives** of an initial value problem PDE using **standard strategies for solving ordinary differential equations (ODEs)**, so long as the initial value problem PDE can be written in the form
$$\partial_t \vec{f} = \mathbf{M}\ \vec{f},$$
where $\mathbf{M}$ is an $N\times N$ matrix filled with differential operators that act on the $N$-element column vector $\vec{f}$. $\mathbf{M}$ may not contain $t$ or time derivatives explicitly; only *spatial* partial derivatives are allowed to appear inside $\mathbf{M}$. The scalar wave equation as written in the [previous module](Tutorial-ScalarWave.ipynb)
\begin{equation}
\partial_t
\begin{bmatrix}
u \\
v
\end{bmatrix}=
\begin{bmatrix}
0 & 1 \\
c^2 \nabla^2  & 0
\end{bmatrix}
\begin{bmatrix}
u \\
v
\end{bmatrix}
\end{equation}
satisfies this requirement.

Thus we can treat the spatial derivatives $\nabla^2 u$ of the scalar wave equation  using **standard finite-difference approaches**, and the temporal derivatives $\partial_t u$ and $\partial_t v$ using **standard approaches for solving ODEs**. In [the next module](Tutorial-Start_to_Finish-ScalarWave.ipynb), we will apply the highly robust [explicit Runge-Kutta fourth-order scheme](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods) (RK4), used widely for numerically solving ODEs, to "march" (integrate) the solution vector $\vec{f}$ forward in time from its initial value ("initial data").

### Basic Algorithm

The basic algorithm for solving the scalar wave equation [initial value problem](https://en.wikipedia.org/wiki/Initial_value_problem), based on the Method of Lines (see section above) is outlined below, with NRPy+-based components highlighted in green. We will review how NRPy+ generates these core components in this module.

1. Allocate memory for gridfunctions, including temporary storage for the RK4 time integration.
1. Set gridfunction values to initial data.
1. Evolve the system forward in time using RK4 time integration. At each RK4 substep, do the following:
    1. Evaluate scalar wave RHS expressions.
    1. Apply boundary conditions.

**We refer to the right-hand side of the equation $\partial_t \vec{f} = \mathbf{M}\ \vec{f}$ as the RHS. In this case, we refer to the $\mathbf{M}\ \vec{f}$ as the "scalar wave RHSs".** In the following sections we will

1. Use NRPy+ to cast the scalar wave RHS expressions -- in finite difference form -- into highly efficient C code,
    1. first in one spatial dimension with fourth-order finite differences,
    1. and then in three spatial dimensions with tenth-order finite differences.
1. Use NRPy+ to generate monochromatic plane-wave initial data for the scalar wave equation, where the wave propagates in an arbitrary direction.

As for the $\nabla^2 u$ term, spatial derivatives are handled in NRPy+ via [finite differencing](https://en.wikipedia.org/wiki/Finite_difference).

We will sample the solution $\{u,v\}$ at discrete, uniformly-sampled points in space and time. For simplicity, let's assume that we consider the wave equation in one spatial dimension. Then the solution at any sampled point in space and time is given by
$$u^n_i = u(t_n,x_i) = u(t_0 + n \Delta t, x_0 + i \Delta x),$$
where $\Delta t$ and $\Delta x$ represent the temporal and spatial resolution, respectively. $v^n_i$ is sampled at the same points in space and time.

# Table of Contents
$$\label{toc}$$

1. [Step 1](#initializenrpy): Initialize core NRPy+ modules
1. [Step 2](#step2): Scalar Wave RHSs in One Spatial Dimension, Fourth-Order Finite Differencing
1. [Step 3](#step3): Scalar Wave RHSs in Three Spatial Dimensions, Tenth-Order Finite Differencing
    1. [Step 3.a](#code_validation1): Code Validation against ScalarWave.ScalarWave_RHSs NRPy+ module
1. [Step 4](#step4): Plane-Wave Initial Data for the Scalar Wave Equation
    1. [Step 4.a](#code_validation2): Code Validation against ScalarWave.InitialData_PlaneWave NRPy+ module
1. [Step 5](#latex_pdf_output): Output this module to $\LaTeX$-formatted PDF file

# Step 1: Initialize core NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

Let's start by importing all the needed modules from NRPy+:

```python
# Step P1: Import needed NRPy+ core modules:
import NRPy_param_funcs as par
import indexedexp as ixp
import grid as gri
import finite_difference as fin
from outputC import *
```

# Step 2: Scalar Wave RHSs in One Spatial Dimension, Fourth-Order Finite Differencing \[Back to [top](#toc)\]
$$\label{step2}$$

To minimize complication, we will first restrict ourselves to solving the wave equation in one spatial dimension, so
$$\nabla^2 u = \partial_x^2 u.$$
Extension of this operator to higher spatial dimensions is straightforward, particularly when using NRPy+.

As was discussed in [the finite difference section of the tutorial](Tutorial-Finite_Difference_Derivatives.ipynb), NRPy+ approximates derivatives using [finite difference methods](),  the second-order derivative $\partial_x^2$ accurate to fourth-order in uniform grid spacing $\Delta x$ (from fitting the unique 4th-degree polynomial to 5 sample points of $u$) is given by
\begin{equation}
\left[\partial_x^2 u(t,x)\right]_j = \frac{1}{(\Delta x)^2}
\left(
-\frac{1}{12} \left(u_{j+2} + u_{j-2}\right)
+ \frac{4}{3}  \left(u_{j+1} + u_{j-1}\right)
- \frac{5}{2} u_j \right)
+ \mathcal{O}\left((\Delta x)^4\right).
\end{equation}

```python
# Step P2: Define the C parameter wavespeed. The `wavespeed`
#          variable is a proper SymPy variable, so it can be
#          used in below expressions. In the C code, it acts
#          just like a usual parameter, whose value is
#          specified in the parameter file.
thismodule = "ScalarWave"
wavespeed = par.Cparameters("REAL",thismodule,"wavespeed")

# Step 1: Set the spatial dimension parameter, and then read
#         the parameter as DIM.
par.set_parval_from_str("grid::DIM",1)
DIM = par.parval_from_str("grid::DIM")

# Step 2: Set the finite differencing order to 4.
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",4)

# Step 3: Register gridfunctions that are needed as input
#         to the scalar wave RHS expressions.
uu, vv = gri.register_gridfunctions("EVOL",["uu","vv"])

# Step 4: Declare the rank-2 indexed expression \partial_{ij} u,
#         which is symmetric about interchange of indices i and j
#         Derivative variables like these must have an underscore
#         in them, so the finite difference module can parse the
#         variable name properly.
uu_dDD = ixp.declarerank2("uu_dDD","sym01")

# Step 5: Define right-hand sides for the evolution.
uu_rhs = vv
vv_rhs = 0
for i in range(DIM):
    vv_rhs += wavespeed*wavespeed*uu_dDD[i][i]

vv_rhs = sp.simplify(vv_rhs)

# Step 6: Generate C code for scalarwave evolution equations,
#         print output to the screen (standard out, or stdout).
fin.FD_outputC("stdout",
               [lhrh(lhs=gri.gfaccess("rhs_gfs","uu"),rhs=uu_rhs),
                lhrh(lhs=gri.gfaccess("rhs_gfs","vv"),rhs=vv_rhs)])
```

    {
       /*
        * NRPy+ Finite Difference Code Generation, Step 1 of 2: Read from main memory and compute finite difference stencils:
        */
       /*
        *  Original SymPy expression:
        *  "const double uu_dDD00 = invdx0**2*(-5*uu/2 + 4*uu_i0m1/3 - uu_i0m2/12 + 4*uu_i0p1/3 - uu_i0p2/12)"
        */
       const double uu_i0m2 = in_gfs[IDX2(UUGF, i0-2)];
       const double uu_i0m1 = in_gfs[IDX2(UUGF, i0-1)];
       const double uu = in_gfs[IDX2(UUGF, i0)];
       const double uu_i0p1 = in_gfs[IDX2(UUGF, i0+1)];
       const double uu_i0p2 = in_gfs[IDX2(UUGF, i0+2)];
       const double vv = in_gfs[IDX2(VVGF, i0)];
       const double uu_dDD00 = pow(invdx0, 2)*(-5.0/2.0*uu + (4.0/3.0)*uu_i0m1 - 1.0/12.0*uu_i0m2 + (4.0/3.0)*uu_i0p1 - 1.0/12.0*uu_i0p2);
       /*
        * NRPy+ Finite Difference Code Generation, Step 2 of 2: Evaluate SymPy expressions and write to main memory:
        */
       /*
        *  Original SymPy expressions:
        *  "[rhs_gfs[IDX2(UUGF, i0)] = vv,
        *    rhs_gfs[IDX2(VVGF, i0)] = uu_dDD00*wavespeed**2]"
        */
       rhs_gfs[IDX2(UUGF, i0)] = vv;
       rhs_gfs[IDX2(VVGF, i0)] = uu_dDD00*pow(wavespeed, 2);
    }

**Success!** Notice that indeed NRPy+ was able to compute the spatial derivative operator,
\begin{equation}
\left[\partial_x^2 u(t,x)\right]_j \approx \frac{1}{(\Delta x)^2}
\left(
-\frac{1}{12} \left(u_{j+2} + u_{j-2}\right)
+ \frac{4}{3}  \left(u_{j+1} + u_{j-1}\right)
- \frac{5}{2} u_j \right),
\end{equation}
correctly (easier to read in the "Original SymPy expressions" comment block at the top of the C output. Note that $\texttt{invdx0}=1/\Delta x_0$, where $\Delta x_0$ is the (uniform) grid spacing in the zeroth, or $x_0$ direction.

# Step 3: Scalar Wave RHSs in Three Spatial Dimensions, Tenth-Order Finite Differencing \[Back to [top](#toc)\]
$$\label{step3}$$

Let's next repeat the same process, only this time at **10th** finite difference order, for the **3-spatial-dimension** scalar wave equation, with SIMD enabled:

```python
# Step 1: Define the C parameter wavespeed. The `wavespeed`
#         variable is a proper SymPy variable, so it can be
#         used in below expressions. In the C code, it acts
#         just like a usual parameter, whose value is
#         specified in the parameter file.
wavespeed = par.Cparameters("REAL",thismodule,"wavespeed")

# Step 2: Set the spatial dimension parameter
#         to *FOUR* this time, and then read
#         the parameter as DIM.
par.set_parval_from_str("grid::DIM",3)
DIM = par.parval_from_str("grid::DIM")

# Step 3: Set the finite differencing order to 10.
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",10)

# Step 4a: Reset gridfunctions registered in 1D case above,
#          to avoid NRPy+ throwing an error about double-
#          registering gridfunctions, which is not allowed.
gri.glb_gridfcs_list = []

# Step 4b: Register gridfunctions that are needed as input
#          to the scalar wave RHS expressions.
uu, vv = gri.register_gridfunctions("EVOL",["uu","vv"])

# Step 5: Declare the rank-2 indexed expression \partial_{ij} u,
#         which is symmetric about interchange of indices i and j
#         Derivative variables like these must have an underscore
#         in them, so the finite difference module can parse the
#         variable name properly.
uu_dDD = ixp.declarerank2("uu_dDD","sym01")

# Step 6: Define right-hand sides for the evolution.
uu_rhs = vv
vv_rhs = 0
for i in range(DIM):
    vv_rhs += wavespeed*wavespeed*uu_dDD[i][i]

# Step 7: Simplify the expression for c^2 \nabla^2 u (a.k.a., vv_rhs):
vv_rhs = sp.simplify(vv_rhs)

# Step 8: Generate C code for scalarwave evolution equations,
#         print output to the screen (standard out, or stdout).
fin.FD_outputC("stdout",
               [lhrh(lhs=gri.gfaccess("rhs_gfs","uu"),rhs=uu_rhs),
                lhrh(lhs=gri.gfaccess("rhs_gfs","vv"),rhs=vv_rhs)],params="SIMD_enable=True")
```

    initialize_param() minor warning: Did nothing; already initialized parameter ScalarWave::wavespeed
    {
       /*
        * NRPy+ Finite Difference Code Generation, Step 1 of 2: Read from main memory and compute finite difference stencils:
        */
       /*
        *  Original SymPy expressions:
        *  "[const REAL_SIMD_ARRAY uu_dDD00 = invdx0**2*(-5269*uu/1800 + 5*uu_i0m1_i1_i2/3 - 5*uu_i0m2_i1_i2/21 + 5*uu_i0m3_i1_i2/126 - 5*uu_i0m4_i1_i2/1008 + uu_i0m5_i1_i2/3150 + 5*uu_i0p1_i1_i2/3 - 5*uu_i0p2_i1_i2/21 + 5*uu_i0p3_i1_i2/126 - 5*uu_i0p4_i1_i2/1008 + uu_i0p5_i1_i2/3150),
        *    const REAL_SIMD_ARRAY uu_dDD11 = invdx1**2*(-5269*uu/1800 + 5*uu_i0_i1m1_i2/3 - 5*uu_i0_i1m2_i2/21 + 5*uu_i0_i1m3_i2/126 - 5*uu_i0_i1m4_i2/1008 + uu_i0_i1m5_i2/3150 + 5*uu_i0_i1p1_i2/3 - 5*uu_i0_i1p2_i2/21 + 5*uu_i0_i1p3_i2/126 - 5*uu_i0_i1p4_i2/1008 + uu_i0_i1p5_i2/3150),
        *    const REAL_SIMD_ARRAY uu_dDD22 = invdx2**2*(-5269*uu/1800 + 5*uu_i0_i1_i2m1/3 - 5*uu_i0_i1_i2m2/21 + 5*uu_i0_i1_i2m3/126 - 5*uu_i0_i1_i2m4/1008 + uu_i0_i1_i2m5/3150 + 5*uu_i0_i1_i2p1/3 - 5*uu_i0_i1_i2p2/21 + 5*uu_i0_i1_i2p3/126 - 5*uu_i0_i1_i2p4/1008 + uu_i0_i1_i2p5/3150)]"
        */
       const REAL_SIMD_ARRAY uu_i0_i1_i2m5 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2-5)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2m4 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2-4)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2m3 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2-3)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2m2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2-2)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2m1 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2-1)]);
       const REAL_SIMD_ARRAY uu_i0_i1m5_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1-5,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1m4_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1-4,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1m3_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1-3,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1m2_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1-2,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1m1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1-1,i2)]);
       const REAL_SIMD_ARRAY uu_i0m5_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0-5,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0m4_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0-4,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0m3_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0-3,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0m2_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0-2,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0m1_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0-1,i1,i2)]);
       const REAL_SIMD_ARRAY uu = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0p1_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0+1,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0p2_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0+2,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0p3_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0+3,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0p4_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0+4,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0p5_i1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0+5,i1,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1p1_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1+1,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1p2_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1+2,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1p3_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1+3,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1p4_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1+4,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1p5_i2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1+5,i2)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2p1 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2+1)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2p2 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2+2)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2p3 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2+3)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2p4 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2+4)]);
       const REAL_SIMD_ARRAY uu_i0_i1_i2p5 = ReadSIMD(&in_gfs[IDX4(UUGF, i0,i1,i2+5)]);
       const REAL_SIMD_ARRAY vv = ReadSIMD(&in_gfs[IDX4(VVGF, i0,i1,i2)]);
       const double tmpFD_Rational_1_3150 = 0.0003174603174603174603174603174603175;
       const REAL_SIMD_ARRAY _Rational_1_3150 = ConstSIMD(tmpFD_Rational_1_3150);

       const double tmpFD_Rational_5_126 = 0.03968253968253968253968253968253968;
       const REAL_SIMD_ARRAY _Rational_5_126 = ConstSIMD(tmpFD_Rational_5_126);

       const double tmpFD_Rational_5_3 = 1.666666666666666666666666666666667;
       const REAL_SIMD_ARRAY _Rational_5_3 = ConstSIMD(tmpFD_Rational_5_3);

       const double tmpFD_Rational_m5269_1800 = -2.927222222222222222222222222222222;
       const REAL_SIMD_ARRAY _Rational_m5269_1800 = ConstSIMD(tmpFD_Rational_m5269_1800);

       const double tmpFD_Rational_m5_1008 = -0.004960317460317460317460317460317460;
       const REAL_SIMD_ARRAY _Rational_m5_1008 = ConstSIMD(tmpFD_Rational_m5_1008);

       const double tmpFD_Rational_m5_21 = -0.2380952380952380952380952380952381;
       const REAL_SIMD_ARRAY _Rational_m5_21 = ConstSIMD(tmpFD_Rational_m5_21);

       const REAL_SIMD_ARRAY tmpFD0 = MulSIMD(uu, _Rational_m5269_1800);
       const REAL_SIMD_ARRAY uu_dDD00 = MulSIMD(FusedMulAddSIMD(uu_i0p4_i1_i2, _Rational_m5_1008, FusedMulAddSIMD(uu_i0p3_i1_i2, _Rational_5_126, FusedMulAddSIMD(uu_i0p2_i1_i2, _Rational_m5_21, FusedMulAddSIMD(uu_i0p1_i1_i2, _Rational_5_3, FusedMulAddSIMD(uu_i0m5_i1_i2, _Rational_1_3150, FusedMulAddSIMD(uu_i0m4_i1_i2, _Rational_m5_1008, FusedMulAddSIMD(uu_i0m3_i1_i2, _Rational_5_126, FusedMulAddSIMD(uu_i0m2_i1_i2, _Rational_m5_21, FusedMulAddSIMD(uu_i0m1_i1_i2, _Rational_5_3, FusedMulAddSIMD(uu_i0p5_i1_i2, _Rational_1_3150, tmpFD0)))))))))), MulSIMD(invdx0, invdx0));
       const REAL_SIMD_ARRAY uu_dDD11 = MulSIMD(FusedMulAddSIMD(uu_i0_i1p4_i2, _Rational_m5_1008, FusedMulAddSIMD(uu_i0_i1p3_i2, _Rational_5_126, FusedMulAddSIMD(uu_i0_i1p2_i2, _Rational_m5_21, FusedMulAddSIMD(uu_i0_i1p1_i2, _Rational_5_3, FusedMulAddSIMD(uu_i0_i1m5_i2, _Rational_1_3150, FusedMulAddSIMD(uu_i0_i1m4_i2, _Rational_m5_1008, FusedMulAddSIMD(uu_i0_i1m3_i2, _Rational_5_126, FusedMulAddSIMD(uu_i0_i1m2_i2, _Rational_m5_21, FusedMulAddSIMD(uu_i0_i1m1_i2, _Rational_5_3, FusedMulAddSIMD(uu_i0_i1p5_i2, _Rational_1_3150, tmpFD0)))))))))), MulSIMD(invdx1, invdx1));
       const REAL_SIMD_ARRAY uu_dDD22 = MulSIMD(FusedMulAddSIMD(uu_i0_i1_i2p4, _Rational_m5_1008, FusedMulAddSIMD(uu_i0_i1_i2p3, _Rational_5_126, FusedMulAddSIMD(uu_i0_i1_i2p2, _Rational_m5_21, FusedMulAddSIMD(uu_i0_i1_i2p1, _Rational_5_3, FusedMulAddSIMD(uu_i0_i1_i2m5, _Rational_1_3150, FusedMulAddSIMD(uu_i0_i1_i2m4, _Rational_m5_1008, FusedMulAddSIMD(uu_i0_i1_i2m3, _Rational_5_126, FusedMulAddSIMD(uu_i0_i1_i2m2, _Rational_m5_21, FusedMulAddSIMD(uu_i0_i1_i2m1, _Rational_5_3, FusedMulAddSIMD(uu_i0_i1_i2p5, _Rational_1_3150, tmpFD0)))))))))), MulSIMD(invdx2, invdx2));
       /*
        * NRPy+ Finite Difference Code Generation, Step 2 of 2: Evaluate SymPy expressions and write to main memory:
        */
       /*
        *  Original SymPy expressions:
        *  "[const REAL_SIMD_ARRAY __RHS_exp_0 = vv,
        *    const REAL_SIMD_ARRAY __RHS_exp_1 = wavespeed**2*(uu_dDD00 + uu_dDD11 + uu_dDD22)]"
        */
       const REAL_SIMD_ARRAY __RHS_exp_0 = vv;
       const REAL_SIMD_ARRAY __RHS_exp_1 = MulSIMD(AddSIMD(uu_dDD11, AddSIMD(uu_dDD00, uu_dDD22)), MulSIMD(wavespeed, wavespeed));
       WriteSIMD(&rhs_gfs[IDX4(UUGF, i0, i1, i2)], __RHS_exp_0);
       WriteSIMD(&rhs_gfs[IDX4(VVGF, i0, i1, i2)], __RHS_exp_1);
    }

## Step 3.a:  Code Validation against ScalarWave.ScalarWave_RHSs NRPy+ module \[Back to [top](#toc)\]
$$\label{code_validation1}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the RHSs of the three-spatial-dimension Scalar Wave equation (i.e., uu_rhs and vv_rhs) between

1. this tutorial and
2. the [NRPy+ ScalarWave.ScalarWave_RHSs](../edit/ScalarWave/ScalarWave_RHSs.py) module.

```python
# Step 10: We already have SymPy expressions for uu_rhs and vv_rhs in
#          terms of other SymPy variables. Even if we reset the list
#          of NRPy+ gridfunctions, these *SymPy* expressions for
#          uu_rhs and vv_rhs *will remain unaffected*.
#
#          Here, we will use the above-defined uu_rhs and vv_rhs to
#          validate against the same expressions in the
#          ScalarWave/ScalarWave_RHSs.py module,
#          to ensure consistency between this tutorial
#          (historically speaking, the tutorial was written first)
#          and the ScalarWave_RHSs.py module itself.
#
# Reset the list of gridfunctions, as registering a gridfunction
#   twice will spawn an error.
gri.glb_gridfcs_list = []

# Step 11: Call the ScalarWave_RHSs() function from within the
#         ScalarWave/ScalarWave_RHSs.py module,
#         which should do exactly the same as in Steps 1-10 above.
import ScalarWave.ScalarWave_RHSs as swrhs
swrhs.ScalarWave_RHSs()

# Step 12: Consistency check between the tutorial module above
#         and the ScalarWave_RHSs() function from within the
#         ScalarWave/ScalarWave_RHSs.py module.
# It is SAFE to ignore the warning from re-initializing the parameter RMAX.
print("^^^ Ignore the minor warning above. ^^^\n")
print("Consistency check between ScalarWave tutorial and NRPy+ module:")
print("uu_rhs - swrhs.uu_rhs: Should be zero: ",sp.simplify(uu_rhs - swrhs.uu_rhs))
print("vv_rhs - swrhs.vv_rhs: Should be zero: ",sp.simplify(vv_rhs - swrhs.vv_rhs))
```

    ^^^ Ignore the minor warning above. ^^^

    Consistency check between ScalarWave tutorial and NRPy+ module:
    uu_rhs - swrhs.uu_rhs: Should be zero:  0
    vv_rhs - swrhs.vv_rhs: Should be zero:  0

# Step 4: Plane-Wave Initial Data for the Scalar Wave Equation \[Back to [top](#toc)\]
$$\label{step4}$$

The solution to the scalar wave equation for a monochromatic (single-wavelength) wave traveling in the $\hat{k}$ direction is
$$u(\vec{x},t) = f(\hat{k}\cdot\vec{x} - c t),$$
where $\hat{k}$ is a unit vector. We choose $f(\hat{k}\cdot\vec{x} - c t)$ to take the form
$$
f(\hat{k}\cdot\vec{x} - c t) = \sin\left(\hat{k}\cdot\vec{x} - c t\right) + 2,
$$
where we add the $+2$ to ensure that the exact solution never crosses through zero. In places where the exact solution passes through zero, the relative error (i.e., the measure of error to compare numerical with exact results) is undefined. Also, $f(\hat{k}\cdot\vec{x} - c t)$ plus a constant is still a solution to the wave equation.

```python
# Step 1: Set parameters defined in other modules
xx = gri.xx

# Step 2: Declare free parameters intrinsic to these initial data
time = par.Cparameters("REAL",thismodule,"time")
kk   = par.Cparameters("REAL",thismodule,["kk0","kk1","kk2"])

# Step 3: Normalize the k vector
kk_norm = sp.sqrt(kk[0]**2 + kk[1]**2 + kk[2]**2)

# Step 4: Compute k.x
dot_product = sp.sympify(0)
for i in range(DIM):
    dot_product += xx[i]*kk[i]
dot_product /= kk_norm

# Step 5: Set initial data for uu and vv, where vv_ID = \partial_t uu_ID.
uu_ID = sp.sin(dot_product - wavespeed*time)+2
vv_ID = sp.diff(uu_ID, time)
```

Next we verify that $f(\hat{k}\cdot\vec{x} - c t)$ satisfies the wave equation, by computing
$$\left(c^2 \nabla^2 - \partial_t^2 \right)\ f\left(\hat{k}\cdot\vec{x} - c t\right),$$
and confirming the result is exactly zero.

```python
sp.simplify(wavespeed**2*(sp.diff(uu_ID,xx[0],2) +
                          sp.diff(uu_ID,xx[1],2) +
                          sp.diff(uu_ID,xx[2],2))
            - sp.diff(uu_ID,time,2))
```

$\displaystyle 0$

## Step 4.a: Code Validation against ScalarWave.InitialData_PlaneWave NRPy+ module  \[Back to [top](#toc)\]
$$\label{code_validation2}$$

As a code validation check, we will verify agreement in the SymPy expressions for plane-wave initial data for the Scalar Wave equation between
1. this tutorial and
2. the NRPy+ [ScalarWave.InitialData_PlaneWave](../edit/ScalarWave/InitialData_PlaneWave.py) module.

```python
# We just defined SymPy expressions for uu_ID and vv_ID in
# terms of other SymPy variables. Here, we will use the
# above-defined uu_ID and vv_ID to validate against the
# same expressions in the ScalarWave/InitialData_PlaneWave.py
# module, to ensure consistency between this tutorial
# (historically speaking, the tutorial was written first)
# and the PlaneWave ID module itself.
#
# Step 6: Call the InitialData_PlaneWave() function from within the
#         ScalarWave/InitialData_PlaneWave.py module,
#         which should do exactly the same as in Steps 1-5 above.
import ScalarWave.InitialData_PlaneWave as swid
swid.InitialData_PlaneWave()

# Step 7: Consistency check between the tutorial module above
#         and the InitialData_PlaneWave() function from within the
#         ScalarWave/InitialData_PlaneWave.py module.
# It is SAFE to ignore the warning from re-initializing the parameter RMAX.
print("Consistency check between ScalarWave tutorial and NRPy+ module:")
print("uu_ID - swid.uu_ID: Should be zero: ",sp.simplify(uu_ID - swid.uu_ID))
print("vv_ID - swid.vv_ID: Should be zero: ",sp.simplify(vv_ID - swid.vv_ID))
```

    Consistency check between ScalarWave tutorial and NRPy+ module:
    uu_ID - swid.uu_ID: Should be zero:  0
    vv_ID - swid.vv_ID: Should be zero:  0

# Step 5: Output this module to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-ScalarWave.pdf](Tutorial-ScalarWave.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx Tutorial-ScalarWave.ipynb
!pdflatex -interaction=batchmode Tutorial-ScalarWave.tex
!pdflatex -interaction=batchmode Tutorial-ScalarWave.tex
!pdflatex -interaction=batchmode Tutorial-ScalarWave.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-ScalarWave.ipynb to latex
    [NbConvertApp] Writing 63651 bytes to Tutorial-ScalarWave.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


# Tutorial: Torque magnetometry

In this tutorial, the Ubermag micromagnetic simulation software is used to create 3-dimensional magnetic structure, on which torque magnetometry can be performed. Torque magnetometry is a technique where an external magnetic field is applied in a direction relative to a magnetic sample, an the torque produced measured. This technique can be used to obtain information on the characteristics of the magnetism in the sample like anisotropy.

Torque arises from the cross product of the magnetic moment ${\bf m}$, and the magnetic flux density ${\bf B}$,
\begin{equation}
{\bf \tau} = {\bf m} \times {\bf B}.
\end{equation}
This equation can be rewritten as
\begin{equation}
{\bf \tau} = \mu_0 {\bf m} \times \left( {\bf H}_{\rm app} + {\bf H}_{\rm demag} \right),
\end{equation}
where ${\bf H}_{app}$ is the applied magnetic field, and ${\bf H}_{demag}$ is the demagnetisation field.

The magnetic moment can be obtain from the magnetisation using
\begin{align}
{\bf M} &= \frac{d{\bf m}}{dV}, \\
{\bf m} &= \int {\bf M} dV,\\
{\bf m} &=  {\bf M} V,
\end{align}
where ${\bf M}$ is magnetisation, ${\bf m}$ is the magnetic moment, and $V$ is the volume.

## Magnetometry Reference frame
In `mag2exp` the experimental reference for magnetometry is defined to be congruent as the sample reference frame.

## The micromagnetic simulation
A micromagnetic simulation can be set up using Ubermag to obtain a 3-dimensional magnetic structure.

```python
# NBVAL_IGNORE_OUTPUT
import matplotlib.pyplot as plt
import oommfc as oc
import discretisedfield as df
import micromagneticmodel as mm
import numpy as np
import ubermagutil.units as uu
np.random.seed(1)

region = df.Region(p1=(-150e-9, -150e-9, 0), p2=(150e-9, 150e-9, 20e-9))
mesh = df.Mesh(region=region, cell=(5e-9, 5e-9, 5e-9))

Ms = 9.65e5
H_mag = 2.38e6

system = mm.System(name='Box2')

system.energy = (mm.Exchange(A=8.78e-13)
                 + mm.UniaxialAnisotropy(K=1.18e6, u=(0, 0, 1))
                 + mm.Demag()
                 + mm.Zeeman(H=(0, 0, 0)))

# create system with above geometry and initial magnetisation
system.m = df.Field(mesh, dim=3, value=(0, 0, 1), norm=Ms)
system.m.z.plane('z').mpl.scalar()
```

Relax the system and plot its magnetisation.

```python
# NBVAL_IGNORE_OUTPUT
# minimize the energy
md = oc.MinDriver()
md.drive(system)

# Plot relaxed configuration: vectors in z-plane
system.m.plane('z').mpl()
```

## Computing Torque Magnetometry
The torque can be calculated using
\begin{equation}
{\bf \tau} = \mu_0 {\bf m} \times \left( {\bf H}_{\rm app} + {\bf H}_{\rm demag} \right).
\end{equation}
This method is included as part of the `mag2exp.magnetometry` module.

```python
# NBVAL_IGNORE_OUTPUT
import mag2exp
mag2exp.magnetometry.torque(system)
```

    Running OOMMF (DockerOOMMFRunner) [2021/09/10 17:30]... (2.6 s)

    (-1.7763568394002505e-14, -4.565237077258644e-13, 3.4938953516920993e-14)

If the demagnetisation term is not required for any reason, then the use_demag=False argument can be passed.

```python
mag2exp.magnetometry.torque(system, use_demag=False)
```

    (0.0, 0.0, 0.0)

## Recreating experiments
A common experiment with torque magnetometry is to rotate the magnetic field relative to the sample and measure the torque. Doing this at high applied magnetic fields can reveal the magnetic anisotropy of the sample.

Here we rotate the applied magnetic field a full $2\pi$ perpendicular to the $x$ axis, with our initial field in the $z$ direction. The sample has uniaxial anisotropy in the $z$ direction, hence the torque forms a periodic pattern.

```python
%%capture
angles = np.linspace(0, 2*np.pi, 71)

torque_val_x = np.full_like(angles, np.nan)
torque_val_y = np.full_like(angles, np.nan)
torque_val_z = np.full_like(angles, np.nan)
for i, angle in enumerate(angles):
    system.energy.zeeman.H= (0,H_mag*np.sin(angle),H_mag*np.cos(angle))
    md.drive(system)
    torque_val_x[i], torque_val_y[i], torque_val_z[i] = mag2exp.magnetometry.torque(system)
```

```python
# NBVAL_IGNORE_OUTPUT
fig, ax = plt.subplots(dpi=100)
ax.scatter(angles, torque_val_x)
ax.set_xlabel("Angle (rad.)")
ax.set_ylabel(r"Torque (Nm$^{-2}$)")
plt.show()
```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Stretch Processor
***

Referring to Section 8.4, stretch processing is an intrapulse compression technique used for processing waveforms of very large bandwidth.  It was originally developed to reduce the sampling requirements for large bandwidth, high range resolution systems such as synthetic aperture radar (SAR) systems.  While stretch processing may be applied to various forms of wide bandwidth waveforms, this discussion is focused on linear frequency modulation (LFM) waveforms.  A block diagram of a stretch processor is given in Figure 8.7, and in general, stretch processing consists of the following basic steps:

1. Mix the return signal with a reference signal.
2. Low-pass filter the mixed signals.
3. Analog-to-digital conversion of the low-pass filtered signals.
4. Fourier transform of the digital signals.

The first step in stretch processing is to mix the return signal with a reference signal.  The reference signal is typically generated with the same frequency slope as the transmitted signal.  The transmitted LFM signal may be written as (Equation 8.36)

\begin{equation}
    s(t) = \exp\left[j\, 2 \pi \left(f_0 t + \frac{\beta}{2} t ^ 2\right)\right] \hspace{0.5in} 0\le t \le \tau_p,
\end{equation}

The return signal from a point scatterer is written as (Equation 8.37)

\begin{equation}
    s_r(t) = A\, \exp\left[j\, 2 \pi \left(f_0(t - t_0) + \frac{\beta}{2}(t - t_0) ^ 2\right)\right],
\end{equation}

The factor, $A$, is a constant related to the target's radar cross section, antenna gain, path loss, and other factors.  The reference signal is written as (Equation 8.38)

\begin{equation}
    s_{ref}(t) = \exp\left[ j\, 2 \pi \left( f_0(t - t_{ref}) + \frac{\beta}{2}(t - t_{ref}) ^ 2 \right) \right],
\end{equation}

where $t_{ref}$ is the reference time of the signal, which must closely match $t_0$. This leads to the following requirement on the pulsewidth of the reference signal (Equation 8.39)

\begin{equation}
    \tau_{ref} \ge t_r + \tau_p \hspace{0.5in} \text{(s)},
\end{equation}

where $t_r$ is the range of time over which stretch processing is to be employed.  This is expressed as $t_r = t_{max} - t_{min}$.  The second requirement is $ t_{ref} - t_{min} \le t_0 \le t_{max} + t_{ref}$. Now that the requirements on the reference signal have been specified, the output of the mixing step is written as (Equation 8.40)

\begin{equation}\label{eq:sp_output_mixed}
    s_o(t) = A\, \exp\Big[ j\, 2 \pi f_0 (t_0 - t_{ref}) + j\, 2 \pi \beta (t_0 - t_{ref})t + j\, \pi \beta (t_{ref}^2 - t_0^2) \Big].
\end{equation}

The instantaneous frequency is the time derivative of the phase, which is written as (Equation 8.41)

\begin{equation}
    f_i = \frac{1}{2 \pi}\frac{d\phi(t)}{dt} \hspace{0.5in} \text{(Hz)},
\end{equation}

where $\phi(t)$ is the phase of the signal which is simply (Equation 8.42)

\begin{equation}
    \phi(t) = 2 \pi f_0 (t_0 - t_{ref}) + 2 \pi \beta (t_0 - t_{ref})t + \pi \beta \left(t_{ref}^2 - t_0^2\right) \hspace{0.25in} \text{(rad)}.
\end{equation}

The instantaneous frequency is then (Equation 8.43)

\begin{equation}\label{eq:inst_freq_sp}
    f_i = \beta(t_0 - t_{ref}) \hspace{0.5in} \text{(Hz)}.
\end{equation}

This allows the time delay of the target to be written as (Equation 8.44)

\begin{equation}
    t_0 = \frac{f_i}{\beta} + t_{ref} \hspace{0.5in} \text{(s)}.
\end{equation}

Therefore, the target range is related to the instantaneous frequency as (Equation 8.45)
\begin{equation}
    r_0 = \frac{c}{2}\bigg(\frac{f_i}{B}\tau_p + t_{ref}\bigg) \hspace{0.5in} \text{(m)}.
\end{equation}

The expression in (Equation 8.43) indicates the resulting instantaneous frequency is constant and related to the range to the target. Performing analog-to-digital sampling on the signal output from the mixer only requires sampling at the maximum instantaneous frequency in (Equation 8.43), rather than the full bandwidth.
***

Set the bandwidth (Hz), the pulsewidth (s), the length of the range window (m), the target range (m) and the target radar cross section (m^2)

```python
bandwidth = 1e9

pulsewidth = 10e-4

range_window_length = 50.0

target_range = [10.0, 20.0, 30.0]

target_rcs = [1.0, 10.0, 3.0]
```

Calculate the number of samples for stretch processing

```python
from numpy import ceil

from scipy.constants import c

number_of_samples = int(ceil(4 * bandwidth * range_window_length / c))
```

Set the window type

```python
window_type = 'Rectangular'
```

Get the window coefficients using `scipy.signal.windows`

```python
from scipy.signal.windows import hann, hamming, blackmanharris, kaiser

from numpy import ones

if window_type == 'Kaiser':

    coefficients = kaiser(number_of_samples, 6, True)

elif window_type == 'Blackman-Harris':

    coefficients = blackmanharris(number_of_samples, True)

elif window_type == 'Hanning':

    coefficients = hann(number_of_samples, True)

elif window_type == 'Hamming':

    coefficients = hamming(number_of_samples, True)

elif window_type == 'Rectangular':

    coefficients = ones(number_of_samples)
```

Get the time sampling using the `linspace` routine from `scipy`

```python
from numpy import linspace

t, dt = linspace(-0.5 * pulsewidth, 0.5 * pulsewidth, number_of_samples, retstep=True)
```

Calculate the sampled signal after the mixing process

```python
from numpy import zeros, sqrt, exp

from scipy.constants import pi

so = zeros(number_of_samples, dtype=complex)

for r, rcs in zip(target_range, target_rcs):

    so += sqrt(rcs) * exp(1j * 2.0 * pi * bandwidth / pulsewidth * (2 * r / c) * t)
```

Perform the FFT on the sampled signal using the routines from `fftpack`

```python
from scipy.fftpack import fftfreq, fft, fftshift

# Fourier transform

so = fftshift(fft(so * coefficients, 4 * number_of_samples))

# FFT frequencies

frequencies = fftshift(fftfreq(4 * number_of_samples, dt))
```

Calculate the range window (m) from the frequencies (Hz)

```python
# Range window

range_window = 0.5 * frequencies * c * pulsewidth / bandwidth
```

Display the results of stretch processing using the `matplotlib` routines

```python
from matplotlib import pyplot as plt

from numpy import log10, finfo

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Create the line plot

plt.plot(range_window, 20.0 * log10(abs(so) / number_of_samples + finfo(float).eps), '')

plt.xlim(min(target_range) - 5, max(target_range) + 5)

plt.ylim(-60, max(20.0 * log10(abs(so) / number_of_samples)) + 10)

# Set the x and y axis labels

plt.xlabel("Range (m)", size=12)

plt.ylabel("Amplitude (dBsm)", size=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the plot title and labels

plt.title('Stretch Processor Range Profile', size=14)

# Set the tick label size

plt.tick_params(labelsize=12)
```

```python

```