# Data Science Basics in Python Series

## Chapter VI: Basic Statistical Analysis in Python

### Michael Pyrcz, Associate Professor, The University of Texas at Austin

*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

####  Basic Univariate Statistics

Here's a demonstration of the calculation and visualization for basic statistical analysis in Python.

We will use the following Python packages:

* [statistics](https://docs.python.org/3/library/statistics.html)

* [SciPy](https://www.scipy.org/)

* [MatPlotLib](https://matplotlib.org/)

We will cover a variety of common, basic statistical analyses and displays.

####  Basic Univariate Statistics

This tutorial includes the methods and operations that would commonly be required for Engineers and Geoscientists working with Regularly Gridded Data Structures for the purpose of:

1. Data Checking and Cleaning
2. Data Mining / Inferential Data Analysis
3. Predictive Modeling

for Data Analytics, Geostatistics and Machine Learning.

#### General Definitions

**Statistics**

collecting, organizing, and interpreting data, as well as drawing conclusions and making decisions.

**Geostatistics**

a branch of applied statistics that integrates:

1. the spatial (geological) context,

2. the spatial relationships,

3. volumetric support / scale, and

4.  uncertainty.

**Data Analytics**

use of statistics [with visualization] to support decision making.

**Big Data Analytics**

process of examining large and varied data sets (big data) to discover patterns and make decisions.

#### General Definitions

**Variable** or **Feature**

* any property measured / observed in a study (e.g., porosity, permeability, mineral concentrations, saturations, contaminant concentration)
* the measure often requires significant analysis, interpretation and uncertainty, 'data softness'

#### General Definitions

**Population**

exhaustive, finite list of property of interest over area of interest. Generally the entire population is not accessible.

* exhaustive set of porosity at each location within a reservoir

**Sample**

set of values, locations that have been measured

* porosity data from well-logs within a reservoir

#### General Definitions

**Parameters**

summary measure of a population

* population mean, population standard deviation, we rarely have access to this

* model parameters is different and we will cover later.

**Statistics**

summary measure of a sample

* sample mean, sample standard deviation, we use statistics as estimates of the parameters

#### Covered Parameters / Statistics

We cover the following parameters and statistics.

| Central Tendency | Dispersion | Outliers | Distributions Shape |
| :--------------: | :--------: | :------: | :-----------------: |
| Arithmetic Average / Mean | Variance | Tukey Outlier Test| Skew            |
| Median            | Standard Deviation |                 | Excess Kurtosis |
| Mode              | Range |                              | Person's Mode Skewness |
| Geometric Mean    | Percentile |                         | Quartile Skew Coefficient |
| Harmonic Mean     | Interquartile Range |                |                           |
| Power Law Average |                     |                |                           |

I have a lecture on these univariate statistics available on [YouTube](https://www.youtube.com/watch?v=wAcbA2cIqec&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=11&t=0s).

#### Nonparmetric Cumulative Distribution Functions (CDFs)

**nonparametric CDF**
* plotting nonparametric distributions

**fitting CDFs**
* fitting a parametric distribution and plotting

#### Getting Started

Here's the steps to get setup to run this demonstration:

1. **Install Anaconda 3** on your machine from https://www.anaconda.com/download/.
2. **Open Jupyter Notebook**, look for the Jupyter app on your system after installing Anaconda 3.
3. **Load this Workflow** found here [PythonDataBasics_Statistics.ipynb](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/PythonDataBasics_PedictiveMachineLearning.ipynb).
4. **Load the data**, this workflow retreives the data from my GitHub [GeoDataSets Repository](https://github.com/GeostatsGuy/GeoDataSets). If you want to work locally, you will need to first download the data file to your working directory. The data file is found here, [2D_MV_200wells.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/2D_MV_200wells.csv). Code is provided below to set the working directory and to load the data locally.

#### Load the required libraries

The following code loads the required libraries.

```python
import numpy as np                              # ndarrys for gridded data
import pandas as pd                             # DataFrames for tabular data
import os                                       # set working directory, run executables
import matplotlib.pyplot as plt                 # plotting
import scipy                                    # statistics
import statistics as stats                      # statistics like the mode
from scipy.stats import norm                    # fitting a Gaussian distribution
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time). Set this to your working directory, with the above mentioned data file.

```python
os.chdir("c:/PGE383")                           # set the working directory
```

#### Loading Data

Let's load the provided multivariate, spatial dataset [2D_MV_200wells.csv](https://github.com/GeostatsGuy/GeoDataSets). These are the features:

| Feature  | Units | Descriptions |
| :------: | :--------------: | :--------- |
| X, Y  | $meters$     | Location |
| porosity | $fraction$   | rock void fraction |
| permeability | $mDarcy$ | capability of a porous rock to permit the flow of fluids through its pore spaces |
| acoustic impedance | $\frac{kg}{m^2s} 10^6$ | rock bulk density times rock acoustic velocity |

* load use the Pandas 'read_csv' function and rename the features for readable code

```python
df = pd.read_csv("2D_MV_200wells.csv")          # read a .csv file in as a DataFrame
df = df.rename(columns={'facies_threshold_0.3': 'facies','permeability':'perm','acoustic_impedance':'ai'}) # rename columns of the
df.head()
```

      X
      Y
      facies
      porosity
      perm
      ai

      0
      565
      1485
      1
      0.1184
      6.170
      2.009

      1
      2585
      1185
      1
      0.1566
      6.275
      2.864

      2
      2065
      2865
      2
      0.1920
      92.297
      3.524

      3
      3575
      2655
      1
      0.1621
      9.048
      2.157

      4
      1835
      35
      1
      0.1766
      7.123
      3.979

#### Extract a Feature

Let's extract one of the features, porosity, into a 1D ndarray and do our statistics on porosity.

* then we can use NumPy's statistics methods

* note this is a **shallow copy**, any changes to the array will change the feature in the DataFrame

```python
feature = 'Porosity'; fcol = 'porosity'; funits = '(fraction)'; f2units = '(fraction^2)'; fmin = 0.0; fmax = 0.25
X = df[fcol].values
print('We are working with ' + feature + ' ' + funits + ' from column ' + fcol +  ' .')
```

    We are working with Porosity (fraction) from column porosity .

#### Visualize the Feature Histogram

To improve our understanding of the feature, let's visualize the featre distribuiton as a histogram.

```python
plt.hist(X,color='red',alpha=0.2,edgecolor='black',bins=np.linspace(fmin,fmax,20))
plt.xlabel(feature + ' ' + funits); plt.ylabel('Frequency');
```

#### Measures of Central Tendency

##### The Arithmetic Average / Mean

\begin{equation}
\overline{x} = \frac{1}{n}\sum^n_{i=1} x_i
\end{equation}

```python
average = np.average(X)
print(feature + ' average is ' + str(round(average,2)) + ' ' + funits + '.')
```

    Porosity average is 0.15 (fraction).

#### Measures of Central Tendency

##### The Weighted Arithmetic Average / Mean

Many of the following methods accept data weights, e.g. declustering

\begin{equation}
\overline{x} = \frac{\sum^n_{i=1} \lambda_i x_i}{\sum^n_{i=1} \lambda_i}
\end{equation}

```python
weights = np.ones(X.shape)
wt_average = np.average(X,weights = weights)
print(feature + ' average is ' + str(round(wt_average,2)) + ' ' + funits + '.')
```

    Porosity average is 0.15 (fraction).

#### Measures of Central Tendency

##### Median

\begin{equation}
P50_x = F^{-1}_{x}(0.50)
\end{equation}

```python
median = np.median(X)
print(feature + ' median is ' + str(round(median,2)) + ' ' + funits + '.')
```

    Porosity median is 0.15 (fraction).

#### Measures of Central Tendency

##### Mode

The most common value. To do this we should bin the data, like into histogram bins/bars.  To do this we will round the data to the 2nd decimal place.  We are assume bin boundaries, $0.01, 0.02,\ldots, 0.30$.

```python
mode = stats.mode(np.round(X,2))
print(feature + ' mode is ' + str(round(mode,2)) + ' ' + funits + '.')
```

    Porosity mode is 0.14 (fraction).

#### Measures of Central Tendency

##### Geometric Mean

\begin{equation}
\overline{x}_G = ( \prod^n_{i=1} x_i )^{\frac{1}{n}}
\end{equation}

```python
geometric = scipy.stats.mstats.gmean(X)
print(feature + ' geometric mean is ' + str(round(geometric,2)) + ' ' + funits + '.')
```

    Porosity geometric mean is 0.15 (fraction).

#### Measures of Central Tendency

##### Harmonic Mean

\begin{equation}
\overline{x}_H = \frac{n}{\sum^n_{i=1} \frac{1}{x_i}}
\end{equation}

```python
hmean = scipy.stats.mstats.hmean(X)
print(feature + ' harmonic mean is ' + str(round(hmean,2)) + ' ' + funits + '.')
```

    Porosity harmonic mean is 0.14 (fraction).

##### Power Law Average

\begin{equation}
\overline{x}_p = (\frac{1}{n}\sum^n_{i=1}{x_i^{p}})^\frac{1}{p}
\end{equation}

```python
power = -0.5
power_avg = np.average(np.power(X,power))**(1/power)
print(feature + ' power law average for p = ' + str(power) + ' is ' + str(round(power_avg,2)) + ' ' + funits + '.')
```

    Porosity power law average for p = -0.5 is 0.14 (fraction).

#### Let's Visualize Some of the Measures of Central Tendency

To visualize ocmpare these statistics, parameters

```python
plt.hist(X,color='red',alpha=0.2,edgecolor='black',bins=np.linspace(fmin,fmax,20))
plt.xlabel(feature + ' ' + funits); plt.ylabel('Frequency')
plt.axvline(x=average, ymin=0, ymax=1,color='black',label='Average')
plt.axvline(x=median, ymin=0, ymax=1,color='black',label='Median',linestyle='--')
plt.axvline(x=mode, ymin=0, ymax=1,color='black',label='Mode',linestyle='dashdot')
plt.axvline(x=power_avg, ymin=0, ymax=1,color='black',label='Power',linestyle='dotted');
plt.legend(loc='upper left');
```

#### Measures of Dispersion

##### Population Variance

\begin{equation}
\sigma^2_{x} = \frac{1}{n}\sum^n_{i=1}(x_i - \mu)
\end{equation}

```python
varp = stats.pvariance(X)
print(feature + ' population variance is ' + str(round(varp,4)) + ' ' + f2units + '.')
```

    Porosity population variance is 0.0011 (fraction^2).

##### Sample Variance

\begin{equation}
\sigma^2_{x} = \frac{1}{n-1}\sum^n_{i=1}(x_i - \overline{x})^2
\end{equation}

```python
var = stats.variance(X)
print(feature + ' sample variance is ' + str(round(var,4)) + ' ' + f2units + '.')
```

    Porosity sample variance is 0.0011 (fraction^2).

##### Population Standard Deviation

\begin{equation}
\sigma_{x} = \sqrt{ \frac{1}{n}\sum^n_{i=1}(x_i - \mu)^2 }
\end{equation}

```python
stdp = stats.pstdev(X)
print(feature + ' sample variance is ' + str(round(stdp,4)) + ' ' + f2units + '.')
```

    Porosity sample variance is 0.0329 (fraction^2).

##### Sample Standard Deviation

\begin{equation}
\sigma_{x} = \sqrt{ \frac{1}{n-1}\sum^n_{i=1}(x_i - \mu)^2 }
\end{equation}

```python
std = stats.stdev(X)
print(feature + ' sample variance is ' + str(round(std,4)) + ' ' + f2units + '.')
```

    Porosity sample variance is 0.0329 (fraction^2).

##### Range

\begin{equation}
range_x = P100_x - P00_x
\end{equation}

```python
range = np.max(X) - np.min(X)
print(feature + ' range is ' + str(round(range,2)) + ' ' + funits + '.')
```

    Porosity range is 0.17 (fraction).

##### Percentile

\begin{equation}
P(p)_x = F^{-1}_{x}(p)
\end{equation}

```python
p_value = 13
percentile = np.percentile(X,p_value)
print(feature + ' ' + str(int(p_value)) + 'th percentile is ' + str(round(percentile,2)) + ' ' + funits + '.')
```

    Porosity 13th percentile is 0.11 (fraction).

##### Inter Quartile Range

\begin{equation}
IQR = P(0.75)_x - P(0.25)_x
\end{equation}

```python
iqr = scipy.stats.iqr(X)
print(feature + ' interquartile range is ' + str(round(iqr,2)) + ' ' + funits + '.')
```

    Porosity interquartile range is 0.04 (fraction).

#### Tukey Test for Outliers

Let's demonstrate the Tukey test for outliers based on the lower and upper fences.

\begin{equation}
fence_{lower} = P_x(0.25) - 1.5 \times [P_x(0.75) - P_x(0.25)]
\end{equation}

\begin{equation}
fence_{upper} = P_x(0.75) + 1.5 \times [P_x(0.75) - P_x(0.25)]
\end{equation}

Then we declare samples values above the upper fence or below the lower fence as outliers.

```python
p25, p75 = np.percentile(X, [25, 75])
lower_fence = p25 - iqr * 1.5
upper_fence = p75 + iqr * 1.5
outliers = X[np.where((X > upper_fence) | (X < lower_fence))[0]]
print(feature + ' outliers by Tukey test include ' + str(outliers) + '.')
outliers_indices = np.where((X > upper_fence) | (X < lower_fence))[0]
print(feature + ' outlier indices by Tukey test are ' + str(outliers_indices) + '.')
```

    Porosity outliers by Tukey test include [0.06726 0.05    0.06092].
    Porosity outlier indices by Tukey test are [110 152 198].

#### Let's Visualize Outliers with a Box Plot (Box and Wisker Plot)

The median is the orange line, P25 and P75 are the box and lower and upper fences are the wiskers.

```python
plt.boxplot(X); plt.ylabel(feature + ' ' + funits)
plt.xticks([1], [feature + ' Boxplot and Outliers']); plt.show()
```

#### Measures of Shape

##### Pearson's Mode Skewness

\begin{equation}
skew = \frac{3 (\overline{x} - P50_x)}{\sigma_x}
\end{equation}

```python
skew = (average - median)/std
print(feature + ' skew is ' + str(round(skew,2)) + '.')
```

    Porosity skew is -0.03.

##### Population Skew, 3rd Central Moment

\begin{equation}
\gamma_{x} = \frac{1}{n}\sum^n_{i=1}(x_i - \mu)^3
\end{equation}

```python
cm = scipy.stats.moment(X,moment=3)
print(feature + ' 3rd cenral moment is ' + str(round(cm,7)) + '.')
```

    Porosity 3rd cenral moment is -1.22e-05.

##### Quartile Skew Coefficient

\begin{equation}
QS = \frac{(P75_x - P50_x) - (P50_x - P25_x)}{(P75_x - P25_x)}
\end{equation}

```python
qs = ((np.percentile(X,75)-np.percentile(X,50))
          -(np.percentile(X,50)-np.percentile(X,25))) /((np.percentile(X,75))-np.percentile(X,25))
print(feature + ' quartile skew coefficient is ' + str(round(qs,2)) + '.')
```

    Porosity quartile skew coefficient is 0.14.

#### Plot the Nonparametric CDF

Let's demonstrate plotting a nonparametric cumulative distribution function (CDF) in Python

```python
# sort the data:
sort = np.sort(X)

# calculate the cumulative probabilities assuming known tails
p = np.arange(len(X)) / (len(X) - 1)

# plot the cumulative probabilities vs. the sorted porosity values
plt.subplot(122)
plt.scatter(sort, p, c = 'red', edgecolors = 'black', s = 10, alpha = 0.7)
plt.xlabel(feature + ' ' + funits); plt.ylabel('Cumulative Probability'); plt.grid();
plt.title('Nonparametric CDF')
plt.ylim([0,1]); plt.xlim([0,0.25])
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
```

#### Fit a Gaussian Distribution

Let's fit a Gaussian distribution

* we get fancy with Maximuum Likelihood Estimation (MLE) for the Gaussian parametric distribution fit mean and standard deviation

```python
values = np.linspace(fmin,fmax,100)
fit_mean, fit_stdev = norm.fit(X,loc = average, scale = std) # fit MLE of the distribution
cumul_p = norm.cdf(values, loc = fit_mean, scale = fit_stdev)

# plot the cumulative probabilities vs. the sorted porosity values
plt.subplot(122)
plt.scatter(sort, p, c = 'red', edgecolors = 'black', s = 10, alpha = 0.7,label='data')
plt.plot(values,cumul_p, c = 'black',label='fit'); plt.legend(loc='upper left')
plt.xlabel(feature + ' ' + funits); plt.ylabel('Cumulative Probability'); plt.grid();
plt.title('Nonparametric and Fit Gaussian CDFs')
plt.ylim([0,1]); plt.xlim([0,0.25])
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
```

#### Comments

This was a basic demonstration of univariate statistics in Python.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


#Fourier Analysis

```python
import math
import scipy # Has linear algebra
import scipy.ndimage
import numpy as np
import sympy as sm
sm.init_printing(use_latex='mathjax')
import matplotlib.pyplot as plt
import matplotlib.patches as pch
%matplotlib inline
from matplotlib.ticker import MaxNLocator # to make integer ticks on x axis
from sympy import *
from sympy import series
from sympy import lambdify, Piecewise, fourier_series
from sympy.abc import x
```

### 1. Exponential form of Complex Numbers

Express each of the following complex numbers in the form $r e^{i\theta}$. Do this by hand (or in Latex) and check your work with Python.

$$0$$
$$1$$
$$-1$$
$$i$$
$$1-2i$$
$$-8+7i$$

1.a) $z = 0$ then
$$
r = \sqrt{0^2 + 0^2} = \sqrt{0} = 0
$$
and
$$
\theta = \arctan(0,0) = undefined, we\:consider\:it\:0
$$
so in $r.e^{i\theta}$ form,
$$
0 = 0.e^{0.i} = 0.1= 0
$$

----------------------------------------------------------------------------------------------------------------------------------
1.b) $z = 1$ then
$$
r = \sqrt{1^2 + 0^2} = \sqrt{1} = 1
$$
and
$$
\theta = \arctan(0,1) = 0
$$
so in $r.e^{i\theta}$ form,
$$
1 = 1.e^{0.i} = 1.1= 1
$$

----------------------------------------------------------------------------------------------------------------------------------
1.c) $z = -1$ then
$$
r = \sqrt{-1^2 + 0^2} = \sqrt{1} = 1
$$
and
$$
\theta = \arctan(0,-1) = \pi \:because\:direction\:is\:negative
$$
so in $r.e^{i\theta}$ form,
$$
-1 = 1.e^{\pi.i} = e^{i.\pi} = -1 (From Euler's formula)
$$

----------------------------------------------------------------------------------------------------------------------------------
1.d) $z = i$ then
$$
r = \sqrt{0^2 + 1^2} = \sqrt{1} = 1
$$
and
$$
\theta = \arctan(1,0) = \frac{\pi}{2}
$$
so in $r.e^{i\theta}$ form,
$$
i = 1.e^{\frac{\pi}{2}{i}} = e^{i.\frac{\pi}{2}} = i (From Euler's formula)
$$

----------------------------------------------------------------------------------------------------------------------------------
1.e) $z = 1-2i$ then
$$
r = \sqrt{1^2 + (-2)^2} = \sqrt{1+4} = \sqrt{5}
$$
and
$$
\theta = \arctan(-2,1) = -arctan(2) = -63.435 deg\:or\:-1.107 rad
$$
so in $r.e^{i\theta}$ form,
$$
1-2i = \sqrt{5}.e^{-i.tan^{-1}(2)}
$$

----------------------------------------------------------------------------------------------------------------------------------
1.f) $z = -8+7i$ then
$$
r = \sqrt{(-8)^2 + 7^2} = \sqrt{64+49} = \sqrt{113}
$$
and
$$
\theta = \arctan(7,-8) = \pi-arctan(7/8) = 180-41.186 deg\:or 3.14-0.718 rad = 138.814 deg \:or\:2.422 rad
$$
so in $r.e^{i\theta}$ form,
$$
-8+7i = \sqrt{113}.e^{i(\pi-tan^{-1}(\frac{7}{8}))}
$$

```python
#1.a)
z_a = 0
r_a = np.abs(z_a)
theta_a = np.arctan2(np.imag(z_a),np.real(z_a))
p_a = r_a * np.exp(1j*theta_a)
print('1.a)', 'r:', r_a,',theta:', theta_a, ',Polar form:', p_a)

#1.b)
z_b = 1
r_b = np.abs(z_b)
theta_b = np.arctan2(np.imag(z_b),np.real(z_b))
p_b = r_b * np.exp(1j*theta_b)
print('1.b)', 'r:', r_b,',theta:', theta_b, ',Polar form:', p_b)

#1.c)
z_c = -1
r_c = np.abs(z_c)
theta_c = np.arctan2(np.imag(z_c),np.real(z_c))
p_c = r_c * np.exp(1j*theta_c)
print('1.c)', 'r:', r_c,',theta:', theta_c, ',Polar form:', p_c)

#1.d)
z_d = 1j
r_d = np.abs(z_d)
theta_d = np.arctan2(np.imag(z_d),np.real(z_d))
p_d = r_d * np.exp(1j*theta_d)
print('1.d)', 'r:', r_d,',theta:', theta_d, ',Polar form:', p_d)

#1.e)
z_e = 1-2j
r_e = np.abs(z_e)
theta_e = np.arctan2(np.imag(z_e),np.real(z_e))
p_e = r_e * np.exp(1j*theta_e)
print('1.e)', 'r:', r_e,',theta:', theta_e, ',Polar form:', p_e)

#1.f)
z_f = -8+7j
r_f = np.abs(z_f)
theta_f = np.arctan2(np.imag(z_f),np.real(z_f))
p_f = r_f * np.exp(1j*theta_f)
print('1.f)', 'r:', r_f,',theta:', theta_f, ',Polar form:', p_f)
```

    1.a) r: 0 ,theta: 0.0 ,Polar form: 0j
    1.b) r: 1 ,theta: 0.0 ,Polar form: (1+0j)
    1.c) r: 1 ,theta: 3.141592653589793 ,Polar form: (-1+1.2246467991473532e-16j)
    1.d) r: 1.0 ,theta: 1.5707963267948966 ,Polar form: (6.123233995736766e-17+1j)
    1.e) r: 2.23606797749979 ,theta: -1.1071487177940904 ,Polar form: (1.0000000000000002-2j)
    1.f) r: 10.63014581273465 ,theta: 2.4227626539681686 ,Polar form: (-7.999999999999999+7.000000000000001j)

### 2. A Trigonometric Identity

The goal of this exercise is to find formulas for $\cos(\alpha+\beta)$ and $\sin(\alpha + \beta)$. in terms of $\sin\alpha$, $\sin\beta$, $\cos\alpha$, and $\cos\beta$.

a) First, express $e^{(\alpha+\beta)i}$ directly in terms of the $\sin$ and $\cos$ using Euler's formula.

b) Next, express $e^{(\alpha+\beta)i}$ as a product of two exponentials. Then convert each term in the product into $\sin$ and $\cos$ using Euler's formula. Expand the expression and group the terms in the real part and the imaginary part.

c) Equate the real parts and imaginary parts from (a) and (b).

2.a) Using Euler's formula: $e^{i \theta} = \cos \theta + i \sin \theta$

We can say that,

$e^{(\alpha+\beta)i}=cos(\alpha+\beta)+isin(\alpha+\beta)$-----(1)

b) Also, $e^{(\alpha+\beta)i}$ can be expressed as $e^{i\alpha}.e^{i\beta}$----(2)

From Euler's formula, $e^{i\alpha}= \cos \alpha + i \sin \alpha$

and, $e^{i\beta}= \cos \beta + i \sin \beta$

So, their product $e^{i\alpha}.e^{i\beta}$= $(\cos \alpha + i \sin \alpha)$$(\cos \beta + i \sin \beta)$= $(\cos \alpha.\cos \beta+ i \cos \alpha.\sin \beta+i \cos \beta.\sin \alpha + i^2 \sin \alpha\sin \beta)$

$e^{i\alpha}.e^{i\beta}$= $(\cos \alpha.\cos \beta- \sin \alpha\sin \beta)+i( \cos \alpha.\sin \beta+ \cos \beta.\sin \alpha )$-----(3)

c) From eqns. (1), (2) and (3), we have,

$cos(\alpha+\beta)+isin(\alpha+\beta)$= $(\cos \alpha.\cos \beta- \sin \alpha\sin \beta)+i( \cos \alpha.\sin \beta+ \cos \beta.\sin \alpha )$----(4)
Thus, by seperating the real and imaginary parts from eqn. (4), we have,
$$cos(\alpha+\beta)= (\cos \alpha.\cos \beta- \sin \alpha\sin \beta)$$
$$sin(\alpha+\beta)= ( \cos \alpha.\sin \beta+ \cos \beta.\sin \alpha )$$

### 3. Fourier Series Approximation

Similar to the example of the square wave done in class, find a Fourier Series approximation to the function

$$
f(t) = \left |cos(2\pi t) \right|
$$

which is a *rectified cosine*. Make sure you get the period of this function correct. Make a plot with three subplots for $N=5$, $N=10$, and $N=20$ showing better and better approximations.

```python
t = sm.symbols("t", real=True)
n = sm.symbols("n", integer=True, positive=True)
q = 2*sm.pi*sm.I*n*t
p = 2*sm.pi*n*t
T = sm.Rational(1,2)

fn = sm.integrate(sm.cos(p)*sm.exp(q),(t,-T/2,T/2)) #cos is positive from -pi/2 to pi/2
display("Expression:", fn)
fn = sm.simplify(fn)
display("Simplified Expresion:",fn)
```

    'Expression:'

$\displaystyle \frac{1}{4} - \frac{i e^{i \pi n}}{8 \pi n} + \frac{i e^{- i \pi n}}{8 \pi n}$

    'Simplified Expresion:'

$\displaystyle \frac{1}{4}$

```python
N1 = range(-2,3)
N2 = range(-5,6)
N3 = range(-10,11)
terms_N1 = [0 if k == 0 else fn.subs(n,k)*sm.exp(-2*sm.pi*sm.I*k*t) for k in N1]
terms_N2 = [0 if k == 0 else fn.subs(n,k)*sm.exp(-2*sm.pi*sm.I*k*t) for k in N2]
terms_N3 = [0 if k == 0 else fn.subs(n,k)*sm.exp(-2*sm.pi*sm.I*k*t) for k in N3]

fapprox_N1 = sum(terms_N1)
fapprox_N2 = sum(terms_N2)
fapprox_N3 = sum(terms_N3)

#Plotting
line1, line2, line3 = sm.plot(fapprox_N1, fapprox_N2, fapprox_N3, (t,-1,1), show=False)
x1, y1 = line1.get_points()
x2, y2 = line2.get_points()
x3, y3 = line3.get_points()
fig = plt.figure(figsize = (10,10))
plt.plot(x1, y1, color='r', label='N=5')
plt.plot(x2, y2, color='y', label='N=10')
plt.plot(x3, y3, color='g', label='N=20')
plt.title("Finite Fourier Series Approximation")
plt.xlabel('n')
plt.ylabel('f(n)')
plt.axhline(y=0, color='gray')
plt.axvline(x=0, color='gray')
plt.legend();
plt.show()
```

### 4. Orthogonality

Show that $\cos(2\pi t)$ and $\sin(2 \pi t)$ are orthogonal by showing that their inner product (a.k.a. dot product) is zero. Hint: Express the two functions in exponential form, and expand their product inside the integral.

4) Let $f(t)=cos(2\pi t)$

Representing $f(t)$ using Euler's representation-

$cos(2\pi t) =\frac{e^{2\pi it}+e^{-2\pi it}}{2}$-----(1)

And, let $g(t)=sin(2\pi t)$

Representing $g(t)$ using Euler's representation-

$sin(2\pi t) =\frac{e^{2\pi it}-e^{-2\pi it}}{2i}$-----(2)

Now, the inner product or dot product of the 2 functions is-

$=\int_0^1 f(t)\overline{g(t)} dt$

From eqn (1) and (2), we get,

$=\int_0^1 \frac{e^{2\pi it}+e^{-2\pi it}}{2}.\int_0^1 \overline{\frac{e^{2\pi it}-e^{-2\pi it}}{2i}}$

$=\int_0^1 \frac{e^{2\pi it}+e^{-2\pi it}}{2}.\int_0^1 \frac{e^{-2\pi it}-e^{2\pi it}}{2i}$

$=\frac{1}{4i}\int_0^1 (e^{2\pi it}+e^{-2\pi it}).(e^{-2\pi it}-e^{2\pi it})$

$=\frac{-1}{4i}\int_0^1 (e^{2\pi it}+e^{-2\pi it}).(e^{2\pi it}-e^{-2\pi it})$

Now, as we know that $(a+b)(a-b)=(a^2-b^2)$ So the above eqn can be simplified as,

$=\frac{-1}{4i}\int_0^1 (e^{4\pi it}-e^{-4\pi it})$

Now, again from Euler's representation,

$=\frac{-1}{4i}\int_0^1 2i.sin(4\pi t)$

$=\frac{-2i}{4i}\int_0^1 sin(4\pi t)$

$=\frac{-1}{2}\frac{-cos(4\pi t)}{4 \pi}|_0^1$

$=\frac{1}{8 \pi}(cos 4\pi-cos 0)$

$=\frac{1}{8 \pi}(1-1)=0$

Hence, we can conclude as the inner or dot product of $\cos(2\pi t)$ and $\sin(2 \pi t)$ is 0, they both are Orthogonal to each other.

### 5. Fourier Transforms

Find the Fourier Transforms of the following functions. You can use `sympy`.
Make sure to simplify, these shouldn't give complicated results.

Plot original functions and their transforms in a nice 3 $\times$ 2 grid of plots. Label the sub plots.

$$
f(t) = \left \{
\begin{array}{l}
1 \;\mathrm{if} \;t \geq 0 \\
0 \;\mathrm{else}
\end{array}
\right .
$$

$$
f(t) = \frac{1}{1+t^2}
$$

$$
f(t) = e^{-\pi t^2}
$$

```python
n = 2
#First given function
a = Piecewise((1, t >= 0), (0, t < 0), (0, True))
fn1 = lambdify(t, a)
print('Function 1:')
display(a)
s = fourier_series(a, (t, -1, 1))
print('Fourier series of Function 1:')
s = s.truncate(n=n)
display(s.trigsimp())
exp1 = lambdify(t,s.trigsimp())

#Second given function
b = 1/(1+t**2)
fn2 = lambdify(t,b)
print('Function 2:')
display(b)
s2 = fourier_series(b, (t, -1, 1))
print('Fourier series of Function 2:')
s2 = s2.truncate(n=n)
display(s2.simplify())
exp2 = lambdify(t,s2)

#Third given function
c = sm.exp(-sm.pi*(t**2))
fn3 = lambdify(t,c)
print('Function 3:')
display(c)
s3 = fourier_series(c, (t, -1, 1))
print('Fourier series of Function 3:')
s3 = s3.truncate(n=n)
display(s3.simplify())
exp3 = lambdify(t,s3)

#Plotting parameters
fig = plt.figure(figsize =(10,10))
x = np.linspace(-3,3, 500)

fig.add_subplot(3,2,1)
plt.plot(x, fn1(x), 'b')
plt.title('Function 1')
plt.xlabel('t')
plt.ylabel('$f(t)$')

fig.add_subplot(3,2,2)
plt.plot(x, exp1(x), 'b')
plt.title('Fourier of Function 1')
plt.xlabel('s')
plt.ylabel('$f(s)$')

fig.add_subplot(3,2,3)
plt.plot(x, fn2(x), 'r')
plt.title('Function 2')
plt.xlabel('t')
plt.ylabel('$f(t)$')

fig.add_subplot(3,2,4)
plt.plot(x, exp2(x), 'r')
plt.title('Fourier of Function 2')
plt.xlabel('s')
plt.ylabel('$f(s)$')

fig.add_subplot(3,2,5)
plt.plot(x, fn3(x), 'g')
plt.title('Function 3')
plt.xlabel('t')
plt.ylabel('$f(t)$')

fig.add_subplot(3,2,6)
plt.plot(x, exp3(x), 'g')
plt.title('Fourier of Function 3')
plt.xlabel('s')
plt.ylabel('$f(s)$')
```

### 6. Solve a system of ODEs

Similar to the ODE example in the notes, find a solution to the two differential equations

\begin{eqnarray}
\dot x & = & -y \\
\dot y & = & -x + u
\end{eqnarray}

a) First take the Fourier Transform of both equations.

b) Solve for $Y(s)$ in terms of $U(s)$.

c) Note that the solution for $Y(s)$ involves a function of $s$ times $U(s)$. Figure out the inverse Fourier Transform of that function.

d) Use a convolution to figure out $y(t)$ in terms of $u(t)$.

e) Find $X(s)$ in terms of $Y(s)$ and then $x(t)$ in terms of $y(t)$.

6.a) Given eqns are:
$\dot x  =  -y$ and $\dot y  =  -x + u$
Taking the Fourier Transform of both equations we get-

$ 2\pi i sX(s) = -Y(s) ---(1)$

$ 2\pi i sY(s) = -X(s) + U(s) ---(2)$

b) From eqn (1), we have

$ X(s) = \frac{-Y(s)}{2\pi i s}$ and substituting this in eqn (2),

$ 2\pi i sY(s) = \frac{Y(s)}{2\pi i s} + U(s) $

$ 2\pi i sY(s) - \frac{Y(s)}{2\pi i s}=   U(s) $

$ Y(s)  (2\pi i s - \frac{1}{2\pi i s}) =   U(s) $

$ Y(s) =  (2\pi i s - \frac{1}{2\pi i s})^{-1} * U(s) ----(3)$

$ Y(s) =  F(s) * U(s)$ where $F(s)=(2\pi i s - \frac{1}{2\pi i s})^{-1}$

c) Inverse fourier transform of the function $F(s)$ obtained in eqn (3),

$\mathcal{F}^{-1}F(s)= f(t) = \int_{-\infty}^{\infty} (2\pi i s - \frac{1}{2\pi i s})^{-1} e^{i2πst}ds $

$= \int_{-\infty}^{\infty} {\frac{2\pi i s} {(2\pi i s)^2 - 1}} e^{i2πst}ds $

$= \int_{-\infty}^{\infty} { \frac{2\pi i s} {(2\pi i s + 1) (2\pi i s - 1)}} e^{i2πst}ds $

$= \int_{-\infty}^{\infty} {\frac{ 2\pi i s} {2} \left(\frac{1} {(2\pi i s - 1)} - \frac{1}{(2\pi i s + 1)}\right)} e^{ i 2πst}ds $

$= \int_{-\infty}^{\infty} {\frac{ 2\pi i s} {2} \left(\frac{1} {(2\pi i s - 1)}\right)e^{ i 2πst}ds - \int_{-\infty}^{\infty} \frac{ 2\pi i s} {2} \left(\frac{1}{(2\pi i s + 1)}\right)} e^{i2πst}ds $

$f(t)=\frac{e^{\frac{-t}{2\pi}}\theta (t)}{2\sqrt{2\pi}}+\frac{e^{\frac{t}{2\pi}}\theta (-t)}{2\sqrt{2\pi}}$

d) Using convolution,

$y(t)= f(t)*u(t)$

$y(t)= (\frac{e^{\frac{-t}{2\pi}}\theta (t)}{2\sqrt{2\pi}}+\frac{e^{\frac{t}{2\pi}}\theta (-t)}{2\sqrt{2\pi}})u(t)$

e)From (b) we have-

$ X(s) = \frac{-Y(s)}{2\pi i s}$

Now, taking the inverse Fourier transform of the above eqn we get,

$\mathcal{F}^{-1}X(s)=x(t)=g(t)*y(t)$

where, $g(t)= \mathcal{F}^{-1}(\frac{-1}{2\pi i s})=-\frac{sgn(t)}{4\pi}$

Therefore, $x(t)=-\frac{sgn(t)}{4\pi}y(t)$

### 7.  Image Processing

Find a nice gray scale image to load, or use the Minion image from class. Call the resulting matrix $m$.

a) Render the image using `matplotlib`.

b) Take the DFT of the image $m$ to get a matrix $M$. Then make a new matrix $M_\mathrm{new}$ in which each entry is just the magnitude of the corresponding entry in $M$. Create a new image from the inverse Fourier transform of this image and render it.

c) Do the same thing, except set the entries in $M_\mathrm{new}$ to the angle of the entries in $M$. Create a new image from the inverse Fourier transform of this image and render it.

d) Do the same thing, except normalize all the entries in $M_\mathrm{new}$ to have magnitude 1. Create a new image from the inverse Fourier transform of this image and render it.

What might you conclude about the information contained in the magnitude of the transformed image? What might you conclude about the information contained in the angle?

```python
#a)
import imageio
m = imageio.imread('sample_data/Minion.jpg', as_gray=True)

#b) DFT
M = np.fft.fft2(m)
M = np.fft.fftshift(M) # Shift it so the low frequencies are in the middle
M_new = 20*np.log(np.abs(M))

#c) Setting M_new to the angles of entries in M and IFT
M_new_M = 20*np.abs(np.angle(M, deg = True))
M_new_M = np.fft.fft2(M_new_M)
M_new_M = np.fft.fftshift(M_new_M) # Shift it so the low frequencies are in the middle
M_new_M = 20*np.log(np.abs(M_new_M))

#d) Normalize to magnitude 1 and IFT
norm_M = M/abs(M)
M_norm = np.fft.fft2(norm_M)
M_norm= np.fft.fftshift(norm_M) # Shift it so the low frequencies are in the middle
M_norm = 20*np.log(np.abs(M_norm))

#Plotting
fig, ax = plt.subplots(1,4,figsize=(20,5))
ax[0].imshow(m, cmap = 'gray'),ax[0].set_title('Original Image')
ax[1].imshow(M_new, cmap = 'gray'),ax[1].set_title('DFT of Image')
ax[2].imshow(M_new_M, cmap = 'gray'),ax[2].set_title('DFT using angles in M')
ax[3].imshow(M_norm, cmap = 'gray'),ax[3].set_title('Normalized Image')
```


# Lesson - Machine Learning VI:Calculus for Machine Learning

## Sec-1 - Linear and Non Linear Functions

Earlier, we explored the machine learning workflow using the k-nearest neighbors algorithm. We chose the k-nearest neighbors algorithm because building the intuition for how the algorithm works doesn't require any mathematics. While the algorithm is easy to grasp, we can't use it for larger datasets because the model itself is represented using the entire training set. Each time we want to make a prediction on a new observation, we need to calculate the distance between each observation in our training set and our new observation, then rank by ascending distance. This is a computationally intensive technique.

For most of the machine learning techniques, the model is represented as a mathematical function. This mathematical function approximates the underlying function that describes how the features are related to the target attribute. Once we derive this mathematical function using the training dataset, making predictions on the test dataset (or on a future dataset) is computationally cheap. The following diagram shows 2 different linear regression functions that approximate the dataset (note that the values in this dataset are random).

Calculus provides a framework for understanding how mathematical functions behave. Calculus helps us:

- understand the steepness at various points
- find the extreme points in a function
- determine the optimal function that best represents a dataset

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# Setting pandas display options for large data
pd.options.display.max_rows = 200
pd.options.display.max_columns = 150
```

### Linear Function

A simple, straight line is more clearly defined as a linear function. All linear functions can be written in the following form:

$$y = mx + b$$

For a specific linear function, m and b are constant values while x and y are variables. y=3x+1 and y=5 are both examples of linear function.

Let's focus on the function y=3x+1 for now. This function multiples any x value we pass in by 3 then adds 1 to it.

The constant m determines how steep the line is. When m is equal to 0, the line is completely flat and is parallel to the x-axis. When m and b are both set to 0, the line is equivalent to the x-axis.

The m value controls a line's slope while the b value controls a line's y-intercept. The y-intercept is the value where the line intersects the y-axis. When we fix m but vary b, the point at which the line intersected the y-axis varies. The line intersects the y-axis when x is 0. For y=3x+1, the y-intercept is 1:
$$f(0) = 3(0) + 1 = 1$$

While the y-intercept describes where a line intersects the y-axis, the slope describes the steepness of the line. Whenever we set m to a positive value, the y values increase as the x values increase. Whenever we set m to a negative value, the y values decrease as the x values increase. When m was 0, the y value was consistent throughout (and equivalent to the y-intercept).

Another way to think about slope is **rate of change**. The rate of change is how much the y axis changes for a specific change in the x axis.

$$m = \frac {\text {change in y}} {\text{change in x}}  = \frac{\Delta y}{ \Delta x}$$

Following is the slope calculation between the points (1,4) and (4,13), which are both points on the line y=3x+1:
$$m = \frac {\text {change in y}} {\text{change in x}}  = \frac {\text {13 - 4}} {\text{4 - 1}} = 3$$

When referring to unique x and y coordinates, an integer value is usually added to the subscripts. For example, the points (1,4) and (4,13) from the last calculation can be referred to as (x1,y1) and (x2,y2). If we don't specify what (x1,y1) and (x2,y2) are, they act as variables.

### Slope

If we calculate the rate of change between any points on a line, it will always equal the function's slope. Let's show this for the more general case. In the following math block, (x1,y1) and (x2,y2) represent any 2 coordinates on a line:
$$y_1 = mx_1 + b$$
$$y_2 = mx_2 + b$$

We can subtract both of these equations to get:
$$y_1 - y_2 = mx_1 - mx_2 + b - b$$
$$y_1 - y_2 = m(x_1 - x_2)$$
$$m = (y_1 - y_2)/(x_1 - x_2)$$

where y1−y2 represents Δy, and x1−x2 represents Δx.

Below, we will write a function for calculating slope of a line.

```python
def slope(x1, x2, y1, y2):
    slope = (y2-y1) / (x2-x1)
    return slope

slope_one = slope(0,4,1,13)
slope_two = slope(5,-1,16,-2)
print(slope_one, slope_two)

```

    3.0 3.0

### Non Linear Function

Suppose, following equation, which describes the trajectory of a ball after it's kicked by a football player:

$$y = -(x^2) + 3x - 1$$

x is time in seconds while y is the vertical position of the ball. Naturally, we'd like to know the highest position the ball reached and at what time that happened. While we can graph the equation and estimate the result visually, if we want the precise time and vertical position we'll need to use calculus.

If we analyze the above equation, we'll notice that it doesn't match the form y=mx+b. This is because this function is a nonlinear function. Nonlinear functions don't represent straight lines -- they represent curves. The outputs y of a nonlinear function are not proportional to the input values x. An increment in x doesn't result in a constant increment in y.

**Whenever x is raised to a power not equal to 1, we have a non-linear function**. Here are some more examples of nonlinear functions:

$$y=x3$$
$$y = x^3 + 3x^2 + 2x - 1$$
$$y = \frac{1} {-x^2}$$
$$y = \sqrt x$$

```python
x = np.linspace(start = 0, stop= 3, num = 100, endpoint = True )
y = -1* (x**2) + 3*x - 1

fig = plt.figure(figsize=(8, 6))
plt.style.use('fivethirtyeight')
plt.plot(x, y, color = 'red')
plt.xlabel("x") # k is the order of listing not k value
plt.ylabel('y')
plt.show()
```

### Secant Lines

When we calculate the slope between 2 points on a curve, we're really calculating the slope between the line that intersects both of those points. The equation for a linear function is y=mx+b. If we know the values for m, x, and y, we can calculate b, the y-intercept. For example for the following function if x1 = 0 and x2 = 1:
$$y = -(x^2) + 3x - 1$$
$$x_1 = 0$$
$$x_2 = 1$$
We use y=−(x2)+3x−1 to calculate y1 and y2 which come out to be y1= -1 and y2 = 1 and the slope m = (y2-y1)/ (x2-x1) = 2 and:
$$y_1 = mx_1 + b$$
$$-1 = 2(0) + b$$
b = -1

Finally, we can use the slope and the y-intercept to declare the equation for the line connecting both points:
$$y = 2x - 1$$

A line that intersects 2 points on a curve is known as a **secant line**.

**Exercise**

Writ the function draw_secant() for the above nonlinear function between the x-values -20 and 30.

```python
import seaborn
seaborn.set(style='darkgrid')

def draw_secant(x_values):
    x = np.linspace(-20,30,100)
    y = -1*(x**2) + x*3 - 1
    plt.plot(x,y)
    y_values = []
    for values in x_values:
        y_func = -1*(values**2) + values*3 - 1
        y_values.append(y_func)
    m = (y_values[1] - y_values[0]) / (x_values[1] - x_values[0])
    b = y_values[1] - m*x_values[1]
    y_sec_line = m*x + b
    plt.plot(x, y_sec_line, c='green')
    plt.show()
```

```python
draw_secant([3,5])
draw_secant([3,10])
draw_secant([3,15])

```

### Secant Line and Slope

Above, we drew secant lines between the fixed x1 value and increasingly larger x2 values. When the gap between x1 and x2 was small, the secant line intersected comfortably with our curve. As we increased the gap between the 2 x-values, the secant line started to diverge from our curve. This observation is critical to understanding how slope works for curves.

The slope of a curve at a specific point, x1 is best understood as the slope of the secant line at increasingly smaller intervals of $[x1,x2]$. The smaller the difference between x1 and x2, the more precise the secant line approximates the slope at that point on our curve.

### Tangent Line

When x1 and x2 are equivalent,the slope value is undefined. This is because division by 0 has no meaning in mathematical calculations.
$$m = \frac{-1 - -1}{3 - 3} = \frac{0}{0} = \hbox{ undefined }$$

As x2 approaches x1 from either side of the curve, however, the slope of the secant line approaches −3. Even though we can't calculate the slope at the point where x0=x1=3, we can tell from calculations that it approaches −3. This value is known as the instantaneous rate of change, because it describes the slope at a particular point. For linear functions, the instantaneous rate of change at any point on the line is the same. For nonlinear functions, the instantaneous rate of change describes the steepness of the graph of the function at a specific point.

This line is known as the tangent, and, unlike the secant line, it only intersects our function at one point. So far, we've been working with secant lines that connect 2 points that are increasingly close together. We can think of the tangent line as the secant line when both points are the same.

## Sec-2 - Limits

Above, we observed what happened when we moved the second point closer to the first point along the curve. The larger the interval between the 2 points on the x-axis, the more the steepness of the secant line diverged from the steepness of the curve. The closer the interval, the more the secant line started to match the steepness at the first point on the curve.

Earlier we saw that if  we try to state the observation by plugging in values to the slope equation, we'll run into the division-by-zero problem:

$$m = \frac{f(x_2) - f(x_1)}{x_2 - x_1} = 0 / 0$$

Even though the slope is undefined when x1 and x2 are equivalent, we still want to be able to state and reason about what value the slope approaches as x2 approaches x1. To do that, we need to reframe the problem as a limit. A limit describes the value a function approaches when the input variable to the function approaches a specific value. In our case, the input variable is x2 and our function is $m = \frac{f(x_2) - f(x_1)}{x_2 - x_1}$. The following mathematical notation formalizes the statement "As x2 approaches 3, the slope between x1 and x2 approaches −3" using a limit:

$$\lim_{x_2 \to 3} \frac{f(x_2) - f(x_1)}{x_2 - x_1} = -3$$

$\lim_{x_2 \to 3}$ is another way of saying "As x2 approaches 3". Because we fixed x1 to 3, we can replace x1 with 3 in the function:
$$\lim_{x_2 \to 3} \frac{f(x_2) - f(3)}{x_2 - 3} =  \lim_{x_2 \to 3} \frac{f(x_2) + 1}{x_2 - 3} = -3$$

### Defined versus Undefined Limits

There are 2 kinds of limits -- defined limits and undefined limits. The limit above was an undefined limit because plugging in 3 for x2 results in an undefined value. A defined limit can be evaluated just by substituting the value into the limit.

Following are the slopes of the secant lines between closer and closer points to x1=3:

>
   x1	x0	m
    2.9	3	-2.9
   2.99	3	-2.99
   2.999 3	-2.999
   3.01	3	-3.01
   3.001 3	-3.001
   3.0001 3	-3.0001
   >

 To calculate each of these slopes, we plugged in both x1 and x2 values into the slope equation. Here's the calculation for the slope of the secant line between the points at x1=3 and x2=2.9 where $f(x) = -(x^2) + 3x - 1$
 $$m = \frac{f(2.9) - f(3)}{2.9 - 3} = -2.9$$

 We can actually rewrite each of these as defined limits. For example, the following defined limit describes the slope of the secant line between the same 2 points:
 $$\lim_{x_2 \to 2.9} \frac{f(x_2) - f(x_1)}{x_2 - x_1}$$

 Since we've fixed x1 to the value 3, we can replace x1 with 3 and f(x1) with −1:
 $$\lim_{x_2 \to 2.9} \frac{f(x_2) - (-1)}{x_2 - 3}$$

 To finish solving the limit, we can use direct substitution and plug in 2.9 for x2 into the limit. This is because the limit is defined at the specific value (2.9) that the input variable approaches (x2). When we can replace the input variable with the value it approaches in the limit, the problem is converted to a regular arithmetic problem:
 $$\frac{f(2.9) - (-1)}{2.9 - 3} = -2.9$$
By rewriting and converting an undefined limit to a defined limit, we can solve the limit using direct substitution and standard arithmetic.

### Python Sympy Library
The [SymPy](https://docs.sympy.org/dev/tutorial/calculus.html#limits) library has a suite of functions that let us calculate limits. The syntax for SymPy closely follows mathematical notation. To start, we need to import the library and use sympy.symbols() to declare the variables we want to be treated as symbols for SymPy to parse:

```
import sympy
x,y = sympy.symbols('x y')
```

Previously, we plotted a function by generating many x values (and assigning them to x), transforming those x values to y values (assigned to y) using Python arithmetic operators, and using matplotlib to plot both lists:
```
x = np.linspace(0, 3, 100)
y = -1*(x**2) + x*3 - 1
plt.plot(x,y)
```
In SymPy, the workflow is different and the Python variables we use don't directly map to specific values. In SymPy, Python variables map directly to variables in math. We express a mathematical function as a transformation of the Python variable. When we called sympy.symbols() and passed in x, x points to a special SymPy object (not a list). We then need to express the mathematical function in SymPy. We use standard Python arithmetic operators for this:

```python
import sympy
x,y = sympy.symbols('x y')

print(type(x))
y = x**2 + 3*x
y
```

    <class 'sympy.core.symbol.Symbol'>

$\displaystyle x^{2} + 3 x$

```python
x, y = sympy.symbols('x y')
y = x^2 + 1
print(y)
y = 3*x
print(y)
y = x**2 + 1
print(y)
```

    ~x
    3*x
    x**2 + 1

After we've defined the variables and the mathematical function in SymPy, we use the sympy.limit() function to calculate the limit. This function takes in 3 parameters:

- the function we're taking the limit for
- the input variable
- the value the input variable approaches

For example if we wanted to solve the following defined limit:
$$\lim_{x \to 1} x^2 + 1$$

For this problem, the input variable is x and the limit criteria is $limx→1$. Here's how to express that in SymPy:

```
limit_one = sympy.limit(x**2 +1, x, 1)
```

**Exercise**

- Use SymPy to confirm that $\lim_{x_2 \to 2.9} \frac{f(x_2) - (-1)}{x_2 - 3} = -2.9$

- plug in $x2=2.9$ into $f(x)=−(x2)+3x−1$ before passing in the limit function to sympy.limit().
- Assign the result of the limit calculation to `limit_one` and display it using the print() function.

```python
import sympy
x2, y = sympy.symbols('x2 y')
x2 = 2.9
y = -(x2**2) + 3*x2 -1
limit_one = sympy.limit((y-(-1))/ (x2-3), x2, 2.9)
print(limit_one)

```

    -2.89999999999999

### Properties of Limits I
Above is the slope of the secant line that we calculated earlier. Before we can convert undefined limits to defined limits, we need to first understand some of the properties of limits. Using these properties, we can rework undefined limits to defined limits and solve them.

- First Property:
$$\lim_{x \to a} [f(x) + g(x)] = \lim_{x \to a} f(x) + \lim_{x \to a} g(x)$$

This property allows us to break up limits with terms that are being summed into their individual components. This also holds true for subtraction:
$$\lim_{x \to a} [f(x) - g(x)] = \lim_{x \to a} f(x) - \lim_{x \to a} g(x)$$

We can apply these rules to break up the following limit into smaller components:

$$\lim_{x \to 1} [3x^2 + 3x - 3]$$
Even though this is a defined limit and we can solve it using direct substitution, it's a good example for understanding how to apply these properties.
$$\lim_{x \to 1} [3x^2] + \lim_{x \to 1} [3x] - \lim_{x \to 1} [3]$$
If we plug in x=1 to each of these limit terms:
3 + 3 - 3 = 3

**Exercise**

- Use SymPy to calculate $\lim_{x \to 1} [3x^2 + 3x - 3]$
- Assign the result to limit_two and display using the print() function.

```python
import sympy
x, y = sympy.symbols('x y')
x = 1
y = 3*(x**2) + 3*x -3
limit_two = sympy.limit(y, x, 1)
print(limit_two)
```

    3

### Properties of Limits II

The second property allows us to break up terms that are being multiplied by a constant:
$$\lim_{x \to a} [cf(x)] = c \lim_{x \to a} f(x)$$

$$\lim_{x \to 1} [3x^2] + \lim_{x \to 1} [3x] - \lim_{x \to 1} [3]$$

If we apply this limit property:
$$3 \lim_{x \to 1} [x^2] + 3 \lim_{x \to 1} [x] - 3 \lim_{x \to 1} [1]$$

If x = 1:
$$3 + 3 - 3 = 3$$

We can apply multiple properties at once:
$$\lim_{x \to 1} [3x^2 + 3x - 3]$$
$$3 \lim_{x \to 1} [x^2] + 3 \lim_{x \to 1} [x] - 3 \lim_{x \to 1} [1]$$

**Exercise**

solve the following limit:
$$\lim_{x \to -1} [x^3 + 2x^2 - 10x]$$

Assign the result to `limit_three` and display using the `print()` function.

```python
import sympy
x, y = sympy.symbols('x y')
y = x**3 + 2*x**2 - 10*x
limit_three = sympy.limit(x**3, x, -1) + 2*sympy.limit(x**2, x, -1) - 10*sympy.limit(x, x, -1)
print(limit_three)
```

    11

### Undefined Limit to Defined Limit
We will now convert the original undefined limit that calculated the slope at x=3 into a defined limit:
$$\lim_{x_2 \to 3} \frac{f(x_2) + 1}{x_2 - 3} =$$

Since $f(x) = -(x^2) + 3x - 1$

$$\lim_{x_2 \to 3} \frac{-x_2^2+3x_2 -1 + 1}{x_2 - 3}$$

$$\lim_{x_2 \to 3} \frac{-x_2^2+3x_2 }{x_2 - 3}$$

We still can't use direct substitution to solve this limit because of the denominator term. We need a way to cancel the denominator or convert it into a form that lets us plug in 3 for x2. If we factor out x2 on the numerator, we'll notice something interesting:

$$\lim_{x_2 \to 3} \frac{x_2(-x_2 + 3) }{x_2 - 3}$$

If we factor out −1 as well, we can cancel x2−3 on the denominator with the same term on the numerator:
$$\lim_{x_2 \to 3} \frac{-x_2(x_2 - 3) }{x_2 - 3}$$

This simplifies to:
$$\lim_{x_2 \to 3} -x_2$$

Now we can use direct substitution to arrive at −3. In this case, we didn't need to use the properties of limits that we explored.

**Exercise**

Use SymPy to solve the limit:$\lim_{x_2 \to 3} \frac{f(x_2) + 1}{x_2 - 3}$
Assign the result to `limit_four`.

```python
import sympy
x2,y = sympy.symbols('x2 y')
y = (-x2**2 + 3*x2 - 1)
limit_four = sympy.limit((y+1)/ (x2-3), x2, 3)
limit_four
```

$\displaystyle -3$

## Sec-3 - Finding Extreme Points

Above, we used limits to calculate the point a function approaches when the input value approaches a specific value. We applied this technique to calculate the slope of the tangent line at a specific point on our nonlinear function  y=−(x2)+3x−1. and now we are interested in determining the highest point on this curve.

Below, it can be seen that when x is less than 1.5, the slope is positive. As x approaches 1.5 from the left side, the slope decreases and approaches 0. At x=1.5, the slope of the tangent line is exactly 0. When we increased x past 1.5 the slope of the tangent line becomes increasingly negative. The highest point on our curve acts as a kind of transition point from positive slope to negative slope.

```python
import seaborn
seaborn.set(style='darkgrid')

def draw_secant(x_values):
    x = np.linspace(-20,30,100)
    y = -1*(x**2) + x*3 - 1
    plt.plot(x,y)
    y_values = []
    for values in x_values:
        y_func = -1*(values**2) + values*3 - 1
        y_values.append(y_func)
    m = (y_values[1] - y_values[0]) / (x_values[1] - x_values[0])
    b = y_values[1] - m*x_values[1]
    y_sec_line = m*x + b
    plt.plot(x, y_sec_line, c='green')
    plt.show()
```

```python
draw_secant([-5.4, -5.45])
```

```python
draw_secant([1.45, 1.5])
```

```python
draw_secant([5.4, 5.45])
```

By plotting the slopes at many different x values, we can tell that they follow a straight line. This line can be described as its own function, and is known as the derivative. **A function's derivative can tell us the slope of the tangent line for any x value along the function**. To find the general function that describes the slope at any point, we need to solve the limit using the original variable instead of replacing with a specific x value. We can write the slope equation to the more common form. Instead of using x1 and x2 in the slope equation, we will use x and x+h.
$$\lim_{h \to 0} \frac{f(x + h) - f(x)}{h}$$

In this form, h represents the distance between the 2 points: x and x+h. The slope is the value the limit approaches as h approaches 0.

### Differentiation
The process of finding a function's derivative is known as differentiation. Instead of selecting a specific x value, we'll leave x where it is and expand the equation to solve for the derivative.

$f(x) = -(x)^2 + 3x - 1$
$\lim_{h \to 0} \frac{(-(x + h)^2 + 3(x + h) - 1) - (-(x)^2 + 3x - 1)}{h}$
$\lim_{h \to 0} \frac{-x^2 - 2xh - h^2 +3x + 3h - 1 + x^2 - 3x + 1}{h}$
$\lim_{h \to 0} \frac{-2xh - h^2 + 3h}{h}$
$\lim_{h \to 0} \frac{h(-2x - h + 3)}{h}$
$\lim_{h \to 0} -2x - h + 3$

**Exercise**

- Simplify the equation by using direct substitution to solve the limit.
- Use numpy.linspace() to generate 110 values from -5 to 6 and assign to x.
- Transform x using the equation solved above and assign to y.
- Generate a line plot using x and y.

This will be the straight line representing slope values for x as $\lim_{h \to 0}$

```python
# Using substitution, above slope equation becomes
x = np.linspace(-5, 6, 110)
y = -2*x + 3

plt.plot(x, y)
```

### Critical Points
We use the following notation to specify that −2x+3 is our curve's derivative:
$$y' = -2x + 3$$

The following is an alternative form of notation, which makes it clear that the derivative of y is a function of x:
$$f'(x) = -2x + 3$$

The last common notation is the following, which can be read as "the derivative with respect to x is"
$$\frac{d}{dx}[ -x^2 + 3x - 1 ] =  -2x + 3$$

Earlier we discussed how the slope at x=1.5 was 0 and acted as a transition point from positive slope to negative slope. This is formally known as a **critical point**. **At a function's critical point**, the slope is either 0 or undefined. Not all critical points act as transition points.

### Extreme Values

Above, we observed how the slope didn't change from positive to negative at the critical point. Critical points are interesting to us in data science when they represent extreme values. Extreme values can be split into 2 categories -- minimum and maximum. As we noticed earlier, a critical point can be a minimum value if the slope transitions from negative to positive at that point. When the slope transitions from positive to negative at a point, it can be a maximum value. If the slope doesn't transition at a critical point, like at x=0 for y=x3 in the last screen, then it can't be a minimum or a maximum point.

A curve can have multiple critical points and multiple candidates for minimum and maximum values. The following curve has 3 critical points and all 3 meet the criteria of transitioning slope from positive to negative or negative to positive. At the same time, none of the values are the highest points on the curve (while one of them seems to be the lowest).

These values are known as local extrema, since they're the lowest or highest value in the interval surrounding each point. If a critical point is the lowest point in the interval surrounding it, it's known as a relative minimum. If it's the highest point in the neighborhood, it's known as a relative maximum.

Since we know that relative minimums and maximums can occur when the slope is 0, we can solve for the critical points using arithmetic (instead of relying on just the graph). To calculate the critical point for our original curve, we need to set our derivative function to 0 and solve for x.

$$-2x + 3 = 0$$
$x = 1.5$

This matches the value we observed visually in the graphs we generated above. Since this curve only had 1 critical point and the slope transitioned from positive to negative (as x is increased), we can safely say that this is a relative maximum.

### Power Rule for Derivatives
We calculated the derivative for our curve using a limit and working out the arithmetic. We then set the derivative function to 0 and calculated the critical points. This approach can be painful to do for more complex functions like $y = x^5 - x$ because we'll need to replace x with x+h and expand the exponent (x+h)5:

$$f'(x) = \lim_{h \to 0} \frac{((x + h)^5 - (x + h)) - (x^5 - x) }{h}$$

We can instead apply derivative rules to more easily calculate the derivative function.

The first rule is the power rule, which helps us take the derivative for functions with exponents.

$$f'(x) = r x^{r-1}$$

If our original function were $f(x) = x^2$ r would be 2, Its derivative would be $f'(x) = 2x^{2-1}$ or $f'(x) = 2x$ for short.

When we calculate the derivative of a function, we're really computing and solving a limit. We can use the properties of limits to break down the function into 3 different derivatives. We can then apply the power rule to each component:

The function for our curve above is $f(x) = -(x)^2 + 3x - 1$ and $f'(x) = -2x + 3$

**Exercise**

- Calculate the derivative of `f(x)=x**5` using pencil and paper. Use the derivative function to look up the slope at x=2 and assign to `slope_one`.
- Calculate the derivative of `f(x)=x**9` using pencil and paper. Use the derivative function to look up the slope at x=0 and assign to `slope_two`.

```python
x = 2
slope_one = 5*(x**4)
slope_one
```

    80

```python
x = 0
slope_two = 9*(x**8)
slope_two
```

    0

### Linearity of Differentiation

Another important set of rules is linearity of differentiation. Like with limit functions, we can separate mathematical terms that are being added or subtracted and differentiate them separately. Linearity of differentiation consists of 2 rules. First is the sum rule:

$$\frac {d}{dx} [f(x) + g(x)] = \frac {d}{dx} [f(x)] + \frac {d}{dx} [g(x)]$$

Second is the constant factor rule, which lets us pull out constants from the derivative:

$$\frac {d}{dx} [cf(x)] = c\frac {d}{dx} [f(x)]$$

By combining both of these rules, we can find the derivative for a wide range of functions. For example:

$$\frac {d}{dx} [-x^3 + x^2 ] =  \frac {d}{dx} [-x^3] + \frac {d}{dx} [x^2] = -3x^2 + 2x$$

**Exercise**

- Calculate the derivative of `f(x)=x**5−x` using pencil and paper. Use the derivative function to look up the slope at `x=1` and assign to `slope_three`.
- Calculate the derivative of `f(x)=x**3−x**2` using pencil and paper. Use the derivative function to look up the slope at `x=2` and assign to `slope_four`.

```python
x = 1
x_1 = 2

slope_three = 5*(x**4) - x
slope_four = 3*(x_1**2) - 2*x_1

print(slope_three, slope_four)
```

    4 8

### Finding Extreme Values
Earlier, we started by plotting the curve and visually observing where the critical points could be. We can aslo do so by finding the derivatives and setting them to 0.

In the following function:
$$f(x) = x^3 - x^2$$

the derivative is:

$$f'(x) = 3x^2 - 2x$$

We have to set the derivative to zero and find relative minimum and relative maximum by analyzing the slope around the point(s) to see if slope transitions from positive to negative or vice versa (when x is increased). We can use a [sign chart](http://www.rasmus.is/uk/t/F/Su53k02.htm) for this.

```python
derivative = "3x^2 - 2x"
critical_points = [0, 2/3]
rel_min = [2/3]
rel_max = [0]
```

Lesson-ML VI-Calculus for Machine Learning.ipynb


# Problem: Reflection unitaries

In the previous two problems, the output of a classical Boolean function is computed into an additional qubit. However, quantum computers can also compute Boolean functions into the phase of a quantum state, which we explore in this problem.

Suppose $f:\{0,1\}^n\rightarrow\{0,1\}$ is a Boolean function. For $\beta \in R$, define an $\textit{n}$-qubit unitary

$$
\large
\begin{equation*}
    U_f(\beta) = \sum_{x\in\{0,1\}^n} e^{i\beta f(x)}|x\rangle \langle x|
\tag{1}
\end{equation*}
$$
When $U_f$ is applied to computational basis state $|x\rangle$ it adds a phase $e^{i\beta}$ if $f(x) = 1$ and does  nothing otherwise.

Your goal in this problem is to implement $U_f$ for a variety of functions with $\textit{n}$= 3

\begin{align}
f_0(x_1x_2x_3)&=0,\quad f_1(x_1x_2x_3)=x_1x_2x_3,\quad f_2(x_1x_2x_3)=x_1x_2x_3+(1-x_1)(1-x_2)(1-x_3),\\\nonumber
f_3(x_1x_2x_3)&=x_1+x_2+x_3+x_1x_2x_3,\quad f_4(x_1x_2x_3)=x_1+x_2+x_3.
\tag{2}
\end{align}

Here, $f_m$ has m solutions to f(x) = 1. For instance,$f_2$ evaluates to 1 on the inputs x = 000 and x = 111.You may use $A \geq 0$ ancilla qubits in your implementation initialized in $∣0^A\rangle$ as long as they are returned to $∣0^A\rangle$ by the end of the computation. Two examples are shown in Fig.2 for n= 2 functions.

The "reflection unitaries" $U_f(\beta)$ are useful in some quantum algorithms. In particular, the famous Grover search algorithm uses these reflections to outperform classical search algorithms for unstructured lists.

    $f_m$ for m = 0

```python

# Importing the qiskit module
from qiskit import *
from qiskit.circuit import Parameter
params = [Parameter('beta')]

q0 =    3 # number of required qubits
a0 =    0 # number of ancilla qubit

# Defining Quantum Circuit with the given circuits
def Circuit_0(q,a):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########
    params = [Parameter('beta')]
    Circ.p(params[0], 0)
    Circ.p(-params[0], 0)

    ########################################

    # Uncomment to draw quantum circuit
   # display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_0 = Circuit_0(q0,a0)
```

    $f_m$ for m = 1

```python
# Importing the qiskit module
from qiskit import *
from qiskit.circuit.library import CU1Gate
from qiskit.circuit.library.standard_gates import XGate
crz = CU1Gate(Parameter('beta'))
c3x_gate = XGate().control(3)
q1 =    3 # number of required qubits
a1 =    1 # number of ancilla qubit

# Defining Quantum Circuit with the given circuits
def Circuit_1(q,a):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q, 'input')

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q)

    ##### Create you circuit below #########
    Circ.append(c3x_gate,[Q[0],Q[1],Q[2],A[0]])
    Circ.p(params[0], A[0])
    Circ.append(c3x_gate,[Q[0],Q[1],Q[2],A[0]])

    ########################################

    # Uncomment to draw quantum circuit
    display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_1 = Circuit_1(q1,a1)
```

    $f_m$ for m = 2

```python
# Importing the qiskit module
from qiskit import *

q2 =    3 # number of required qubits
a2 =  4  # number of ancilla qubit

# Defining Quantum Circuit with the given circuits
def Circuit_2(q,a):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)

    ##### Create you circuit below #########
    Circ.append(c3x_gate,[Q[0],Q[1],Q[2],A[3]])
    Circ.x(0)
    Circ.cx(0,A[0])
    Circ.x(0)
    Circ.x(1)
    Circ.cx(1,A[1])
    Circ.x(1)
    Circ.x(2)
    Circ.cx(2,A[2])
    Circ.x(2)
    Circ.append(c3x_gate,[A[0],A[1],A[2],A[3]])
    #Circ.append(crz,[A[3],Q[0]])
    Circ.p(params[0], A[3])
    #Circ.append(crz,[A[3],Q[1]])
    #Circ.append(crz,[A[3],Q[2]])
    Circ.append(c3x_gate,[A[0],A[1],A[2],A[3]])
    Circ.x(2)
    Circ.cx(2,A[2])
    Circ.x(2)
    Circ.x(1)
    Circ.cx(1,A[1])
    Circ.x(1)
    Circ.x(0)
    Circ.cx(0,A[0])
    Circ.x(0)
    Circ.append(c3x_gate,[Q[0],Q[1],Q[2],A[3]])

    ########################################

    # Uncomment to draw quantum circuit
    display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_2 = Circuit_2(q2,a2)
```

    $f_m$ for m = 3

```python
# Importing the qiskit module
from qiskit import *

q3 =   3  # number of required qubits
a3 =   2 # number of ancilla qubit

# Defining Quantum Circuit with the given circuits
def Circuit_3(q,a):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)

    ##### Create you circuit below #########
    Circ.append(c3x_gate,[0,1,2,A[0]])
    Circ.cx(A[0],A[1])
    Circ.cx(0,A[1])
    Circ.cx(1,A[1])
    Circ.cx(2,A[1])
    #Circ.append(crz,[A[1],Q[2]])
    Circ.p(params[0], A[1])
    #Circ.append(crz,[A[1],Q[1]])
    #Circ.append(crz,[A[1],Q[2]])
    Circ.cx(2,A[1])
    Circ.cx(1,A[1])
    Circ.cx(0,A[1])
    Circ.cx(A[0],A[1])
    Circ.append(c3x_gate,[0,1,2,A[0]])

    ########################################

    # Uncomment to draw quantum circuit
    display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_3 = Circuit_3(q3,a3)
```

    $f_m$ for m = 4

```python
# Importing the qiskit module
from qiskit import *

q4 =   3  # number of required qubits
a4 =  1  # number of ancilla qubit

# Defining Quantum Circuit with the given circuits
def Circuit_4(q,a):
    if a != 0:
        # initiating required and ancilla qubits
        Q = QuantumRegister(q, 'input')
        A = QuantumRegister(a)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)
    else:
        # initiating required qubits
        Q = QuantumRegister(q)

        # creating circuit with above qubits
        Circ = QuantumCircuit(Q,A)

    ##### Create you circuit below #########
    Circ.cx(0,A[0])
    Circ.cx(1,A[0])
    Circ.cx(2,A[0])
    #Circ.append(crz,[A[0],Q[0]])
    Circ.p(params[0], A[0])
    #Circ.append(crz,[A[0],Q[1]])
    #Circ.append(crz,[A[0],Q[2]])
    Circ.cx(2,A[0])
    Circ.cx(1,A[0])
    Circ.cx(0,A[0])

    ########################################

    # Uncomment to draw quantum circuit
    display(Circ.draw('mpl'))

    # Transpiling the circuit into u, cnot
    Circ = transpile(Circ, basis_gates=['u3','cx'])

    # Uncomment to draw transpiled circuit
#     display(Circ.draw('mpl'))

    return Circ

qc_4 = Circuit_4(q4,a4)
```

### Grader

```python
all_circuits = [qc_0, qc_1, qc_2, qc_3, qc_4]
from qc_grader import grade_ex3
import time
start_time = time.time()
grade_ex3(all_circuits)
print("--- %s seconds ---" % (time.time() - start_time))
```

    Checking for circuit 0 ...
    Checking for circuit 1 ...
    Checking for circuit 2 ...
    Checking for circuit 3 ...
    Checking for circuit 4 ...
    Grading your answer for ex3. Please wait...

    Congratulations 🎉! Your answer is correct.
    Your cost is 2632.
    Feel free to submit your answer.

    --- 13.577566385269165 seconds ---

# Circuit verification and Cost Metric

$\textbf{Costing metric:}$ For the $m^{th}$ circuit, let $G_m$ be the number of 2-qubit gates, $D_m$ the depth,  and $A_m$ the number of ancillas. Your solution quality will be judged on the following cost metric:
$$\text{Cost} = \sum_{m=0}^4 10G_m + D_m(n+A_m)$$ .
The reasoning behind this metric relates to the error rates in quantum circuits. Notice that $D_m(n+A_m)$ is roughly the spacetime volume of the circuit (the collective number of timesteps qubits participating in thecircuit must be kept alive). Also, two-qubit gates are roughly (optimistically) 10 times as likely to fail as idling qubits. Thus, $Cost_n$ is roughly proportional to error rate of all of your circuits combined.

### Submission for evaluation

```python
## Uncomment and run the following code to submit
#from qc_grader import submit_ex3

#submit_ex3(all_circuits)
```

```python

```


# Goldbach conjecture

```python
import pybryt
```

# helper function

In this reference implementation we mark many values that we expect a good solution *not* to have. Therefore, we use this helper function simplifies the notation.

```python
def notify_when_value_exists(value, message_when_exists, message_when_missing = None):
    v = pybryt.Value(value)
    n = ~v
    n.success_message = message_when_missing
    n.failure_message = message_when_exists
    return(n)
```

The function above can be written in a concise way but it might be less obvious to understand why it works:

```python
def notify_when_value_exists(value, message_when_exists, message_when_missing = None):
    return (~pybryt.Value(value,
        failure_message = message_when_missing,
        success_message = message_when_exists)
```

The strong Goldbach conjecture states that every even integer $\geq$ 4 is a sum of two prime numbers. Proving this conjecture is an open problem in mathematics for almost 300 years. In this exercise you are asked to write a function `check_goldbach_conjecture(n, primes)` which receives an integer `n` and a set of primes and returns `True` if `n` is a sum of two primes from the set or `False` otherwise. Try to make your solution efficient in terms of computational complexity and memory usage.

```python
def check_goldbach_conjecture(n, primes):
    """
    Checks if n can be written as a sum of two elements from the set primes.

    Args:
        n (``int``): an integer expected to be even greater than 4
        primes (``set`` of ``int``): a set of prime numbers

    Returns:
        ``bool``: True of n is a sum of two primes
    """
    relevant_primes_set={p for p in primes if p<n}
    notify_when_value_exists(
        relevant_primes_set,
        "ADVICE: it is good that you are looking only at primes smaller than n but to save memory you can do that without creating a new set"
    )

    notify_when_value_exists(sorted(primes), "ADVICE: sorting the list of primes only increases the complexity of the solution")

    for prime in primes:
        if n - prime in relevant_primes_set:
            return True
    return False
```

```python
from sympy.ntheory.generate import primerange
primes = set(primerange(50,1000))
n = 116
check_goldbach_conjecture(n, primes)

notify_when_value_exists(71 + 73, "ADVICE: instead of going over all primes and checking if they sum to n you can check if n-p is in the list of primes")

notify_when_value_exists(n - 997, "ADVICE: if the prime is bigger then n it cannot be a member of a pair that sum to n")

notify_when_value_exists(35, "ADVICE: the program checks if 35 is prime (35 is just an example), try to see if this can be avoided")

```

    pybryt.NotAnnotation


# Linear Regression
## The Solution Space
** November 2017 **

** Andrew Riberio @ [AndrewRib.com](http://www.andrewrib.com) **

In this notebook we will explore the solution space of linear regression with the mean squared error.

** Note: ** This notebook contains interactive elements and certain latex snippets that will not render in github markdown. You must run this notebook on your local Jupyter notebook environment.

## Libraries

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
import sklearn as sk
import sklearn.datasets as skd
from IPython.display import display
from sympy import MatrixSymbol, Matrix
from numpy.linalg import inv
from ipywidgets import interact, interactive, fixed, interact_manual
from mpl_toolkits.mplot3d import Axes3D

sp.init_printing(order='rev-lex',use_latex='mathjax')
```

## Linear Regression via Linear Least Squares

Let's begin with an interactive plot that will allow you to explore the relationship between:
* **The Regression Line**
    * Slope: The angle of the line.
    * Y-Intercept: Where the line intercepts the y axis.
    * Predicted Value = Slope * Dependent Variable + Y-Intercept

* **Dataset**
    * Generating function: Our dataset is synthetically generated by sampling a linear function, then adding noise.
    * Noise: a certain ammount of noise will be added to each sampling of the generating function.

* **Mean Squared Error**
    * How well our regression line fits the data. The smaller the value, the better the fit. See the definition of the calculation below. In machine learning terms, this is our cost function.

```python
def mse(yPredict,yActual):
    return np.square(yPredict.T-yActual.T).mean()

def h(slope,y_intercept):
    return lambda x:slope*x+y_intercept

def interactiveLine(slope,y_intercept,noise=50):
    X,Y = skd.make_regression(100,1,random_state=0, noise=noise)

    ys = np.apply_along_axis(h(slope,y_intercept), 0, X)

    plt.figure(figsize=(10,10))

    plt.title("Mean Squared Error: {0}".format(mse(ys,Y)),fontsize=15)
    plt.ylabel("Dependent Variable (Y)",fontsize=15)
    plt.xlabel("Independent Variable (X)",fontsize=15)
    plt.scatter(X,Y)
    plt.plot(X,ys)

    plt.show()

interact(interactiveLine, slope=(-150,150),y_intercept=(-100,100),noise=(0,150));
```

### Plotting the solution space.
We will now enumerate all linear functions with all combinations of integer slopes and y-intercepts within [-150,150]. We therefore have:

* The X Axis:    150*2=300 unique slopes
* The Y Axis:    150*2=300 unique y-intercepts
* The X*Y Plane: 300^2 = 9,0000 unique linear regression lines.

The $slope * yIntercept$ space creates a plane, the x and y axes, where each point represents a particular regression line.

The z axis will represent the MSE for that regression line on the data:

$$
\large MSE(\mathbf{x},\mathbf{y},h_\theta) = \frac{1}{m} \sum_{i=1}^m{(h_\theta(x^i) - y^i})^2  \tag{1} \\
$$

Where:
* $\theta_0$ = y_intercept
* $\theta_1$ = slope
* $h_\theta(\mathbf{x})= \mathbf{\theta_{0}}+ \mathbf{\theta_{1}}\mathbf{x}$

In this section we will use the following synthetic dataset.

```python
X,Y = skd.make_regression(100,1,random_state=0, noise=50)

plt.scatter(X,Y)
plt.show()
```

```python
# This can be done much more efficiently with matrix algebra, but this is the clearest way of doing this.
# Enumerate over the entire space of linear functions within the ranges.
# This cell will take a while to compute.

# Uses h(slope,yInter)
def enumFnsAndMSE(possibleSlopes,possibleYInter,X_Data,Y_Data,scaleFactor=1):
    errorOverLines = []

    for slope in possibleSlopes:
        row = []

        for yInter in possibleYInter:
            lFN = h(slope,yInter)
            regressionYs = np.apply_along_axis(lFN, 0, X_Data)
            row.append( mse(regressionYs,Y_Data)/scaleFactor )

        errorOverLines.append(row)

    return errorOverLines

possibleSlopes = range(-150,150)
possibleYInter = range(-150,150)

errorOverLines = enumFnsAndMSE(possibleSlopes,possibleYInter,X,Y,600)
```

```python
# Plot figure
xx, yy = np.mgrid[-150:150, -150:150]

fig = plt.figure(figsize=(14,14))
ax = fig.gca(projection='3d',facecolor='gray')

ax.set_xlabel('Slope',fontSize=16)
ax.set_ylabel('Y-Intercept',fontSize=16)
ax.set_zlabel('Mean Squared Error',fontSize=16)

ax.plot_surface(xx, yy, errorOverLines, cmap=plt.cm.Reds, linewidth=0.2)
plt.show()
```

The plot you see above is characteristic of convex optimization problems. If we put a ball anywhere on this surface, it will roll down to the bottom AKA the global minimum. A ball with no force applied will follow the steepest path -- the path where the gradient is largest. If we did apply some sideways force, it will roll across the sides until it settles to the bottom -- it may jump out of the bottom a few times untill it loses all of its kenetic energy. This method of using [momentum is actually usefull in some non-convex optimization problems, because it forces us to explore a wider area](https://distill.pub/2017/momentum/).

I'd like to make something clear that wasn't made so clear to me when I was first learning about convex optimization. Why do we care about things like a ball rolling down the slopes? The figure you see above is a plot of all possible solutions within some precision ( i.e. we had integer ranges for our possible slopes and y-intercepts, we could have had more precise floating point ranges ). Because this is a toy problem on a toy dataset, enumerating all possible solutions doesn't take too much time. When we start working with real world data with millions of datapoints, and much higher dimensions (i.e. polynomial regression) enumerating the entire solution space becomes combinatorially explosive. Therefore, in the absense of a closed-form solution, we are concerned with starting at some point, some solution, and making it better until we arrive at a solution we can't make better. Focusing on making a solution better and better until we can't is computationally cheap, expecially in the case of convex optimizaiton.

In order to formalize this notion of the motion of a ball rolling around this space of solutions, we must introduce the notion of the gradient. We will explore this in the gradient descent notebook.

### Producing Optimal Linear Regression Lines
There are two primary optimization methods to produce a line that minimizes the MSE (as explored above):
#### 1. Itterative Optimizaiton Methods
Itterative methods behave simmilarly to how you adjusted to the parameters above to get the best fitting line: we adjust the slope and bias so that it keeps getting smaller. Because linear regression is a convex optimization problem, we are always garenteed that if we follow this gradient, the value will always get smaller until it reaches it's minimum value -- which is garenteed to be a global minimum in the case of convex optimization.

Examples:
    * Gradient Descent
    * Newton's Method
    * Exhaustive Search
#### 2. Closed-Form Methods
In some simpler optimization methods, as in linear least squares, we can actually derive the optimal solution analytically -- where we solve an equation and get the optimal result.

Examples:
    * Normal Equations

** We will explore Gradient Descent and the Normal Equations in other notebooks. **

### Example - Exhaustive Search
When plotting the solition space above, we performed an exhaustive search within some precision -- being the intervals of the candidate slopes and y-intercepts. Let's use this and plot which line best fit our data in the exhaustive search.

```python
def findMinimum(eol):
    rowLen = len(eol)
    colLen = len(eol[0])

    minVal = 100000000
    minCord = []

    for r_i in range(rowLen):
        for c_i in range(colLen):
            if(eol[r_i][c_i] < minVal):
                minVal = eol[r_i][c_i]
                # We split the range in half to account for the negative values.
                minCord = [r_i-rowLen/2,c_i-colLen/2]

    return [minCord,minVal]

bestLine = findMinimum(errorOverLines)
minSlope,minIntercept = findMinimum(errorOverLines)[0]
ys = np.apply_along_axis(h(minSlope,minIntercept), 0, X)

plt.scatter(X,Y)
plt.plot(X,ys)

plt.show()
print("MSE of line: {0}".format(bestLine[1]))
```

Notebooks/Linear Regression - The Solution Space.ipynb


# 01 - Introduction to seismic modelling

This notebook is the first in a series of tutorials highlighting various aspects of seismic inversion based on Devito operators. In this first example we aim to highlight the core ideas behind seismic modelling, where we create a numerical model that captures the processes involved in a seismic survey. This forward model will then form the basis for further tutorials on the implementation of inversion processes using Devito operators.

## Modelling workflow

The core process we are aiming to model is a seismic survey, which consists of two main components:

- **Source** - A source is positioned at a single or a few physical locations where artificial pressure is injected into the domain we want to model. In the case of land survey, it is usually dynamite blowing up at a given location, or a vibroseis (a vibrating engine generating continuous sound waves). For a marine survey, the source is an air gun sending a bubble of compressed air into the water that will expand and generate a seismic wave.
- **Receiver** - A set of microphones or hydrophones are used to measure the resulting wave and create a set of measurements called a *Shot Record*. These measurements are recorded at multiple locations, and usually at the surface of the domain or at the bottom of the ocean in some marine cases.

In order to create a numerical model of a seismic survey, we need to solve the wave equation and implement source and receiver interpolation to inject the source and record the seismic wave at sparse point locations in the grid.

## The acoustic seismic wave equation
The acoustic wave equation for the square slowness $m$, defined as $m=\frac{1}{c^2}$, where $c$ is the speed of sound in the given physical media, and a source $q$ is given by:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) = q \ \text{in } \Omega \\
 &u(.,t=0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

with the zero initial conditions to guarantee unicity of the solution.
The boundary conditions are Dirichlet conditions:
\begin{equation}
 u(x,t)|_\delta\Omega = 0
\end{equation}

where $\delta\Omega$ is the surface of the boundary of the model $\Omega$.

# Finite domains

The last piece of the puzzle is the computational limitation. In the field, the seismic wave propagates in every direction to an "infinite" distance. However, solving the wave equation in a mathematically/discrete infinite domain is not feasible. In order to compensate, Absorbing Boundary Conditions (ABC) or Perfectly Matched Layers (PML) are required to mimic an infinite domain. These two methods allow to approximate an infinite media by damping and absorbing the waves at the limit of the domain to avoid reflections.

The simplest of these methods is the absorbing damping mask. The core idea is to extend the physical domain and to add a Sponge mask in this extension that will absorb the incident waves. The acoustic wave equation with this damping mask can be rewritten as:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) + \eta \frac{d u(x,t)}{dt}=q  \ \text{in } \Omega \\
 &u(.,0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

where $\eta$ is the damping mask equal to $0$ inside the physical domain and increasing inside the sponge layer. Multiple choice of profile can be chosen for $\eta$ from linear to exponential.

# Seismic modelling with devito

We describe here a step by step setup of seismic modelling with Devito in a simple 2D case. We will create a physical model of our domain and define a single source and an according set of receivers to model for the forward model. But first, we initialize some basic utilities.

```python
#NBVAL_IGNORE_OUTPUT
# Adding ignore due to (probably an np notebook magic) bug
import numpy as np
%matplotlib inline
```

## Define the physical problem

The first step is to define the physical model:

- What are the physical dimensions of interest
- What is the velocity profile of this physical domain

We will create a simple velocity model here by hand for demonstration purposes. This model essentially consists of two layers, each with a different velocity: $1.5km/s$ in the top layer and $2.5km/s$ in the bottom layer. We will use this simple model a lot in the following tutorials, so we will rely on a utility function to create it again later.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Model, plot_velocity

# Define a physical size
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # What is the location of the top left corner. This is necessary to define
# the absolute location of the source and receivers

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:, :51] = 1.5
v[:, 51:] = 2.5

# With the velocity and model size defined, we can create the seismic model that
# encapsulates this properties. We also define the size of the absorbing layer as 10 grid points
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=2, nbl=10, bcs="damp")

plot_velocity(model)
```

# Acquisition geometry

To fully define our problem setup we also need to define the source that injects the wave to model and the set of receiver locations at which to sample the wavefield. The source time signature will be modelled using a Ricker wavelet defined as

\begin{equation}
  q(t) = (1-2\pi^2 f_0^2 (t - \frac{1}{f_0})^2 )e^{- \pi^2 f_0^2 (t - \frac{1}{f_0})}
\end{equation}

To fully define the source signature we first need to define the time duration for our model and the timestep size, which is dictated by the CFL condition and our grid spacing. Luckily, our `Model` utility provides us with the critical timestep size, so we can fully discretize our model time axis as an array:

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 1000.  # Simulation last 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing

time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

The source is positioned at a $20m$ depth and at the middle of the $x$ axis ($x_{src}=500m$), with a peak wavelet frequency of $10Hz$.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import RickerSource

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

# First, position source centrally in all dimensions, then set depth
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 20.  # Depth is 20m

# We can plot the time signature to see the wavelet
src.show()
```

Similarly to our source object, we can now define our receiver geometry as a symbol of type `Receiver`. It is worth noting here that both utility classes, `RickerSource` and `Receiver` are thin wrappers around the Devito's `SparseTimeFunction` type, which encapsulates sparse point data and allows us to inject and interpolate values into and out of the computational grid. As we have already seen, both types provide a `.coordinates` property to define the position within the domain of all points encapsulated by that symbol.

In this example we will position receivers at the same depth as the source, every $10m$ along the x axis. The `rec.data` property will be initialized, but left empty, as we will compute the receiver readings during the simulation.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Receiver

# Create symbol for 101 receivers
rec = Receiver(name='rec', grid=model.grid, npoint=101, time_range=time_range)

# Prescribe even spacing for receivers along the x-axis
rec.coordinates.data[:, 0] = np.linspace(0, model.domain_size[0], num=101)
rec.coordinates.data[:, 1] = 20.  # Depth is 20m

# We can now show the source and receivers within our domain:
# Red dot: Source location
# Green dots: Receiver locations (every 4th point)
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])
```

# Finite-difference discretization

Devito is a finite-difference DSL that solves the discretized wave-equation on a Cartesian grid. The finite-difference approximation is derived from Taylor expansions of the continuous field after removing the error term.

## Time discretization

We only consider the second order time discretization for now. From the Taylor expansion, the second order discrete approximation of the second order time derivative is:
\begin{equation}
\begin{aligned}
 \frac{d^2 u(x,t)}{dt^2} = \frac{\mathbf{u}(\mathbf{x},\mathbf{t+\Delta t}) - 2 \mathbf{u}(\mathbf{x},\mathbf{t}) + \mathbf{u}(\mathbf{x},\mathbf{t-\Delta t})}{\mathbf{\Delta t}^2} + O(\mathbf{\Delta t}^2).
\end{aligned}
\end{equation}

where $\mathbf{u}$ is the discrete wavefield, $\mathbf{\Delta t}$ is the discrete
time-step (distance between two consecutive discrete time points) and $O(\mathbf{\Delta
  t}^2)$ is the discretization error term. The discretized approximation of the
second order time derivative is then given by dropping the error term. This derivative is represented in Devito by `u.dt2` where u is a `TimeFunction` object.

## Spatial discretization

We define the discrete Laplacian as the sum of the second order spatial
derivatives in the three dimensions:
\begin{equation}
\begin{aligned}
\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t})= \sum_{j=1}^{j=\frac{k}{2}} \Bigg[\alpha_j \Bigg(&
\mathbf{u}(\mathbf{x+jdx},\mathbf{y},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x-jdx},\mathbf{y},\mathbf{z},\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y+jdy},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y-jdy},\mathbf{z}\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z+jdz},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z-jdz},\mathbf{t})\Bigg) \Bigg] + \\
&3\alpha_0 \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}).
\end{aligned}
\end{equation}

This derivative is represented in Devito by `u.laplace` where u is a `TimeFunction` object.

## Wave equation

With the space and time discretization defined, we can fully discretize the wave-equation with the combination of time and space discretizations and obtain the following second order in time and $k^{th}$ order in space discrete stencil to update one grid point at position $\mathbf{x}, \mathbf{y},\mathbf{z}$ at time $\mathbf{t}$, i.e.
\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

```python
# In order to represent the wavefield u and the square slowness we need symbolic objects
# corresponding to time-space-varying field (u, TimeFunction) and
# space-varying field (m, Function)
from devito import TimeFunction

# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=2)

# We can now write the PDE
pde = model.m * u.dt2 - u.laplace + model.damp * u.dt

# The PDE representation is as on paper
pde
```

$\displaystyle \operatorname{damp}{\left(x,y \right)} \frac{\partial}{\partial t} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial x^{2}} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial y^{2}} u{\left(t,x,y \right)} + \frac{\frac{\partial^{2}}{\partial t^{2}} u{\left(t,x,y \right)}}{\operatorname{vp}^{2}{\left(x,y \right)}}$

```python
# This discrete PDE can be solved in a time-marching way updating u(t+dt) from the previous time step
# Devito as a shortcut for u(t+dt) which is u.forward. We can then rewrite the PDE as
# a time marching updating equation known as a stencil using customized SymPy functions
from devito import Eq, solve

stencil = Eq(u.forward, solve(pde, u.forward))
```

# Source injection and receiver interpolation

With a numerical scheme to solve the homogenous wave equation, we need to add the source to introduce seismic waves and to implement the measurement operator, and interpolation operator. This operation is linked to the discrete scheme and needs to be done at the proper time step. The semi-discretized in time wave equation with a source reads:

\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

It shows that in order to update $\mathbf{u}$ at time $\mathbf{t+\Delta t}$ we have to inject the value of the source term $\mathbf{q}$ of time $\mathbf{t}$. In Devito, it corresponds the update of $u$ at index $t+1$ (t = time implicitly) with the source of time $t$.
On the receiver side, the problem is either as it only requires to record the data at the given time step $t$ for the receiver at time $time=t$.

```python
# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)

# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=u.forward)
```

# Devito operator and solve
After constructing all the necessary expressions for updating the wavefield, injecting the source term and interpolating onto the receiver points, we can now create the Devito operator that will generate the C code at runtime. When creating the operator, Devito's two optimization engines will log which performance optimizations have been performed:
* **DSE:** The Devito Symbolics Engine will attempt to reduce the number of operations required by the kernel.
* **DLE:** The Devito Loop Engine will perform various loop-level optimizations to improve runtime performance.

**Note**: The argument `subs=model.spacing_map` causes the operator to substitute values for our current grid spacing into the expressions before code generation. This reduces the number of floating point operations executed by the kernel by pre-evaluating certain coefficients.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

op = Operator([stencil] + src_term + rec_term, subs=model.spacing_map)
```

Now we can execute the create operator for a number of timesteps. We specify the number of timesteps to compute with the keyword `time` and the timestep size with `dt`.

```python
#NBVAL_IGNORE_OUTPUT
op(time=time_range.num-1, dt=model.critical_dt)
```

    Operator `Kernel` run in 0.01 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=0.0032300000000000237, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section1', rank=None),
                         PerfEntry(time=1.4000000000000003e-05, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section2', rank=None),
                         PerfEntry(time=0.0006070000000000032, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

After running our operator kernel, the data associated with the receiver symbol `rec.data` has now been populated due to the interpolation expression we inserted into the operator. This allows us the visualize the shot record:

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

plot_shotrecord(rec.data, model, t0, tn)
```

```python
assert np.isclose(np.linalg.norm(rec.data), 370, rtol=1)
```