## GeostatsPy: Basic Univariate Distribution Transformations  Subsurface Data Analytics in Python

### Michael Pyrcz, Associate Professor, University of Texas at Austin

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Basic Univariate Distribution Transformations  Subsurface Data Analytics in Python with GeostatsPy

Here's a simple workflow with some basic univariate distribution transformations for subsurface modeling workflows. This should help you get started data transformations.

#### Data Distribution Transformations

Why?:
* variable has expected shape / correcting for too few data
* a specific distribution assumption is required
* correct for outliers

How?:

There are a variety of transformations. In general we are transforming the values from the cumulative distribution function (CDF), $F_{X}$, to a new CDF , $G_{Y}$. This can be generalized with the quantile - quantile transformation applied to all the sample data:

* The forward transform:

\begin{equation}
Y = G_{Y}^{-1}(F_{X}(X))
\end{equation}

* The reverse transform:

\begin{equation}
X = F_{X}^{-1}(G_{Y}(Y))
\end{equation}

This may be applied to any data, nonparametric or samples from a parametric distribution. We just need to be able to map from one distribution to another through percentiles, so it is a:

* Rank preserving transform

We will cover three examples including:

1. Distribution rescaling
2. Normal score transform

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383/Examples")             # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv')     # load our data table (wrong name!)
```

It worked, we loaded our file into our DataFrame called 'df'. But how do you really know that it worked? Visualizing the DataFrame would be useful and we already leard about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
print(df.iloc[0:5,:])                   # display first 4 samples in the table as a preview
df.head(n=13)                           # we could also use this command for a table preview
```

         X    Y  Facies  Porosity       Perm
    0  100  900       1  0.115359   5.736104
    1  100  800       1  0.136425  17.211462
    2  100  600       1  0.135810  43.724752
    3  100  500       0  0.094414   1.609942
    4  100  100       0  0.113049  10.886001

      X
      Y
      Facies
      Porosity
      Perm

      0
      100
      900
      1
      0.115359
      5.736104

      1
      100
      800
      1
      0.136425
      17.211462

      2
      100
      600
      1
      0.135810
      43.724752

      3
      100
      500
      0
      0.094414
      1.609942

      4
      100
      100
      0
      0.113049
      10.886001

      5
      200
      800
      1
      0.154648
      106.491795

      6
      200
      700
      1
      0.153113
      140.976324

      7
      200
      500
      1
      0.126167
      12.548074

      8
      200
      400
      0
      0.094750
      1.208561

      9
      200
      100
      1
      0.150961
      44.687430

      10
      300
      800
      1
      0.199227
      1079.709291

      11
      300
      700
      1
      0.154220
      179.491695

      12
      300
      500
      1
      0.137502
      38.164911

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 2 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      289.0
      475.813149
      254.277530
      0.000000
      300.000000
      430.000000
      670.000000
      990.000000

      Y
      289.0
      529.692042
      300.895374
      9.000000
      269.000000
      549.000000
      819.000000
      999.000000

      Facies
      289.0
      0.813149
      0.390468
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      289.0
      0.134744
      0.037745
      0.058548
      0.106318
      0.126167
      0.154220
      0.228790

      Perm
      289.0
      207.832368
      559.359350
      0.075819
      3.634086
      14.908970
      71.454424
      5308.842566

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;           # range of porosity values
cmap = plt.cm.plasma                    # color map
```

Let's try out locmap. This is a reimplementation of GSLIB's locmap program that uses matplotlib. I hope you find it simpler than matplotlib, if you want to get more advanced and build custom plots lock at the source. If you improve it, send me the new code. Any help is appreciated. To see the parameters, just type the command name:

```python
GSLIB.locmap
```

    <function geostatspy.GSLIB.locmap(df, xcol, ycol, vcol, xmin, xmax, ymin, ymax, vmin, vmax, title, xlabel, ylabel, vlabel, cmap, fig_name)>

Now we can populate the plotting parameters and visualize the porosity data.

```python
GSLIB.locmap(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap,'locmap_Porosity')
```

#### Decluster the Data

Look carefully, and you'll notice the the spatial samples are more dense in the high porosity regions and less dense in the low porosity regions. There is preferential sampling. We cannot use the naive statistics to represent this region. We have to correct for the clustering of the samples in the high porosity regions.

Let's try cell declustering. We can interpret that we will want to minimize the declustering mean and that a cell size of between 100 - 200m is likely a good cell size, this is 'an ocular' estimate of the largest average spacing in the sparsely sampled regions.

Let's check out the declus program reimplimented from GSLIB.

```python
geostats.declus
```

    <function geostatspy.geostats.declus(df, xcol, ycol, vcol, iminmax, noff, ncell, cmin, cmax)>

We can now populate the parameters. We will run a very wide range of cell sizes, from 10m to 2,000m ('cmin' and 'cmax') and take the cell size that minimizes the declustered mean ('iminmax' = 1 minimize, and = 0 maximize). Multiple offsets (number of these is 'noff') uses multiple grid origins and averages the results to remove sensitivity to grid position.  The ncell is the number of cell sizes.

The output from this program is:

* wts - an array with the weigths for each data (they sum to the number of data, 1 indicates nominal weight)
* cell_sizes - an array with the considered cell sizes
* dmeans - de an

```python
wts, cell_sizes, dmeans = geostats.declus(df,'X','Y','Porosity',iminmax = 1, noff= 10, ncell=100,cmin=10,cmax=2000)
df['Wts'] = wts                            # add weights to the sample data DataFrame
df.head()                                  # preview to check the sample data DataFrame
```

    There are 289 data with:
       mean of      0.13474387540138408
       min and max  0.058547873 and 0.228790002
       standard dev 0.03767982164385207

      X
      Y
      Facies
      Porosity
      Perm
      Wts

      0
      100
      900
      1
      0.115359
      5.736104
      3.064286

      1
      100
      800
      1
      0.136425
      17.211462
      1.076608

      2
      100
      600
      1
      0.135810
      43.724752
      0.997239

      3
      100
      500
      0
      0.094414
      1.609942
      1.165119

      4
      100
      100
      0
      0.113049
      10.886001
      1.224164

Let's look at the location map of the weights.

```python
GSLIB.locmap(df,'X','Y','Wts',xmin,xmax,ymin,ymax,0.5,2.5,'Well Data Weights','X(m)','Y(m)','Weights',cmap,'locmap_Weights')
```

Does it look correct?  See the weight varies with local sampling density?

Now let's add the distribution of the weights and the naive and declustered porosity distributions. You should see the histogram bars adjusted by the weights. Also note the change in the mean due to the weights. There is a significant change.

```python
plt.subplot(121)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(122)
GSLIB.hist_st(df['Porosity'],0.05,0.25,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Porosity",title="Declustered Porosity")
plt.ylim(0.0,40)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.5, wspace=0.2, hspace=0.2)
plt.show()
```

We are now ready to do some data transformations.

#### Distribution Rescaling

Distribution rescaling can be thought of as shifting, and stretching and squeezing a distribution. The common method is known as affine correction:

\begin{equation}
y = \frac{\sigma_y}{\sigma_x}(y - \overline{y}) + \overline{x}
\end{equation}

We can see that the metho first centers the distribution, the rescales the dispersion based on the ratio of the new standard deviation to the original standard deviation and then shifts the distribution to centered on the target mean.

We have a function in GeostatsPy to do the affine correction of the distribution.

```python
GSLIB.affine
```

    <function geostatspy.GSLIB.affine(array, tmean, tstdev)>

We just need to specify the new target mean and variance.  Let's make 2 new rescaled distributions and then plot the results.

```python
por_original = df['Porosity'].values      # extract the pororsity data as a ndarray
por1 = GSLIB.affine(por_original,0.0,1.0) # rescale the porosity to have a standard distribution
df['standPor'] = por1

plt.subplot(221)
GSLIB.hist_st(df['Porosity'],0.05,0.25,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Porosity (fraction)",title="Declustered Porosity")
plt.ylim(0.0,40)

plt.subplot(222)
GSLIB.hist_st(df['standPor'],-3.0,3.0,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Porosity Standardized",title="Declustered Porosity Standardized")
plt.ylim(0.0,40)

plt.subplot(223)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(224)
GSLIB.locmap_st(df,'X','Y','standPor',xmin,xmax,ymin,ymax,-3,3,'Well Data - Porosity - Standardized','X(m)','Y(m)','Standardized Porosity',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.5, wspace=0.2, hspace=0.2)
plt.show()
```

Notice that I did not say standard normal? A standard distribution has a mean of 0.0 and standard deviation of 1.0.  The rescaling does not change the distribution shape; therefore, a non-normal (non-Gaussian) distribution cannot become normal just by rescaling.  We'll cover that method in a bit.

Also, notice that the shape is the same and the location maps look exactly the same? By adjusting the minimum and maximum values in the histogram x-axis and the location map color bar, we made them look unchanged! There are minor differences in bars due to the precise locations of the bin boundaries.

Let's try a minor adjustment as in the case of correcting the porosity from well logs to a more reliable mean and standard deviation or in the case to produce multiple scenarios of the porosity distribution (more on these uncertainty methods later).

```python
por_original = df['Porosity'].values      # extract the pororsity data as a ndarray
por2 = GSLIB.affine(por_original,0.11,0.02) # rescale the porosity to have a standard distribution
df['adjustedPor'] = por2

plt.subplot(221)
GSLIB.hist_st(df['Porosity'],0.05,0.25,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Porosity (fraction)",title="Declustered Porosity")
plt.ylim(0.0,60)

plt.subplot(222)
GSLIB.hist_st(df['adjustedPor'],0.05,0.25,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Porosity Corrected",title="Declustered Porosity Corrected")
plt.ylim(0.0,60)

plt.subplot(223)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(224)
GSLIB.locmap_st(df,'X','Y','adjustedPor',xmin,xmax,ymin,ymax,0.05,0.25,'Well Data - Porosity - Corrected','X(m)','Y(m)','Corrected Porosity',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.5, wspace=0.2, hspace=0.2)
plt.show()
```

#### Normal Score Transform / Gaussian Anamorphosis

We showed that the correction of the mean to 0.0 and standard deviation to 1.0 with affine correction does not change the shape; therefore, does not make a Gaussian distributed property. For many statistic / geostatistical methods the assumption of Gaussian distributed is required. We need normal score transforms in many subsurface modeling workflows.

Let's check out the GSLIB NSCORE program translated to Python in GeostatsPy.

```python
geostats.nscore
```

    <function geostatspy.geostats.nscore(df, vcol, wcol=0, ismooth=0, dfsmooth=0, smcol=0, smwcol=0)>

The inputs are primarily the DataFrame, the variable and the data weight columns ('df', 'vcol' and 'wcol'). The remainder of the variables are for the use of a reference distribution. When would you use a reference distribution? This would be the case when you have too few data to perform a reliable transformation and use analog information to inform a more complete distribution to support the transformation.

As you can see the inputs from weights column ('wcol') have defaults of 0. You can run the function omitting these (e.g. just DataFrame and variable column etc.).

The output form the program include the transformed data, and the trasformation table (discretized values in original and associated Gaussian space).

```python
ns_por,trans_vr,trans_ns = geostats.nscore(df,'Porosity','Wts')
df['NPor'] = ns_por

plt.subplot(221)
GSLIB.hist_st(df['Porosity'],0.05,0.25,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Porosity (fraction)",title="Declustered Porosity")
plt.ylim(0.0,40)

plt.subplot(222)
GSLIB.hist_st(df['NPor'],-3.0,3.0,log=False,cumul=False,bins=30,weights=df['Wts'],xlabel="Normal Scores Porosity",title="Declustered Normal Scores Porosity")
plt.ylim(0.0,40)

plt.subplot(223)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)

plt.subplot(224)
GSLIB.locmap_st(df,'X','Y','NPor',xmin,xmax,ymin,ymax,-3.0,3.0,'Well Data - Normal Scores Porosity','X(m)','Y(m)','Normal Scores Porosity',cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.5, wspace=0.2, hspace=0.2)
plt.show()
```

That is interesting! Why is the new distribution not perfectly Gaussian in shape? Because it is the declustered distribution of the data transformed to Gaussian. It accounts for the spatial bias in the sampling.

I'm not completely satified with the behavoir of the transformed data distribution at the tails.  The NSCORE programs does not have any tail extrapolation model as found with simulation methods. The transform at the tails is hard to do just from the data alone. When we get into simulation methods we'll check that out.

We should also visualize the transformation table.

```python
plt.subplot(111)
plt.scatter(trans_vr,trans_ns, c = "black", marker='o', alpha = 0.2, edgecolors = "none")
plt.xlabel('Porosity (%)')
plt.ylabel('Normal Score Transformed Porosity')
plt.title('Normal Score Transformed Porosity vs Untransformed Porosity p-p Plot')
plt.ylim(-4,4)
plt.xlim(0,.30)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

This is a q-q plot that maps the transform from our original distribution to the Gaussian distribution. Notice how the declustering weights have shift up the lowe quantiles as they received more weight.

As a final step we should check out the summary statistics of all the variants of porosity from our various data transformations.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      289.0
      4.758131e+02
      254.277530
      0.000000
      300.000000
      430.000000
      670.000000
      990.000000

      Y
      289.0
      5.296920e+02
      300.895374
      9.000000
      269.000000
      549.000000
      819.000000
      999.000000

      Facies
      289.0
      8.131488e-01
      0.390468
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      289.0
      1.347439e-01
      0.037745
      0.058548
      0.106318
      0.126167
      0.154220
      0.228790

      Perm
      289.0
      2.078324e+02
      559.359350
      0.075819
      3.634086
      14.908970
      71.454424
      5308.842566

      Wts
      289.0
      1.000000e+00
      0.639743
      0.281976
      0.670642
      0.789486
      1.174123
      3.984325

      standPor
      289.0
      1.600027e-16
      1.001735
      -2.022196
      -0.754414
      -0.227615
      0.516880
      2.495928

      adjustedPor
      289.0
      1.100000e-01
      0.020035
      0.069556
      0.094912
      0.105448
      0.120338
      0.159919

      NPor
      289.0
      3.876516e-01
      1.042739
      -2.336338
      -0.330268
      0.331970
      1.118825
      3.157063

#### Comments

This was a basic demonstration of data transformations. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


**This notebook is an exercise in the [Computer Vision](https://www.kaggle.com/learn/computer-vision) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/convolution-and-relu).**

---

# Introduction #

In this exercise, you'll work on building some intuition around feature extraction. First, we'll walk through the example we did in the tutorial again, but this time, with a kernel you choose yourself. We've mostly been working with images in this course, but what's behind all of the operations we're learning about is mathematics. So, we'll also take a look at how these feature maps can be represented instead as arrays of numbers and what effect convolution with a kernel will have on them.

Run the cell below to get started!

```python
# Setup feedback system
from learntools.core import binder
binder.bind(globals())
from learntools.computer_vision.ex2 import *

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt

plt.rc('figure', autolayout=True)
plt.rc('axes', labelweight='bold', labelsize='large',
       titleweight='bold', titlesize=18, titlepad=10)
plt.rc('image', cmap='magma')

tf.config.run_functions_eagerly(True)
```

# Apply Transformations #

The next few exercises walk through feature extraction just like the example in the tutorial. Run the following cell to load an image we'll use for the next few exercises.

```python
image_path = '../input/computer-vision-resources/car_illus.jpg'
image = tf.io.read_file(image_path)
image = tf.io.decode_jpeg(image, channels=1)
image = tf.image.resize(image, size=[400, 400])

img = tf.squeeze(image).numpy()
plt.figure(figsize=(6, 6))
plt.imshow(img, cmap='gray')
plt.axis('off')
plt.show();
```

You can run this cell to see some standard kernels used in image processing.

```python
import learntools.computer_vision.visiontools as visiontools
from learntools.computer_vision.visiontools import edge, bottom_sobel, emboss, sharpen

kernels = [edge, bottom_sobel, emboss, sharpen]
names = ["Edge Detect", "Bottom Sobel", "Emboss", "Sharpen"]

plt.figure(figsize=(12, 12))
for i, (kernel, name) in enumerate(zip(kernels, names)):
    plt.subplot(1, 4, i+1)
    visiontools.show_kernel(kernel)
    plt.title(name)
plt.tight_layout()
```

# 1) Define Kernel #

Use the next code cell to define a kernel. You have your choice of what kind of kernel to apply. One thing to keep in mind is that the *sum* of the numbers in the kernel determines how bright the final image is. Generally, you should try to keep the sum of the numbers between 0 and 1 (though that's not required for a correct answer).

In general, a kernel can have any number of rows and columns. For this exercise, let's use a $3 \times 3$ kernel, which often gives the best results. Define a kernel with `tf.constant`.

```python
# YOUR CODE HERE: Define a kernel with 3 rows and 3 columns.
kernel = tf.constant([
    #____,
    [-1, -1, -1],
    [-1, 8, -1],
    [-1, -1, -1]
])
# Uncomment to view kernel
# visiontools.show_kernel(kernel)

# Check your answer
q_1.check()
```

    <IPython.core.display.Javascript object>

Correct

```python
# Lines below will give you a hint or solution code
#q_1.hint()
#q_1.solution()
```

Now we'll do the first step of feature extraction, the filtering step. First run this cell to do some reformatting for TensorFlow.

```python
# Reformat for batch compatibility.
image = tf.image.convert_image_dtype(image, dtype=tf.float32)
image = tf.expand_dims(image, axis=0)
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

# 2) Apply Convolution #

Now we'll apply the kernel to the image by a convolution. The *layer* in Keras that does this is `layers.Conv2D`. What is the *backend function* in TensorFlow that performs the same operation?

```python
# YOUR CODE HERE: Give the TensorFlow convolution function (without arguments)
conv_fn = tf.nn.conv2d

# Check your answer
q_2.check()
```

    <IPython.core.display.Javascript object>

Correct

```python
# Lines below will give you a hint or solution code
#q_2.hint()
#q_2.solution()
```

Once you've got the correct answer, run this next cell to execute the convolution and see the result!

```python
image_filter = conv_fn(
    input=image,
    filters=kernel,
    strides=1, # or (1, 1)
    padding='SAME',
)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_filter)
)
plt.axis('off')
plt.show();
```

Can you see how the kernel you chose relates to the feature map it produced?

# 3) Apply ReLU #

Now detect the feature with the ReLU function. In Keras, you'll usually use this as the activation function in a `Conv2D` layer. What is the *backend function* in TensorFlow that does the same thing?

```python
# YOUR CODE HERE: Give the TensorFlow ReLU function (without arguments)
relu_fn = tf.nn.relu

# Check your answer
q_3.check()
```

    <IPython.core.display.Javascript object>

Correct

```python
# Lines below will give you a hint or solution code
#q_3.hint()
#q_3.solution()
```

Once you've got the solution, run this cell to detect the feature with ReLU and see the result!

The image you see below is the feature map produced by the kernel you chose. If you like, experiment with some of the other suggested kernels above, or, try to invent one that will extract a certain kind of feature.

```python
image_detect = relu_fn(image_filter)

plt.imshow(
    # Reformat for plotting
    tf.squeeze(image_detect)
)
plt.axis('off')
plt.show();
```

In the tutorial, our discussion of kernels and feature maps was mainly visual. We saw the effect of `Conv2D` and `ReLU` by observing how they transformed some example images.

But the operations in a convolutional network (like in all neural networks) are usually defined through mathematical functions, through a computation on numbers. In the next exercise, we'll take a moment to explore this point of view.

Let's start by defining a simple array to act as an image, and another array to act as the kernel. Run the following cell to see these arrays.

```python
# Sympy is a python library for symbolic mathematics. It has a nice
# pretty printer for matrices, which is all we'll use it for.
import sympy
sympy.init_printing()
from IPython.display import display

image = np.array([
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 1, 0, 1, 1, 1],
    [0, 1, 0, 0, 0, 0],
])

kernel = np.array([
    [1, -1],
    [1, -1],
])

display(sympy.Matrix(image))
display(sympy.Matrix(kernel))
# Reformat for Tensorflow
image = tf.cast(image, dtype=tf.float32)
image = tf.reshape(image, [1, *image.shape, 1])
kernel = tf.reshape(kernel, [*kernel.shape, 1, 1])
kernel = tf.cast(kernel, dtype=tf.float32)
```

    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left[\begin{matrix}0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 0 & 0 & 0\\0 & 1 & 0 & 1 & 1 & 1\\0 & 1 & 0 & 0 & 0 & 0\end{matrix}\right]$

    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left[\begin{matrix}1 & -1\\1 & -1\end{matrix}\right]$

# 4) Observe Convolution on a Numerical Matrix #

What do you see? The image is simply a long vertical line on the left and a short horizontal line on the lower right. What about the kernel? What effect do you think it will have on this image? After you've thought about it, run the next cell for the answer.

```python
# View the solution (Run this code cell to receive credit!)
q_4.check()
```

    <IPython.core.display.Javascript object>

Correct:

In the tutorial, we talked about how the pattern of positive numbers will tell you the kind of features the kernel will extract. This kernel has a vertical column of 1's, and so we would expect it to return features of vertical lines.

Now let's try it out. Run the next cell to apply convolution and ReLU to the image and display the result.

```python
image_filter = tf.nn.conv2d(
    input=image,
    filters=kernel,
    strides=1,
    padding='VALID',
)
image_detect = tf.nn.relu(image_filter)

# The first matrix is the image after convolution, and the second is
# the image after ReLU.
display(sympy.Matrix(tf.squeeze(image_filter).numpy()))
display(sympy.Matrix(tf.squeeze(image_detect).numpy()))
```

    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left[\begin{matrix}-2.0 & 2.0 & 0.0 & 0.0 & 0.0\\-2.0 & 2.0 & 0.0 & 0.0 & 0.0\\-2.0 & 2.0 & 0.0 & 0.0 & 0.0\\-2.0 & 2.0 & -1.0 & 0.0 & 0.0\\-2.0 & 2.0 & -1.0 & 0.0 & 0.0\end{matrix}\right]$

    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /opt/conda/lib/python3.7/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left[\begin{matrix}0.0 & 2.0 & 0.0 & 0.0 & 0.0\\0.0 & 2.0 & 0.0 & 0.0 & 0.0\\0.0 & 2.0 & 0.0 & 0.0 & 0.0\\0.0 & 2.0 & 0.0 & 0.0 & 0.0\\0.0 & 2.0 & 0.0 & 0.0 & 0.0\end{matrix}\right]$

Is the result what you expected?

# Conclusion #

In this lesson, you learned about the first two operations a convolutional classifier uses for feature extraction: **filtering** an image with a **convolution** and **detecting** the feature with the **rectified linear unit**.

# Keep Going #

Move on to [**Lesson 3**](https://www.kaggle.com/ryanholbrook/maximum-pooling) to learn the final operation: **condensing** the feature map with **maximum pooling**!

---

*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/196537) to chat with other Learners.*


# 监督学习 Supervised Learning

引入科学计算和绘图相关包

```python
import logging
from collections import namedtuple

import numpy as np
from pandas import DataFrame
from IPython.display import HTML
from numpy.linalg import inv
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import cross_val_score

# 引入绘图包
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import animation
from matplotlib import cm
from matplotlib.colors import LogNorm
import seaborn as sns
sns.set_style('whitegrid')
logging.basicConfig(level=logging.INFO)
%matplotlib inline
```

引入波士顿房价数据

```python
from sklearn.datasets import load_boston
boston = load_boston()
print(boston.DESCR)
```

    Boston House Prices dataset
    ===========================

    Notes
    ------
    Data Set Characteristics:

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive

        :Median Value (attribute 14) is usually the target

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    http://archive.ics.uci.edu/ml/datasets/Housing

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    **References**

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
       - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

我们仅使用RM（平均房间数）和Price（按1000美元计的房价中位数）两个指标。

绘制平均房间数和房价中位数的散点图

```python
df = DataFrame(data=boston.data, columns=boston.feature_names)[["RM"]].join(DataFrame(data=boston.target, columns=["Price"]))
plt.scatter(df["RM"], df["Price"])
plt.xlabel("Number of rooms")
plt.ylabel("Price in $1000")
plt.show()
```

为了方便描述，我们这里引入一些符号。$x^{(i)}$ 用来表示输入变量，比如这里的平均房间数，它也被称为**特征features**。$y^{(i)}$ 用来表示输出变量，或我们希望预测的**目标target**变量。一对 $(x^{(i)}, y^{(i)})$ 称为一个**训练样本training example**，而我们将使用的 $m$ 个训练样本 $\{(x^{(i)}, y^{(i)});i=1,...,m\}$ 的集合称为**训练集training set**。注意这里的上标 $(i)$ 是对训练集的一个索引编号，不是幂运算。我们用 $\chi$ 表示输入空间，$\gamma$ 表示输出空间。在这个例子中 $\chi=\gamma=\mathbb{R}$

正式地定义监督学习：给定一个训练集，我们希望学习到一个函数 $h: \chi \rightarrow \gamma$，使得 $h(x)$ 可以很好地预测 $y$。由于历史原因，函数 $h$ 被称为**假设hypothesis**。

当我们希望预测的目标变量是连续型变量时，比如这里的房价，我们称这是一个**回归Regression**问题。当 $y$ 仅仅只取少数离散值时，我们称这是一个**分类Classification**问题。

# 线性回归 Linear Regression

包括以下内容：
    1. 最小二乘法 LMS algorithm
    2. 正规方程 The normal equations
    3. 概率诠释 Probabilistic interpretation
    4. 局部加权线性回归 Locally weighted linear regression

```python
# 预览数据
print(df.head())
```

          RM  Price
    0  6.575   24.0
    1  6.421   21.6
    2  7.185   34.7
    3  6.998   33.4
    4  7.147   36.2

对于这个数据集，$x$ 是一个$\mathbb{R}$上的一维向量。$x_1^i$是该区域的平均房间数。

要进行监督学习，我们需要首先决定如何来表示假设/函数 $h$。较为简单的情况下，我们选取 $x$ 的线性函数来预测 $y$ ：
$$ h_\theta(x) = \theta_0 + \theta_1x_1 $$
这里，$\theta_i$称为**参数parameters**（有时也称为**权重weights**），通过这组参数，我们将 $\chi$ 线性映射到 $\gamma$。在不引发误解的情况下，我们将 $h_\theta(x)$简写为 $h(x)$。习惯上，我们设 $x_0=1$ （称为**截距项intercept term**），这样
$$ h(x) = \sum_{i=0}^n \theta_ix_i = \theta^Tx $$
对于右式，我们将 $\theta$ 和 $x$ 看做向量，$n$ 是特征数量（不包括 $x_0$ )

这样，给定训练集，我们需要挑选，或者说学习出参数 $\theta$ 的值。一个直观的方法是使 $h(x)$ 尽量接近 $y$，至少对于训练集来说，尽量地接近。为了更正式地定义“接近”，我们定义**成本函数cost function**：
$$ J(\theta) = \frac{1}{2}\sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2$$

## 1. 最小二乘法 Linear Regression

我们希望挑选出使 $J(\theta)$ 最小化的值 $\theta$。一种搜索算法的思路是，我们随机猜测一个 $\theta$，然后按某种规则重复地改变 $\theta$ 使得 $J(\theta)$ 不断变小，直到 $\theta$ 收敛到某个值使得 $J(\theta)$ 取得最小值。

特别地，我们将考虑**梯度下降Gradient Descent**算法。如下图所示，$J(\theta)$ 是关于 $\theta_0$ 和 $\theta_1$ 的函数。首先我们随机选择一个 $ \theta = [-10, 17.5]$，想象在图形中下坡，我们需要选择一个下降最快的方向。不断地前进，直到来到某个**局域最小值local minima**。在我们的图中，只有一个**全局最小值global minima**（以红星标记）。但梯度下降的一个特性是，如果存在局域最小值时，根据随机选择初始化 $\theta$ 的不同，我们可能会到达不同的局域最小值。局部最小值的问题，在今后的模型中会遇到。

```python
# 读取训练集和目标变量
X, y = df[["RM"]].values, df["Price"].values
m, n = X.shape
# 增加截距列
X_intercept = np.column_stack((np.ones((m, 1)), X))

@np.vectorize
def cost_function(*args):
    """成本函数"""
    theta = np.array(args)
    return 0.5 * ((X_intercept @ theta - y) ** 2).sum()

def gradient_descent_optimize(cost_func, init_theta=None):
    """使用梯度下降寻找最优的theta"""
    alpha = 0.00003

    def good_enough(old_cost, new_cost):
        return np.abs(new_cost - old_cost) / old_cost < 0.00000001

    def improve_theta(old_theta):
        return old_theta - alpha * ((X_intercept @ old_theta - y) @ X_intercept)

    if init_theta is None:
        init_theta = np.array([0, 0])
    path_ = [init_theta]
    o_theta, o_cost = init_theta, cost_function(*init_theta)
    while True:
        n_theta = improve_theta(o_theta)
        n_cost = cost_func(*n_theta)
        path_.append(n_theta)
        logging.debug((o_theta, n_theta, o_cost, n_cost))
        if good_enough(o_cost, n_cost):
            break
        o_theta, o_cost = n_theta, n_cost
    Result = namedtuple('Result', ['theta', 'path'])
    return Result(theta=n_theta, path=np.array(path_))

# 使用梯度下降获取优化结果
init_guess = np.array([-10, 17.5])
gradient_descent_result = gradient_descent_optimize(cost_function, init_theta=init_guess)
path = gradient_descent_result.path
sampled_path = np.concatenate((np.array([path[0, ]]),
                               path[(2 ** np.array(range(int(np.ceil(np.log2(path.shape[0])))))).astype(np.int32), ]))
sampled_path_mat = sampled_path.T

# 使用sklearn的LinearRegression模型，直接获取最终的theta值
regr = LinearRegression()
regr.fit(X, y)
theta0_optimize, theta1_optimize = regr.intercept_, regr.coef_[0]
minima = np.array([theta0_optimize, theta1_optimize])
minima_ = minima.reshape(-1, 1)

# 根据最优化的theta，生成以此为中心的网格数据
step = 0.2
theta0_min, theta0_max = theta0_optimize - np.abs(theta0_optimize), theta0_optimize + np.abs(theta0_optimize)
theta1_min, theta1_max = theta1_optimize - np.abs(theta1_optimize), theta1_optimize + np.abs(theta1_optimize)
theta0_range, theta1_range = np.arange(theta0_min, theta0_max + step, step), np.arange(
    theta1_min, theta1_max + step, step)
theta0_mat, theta1_mat = np.meshgrid(theta0_range, theta1_range)
z = cost_function(theta0_mat.reshape(-1), theta1_mat.reshape(-1)).reshape(theta0_mat.shape)

fig = plt.figure(figsize=(12, 6))
# 绘制surface平面图
ax1 = fig.add_subplot(1, 2, 1, projection='3d', elev=50, azim=-50)
ax1.plot_surface(theta0_mat, theta1_mat, z, norm=LogNorm(), cmap=cm.jet, rstride=1, cstride=1, edgecolor='none',
                 alpha=.8)
ax1.scatter(sampled_path_mat[0, :-1], sampled_path_mat[1, :-1], cost_function(*sampled_path_mat[::, :-1]))
ax1.plot(*minima_, cost_function(*minima_), 'r*', markersize=10)
ax1.set_xlabel('$theta_0$')
ax1.set_ylabel('$theta_1$')
ax1.set_zlabel('$J(theta)$')
ax1.set_xlim((theta0_min, theta0_max))
ax1.set_ylim((theta1_min, theta1_max))
ax1.set_title("Surface Map of J(theta)")

# 绘制contour轮廓图
ax2 = fig.add_subplot(1, 2, 2)
ax2.contour(theta0_mat, theta1_mat, z, levels=np.logspace(0, 5, 35), norm=LogNorm(), cmap=cm.jet)
ax2.plot(*minima_, 'r*', markersize=18)
ax2.quiver(sampled_path_mat[0, :-1], sampled_path_mat[1, :-1], sampled_path_mat[0, 1:] - sampled_path_mat[0, :-1],
           sampled_path_mat[1, 1:] - sampled_path_mat[1, :-1],
           scale_units='xy', angles='xy', scale=1, color='k')
ax2.set_xlabel('$theta0$')
ax2.set_ylabel('$theta1$')
ax2.set_xlim((theta0_min, theta0_max))
ax2.set_ylim((theta1_min, theta1_max))
ax2.set_title("Contour Map of J(theta)")

plt.tight_layout()
plt.show()
```

下面以数学形式定义梯度下降，随机选择某个初始 $\theta$，之后重复以下更新：
$$ \theta_j = \theta_j - \alpha\frac{\partial}{\partial\theta_j}J(\theta) $$
（同时对所有值进行更新，$j = 0,...,n$）

这里，$\alpha$ 称为**学习速率learning rate**。这个算法非常自然地每次往J下降幅度最为陡峭的方向前进一小步。当 $\alpha$ 过小时，算法的收敛速度会比较慢；而 $\alpha$ 过大时，会出现算法不收敛的情况。另外，梯度下降过程中，梯度 $\frac{\partial}{\partial\theta_j}J(\theta)$ 本身会不断变小，因此更新的幅度也会越来越小：

$$\begin{equation}
\begin{split}
\frac{\partial}{\partial\theta_j}J(\theta) & = \sum_{i=1}^m\frac{\partial}{\partial\theta_j}\frac{1}{2}(h_\theta(x^{(i)}) - y^{(i)})^2 \\
 & = \sum_{i=1}^m2 \cdot \frac{1}{2}(h_\theta(x^{(i)}) - y^{(i)}) \cdot \frac{\partial}{\partial\theta_j}(h_\theta(x^{(i)}) - y^{(i)}) \\
 & = \sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)}) \cdot \frac{\partial}{\partial\theta_j}(\sum_{k=0}^{n}\theta_kx_k^{(i)}-y^{(i)}) \\
 & = \sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)}
\end{split}
\end{equation}$$

我们根据以下规则进行更新：
$$ \theta_j = \theta_j - \alpha\sum_{i=1}^m(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} $$
重复直至 $\theta$ 收敛即可。

这个规则称为**最小二乘 LMS least mean square**，也被称为**Widrow-Hoff学习规则**。

每次更新过程中都检查整个训练集，我们称这种算法为**批量梯度下降 batch gradient descent**。对于线性回归来说，成本函数只有唯一一个全局最小值，不存在局域最小值。也就是说，$J$ 是一个凸二次函数。梯度下降对于线性回归总能收敛（唯一的前提，学习速率 $\alpha$ 没有选得过大）。

下图展现了动画形式的梯度下降训练过程：

```python
fig, ax = plt.subplots()
ax.scatter(X.reshape(-1), y, alpha=0.5)
plt.xlabel("Number of rooms")
plt.ylabel("Price in $1000")
x = np.arange(3, 10, 1)
line, = ax.plot([], [], lw=2)

def animate(i):
    line.set_data(x, np.column_stack((np.ones((x.size, 1)), x)) @ sampled_path[i])
    return line,

def init():
    line.set_data([], [])
    return line,

anim = animation.FuncAnimation(fig, animate, frames=sampled_path.shape[0], init_func=init, interval=800,
                               repeat_delay=3000, blit=True)

HTML(anim.to_html5_video())
```

```python
# 使用批量梯度下降训练出的theta
print(gradient_descent_result.theta)
```

    [-34.35932859   9.05317227]

```python
# 使用sklearn的LinearRegression模型训练出的theta
print(minima)
```

    [-34.67062078   9.10210898]

可以看出，梯度下降训练出的 $\theta$ 十分接近使用高级优化算法计算出的最优 $\theta$；进一步增加good_enough函数中的精度，可以使梯度下降算法训练出更精确的 $\theta$（同时也需要花费更多的迭代次数，意味着更长的训练时间）。

批量梯度下降的执行效率较低，因为每一次迭代更新，都需要使用全量数据来计算梯度。当 $m$ 非常大的时候，这个操作十分耗时。考虑另一种效果同样不错，但是效率更高的替代方案：
$$ \theta_j = \theta_j - \alpha(h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} $$
对于 $i = 1, 2, ..., m$ 重复执行，每次执行同时对所有值进行更新，$j = 0,...,n$

在该算法中，我们遍历整个训练集，对每一个训练样本，仅以该样本的梯度进行参数更新。这种算法叫做**随机梯度下降 stochastic gradient descent**，也称为**增量梯度下降 incremental gradient descent**。

随机梯度下降通常会比批量梯度下降更快地接近最优 $\theta$，但需要注意的是，随机梯度下降可能永远不会“收敛”到最小值，其训练的参数 $ \theta $会在 $J(\theta)$ 取得最小值处震荡。但是在实际情况中，最小值附近的值就已经足够好了。由于这些原因，当训练集非常大时，我们通常更趋向于使用随机梯度下降。

## 2. 正规方程 The normal equations

梯度下降是一种迭代算法，从算法描述就可以看出它非常符合计算机科学的思维方式。而正规方程是一种直接求最小值的方法，我们令 $J(\theta)$ 的导数为0，进一步通过推导的形式计算出此时的 $\theta$，显然这是数学的思维方式。

### 2.1 矩阵导数 Matrix derivatives

对于一个将 $m \times n$ 矩阵映射到实数的函数 $f: \mathbb{R}^{m \times n} \rightarrow \mathbb{R} $，我们定义 $f$ 对 $A$ 的导数为：

$$
\nabla_Af(A) =
\begin{bmatrix}
    \frac{\partial f}{\partial A_{11}} & \dots & \frac{\partial f}{\partial A_{1n}} \\
    \vdots & \ddots & \vdots \\
    \frac{\partial f}{\partial A_{m1}} & \dots & \frac{\partial f}{\partial A_{mn}}
\end{bmatrix}
$$

因此，梯度 $\nabla_Af(A)$ 本身是一个 $m \times n$ 的矩阵，其第 $(i, j)$ 个元素为 $\frac{\partial f}{\partial A_{ij}}$。

以一个 $2 \times 2$ 的矩阵为例，假设 $f(A) = \frac{3}{2}A_{11} + 5A_{12}^2 + A_{21}A_{22}$，那么：
$$
\nabla_Af(A) =
\begin{bmatrix}
    \frac{\partial f}{\partial A_{11}} & \frac{\partial f}{\partial A_{12}} \\
    \frac{\partial f}{\partial A_{21}} & \frac{\partial f}{\partial A_{22}}
\end{bmatrix} =
\begin{bmatrix}
    \frac{3}{2} & 10A_{12} \\
    A_{22} & A_{21}
\end{bmatrix}
$$

对于一个 $n \times n$ 的矩阵（方阵）A，矩阵的**迹 trace**，记做 $trA$，定义为对角线上的元素之和：
$$ trA = \sum_{i=1}^n A_{ii} $$

如果 $a$ 是一个实数（$ 1 \times 1$ 矩阵），$tr a = a$

对于矩阵 $A$ 和矩阵 $B$，如果矩阵 $AB$ 是方阵，那么 $trAB = trBA$。证明如下：

设 $A$ 是一个 $ m \times n$ 的矩阵，为了使 $AB$ 是方阵，则 $B$ 必须是 $n \times m$的矩阵，根据矩阵相乘和迹的定义
$$ trAB =  \sum_{i=1}^m(\sum_{j=1}^n A_{ij}B_{ji}) = \sum_{j=1}^n(\sum_{i=1}^m B_{ji}A_{ij}) = trBA$$

显然上述结论具有以下推论：
$$ trABC = trCAB = trBCA $$
$$ trABCD = trDABC = tr CDAB = tr BCDA$$

对于方阵 $A$，方阵 $B$ 和实数a：
$$ trA = trA^T $$
$$ tr(A + B) = trA + trB $$
$$ traA = atrA $$

矩阵的导数具有以下性质：
$$
\begin{align*}
\nabla_AtrAB &= B^T  &(1) \\
\nabla_{A^T}f(A) &= (\nabla_Af(A))^T &(2) \\
\nabla_A trABA^TC &= CAB + C^TAB^T &(3) \\
\nabla_A |A| &= |A|(A^{-1})^T &(4)
\end{align*}
$$

依然沿用 $A$ 是 $m \times n$ 的矩阵，$B$ 是 $n \times m$的矩阵的假设，证明如下：

1) 设 $ f(A) = trAB$，显然 $f$ 满足条件 $\mathbb{R}^{m \times n} \rightarrow \mathbb{R}$的映射
$$ \nabla_AtrAB = \nabla_Af(A) = \nabla_A \sum_{i=1}^m\sum_{j=1}^nA_{ij}B_{ji} =
\begin{bmatrix}
    B_{11} & \dots & B_{n1} \\
    \vdots & \ddots & \vdots \\
    B_{1m} & \dots & B_{nm}
\end{bmatrix}
= B^T
$$

2) 设 $ X = A^T $，即 $X_{ji} = A_{ij}$:
$$
\begin{equation}
\begin{split}
\nabla_{A^T}f(A) & = \nabla_{A^T}f(A_{11}, ..., A_{1n}, A_{21}, ..., A_{2n},..., A_{m1}, ..., A_{mn}) \\
& = \nabla_{X}f(X_{11}, ..., X_{n1}, X_{12}, ..., X_{n2},..., X_{1m}, ..., X_{nm}) \\
& = \nabla_{X}f(X_{11}, ..., X_{1m}, X_{21}, ..., X_{2n},..., X_{1n}, ..., X_{nm}) \\
& = \begin{bmatrix}
    \frac{\partial f}{\partial X_{11}} & \dots & \frac{\partial f}{\partial X_{1m}} \\
    \vdots & \ddots & \vdots \\
    \frac{\partial f}{\partial X_{n1}} & \dots & \frac{\partial f}{\partial X_{nm}}
\end{bmatrix} \\
& = \begin{bmatrix}
    \frac{\partial f}{\partial X_{11}} & \dots & \frac{\partial f}{\partial X_{n1}} \\
    \vdots & \ddots & \vdots \\
    \frac{\partial f}{\partial X_{1m}} & \dots & \frac{\partial f}{\partial X_{nm}}
\end{bmatrix}^T \\
& = \begin{bmatrix}
    \frac{\partial f}{\partial A_{11}} & \dots & \frac{\partial f}{\partial A_{1n}} \\
    \vdots & \ddots & \vdots \\
    \frac{\partial f}{\partial A_{m1}} & \dots & \frac{\partial f}{\partial A_{mn}}
\end{bmatrix}^T \\
& = (\nabla_{A}f(A))^T
\end{split}
\end{equation}
$$

3) 略

4) 略

### 2.2 最小二乘法回顾 Least squares revisited

将整个训练集想象成一个 $m \times n$ 的矩阵（如果加入截距项的话，实际是 $m \times n+1$ 的矩阵），$y$ 想象成是 $m$ 维的向量，$h_\theta(x) = X \theta$。于是，成本函数可以表示为：
$$
J(\theta) = \frac{1}{2}\sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})^2 = \frac{1}{2}(X\theta-y)^T(X\theta-y)
$$

其导数可以表示为：
$$
\begin{split}
\nabla_{\theta}J(\theta) & = \nabla_{\theta}\frac{1}{2}(X\theta-y)^T(X\theta-y) \\
& = \frac{1}{2}\nabla_{\theta}(\theta^TX^TX\theta - \theta^TX^Ty - y^TX\theta + y^Ty) \\
& = \frac{1}{2}\nabla_{\theta}tr(\theta^TX^TX\theta - \theta^TX^Ty - y^TX\theta + y^Ty) \\
& = \frac{1}{2}\nabla_{\theta}(tr\theta^TX^TX\theta - 2try^TX\theta) \\
& = \frac{1}{2}(X^TX\theta + X^TX\theta - 2X^Ty) \\
& = X^TX\theta - X^Ty
\end{split}
$$

上述推导过程中，第二步是矩阵相乘；第三步利用了实数的迹等于实数本身；第四步利用了 $trA = trA^T$，而 $y^Ty$ 与 $\theta$ 无关；第五步首先利用了公式 $\nabla_AtrA^TCAB = \nabla_A trABA^TC = CAB + C^TAB^T$，令 $A=\theta, C=X^TX, B=I$。其次利用了 $\nabla_AtrBA = \nabla_AtrAB = B^T$。

为了使 $J(\theta)$ 最小化，我们将其导数设为零，于是得到正规方程：
$$ X^TX\theta = X^Ty $$

所以，使 $J(\theta)$ 取得最小值的 $\theta$ 为：
$$ \theta = (X^TX)^{-1}X^Ty $$

```python
# 使用正规方程计算出的theta
print(inv(X_intercept.T @ X_intercept) @ X_intercept.T @ y)
```

    [-34.67062078   9.10210898]

可以看到，使用正规方程计算的最优 $\theta$ 值，和sklearn线性模型计算出来的是一样的。

## 3. 概率诠释 Probabilistic interpretation

面对一个回归问题，一个疑惑是，为什么我们选择了最小二乘作为成本函数？本节将引入一些概率论的假设，在这些假设之上，最小二乘的线性回归将会是非常自然的选择。

假设目标变量和输入变量的关系如下：
$$ y^{(i)} = \theta^Tx^{(i)} + \epsilon^{(i)} $$

这里 $\epsilon$ 是误差项，用来描述模型未引入的影响因素（比如某些对 $y$ 有重要影响的指标）或随机噪声。并且我们假设 $\epsilon$ 是满足以 $0$ 为均值，$\sigma^2$ 为方差的高斯分布的**独立同分布随机变量 IID independently and identically distributed random variable**。写做 $\epsilon^{(i)} \sim N(0, \sigma^2)$，即 $\epsilon^{(i)}$ 的概率密度为：

$$ f(\epsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(\epsilon^{(i)})^2}{2\sigma^2}) $$

也就是说：

$$ f(y^{(i)}|x^{(i)};\theta) = \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}) $$

这里 $f(y^{(i)}|x^{(i)};\theta)$ 表示以 $\theta$ 为参数，给定 $x^{(i)}$ 时 $y^{(i)}$ 的概率密度。注意 $\theta$ 不是随机变量，而是描述 $y^{(i)}$所服从分布的参数，不能写到条件中去。

以整个训练集来描述，这个概率变为 $f(y|X; \theta)$。如果我们将其视作关于 $\theta$ 的函数，这个函数就被称为**似然函数 likelyhood function**：

$$ L(\theta) = L(\theta; X, y) = f(y|X; \theta) $$

由 $\epsilon^{(i)}$ 是独立变量的假设，似然函数可以写成：

$$ L(\theta) = \prod_{i=1}^{m} f(y^{(i)}|x^{(i)};\theta) = \prod_{i=1}^{m} \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2})$$

**最大似然估计法 maximum likelyhood**要求我们选择使 $L(\theta)$ 最大的 $\theta$，以使得事件发生的可能性最大。

任何关于 $L(\theta)$ 的增函数取得最大值时，$L(\theta)$也会同时取得最大值。因此，很多时候我们都最大化**对数似然函数log likelyhood** $\ell(\theta)$:

$$
\begin{split}
\ell(\theta) & = logL(\theta) \\
& = log\prod_{i=1}^{m} \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}) \\
& = \sum_{i=1}^{m} log\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}) \\
& = mlog\frac{1}{\sqrt{2\pi}\sigma} - \frac{1}{2\sigma^2}\sum_{i=1}^{m} (y^{(i)} - \theta^Tx^{(i)})^2
\end{split}
$$

于是，最大化 $\ell(\theta)$ 等价于最小化 $\frac{1}{2}\sum_{i=1}^{m} (y^{(i)} - \theta^Tx^{(i)})^2$，正是成本函数 $J(\theta)$。

总结一下：给定我们关于数据的概率假设，最小二乘回归对应着寻找 $\theta$ 的最大似然估计值。

## 4. 局部加权线性回归 Locally weighted linear regression

```python
np.random.seed(0)

n_samples = 30
degrees = [1, 4, 15]

true_fun = lambda X: np.cos(1.5 * np.pi * X)
X = np.sort(np.random.rand(n_samples))
y = true_fun(X) + np.random.randn(n_samples) * 0.1

plt.figure(figsize=(14, 5))
for i in range(len(degrees)):
    ax = plt.subplot(1, len(degrees), i + 1)
    plt.setp(ax, xticks=(), yticks=())

    polynomial_features = PolynomialFeatures(degree=degrees[i],
                                             include_bias=False)
    linear_regression = LinearRegression()
    pipeline = Pipeline([("polynomial_features", polynomial_features),
                         ("linear_regression", linear_regression)])
    pipeline.fit(X[:, np.newaxis], y)

    # Evaluate the models using crossvalidation
    scores = cross_val_score(pipeline, X[:, np.newaxis], y,
                             scoring="neg_mean_squared_error", cv=10)

    X_test = np.linspace(0, 1, 100)
    plt.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
    plt.plot(X_test, true_fun(X_test), label="True function")
    plt.scatter(X, y, label="Samples")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.xlim((0, 1))
    plt.ylim((-2, 2))
    plt.legend(loc="best")
    plt.title("Degree {}\nMSE = {:.2e}(+/- {:.2e})".format(
        degrees[i], -scores.mean(), scores.std()))
plt.show()
```

考虑这个问题，我们试图用 $x \in \mathbb{R}$ 来预测变量 $y$，上图可以看出 $y$ 与 $x$ 并不是线性关系。左侧的图中，我们试图通过线性回归来预测，拟合的效果较差。如果我们最高使用 $x^4$ 作为特征，即 $y=\theta_0 + \theta_1x + \theta_2x^2 + \theta_3x^3 + \theta_4x^4$，拟合的效果如中图所示，非常接近于真实的函数关系；然而，不断添加高阶特征也非常危险，右侧的图上添加了最高15阶的特征，即 $y = \sum_{j=0}^{15} \theta_jx^j$，这时曲线拟合的效果甚至更好，但这显然不是一个很好的模型。

左侧的模型属于**欠拟合 underfitting**，模型并没能捕捉到数据的特征；而右侧的模型属于**过拟合 overfitting**，模型因为对训练数据做了过多的优化，捕捉了非常多噪声，从而缺少泛化能力。（之后，在学习理论中，我们会更加正式地定义这两个概念）。

从中我们可以得到的另一个结论是，特征的选择，对于模型表现具有至关重要的影响。

本节我们将简单地介绍**局部加权线性回归 Locally Weighted Linear Regression(LWR)**，当有足够的数据量时，这个模型对特征的选择相对不那么敏感。

在线性回归模型中，为了根据 $x$ 预测 $y$，我们会：
1. 找到使得 $\sum_i (y^{(i)}-\theta^Tx^{(i)})^2$ 最小的 $\theta$
2. 计算 $\theta^Tx$

而在局部加权线性模型中，我们进行以下步骤：
1. 找到使得 $\sum_i w^{(i)}(y^{(i)}-\theta^Tx^{(i)})^2$ 最小的 $\theta$
2. 计算 $\theta^Tx$

这里，$w^{(i)}$ 是非负的**权重 weights**。对于某个样本 $i$ 而言，如果 $w^{(i)}$ 非常大，这个样本的平方误差，对于最终的成本函数来说，显然具有更大的话语权。

一个常见的权重选择是：
$$ w^{(i)} = exp(-\frac{(x^{(i)}-x)^2}{2\tau^2}) $$

权重取决于我们具体将要预测的点 $x$，并且给定点到预测点的距离越近，权重越接近1；反之权重越接近0；参数 $\tau$ 决定了随着 $x^{(i)}$ 远离预测点 $x$，权重下降得有多块。因而 $\tau$ 被称为**带宽 bandwidth**参数。

设有对角矩阵 $W$，即$ w(i, i) = exp(-\frac{(x^{(i)}-x)^2}{2\tau^2}) $，则局部加权线性模型的正规方程为：
$$ w = (X^TWX)^{-1}X^TWy $$

局部加权线性是一个**非参数算法 non-parametric**，而线性回归是**参数学习算法 parametric learning algorithm**，因为线性回归有固定有限个参数 $\theta$。一旦我们计算好线性回归的参数 $\theta$ 并保存下来，预测过程中我们将不再需要使用训练集。相反，对于局部加权线性模型，我们始终需要保存整个训练集。非参数算法大致可以等同于以下事实：为表示假设函数 $h(x)$，我们需要保存的数据量通常随着训练集的大小线性增长。

## 附：参考资料

1. [cs229-notes1](https://see.stanford.edu/materials/aimlcs229/cs229-notes1.pdf)
2. [Scikit Learn Document](http://scikit-learn.org/stable/documentation.html)
3. [机器学习实战](https://book.douban.com/subject/24703171/)

```python

```


# Linear Shooting Method
#### John S Butler john.s.butler@tudublin.ie   [Course Notes](https://johnsbutler.netlify.com/files/Teaching/Numerical_Analysis_for_Differential_Equations.pdf)    [Github](https://github.com/john-s-butler-dit/Numerical-Analysis-Python)

## Overview
This notebook illustates the implentation of a the linear shooting method to a linear boundary value problem.

## Introduction
To numerically approximate the Boundary Value Problem
$$
y^{''}=p(x)y^{'}+q(x)y+r(x) \ \ \ a < x < b $$
$$y(a)=\alpha$$
$$y(b) =\beta$$

The Boundary Value Problem is divided into two
Initial Value Problems:
1. The first 2nd order Initial Value Problem is the same as the original Boundary Value Problem with an extra initial condtion $y_1^{'}(a)=0$.
\begin{equation}
 y^{''}_1=p(x)y^{'}_1+q(x)y_1+r(x), \ \    y_1(a)=\alpha, \ \ \color{green}{y^{'}_1(a)=0},\\
\end{equation}
2. The second 2nd order Initial Value Problem is the homogenous form of the original Boundary Value Problem, by removing $r(x)$, with the initial condtions $y_2(a)=0$ and $y_2^{'}(a)=1$.

\begin{equation}
y^{''}_2=p(x)y^{'}_2+q(x)y_2, \ \ \color{green}{y_2(a)=0, \ \ y^{'}_2(a)=1}.
\end{equation}

combining these intial values problems together to get the unique solution
\begin{equation}
y(x)=y_1(x)+\frac{\beta-y_1(b)}{y_2(b)}y_2(x),
\end{equation}
provided that $y_2(b)\not=0$.

The truncation error for the shooting method  is
$$ |y_i - y(x_i)| \leq K h^n\left|1+\frac{y_{2 i}}{y_{2 i}}\right| $$
$O(h^n)$ is the order of the numerical method used to approximate the solution of the Initial Value Problems.

```python
import numpy as np
import math
import matplotlib.pyplot as plt
import pandas as pd
import warnings
warnings.filterwarnings("ignore")

from IPython.display import HTML

```

<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>

## Example Boundary Value Problem
To illustrate the shooting method we shall apply it to the Boundary Value Problem:
$$ y^{''}=2y^{'}+3y-6, $$
with boundary conditions
$$y(0) = 3, $$
$$y(1) = e^3+2, $$
with the exact solution is
$$y=e^{3x}+2. $$
The __boundary value problem__ is broken into two second order __Initial Value Problems:__
1. The first 2nd order Intial Value Problem is the same as the original Boundary Value Problem with an extra initial condtion $u^{'}(0)=0$.
\begin{equation}
u^{''} =2u'+3u-6, \ \ \ \ u(0)=3, \ \ \ \color{green}{u^{'}(0)=0}
\end{equation}
2. The second 2nd order Intial Value Problem is the homogenous form of the original Boundary Value Problem with the initial condtions $w^{'}(0)=0$ and $w^{'}(0)=1$.
\begin{equation}
w^{''} =2w^{'}+3w, \ \ \ \ \color{green}{w(0)=0}, \ \ \ \color{green}{w^{'}(0)=1}
\end{equation}

combining these results of these two intial value problems as a linear sum
\begin{equation}
y(x)=u(x)+\frac{e^{3x}+2-u(1)}{w(1)}w(x)
\end{equation}
gives the solution of the Boundary Value Problem.

## Discrete Axis
The stepsize is defined as
$$h=\frac{b-a}{N}$$
here it is
$$h=\frac{1-0}{10}$$
giving
$$x_i=0+0.1 i$$
for $i=0,1,...10.$

```python
## BVP
N=10
h=1/N
x=np.linspace(0,1,N+1)
fig = plt.figure(figsize=(10,4))
plt.plot(x,0*x,'o:',color='red')
plt.xlim((0,1))
plt.title('Illustration of discrete time points for h=%s'%(h))

plt.show()
```

## Initial conditions
The initial conditions for the discrete equations are:
$$ u_1[0]=3$$
$$ \color{green}{u_2[0]=0}$$
$$ \color{green}{w_1[0]=0}$$
$$ \color{green}{w_2[0]=1}$$

```python
U1=np.zeros(N+1)
U2=np.zeros(N+1)
W1=np.zeros(N+1)
W2=np.zeros(N+1)

U1[0]=3
U2[0]=0

W1[0]=0
W2[0]=1
```

## Numerical method
The Euler method is applied to numerically approximate the solution of the system of the two second order initial value problems they are converted in to two pairs of two first order initial value problems:

### 1. Inhomogenous Approximation
The plot below shows the numerical approximation for the two first order Intial Value Problems
\begin{equation}
u_1^{'} =u_2, \ \ \ \ u_1(0)=3,
\end{equation}
\begin{equation}
u_2^{'} =2u_2+3u_1-6, \ \ \ \color{green}{u_2(0)=0},
\end{equation}

that Euler approximate of the inhomogeneous two Initial Value Problems is :
$$u_{1}[i+1]=u_{1}[i] + h u_{2}[i]$$
$$u_{2}[i+1]=u_{2}[i] + h (2u_{2}[i]+3u_{1}[i] -6)$$
with $u_1[0]=3$ and $\color{green}{u_2[0]=0}$.

```python
for i in range (0,N):
    U1[i+1]=U1[i]+h*(U2[i])
    U2[i+1]=U2[i]+h*(2*U2[i]+3*U1[i]-6)

```

### Plots
The plot below shows the Euler approximation of the two intial value problems $u_1$ on the left and $u2$ on the right.

```python
fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(1,2,1)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,U1,'^')
plt.title(r"$u_1'=u_2, \ \  u_1(0)=3$",fontsize=16)
plt.grid(True)

ax = fig.add_subplot(1,2,2)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,U2,'v')
plt.title(r"$u_2'=2u_2+3u_1-6, \ \  u_2(0)=0$", fontsize=16)

plt.grid(True)
plt.show()
```

### 2. Homogenous Approximation
The homogeneous Bounday Value Problem is divided into two first order Intial Value Problems
\begin{equation}
w_1^{'} =w_2, \ \ \ \ \color{green}{w_1(1)=0}
\end{equation}
\begin{equation}
w_2^{'} =2w_2+3w_1,  \ \ \ \color{green}{w_2(1)=1}
\end{equation}

The Euler approximation of the homogeneous of the two Initial Value Problem is
$$w_{1}[i+1]=w_{1}[i] + h w_{2}[i]$$
$$w_{2}[i+1]=w_{2}[i] + h (2w_{2}[i]+3w_{1}[i])$$
with $\color{green}{w_1[0]=0}$ and $\color{green}{w_2[1]=1}$.

```python
for i in range (0,N):
    W1[i+1]=W1[i]+h*(W2[i])
    W2[i+1]=W2[i]+h*(2*W2[i]+3*W1[i])
```

### Homogenous Approximation

### Plots
The plot below shows the Euler approximation of the two intial value problems $u_1$ on the left and $u2$ on the right.

```python
fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(1,2,1)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,W1,'^')
plt.grid(True)
plt.title(r"$w_1'=w_2, \ \  w_1(0)=0$",fontsize=16)

ax = fig.add_subplot(1,2,2)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,W2,'v')
plt.grid(True)
plt.title(r"$w_2'=2w_2+3w_1, \ \  w_2(0)=1$",fontsize=16)
plt.tight_layout()
plt.subplots_adjust(top=0.85)

plt.show()
beta=math.exp(3)+2
y=U1+(beta-U1[N])/W1[N]*W1
```

## Approximate Solution
Combining together the numerical approximation of $u_1$ and $w_1$ as a weighted sum
$$y(x[i])\approx u_{1}[i] + \frac{e^3+2-u_{1}[N]}{w_1[N]}w_{1}[i]$$
gives the approximate solution of the Boundary Value Problem.

The truncation error for the shooting method using the Euler method is
$$ |y_i - y(x[i])| \leq K h\left|1+\frac{w_{1}[i]}{u_{1}[i]}\right| $$
$O(h)$ is the order of the method.

The plot below shows the approximate solution of the Boundary Value Problem (left), the exact solution (middle) and the error (right)

```python
Exact=np.exp(3*x)+2
fig = plt.figure(figsize=(12,4))
ax = fig.add_subplot(2,3,1)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,y,'o')

plt.grid(True)
plt.title(r"Numerical: $u_1+\frac{e^3+2-u_1(N)}{w_1(N)}w_1$",
          fontsize=16)

ax = fig.add_subplot(2,3,2)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,Exact,'ks-')

plt.grid(True)
plt.title(r"Exact: $y=e^{3x}+2$",
          fontsize=16)

ax = fig.add_subplot(2,3,3)
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.plot(x,abs(y-Exact),'ro')
plt.grid(True)
plt.title(r"Error ",fontsize=16)

plt.tight_layout()
plt.subplots_adjust(top=0.85)
plt.show()
```

### Data
The Table below shows that output for $x$, the Euler numerical approximations $U1$, $U2$, $W1$ and $W2$ of the system of four Intial Value Problems, the shooting methods approximate solution $y_i=u_{1 i} + \frac{e^3+2-u_{1}(x_N)}{w_1(x_N)}w_{1 i}$ and the exact solution of the Boundary Value Problem.

```python
table = ListTable()
table.append(['x', 'U1','U2','W1','W2','Approx','Exact'])
for i in range (0,len(x)):
    table.append([round(x[i],3), round(U1[i],3), round(U2[i],3),
                  round(W1[i],5),round(W2[i],3),
                  round(y[i],5),
                 round(Exact[i],5)])
table

d = {'time x_i': x[0:10],
     'Runge Kutta':round(U1[0:10],3),

    round(U1[i],3), round(U2[i],3),
                  round(W1[i],5),round(W2[i],3),
                  round(y[i],5),
                 round(Exact[i],5)
    }
df = pd.DataFrame(data=d)
df
```

x	U1	U2	W1	W2	Approx	Exact

0.0	3.0	0.0	0.0	1.0	3.0	3.0

0.1	3.0	0.3	0.1	1.2	3.48753	3.34986

0.2	3.03	0.66	0.22	1.47	4.10257	3.82212

0.3	3.096	1.101	0.367	1.83	4.88524	4.4596

0.4	3.206	1.65	0.55	2.306	5.88752	5.32012

0.5	3.371	2.342	0.78061	2.932	7.17681	6.48169

0.6	3.605	3.222	1.07384	3.753	8.84059	8.04965

0.7	3.927	4.347	1.44914	4.826	10.99242	10.16617

0.8	4.362	5.795	1.93171	6.226	13.77985	13.02318

0.9	4.942	7.663	2.55427	8.05	17.39453	16.87973

1.0	5.708	10.078	3.35929	10.427	22.08554	22.08554

```python

```

Chapter 06 - Boundary Value Problems/.ipynb_checkpoints/601_Linear Shooting Method-checkpoint.ipynb


# Занятие 6
# Прикладная алгебра и численные методы
## Интерполяция: оценка близости функций

```
!python -m pip install sympy --upgrade
!python -m pip install -U matplotlib
!pip install bezier
```

    Requirement already up-to-date: sympy in /usr/local/lib/python3.6/dist-packages (1.7.1)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy) (1.1.0)
    Requirement already up-to-date: matplotlib in /usr/local/lib/python3.6/dist-packages (3.3.4)
    Requirement already satisfied, skipping upgrade: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.19.0)
    Requirement already satisfied, skipping upgrade: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (0.10.0)
    Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.3.1)
    Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.4.7)
    Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.8.1)
    Requirement already satisfied, skipping upgrade: pillow>=6.2.0 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (7.0.0)
    Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib) (1.15.0)
    Collecting bezier
      Downloading https://files.pythonhosted.org/packages/39/2c/5e002f45ddb1c531e9f8cf54fe6dd018da28ed4a1071ababd5333c16819d/bezier-2020.5.19-cp36-cp36m-manylinux2010_x86_64.whl (1.4MB)
         |████████████████████████████████| 1.4MB 4.4MB/s
    Requirement already satisfied: numpy>=1.18.1 in /usr/local/lib/python3.6/dist-packages (from bezier) (1.19.0)
    Installing collected packages: bezier
    Successfully installed bezier-2020.5.19

```
import numpy as np
import scipy.linalg
import sympy
import bezier
from sympy import S, interpolating_spline
from sympy.polys.specialpolys import interpolating_poly
import matplotlib.pyplot as plt
from copy import deepcopy
from scipy.interpolate import lagrange, interp1d, splrep, splev, InterpolatedUnivariateSpline, BPoly, CubicSpline
from scipy.linalg import norm
import pandas as pd
from google.colab import files
%matplotlib inline
```

```
import sympy
import numpy
sympy.__version__, numpy.__version__
```

    ('1.7.1', '1.19.0')

```
sympy.init_printing(use_unicode=False, wrap_line=False, no_global=True)
```

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html#scipy.linalg.norm

https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html

## Многочлен Лагранжа
$$
f(x) = L(x) = \sum_{i=1}^n y_i
\frac{(x - x_0)(x - x_1)...(x - x_{i-1})(x - x_{i+1})...(x - x_n)}{(x_i - x_0)(x_i - x_1)...(x_i - x_{i-1})(x_i - x_{i+1})...(x_i - x_n)},
\quad y_i = f(x_i), \quad i = 0, ..., n.
$$
## Пример 1
Построим полином Лагранжа для данных

https://showdata.gks.ru/report/278934/

файл 6_population.xlsx

Из файла считать данные общего прироста населения РФ в 2013-2019 гг., построить полином Лагранжа в помощью scipy.interpolate

Построить график полученного полинома, отметить соответствующие данные точками.

Для чтения из файла будем пользоваться read_excel

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html

Для загрузки файла с компьютера используем средства colab:

  from google.colab import files

  for fn in uploaded.keys():

    print('User uploaded file "{name}" with length {length} bytes'.format(
          name=fn, length=len(uploaded[fn])))

```
uploaded = files.upload()

for fn in uploaded.keys():
  print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))
```

<input type="file" id="files-c8da3359-ba91-4166-a499-2aa7384b3358" name="files[]" multiple disabled
   style="border:none" />
<output id="result-c8da3359-ba91-4166-a499-2aa7384b3358">
 Upload widget is only available when the cell has been executed in the
 current browser session. Please rerun this cell to enable.
 </output>

    Saving 6_population.xlsx to 6_population.xlsx
    User uploaded file "6_population.xlsx" with length 18683 bytes

После успешной загрузки файла считаем из него данные.

Выведем на экран основные атрибуты полученных данных:

```
rexcel = pd.read_excel('6_population.xlsx', index_col=0, skiprows=[0, 1], dtype={'Name': str, 'Value': int})
display('index', rexcel[:10].index, 'columns', rexcel[:10].columns, 'values', rexcel[:10].values, 'axes', rexcel[:10].axes,
        'ndim', rexcel.ndim, 'size', rexcel.size, 'shape', rexcel.shape, 'empty', rexcel.empty)
```

Выделим данные о приросте численности населения РФ с 2013 по 2019:

```
Y1 = rexcel.values[0,1:]
display(Y1)
X1 = [k for k in range(len(Y1))]
poly1 = lagrange(X1, Y1)
X1ls = np.linspace(X1[0], X1[-1])
plt.plot(X1, Y1, 'ro', X1ls, poly1(X1ls), 'g--')
```

## Пример 2.
Построим кубический сплайн для данных из Примера 1.

Построим на одном графике сплайн и полином Лагранжа Примера 1, отметим на нем  точки данных.

Вычислим значения сплайна и полинома Лагранжа в серединах отрезков между соседними точками данных (т.е. в 0.5, 1.5, 2.5 и т.п.), найдем нормы разности полученных векторов.

```
spl2_cubic = interp1d(X1, Y1, kind='quadratic')
xs = np.linspace(X1[0], X1[-1], 1000)
plt.plot(X1, Y1, 'ro', xs, poly1(xs), 'c-', xs, spl2_cubic(xs), 'm--', lw=3)
X2 = np.arange(0.5, 6.1, 1)
difference2 = poly1(X2) - spl2_cubic(X2)
print(*[round(norm(difference2, item), 4) for item in list(range(-1, 4)) +[np.inf, -np.inf]])
```

```
npsympyLagrange = sympy.lambdify(x, sympyLagrange(x))
```

## Пример 3

Вычислим интеграл от квадрата разности кубического сплайна и полинома Лагранжа, воспользуемся sympy.

```
x = sympy.Symbol('x')
spline3 = interpolating_spline(3, x, list(X1), list(Y1))
sympyLagrange = interpolating_poly(len(X1), x, X1, Y1)

difference3 = (sympyLagrange - spline3)**2
sympy.integrate(difference3, (x, 0, 6))
```

Для того, чтобы выражениями sympy воспользоваться для построения графика matplotlib, воспользуемся инструментом lambdify, возвращающим функцию, в которую можно в качестве аргумента подставлять, например, np.linspace.

```
npsympyLagrange = sympy.lambdify(x, sympyLagrange)

npsympyspline3 = sympy.lambdify(x, spline3)
xs = np.linspace(X1[0], X1[-1], 1000)
plt.plot(X1, Y1, 'ro', xs, npsympyLagrange(xs), 'c-', xs, npsympyspline3(xs), 'm--', lw=3)
```

## Кривые Безье

https://bezier.readthedocs.io/en/stable/python/reference/bezier.curve.html

## Пример 4
Построим кривую Безье с помощью модуля bezier:

https://bezier.readthedocs.io/en/stable/python/reference/bezier.curve.html

Кривая должна аппроксимировать $f(x) = \sin(x/3)$, проходя через точки $\pi/2$ и $3\pi/2$, при этом в этих точках касательная кривой Безье должна совпадать с касательной $f(x)$.

Построим на общем графике $f(x)$ и кривую Безье, отметим узловые точки.

Вначале определим функцию $f4(x)$, затем составим список точек $x_1, x_2, x_3, x_4$ для построения кривой Безье. Концы $x_1, x_4$ берем из условия, это $\pi/2$ и $3\pi/2$, $\pi$ берем из sympy. Промежуточные точки $x_2, x_3$ определим из условия на производные: прямая, соединяющая первую точку со второй должна быть касательной к $f(x)$, так что горизонтальную координату второй точки для кривой Безье возьмем на 1 больше, чем у первой, $x_2 = x_1 + 1$, тогда вертикальная координата второй точки $y_2 = y_1 + f'(x_1)$. Аналогично $x_3 = x_4 - 1$, $y_3 = y_4 - f'(x_4)$.

```
x = sympy.Symbol('x')
def f4(x):
  return sympy.sin(x/3)

X4 = [sympy.pi/2, sympy.pi/2 + 1, 3*sympy.pi/2 - 1, 3*sympy.pi/2]
```

Нужно вычислить производную $f(x)$ в произвольной точке и в точках $x_1, x_4$.

```
df4 = f4(x).diff(x)
Y4 = [f4(X4[0]), f4(X4[0]) + df4.subs(x, X4[0]), f4(X4[-1]) - df4.subs(x, X4[-1]), f4(X4[-1])]
nodes = np.asfarray([X4, Y4])
curve4 = bezier.Curve(nodes, degree=3)
```

Теперь осталось построить график. Для того, чтобы кривая Безье попала на один график с f4, нужно передать методу plot класса кривых Безье параметр ax, отвечающий за систему координат.

```
X4plt = np.linspace(float(X4[0]), float(X4[-1]), 1000)
npf4 = sympy.lambdify(x, f4(x))
fig, ax = plt.subplots()
ax.plot(X4, Y4, 'ro', X4plt, npf4(X4plt), 'k:', lw=3)
curve4.plot(100, color='green', ax=ax)
```

## Пример 5

Для кривой Безье, построенной по точкам A(1, 2), B(3, 3), C(5, 5), D(7, 1) вычислить интеграл квадрата разности с  квадратичным сплайном, соединяющим эти точки, а также нормы разности значений вертикальной координаты кривой Безье и сплайна на сетке с шагом 0.1.

Для вычисления кординат точек на кривой Безье воспользуемся методом implicitize класса кривых Безье и sympy.solve.

```
X5 = [1, 3, 5, 7]
Y5 =  [2, 3, 5, 1]
nodes = np.array([X5, Y5])
curve5 = bezier.Curve(nodes, degree=3)
```

```
curve5_equation = curve5.implicitize()#.evalf(3)
curve5_equation
```

```
curve5_y = sympy.solve(curve5_equation, sympy.Symbol('y'))[0]
curve5_y
```

Теперь построим квадратичный сплайн

```
spl5_cubic = interpolating_spline(2, x, X5, Y5)
```

```
print('integral', round(sympy.integrate((spl5_cubic - curve5_y)**2, (x, X5[0], X5[-1])), 3))

curve5_y_np = sympy.lambdify(x, curve5_y)
spl5_cubic_np = sympy.lambdify(x, spl5_cubic)
X5_1 = np.arange(X5[0] + 0.00001, X5[-1], 0.1)
difference5 = spl5_cubic_np(X5_1) - curve5_y_np(X5_1)
print('norm', *[round(norm(difference5, item), 4) for item in list(range(-1, 4)) +[np.inf, -np.inf]])
```

    integral 5.919
    norm 0.0 60.0 46.6398 7.6933 4.4318 1.7535 0.0


# Computación III - Contenido
### UNIDAD II Análisis Matricial
***
+ Autor: Ulises Olivares
+uolivares@unam.mx
+ Oct 09, 2020
***
### Contenido
***
1. Determinantes
2. Trazas
3. Operaciones elementales
4. Multiplicación de matrices
5. Normas de vectores
6. Normas de matrices
7. Inversa de una Matriz
8. Factorización de matrices
***

## 0. *Numpy - Repaso*

Demostrar el uso de arreglos unidimensionales y bi-dimensionales utilizando numpy.

```
# importar librería de numpy
import numpy as np

# Creara un arreglo bi-dimensional
a = np.array([[1.0,2,3],
              [4,5,6]],
             dtype = np.float16)
# Imprimir matriz
print(a)

# IMPRIMIR ELEMENTOS
print("Dimensiones: " + str(a.ndim))
print("Forma: " + str(a.shape))
print("Tipo de datos: " + str(a.dtype))
```

```
# Creación de arreglos
# Arreglo de 1s
ones = np.ones((3,5), dtype=np.float32)
# Arreglo de 0s
zeros = np.zeros((5,5), dtype=np.float32)

#Impresión de matrices
print(ones)
print(zeros)

print("Matriz 1s tiene: " + str(len(ones)) + " filas y " + str(len(ones[0])) + " columnas")

# Acceso a la matriz usando índices
filas = len(ones)
columnas = len(ones[0])

print("Impresión utilizando índices")
for i in range(filas):
  for j in range(columnas):
    print(ones[i][j], end= " ")
  print(" ")

print("Impresión utilizando elementos")
# Acceso usando elementos
for i in ones:
  for j in i:
    print(j , end =" ")
  print(" ")
```

```
# Generación de un vector a través del uso de rangos
np.arange(1, 100)
```

```
# Concatenando arreglos o matrices
A = np.ones((5,3), dtype = np.float32)
B = np.zeros((5,4), dtype = np.float32)

# concatenar matrices
np.concatenate([A,B], axis = 1)
```

```
# Uso de astype para hacer un cast de tipos de datos
A = np.zeros((5,4), dtype = np.float32)
print(len(A[0]))

for i in range(len(A)):
  for j in range(len(A[0])):
    A[i][j] = np.pi
print(A)

print(A.astype(np.int16))
```

```
# Uso de np.zeros_like y np.ones_like
A = np.ones((4,4))
print(A)
# zeros_like genera un arreglo de las
# mismas dimensiones que el original
B = np.zeros_like(A)
print(B)
```

```
# Generación de arreglos aleatorios
A = np.random.random((5, 5))
print(A)
# Generación de números enteros del 0 al n-1
n = 10
B= np.random.randint(10, size = (7,7))
print(B)
```

```
# Transformaciones de arreglos
A = np.array([1,2,3,4,5,6])
print(A)

# Transformar a 3 filas, 2 columnas
A = A.reshape(3, 2)
print(A)

# Se utiliza -1 para determinar cols
A = A.reshape(2, -1)
print(A)

# Transormar a un arreglo lineal
A = A.ravel()
print(A)
```

```
# Transpuesta de una matriz
A = np.arange(25).reshape(5,5)
print("Matriz Original: \n" + str(A))
# Imprimir matriz transpuesta
print("Matriz Transpuesta: \n" + str(A.T))
# Imprimir matriz
print("Matriz Transpuesta: \n" + str(A.transpose((1,0))))

```

```
# Operadores Matemáticos
A = np.arange(100).reshape(10,10)
print("Primer Matriz: \n" + str(A))

B = np.arange(100, 200).reshape(10, 10)
print("Segunda Matriz: \n" + str(B))

# Calcular la suma de ambas matrices
print("Suma: "  + str(A + B))

# Calcular la resta de ambas matrices
print("resta: "  + str(B - A))

# Calcular la multiplicación de ambas matrices
print("multiplicación: "  + str(A * B))
```

```
# Operadores lógicos
A = np.random.random((5,5))
print("Matriz Original: \n" + str(A))

# Operador lógico
print("Operador lógico a >5 \n" + str(A > 0.5))
```

```
# Operaciones in-situ
A = np.random.randint(10, size= (5,5))
print("Matriz A Original: \n" + str(A))

B = A = np.random.randint(10, size= (5,5))
print("Matriz B Original: \n" + str(B))

# Operación in-situ
A += B
print("Matriz A Modificada: \n" + str(A))
```

```
# Funciones universales
A = np.random.randint(10, 100, size= (5,5))
print("Matriz A Original: \n" + str(A))

# Calcular Raiz cuadrada
print("Raiz Cuadrada: \n" + str(np.sqrt(A)))

# Calcular raiz a elemento de la matriz
print(np.sqrt(A[0][0]))

```

#### Inversa de una matriz

```
# Inversa de una matriz
import numpy as np

# Inicializar una matriz
A = np.random.randint(10, size= (5,5))
print("Matriz A Original: \n" + str(A))

# Calcular matriz inversa
B = np.linalg.inv(A)
print("Matriz Inversa A^-1: \n" + str(B))
```

***
Termina repaso de Numpy
***

## 1. *Determinantes de una matriz*

```
import sys
import os

class Matriz:

    def determinante2x2(self, A):
        if (len(A) == len(A[0]) and len(A) == 2):
            return (A[0][0] * A[1][1]) - (A[0][1] * A[1][0])
        elif len(A) > 2:
            print("La dimensión de la matriz excede el límite de 2x2!, No fue posible calcular el determinante")
        else:
            print("No es posible calcular determinante, filas y columnas son distintos")

    """def determinante3x3(self, A):
        det = 0
        if (len(A) == len(A[0]) and len(A) == 3):
            # Utilizando la primer fila para obtener el determinante
            for j in range(len(A)):
                if (j % 2 ) != 0:
                    a = A[0][j] * -1

                # Inicializar matriz menor
                M = np.zeros(len(n)-1,len(n)-1)
                # Utilizar indexación de numpy para evitar el segundo ciclo.
                for k in range(len(A)):
                    ## Establecer los valores de la matriz menor (M)
                ##detM = self.determinante2x2(M)
                #det += a * detM
        return det"""

    def sumaMatrices(self, A, B):
        # Definir una matriz de las mismas dimensiones de A
        C = np.zeros_like(A);
        # Verificar que las matrices sean iguales
        if (len(A) == len(B) and len(A[0] == len(B[0]))):
            for i in range(len(A)):
                for j in range(len(A[0])):
                    C[i][j] = A[i][j] + B[i][j]
        return C

    def sumaMatricesNp(self, A, B):
        return A + B

    def restaMatricesNp(self, A, B):
        return A - B

    def restaMatrices(self, A, B):
        # Definir una matriz de las mismas dimensiones de A
        C = np.zeros_like(A);
        # Verificar que las matrices sean iguales
        if (len(A) == len(B) and len(A[0] == len(B[0]))):
            for i in range(len(A)):
                for j in range(len(A[0])):
                    C[i][j] = A[i][j] - B[i][j]
        return C

    def multiplicaciónMatrices(self, A, B):
        if(len(A[0]) == len(B)):
            for (for i in range(len(A)):
                for j in range(len(A[0]):
                    for k in rang(len (A[0])):
                        ## Completar multiplicación de matrices
            ######
        else:
            print("No es posible realizar la multiplicación matricial")

    def multiplicacionMatNp(self, A, B):
        retrun A * B

    def divisionMatElement(self, A,B)
        # Definir una matriz de las mismas dimensiones de A
        C = np.zeros_like(A);
        # Verificar que las matrices sean iguales
        if (len(A) == len(B) and len(A[0] == len(B[0]))):
            for i in range(len(A)):
                for j in range(len(A[0])):
                    C[i][j] = A[i][j] / B[i][j]
        return C

    ## Hacer la división matricial utilizando al definición de la inversa de una matriz

```

```
import numpy as np
import sympy
from sympy.abc import x,y,z
from sympy.solvers import solve

# Método main
def main():

    # Definir e inicializar una matriz de 2 x 2
    A = np.array([[1,2,3],
              [3,4,5]],
             dtype = np.float32)
    # inicio, fin, tamaño

    #A = np.random.randint(1, 10, size= (5,5))

    # Creación de objeto
    objMat = Matriz()

    # Cálculo de determinante de 2 x2
    detA = objMat.determinante2x2(A)

    # ¿El cálculo fué válido?
    if detA != None:
        print("El determinante de la matriz A \n", A,   "Es: ", detA)

    # definir matriz de 3x3
    B = np.array([[0,1,3],
                  [5,3,1],
                  [-1,2,0]],
                 dtype = np.float32)
    print("Matriz de 3x3\n", B)
    #detB = objMat.determinante3x3(B)

    # despejando z de la ecuación
    f = 4*x - 3*y + 5*z -3
    f1 = solve(f, z)
    print(f1)

if __name__ == "__main__":
    main()
```

### 2 . *Vectorización de matrices*

La vectorización de matrices consiste en disminuir la dimensionalidad de una matriz. Convertir una matriz a un vector y acceder los elementos a través de un solo índice.

```
import time
import  numpy as np

class Matrix:
    def __init__(self, f, c, M):
        self.f = f
        self.c = c
        self.M = M;

    def suma(self):
        start = time.time()
        z = np.zeros_like(self.M)
        for i in range(len(self.M)):
            for j in range(len(self.M[0])):
                z[i][j] = self.M[i][j] + self.M[i][j]
        end = time.time()
        print("Total matrix sum time: ", (end-start)*1000, "ms")
        return z

    def sumaNumpy(self):
        start = time.time()
        z = np.zeros_like(self.M)
        z = self.M + self.M
        end = time.time()
        print("Total matrix Numpy sum time: ", (end-start)*1000, "ms")
        return z

    def vectorizedSum(self, V):
        start = time.time()
        Z = np.zeros(self.f*self.c)
        for i in range (self.c*self.f):
            Z[i] = V[i] + V[i]
        end = time.time()
        print("Total vectorized matrix sum time: ", (end-start)*1000, "ms")

    def vectorizedSumNumpy(self, V):
        start = time.time()
        Z = V + V
        end = time.time()
        print("Total vectorized matrix Numpy sum time: ", (end-start)*1000, "ms")

    def vectorizedMult(self, V):
        # Implementar multiplicación

def main():
    f = 500
    c = 500

    M = np.random.randint(10, size= (f,c))
    N = np.random.randint(10, size= (f,c))

    # Definición manual de vector
    V1 = np.zeros(f*c)
    for i in range(len(M)):
        for j in range(len(M[0])):
            V1[j + i * len(M[0])] = M[i][j]

    # Vectorización automática
    V = M.reshape(f*c)

    objM = Matrix(f, c, M)
    objM.suma()
    objM.sumaNumpy()
    objM.vectorizedSum(V)
    objM.vectorizedSumNumpy(V)

if __name__ == "__main__":
    main()
```

# *3. Matrices dispersas*

Una matriz dispersa es aquella que contiene muy pocos elementos distintos de 0.

Referencias adicionales:

1. [Sparse Matrix](https://en.wikipedia.org/wiki/Sparse_matrix)

2. [Aplicaciones](https://machinelearningmastery.com/sparse-matrices-for-machine-learning/#:~:text=A%20sparse%20matrix%20is%20a,of%20its%20coefficients%20are%20zero.)

```
import numpy as np
import random
import time

class Matrix:
    def __init__(self, f, c, M):
        self.f = f
        self.c = c
        self.M = M;

    def suma(self):
        start = time.time()
        z = np.zeros_like(self.M)
        for i in range(len(self.M)):
            for j in range(len(self.M[0])):
                z[i][j] = self.M[i][j] + self.M[i][j]
        end = time.time()
        print("Tiempo total suma convencional: ", (end-start)*1000, "ms")
        return z, (end-start)*1000

    def sumaNumpy(self):
        start = time.time()
        z = np.zeros_like(self.M)
        z = self.M + self.M
        end = time.time()
        print("Tiempo total suma NUMPY: ", (end-start)*1000, "ms")
        return z

    def vectorizedSum(self, V):
        start = time.time()
        Z = np.zeros(self.f*self.c)
        for i in range (self.c*self.f):
            Z[i] = V[i] + V[i]
        end = time.time()
        print("Tiempo total suma vectorizada: ", (end-start)*1000, "ms")

    def vectorizedSumNumpy(self, V):
        start = time.time()
        Z = V + V
        end = time.time()
        print("Tiempo total suma vectorizada con NUMPY: ", (end-start)*1000, "ms")

    def vectorizedMult(self, V):
        # Implementar multiplicación
        print("Multiplication")

"""
    Estructura para representar un elemento disperso
"""
class SparseStructure:
    def __init__(self, f, c, value):
        self.fila = f
        self.col = c
        self.value = value

class SparseMatrix:

    """
        Suma de una matriz dispersa
    """
    def suma(self, M):
        z = list()
        start = time.time()

        # Misma matriz
        for i in range(len(M)):
            z.append(M[i].value + M[i].value)

        # Matrices distintas??
        end = time.time()

        print("Tiempo total suma dispersa: ", (end-start)*1000, "ms")
        return z, (end-start)*1000

    """
        Imprimir matriz dispersa
    """
    def printSparseMatrix(self,vectorSparse):
        #Impresión sólo de elementos !=0
        for i in vectorSparse:
            #Acceso eficiente
            print(i.fila, i.col ,i.value)
            # Acceder a la matriz original (No eficiente)
            # print("Valor Matriz: ", M[i.fila][i.col])

    """
        Calcular la transpuesta de una matriz dispersa
    """
    def transposeSparseMatrix(self, vectorSparse):
        transp = list()
        for i in vectorSparse:
            tmp = i.fila
            i.fila = i.col
            i.col = tmp
            transp.append(i)
        return transp

def initMatrix(f, c, M):
        elements = int ((f * c) * 0.1)
        cont = 0
        for i in range(len(M)):
            for j in range(len(M[0])):
                if(random.random() >= 0.75 ):
                    if (cont >= elements):
                        break;
                    M[i][j] = random.randint(1,10);
                    cont += 1

def main():
    # Inicializar variables
    f = 10
    c = 10

    # Definición de matrices
    M = np.zeros((f, c), dtype=np.float)
    N = np.zeros((f, c), dtype=np.float)

    vectorSparseM = list() # Inicializando una lista vacía

    # iniclizar de forma aleatoria el 10 -15% de la matriz
    initMatrix(f, c, M)
    initMatrix(f, c, N)

    # inicializar estructura dispersa
    for i in range(len(M)):
        for j in range(len(M[0])):
            if(M[i][j] != 0):
                vectorSparseM.append(SparseStructure(i,j, M[i][j])); # Se almacena un objeto en cada posición de la lista

    #print("Matriz original")
    print (M)
    print (N)

    objSparse = SparseMatrix()
    printSparseMatrix(vectorSparseM)

    # Suma convencional
    objMat = Matrix(f, c, M)
    M1, sumConv = objMat.suma()

    # Suma dispersa
    objSparse = SparseMatrix()
    M2, sumSparse =  objSparse.suma(vectorSparseM)
    print("SpeedUp: ", sumConv/sumSparse)

    transp = objSparse.transposeSparseMatrix(vectorSparseM)
    print("Matrix transpuesta")
    printSparseMatrix(transp)

    # ¿Transpuesta de una matriz?

if __name__ == "__main__":
    main()
```

# *4. Normas de Matrices y vectores*

```
import time
import numpy as np

class Norm:

    def vectorNorm():
        return 1
    """
        Realizar sumatoria de valores absolutos por fila
    """
    def rowSum(self, M):
        sum = np.zeros(len(M));
        for i in range (len(M)):
            for j in range (len(M[0])):
                sum[i] += abs(M[i][j]);

        return sum;

    """
        Calcular norma l1 e inf
    """
    def matrixNorm(self, M, p):
        if (p == 'inf'):
            sumVec = self.rowSum(M)
        elif (p == 'l1'):
            M = M.T
            sumVec = self.rowSum(M)

        # Elemento mayor
        mayor = -9999
        for i in sumVec:
            if (i > mayor):
                mayor = i

        print("La norma", p, "es", mayor)

def main():
    # Inicializar variables
    f = 10
    c = 10

    # Definición de matrices
    M = np.random.randint(10, size = (f, c))

    objNorm = Norm()

    objNorm.matrixNorm(M, 'inf')

if __name__ == "__main__":
    main()
```

# *5. Inversa de una matriz*

```
import time

# Clase para calcular la inversa de una matriz
class Inverse:

def main():

if __name__ == "__main__":
    main()
```


In this notebook we will look how we can use Cython to generate a faster callback and hopefully shave off some running time from our integration.

```python
import json
import numpy as np
from scipy2017codegen.odesys import ODEsys
from scipy2017codegen.chem import mk_rsys
```

The `ODEsys` class and convenience functions from previous notebook (35) has been put in two modules for easy importing. Recapping what we did last:

```python
watrad_data = json.load(open('../scipy2017codegen/data/radiolysis_300_Gy_s.json'))
watrad = mk_rsys(ODEsys, **watrad_data)
tout = np.logspace(-6, 3, 200)  # close to one hour of operation
c0 = {'H2O': 55.4e3, 'H+': 1e-4, 'OH-': 1e-4}
y0 = [c0.get(symb.name, 0) for symb in watrad.y]
```

```python
%timeit yout, info = watrad.integrate_odeint(tout, y0)
```

so that is the benchmark to beat, we will export our expressions as Cython code. We then subclass `ODEsys` to have it render, compile and import the code:

```python
# %load ../scipy2017codegen/odesys_cython.py
import uuid
import numpy as np
import sympy as sym
import setuptools
import pyximport
from scipy2017codegen import templates
from scipy2017codegen.odesys import ODEsys

pyximport.install()

cython_template = """
cimport numpy as cnp
import numpy as np

def f(cnp.ndarray[cnp.float64_t, ndim=1] y, double t, %(args)s):
    cdef cnp.ndarray[cnp.float64_t, ndim=1] out = np.empty(y.size)
    %(f_exprs)s
    return out

def j(cnp.ndarray[cnp.float64_t, ndim=1] y, double t, %(args)s):
    cdef cnp.ndarray[cnp.float64_t, ndim=2] out = np.empty((y.size, y.size))
    %(j_exprs)s
    return out

"""

class CythonODEsys(ODEsys):

    def setup(self):
        self.mod_name = 'ode_cython_%s' % uuid.uuid4().hex[:10]
        idxs = list(range(len(self.f)))
        subs = {s: sym.Symbol('y[%d]' % i) for i, s in enumerate(self.y)}
        f_exprs = ['out[%d] = %s' % (i, str(self.f[i].xreplace(subs))) for i in idxs]
        j_exprs = ['out[%d, %d] = %s' % (ri, ci, self.j[ri, ci].xreplace(subs)) for ri in idxs for ci in idxs]
        ctx = dict(
            args=', '.join(map(str, self.p)),
            f_exprs = '\n    '.join(f_exprs),
            j_exprs = '\n    '.join(j_exprs),
        )
        open('%s.pyx' % self.mod_name, 'wt').write(cython_template % ctx)
        open('%s.pyxbld' % self.mod_name, 'wt').write(templates.pyxbld % dict(
            sources=[], include_dirs=[np.get_include()],
            library_dirs=[], libraries=[], extra_compile_args=[], extra_link_args=[]
        ))
        mod = __import__(self.mod_name)
        self.f_eval = mod.f
        self.j_eval = mod.j

```

```python
cython_sys = mk_rsys(CythonODEsys, **watrad_data)
```

```python
%timeit cython_sys.integrate(tout, y0)
```

That is a considerable speed up from before. But the solver still has to
allocate memory for creating new arrays at each call, and each evaluation
has to pass the python layer which is now the bottleneck for the integration.

In order to speed up integration further we need to make sure the solver can evaluate the function and Jacobian without calling into Python.

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

Just to see that everything looks alright:

```python
fig, ax = plt.subplots(1, 1, figsize=(14, 6))
cython_sys.plot_result(tout, *cython_sys.integrate_odeint(tout, y0), ax=ax)
ax.set_xscale('log')
ax.set_yscale('log')
```