# Logistic Regression

Researchers are often interested in setting up a model to analyze the relationship between predictors (i.e., independent variables) and it's corresponsing response (i.e., dependent variable). Linear regression is commonly used when the response variable is continuous.  One assumption of linear models is that the residual errors follow a normal distribution. This assumption fails when the response variable is categorical, so an ordinary linear model is not appropriate. This newsletter presents a regression model for response variable that is dichotomous–having two categories. Examples are common: whether a plant lives or dies, whether a survey respondent agrees or disagrees with a statement, or whether an at-risk child graduates or drops out from high school.

In ordinary linear regression, the response variable (Y) is a linear function of the coefficients (B0, B1, etc.) that correspond to the predictor variables (X1, X2, etc.,). A typical model would look like:

    Y = B0 + B1*X1 + B2*X2 + B3*X3 + … + E

For a dichotomous response variable, we could set up a similar linear model to predict individual category memberships if numerical values are used to represent the two categories. Arbitrary values of 1 and 0 are chosen for mathematical convenience. Using the first example, we would assign Y = 1 if a plant lives and Y = 0 if a plant dies.

This linear model does not work well for a few reasons. First, the response values, 0 and 1, are arbitrary, so modeling the actual values of Y is not exactly of interest. Second, it is the probability that each individual in the population responds with 0 or 1 that we are interested in modeling. For example, we may find that plants with a high level of a fungal infection (X1) fall into the category “the plant lives” (Y) less often than those plants with low level of infection. Thus, as the level of infection rises, the probability of plant living decreases.

Thus, we might consider modeling P, the probability, as the response variable. Again, there are problems. Although the general decrease in probability is accompanied by a general increase in infection level, we know that P, like all probabilities, can only fall within the boundaries of 0 and 1. Consequently, it is better to assume that the relationship between X1 and P is sigmoidal (S-shaped), rather than a straight line.

It is possible, however, to find a linear relationship between X1 and function of P. Although a number of functions work, one of the most useful is the logit function. It is the natural log of the odds that Y is equal to 1, which is simply the ratio of the probability that Y is 1 divided by the probability that Y is 0. The relationship between the logit of P and P itself is sigmoidal in shape. The regression equation that results is:

    ln[P/(1-P)] = B0 + B1*X1 + B2*X2 + …

Although the left side of this equation looks intimidating, this way of expressing the probability results in the right side of the equation being linear and looking familiar to us. This helps us understand the meaning of the regression coefficients. The coefficients can easily be transformed so that their interpretation makes sense.

The logistic regression equation can be extended beyond the case of a dichotomous response variable to the cases of ordered categories and polytymous categories (more than two categories).

# Mathematics behind Logistic Regression

## Notation

The problem structure is the classic classification problem. Our data set $\mathcal{D}$ is composed of $N$ samples. Each sample is a tuple containing a feature vector and a label. For any sample $n$ the feature vector is a $d+1$ dimensional column vector denoted by ${\bf x}_n$ with $d$ real-valued components known as features. Samples are represented in homogeneous form with the first component equal to $1$: $x_0=1$. Vectors are bold-faced. The associated label is denoted $y_n$ and can take only two values: $+1$ or $-1$.

$$
\mathcal{D} = \lbrace ({\bf x}_1, y_1), ({\bf x}_2, y_2), ..., ({\bf x}_N, y_N) \rbrace \\
{\bf x}_n = \begin{bmatrix} 1 & x_1 & ... & x_d \end{bmatrix}^T
$$

## Derivation

Despite the name logistic *regression* this is actually a probabilistic classification model. It is also a linear model which can be subjected to nonlinear transforms.

All linear models make use of a "signal" $s$ which is a linear combination of the input vector ${\bf x}$ components weighed by the corresponding components in a weight vector ${\bf w}$.

$$
{\bf w} = \begin{bmatrix} w_0 & w_1 & ... & w_d \end{bmatrix}^T \\
s = w_0 + w_1 x_1 + \;...\; + w_d x_d = \sum_{i=0}^d w_i x_i = {\bf w} \cdot {\bf x} = {\bf w}^T {\bf x}
$$

Note that the homogeneous representation (with the $1$ at the first component) allows us to include a constant offset using a more compact vector-only notation (instead of ${\bf w}^T {\bf x}+b$).

Linear classification passes the signal through a harsh threshold:

$$
h({\bf x}) = \operatorname{sign}(s)
$$

Linear regression uses the signal directly without modification:

$$
h({\bf x}) = s
$$

Logistic regression passes the signal through the logistic/sigmoid but then treats the result as a probability:

$$
h({\bf x}) = \theta(s)
$$

The [logistic function](http://en.wikipedia.org/wiki/Logistic_function) is

$$
\theta(s) = \frac{e^s}{1+e^s} = \frac{1}{1+e^{-s}}
$$

There are many other formulas that can achieve a soft threshold such as the hyperbolic tangent, but this function results in some nice simplification.

We say that the data is generated by a noisy target.

$$
P(y\mid{\bf x})=\begin{cases}
f({\bf x}) & \text{for }y=+1 \\
1-f({\bf x}) & \text{for }y=-1
\end{cases}
$$

With this noisy target we want to learn a hypothesis $h({\bf x})$ that best fits the above noisy target according to some error function.

$$
h({\bf x})=\theta({\bf w}^T {\bf x})\approx f({\bf x})
$$

It's important to note that the data does not tell you the probability of a label, rather it tells what label the sample has after being generated by the target distribution.

## Error Measure

To learn a good hypothesis we want to find a hypothesis parameterization ${\bf w}$ (the weight vector) that minimizes some in-sample error measure $E_\text{in}$.

$$
{\bf w}_h = \underset{{\bf w}}{\operatorname{argmin}} \; E_\text{in}({\bf w})
$$

The error measure will be used is both plausible and nice. It is based on likelihood which is the probability of generating the data given a model.

If our hypothesis is close to our target distribution ($h\approx f$) then we expect that probability of generating the data to be high.

There is some controversy with using likelihood. We are really looking for the most probable hypothesis given the data: $\underset{h}{\operatorname{argmax}} P(h\mid{\bf x})$. The likelihood approach is looking for the hypothesis that makes the data most probable: $\underset{h}{\operatorname{argmax}} P({\bf x}\mid h)$.

The Bayesian approach tackles this issue using Bayes' Theorem but introduces other issues such as choosing priors.

To determine the likelihood we assume the data was generated with our hypothesis $h$:

$$
P(y\mid{\bf x})=\begin{cases}
h({\bf x}) & \text{for }y=+1 \\
1-h({\bf x}) & \text{for }y=-1
\end{cases} \\
$$

where $h({\bf x})=\theta({\bf w}^T {\bf x})$.

We don't want to deal with cases so we take advantage of a nice property of the logistic function: $\theta(-s)=1-\theta(s)$.

$$
\text{if } y = +1 \text{ then } h({\bf x}) = \theta({\bf w}^T {\bf x}) = \theta(y \; {\bf w}^T {\bf x}) \\
\text{if } y = -1 \text{ then } 1 - h({\bf x}) = 1 - \theta({\bf w}^T {\bf x}) = \theta(- {\bf w}^T {\bf x}) = \theta(y \; {\bf w}^T {\bf x}) \\
$$

Using this simplification,

$$
P(y\mid{\bf x})=\theta(y\; {\bf w}^T {\bf x})
$$

The likelihood is defined for a data set $\mathcal{D}$ with $N$ samples given a hypothesis (denoted arbitrarily $g$ here):

$$
L(\mathcal{D} \mid g) =
\prod_{n=1}^{N} P(y_n \mid {\bf x}_n) =
\prod_{n=1}^{N} \theta(y_n \; {\bf w}_g^T {\bf x}_n)
$$

Now finding a good hypothesis is a matter of finding a hypothesis parameterization ${\bf w}$ that maximizes the likelihood.

$$
{\bf w}_h =
\underset{{\bf w}}{\operatorname{argmax}} \; L(\mathcal{D} \mid h) =
\underset{{\bf w}}{\operatorname{argmax}} \; \theta(y_n \; {\bf w}^T {\bf x}_n)
$$

Maximizing the likelihood is equivalent to maximizing the log of the function since the natural logarithm is a monotonically increasing function:

$$
\underset{{\bf w}}{\operatorname{argmax}} \; \ln \left( \prod_{n=1}^{N} \theta(y_n \; {\bf w}^T {\bf x}_n) \right)
$$

We can maximize the above proportional to a constant as well so we'll tack on a $\frac{1}{N}$:

$$
\underset{{\bf w}}{\operatorname{argmax}} \; \frac{1}{N} \ln \left( \prod_{n=1}^{N} \theta(y_n \; {\bf w}^T {\bf x}_n) \right)
$$

Now maximizing that is the same as minimizing its negative:

$$
\underset{{\bf w}}{\operatorname{argmin}} \left[ -\frac{1}{N} \ln \left( \prod_{n=1}^{N} \theta(y_n \; {\bf w}^T {\bf x}_n) \right) \right]
$$

If we move the negative into the log and the log into the product we turn the product into a sum of logs:

$$
\underset{{\bf w}}{\operatorname{argmin}} \;\frac{1}{N} \sum_{n=1}^{N} \ln \left( \frac{1}{\theta(y_n \; {\bf w}^T {\bf x}_n)} \right)
$$

Expanding the logistic function,

$$
\underset{{\bf w}}{\operatorname{argmin}} \;\frac{1}{N} \sum_{n=1}^{N} \ln \left( 1 + e^{y_n \; {\bf w}^T {\bf x}_n} \right)
$$

Now we have a much nicer form for the error measure known as the "cross-entropy" error.

$$
E_\text{in}({\bf w}) = \frac{1}{N} \sum_{n=1}^{N} \ln \left( 1+e^{-y_n \; {\bf w}^T {\bf x}_n} \right)
$$

This is nice because it can be interpreted as the average point error where the point error function is

$$
e(h({\bf x}_n), y_n) = \ln \left( 1+e^{-y_n \; {\bf w}^T {\bf x}_n} \right) \\
E_\text{in}({\bf w}) = \frac{1}{N} \sum_{n=1}^{N} e(h({\bf x}_n), y_n)
$$

## OR in Simple Equation ,

So to learn a hypothesis we'll want to perform the following optimization:

$$
{\bf w}_h =
\underset{{\bf w}}{\operatorname{argmin}} \; E_\text{in}({\bf w}) =
\underset{{\bf w}}{\operatorname{argmin}} \;\frac{1}{N} \sum_{n=1}^{N} \ln \left( 1 + e^{y_n \; {\bf w}^T {\bf x}_n} \right)
$$

## Learning Algorithm

The learning algorithm is how we search the set of possible hypotheses (hypothesis space $\mathcal{H}$) for the best parameterization (in this case the weight vector ${\bf w}$). This search is an optimization problem looking for the hypothesis that optimizes an error measure.

There is no sophisticted, closed-form solution like least-squares linear, so we will use gradient descent instead. Specifically we will use batch gradient descent which calculates the gradient from all data points in the data set.

Luckily, our "cross-entropy" error measure is convex so there is only one minimum. Thus the minimum we arrive at is the global minimum.

Gradient descent is a general method and requires twice differentiability for smoothness. It updates the parameters using a first-order approximation of the error surface.

$$
{\bf w}_{i+1} = {\bf w}_i + \nabla E_\text{in}({\bf w}_i)
$$

To learn we're going to minimize the following error measure using batch gradient descent.

$$
e(h({\bf x}_n), y_n) = \ln \left( 1+e^{-y_n \; {\bf w}^T {\bf x}_n} \right) \\
E_\text{in}({\bf w}) = \frac{1}{N} \sum_{n=1}^{N} e(h({\bf x}_n), y_n) = \frac{1}{N} \sum_{n=1}^{N} \ln \left( 1+e^{-y_n \; {\bf w}^T {\bf x}_n} \right)
$$

We'll need the derivative of the point loss function and possibly some abuse of notation.

$$
\frac{d}{d{\bf w}} e(h({\bf x}_n), y_n)
= \frac{-y_n \; {\bf x}_n \; e^{-y_n {\bf w}^T {\bf x}_n}}{1 + e^{-y_n {\bf w}^T {\bf x}_n}}
= -\frac{y_n \; {\bf x}_n}{1 + e^{y_n {\bf w}^T {\bf x}_n}}
$$

With the point loss derivative we can determine the gradient of the in-sample error:

$$
\begin{align}
\nabla E_\text{in}({\bf w})
&= \frac{d}{d{\bf w}} \left[ \frac{1}{N} \sum_{n=1}^N e(h({\bf x}_n), y_n) \right] \\
&= \frac{1}{N} \sum_{n=1}^N \frac{d}{d{\bf w}} e(h({\bf x}_n), y_n) \\
&= \frac{1}{N} \sum_{n=1}^N \left( - \frac{y_n \; {\bf x}_n}{1 + e^{y_n {\bf w}^T {\bf x}_n}} \right) \\
&= - \frac{1}{N} \sum_{n=1}^N \frac{y_n \; {\bf x}_n}{1 + e^{y_n {\bf w}^T {\bf x}_n}} \\
\end{align}
$$

Our weight update rule per batch gradient descent becomes

$$
\begin{align}
{\bf w}_{i+1} &= {\bf w}_i - \eta \; \nabla E_\text{in}({\bf w}_i) \\
&= {\bf w}_i - \eta \; \left( - \frac{1}{N} \sum_{n=1}^N \frac{y_n \; {\bf x}_n}{1 + e^{y_n {\bf w}_i^T {\bf x}_n}} \right) \\
&= {\bf w}_i + \eta \; \left( \frac{1}{N} \sum_{n=1}^N \frac{y_n \; {\bf x}_n}{1 + e^{y_n {\bf w}_i^T {\bf x}_n}} \right) \\
\end{align}
$$

where $\eta$ is our learning rate.

### Enough with the theory, now jump to the implimentation. We will look at 2 libraries for the same.

## Logistic Regression with statsmodel

We'll be using the same dataset as UCLA's Logit Regression tutorial to explore logistic regression in Python. Our goal will be to identify the various factors that may influence admission into graduate school.

The dataset contains several columns which we can use as predictor variables:

   * gpa
   * gre score
   * rank or prestige of an applicant's undergraduate alma mater
   * The fourth column, admit, is our binary target variable. It indicates whether or not a candidate was admitted our not.

```python
import pandas as pd
import statsmodels.api as sm
import pylab as pl
import numpy as np
```

```python
# read the data in
df = pd.read_csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
```

```python
df.head()
```

      admit
      gre
      gpa
      rank

      0
      0
      380
      3.61
      3

      1
      1
      660
      3.67
      3

      2
      1
      800
      4.00
      1

      3
      1
      640
      3.19
      4

      4
      0
      520
      2.93
      4

```python
# rename the 'rank' column because there is also a DataFrame method called 'rank'
df.columns = ["admit", "gre", "gpa", "prestige"]
```

### Summary Statistics & Looking at the data
Now that we've got everything loaded into Python and named appropriately let's take a look at the data. We can use the pandas function which describes a summarized view of everything. There's also function for calculating the standard deviation, std.

A feature I really like in pandas is the pivot_table/crosstab aggregations. crosstab makes it really easy to do multidimensional frequency tables. You might want to play around with this to look at different cuts of the data.

```python
df.describe()
```

      admit
      gre
      gpa
      prestige

      count
      400.000000
      400.000000
      400.000000
      400.00000

      mean
      0.317500
      587.700000
      3.389900
      2.48500

      std
      0.466087
      115.516536
      0.380567
      0.94446

      min
      0.000000
      220.000000
      2.260000
      1.00000

      25%
      0.000000
      520.000000
      3.130000
      2.00000

      50%
      0.000000
      580.000000
      3.395000
      2.00000

      75%
      1.000000
      660.000000
      3.670000
      3.00000

      max
      1.000000
      800.000000
      4.000000
      4.00000

```python
# frequency table cutting presitge and whether or not someone was admitted
pd.crosstab(df['admit'], df['prestige'], rownames=['admit'])
```

      prestige
      1
      2
      3
      4

      admit

      0
      28
      97
      93
      55

      1
      33
      54
      28
      12

```python
# plot all of the columns
df.hist()
pl.show()
```

 ### dummy variables
pandas gives you a great deal of control over how categorical variables can be represented. We're going dummify the "prestige" column using get_dummies.

get_dummies creates a new DataFrame with binary indicator variables for each category/option in the column specified. In this case, prestige has four levels: 1, 2, 3 and 4 (1 being most prestigious). When we call get_dummies, we get a dataframe with four columns, each of which describes one of those levels.

```python
# dummify rank
dummy_ranks = pd.get_dummies(df['prestige'], prefix='prestige')
dummy_ranks.head()
```

      prestige_1
      prestige_2
      prestige_3
      prestige_4

      0
      0
      0
      1
      0

      1
      0
      0
      1
      0

      2
      1
      0
      0
      0

      3
      0
      0
      0
      1

      4
      0
      0
      0
      1

```python
# create a clean data frame for the regression
cols_to_keep = ['admit', 'gre', 'gpa']
data = df[cols_to_keep].join(dummy_ranks.ix[:, 'prestige_2':])
data.head()
```

      admit
      gre
      gpa
      prestige_2
      prestige_3
      prestige_4

      0
      0
      380
      3.61
      0
      1
      0

      1
      1
      660
      3.67
      0
      1
      0

      2
      1
      800
      4.00
      0
      0
      0

      3
      1
      640
      3.19
      0
      0
      1

      4
      0
      520
      2.93
      0
      0
      1

```python
# manually add the intercept
data['intercept'] = 1.0
```

Once that's done, we merge the new dummy columns with the original dataset and get rid of the prestige column which we no longer need.

Lastly we're going to add a constant term for our logistic regression. The statsmodels function we would use requires intercepts/constants to be specified explicitly.

### Performing the regression
Actually doing the logistic regression is quite simple. Specify the column containing the variable you're trying to predict followed by the columns that the model should use to make the prediction.

In our case we'll be predicting the admit column using gre, gpa, and the prestige dummy variables prestige_2, prestige_3 and prestige_4. We're going to treat prestige_1 as our baseline and exclude it from our fit. This is done to prevent multicollinearity, or the dummy variable trap caused by including a dummy variable for every single category.

```python
train_cols = data.columns[1:]
# Index([gre, gpa, prestige_2, prestige_3, prestige_4], dtype=object)

logit = sm.Logit(data['admit'], data[train_cols])

# fit the model
result = logit.fit()
```

    Optimization terminated successfully.
             Current function value: 0.573147
             Iterations 6

Since we're doing a logistic regression, we're going to use the statsmodels Logit function. For details on other models available in statsmodels, check out their docs here.

### Interpreting the results
One of my favorite parts about statsmodels is the summary output it gives. If you're coming from R, I think you'll like the output and find it very familiar too.

```python
result.summary()
```

Logit Regression Results

  Dep. Variable:	       admit	        No. Observations:  	     400

  Model:	               Logit	        Df Residuals:      	     394

  Method:	               MLE	         Df Model:          	       5

  Date:	          Sat, 09 Sep 2017	   Pseudo R-squ.:     	  0.08292

  Time:	              20:24:18	       Log-Likelihood:    	  -229.26

  converged:	           True	         LL-Null:           	  -249.99

   	                       	          LLR p-value:       	 7.578e-08

       	         coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  gre	            0.0023	     0.001	     2.070	  0.038	     0.000	     0.004

  gpa	            0.8040	     0.332	     2.423	  0.015	     0.154	     1.454

  prestige_2	    -0.6754	     0.316	    -2.134	  0.033	    -1.296	    -0.055

  prestige_3	    -1.3402	     0.345	    -3.881	  0.000	    -2.017	    -0.663

  prestige_4	    -1.5515	     0.418	    -3.713	  0.000	    -2.370	    -0.733

  intercept	     -3.9900	     1.140	    -3.500	  0.000	    -6.224	    -1.756

# Logistic Regression with scikit-learn

## Dataset

The dataset I chose is the [affairs dataset](http://statsmodels.sourceforge.net/stable/datasets/generated/fair.html) that comes with [Statsmodels](http://statsmodels.sourceforge.net/). It was derived from a survey of women in 1974 by Redbook magazine, in which married women were asked about their participation in extramarital affairs. More information about the study is available in a [1978 paper](http://fairmodel.econ.yale.edu/rayfair/pdf/1978a200.pdf) from the Journal of Political Economy.

## Description of Variables

The dataset contains 6366 observations of 9 variables:

* `rate_marriage`: woman's rating of her marriage (1 = very poor, 5 = very good)
* `age`: woman's age
* `yrs_married`: number of years married
* `children`: number of children
* `religious`: woman's rating of how religious she is (1 = not religious, 4 = strongly religious)
* `educ`: level of education (9 = grade school, 12 = high school, 14 = some college, 16 = college graduate, 17 = some graduate school, 20 = advanced degree)
* `occupation`: woman's occupation (1 = student, 2 = farming/semi-skilled/unskilled, 3 = "white collar", 4 = teacher/nurse/writer/technician/skilled, 5 = managerial/business, 6 = professional with advanced degree)
* `occupation_husb`: husband's occupation (same coding as above)
* `affairs`: time spent in extra-marital affairs

## Problem Statement

I decided to treat this as a classification problem by creating a new binary variable `affair` (did the woman have at least one affair?) and trying to predict the classification for each woman.

## Import modules

```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
from patsy import dmatrices
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import train_test_split
from sklearn import metrics
from sklearn.cross_validation import cross_val_score
```

    /home/abhay/.virtualenvs/Data_Science/local/lib/python2.7/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools
    /home/abhay/.virtualenvs/Data_Science/local/lib/python2.7/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)

## Data Pre-Processing

First, let's load the dataset and add a binary `affair` column.

```python
# load dataset
dta = sm.datasets.fair.load_pandas().data

dta.head()
```

      rate_marriage
      age
      yrs_married
      children
      religious
      educ
      occupation
      occupation_husb
      affairs

      0
      3.0
      32.0
      9.0
      3.0
      3.0
      17.0
      2.0
      5.0
      0.111111

      1
      3.0
      27.0
      13.0
      3.0
      1.0
      14.0
      3.0
      4.0
      3.230769

      2
      4.0
      22.0
      2.5
      0.0
      1.0
      16.0
      3.0
      5.0
      1.400000

      3
      4.0
      37.0
      16.5
      4.0
      3.0
      16.0
      5.0
      5.0
      0.727273

      4
      5.0
      27.0
      9.0
      1.0
      1.0
      14.0
      3.0
      4.0
      4.666666

```python
# add "affair" column: 1 represents having affairs, 0 represents not
dta['affair'] = (dta.affairs > 0).astype(int)
dta.head()
```

      rate_marriage
      age
      yrs_married
      children
      religious
      educ
      occupation
      occupation_husb
      affairs
      affair

      0
      3.0
      32.0
      9.0
      3.0
      3.0
      17.0
      2.0
      5.0
      0.111111
      1

      1
      3.0
      27.0
      13.0
      3.0
      1.0
      14.0
      3.0
      4.0
      3.230769
      1

      2
      4.0
      22.0
      2.5
      0.0
      1.0
      16.0
      3.0
      5.0
      1.400000
      1

      3
      4.0
      37.0
      16.5
      4.0
      3.0
      16.0
      5.0
      5.0
      0.727273
      1

      4
      5.0
      27.0
      9.0
      1.0
      1.0
      14.0
      3.0
      4.0
      4.666666
      1

## Data Exploration

```python
dta.groupby('affair').mean()
```

      rate_marriage
      age
      yrs_married
      children
      religious
      educ
      occupation
      occupation_husb
      affairs

      affair

      0
      4.329701
      28.390679
      7.989335
      1.238813
      2.504521
      14.322977
      3.405286
      3.833758
      0.000000

      1
      3.647345
      30.537019
      11.152460
      1.728933
      2.261568
      13.972236
      3.463712
      3.884559
      2.187243

We can see that on average, women who have affairs rate their marriages lower, which is to be expected. Let's take another look at the `rate_marriage` variable.

```python
dta.groupby('rate_marriage').mean()
```

      age
      yrs_married
      children
      religious
      educ
      occupation
      occupation_husb
      affairs
      affair

      rate_marriage

      1.0
      33.823232
      13.914141
      2.308081
      2.343434
      13.848485
      3.232323
      3.838384
      1.201671
      0.747475

      2.0
      30.471264
      10.727011
      1.735632
      2.330460
      13.864943
      3.327586
      3.764368
      1.615745
      0.635057

      3.0
      30.008056
      10.239174
      1.638469
      2.308157
      14.001007
      3.402820
      3.798590
      1.371281
      0.550856

      4.0
      28.856601
      8.816905
      1.369536
      2.400981
      14.144514
      3.420161
      3.835861
      0.674837
      0.322926

      5.0
      28.574702
      8.311662
      1.252794
      2.506334
      14.399776
      3.454918
      3.892697
      0.348174
      0.181446

An increase in `age`, `yrs_married`, and `children` appears to correlate with a declining marriage rating.

## Data Visualization

```python
# show plots in the notebook
%matplotlib inline
```

Let's start with histograms of education and marriage rating.

```python
# histogram of education
dta.educ.hist()
plt.title('Histogram of Education')
plt.xlabel('Education Level')
plt.ylabel('Frequency')
```

```python
# histogram of marriage rating
dta.rate_marriage.hist()
plt.title('Histogram of Marriage Rating')
plt.xlabel('Marriage Rating')
plt.ylabel('Frequency')
```

Let's take a look at the distribution of marriage ratings for those having affairs versus those not having affairs.

```python
# barplot of marriage rating grouped by affair (True or False)
pd.crosstab(dta.rate_marriage, dta.affair.astype(bool)).plot(kind='bar')
plt.title('Marriage Rating Distribution by Affair Status')
plt.xlabel('Marriage Rating')
plt.ylabel('Frequency')
```

Let's use a stacked barplot to look at the percentage of women having affairs by number of years of marriage.

```python
affair_yrs_married = pd.crosstab(dta.yrs_married, dta.affair.astype(bool))
affair_yrs_married.div(affair_yrs_married.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Affair Percentage by Years Married')
plt.xlabel('Years Married')
plt.ylabel('Percentage')
```

## Prepare Data for Logistic Regression

To prepare the data, I want to add an intercept column as well as dummy variables for `occupation` and `occupation_husb`, since I'm treating them as categorial variables. The dmatrices function from the [patsy module](http://patsy.readthedocs.org/en/latest/) can do that using formula language.

```python
# create dataframes with an intercept column and dummy variables for
# occupation and occupation_husb
y, X = dmatrices('affair ~ rate_marriage + age + yrs_married + children + \
                  religious + educ + C(occupation) + C(occupation_husb)',
                  dta, return_type="dataframe")
X.columns
```

    Index(['Intercept', 'C(occupation)[T.2.0]', 'C(occupation)[T.3.0]',
           'C(occupation)[T.4.0]', 'C(occupation)[T.5.0]', 'C(occupation)[T.6.0]',
           'C(occupation_husb)[T.2.0]', 'C(occupation_husb)[T.3.0]',
           'C(occupation_husb)[T.4.0]', 'C(occupation_husb)[T.5.0]',
           'C(occupation_husb)[T.6.0]', 'rate_marriage', 'age', 'yrs_married',
           'children', 'religious', 'educ'],
          dtype='object')

The column names for the dummy variables are ugly, so let's rename those.

```python
# fix column names of X
X = X.rename(columns = {'C(occupation)[T.2.0]':'occ_2',
                        'C(occupation)[T.3.0]':'occ_3',
                        'C(occupation)[T.4.0]':'occ_4',
                        'C(occupation)[T.5.0]':'occ_5',
                        'C(occupation)[T.6.0]':'occ_6',
                        'C(occupation_husb)[T.2.0]':'occ_husb_2',
                        'C(occupation_husb)[T.3.0]':'occ_husb_3',
                        'C(occupation_husb)[T.4.0]':'occ_husb_4',
                        'C(occupation_husb)[T.5.0]':'occ_husb_5',
                        'C(occupation_husb)[T.6.0]':'occ_husb_6'})
```

We also need to flatten `y` into a 1-D array, so that scikit-learn will properly understand it as the response variable.

```python
# flatten y into a 1-D array
y = np.ravel(y)
```

## Logistic Regression

Let's go ahead and run logistic regression on the entire data set, and see how accurate it is!

```python
# instantiate a logistic regression model, and fit with X and y
model = LogisticRegression()
model = model.fit(X, y)

# check the accuracy on the training set
model.score(X, y)
```

    0.72588752748978946

73% accuracy seems good, but what's the null error rate?

```python
# what percentage had affairs?
y.mean()
```

    0.32249450204209867

Only 32% of the women had affairs, which means that you could obtain 68% accuracy by always predicting "no". So we're doing better than the null error rate, but not by much.

Let's examine the coefficients to see what we learn.

```python
# examine the coefficients
X.columns, np.transpose(model.coef_)
```

    (Index(['Intercept', 'occ_2', 'occ_3', 'occ_4', 'occ_5', 'occ_6', 'occ_husb_2',
            'occ_husb_3', 'occ_husb_4', 'occ_husb_5', 'occ_husb_6', 'rate_marriage',
            'age', 'yrs_married', 'children', 'religious', 'educ'],
           dtype='object'), array([[ 1.48986215],
            [ 0.18804152],
            [ 0.49891971],
            [ 0.25064099],
            [ 0.83897693],
            [ 0.8340083 ],
            [ 0.19057989],
            [ 0.29777984],
            [ 0.16135349],
            [ 0.18771784],
            [ 0.19394847],
            [-0.70311586],
            [-0.05841769],
            [ 0.10567657],
            [ 0.01692027],
            [-0.37113376],
            [ 0.00401557]]))

Increases in marriage rating and religiousness correspond to a decrease in the likelihood of having an affair. For both, wife's occupation and the husband's occupation, the lowest likelihood of having an affair corresponds to the baseline occupation (student), since all of the dummy coefficients are positive.

## Model Evaluation Using a Validation Set

So far, we have trained and tested on the same set. Let's instead split the data into a training set and a testing set.

```python
# evaluate the model by splitting into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
model2 = LogisticRegression()
model2.fit(X_train, y_train)
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False)

We now need to predict class labels for the test set. We will also generate the class probabilities, just to take a look.

```python
# predict class labels for the test set
predicted = model2.predict(X_test)
predicted
```

    array([ 1.,  0.,  0., ...,  0.,  0.,  0.])

```python
# generate class probabilities
probs = model2.predict_proba(X_test)
probs
```

    array([[ 0.35148525,  0.64851475],
           [ 0.90955539,  0.09044461],
           [ 0.72569407,  0.27430593],
           ...,
           [ 0.55730636,  0.44269364],
           [ 0.81211049,  0.18788951],
           [ 0.74732836,  0.25267164]])

As you can see, the classifier is predicting a 1 (having an affair) any time the probability in the second column is greater than 0.5.

Now let's generate some evaluation metrics.

```python
# generate evaluation metrics
print(metrics.accuracy_score(y_test, predicted))

```

    0.729319371728
    0.745948078253

The accuracy is 73%, which is the same as we experienced when training and predicting on the same data.

We can also see the confusion matrix and a classification report with other metrics.

```python
print(metrics.confusion_matrix(y_test, predicted))
print(metrics.classification_report(y_test, predicted))
```

    [[1169  134]
     [ 383  224]]
                 precision    recall  f1-score   support

            0.0       0.75      0.90      0.82      1303
            1.0       0.63      0.37      0.46       607

    avg / total       0.71      0.73      0.71      1910

## Model Evaluation Using Cross-Validation

Now let's try 10-fold cross-validation, to see if the accuracy holds up more rigorously.

```python
# evaluate the model using 10-fold cross-validation
scores = cross_val_score(LogisticRegression(), X, y, scoring='accuracy', cv=10)
scores, scores.mean()
```

    (array([ 0.72100313,  0.70219436,  0.73824451,  0.70597484,  0.70597484,
             0.72955975,  0.7327044 ,  0.70440252,  0.75157233,  0.75      ]),
     0.7241630685514876)

Looks good. It's still performing at 73% accuracy.

## Predicting the Probability of an Affair

Just for fun, let's predict the probability of an affair for a random woman not present in the dataset. She's a 25-year-old teacher who graduated from college, has been married for 3 years. She has 1 child, rates herself as strongly religious, rates her marriage as fair, and her husband is a farmer.

```python
model.predict_proba(np.array([[1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 3, 25, 3, 1, 4,
                              16]]))
```

    array([[ 0.77472403,  0.22527597]])

The predicted probability of an affair is 23%.


```python
%pylab inline
import sympy as sym
```

    Populating the interactive namespace from numpy and matplotlib

We'll need to compute

$$
A_{ij} := \int_a^b v_j' v_i'
$$

and

$$
f_i := \int_a^b v_i f
$$

We split the interval in $M$ elements (segments), with $M+1$ vertices (element boundaries). Define the element boundaries $q := [a, q_1, q_2, ..., q_{M-1}, b]$, and the elements sizes $h_k := v_{k+1}-q_k$.
In $[a,b]$ we have a total of $N$ basis functions, which are piecewise polynomials of degree $d$, so that in each segment there are always at most $d+1$ non-zero basis functions, and $d+1$ support points.

We assume that each support point can be interpreted as the image of reference support points $\hat a_\alpha$, where $\alpha \in [0,d]$, throught the mapping $F_k(s) := q_k+h_k s$ which maps $[0,1]$ to $T_k := [q_k, q_{k+1}]$.

Similarly, every global basis function $v_j$ can be seen as the composition of a *reference* basis function $\hat v_\alpha$ defined on the reference interval $[0,1]$, and the inverse of the element transformation $F_k(s) := q_k+h_k s$ which maps $[0,1]$ to $[q_k, q_{k+1}]$, that is:

$$
v_i(F_k(s)) = P_{ki\alpha} \hat v_\alpha(s)
$$

Where $P_{i\alpha}$ represents the numbering of the local basis function $\alpha \in [0,d]$, i.e., given the $(d+1)$ reference basis functions in $0,1$ of degree $d$, $P_{ki\alpha}$.

We will implement the numbering as a matrix $P \in R^{M,d+1}$, which returns the global index $i$, given the local element index $k$ and the local basis function index $i$, i.e.,

$$
v_{P_{k\alpha}}(F_k(s)) = \hat v_\alpha(s)
$$

Notice that, if we want to compute the derivative w.r.t. $x$ of $v_i$, computed in $F_k(s)$, as a function of the derivative of $\hat v_\alpha$ w.r.t. $s$, we need to take into account also the derivative of $F_k$, i.e., since $(v\circ F_k)' = (v' \circ F_k) F_k'$, we have

$$
v'_{P_{k\alpha}}(F_k(s)) = \hat v'_\alpha(s)/F'_k(s) = \hat v'_\alpha(s)/h_k
$$

```python
# Let's define the domain discretisazion. In 1D, this is just a set of vertices, identifying edges
a = 0
b = 1
M = 4 # Number of elements
degree = 3
# Make sure we don't choose degree = 0. This won't work. Piecewise constants cannot be continuous
assert degree > 0

# We now choose the number of quadrature points, in order to integrate *exactly*
# both (v_i, v_j) and (v'_i, v'_j)
n_quadrature_points = 2*degree+1

# To get a continuous space, we construct piecewise polynomials with support points on the boundary of the
# elements. If the degree is greater than 1, then we pick (d-1) equispaced points in the interior of the
# elements as additional support points.

# Notice that the total number of degrees of freedom is equal to the number of vertices (M+1) plus the
# number of *interior*  basis functions (d+1-2), that is: (M+1) + (d-1)*M = M*d+1
N = M*degree+1

ref_vertices = linspace(0,1,degree+1)

vertices = linspace(a,b,M+1) # Vertices of our triangulation
```

```python
# The reference element is [0,1]. We construct the mappings, the determinant of their Jacobians, and the
# reference Basis functions

def mapping(q, i):
    """
    Returns the mapping from [0,1] to T_k := [q[k], q[k+1]]
    """
    assert i < len(q)-1
    assert i >= 0
    return lambda x: q[i]+x*(q[i+1]-q[i])

def mapping_J(q,i):
    assert i < len(q)-1
    assert i >= 0
    return (q[i+1]-q[i])

def lagrange_basis(q, i):
    assert i < len(q)
    assert i >= 0
    return lambda x: prod([(x-q[j])/(q[i]-q[j]) for j in range(len(q)) if i!=j], axis=0)

# Workaround, to allow lambdify to work also on constant expressions
def np_lambdify(varname, func):
    lamb = sym.lambdify(varname, func, modules=['numpy'])
    if func.is_constant():
        return lambda t: full_like(t, lamb(t))
    else:
        return lambda t: lamb(np.array(t))

def lagrange_basis_derivative(q,i,order=1):
    t = sym.var('t')
    return np_lambdify(t, lagrange_basis(q,i)(t).diff(t,order))
```

```python
# Let's check that, on each element, F_k(0) = q[k] and F_k(1) = q[k+1]
assert abs(array([mapping(vertices, i)(0) for i in range(M)]) - vertices[:-1]).max() < 1e-16
assert abs(array([mapping(vertices, i)(1) for i in range(M)]) - vertices[1:]).max() < 1e-16
```

```python
x = linspace(0,1,51)

V = array([lagrange_basis(ref_vertices,i)(x) for i in range(degree+1)]).T
Vp = array([lagrange_basis_derivative(ref_vertices,i)(x) for i in range(degree+1)]).T
_ = [plot(x, V)]
show()
_ = [plot(x, Vp)]
show()
```

```python
# Now construct an interpolatory quadrature formula on [0,1]
q, w = numpy.polynomial.legendre.leggauss(n_quadrature_points)

q = (q+1)/2
w = w/2
```

```python
# And build a global numbering of the basis functions i = P[k,alpha]. Keep in mind that, to ensure continuity,
# we identify the global index the first basis function of each element, with the global index of the
# last basis function of the previous element

P = zeros((M,degree+1), dtype=int)

for k in range(M):
    start = k*degree
    P[k] = array(range(start,start+degree+1))

assert P.max() == N-1
print(P)
```

    [[ 0  1  2  3]
     [ 3  4  5  6]
     [ 6  7  8  9]
     [ 9 10 11 12]]

```python
# Now we build, for each segment, the transformation of quadrature points and weights, so that we can
# integrate the rhs and the matrices

Q = array([mapping(vertices,k)(q) for k in range(M)])
JxW = array([mapping_J(vertices,k)*w for k in range(M)])
```

```python
# Let's test that everything works: the integral between 0 and 1 of the function f(x) = x should return 0.5.
# Then sum_k q[k].dot(w_k) should be 0.5.

integral = 0
for k in range(M):
    integral = integral + Q[k].dot(JxW[k])

assert abs(integral - .5) < 1e-16

#same as
einsum('kq,kq', Q, JxW)
```

    0.49999999999999994

```python
# Construct the matrix B[k,j,i], defined as the value of v_i(F_k(x[j]))
B = zeros((M, len(x), N))

for k in range(M):
    B[k,:,P[k]] = V.T

# To evaluate functions and to do some plotting, also gather together all F_k(x[j]) in X[k,j]
X = array([mapping(vertices,k)(x) for k in range(M)])
```

```python
X.shape, B.shape
```

    ((4, 51), (4, 51, 13))

```python
# Reshaping X and B, we can use them to compute piecewise interpolation
X = X.flatten()
B = B.reshape((len(X),-1))

X.shape, B.shape
```

    ((204,), (204, 13))

```python
_ = plot(X, B[:,0:3])
_ = plot(vertices, 0*vertices,'ro')
```

```python
# Notice that the global support points are the image through F_k of the reference support points,
# Numbered according to the matrix P, i.e..

support_points = zeros((N,))

for k in range(M):
    support_points[P[k]] = mapping(vertices,k)(ref_vertices)

# If we chose equispaced vertices, they should be identical to linspace(a,b,N):
abs(support_points - linspace(a,b,N)).max()
```

    1.1102230246251565e-16

```python
# Let's use the runge function as an example, and compute its *piecewise* polynomial interpolation
def runge(x):
    return 1/(1+50*(x-.5)**2)
```

```python
plot(X, runge(X))
plot(support_points, runge(support_points), 'ro')
plot(X, B.dot(runge(support_points)),'r')
```

```python
# Construct a arrays Bq and Bprimeq: Bq[k,j,i] is v_i(T_k(q[j])),
# and Bprimeq[k,j,i] is v'_i(T_k(q[j]))/T'_k(q[j])
Bq = zeros((M, n_quadrature_points, N))
Bprimeq = zeros((M, n_quadrature_points, N))

Vq = array([lagrange_basis(ref_vertices,i)(q) for i in range(degree+1)]).T
Vprimeq = array([lagrange_basis_derivative(ref_vertices,i)(q) for i in range(degree+1)]).T

for k in range(M):
    Bq[k,:,P[k]] = Vq.T
    Bprimeq[k,:,P[k]] = Vprimeq.T/mapping_J(vertices,k)

Xq = Q.flatten()
Bq = Bq.reshape((len(Xq),-1))
Bprimeq = Bprimeq.reshape((len(Xq),-1))
JxWq = JxW.flatten()
```

```python
# Now compute the integrals for the mass and stiffness matrices
mass_matrix = einsum('qi,qj,q',Bq,Bq,JxWq)
stiffness_matrix = einsum('qi,qj,q',Bprimeq,Bprimeq,JxWq)
```

```python
# Notice that the stiffness matrix is non-invertible (Pure neumann problem in H1 is not uniquely solvable)
# In fact we are not imposing the boundary conditions on the space, and we need to enforce them numerically
linalg.cond(stiffness_matrix)
```

    5.05142981750785e+16

```python
# We do the same thing we did in the finite difference case: eliminate the rows corresponding to the boundary
# points, set the diagonal to 1, and set the rhs to the desired boundary condition
stiffness_matrix[0,:] = stiffness_matrix[-1,:] = 0
stiffness_matrix[0,0] = stiffness_matrix[-1,-1] = 1

# Let's check now if the matrix is well conditioned
linalg.cond(stiffness_matrix)
```

    250.25346068360355

```python
# Now let's first compute an L2 projection of the runge function, and compare with the interpolation
mass_rhs = einsum('qi,q,q', Bq, runge(Xq), JxWq)
u_projection = linalg.solve(mass_matrix, mass_rhs)
```

```python
plot(X, runge(X))
u_interpolate = runge(support_points)
plot(support_points, u_interpolate, 'ro')
plot(X, B.dot(runge(support_points)),'r')
plot(X, B.dot(u_projection),'g')
```

```python
# Finally, let's solve non trivial problem: -u'' = sin(2 pi x) with zero bc.

def rhs_function(x):
    return sin(2*pi*x)

def exact(x):
    return sin(2*pi*x)/(4*pi**2)

# Assemble the rhs, and make sure we set to zero the boundary conditions
rhs = einsum('qi,q,q', Bq, rhs_function(Xq), JxWq)
rhs[0] = rhs[-1] = 0
```

```python
u = linalg.solve(stiffness_matrix, rhs)
```

```python
plot(X, exact(X))
plot(X, B.dot(u))
plot(support_points, exact(support_points), 'o')
```

```python
# Let's compute also the L2 error:
error = sqrt(einsum('q,q', (Bq.dot(u)-exact(Xq))**2, JxWq))
error
```

    3.51606167731296e-05

```python

```

```python

```

slides/.ipynb_checkpoints/Lecture 12 - LH - LAB - Introduction to PDEs - Finite Elements in 1D - Implementation-checkpoint.ipynb


## Parte 1. Introduce un Sistema de Ecuaciones Lineales

```python
#importamos las bibliotecas que utilizaremos
%matplotlib inline
from matplotlib import pyplot as plt
from IPython.display import display, Math, Latex, Markdown
from sympy.interactive import printing
from sympy import Eq, solve_linear_system, Matrix,Symbol, init_printing, init_session
from numpy import linalg
import numpy as np
import sympy as sp
printing.init_printing(use_latex=True)
init_printing(use_latex='mathjax')

#Definimos las variables que se utilizarán guardar datos

def printmd(string):
    display(Markdown(string))

#Definimos las variables para los coeficientes

x, y=sp.symbols ('x,y')
a,b,c,d,e,f,g,h,r,j,k,l,m,n,a1,b1,c1=sp.symbols ('a,b,c,d,e,f,g,h,r,j,k,l,m,n,a1,b1,c1')

to= r'$$Introduce\ los\ coeficientes\ de\ tu\ primera\ ecuación\ lineal,\ de\ la\ forma\ ax+by=c$$'

printmd(to)

#Para que Python interprete la entrada como un número entero, se debe utilizar la función int() para decimal se usa float()

a= int(input("Cuál es el coeficiente de x: "))
a
b=int(input("Cuál es el coeficiente de y: "))
b
c=int(input("Cuál es el término constante: "))

#
printmd('$$Tu\ ecuación\ 1\ es:$$')

ec1= sp.Function('ec1')
ec2= sp.Function('ec2')

ec1= Eq(a*x + b*y, c)

display(ec1)

ta= r'$$Introduce\ los\ datos\ de\ la\ segunda\ ecuación\ de\ la\ forma\ dx+ey=f:$$'

printmd(ta)

d= int(input("Cuál es el coeficiente de x: "))
d
e=int(input("Cuál es el coficiente de y: "))
e
f=int(input("Cuál es el termino constante: "))

printmd('$$Tu\ ecuación\ 2\ es:$$')

ec2= Eq(d*x + e*y, f)

display(ec2)

printmd ('$$Por\ lo\ tanto\ tu\ sistema\ es:$$')

display(ec1, ec2)

printmd('$$Por\ lo\ tanto\ el\ arreglo\ rectangular\ de\ tu\ sistema\ es:$$')

row1=[a,b,c]
row2=[d,e,f]

system = Matrix ((row1,row2))

display (system)

#Gráficamos el sistema

printmd ('**La gráfica del sistema es:**')

def y1(x):
    return (c - a*x)/b
# Función lineal.

def y2(x):
    return  (f - d*x)/f
# Valores del eje X que toma el gráfico.
x = range(-10, 15)
# Graficar ambas funciones.
plt.plot(x, [y1(i) for i in x])
plt.plot(x, [y2(i) for i in x])
# Establecer el color de los ejes.
plt.axhline(0, color="black")
plt.axvline(0, color="black")
# Limitar los valores de los ejes.
plt.xlim(-15, 15)
plt.ylim(-15, 15)
# Guardar gráfico como imágen PNG.
plt.savefig("output.png")
# Mostrarlo.
plt.show()
```

## Parte 2. Modifica un Sistema de Ecuaciones Lineales

**-Tomando en cuenta el sistema que introdujiste anteriormente:**

**-Elije una ecuación y multiplicala por un número entero**

**-Observa cómo cambia el sistema respecto al original**

```python
x,y = sp.symbols('x,y')
m=int(input('Qué ecuación quieres multiplicar por un número (introduce 1 o 2 según la ecuación que desees multiplicar):'))
m

if m == 1:
    n=int(input('Por qué número deseas multiplicar: '))
    n

    a1=a*n
    b1=b*n
    c1=c*n

    printmd('**Tu ecuación 1 ahora es:**')

    display(ec1*n)

    eq3 = sp.Function('eq3')

    printmd('**Es decir:**')

    eq3 = Eq(a1*x+b1*y, c1)

    display(eq3)

    printmd('**Por lo tanto tu sistema es el siguiente:**')

    display (eq3,ec2)

    printmd('$$Por\ lo\ tanto\ el\ arreglo\ rectangular\ de\ tu\ sistema\ es:$$')

    row3=[a1,b1,c1]
    row4=[d,e,f]

    system1 = Matrix ((row3,row4))

    display (system1)

    #Gráficamos el sistema

    printmd ('**La gráfica del sistema es:**')

    def y11(x):
        return (c1 - a1*x)/b1
    # Función lineal.

    def y2(x):
        return  (f - d*x)/f
    # Valores del eje X que toma el gráfico.
    x = range(-10, 15)
    # Graficar ambas funciones.
    plt.plot(x, [y1(i) for i in x])
    plt.plot(x, [y2(i) for i in x])
    # Establecer el color de los ejes.
    plt.axhline(0, color="black")
    plt.axvline(0, color="black")
    # Limitar los valores de los ejes.
    plt.xlim(-15, 15)
    plt.ylim(-15, 15)
    # Guardar gráfico como imágen PNG.
    plt.savefig("output.png")
    # Mostrarlo.
    plt.show()

elif m==2:
    n=int(input('Por qué número deseas multiplicar: '))
    n

    d1=d*n
    e1=e*n
    f1=f*n

    printmd('**Tu ecuación 2 ahora es:**')

    display(ec2*n)

    eq4 = sp.Function('eq4')

    printmd('**Es decir:**')

    eq4 = Eq(d1*x+e1*y, f1)

    display(eq4)

    printmd('**Por lo tanto tu sistema es el siguiente:**')

    display (ec1,eq4)

    printmd('$$Por\ lo\ tanto\ el\ arreglo\ rectangular\ de\ tu\ sistema\ es:$$')

    row4=[a,b,c]
    row5=[d1,e1,f1]

    system2 = Matrix ((row4,row5))

    display (system2)

    #Gráficamos el sistema

    printmd ('**La gráfica del sistema es:**')

    def y1(x):
        return (c - a*x)/b
    # Función lineal.

    def y22(x):
        return  (f1 - d1*x)/f1
    # Valores del eje X que toma el gráfico.
    x = range(-10, 15)
    # Graficar ambas funciones.
    plt.plot(x, [y1(i) for i in x])
    plt.plot(x, [y2(i) for i in x])
    # Establecer el color de los ejes.
    plt.axhline(0, color="black")
    plt.axvline(0, color="black")
    # Limitar los valores de los ejes.
    plt.xlim(-15, 15)
    plt.ylim(-15, 15)
    # Guardar gráfico como imágen PNG.
    plt.savefig("output.png")
    # Mostrarlo.
    plt.show()

else:
    printmd('**Introdujiste un número distinto de 1 o 2, vuelve a intentarlo**')

```

## Parte 3. Muestra la solución de un SEL

```python

```

## Parte 4. Introduce un Parámetro para resolver un SEL

```python

```

```python

```


## Griewank

The Griewank function has many widespread local minima, which are regularly distributed. The complexity is shown in the zoomed-in plots.

**Definition**

\begin{align}
\begin{split}
f(x) & = & \; \sum_{n=1}^n \; \frac{x_i^2}{4000} - \prod_{n=1}^n \cos \Bigg( \frac{x_i}{\sqrt{i}} \Bigg) + 1 \\[4mm]
&&-600 \leq x_i \leq 600 \quad i=1 \ldots,n \\[4mm]
\end{split}
\end{align}

**Optimum**

$$f(x^*) = 0 \; \text{at} \; x^* = (0,\ldots,0) $$

**Contour**

```python
import numpy as np
from pymoo.factory import get_problem, get_visualization

problem = get_problem("griewank", n_var=1)
plot = get_visualization("fitness-landscape", problem, _type="surface", n_samples=1000)
plot.do()
plot.apply(lambda ax: ax.set_xlim(-200,200))
plot.apply(lambda ax: ax.set_ylim(-1,13))
plot.show()
```


# Introduction to Gaussian Process Models
Gaussian process (GP) models serve as approximations of computationally expensive (time-consuming) black-box functions. To reduce the number of times the expensive function must be queried during optimization, the GP is used to guide the sampling decisions in the parameter space and only the "most promising" parameters are selected for evaluation.

A GP model treats the function it approximates like the realization of a stochastic process:
$m_{GP}(\theta) = \mu + Z(\theta)$,
where $\mu$ represents the mean of the stochastic process and $Z(\theta) \sim \mathcal{N}(0,\sigma^2)$ is the deviation from the mean.

The correlation between two random variables $Z(\theta_k)$ and $Z(\theta_l)$ is defined by a kernel, e.g., the squared exponential (also Radial basis function) kernel:
\begin{equation}
Corr(Z(\theta_k),Z(\theta_l)) = \exp(-\sum_{i=1}^d \gamma_i|\theta_k^{(i)}-\theta_l^{(i)}|^{q_i})
\end{equation},
with $\gamma_i$ determining how quickly the correlation in dimension $i$ decreases, and $q_i$ refelcts the smoothness of the function in dimension $i$

Denoting $\mathbf{R}$ as the matrix whose $(k,l)$-th element is given as the correlation above, maximum likelihood estimation is used to determine the GP parameters $\mu$, $\sigma^2$, and $\gamma_i$. Then, at an unsampled point $\theta^{new}$, the GP prediction is \begin{equation}
    m_{\text{GP}}(\theta^{\text{new}})=\hat{\mu}+\mathbf{r}^T\mathbf{R}^{-1}(\mathbf{f}-\mathbf{1}\hat\mu),
\end{equation}
where $\mathbf{1}$ is a vector of ones of appropriate dimension and $\mathbf{f}$ is the vector of function values obtained so far, and
\begin{equation}
    \boldsymbol{r}=
    \begin{bmatrix}
    Corr\left(Z(\theta^{\text{new}}), Z(\theta_1)\right)\\
    \vdots\\
    Corr\left(Z(\theta^{\text{new}}
    ), Z(\theta_n)\right)
    \end{bmatrix}.
\end{equation}
The  corresponding  mean squared error is
\begin{equation}
    s^2(\theta^{\text{new}})=\hat{\sigma}^2\left(   1-\boldsymbol{r}^T\boldsymbol{R}^{-1}\boldsymbol{r} +\frac{(1-\boldsymbol{1}^T\boldsymbol{R}^{-1}\boldsymbol{r})^2}{\mathbf{1}^T\boldsymbol{R}^{-1}\mathbf{1}}\right)
\end{equation}
with
 \begin{equation}
     \hat{\mu} = \frac{\mathbf{1}^T\boldsymbol{R}^{-1}\mathbf{f}}{\mathbf{1}^T\boldsymbol{R}^{-1}\mathbf{1}}
 \end{equation}
 and
 \begin{equation}
     \hat{\sigma}^2=\frac{(\mathbf{f}-\mathbf{1}\hat{\mu})^T\boldsymbol{R}^{-1}(\mathbf{f}-\mathbf{1}\hat{\mu})}{n}.
 \end{equation}

Python has a good implementation of GPs where you can choose different kernels.

First, we need (input, output) data pairs. Inputs are parameters where we query the function (for simplicity, the example has an inexpensive function). From the Sckit-Learn website: https://scikit-learn.org/stable/auto_examples/gaussian_process/plot_gpr_noisy_targets.html

```python
import numpy as np
from matplotlib import pyplot as plt
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel, Matern, RationalQuadratic, ExpSineSquared, WhiteKernel
from scipy.optimize import minimize
from scipy.spatial import distance
import scipy.spatial as scp
from scipy.stats import norm
from pyDOE import * #needed if Latin hypercube design is used
import warnings
warnings.filterwarnings("ignore")
```

```python
def f(x):
    """The function we want to approximate."""
    return x * np.sin(x)
```

```python
xlow = 0 #lower bound on x
xup = 10 #upper bound on x
dim = 1 #dimension of the problem
lhs_wanted = False
np.random.seed(420)
```

```python
if not(lhs_wanted): #when not using space-filling design
    X = np.atleast_2d([1., 3., 7., 8.]).T #select some points where we evaluate the function

    # Function evaluations
    y = f(X).ravel()
```

Other options for creating space filling designs is latin hypercube sampling:

```python
if lhs_wanted:
    ninit=6 #6 initial evaluations
    init_design = lhs(dim, samples =ninit, criterion='maximin') #initial design in [0,1]^dim
    X = xlow+(xup-xlow)*init_design #scale to [xlow,xup]
    # Function evaluations
    y = f(X).ravel()
```

**Exercise:** run the code with different initial samples, i.e., try lhs_wanted = False and lhs_wanted = True and compare the sampling history

```python
# Select a GP kernel (here RBF or squared exponential)
kernel = RBF()
gp = GaussianProcessRegressor(kernel=kernel, normalize_y=True,n_restarts_optimizer=9)

# Fit the GP to the input-output data
gp.fit(X, y)

```

Make some good-looking plots

```python
def plot_the_gp(X, y, gp, xnew):
    #select a bunch of points where we want to make predictions wioth the GP
    x = np.atleast_2d(np.linspace(0, 10, 1000)).T
    # Make the GP prediction at the points where no evaluations were taken - also return predicted uncertainty
    y_pred, sigma = gp.predict(x, return_std=True)
    plt.figure()
    plt.plot(x, f(x), 'r:', label=r'$f(x) = x\,\sin(x)$')
    plt.plot(X, y, 'r.', markersize=10, label='Observations')
    plt.plot(x, y_pred, 'b-', label='Prediction')
    if len(xnew)>0:
        plt.plot(X[-1], y[-1], 'gs', markersize=10, label='Newest sample')
    plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate([y_pred - 1.9600 * sigma,
                        (y_pred + 1.9600 * sigma)[::-1]]),
         alpha=.5, fc='b', ec='None', label='95% confidence interval')
    plt.xlabel('$x$')
    plt.ylabel('$f(x)$')
    plt.ylim(-10, 20)
    plt.legend(loc='upper left')
    plt.show()
```

```python
plot_the_gp(X, y, gp, [])
```

**Optional Exercise:** check out the Scikit-Learn website https://scikit-learn.org/stable/modules/gaussian_process.html#kernels-for-gaussian-processes and experiment around with different basic kernels, kernel parameters and kernel combinations, e.g.,
- does using "kernel = RBF(10, (1e-2, 1e2))" change anything?
- what happens when you use "kernel = Matern(length_scale=1.0, nu=1.5)"
- try "kernel = 1.0 * RationalQuadratic(length_scale=1.0, alpha=0.1, alpha_bounds=(1e-5, 1e15))"
- "kernel = 1.0 * ExpSineSquared(
    length_scale=1.0,
    periodicity=3.0,
    length_scale_bounds=(0.1, 10.0),
    periodicity_bounds=(1.0, 10.0),)"
- use a combination of kernels: "kernel = RBF()+WhiteKernel(noise_level=.001)" using different noise_levels

**Exercise:** Change the inputs of the GP (i.e., the training samples) and see how the GP predictions change (use fewer or more points, use different points in [0,10], e.g., "X=np.atleast_2d(np.random.uniform(0,10,5)).T"

Takeaway: the quality and accuracy of the GP highly depends on the trianing data and the kernel used

# Adaptive Optimization with the GP

GP models are often used in optimization algorithms. In each iteration of the optimization, a new sample point is selected by maximizing the expected improvement (EI):
\begin{equation}
    \mathbb{E}(I)(\theta) = s(\theta)\left(v\Phi(v)+\phi(v)   \right),
\end{equation}
where
\begin{equation}
    v=\frac{f^{\text{best}}-m_{\text{GP}}(\theta)}{s(\theta)}
\end{equation}
and $\Phi$ and $\phi$ are the normal cumulative distribution and density functions, respectively, and $s(\theta)=\sqrt{s^2(\theta)}$ is the square root of the mean squared error.

The function $\mathbb{E}(I)(\theta)$ can be maximized with any python optimization library. The point $\theta^{\text{new}}$ where it reaches its maximum will be the new point where $f$ is evaluated.

```python
# define expected improvement function
def ei(x, gpr_obj, Xsamples, Ysamples): #expected improvement
    dim = len(x)
    x= x.reshape(1, -1)

    min_dist=np.min(scp.distance.cdist(x, Xsamples))
    if min_dist<1e-6: #threshold for when points are so close that they are considered indistinguishable
        expected_improvement=0.0
        return expected_improvement

    mu, sigma = gpr_obj.predict(x.reshape(1, -1), return_std=True)
    mu_sample = gpr_obj.predict(Xsamples)
    mu_sample_opt = np.min(Ysamples)

    # In case sigma equals zero
    with np.errstate(divide='ignore'):
        Z = (mu_sample_opt-mu) / sigma
        expected_improvement = (mu_sample_opt-mu) * norm.cdf(Z) + sigma * norm.pdf(Z)
        expected_improvement[sigma == 0.0] == 0.0
    answer=-1.*expected_improvement #to maximize EI, you minimize the negative of it
    return answer
```

```python
def plot_the_ei(gpr_obj, X, Y):
    x = np.atleast_2d(np.linspace(0, 10, 1000)).T
    expimp=np.zeros(1000)
    for ii in range(1000):
        expimp[ii] = -ei(x[ii], gpr_obj, X, Y)
    plt.figure()
    plt.plot(x, expimp, 'k--', label='Expected improvement')
    plt.plot(X, np.zeros(X.shape[0]), 'rx', markersize=10, label='Observation sites')
    #plt.plot(X[-1],0, 'gs', markersize=10, label='Newest sample')
    plt.xlabel('$x$')
    plt.ylabel('$EI(x)$')
    #plt.ylim(-10, 20)
    plt.legend(loc='upper left')
    plt.show()
```

```python
# do your GP iterations: maximize EI, select new point, evaluate new point, update GP, maximize EI, ....
n_GP_samples = 20 # allow 50 evaluations of f
bound_list = np.array([[xlow, xup]])
xnew=[]
while  X.shape[0]< n_GP_samples:
    gpr_obj = GaussianProcessRegressor(kernel=kernel, random_state=0,normalize_y=True, n_restarts_optimizer=10).fit(X, y) #create the GP
    plot_the_gp(X, y, gpr_obj, xnew)
    plot_the_ei(gpr_obj, X, y)
    #compute next point by maximizing expected improvement, multi-start optimization
    xnew = []
    fnew =np.inf
    for ii in range(10):
        x0 = xlow + (xup-xlow) * np.random.rand(1,dim) #random starting point for optimizing expected improvement
        res= minimize(ei,np.ravel(x0),method='SLSQP',bounds=bound_list, args=(gpr_obj, X, y))
        dist = np.min(scp.distance.cdist(np.asmatrix(res.x), X)) #make sure new point is sufficiently far away from already sampled points
        if np.min(dist)>1e-6 and res.success: #1e-3 is tunable
            x_ = np.asmatrix(res.x)
            if res.fun< fnew:
                xnew = x_
                fnew = res.fun
        else: #use random point as new point
            x_ = np.asarray(xlow) + np.asarray(xup-xlow) * np.asarray(np.random.rand(1,dim)) #random starting point
            fv= ei(x_, gpr_obj, X, y)
            if len(xnew)== 0 or fv < fnew:
                xnew = np.asmatrix(x_)
                fnew= fv

    fval = f(np.ravel(xnew))

    #update Xsamples and Ysamples arrays
    X=np.concatenate((X, np.asmatrix(xnew)), axis = 0)
    Y_ = np.zeros(len(y)+1)
    Y_[0:len(y)]= y
    Y_[-1]=fval
    y =Y_
    minID=np.argmin(y) #find index of best point
    print('best point: ', X[minID])
    print('best value: ', y[minID])
    print('Number evaluations: ', X.shape[0])
```

From the images of the expected improvement we can see that the peaks are becoming increasingly narrow, to almost looking like jump-discontinuities. This means that for an optimizer that tries to find the maximum of the expected improvement function, it becomes increasingly harder to find the optimum and sampling becomes more "random" in the space because the function is flat and EI values are the same everywhere except at the jumps.

Takeaways:
- GPs can be useful to guide the search during optimization
- They shine when the number of function evaluations is severely limited
- The expected improvement function helps to select points that are the "most promising" next evaluations
- The expected improvement function is multimodal and becomes increasingly harder to optimize


# 微積分の計算について　　　N0.3　不定積分の内容－５ 部分積分

###  学籍番号[＿＿＿＿＿＿＿＿＿]クラス[＿＿＿＿＿]　クラス番号[＿＿＿＿＿] 名前[＿＿＿＿＿＿＿＿＿＿＿＿＿＿＿]

##### 積分の手法　（１）部分積分法

$$ \int f'(x)g(x)dx =f(x)g(x)- \int f(x)g'(x)dx $$

```python
from sympy import *
x, n , y, a = symbols('x n y a')
init_printing()
m ='3//5'
i =0
```

 ### 例題ー2

```python
expr = x*cos(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = log(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x**2*sin(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x**3*exp(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = atan(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = log(x)**2
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x**3*exp(2*x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python
expr = x**2*sin(x)
itg = Integral(expr,x)
i=i+1
print( 'No.',m,'---',i)
itg
```

```python
simplify(itg.doit())
```

```python

```


# 2 - Damping

# 2.1 - Introduction

In this notebook we describe a simple method for reduction of reflections at the computational boundaries of the domain $\Omega$ when we simulate the acoustic wave equation. This method, called *Damping*, has been proposed by Sochaki. It adds a damping term, modifying the original wave equation at a boundary layer. We saw in the notebook Introduction to Acoustic Problem that the (artificial) wave reflections on the computational boundaries lead to a very noisy solution of the acoustic problem.

We describe this method in the next Sections, omitting information already discussed in the notebook Introduction to Acoustic Problem, highlighting  only the new elements necessary to apply Damping.

# 2.2 - Acoustic Problem with Damping

We define an extension of the spatial domain $\Omega=\left[x_{I}-L_{x},x_{F}+L_{x}\right] \times\left[z_{I},z_{F}+L_{z}\right]$, in which we added an *absorption region* to the previous spatial domain
$\Omega_{0}=\left[x_{I},x_{F}\right]\times\left[z_{I},z_{F}\right]$.
The *absorption region* is composed by two bands of length $L_{x}$ at the beginning and end of the domain in the direction $x$ and of a band of length $L_{z}$ at the end of the domain in the $z$ direction. Again, $\partial\Omega$ denotes the boundary of $\Omega$. The figure below shows the extended domain $\Omega$, with the absorption region highlighted in blue.

The damping acoustic problem equation is given by:

\begin{equation}
u_{tt}(x,z,t)+c^2(x,z)\zeta(x,z)u_t(x,z,t)-c^2(x,z)\Delta(u(x,z,t))=c^2(x,z)f(x,z,t),
\end{equation}

where $u(x,z,t)$, $f(x,z,t)$ and $c(x,z)$ are as before. The wave equation has been modified by the introduction of the damping term $c^2(x,z)\zeta(x,z)u_t(x,z,t)$, where $\zeta$ is different from zero only in the absorption region, growing smoothly along the absorption bands from zero to its maximum at the outer boundary. The actual form of
$\zeta$ used in this notebook will be given ahead. We still use the same initial conditions

\begin{equation}
u(x,z,0) = 0.0 \hspace{.5cm} \mbox{ and } \hspace{.5cm} u_t(x,z,0)= 0.0.
\end{equation}

and Dirichlet null boundary conditions at the (outer) bottom and lateral boundaries. At the surface we
use a zero Neumman boundary condition.

The source term and the velocity field are defined as before.

# 2.3 - Finite Difference Operators and Discretization of Spatial and Temporal Domains

The only difference with respect to the discretization used in the notebook Introduction to Acoustic Problem is the extra damping term. The temporal derivative of $u$ is approximated by a centered difference:
$$ u_t(x_i,z_j,t_k) = \frac{u_{i,j,k+1}-u_{i,j,k-1}}{2\Delta t} $$. All the other terms are discretized as before.

# 2.4 - Standard Problem

Redeeming the Standard Problem definitions discussed on the notebook Introduction to Acoustic Problem we have that:

- $x_{I}$ =  0.0 Km;
- $x_{F}$ =  1.0 Km = 1000 m;
- $z_{I}$ =  0.0 Km;
- $z_{F}$ =  1.0 Km = 1000 m;
- $L_x$ and $L_z$ will be defined ahead;

The spatial discretization parameters are given by:
- $\Delta x$ = 0.01 km = 10m;
- $\Delta z$ = 0.01 km = 10m;

Let's consider a $I$ the time domain with the following limitations:

- $t_{I}$ = 0 s = 0 ms;
- $t_{F}$ = 1 s = 1000 ms;

The temporal discretization parameters are given by:

- $\Delta t$ $\approx$ 0.0016 s = 1.6 ms;
- $NT$ = 626.

With respect to the $f(x,z,t)$ external force term, we will consider a Ricker source with the following properties:

- Position at $x:$ $\bar{x} = 500 m =  0.5 Km$;
- Position at $z:$ $\bar{z} =  10 m = 0.01 Km$;
- Peak frequency: $f_{0} = 10 Hz = 0.01 Khz$;

The graph of $f(\bar{x},\bar{z},t)$ will be generated when building the code. We will use a velocity profile $c(x, z)$ with the following properties:

- Minimum propagation velocity: $v_{min} = 1500 m/s = 1,5 Km/s$;
- Maximum propagation velocity: $v_{max} = 2500 m/s = 2,5 Km/s$;

The figure of the velocity profile will be generated when building the code. We introduce receivers along the $x$ direction, that is, at all discrete points between $0.0$ Km and $1.0$ Km , at depth $z=0.01$ Km to generate the seismogram.

# 2.5 - Damping Functions

Sochaki proposed various forms for the damping function $\zeta$, including linear, cubic or exponential functions. In general, the damping functions have a similar characteristic: they are zero in the "interior" domain $\Omega_{0}$ and increase toward the outer boundary $\partial\Omega$.

Our particular damping function will be chosen as follows.
 We define the pair of functions $\zeta_{1}(x,z)$ and $\zeta_{2}(x,z)$ given, respectively, by:

\begin{equation}
\zeta_{1}(x,z)=\left\{ \begin{array}{ll}
0, & \textrm{if $x\in \left(x_{I},x_{F}\right)$,}\\ \bar{\zeta}_{1}(x,z)\left(\displaystyle\frac{\vert x-x_{I} \vert}{L_{x}}-\displaystyle\frac{1}{2\pi}\sin\left(\displaystyle\frac{2\pi\vert x-x_{I} \vert}{L_{x}}\right)\right) , & \textrm{if $x_{I}-L_{x}\leq x \leq x_{I}$,}\\ \bar{\zeta}_{1}(x,z)\left(\displaystyle\frac{\vert x-x_{F} \vert}{L_{x}}-\displaystyle\frac{1}{2\pi}\sin\left(\displaystyle\frac{2\pi\vert x-x_{F} \vert}{L_{x}}\right)\right) , & \textrm{if $x_{F}\leq x \leq x_{F}+L_{x}$.}\end{array}\right.
\end{equation}

\begin{equation}
\zeta_{2}(x,z)=\left\{ \begin{array}{ll}
0, & \textrm{if $z\in \left(z_{I},z_{F}\right)$,} \\ \bar{\zeta}_{2}(x,z)\left(\displaystyle\frac{\vert z-z_{F} \vert}{L_{z}}-\displaystyle\frac{1}{2\pi}\sin\left(\displaystyle\frac{2\pi\vert z-z_{F} \vert}{L_{z}}\right)\right) , & \textrm{if $z_{F}\leq z \leq z_{F}+L_{z}$.}\end{array}\right.
\end{equation}

Thus, we define the function $\zeta(x,z)$ as being the following function:

\begin{equation}
\zeta(x,z) = \displaystyle\frac{1}{v_{max}}\left(\displaystyle\frac{\zeta_{1}(x,z)}{\Delta x}+\displaystyle\frac{\zeta_{2}(x,z)}{\Delta z} \right) ,
\end{equation}

where $v_{max}$denotes the maximum velocity of propagation of $c(x,z)$. Below we display the shape of the function $\zeta_1(x,z)$ with $\bar{\zeta_{1}}(x,z)=0.26$ at the left band of the domain. It is similar at the other ones. The figures of the damping profiles will be generated when building the code.

# 2.6 - Numerical Simulations

In the numerical simulations we import the following Python and Devito packages:

```python
# NBVAL_IGNORE_OUTPUT

import numpy                   as np
import matplotlib.pyplot       as plot
import math                    as mt
import matplotlib.ticker       as mticker
from   mpl_toolkits.axes_grid1 import make_axes_locatable
from   matplotlib              import cm
```

From Devito's library of examples we import the following structures:

```python
# NBVAL_IGNORE_OUTPUT

%matplotlib inline
from   examples.seismic  import TimeAxis
from   examples.seismic  import RickerSource
from   examples.seismic  import Receiver
from   devito            import SubDomain, Grid, NODE, TimeFunction, Function, Eq, solve, Operator
```

The mesh parameters define the domain $\Omega_{0}$. The absorption region will be included bellow.

```python
nptx   =  101
nptz   =  101
x0     =  0.
x1     =  1000.
compx  =  x1-x0
z0     =  0.
z1     =  1000.
compz  =  z1-z0;
hxv    =  (x1-x0)/(nptx-1)
hzv    =  (z1-z0)/(nptz-1)
```

**Observation:** In this code we need to work with symbolic values and the real values of $\Delta x$ and $\Delta z$, then the numerica values of $\Delta x$ and $\Delta z$ are represented by *hxv* and *hzv*, respectively. The symbolic values of $\Delta x$ and $\Delta z$ will be given after.

In this case, we need to define the size of the bands $L_{x}$ and $L_{z}$ that extend the domain $\Omega_{0}$ for $\Omega$. The code that we will implement will build the values $L_{x}$ and $L_{z}$ from choosing a certain amount of points in each direction. Without loss of generality, we say that the size $L_{x}$ is such that:

- $L_{x}$ = npmlx*$\Delta x$;
- *0<npmlx<nptx;*

Similarly, we have $L_{z}$ such that:

- $L_{z}$ = npmlz*$\Delta z$;
- *0<npmlz<nptz*;

So, we can explicitly define the lengths $L_{x}$ and $L_{z}$ depending on the number of points *npmlx* and *npmlz*. Thus, we choose these values as being:

```python
npmlx  = 20
npmlz  = 20
```

And we define $L_{x}$ and $L_{z}$ as beeing:

```python
lx = npmlx*hxv
lz = npmlz*hzv
```

Thus, from the *nptx* points, the first and the last *npmlx* points are in the absorption region of the *x* direction. Similarly, from the *nptz* points, the last *npmlz* points are in the absorption region of the *z* direction. Considering the construction of *grid*, we also have the following elements:

```python
nptx    =  nptx + 2*npmlx
nptz    =  nptz + 1*npmlz
x0      =  x0 - hxv*npmlx
x1      =  x1 + hxv*npmlx
compx   =  x1-x0
z0      =  z0
z1      =  z1 + hzv*npmlz
compz   =  z1-z0
origin  = (x0,z0)
extent  = (compx,compz)
shape   = (nptx,nptz)
spacing = (hxv,hzv)
```

The $\zeta(x,z)$ function is non zero only in the blue region in the figure that represents the domain. In this way, the wave equation can be divided into 2 situations:

- In the region in blue:

\begin{equation}
u_{tt}(x,z,t)+c^2(x,z)\zeta(x,z)u_t(x,z,t)-c^2(x,z)^\Delta(u(x,z,t))=c^2(x,z)f(x,z,t),
\end{equation}

- In the white region:

\begin{equation}
u_{tt}(x,z,t)-c^2(x,z)^\Delta(u(x,z,t))=c^2(x,z)f(x,z,t),
\end{equation}

For this reason, we use the structure of the *subdomains* to represent the white region and the blue region.

**Observation:** Note that we can describe the blue region in different ways, that is, the way we choose here is not the only possible discretization for that region.

First, we define the white region, naming this region as *d0*, which is defined by the following pairs of points $(x,z)$:

- $x\in\{npmlx,nptx-npmlx\}$ and $z\in\{0,nptz-npmlz\}$.

In the language of *subdomains* *d0  it is written as:

```python
class d0domain(SubDomain):
    name = 'd0'
    def define(self, dimensions):
        x, z = dimensions
        return {x: ('middle', npmlx, npmlx), z: ('middle', 0, npmlz)}
d0_domain = d0domain()
```

The blue region will be the union of the following regions:

- *d1* represents the left range in the direction *x*, where the pairs $(x,z)$ satisfy: $x\in\{0,npmlx\}$ and $z\in\{0,nptz\}$;
- *d2* represents the rigth range in the direction *x*, where the pairs $(x,z)$ satisfy: $x\in\{nptx-npmlx,nptx\}$ and $z\in\{0,nptz\}$;
- *d3* represents the left range in the direction *y*, where the pairs $(x,z)$ satisfy: $x\in\{npmlx,nptx-npmlx\}$ and $z\in\{nptz-npmlz,nptz\}$;

Thus, the regions *d1*, *d2* and *d3* are described as follows in the language of *subdomains*:

```python
class d1domain(SubDomain):
    name = 'd1'
    def define(self, dimensions):
        x, z = dimensions
        return {x: ('left',npmlx), z: z}
d1_domain = d1domain()

class d2domain(SubDomain):
    name = 'd2'
    def define(self, dimensions):
        x, z = dimensions
        return {x: ('right',npmlx), z: z}
d2_domain = d2domain()

class d3domain(SubDomain):
    name = 'd3'
    def define(self, dimensions):
        x, z = dimensions
        return {x: ('middle', npmlx, npmlx), z: ('right',npmlz)}
d3_domain = d3domain()
```

The figure below represents the division of domains that we did previously:

The advantage of dividing into regions is that the equations will be calculated where they actually operate and thus we gain computational efficiency, as we decrease the number of operations to be done. After defining the spatial parameters and constructing the *subdomains*, we set the *spatial grid* with the following command:

```python
grid = Grid(origin=origin, extent=extent, shape=shape, subdomains=(d0_domain,d1_domain,d2_domain,d3_domain), dtype=np.float64)
```

Again, we use a velocity field given by a binary file. The reading and scaling of the velocity field for the Devito work units is done with the following commands:

```python
v0 = np.zeros((nptx,nptz))
X0 = np.linspace(x0,x1,nptx)
Z0 = np.linspace(z0,z1,nptz)

x10 = x0+lx
x11 = x1-lx

z10 = z0
z11 = z1 - lz

xm = 0.5*(x10+x11)
zm = 0.5*(z10+z11)

pxm = 0
pzm = 0

for i in range(0,nptx):
    if(X0[i]==xm): pxm = i

for j in range(0,nptz):
    if(Z0[j]==zm): pzm = j

p0 = 0
p1 = pzm
p2 = nptz

v0[0:nptx,p0:p1] = 1.5
v0[0:nptx,p1:p2] = 2.5
```

Previously we introduce the local variables *x10,x11,z10,z11,xm,zm,pxm* and *pzm* that help us to create a specific velocity field, where we consider the whole domain (including the absorpion region). Below we include a routine to plot the velocity field.

```python
def graph2dvel(vel):
        plot.figure()
        plot.figure(figsize=(16,8))
        fscale =  1/10**(3)
        scale  = np.amax(vel[npmlx:-npmlx,0:-npmlz])
        extent = [fscale*(x0+lx),fscale*(x1-lx), fscale*(z1-lz), fscale*(z0)]
        fig = plot.imshow(np.transpose(vel[npmlx:-npmlx,0:-npmlz]), vmin=0.,vmax=scale, cmap=cm.seismic, extent=extent)
        plot.gca().xaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
        plot.gca().yaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
        plot.title('Velocity Profile')
        plot.grid()
        ax = plot.gca()
        divider = make_axes_locatable(ax)
        cax = divider.append_axes("right", size="5%", pad=0.05)
        cbar = plot.colorbar(fig, cax=cax, format='%.2e')
        cbar.set_label('Velocity [km/s]')
        plot.show()
```

Below we include the plot of velocity field.

```python
# NBVAL_IGNORE_OUTPUT

graph2dvel(v0)
```

Time parameters are defined and constructed by the following sequence of commands:

```python
t0    = 0.
tn    = 1000.
CFL   = 0.4
vmax  = np.amax(v0)
dtmax = np.float64((min(hxv,hzv)*CFL)/(vmax))
ntmax = int((tn-t0)/dtmax)+1
dt0   = np.float64((tn-t0)/ntmax)
```

With the temporal parameters, we generate the time informations with *TimeAxis* as follows:

```python
time_range = TimeAxis(start=t0,stop=tn,num=ntmax+1)
nt         = time_range.num - 1
```

The symbolic values associated with the spatial and temporal grids that are used in the composition of the equations are given by:

```python
(hx,hz) = grid.spacing_map
(x, z)  = grid.dimensions
t       = grid.stepping_dim
dt      = grid.stepping_dim.spacing
```

We chose a single Ricker source, whose frequency is $ 0.005Khz $. This source is positioned at $\bar{x}$ = 35150m and $\bar{z}$ = 32m. We then defined the following variables that represents our choice:

```python
f0      = 0.01
nsource = 1
xposf   = 0.5*(compx-2*npmlx*hxv)
zposf   = hzv
```

As we know, Ricker's source is generated by the *RickerSource* command. Using the parameters listed above, we generate and position the Ricker source with the following sequence of commands:

```python
src = RickerSource(name='src',grid=grid,f0=f0,npoint=nsource,time_range=time_range,staggered=NODE,dtype=np.float64)
src.coordinates.data[:, 0] = xposf
src.coordinates.data[:, 1] = zposf
```

Below we include the plot of Ricker source.

```python
# NBVAL_IGNORE_OUTPUT

src.show()
```

With respect to receivers, the number of receivers is the same number of discrete points in the $x$ direction. So, we position these receivers along the direction $x$, at height $\bar{z}$ = 10m. In this way, our variables are chosen as:

```python
nrec   = nptx
nxpos  = np.linspace(x0,x1,nrec)
nzpos  = hzv
```

As we know, receivers are generated by the command *Receiver*. Thus, we use the parameters listed above and using the *Receiver* command, we create and position the receivers:

```python
rec = Receiver(name='rec',grid=grid,npoint=nrec,time_range=time_range,staggered=NODE,dtype=np.float64)
rec.coordinates.data[:, 0] = nxpos
rec.coordinates.data[:, 1] = nzpos
```

The displacement field *u* is a second order field in time and space, which uses points of type *non-staggered*. In this way, we construct the displacement field *u* with the command *TimeFunction*:

```python
u = TimeFunction(name="u",grid=grid,time_order=2,space_order=2,staggered=NODE,dtype=np.float64)
```

The velocity field, the source term and receivers are defined as in the previous notebook:

```python
vel0 = Function(name="vel0",grid=grid,space_order=2,staggered=NODE,dtype=np.float64)
vel0.data[:,:] = v0[:,:]
```

```python
src_term = src.inject(field=u.forward,expr=src*dt**2*vel0**2)
```

```python
rec_term = rec.interpolate(expr=u)
```

The next step is to create the sequence of structures that reproduce the function $\zeta(x,z)$. Initially, we define the region $\Omega_{0}$, since the damping function uses the limits of that region. We previously defined the limits of the $\Omega$ region to be *x0*, *x1*, *z0* and *z1*. Now, we define the limits of the region $\Omega_{0}$ as: *x0pml* and *x1pml* in the direction $x$ and *z0pml* and *z1pml* in the direction $z$. These points satisfy the following relationships with the lengths $L_{x}$ and $L_{z}$:

- x0pml = x0 + $L_{x}$;
- x1pml = x1 - $L_{x}$;
- z0pml = z0;
- z1pml = z1 - $L_{z}$;

In terms of program variables, we have the following definitions:

```python
x0pml  = x0 + npmlx*hxv
x1pml  = x1 - npmlx*hxv
z0pml  = z0
z1pml  = z1 - npmlz*hzv
```

Having built the $\Omega$ limits, we then create a function, which we will call *fdamp*, which computationally represents the $\zeta(x,z)$ function. In the *fdamp* function, we highlight the following elements:

- *quibar* represents a constant choice for $\bar{\zeta_{1}}(x,z)$ and $\bar{\zeta_{2}}(x,z)$, satisfying $\bar{\zeta_{1}}(x,z)=\bar{\zeta_{2}}(x,z)$;
- *adamp* denotes the function $\zeta_{1}(x,z)$;
- *bdamp* denotes the function $\zeta_{2}(x,z)$;
- The terms *a* and *b* locate the $(x,z)$ points that are passed as an argument to the *fdamp* function.

The *fdamp* function is defined using the following structure:

```python
def fdamp(x,z):

    quibar  = 1.5*np.log(1.0/0.001)/(40)
    cte     = 1./vmax

    a = np.where(x<=x0pml,(np.abs(x-x0pml)/lx),np.where(x>=x1pml,(np.abs(x-x1pml)/lx),0.))
    b = np.where(z<=z0pml,(np.abs(z-z0pml)/lz),np.where(z>=z1pml,(np.abs(z-z1pml)/lz),0.))
    adamp = quibar*(a-(1./(2.*np.pi))*np.sin(2.*np.pi*a))/hxv
    bdamp = quibar*(b-(1./(2.*np.pi))*np.sin(2.*np.pi*b))/hzv
    fdamp = cte*(adamp+bdamp)

    return fdamp
```

Created the damping function, we define an array that loads the damping information in the entire domain $\Omega$. The objective is to assign this array to a *Function* and use it in the composition of the equations. To generate this array, we will use the function *generatemdamp*. In summary, this function generates a *non-staggered* grid and evaluates that grid in the *fdamp* function. At the end, we generate an array that we call *D0* and which will be responsible for providing the damping value at each of the $\Omega$ points. The *generatemdamp* function is expressed as follows:

```python
def generatemdamp():

    X0     = np.linspace(x0,x1,nptx)
    Z0     = np.linspace(z0,z1,nptz)
    X0grid,Z0grid = np.meshgrid(X0,Z0)
    D0 = np.zeros((nptx,nptz))
    D0 = np.transpose(fdamp(X0grid,Z0grid))

    return D0
```

Built the function *generatemdamp* we will execute it using the command:

```python
D0 = generatemdamp();
```

Below we include a routine to plot the damping field.

```python
def graph2damp(D):
    plot.figure()
    plot.figure(figsize=(16,8))
    fscale = 1/10**(-3)
    fscale = 10**(-3)
    scale  = np.amax(D)
    extent = [fscale*x0,fscale*x1, fscale*z1, fscale*z0]
    fig = plot.imshow(np.transpose(D), vmin=0.,vmax=scale, cmap=cm.seismic, extent=extent)
    plot.gca().xaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
    plot.gca().yaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
    plot.title('Absorbing Layer Function')
    plot.grid()
    ax = plot.gca()
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    cbar = plot.colorbar(fig, cax=cax, format='%.2e')
    cbar.set_label('Damping')
    plot.show()
```

Below we include the plot of damping field.

```python
# NBVAL_IGNORE_OUTPUT

graph2damp(D0)
```

Like the velocity function $c(x,z)$, the damping function $\zeta(x,z)$ is constant in time. Therefore, the damping function will be a second-order *Function* in space, which uses points of the non-staggered type and which we will evaluate with the D0 array. The symbolic name *damp* will be assigned to this field.

```python
damp = Function(name="damp",grid=grid,space_order=2,staggered=NODE,dtype=np.float64)
damp.data[:,:] = D0
```

The expressions for the acoustic equation with damping can be separeted between the white and blue regions.

Translating these expressions in terms of an *eq* that can be inserted in a Devito code, we have that in the white region the equation takes the form:

- eq1 = u.dt2 - vel0 * vel0 * u.laplace,

and in the blue region we have the following equation:

- eq2 = u.dt2 + vel0 * vel0 * damp * u.dtc - vel0 * vel0 * u.laplace.

Here *u.dtc* represents the centered derivative with respect to the variable $t$ for the field *u*. Then, we set the two pdes for the two regions

```python
pde0 = Eq(u.dt2 - u.laplace*vel0**2)
pde1 = Eq(u.dt2 - u.laplace*vel0**2 + vel0**2*damp*u.dtc)
```

As we did on the notebook Introduction to Acoustic Problem, we define the *stencils* for each of the *pdes* that we created previously. In the case of *pde0* it is defined only in the white region, which is represented by *subdomain* *d0*. Then, we define the *stencil0* which resolves *pde0* in *d0* and it is defined as follows:

```python
stencil0 =  Eq(u.forward, solve(pde0,u.forward),subdomain = grid.subdomains['d0'])
```

The *pde1* will be applied in the blue region, the union of the subdomains *d1*, *d2* and *d3*. In this way, we create a vector called *subds* that comprises these three *subdomains*, and we are ready to set the corresponding stencil

```python
subds = ['d1','d2','d3']
```

```python
stencil1 = [Eq(u.forward, solve(pde1,u.forward),subdomain = grid.subdomains[subds[i]]) for i in range(0,len(subds))]
```

The boundary conditions of the problem are kept the same as the notebook Introduction to Acoustic Problem. So these are placed in the term *bc* and have the following form:

```python
bc  = [Eq(u[t+1,0,z],0.),Eq(u[t+1,nptx-1,z],0.),Eq(u[t+1,x,nptz-1],0.),Eq(u[t+1,x,0],u[t+1,x,1])]
```

We then define the operator (*op*) that join the acoustic equation, source term, boundary conditions and receivers.

- 1. The acoustic wave equation in the *d0* region: *[stencil0];*
- 2. The acoustic wave equation in the *d1*, *d2* and *d3* region: *[stencil1];*
- 3. Source term: *src_term;*
- 4. Boundary conditions: *bc;*
- 5. Receivers: *rec_term;*

```python
# NBVAL_IGNORE_OUTPUT

op  = Operator([stencil0,stencil1] + src_term + bc + rec_term,subs=grid.spacing_map)
```

We reset the field *u*:

```python
u.data[:] = 0.
```

We assign in *op* the number of time steps it must execute and the size of the time step in the local variables *time* and *dt*, respectively.

```python
# NBVAL_IGNORE_OUTPUT

op(time=nt,dt=dt0)
```

    Operator `Kernel` run in 0.02 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=0.013632000000000042, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section1', rank=None),
                         PerfEntry(time=3.799999999999999e-05, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section2', rank=None),
                         PerfEntry(time=3.3e-05, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section3', rank=None),
                         PerfEntry(time=0.00027800000000000025, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section4', rank=None),
                         PerfEntry(time=0.0010460000000000092, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

To view the result of the displacement field at the end time, we use the *graph2d* routine given by:

```python
def graph2d(U):
    plot.figure()
    plot.figure(figsize=(16,8))
    fscale =  1/10**(3)
    scale  = np.amax(U[npmlx:-npmlx,0:-npmlz])/10.
    extent = [fscale*x0pml,fscale*x1pml,fscale*z1pml,fscale*z0pml]
    fig = plot.imshow(np.transpose(U[npmlx:-npmlx,0:-npmlz]),vmin=-scale, vmax=scale, cmap=cm.seismic, extent=extent)
    plot.gca().xaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
    plot.gca().yaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
    plot.axis('equal')
    plot.title('Map - Acoustic Problem with Devito')
    plot.grid()
    ax = plot.gca()
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    cbar = plot.colorbar(fig, cax=cax, format='%.2e')
    cbar.set_label('Displacement [km]')
    plot.draw()
    plot.show()
```

```python
# NBVAL_IGNORE_OUTPUT

graph2d(u.data[0,:,:])
```

Note that the solution obtained here has a reduction in noise when compared to the results displayed on the notebook Introduction to Acoustic Problem. To plot the result of the Receivers we use the *graph2drec* routine.

```python
def graph2drec(rec):
        plot.figure()
        plot.figure(figsize=(16,8))
        fscaled = 1/10**(3)
        fscalet = 1/10**(3)
        scale   = np.amax(rec[:,npmlx:-npmlx])/10.
        extent  = [fscaled*x0pml,fscaled*x1pml, fscalet*tn, fscalet*t0]
        fig = plot.imshow(rec[:,npmlx:-npmlx], vmin=-scale, vmax=scale, cmap=cm.seismic, extent=extent)
        plot.gca().xaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f km'))
        plot.gca().yaxis.set_major_formatter(mticker.FormatStrFormatter('%.1f s'))
        plot.axis('equal')
        plot.title('Receivers Signal Profile with Damping - Devito')
        ax = plot.gca()
        divider = make_axes_locatable(ax)
        cax = divider.append_axes("right", size="5%", pad=0.05)
        cbar = plot.colorbar(fig, cax=cax, format='%.2e')
        plot.show()
```

```python
# NBVAL_IGNORE_OUTPUT

graph2drec(rec.data)
```

```python
assert np.isclose(np.linalg.norm(rec.data), 990, rtol=1)
```

# 2.6 - Conclusions

- The damping strategy is a simple way to reduce artificial wave reflections coming from the computational boundaries, leading to a solution with less noise at the end of the simulation, when compared to the results of the notebook Introduction to Acoustic Problem. However, the level of artificial reflections on the boundaries is still high. In the following notebooks we present methods which are more effective.

# 2.7 - Reference

- Sochaki, J., Kubichek, R., George, J., Fletcher, W.R. and Smithson, S. (1987). "Absorbing boundary conditions and surface waves," Geophysics, 52(1), 60-71. DOI: 10.1190/1.1442241. Reference Link.