# Class 4: Plotting with Matplotlib

Matplotlib is a powerful plotting module that is part of Python's standard library. The website for matplotlib is at http://matplotlib.org/. And you can find a bunch of examples at the following two locations: http://matplotlib.org/examples/index.html and http://matplotlib.org/gallery.html.

Matplotlib contains a module called pyplot that was written to provide a Matlab-style plotting interface.

```python
# Import matplotlib.pyplot
import matplotlib.pyplot as plt
```

Next, we want to make sure that the plots that we create are displayed in this notebook. To achieve this we have to issue a command to be interpretted by Jupyter -- called a *magic* command. A magic command is preceded by a `%` character. Magics are *not* Python and will create errors if used in a Python program outside of the Jupyter notebook (e.g., if you are running Python in IDLE or executing a Python script form the command line).

```python
# Magic command for the Jupyter Notebook to display plots in the notebook
%matplotlib inline
```

## An Introductory Example

Create a plot of the sine function for x values between -6 and 6. Add axis labels and a title.

```python
# Import numpy as np
import numpy as np

# Create an array of x values from -6 to 6
x = np.arange(-6,6,0.001)

# Create a variable y equal to the sine of x
y = np.sin(x)

# Use the plot function to plot x on the horizontal axis and y on the vertical axis
plt.plot(x,y)

# Add a title and axis labels
plt.title('sin(x)')
plt.xlabel('x')
plt.ylabel('y')
```

## Style Sheets

Plots made using Matplotlib are highly customizable. Users can issue commands to control the limits of the axes, location of axis ticks, labels associated with axis ticks, and the color, thickness, opacity, and style of plotted data points.

But as a starting-point, Matplotlib uses a *style sheet* to establish initial plot settings until the user specifies different alternatives. The release of Matplotlib 2.0 included a major change in the default syle and the new default doesn't appeal to me.

```python
# Set the Matplotlib style sheet to 'classic'
plt.style.use('classic')

# Re-run the magic command for the Jupyter Notebook to display plots in the notebook
%matplotlib inline

# Use the plot function to plot the sine function
plt.plot(x,y)

# Add a title and axis labels
plt.title('sin(x)')
plt.xlabel('x')
plt.ylabel('y')
```

The matplotlib attribute `plt.style.available` stores as a `list` the default sstyle sheets that you can choose from. Here's a gallery of the different sytles compared with each other. Aside from `classic`, other styles that I think look nice include: `bmh`, `ggplot`, and `seaborn`. But try them all out and see what you like!

```python
#Print the available matplotlib style sheets. PROVIDED
print(plt.style.available)
```

    ['seaborn-dark', 'seaborn-darkgrid', 'seaborn-ticks', 'fivethirtyeight', 'seaborn-whitegrid', 'classic', '_classic_test', 'fast', 'seaborn-talk', 'seaborn-dark-palette', 'seaborn-bright', 'seaborn-pastel', 'grayscale', 'seaborn-notebook', 'ggplot', 'seaborn-colorblind', 'seaborn-muted', 'seaborn', 'Solarize_Light2', 'seaborn-paper', 'bmh', 'tableau-colorblind10', 'seaborn-white', 'dark_background', 'seaborn-poster', 'seaborn-deep']

## The `plot` Function

The `plot` function creates a two-dimensional plot of one variable against another.

```python
# Use the help function to see the documentation for plt.plot()
help(plt.plot)
```

    Help on function plot in module matplotlib.pyplot:

    plot(*args, scalex=True, scaley=True, data=None, **kwargs)
        Plot y versus x as lines and/or markers.

        Call signatures::

            plot([x], y, [fmt], data=None, **kwargs)
            plot([x], y, [fmt], [x2], y2, [fmt2], ..., **kwargs)

        The coordinates of the points or line nodes are given by *x*, *y*.

        The optional parameter *fmt* is a convenient way for defining basic
        formatting like color, marker and linestyle. It's a shortcut string
        notation described in the *Notes* section below.

        >>> plot(x, y)        # plot x and y using default line style and color
        >>> plot(x, y, 'bo')  # plot x and y using blue circle markers
        >>> plot(y)           # plot y using x as index array 0..N-1
        >>> plot(y, 'r+')     # ditto, but with red plusses

        You can use `.Line2D` properties as keyword arguments for more
        control on the appearance. Line properties and *fmt* can be mixed.
        The following two calls yield identical results:

        >>> plot(x, y, 'go--', linewidth=2, markersize=12)
        >>> plot(x, y, color='green', marker='o', linestyle='dashed',
        ...      linewidth=2, markersize=12)

        When conflicting with *fmt*, keyword arguments take precedence.

        **Plotting labelled data**

        There's a convenient way for plotting objects with labelled data (i.e.
        data that can be accessed by index ``obj['y']``). Instead of giving
        the data in *x* and *y*, you can provide the object in the *data*
        parameter and just give the labels for *x* and *y*::

        >>> plot('xlabel', 'ylabel', data=obj)

        All indexable objects are supported. This could e.g. be a `dict`, a
        `pandas.DataFame` or a structured numpy array.

        **Plotting multiple sets of data**

        There are various ways to plot multiple sets of data.

        - The most straight forward way is just to call `plot` multiple times.
          Example:

          >>> plot(x1, y1, 'bo')
          >>> plot(x2, y2, 'go')

        - Alternatively, if your data is already a 2d array, you can pass it
          directly to *x*, *y*. A separate data set will be drawn for every
          column.

          Example: an array ``a`` where the first column represents the *x*
          values and the other columns are the *y* columns::

          >>> plot(a[0], a[1:])

        - The third way is to specify multiple sets of *[x]*, *y*, *[fmt]*
          groups::

          >>> plot(x1, y1, 'g^', x2, y2, 'g-')

          In this case, any additional keyword argument applies to all
          datasets. Also this syntax cannot be combined with the *data*
          parameter.

        By default, each line is assigned a different style specified by a
        'style cycle'. The *fmt* and line property parameters are only
        necessary if you want explicit deviations from these defaults.
        Alternatively, you can also change the style cycle using the
        'axes.prop_cycle' rcParam.

        Parameters
        ----------
        x, y : array-like or scalar
            The horizontal / vertical coordinates of the data points.
            *x* values are optional. If not given, they default to
            ``[0, ..., N-1]``.

            Commonly, these parameters are arrays of length N. However,
            scalars are supported as well (equivalent to an array with
            constant value).

            The parameters can also be 2-dimensional. Then, the columns
            represent separate data sets.

        fmt : str, optional
            A format string, e.g. 'ro' for red circles. See the *Notes*
            section for a full description of the format strings.

            Format strings are just an abbreviation for quickly setting
            basic line properties. All of these and more can also be
            controlled by keyword arguments.

        data : indexable object, optional
            An object with labelled data. If given, provide the label names to
            plot in *x* and *y*.

            .. note::
                Technically there's a slight ambiguity in calls where the
                second label is a valid *fmt*. `plot('n', 'o', data=obj)`
                could be `plt(x, y)` or `plt(y, fmt)`. In such cases,
                the former interpretation is chosen, but a warning is issued.
                You may suppress the warning by adding an empty format string
                `plot('n', 'o', '', data=obj)`.

        Other Parameters
        ----------------
        scalex, scaley : bool, optional, default: True
            These parameters determined if the view limits are adapted to
            the data limits. The values are passed on to `autoscale_view`.

        **kwargs : `.Line2D` properties, optional
            *kwargs* are used to specify properties like a line label (for
            auto legends), linewidth, antialiasing, marker face color.
            Example::

            >>> plot([1,2,3], [1,2,3], 'go-', label='line 1', linewidth=2)
            >>> plot([1,2,3], [1,4,9], 'rs',  label='line 2')

            If you make multiple lines with one plot command, the kwargs
            apply to all those lines.

            Here is a list of available `.Line2D` properties:

              agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array
          alpha: float
          animated: bool
          antialiased: bool
          clip_box: `.Bbox`
          clip_on: bool
          clip_path: [(`~matplotlib.path.Path`, `.Transform`) | `.Patch` | None]
          color: color
          contains: callable
          dash_capstyle: {'butt', 'round', 'projecting'}
          dash_joinstyle: {'miter', 'round', 'bevel'}
          dashes: sequence of floats (on/off ink in points) or (None, None)
          drawstyle: {'default', 'steps', 'steps-pre', 'steps-mid', 'steps-post'}
          figure: `.Figure`
          fillstyle: {'full', 'left', 'right', 'bottom', 'top', 'none'}
          gid: str
          in_layout: bool
          label: object
          linestyle: {'-', '--', '-.', ':', '', (offset, on-off-seq), ...}
          linewidth: float
          marker: unknown
          markeredgecolor: color
          markeredgewidth: float
          markerfacecolor: color
          markerfacecoloralt: color
          markersize: float
          markevery: unknown
          path_effects: `.AbstractPathEffect`
          picker: float or callable[[Artist, Event], Tuple[bool, dict]]
          pickradius: float
          rasterized: bool or None
          sketch_params: (scale: float, length: float, randomness: float)
          snap: bool or None
          solid_capstyle: {'butt', 'round', 'projecting'}
          solid_joinstyle: {'miter', 'round', 'bevel'}
          transform: matplotlib.transforms.Transform
          url: str
          visible: bool
          xdata: 1D array
          ydata: 1D array
          zorder: float

        Returns
        -------
        lines
            A list of `.Line2D` objects representing the plotted data.

        See Also
        --------
        scatter : XY scatter plot with markers of varying size and/or color (
            sometimes also called bubble chart).

        Notes
        -----
        **Format Strings**

        A format string consists of a part for color, marker and line::

            fmt = '[color][marker][line]'

        Each of them is optional. If not provided, the value from the style
        cycle is used. Exception: If ``line`` is given, but no ``marker``,
        the data will be a line without markers.

        **Colors**

        The following color abbreviations are supported:

        =============    ===============================
        character        color
        =============    ===============================
        ``'b'``          blue
        ``'g'``          green
        ``'r'``          red
        ``'c'``          cyan
        ``'m'``          magenta
        ``'y'``          yellow
        ``'k'``          black
        ``'w'``          white
        =============    ===============================

        If the color is the only part of the format string, you can
        additionally use any  `matplotlib.colors` spec, e.g. full names
        (``'green'``) or hex strings (``'#008000'``).

        **Markers**

        =============    ===============================
        character        description
        =============    ===============================
        ``'.'``          point marker
        ``','``          pixel marker
        ``'o'``          circle marker
        ``'v'``          triangle_down marker
        ``'^'``          triangle_up marker
        ``'<'``          triangle_left marker
        ``'>'``          triangle_right marker
        ``'1'``          tri_down marker
        ``'2'``          tri_up marker
        ``'3'``          tri_left marker
        ``'4'``          tri_right marker
        ``'s'``          square marker
        ``'p'``          pentagon marker
        ``'*'``          star marker
        ``'h'``          hexagon1 marker
        ``'H'``          hexagon2 marker
        ``'+'``          plus marker
        ``'x'``          x marker
        ``'D'``          diamond marker
        ``'d'``          thin_diamond marker
        ``'|'``          vline marker
        ``'_'``          hline marker
        =============    ===============================

        **Line Styles**

        =============    ===============================
        character        description
        =============    ===============================
        ``'-'``          solid line style
        ``'--'``         dashed line style
        ``'-.'``         dash-dot line style
        ``':'``          dotted line style
        =============    ===============================

        Example format strings::

            'b'    # blue markers with default shape
            'ro'   # red circles
            'g-'   # green solid line
            '--'   # dashed line with default color
            'k^:'  # black triangle_up markers connected by a dotted line

        .. note::
            In addition to the above described arguments, this function can take a
            **data** keyword argument. If such a **data** argument is given, the
            following arguments are replaced by **data[<arg>]**:

            * All arguments with the following names: 'x', 'y'.

            Objects passed as **data** must support item access (``data[<arg>]``) and
            membership test (``<arg> in data``).

### Example

Create a plot of $f(x) = x^2$ with $x$ between -2 and 2.
* Set the linewidth to 3 points
* Set the line opacity (alpha) to 0.6
* Set axis labels and title
* Add a grid to the plot

```python
# Create an array of x values from -2 to 2
x = np.arange(-2,2,0.001)

# Create a variable y equal to x squared
y = x**2

# Use the plot function to plot the line
plt.plot(x,y,linewidth=3,alpha = 0.6)

# Add a title and axis labels
plt.title('$f(x) = x^2$')
plt.xlabel('x')
plt.ylabel('y')

# Add grid
plt.grid()
```

### Example

Create plots of the functions $f(x) = \log x$ (natural log) and $g(x) = 1/x$ between 0.01 and 5
* Make the line for $log(x)$ solid blue
* Make the line for $1/x$ dashed magenta
* Set the linewidth of each line to 3 points
* Set the line transparency (alpha) for each line to 0.6
* Set the limits for the $x$-axis to (0,5)
* Set the limits for the $y$-axis to (-2,5)
* Set axis labels and title
* Add a legend
* Add a grid to the plot

```python
# Create an array of x values from 0.01 to 5
x = np.arange(0.01,5,0.011)

# Create y variables
y1 = np.log(x)
y2 = 1/x

# Use the plot function to plot the lines
plt.plot(x,y1,'b-',linewidth=3,alpha = 0.6,label='$\log(x)$')
plt.plot(x,y2,'m--',linewidth=3,alpha = 0.6,label='$1/x$')

# Add a title and axis labels
plt.title('Two functions')
plt.xlabel('x')
plt.ylabel('y')

# Set axis limits
plt.xlim([0,5])
plt.ylim([-2,4])

# Make legend
plt.legend(loc='lower right',ncol=2)

# Add grid
plt.grid()
```

### Example

Consider the linear regression model:
\begin{align}
y_i = \beta_0 + \beta_1 x_i + \epsilon_i
\end{align}
where $x_i$ is the independent variable, $\epsilon_i$ is a random regression error term, $y_i$ is the dependent variable and $\beta_0$ and $\beta_1$ are constants.

Let's simulate the model
* Set $\beta_0=1$ and $\beta_1=-0.5$
* Create an array of $x_i$ values from -5 to 5
* Create an array of $\epsilon_i$ values from the standard normal distribution equal in length to the array of $x_i$ values
* Create an array of $y_i$ values
* Plot y against x with either a circle ('o'), triangle ('^'), or square ('s') marker and transparency (alpha) to 0.5
* Add axis lables, a title, and a grid to the plot

```python
# Set betas
beta0 = 1
beta1 = -0.5

# Create x values
x = np.arange(-5,5,0.01)

# Create epsilon values from the standard normal distribution
epsilon = np.random.normal(size=len(x))

# Create y
y = beta0 + beta1*x+epsilon

# Plot
plt.plot(x,y,'s',alpha = 0.5)

# Add a title and axis labels
plt.title('Data')
plt.xlabel('x')
plt.ylabel('y')

# Set axis limits
plt.xlim([-5,5])

# Add grid
plt.grid()
```

### Example

Create plots of the functions $f(x) = x$, $g(x) = x^2$, and $h(x) = x^3$ for $x$ between -2 and 2
* Use the optional string format argument to format the lines:
  - $x$: solid blue line
  - $x^2$: dashed green line
  - $x^3$: dash-dot magenta line
* Set the linewidth of each line to 3 points
* Set transparency (alpha) for each line to 0.7
* Set axis labels and title
* Add a legend to lower right with 3 columns
* Add a grid to the plot

```python
# Create an array of x values from -2 to 2
x = np.arange(-2,2,0.001)

# Create y variables
y1 = x
y2 = x**2
y3 = x**3

# Use the plot function to plot the lines
plt.plot(x,y1,'b-',lw=3,label='$x$',alpha=0.7)
plt.plot(x,y2,'g--',lw=3,label='$x^2$',alpha=0.7)
plt.plot(x,y3,'m-.',lw=3,label='$x^3$',alpha=0.7)

# Add a title and axis labels
plt.title('Three functions')
plt.xlabel('x')
plt.ylabel('y')

# Make legend
plt.legend(loc='lower right',ncol=3)

# Add grid
plt.grid()
```

## Figures, Axes, and Subplots

Often we want to create plots with multiple axes or we want to modify the size and shape of the plot areas. To be able to do these things, we need to explicity create a figure and then create axes within the figure. The best way to see how this works is by example.

### Example: A Single Plot with Double Width

The default dimensions of a matplotlib figure are 6 inches by 4 inches. As we saw above, this leaves some whitespace on the right side of the figure in the Notebook. Suppose we want to remove that by making the plot area twice as wide.

Plot the sine function on -6 to 6 using a figure with dimensions 12 inches by 4 inches

```python
# Create an array of x values from -6 to 6
x = np.arange(-6,6,0.001)

# Create y variables
y = np.sin(x)

# Create a new figure that is twice as wide as the default
fig = plt.figure(figsize=(12,4))

# Create axis within the figure
ax1 = fig.add_subplot(1,1,1)

# Plot
ax1.plot(x,y,lw=3,alpha = 0.6)

# Add grid
ax1.grid()
```

In the previous example the `fig = plt.figure()` function creates a new figure called `fig` and `fig.add_subplot()` puts a new axis on the figure. The command `ax1 = fig.add_subplot(1,1,1)` means divide the figure `fig` into a 1 by 1 grid and assign the first component of that grid to the variable `ax1`.

### Example: Two Plots Stacked

Create a new figure with two axes in a column and plot the sine function on -6 to 6 on the top axis and the cosine function on -6 to 6 on the bottom axis.

```python
# Create an array of x values from -6 to 6
x = np.arange(-6,6,0.001)

# Create y variables
y1 = np.sin(x)
y2 = np.cos(x)

# Create a new figure that is twice as tall as the default
fig = plt.figure(figsize=(6,8))

# Create axis 1 and plot with title and axis labels
ax1 = fig.add_subplot(2,1,1)
ax1.plot(x,y1,lw=3,alpha = 0.6)
ax1.grid()
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_title('sin')

# Create axis 2 and plot with title and axis labels
ax2 = fig.add_subplot(2,1,2)
ax2.plot(x,y2,lw=3,alpha = 0.6)
ax2.grid()
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('cos')

# Improve the spacing between the plots
fig.tight_layout()
```

### Example: Block of Four Plots

The default dimensions of a matplotlib figure are 6 inches by 4 inches. As we saw above, this leaves some whitespace on the right side of the figure. Suppose we want to remove that by making the plot area twice as wide.

Create a new figure with four axes in a two-by-two grid. Plot the following functions on the interval -2 to 2:
* $y = x$
* $y = x^2$
* $y = x^3$
* $y = x^4$

Set the figure size to 12in. by 8in. Run the command `plt.tight_layout()` to adust the figure's margins after creating your figure, axes, and plots.

```python
# Create data for plot
x = np.arange(-2,2,0.001)
y1 = x
y2 = x**2
y3 = x**3
y4 = x**4

# Create a new figure that is twice as wide and twice as tall as the default
fig = plt.figure(figsize=(12,8))

# Create axis 1 and plot with title
ax1 = fig.add_subplot(2,2,1)
ax1.plot(x,y1,lw=3,alpha = 0.6)
ax1.grid()
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_title('$x$')

# Create axis 2 and plot with title
ax2 = fig.add_subplot(2,2,2)
ax2.plot(x,y2,lw=3,alpha = 0.6)
ax2.grid()
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('$x^2$')

# Create axis 3 and plot with title
ax3 = fig.add_subplot(2,2,3)
ax3.plot(x,y3,lw=3,alpha = 0.6)
ax3.grid()
ax3.set_xlabel('x')
ax3.set_ylabel('y')
ax3.set_title('$x^3$')

# Create axis 4 and plot with title
ax4 = fig.add_subplot(2,2,4)
ax4.plot(x,y4,lw=3,alpha = 0.6)
ax4.grid()
ax4.set_xlabel('x')
ax4.set_ylabel('y')
ax4.set_title('$x^4$')

# Improve the spacing between the plots
plt.tight_layout()
```

## Exporting Figures to Image Files

Use the `plt.savefig()` function to save figures to images.

```python
# Create data to plot sin(x)/x on [-20,20]
x = np.arange(-20,20,0.001)
y = np.sin(x)/x

# Plot y against x
plt.plot(x,y)

# Save figure as 'fig_econ126_class04_sine_over_x.png' at 120 DPI resolution
plt.savefig('fig_econ126_class04_sine_over_x.png',dpi=120)
```

In the previous example, the image is saved as a PNG file with 120 dots per inch. This resolution is high enough to look good even when projected on a large screen. The image format is inferred by the extension on the filename.


# Intertemporal consumer problem

Team: M&M

Members: Markus Gorgone Larsen (hbk716) & Matias Bjørn Frydensberg Hall (pkt593)

Imports and set magics:

```python
import numpy as np
from scipy import optimize
from types import SimpleNamespace
import sympy as sm
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
plt.style.use('seaborn-whitegrid')

# autoreload modules when code is run
%load_ext autoreload
%autoreload 2

# local modules
import modelproject as mp
```

# Model description

We consider a standard **intertemporal consumer utility function in two periods** known from Macro II where:

* $C_t$ is consumption in period t
* $Y_t^L$ is labour income in period t
* $V_1$ is the intial wealth
* $T_t$ is lump sum tax payment in period t
* $\phi$ is the degree of impatience
* $r$ is the real rental rate

all for $t\in\{1,2\}$
We also have $C_t > 0$ and $\phi > 0$

$$ U = u(C_1) + \frac{u(C_2)}{1+\phi}, \qquad u^{''} < 0 $$

Given **saving** in period 1 we can find **wealth in period 2**:

$$ V_2 = (1+r)\bigg(V_1+Y_1^L-T_1-C_1\bigg) = (1+r)S_1 $$

And **comsumption in period 2** becomes:

$$ C_2 = V_2 + Y_2^L - T_2 $$

Consolidating the two we get the **intertemporal budget constraint**:

$$C_1 + \frac{C_2}{1+r} = V_1+Y_1^L-T_1+\frac{Y_2^L-T_2}{1+r} $$

Inserting the expression for $C_2$ into the utility function we get:

$$ U = u(C_1) + \frac{u((1+r)(V_1+Y_1^L-T_1-C_1)+Y_2^L-T_2)}{1+\phi} $$

Assuming our consumer is risk averse we can use the following **utility function**:

$$ u(C_t) = lnC_t $$

Which gives us the following **maximisation problem** to solve:

\begin{equation}
C_1^* = \text{arg}\max_{C_1} \, lnC_1 + \frac{ln((1+r)(V_1+Y_1^L-T_1-C_1)+Y_2^L-T_2)}{1+\phi}
\end{equation}

Further more we know from solving the model analytical that the solution must satisfy:

$$ u'(C_1) = \frac{1+r}{1+\phi}u'(C_2) $$

# Algorithm

Given the model above we can now describe our algorithm:

**Problem:** Solve the maximisation problem above
**Inputs:** The variables mentioned above
**Outputs:** $C_1, \, C_2, \, U$
**Algorithm:** `U_optimize()`
  * Use scipy to numerically optimize U given parameters

# Solving the model

For simplicity, we assume that income is the same in both periods. We set the level of impatience and rental rate equal to each other and close to zero, as the rental rate is close to zero right now. The initial endowment is also picked arbitrarily. Our above stated theory says that consumption must be the same in both periods with the following parameters.

```python
# a. Create simplenamespace and set parameter values

par = SimpleNamespace()

par.r = 0.02
par.V_1 = 5
par.Y_L1 = 2
par.Y_L2 = 2
par.T_1 = 0.5
par.T_2 = 0.5
par.phi = 0.02

# b. Compute optimal housing quality, consumption and utility
c_1star, c_2star, u_star = mp.u_optimise(par)

# c. Print solution
print(f'The agent will choose optimal consumption in period 1 = {c_1star:.2f}, which implies optimal consumption in period 2 = {c_2star:.2f} and utility = {u_star:.2f}')

```

    The agent will choose optimal consumption in period 1 = 4.02, which implies optimal consumption in period 2 = 4.02 and utility = 2.76

Our computation is what we could expect, as the consumer has no preference for consumption in one of the two periods, as the rental rate equals the level of impatience. We infer that the result is correct. Now we will get to the visualising of the results.

# Visualising consumption in both periods

For visualising the results we will show how consumption in the two periods varies with income levels. We will also take a look at utility.

Our method for visualising in 2d will be the following:

1. Create array with relevant variable we want to change
2. Create empty containers for $C_1$, $C_2$ and U
3. Maximise U looping over the array and save values of $C_1$, $C_2$ and U in containers
4. Plot the findings

We have created a function that does the first 2 out of 4 steps. We will start with income:

```python
# a. Set number of objects and create array of Y_L1's/Y_L2's and container for C_1*, C_2* and U*
par.N = 1000

Y_L1_vec, c1_vec, c2_vec, u_vec = mp.array(0.5,5, par.N)
Y_L2_vec, c1_vec, c2_vec, u_vec = mp.array(0.5,5, par.N)
# b. Loop the optimise function over the Y_Lt_vec arrays
for i in range(par.N):
    par.Y_L1 = Y_L1_vec[i]
    c1_vec[i], c2_vec[i], u_vec[i] = mp.u_optimise(par)

par.Y_L1 = 2

for i in range(par.N):
    par.Y_L2 = Y_L2_vec[i]
    c1_vec[i], c2_vec[i], u_vec[i] = mp.u_optimise(par)

par.Y_L2 = 2

# c. Create graphs and plot
mp.two_figures(Y_L1_vec, c1_vec, "Consumption in period 1", "$Y^L_1$", "$C_1$", Y_L1_vec, c2_vec, "Consumption in period 2", "$Y^L_1$", "$C_2$")
mp.one_figure(Y_L1_vec, u_vec, "Utility", "$Y^L_1$", "$U$")

mp.two_figures(Y_L2_vec, c1_vec, "Consumption in period 1", "$Y^L_2$", "$C_1$", Y_L2_vec, c2_vec, "Consumption in period 2", "$Y^L_2$", "$C_2$")
mp.one_figure(Y_L2_vec, u_vec, "Utility", "$Y^L_2$", "$U$")
```

Above we se that both utility curves are concave. This is due to the specification of the utility function. We also see not suprisingly that when labour income increases, consumption in both periods go up. We know that an increase in the labour income in period 1 increases overall income slightly more due to the rental rate. But as we see above it doesn't have a noticeable effect on utility or consumption.

# How the rental rate and impatience affect allocation

Now we would like to show how varying degrees of impatience influence consumption in both periods. Likewise we would like to illustrate how different levels of rental rates affect consumption in both periods. Our method is going to be the same as described above.
Let's start with patience:

```python
# a. Create array of phi's and container for C_1*, C_2* and U*
phi_vec, c1_vec, c2_vec, u_vec = mp.array(0.001, 0.5, par.N)

# b. Loop the optimise function over the phi_vec array
for i in range(par.N):
    par.phi = phi_vec[i]
    c1_vec[i], c2_vec[i], u_vec[i] = mp.u_optimise(par)

# c. Create graph and plot
mp.two_figures(phi_vec, c1_vec, "Consumption in period 1", "$Impatience$", "$C_1$", phi_vec, c2_vec, "Consumption in period 2", "$Impatience$", "$C_2$")
mp.one_figure(phi_vec, u_vec, "Utility", "$Impatience$", "$Utility$")

# d. Reset phi
par.phi = 0.02
```

Hence we find that when the level of impatience increases, consumption in period 1 increases while it decreases in period 2(not suprising), while the overall utility falls. This is a consequense of the concave utility functions, why the increased consumption in period 1 can't fully compensate for the drop in consumption in period 2.
Next we turn to the rental rate:

```python
# a. Create array of r's and container for C_1*, C_2* and U*
r_vec, c1_vec, c2_vec, u_vec = mp.array(0.001,0.2, par.N)

# b. Loop the optimise function over the r_vec array
for i in range(par.N):
    par.r = r_vec[i]
    c1_vec[i], c2_vec[i], u_vec[i] = mp.u_optimise(par)

# c. Create graph and plot
mp.two_figures(r_vec, c1_vec, "Consumption in period 1", "$Rental \: rate$", "$C_1$", r_vec, c2_vec, "Consumption in period 2", "$Rental \: rate$", "$C_2$")
mp.one_figure(r_vec, u_vec, "Utility", "$Rental \: rate$", "$U$")

# d. Reset r
par.r = 0.02
```

Above we see that consumption in period 1 is decreasing in r while consumption in period 2 is increasing, why we have that the substitution effect dominates the income effect. We also see that utility is increasing in the rental rate which is because of direct postive income effect a higher rental rate has.

An interesting point to visualize is the combined effects of the rental rate and impatience on utility. We therefore set up a 3D function, where the combinations $r$ and $\phi$ are visualized.

```python
#To create the 3D graph, we first create our variables to input.
#We need three for the 3-dimensional space, who all need to be 2-dimensional
N = 100
shape_tuple = (N,N)
r_values = np.empty(shape_tuple)
phi_values = np.empty(shape_tuple)
u_values = np.empty(shape_tuple)

#After creating our empty tuples, we can now begin insert values into them.
#We run our optimization function for every combination of phi and r that exists between 0 and 1 with two decimals.

for i in range(N):
        for j in range(N):
            par.r = i/100
            par.phi = j/100
            r_values[i,j] = par.r
            phi_values[i,j] = par.phi
            c1, c2, u = mp.u_optimise(par)
            u_values[i,j] = u

#We now have our three filled tuples, we now can plot them together.
#We use a widget here for better visualization.
#NOTE: Rerun all cells to see the widget.
%matplotlib widget
fig = plt.figure(figsize=(10,7))
ax = plt.axes(projection='3d')
ax.plot_surface(phi_values,r_values, u_values, rstride=1, cstride=1,
                cmap='viridis', edgecolor='none');
ax.set_xlabel('$\phi$')
ax.set_ylabel('$r$')
ax.set_zlabel('$U$')
ax.set_title('Utility for values of $\phi$ and $r$');
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

As we can see in the above graph, we come to the same conclusion as with our 2D graphs. Impatience decreases the overall utility for any given value of the rental rate, while the rental rate increases utility for any given rate of the impatience. We see that the maximum utility given that impatience and the rental rate would be non-fixed is the point, where the consumer has "limitless patience" and the rental rate is as high as it can get.

# Extensions to the baseline model

As an extension to out model we will give the consumer an option of a risk bearing asset, while stile having the option to invest in the risk free assest, which is the equivilent of the rental rate in the baseline model.
The **risky asset**, $R$, give an expected return of:

$$ E[R] = \beta*(1+r_{rb}) + (1-\beta)*0 $$

While the expected return of the **risk-free assest**, $RF$, is:

$$ E[RF] = 1 + r_{rf} $$

We also set the percentage of the savings used on the risk free asset equal to $\alpha$

With this new addition to our model the **new intertemporal budget constraint** becomes:

$$ C_1 + \frac{C_2}{\alpha(1+r_{rf})+\beta(1-\alpha)(1+r_{rb})} = V_1 + Y_1^L - T_1 + \frac{1}{\alpha(1+r_{rf})+\beta(1-\alpha)(1+r_{rb})}(Y_2^L-T_2) $$

And the **new maximisation problem** becomes:

$$ C_1^* = \text{arg}\max_{C_1, \, \alpha} \, lnC_1 + \frac{ln(\alpha(1+r_{rf})(V_1+Y_1^L-T_1-C_1)+\beta(1-\alpha)(1+r_{rb})(V_1+Y_1^L-T_1-C_1)+Y_2^L-T_2)}{1+\phi} \quad st. 0 \leq \alpha \leq 1$$

Now we can begin to solve the model using the same method as before:

```python
# a. Create simplenamespace and set parameter values

par = SimpleNamespace()

par.r_rf = 0.02
par.r_rb = 1.5
par.V_1 = 5
par.Y_L1 = 2
par.Y_L2 = 2
par.T_1 = 0.5
par.T_2 = 0.5
par.phi = 0.02
par.alpha = 0.5
par.beta = 0.5

# b. Compute optimal housing quality, consumption and utility
c_1star, c_2star, alpha_star, u_star = mp.u_optimise_ext(par)

# c. Print solution
#print(f'The agent will choose optimal consumption in period 1 = {c_1star:.2f}, which implies optimal consumption in period 2 = {c_2star:.2f} and utility = {u_star:.2f}')
```

We were not able to able to figure out how to optimise given two variables(both alpha and C_1 in our case) but would like to implement that before the exam, as that's what the extension to the model actually suggests. Regardless the extension show that given the new risk bearing asset and an $\alpha = 0.5$ consumption goes up in period 1 and down in period 2.

```python
print(c_1star, c_2star, alpha_star, u_star )
```

    3.888118512142534 4.764851859821833 0.0 2.888578747449301

```python

```

## An implementation incorporating risk aversion

$$ E[R] = \beta*(1+r_{rb}) + (1-\beta)*0 $$

While the expected return of the **risk-free assest**, $RF$, is:

$$ E[RF] = 1 + r_{rf} $$

$$ E(U) = u(C_1) + \frac{E\left(u(C_2)\right)}{1+\phi}, \qquad u^{''} < 0 $$

Given **saving** in period 1 we can find **wealth in period 2**:

$$S_{1} = V_1+Y_1^L-T_1-C_1 $$

$$
\begin{aligned}
V_2 &= \begin{cases}
\alpha\cdot(1+r_{rf})S_{1}   + \left(1-\alpha\right)\left(1+r_{rb}\right)\cdot S_{1} & \text{with prob. } \beta \\
\alpha\cdot(1+r_{rf})S_{1} & \text{with prob. } 1-\beta
\end{cases}
\end{aligned}
$$

And **comsumption in period 2** becomes:

$$ C_2 = V_2 + Y_2^L - T_2 $$

Thus, utilty in period 2 is:
$$
\begin{aligned}
C_{2} &= \begin{cases}
\ln \left( \alpha\cdot(1+r_{rf})S_{1}   + \left(1-\alpha\right)\left(1+r_{rb}\right)\cdot S_{1} + Y_{2}^{L}-T_{2} \right) & \text{with prob. } \beta \\
\ln \left( \alpha\cdot(1+r_{rf})S_{1}   + Y_{2}^{L}-T_{2} \right) & \text{with prob. } 1-\beta
\end{cases}
\end{aligned}
$$

And the expectation in period 1 is:

$$
E\left(u\left(C_{2}\right)\vert S_{1}, \alpha \right) = \beta\cdot\ln \left( \alpha\cdot(1+r_{rf})S_{1}   + \left(1-\alpha\right)\left(1+r_{rb}\right)\cdot S_{1} + Y_{2}^{L}-T_{2} \right) + (1-\beta)\cdot\ln \left( \alpha\cdot(1+r_{rf})S_{1}   + Y_{2}^{L}-T_{2} \right)
$$
With $S_{1} = V_1+Y_1^L-T_1-C_1 $$

So the optimization problem becomes:

$$ C_1^*,\alpha^* = \text{args}\max_{C_1, \, \alpha} \, ln( C_1) + \frac{\beta\cdot \left(\ln \left( \alpha\cdot(1+r_{rf})\left(V_1+Y_1^L-T_1-C_1\right)  + \left(1-\alpha\right)\left(1+r_{rb}\right)\cdot\left(V_1+Y_1^L-T_1-C_1\right) + Y_{2}^{L}-T_{2} \right) \right) +(1-\beta) \cdot \left( \ln \left( \alpha\cdot(1+r_{rf})\left(V_1+Y_1^L-T_1-C_1\right)   + Y_{2}^{L}-T_{2} \right)   \right )}{1+\phi} \quad st. 0 \leq \alpha \leq 1$$

An upper bound on $C_1$ is also imposed, in either possible states the agent must be able to pay all taxes in period 2 and also pay back any debt. (the possible $C_{2}$ must both be positive):

$$
\alpha\cdot(1+r_{rf})\left(V_1+Y_1^L-T_1-C_1\right)  + \left(1-\alpha\right)\left(1+r_{rb}\right)\cdot\left(V_1+Y_1^L-T_1-C_1\right) + Y_{2}^{L}-T_{2}>0
$$
$$
\land
$$
$$
\alpha\cdot(1+r_{rf})\left(V_1+Y_1^L-T_1-C_1\right)   + Y_{2}^{L}-T_{2} >0
$$

This could possible be implemented as a bound, if one derived C1 in both cases and the upper bound would be the minimum of those two

```python
par = SimpleNamespace()

par.r_rf = 0.02
par.r_rb = 1.2
par.V_1 = 5
par.Y_L1 = 2
par.Y_L2 = 2
par.T_1 = 0.5
par.T_2 = 0.5
par.phi = 0.02
par.alpha = 0.2
par.beta = 0.5
c_1star, c_2star_good,c_2star_bad, alpha_star, u_star = mp.u_optimise_ext2(par)
```

```python
print(c_1star, c_2star_good,c_2star_bad, alpha_star, u_star)
```

    4.024764002719247 4.3403959218957375 3.751886218274879 0.8919277147590657 2.7602232739628265

# Conclusion

```python

```

In this assignment we have started with an two-period intertemporal consumer utility function and attempted to solve the maximisation problem presented using u_optimize in the scipy package. We have then attempted to chart the deeper mechanics of the model by visualizing how the model responds to changes in income levels across the two periods. We found that Consumption is very obviously improved by increased income, with a slighty bigger increase when the income rises in period 1. We have then visualized how the rental rate and impatience affects consumption and utility, finding that impatience decreases overall utility by consuming more in period 1, thereby removing some benefit from the rental rate. The rental rate increases the overall utility by consuming more in period 2. We also showed how different combinations of impatience and the rental rate affect utility, giving us the same conclusions as before. Lastly, we extent our model to encompass risk through risk-free and risky assets that also changes the consumer's behavior. While our implementation is limited, we do find that with an $\alpha$ = 0.5, we achieve increased consumption in period 1 compared to before.

Further extensions to the model could be changing the utility function to be able to vary the degree of risk aversion. We could also have modelled labour, making the consumer have a choice between labour and freetime. To make the model more realistic the tax could also be calculated from a constant tax rate based on the labour income, instead of being lump sum. Also, simulating our extension model could reveal if the agent would get the same utility as in expectation.

```python

```


# Linearization at the differential equation level

The attention is now turned to nonlinear partial differential
equations (PDEs) and application of the techniques explained above for
ODEs.  The model problem is a nonlinear diffusion equation for
$u(\x,t)$:

$$
\begin{equation}
\frac{\partial u}{\partial t} = \nabla\cdot (\dfc(u)\nabla u) + f(u),\quad
\x\in\Omega,\ t\in (0,T],
\label{nonlin:pdelevel:model:pde} \tag{1}
\end{equation}
$$

$$
\begin{equation}
-\dfc(u)\frac{\partial u}{\partial n} = g,\quad \x\in\partial\Omega_N,\
t\in (0,T],
\label{nonlin:pdelevel:model:Neumann} \tag{2}
\end{equation}
$$

$$
\begin{equation}
u = u_0,\quad \x\in\partial\Omega_D,\ t\in (0,T]\thinspace .
\label{nonlin:pdelevel:model:Dirichlet} \tag{3}
\end{equation}
$$

In the present section, our aim is to discretize this problem in time
and then present techniques for linearizing the time-discrete PDE
problem "at the PDE level" such that we transform the nonlinear
stationary PDE problem at each time level into a sequence of linear
PDE problems, which can be solved using any method for linear
PDEs. This strategy avoids the solution of systems of nonlinear
algebraic equations.  In the section [1D stationary nonlinear differential equations](#nonlin:alglevel:1D) we shall take
the opposite (and more common) approach: discretize the nonlinear
problem in time and space first, and then solve the resulting
nonlinear algebraic equations at each time level by the methods of
the section [nonlin:systems:alg](#nonlin:systems:alg).  Very often, the two approaches are
mathematically identical, so there is no preference from a
computational efficiency point of view.  The details of the ideas
sketched above will hopefully become clear through the forthcoming
examples.

## Explicit time integration

The nonlinearities in the PDE are trivial to deal with if we choose an
explicit time integration method for ([1](#nonlin:pdelevel:model:pde)),
such as the Forward Euler method:

$$
[D_t^+ u = \nabla\cdot (\dfc(u)\nabla u) + f(u)]^n,
$$

or written out,

$$
\frac{u^{n+1} - u^n}{\Delta t} = \nabla\cdot (\dfc(u^n)\nabla u^n)
+ f(u^n),
$$

which is a linear equation in the unknown $u^{n+1}$ with solution

$$
u^{n+1} = u^n + \Delta t\nabla\cdot (\dfc(u^n)\nabla u^n) +
\Delta t f(u^n)\thinspace .
$$

The disadvantage with this discretization is
the strict stability criterion $\Delta t \leq h^2/(6\max\alpha)$
for the case $f=0$ and a standard 2nd-order finite difference discretization
in 3D space with mesh cell sizes $h=\Delta x=\Delta y=\Delta z$.

## Backward Euler scheme and Picard iteration

A Backward Euler scheme for ([1](#nonlin:pdelevel:model:pde))
reads

$$
[D_t^- u = \nabla\cdot (\dfc(u)\nabla u) + f(u)]^n\thinspace .
$$

Written out,

$$
\begin{equation}
\frac{u^{n} - u^{n-1}}{\Delta t} = \nabla\cdot (\dfc(u^n)\nabla u^n)
+ f(u^n)\thinspace .
\label{nonlin:pdelevel:pde:BE} \tag{4}
\end{equation}
$$

This is a nonlinear PDE for the unknown function $u^n(\x)$. Such a
PDE can be viewed as a time-independent PDE where
$u^{n-1}(\x)$ is a known function.

We introduce a Picard iteration with $k$ as iteration counter.
A typical linearization of the $\nabla\cdot(\dfc(u^n)\nabla u^n)$ term
in iteration $k+1$ is to use the previously computed $u^{n,k}$
approximation in the diffusion coefficient: $\dfc(u^{n,k})$.
The nonlinear source term is treated similarly: $f(u^{n,k})$.
The unknown function $u^{n,k+1}$ then fulfills the linear PDE

$$
\begin{equation}
\frac{u^{n,k+1} - u^{n-1}}{\Delta t} = \nabla\cdot (\dfc(u^{n,k})
\nabla u^{n,k+1})
+ f(u^{n,k})\thinspace .
\label{nonlin:pdelevel:pde:BE:Picard:k} \tag{5}
\end{equation}
$$

The initial guess for the Picard iteration at this time level can be
taken as the solution at the previous time level: $u^{n,0}=u^{n-1}$.

We can alternatively apply the implementation-friendly
notation where $u$ corresponds to
the unknown we want to solve for, i.e., $u^{n,k+1}$ above, and $u^{-}$
is the most recently computed value, $u^{n,k}$ above. Moreover,
$u^{(1)}$ denotes the unknown function at the previous time level, $u^{n-1}$
above. The PDE to be solved in a Picard iteration then looks like

$$
\begin{equation}
\frac{u - u^{(1)}}{\Delta t} = \nabla\cdot (\dfc(u^{-})
\nabla u)
+ f(u^{-})\thinspace .
\label{nonlin:pdelevel:pde:BE:Picard} \tag{6}
\end{equation}
$$

At the beginning of the iteration we start with the value from the
previous time level: $u^{-}=u^{(1)}$, and after each
iteration, $u^{-}$ is updated to $u$.

**Remark on notation.**

The previous derivations of the numerical scheme for time discretizations
of PDEs have, strictly
speaking, a somewhat sloppy notation, but it is much used and convenient
to read. A more precise notation must
distinguish clearly between the exact solution of the PDE problem,
here denoted $\uex(\x,t)$, and the exact solution of the spatial
problem, arising after time discretization at each time level,
where ([4](#nonlin:pdelevel:pde:BE)) is an example. The latter
is here represented as $u^n(\x)$ and is an approximation to
$\uex(\x,t_n)$. Then we have another approximation $u^{n,k}(\x)$
to $u^n(\x)$ when solving the nonlinear PDE problem for
$u^n$ by iteration methods, as in ([5](#nonlin:pdelevel:pde:BE:Picard:k)).

In our notation, $u$ is a synonym for $u^{n,k+1}$ and $u^{(1)}$ is
a synonym for $u^{n-1}$, inspired by what are natural variable names
in a code.
We will usually state the PDE problem in terms of $u$ and
quickly redefine the symbol $u$ to mean the numerical approximation,
while $\uex$ is not explicitly introduced unless we need to talk about
the exact solution and the approximate solution at the same time.

## Backward Euler scheme and Newton's method

At time level $n$, we have to solve the stationary PDE
([4](#nonlin:pdelevel:pde:BE)). In the previous section, we
saw how this can be done with Picard iterations.
Another alternative is to apply the idea of Newton's method
in a clever way.
Normally, Newton's method is defined for systems of *algebraic equations*,
but the idea of the method can be applied at the PDE level too.

### Linearization via Taylor expansions

Let $u^{n,k}$ be an approximation to the unknown $u^n$. We seek a
better approximation on
the form

$$
\begin{equation}
u^{n} = u^{n,k} + \delta u\thinspace .
\label{nonlin:pdelevel:Newton:ansatz} \tag{7}
\end{equation}
$$

The idea is to insert ([7](#nonlin:pdelevel:Newton:ansatz)) in
([4](#nonlin:pdelevel:pde:BE)), Taylor expand the nonlinearities
and keep only the terms that are
linear in $\delta u$ (which makes ([7](#nonlin:pdelevel:Newton:ansatz))
an approximation for $u^{n}$). Then we can solve a linear PDE for
the correction $\delta u$ and use ([7](#nonlin:pdelevel:Newton:ansatz))
to find a new approximation

$$
u^{n,k+1}=u^{n,k}+\delta u
$$

to $u^{n}$.
Repeating this procedure gives a sequence $u^{n,k+1}$, $k=0,1,\ldots$
that hopefully converges to the goal $u^n$.

Let us carry out all the mathematical details for the nonlinear diffusion
PDE discretized by the Backward Euler method.
Inserting ([7](#nonlin:pdelevel:Newton:ansatz)) in
([4](#nonlin:pdelevel:pde:BE)) gives

$$
\begin{equation}
\frac{u^{n,k} +\delta u - u^{n-1}}{\Delta t} =
\nabla\cdot (\dfc(u^{n,k} + \delta u)\nabla (u^{n,k}+\delta u))
+ f(u^{n,k}+\delta u)\thinspace .
\label{nonlin:pdelevel:pde:BE:Newton1} \tag{8}
\end{equation}
$$

We can Taylor expand $\dfc(u^{n,k} + \delta u)$ and
$f(u^{n,k}+\delta u)$:

$$
\begin{align*}
\dfc(u^{n,k} + \delta u) & = \dfc(u^{n,k}) + \frac{d\dfc}{du}(u^{n,k})
\delta u + \Oof{\delta u^2}\approx \dfc(u^{n,k}) + \dfc^{\prime}(u^{n,k})\delta u,\\
f(u^{n,k}+\delta u) &=  f(u^{n,k}) + \frac{df}{du}(u^{n,k})\delta u
+ \Oof{\delta u^2}\approx f(u^{n,k}) + f^{\prime}(u^{n,k})\delta u\thinspace .
\end{align*}
$$

Inserting the linear approximations of $\dfc$ and $f$ in
([8](#nonlin:pdelevel:pde:BE:Newton1)) results in

$$
\frac{u^{n,k} +\delta u - u^{n-1}}{\Delta t} =
\nabla\cdot (\dfc(u^{n,k})\nabla u^{n,k}) + f(u^{n,k}) + \nonumber
$$

$$
\qquad \nabla\cdot (\dfc(u^{n,k})\nabla \delta u)
+ \nabla\cdot (\dfc^{\prime}(u^{n,k})\delta u\nabla u^{n,k}) + \nonumber
$$

$$
\begin{equation}
\qquad \nabla\cdot (\dfc^{\prime}(u^{n,k})\delta u\nabla \delta u)
+ f^{\prime}(u^{n,k})\delta u\thinspace .
\label{nonlin:pdelevel:pde:BE:Newton2} \tag{9}
\end{equation}
$$

The term $\dfc^{\prime}(u^{n,k})\delta u\nabla \delta u$ is of
order $\delta u^2$
and therefore omitted since we expect the correction $\delta u$
to be small ($\delta u \gg \delta u^2$).
Reorganizing the equation gives a PDE
for $\delta u$ that we can write in short form as

$$
\delta F(\delta u; u^{n,k}) = -F(u^{n,k}),
$$

where

$$
\begin{equation}
F(u^{n,k}) = \frac{u^{n,k} - u^{n-1}}{\Delta t} -
\nabla\cdot (\dfc(u^{n,k})\nabla u^{n,k}) + f(u^{n,k}),
\label{nonlin:pdelevel:pde:BE:Newton2:F} \tag{10}
\end{equation}
$$

$$
\delta F(\delta u; u^{n,k}) =
- \frac{1}{\Delta t}\delta u +
\nabla\cdot (\dfc(u^{n,k})\nabla \delta u) + \nonumber
$$

$$
\begin{equation}
\quad \nabla\cdot (\dfc^{\prime}(u^{n,k})\delta u\nabla u^{n,k})
+ f^{\prime}(u^{n,k})\delta u\thinspace .
\label{_auto1} \tag{11}
\end{equation}
$$

Note that $\delta F$ is a linear function of $\delta u$, and
$F$ contains only terms that are known, such that
the PDE for $\delta u$ is indeed linear.

**Observations.**

The notational form $\delta F = -F$ resembles the Newton system $J\delta u =-F$
for systems of algebraic equations, with $\delta F$ as $J\delta u$.
The unknown vector in a linear system of algebraic equations enters
the system as a linear operator in terms of a
matrix-vector product ($J\delta u$), while at
the PDE level we have a linear differential operator instead
($\delta F$).

### Similarity with Picard iteration

We can rewrite the PDE for $\delta u$ in a slightly different way too
if we define $u^{n,k} + \delta u$ as $u^{n,k+1}$.

$$
\frac{u^{n,k+1} - u^{n-1}}{\Delta t} =
\nabla\cdot (\dfc(u^{n,k})\nabla u^{n,k+1}) + f(u^{n,k})\nonumber
$$

$$
\begin{equation}
\qquad  + \nabla\cdot (\dfc^{\prime}(u^{n,k})\delta u\nabla u^{n,k})
+ f^{\prime}(u^{n,k})\delta u\thinspace .
\label{_auto2} \tag{12}
\end{equation}
$$

Note that the first line is the same PDE as arises in the Picard
iteration, while the remaining terms arise from the differentiations
that are an inherent ingredient in Newton's method.

### Implementation

For coding we want to introduce $u$ for $u^n$, $u^{-}$ for $u^{n,k}$ and
$u^{(1)}$ for $u^{n-1}$. The formulas for $F$ and $\delta F$
are then more clearly written as

$$
\begin{equation}
F(u^{-}) = \frac{u^{-} - u^{(1)}}{\Delta t} -
\nabla\cdot (\dfc(u^{-})\nabla u^{-}) + f(u^{-}),
\label{nonlin:pdelevel:pde:BE:Newton2:F2} \tag{13}
\end{equation}
$$

$$
\delta F(\delta u; u^{-}) =
- \frac{1}{\Delta t}\delta u +
\nabla\cdot (\dfc(u^{-})\nabla \delta u) + \nonumber
$$

$$
\begin{equation}
\quad \nabla\cdot (\dfc^{\prime}(u^{-})\delta u\nabla u^{-})
+ f^{\prime}(u^{-})\delta u\thinspace .
\label{_auto3} \tag{14}
\end{equation}
$$

The form that orders the PDE as the Picard iteration terms plus
the Newton method's derivative terms becomes

$$
\frac{u - u^{(1)}}{\Delta t} =
\nabla\cdot (\dfc(u^{-})\nabla u) + f(u^{-}) + \nonumber
$$

$$
\begin{equation}
\qquad  \gamma(\nabla\cdot (\dfc^{\prime}(u^{-})(u - u^{-})\nabla u^{-})
+ f^{\prime}(u^{-})(u - u^{-}))\thinspace .
\label{_auto4} \tag{15}
\end{equation}
$$

The Picard and full Newton versions correspond to
$\gamma=0$ and $\gamma=1$, respectively.

### Derivation with alternative notation

Some may prefer to derive the linearized PDE for $\delta u$ using
the more compact notation. We start with inserting $u^n=u^{-}+\delta u$
to get

$$
\frac{u^{-} +\delta u - u^{n-1}}{\Delta t} =
\nabla\cdot (\dfc(u^{-} + \delta u)\nabla (u^{-}+\delta u))
+ f(u^{-}+\delta u)\thinspace .
$$

Taylor expanding,

$$
\begin{align*}
\dfc(u^{-} + \delta u) & \approx \dfc(u^{-}) + \dfc^{\prime}(u^{-})\delta u,\\
f(u^{-}+\delta u) & \approx f(u^{-}) + f^{\prime}(u^{-})\delta u,
\end{align*}
$$

and inserting these expressions gives a less cluttered PDE for $\delta u$:

$$
\begin{align*}
\frac{u^{-} +\delta u - u^{n-1}}{\Delta t} &=
\nabla\cdot (\dfc(u^{-})\nabla u^{-}) + f(u^{-}) + \\
&\qquad \nabla\cdot (\dfc(u^{-})\nabla \delta u)
+ \nabla\cdot (\dfc^{\prime}(u^{-})\delta u\nabla u^{-}) + \\
&\qquad \nabla\cdot (\dfc^{\prime}(u^{-})\delta u\nabla \delta u)
+ f^{\prime}(u^{-})\delta u\thinspace .
\end{align*}
$$

## Crank-Nicolson discretization

A Crank-Nicolson discretization of
([1](#nonlin:pdelevel:model:pde)) applies a centered difference
at $t_{n+\frac{1}{2}}$:

$$
[D_t u = \nabla\cdot (\dfc(u)\nabla u) + f(u)]^{n+\frac{1}{2}}\thinspace .
$$

The standard technique is to apply an arithmetic average for
quantities defined between two mesh points, e.g.,

$$
u^{n+\frac{1}{2}}\approx \frac{1}{2}(u^n + u^{n+1})\thinspace .
$$

However, with nonlinear terms we have many choices of formulating
an arithmetic mean:

$$
\begin{equation}
[f(u)]^{n+\frac{1}{2}} \approx f(\frac{1}{2}(u^n + u^{n+1}))
= [f(\overline{u}^t)]^{n+\frac{1}{2}},
\label{_auto5} \tag{16}
\end{equation}
$$

$$
\begin{equation}
[f(u)]^{n+\frac{1}{2}} \approx \frac{1}{2}(f(u^n) + f(u^{n+1}))
=[\overline{f(u)}^t]^{n+\frac{1}{2}},
\label{_auto6} \tag{17}
\end{equation}
$$

$$
\begin{equation}
[\dfc(u)\nabla u]^{n+\frac{1}{2}} \approx
\dfc(\frac{1}{2}(u^n + u^{n+1}))\nabla (\frac{1}{2}(u^n + u^{n+1}))
= [\dfc(\overline{u}^t)\nabla \overline{u}^t]^{n+\frac{1}{2}},
\label{_auto7} \tag{18}
\end{equation}
$$

$$
\begin{equation}
[\dfc(u)\nabla u]^{n+\frac{1}{2}} \approx
\frac{1}{2}(\dfc(u^n) + \dfc(u^{n+1}))\nabla (\frac{1}{2}(u^n + u^{n+1}))
= [\overline{\dfc(u)}^t\nabla\overline{u}^t]^{n+\frac{1}{2}},
\label{_auto8} \tag{19}
\end{equation}
$$

$$
\begin{equation}
[\dfc(u)\nabla u]^{n+\frac{1}{2}} \approx
\frac{1}{2}(\dfc(u^n)\nabla u^n + \dfc(u^{n+1})\nabla u^{n+1})
= [\overline{\dfc(u)\nabla u}^t]^{n+\frac{1}{2}}\thinspace .
\label{_auto9} \tag{20}
\end{equation}
$$

A big question is whether there are significant differences in accuracy
between taking the products of arithmetic means or taking the arithmetic
mean of products. [nonlin:exer:products:arith:mean](#nonlin:exer:products:arith:mean) investigates
this question, and the answer is that the approximation is
$\Oof{\Delta t^2}$ in both cases.

# 1D stationary nonlinear differential equations

The section [Linearization at the differential equation level](#nonlin:pdelevel) presented methods for linearizing
time-discrete PDEs directly prior to discretization in space.  We can
alternatively carry out the discretization in space of the
time-discrete nonlinear PDE problem and get a system of nonlinear
algebraic equations, which can be solved by Picard iteration or
Newton's method as presented in the section [nonlin:systems:alg](#nonlin:systems:alg).
This latter approach will now be described in detail.

We shall work with the 1D problem

$$
\begin{equation}
-(\dfc(u)u^{\prime})^{\prime} + au = f(u),\quad x\in (0,L),
\quad \dfc(u(0))u^{\prime}(0) = C,\ u(L)=D
\thinspace .
\label{nonlin:alglevel:1D:pde} \tag{21}
\end{equation}
$$

The problem ([21](#nonlin:alglevel:1D:pde)) arises from the stationary
limit of a diffusion equation,

$$
\begin{equation}
\frac{\partial u}{\partial t} = \frac{\partial}{\partial x}\left(
\alpha(u)\frac{\partial u}{\partial x}\right) - au + f(u),
\label{nonlin:alglevel:1D:pde:tver} \tag{22}
\end{equation}
$$

as $t\rightarrow\infty$ and $\partial u/\partial t\rightarrow 0$.
Alternatively, the problem ([21](#nonlin:alglevel:1D:pde)) arises
at each time level from implicit time discretization of
([22](#nonlin:alglevel:1D:pde:tver)). For example, a Backward Euler
scheme for ([22](#nonlin:alglevel:1D:pde:tver)) leads to

$$
\begin{equation}
\frac{u^{n}-u^{n-1}}{\Delta t} =
\frac{d}{dx}\left(
\alpha(u^n)\frac{du^n}{dx}\right) - au^n + f(u^n)\thinspace .
\label{nonlin:alglevel:1D:pde:tver:BE} \tag{23}
\end{equation}
$$

Introducing $u(x)$ for $u^n(x)$, $u^{(1)}$ for $u^{n-1}$, and defining $f(u)$
in ([21](#nonlin:alglevel:1D:pde)) to be $f(u)$ in
([23](#nonlin:alglevel:1D:pde:tver:BE)) plus $u^{n-1}/\Delta t$, gives
([21](#nonlin:alglevel:1D:pde)) with $a=1/\Delta t$.

## Finite difference discretization

The nonlinearity in the differential equation
([21](#nonlin:alglevel:1D:pde)) poses no more difficulty than a variable
coefficient, as in the term $(\dfc(x)u^{\prime})^{\prime}$.  We can
therefore use a standard finite difference approach when discretizing
the Laplace term with a variable coefficient:

$$
[-D_x\dfc D_x u +au = f]_i\thinspace .
$$

Writing this out for a uniform mesh with points $x_i=i\Delta x$,
$i=0,\ldots,N_x$, leads to

$$
\begin{equation}
-\frac{1}{\Delta x^2}
\left(\dfc_{i+\frac{1}{2}}(u_{i+1}-u_i) -
\dfc_{i-\frac{1}{2}}(u_{i}-u_{i-1})\right)
+ au_i = f(u_i)\thinspace .
\label{nonlin:alglevel:1D:fd:deq0} \tag{24}
\end{equation}
$$

This equation is valid at all the mesh points $i=0,1,\ldots,N_x-1$.
At $i=N_x$ we have the Dirichlet condition $u_i=0$.
The only difference from the case with $(\dfc(x)u^{\prime})^{\prime}$ and $f(x)$ is that
now $\dfc$ and $f$ are functions of $u$ and not only of $x$:
$(\dfc(u(x))u^{\prime})^{\prime}$ and $f(u(x))$.

The quantity $\dfc_{i+\frac{1}{2}}$, evaluated between two mesh points,
needs a comment. Since $\dfc$ depends on $u$ and $u$ is only known
at the mesh points, we need to express $\dfc_{i+\frac{1}{2}}$ in
terms of $u_i$ and $u_{i+1}$. For this purpose we use an arithmetic
mean, although a harmonic mean is also common in this context if
$\dfc$ features large jumps.
There are two choices of arithmetic means:

$$
\begin{equation}
\dfc_{i+\frac{1}{2}} \approx
\dfc(\frac{1}{2}(u_i + u_{i+1}) =
[\dfc(\overline{u}^x)]^{i+\frac{1}{2}},
\label{nonlin:alglevel:1D:fd:dfc:mean:u} \tag{25}
\end{equation}
$$

$$
\begin{equation}
\dfc_{i+\frac{1}{2}} \approx
\frac{1}{2}(\dfc(u_i) + \dfc(u_{i+1})) = [\overline{\dfc(u)}^x]^{i+\frac{1}{2}}
\label{nonlin:alglevel:1D:fd:dfc:mean:dfc} \tag{26}
\end{equation}
$$

Equation ([24](#nonlin:alglevel:1D:fd:deq0)) with
the latter approximation then looks like

$$
-\frac{1}{2\Delta x^2}
\left((\dfc(u_i)+\dfc(u_{i+1}))(u_{i+1}-u_i) -
(\dfc(u_{i-1})+\dfc(u_{i}))(u_{i}-u_{i-1})\right)\nonumber
$$

$$
\begin{equation}
\qquad\qquad + au_i = f(u_i),
\label{nonlin:alglevel:1D:fd:deq} \tag{27}
\end{equation}
$$

or written more compactly,

$$
[-D_x\overline{\dfc}^x D_x u +au = f]_i\thinspace .
$$

At mesh point $i=0$ we have the boundary condition $\dfc(u)u^{\prime}=C$,
which is discretized by

$$
[\dfc(u)D_{2x}u = C]_0,
$$

meaning

$$
\begin{equation}
\dfc(u_0)\frac{u_{1} - u_{-1}}{2\Delta x} = C\thinspace .
\label{nonlin:alglevel:1D:fd:Neumann:x0} \tag{28}
\end{equation}
$$

The fictitious value $u_{-1}$ can be eliminated with the aid
of ([27](#nonlin:alglevel:1D:fd:deq)) for $i=0$.
Formally, ([27](#nonlin:alglevel:1D:fd:deq)) should be solved with
respect to $u_{i-1}$ and that value (for $i=0$) should be inserted in
([28](#nonlin:alglevel:1D:fd:Neumann:x0)), but it is algebraically
much easier to do it the other way around. Alternatively, one can
use a ghost cell $[-\Delta x,0]$ and update the $u_{-1}$ value
in the ghost cell according to ([28](#nonlin:alglevel:1D:fd:Neumann:x0))
after every Picard or Newton iteration. Such an approach means that
we use a known $u_{-1}$ value in ([27](#nonlin:alglevel:1D:fd:deq))
from the previous iteration.

## Solution of algebraic equations

### The structure of the equation system

The nonlinear algebraic equations ([27](#nonlin:alglevel:1D:fd:deq)) are
of the form $A(u)u = b(u)$ with

$$
\begin{align*}
A_{i,i} &= \frac{1}{2\Delta x^2}(\dfc(u_{i-1}) + 2\dfc(u_{i})
\dfc(u_{i+1})) + a,\\
A_{i,i-1} &= -\frac{1}{2\Delta x^2}(\dfc(u_{i-1}) + \dfc(u_{i})),\\
A_{i,i+1} &= -\frac{1}{2\Delta x^2}(\dfc(u_{i}) + \dfc(u_{i+1})),\\
b_i &= f(u_i)\thinspace .
\end{align*}
$$

The matrix $A(u)$ is tridiagonal: $A_{i,j}=0$ for $j > i+1$ and $j < i-1$.

The above expressions are valid for internal mesh points $1\leq i\leq N_x-1$.
For $i=0$ we need to express $u_{i-1}=u_{-1}$ in terms of $u_1$ using
([28](#nonlin:alglevel:1D:fd:Neumann:x0)):

$$
\begin{equation}
u_{-1} = u_1 -\frac{2\Delta x}{\dfc(u_0)}C\thinspace .
\label{nonlin:alglevel:1D:fd:Neumann:x0:um1} \tag{29}
\end{equation}
$$

This value must be inserted in $A_{0,0}$. The expression for $A_{i,i+1}$
applies for $i=0$, and $A_{i,i-1}$ does not enter the system when $i=0$.

Regarding the last equation, its form depends on whether we include
the Dirichlet condition $u(L)=D$, meaning $u_{N_x}=D$, in the
nonlinear algebraic equation system or not. Suppose we choose
$(u_0,u_1,\ldots,u_{N_x-1})$ as unknowns, later referred to as
*systems without Dirichlet conditions*. The last equation
corresponds to $i=N_x-1$. mathcal{I}_t involves the boundary value $u_{N_x}$,
which is substituted by $D$. If the unknown vector includes the
boundary value, $(u_0,u_1,\ldots,u_{N_x})$, later referred to as
*system including Dirichlet conditions*, the equation for $i=N_x-1$
just involves the unknown $u_{N_x}$, and the final equation becomes
$u_{N_x}=D$, corresponding to $A_{i,i}=1$ and $b_i=D$ for $i=N_x$.

### Picard iteration

The obvious Picard iteration scheme is to use previously computed
values of $u_i$ in $A(u)$ and $b(u)$, as described more in detail in
the section [nonlin:systems:alg](#nonlin:systems:alg). With the notation $u^{-}$ for the
most recently computed value of $u$, we have the system $F(u)\approx
\hat F(u) = A(u^{-})u - b(u^{-})$, with $F=(F_0,F_1,\ldots,F_m)$,
$u=(u_0,u_1,\ldots,u_m)$.  The index $m$ is $N_x$ if the system
includes the Dirichlet condition as a separate equation and $N_x-1$
otherwise.  The matrix $A(u^{-})$ is tridiagonal, so the solution
procedure is to fill a tridiagonal matrix data structure and the
right-hand side vector with the right numbers and call a Gaussian
elimination routine for tridiagonal linear systems.

### Mesh with two cells

mathcal{I}_t helps on the understanding of the details to write out all the
mathematics in a specific
case with a small mesh, say just two cells ($N_x=2$). We use $u^{-}_i$
for the $i$-th component in $u^{-}$.

The starting point is the basic expressions for the
nonlinear equations at mesh point $i=0$ and $i=1$:

$$
\begin{equation}
A_{0,-1}u_{-1} + A_{0,0}u_0 + A_{0,1}u_1 = b_0,
\label{nonlin:alglevel:1D:fd:2x2:x0} \tag{30}
\end{equation}
$$

$$
\begin{equation}
A_{1,0}u_{0} + A_{1,1}u_1 + A_{1,2}u_2 = b_1\thinspace .
\label{nonlin:alglevel:1D:fd:2x2:x1} \tag{31}
\end{equation}
$$

Equation ([30](#nonlin:alglevel:1D:fd:2x2:x0)) written out reads

$$
\begin{align*}
\frac{1}{2\Delta x^2}(& -(\dfc(u_{-1}) + \dfc(u_{0}))u_{-1}\, +\\
& (\dfc(u_{-1}) + 2\dfc(u_{0}) + \dfc(u_{1}))u_0\, -\\
& (\dfc(u_{0}) + \dfc(u_{1})))u_1 + au_0
=f(u_0)\thinspace .
\end{align*}
$$

We must then replace $u_{-1}$ by
([29](#nonlin:alglevel:1D:fd:Neumann:x0:um1)).
With Picard iteration we get

$$
\begin{align*}
\frac{1}{2\Delta x^2}(& -(\dfc(u^-_{-1}) + 2\dfc(u^-_{0})
+ \dfc(u^-_{1}))u_1\, +\\
&(\dfc(u^-_{-1}) + 2\dfc(u^-_{0}) + \dfc(u^-_{1}))u_0
 + au_0\\
&=f(u^-_0) -
\frac{1}{\dfc(u^-_0)\Delta x}(\dfc(u^-_{-1}) + \dfc(u^-_{0}))C,
\end{align*}
$$

where

$$
u^-_{-1} = u_1^- -\frac{2\Delta x}{\dfc(u^-_0)}C\thinspace .
$$

Equation ([31](#nonlin:alglevel:1D:fd:2x2:x1)) contains the unknown $u_2$
for which we have a Dirichlet condition. In case we omit the
condition as a separate equation, ([31](#nonlin:alglevel:1D:fd:2x2:x1))
with Picard iteration becomes

$$
\begin{align*}
\frac{1}{2\Delta x^2}(&-(\dfc(u^-_{0}) + \dfc(u^-_{1}))u_{0}\, + \\
&(\dfc(u^-_{0}) + 2\dfc(u^-_{1}) + \dfc(u^-_{2}))u_1\, -\\
&(\dfc(u^-_{1}) + \dfc(u^-_{2})))u_2 + au_1
=f(u^-_1)\thinspace .
\end{align*}
$$

We must now move the $u_2$ term to the right-hand side and replace all
occurrences of $u_2$ by $D$:

$$
\begin{align*}
\frac{1}{2\Delta x^2}(&-(\dfc(u^-_{0}) + \dfc(u^-_{1}))u_{0}\, +\\
& (\dfc(u^-_{0}) + 2\dfc(u^-_{1}) + \dfc(D)))u_1 + au_1\\
&=f(u^-_1) + \frac{1}{2\Delta x^2}(\dfc(u^-_{1}) + \dfc(D))D\thinspace .
\end{align*}
$$

The two equations can be written as a $2\times 2$ system:

$$
\left(\begin{array}{cc}
B_{0,0}& B_{0,1}\\
B_{1,0} & B_{1,1}
\end{array}\right)
\left(\begin{array}{c}
u_0\\
u_1
\end{array}\right)
=
\left(\begin{array}{c}
d_0\\
d_1
\end{array}\right),
$$

where

$$
\begin{equation}
B_{0,0} =\frac{1}{2\Delta x^2}(\dfc(u^-_{-1}) + 2\dfc(u^-_{0}) + \dfc(u^-_{1}))
+ a,
\label{_auto10} \tag{32}
\end{equation}
$$

$$
\begin{equation}
B_{0,1} =
-\frac{1}{2\Delta x^2}(\dfc(u^-_{-1}) + 2\dfc(u^-_{0})
+ \dfc(u^-_{1})),
\label{_auto11} \tag{33}
\end{equation}
$$

$$
\begin{equation}
B_{1,0} =
-\frac{1}{2\Delta x^2}(\dfc(u^-_{0}) + \dfc(u^-_{1})),
\label{_auto12} \tag{34}
\end{equation}
$$

$$
\begin{equation}
B_{1,1} =
\frac{1}{2\Delta x^2}(\dfc(u^-_{0}) + 2\dfc(u^-_{1}) + \dfc(D)) + a,
\label{_auto13} \tag{35}
\end{equation}
$$

$$
\begin{equation}
d_0 =
f(u^-_0) -
\frac{1}{\dfc(u^-_0)\Delta x}(\dfc(u^-_{-1}) + \dfc(u^-_{0}))C,
\label{_auto14} \tag{36}
\end{equation}
$$

$$
\begin{equation}
d_1 = f(u^-_1) + \frac{1}{2\Delta x^2}(\dfc(u^-_{1}) + \dfc(D))D\thinspace .
\label{_auto15} \tag{37}
\end{equation}
$$

The system with the Dirichlet condition becomes

$$
\left(\begin{array}{ccc}
B_{0,0}& B_{0,1} & 0\\
B_{1,0} & B_{1,1} & B_{1,2}\\
0  & 0 & 1
\end{array}\right)
\left(\begin{array}{c}
u_0\\
u_1\\
u_2
\end{array}\right)
=
\left(\begin{array}{c}
d_0\\
d_1\\
D
\end{array}\right),
$$

with

$$
\begin{equation}
B_{1,1} =
\frac{1}{2\Delta x^2}(\dfc(u^-_{0}) + 2\dfc(u^-_{1}) + \dfc(u_2)) + a,
\label{_auto16} \tag{38}
\end{equation}
$$

$$
\begin{equation}
B_{1,2} = -
\frac{1}{2\Delta x^2}(\dfc(u^-_{1}) + \dfc(u_2))),
\label{_auto17} \tag{39}
\end{equation}
$$

$$
\begin{equation}
d_1 = f(u^-_1)\thinspace .
\label{_auto18} \tag{40}
\end{equation}
$$

Other entries are as in the $2\times 2$ system.

### Newton's method

The Jacobian must be derived in order to use Newton's method. Here it means
that we need to differentiate $F(u)=A(u)u - b(u)$ with respect to
the unknown parameters
$u_0,u_1,\ldots,u_m$ ($m=N_x$ or $m=N_x-1$, depending on whether the
Dirichlet condition is included in the nonlinear system $F(u)=0$ or not).
Nonlinear equation number $i$ has the structure

$$
F_i = A_{i,i-1}(u_{i-1},u_i)u_{i-1} +
A_{i,i}(u_{i-1},u_i,u_{i+1})u_i +
A_{i,i+1}(u_i, u_{i+1})u_{i+1} - b_i(u_i)\thinspace .
$$

Computing the Jacobian requires careful differentiation. For example,

$$
\begin{align*}
\frac{\partial}{\partial u_i}(A_{i,i}(u_{i-1},u_i,u_{i+1})u_i) &=
\frac{\partial A_{i,i}}{\partial u_i}u_i + A_{i,i}
\frac{\partial u_i}{\partial u_i}\\
&=
\frac{\partial}{\partial u_i}(
\frac{1}{2\Delta x^2}(\dfc(u_{i-1}) + 2\dfc(u_{i})
+\dfc(u_{i+1})) + a)u_i +\\
&\quad\frac{1}{2\Delta x^2}(\dfc(u_{i-1}) + 2\dfc(u_{i})
+\dfc(u_{i+1})) + a\\
&= \frac{1}{2\Delta x^2}(2\dfc^\prime (u_i)u_i
+\dfc(u_{i-1}) + 2\dfc(u_{i})
+\dfc(u_{i+1})) + a\thinspace .
\end{align*}
$$

The complete Jacobian becomes

$$
\begin{align*}
J_{i,i} &= \frac{\partial F_i}{\partial u_i}
= \frac{\partial A_{i,i-1}}{\partial u_i}u_{i-1}
+ \frac{\partial A_{i,i}}{\partial u_i}u_i
+ A_{i,i}
+ \frac{\partial A_{i,i+1}}{\partial u_i}u_{i+1}
- \frac{\partial b_i}{\partial u_{i}}\\
&=
\frac{1}{2\Delta x^2}(
-\dfc^{\prime}(u_i)u_{i-1}
+2\dfc^{\prime}(u_i)u_{i}
+\dfc(u_{i-1}) + 2\dfc(u_i) + \dfc(u_{i+1})) +\\
&\quad a
-\frac{1}{2\Delta x^2}\dfc^{\prime}(u_{i})u_{i+1}
- b^{\prime}(u_i),\\
J_{i,i-1} &= \frac{\partial F_i}{\partial u_{i-1}}
= \frac{\partial A_{i,i-1}}{\partial u_{i-1}}u_{i-1}
+ A_{i-1,i}
+ \frac{\partial A_{i,i}}{\partial u_{i-1}}u_i
- \frac{\partial b_i}{\partial u_{i-1}}\\
&=
\frac{1}{2\Delta x^2}(
-\dfc^{\prime}(u_{i-1})u_{i-1} - (\dfc(u_{i-1}) + \dfc(u_i))
+ \dfc^{\prime}(u_{i-1})u_i),\\
J_{i,i+1} &= \frac{\partial A_{i,i+1}}{\partial u_{i-1}}u_{i+1}
+ A_{i+1,i} +
\frac{\partial A_{i,i}}{\partial u_{i+1}}u_i
- \frac{\partial b_i}{\partial u_{i+1}}\\
&=\frac{1}{2\Delta x^2}(
-\dfc^{\prime}(u_{i+1})u_{i+1} - (\dfc(u_{i}) + \dfc(u_{i+1}))
+ \dfc^{\prime}(u_{i+1})u_i)
\thinspace .
\end{align*}
$$

The explicit expression for nonlinear equation number $i$,
$F_i(u_0,u_1,\ldots)$, arises from moving the $f(u_i)$ term in
([27](#nonlin:alglevel:1D:fd:deq)) to the left-hand side:

$$
F_i = -\frac{1}{2\Delta x^2}
\left((\dfc(u_i)+\dfc(u_{i+1}))(u_{i+1}-u_i) -
(\dfc(u_{i-1})+\dfc(u_{i}))(u_{i}-u_{i-1})\right)\nonumber
$$

$$
\begin{equation}
\qquad\qquad + au_i - f(u_i) = 0\thinspace .
\label{nonlin:alglevel:1D:fd:deq2} \tag{41}
\end{equation}
$$

At the boundary point $i=0$, $u_{-1}$ must be replaced using
the formula ([29](#nonlin:alglevel:1D:fd:Neumann:x0:um1)).
When the Dirichlet condition at $i=N_x$ is not a part of the
equation system, the last equation $F_m=0$ for $m=N_x-1$
involves the quantity $u_{N_x-1}$ which must be replaced by $D$.
If $u_{N_x}$ is treated as an unknown in the system, the
last equation $F_m=0$ has $m=N_x$ and reads

$$
F_{N_x}(u_0,\ldots,u_{N_x}) = u_{N_x} - D = 0\thinspace .
$$

Similar replacement of $u_{-1}$ and $u_{N_x}$ must be done in
the Jacobian for the first and last row. When $u_{N_x}$
is included as an unknown, the last row in the Jacobian
must help implement the condition $\delta u_{N_x}=0$, since
we assume that $u$ contains the right Dirichlet value
at the beginning of the iteration ($u_{N_x}=D$), and then
the Newton update should be zero for $i=0$, i.e., $\delta u_{N_x}=0$.
This also forces the right-hand side to be $b_i=0$, $i=N_x$.

We have seen, and can see from the present example, that the
linear system in Newton's method contains all the terms present
in the system that arises in the Picard iteration method.
The extra terms in Newton's method can be multiplied by a factor
such that it is easy to program one linear system and set this
factor to 0 or 1 to generate the Picard or Newton system.


# Kernel Design

It's easy to make new kernels in GPflow. To demonstrate, we'll have a look at the Brownian motion kernel, whose function is
\begin{equation}
k(x, x') = \sigma^2 \text{min}(x, x')
\end{equation}
where $\sigma^2$ is a variance parameter.

```python
import gpflow
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from gpflow.utilities import print_summary, positive

plt.style.use("ggplot")
%matplotlib inline
```

To make this new kernel class, we inherit from the base class `gpflow.kernels.Kernel` and implement the three functions below. **NOTE:** Depending on the kernel to be implemented, other classes can be more adequate. For example, if the kernel to be implemented is isotropic stationary, you can immediately subclass `gpflow.kernels.IsotropicStationary` (at which point you
only have to override `K_r` or `K_r2`; see the `IsotropicStationary` class docstring). Stationary but anisotropic kernels should subclass `gpflow.kernels.AnisotropicStationary` and override `K_d`.

#### `__init__`
In this simple example, the constructor takes no argument (though it could, if that was convenient, for example to pass in an initial value for `variance`). It *must* call the constructor of the superclass with appropriate arguments. Brownian motion is only defined in one dimension, and we'll assume that the `active_dims` are `[0]`, for simplicity.

We've added a parameter to the kernel using the `Parameter` class. Using this class lets the parameter be used in computing the kernel function, and it will automatically be recognised for optimization (or MCMC). Here, the variance parameter is initialized at 1, and constrained to be positive.

#### `K`
This is where you implement the kernel function itself. This takes two arguments, `X` and `X2`. By convention, we make the second argument optional (it defaults to `None`).

Inside `K`, all the computation must be done with TensorFlow - here we've used `tf.minimum`. When GPflow executes the `K` function, `X` and `X2` will be TensorFlow tensors, and parameters such as `self.variance` behave like TensorFlow tensors as well.

#### `K_diag`
This convenience function allows GPflow to save memory at predict time. It's simply the diagonal of the `K` function, in the case where `X2` is `None`. It must return a one-dimensional vector, so we use TensorFlow's reshape command.

```python
class Brownian(gpflow.kernels.Kernel):
    def __init__(self):
        super().__init__(active_dims=[0])
        self.variance = gpflow.Parameter(1.0, transform=positive())

    def K(self, X, X2=None):
        if X2 is None:
            X2 = X
        return self.variance * tf.minimum(X, tf.transpose(X2))  # this returns a 2D tensor

    def K_diag(self, X):
        return self.variance * tf.reshape(X, (-1,))  # this returns a 1D tensor

k_brownian = Brownian()
print_summary(k_brownian, fmt="notebook")
```

name             	class    	transform  	prior  	trainable  	shape  	dtype  	  value

Brownian.variance	Parameter	Softplus   	       	True       	()     	float64	      1

We can now evaluate our new kernel function and draw samples from a Gaussian process with this covariance:

```python
np.random.seed(23)  # for reproducibility

def plotkernelsample(k, ax, xmin=0, xmax=3):
    xx = np.linspace(xmin, xmax, 300)[:, None]
    K = k(xx)
    ax.plot(xx, np.random.multivariate_normal(np.zeros(300), K, 5).T)
    ax.set_title("Samples " + k.__class__.__name__)

def plotkernelfunction(k, ax, xmin=0, xmax=3, other=0):
    xx = np.linspace(xmin, xmax, 100)[:, None]
    ax.plot(xx, k(xx, np.zeros((1, 1)) + other))
    ax.set_title(k.__class__.__name__ + " k(x, %.1f)" % other)

f, axes = plt.subplots(1, 2, figsize=(12, 4), sharex=True)
plotkernelfunction(k_brownian, axes[0], other=2.0)
plotkernelsample(k_brownian, axes[1])
```

## Using the kernel in a model

Because we've inherited from the `Kernel` base class, this new kernel has all the properties needed to be used in GPflow. It also has some convenience features such as allowing the user to call

`k(X, X2)`

which computes the kernel matrix.

To show that this kernel works, let's use it inside GP regression. We'll see that Brownian motion has quite interesting properties. To add a little flexibility, we'll add a `Constant` kernel to our `Brownian` kernel, and the `GPR` class will handle the noise.

```python
np.random.seed(42)
X = np.random.rand(5, 1)
Y = np.sin(X * 6) + np.random.randn(*X.shape) * 0.001

k1 = Brownian()
k2 = gpflow.kernels.Constant()
k = k1 + k2

m = gpflow.models.GPR((X, Y), kernel=k)
# m.likelihood.variance.assign(1e-6)

opt = gpflow.optimizers.Scipy()
opt.minimize(m.training_loss, variables=m.trainable_variables)
print_summary(m, fmt="notebook")

xx = np.linspace(0, 1.1, 100).reshape(100, 1)
mean, var = m.predict_y(xx)
plt.plot(X, Y, "kx", mew=2)
(line,) = plt.plot(xx, mean, lw=2)
_ = plt.fill_between(
    xx[:, 0],
    mean[:, 0] - 2 * np.sqrt(var[:, 0]),
    mean[:, 0] + 2 * np.sqrt(var[:, 0]),
    color=line.get_color(),
    alpha=0.2,
)
```

## See also

For more details on how to manipulate existing kernels (or the one you just created!), we refer to the [Manipulating kernels](../advanced/kernels.ipynb) notebook.


# Click "Edit App" to see the code
# Least squares fitting
In this tutorial we'll learn how to fit a function using the **lmfit** library, which provides an interface to non-linear fitting fitting routines.
Let's consider a set of $N$ observations $(x_{i},y_{i})$, where $x_i$ is the independent variable, $y_i$
is the dependent variable and $i=1,\dots,N$, and let $f(x)$ be fitting function that contains a number of parameters, $p_j$, which is much smaller than the number of observations.

The optimisation of the $p_j$ parameters is done via the minimisation of the sum of the square of the residuals, $r_i$, where the residuals are the differences between observed values, $y_i$, and the values provided by the fitting model, $f_i$

\begin{equation}
S = \sum_i r_i = [f(x_i) - y_i]^2 \tag{1}
\end{equation}

As an illustrative example of how the fitting works, we'll use a set of rate constants measured at different temperatures, which obey the Arrhenius equation

\begin{equation}
k_r = A\exp\bigg[-\frac{E_a}{RT}\bigg] \tag{2}
\end{equation}

The Arrhenius equation is often written in linear form ($y=mx+q$) by taking the logarithm of both sides of the equation

\begin{equation}
\ln\big[k_r\big] = \ln A -\frac{E_a}{R}\frac{1}{T} \tag{3}
\end{equation}

where $y=\ln[k_r]$ and $x=1/T$.

In order to show the limitations of the least square method, we'll fit the same data set using both an exponential and a linear function.

# The Jupyter notebook
As always we start by loading up the packages we will need.

```python
# python packages
import pandas as pd # Dataframes and reading CSV files
import numpy as np # Numerical libraries
import matplotlib.pyplot as plt # Plotting library

from scipy.optimize import curve_fit
from sklearn.metrics import r2_score

from lmfit import Model # Least squares fitting library
```

Then we read the data set from a file and cast it into a DataFrame, and we rename the columns.

```python
data = pd.read_csv("../miscData/arrhenius.csv")
data.columns = ("T","Kr")
print(data)
```

As you may have noticed, the temperatures are not increasing (or decreasing) monotonically.
Although this is not a problem for the fitting itself, we sort the DataFrame to make a nicer plot later.
We also reset the row indices, so that we can access the sorted temperature using the reordered indices of the DataFrame, if needed.

```python
data = data.sort_values("T")
data.reset_index(drop=True, inplace=True)
print(data)
```

Although there are pre-built models, we assume that we are fitting a generic user-defined function.
Hence, we need to define two functions for the two equations that we want to use for the fitting.

```python
R = 8.314 # ideal gas constant in J/mol/K
def arrExp(x,A,Ea):
    return A * np.exp(-Ea/R/x)

def arrLin(x,lnA,Ea):
    return lnA - Ea/R * (1/x)
```

Neither the _scipy.optimize_ nor the _lmfit_ libraries compute the $R^2$, but we can easily do that using its definition.

\begin{equation}
R^2 = 1 - \frac{\sum_i (y_i-f_i)}{\sum_i(y_i-\langle y \rangle)^2}
\end{equation}

where the numerator is the sum of the residuals and the denominator is the variance, _i.e._ the standard deviation squared. The sum of the residuals is already computed by _lmfit_ and we can obtain the variance from NumPy.
The $R^2$ confirms that the linear fit it better.

# Fitting using curve_fit from SCIPY

Fitting a function to a dataset is fairly straightforward using the _curve_fit_ function.
This function provides two outputs, the parameters that give the best fit (_popt_) and the covariance matrix (_pcov_), which can be used to compute the error and $R^2$ score.

```python
popt_exp, pcov_exp = curve_fit(arrExp,data["T"],data["Kr"])
popt_lin, pcov_lin = curve_fit(arrLin,data["T"],np.log(data["Kr"]))
```

We now compute the error on the best fit parameters $\dots$

```python
pError_exp = np.sqrt(np.diag(pcov_exp))
pError_lin = np.sqrt(np.diag(pcov_lin))
```

$\dots$ and the $R^2$ score

```python
y2_exp = arrExp(data["T"], *popt_exp)
Rsquared_exp = r2_score(data["Kr"], y2_exp)

y2_lin = arrLin(data["T"], *popt_lin)
Rsquared_lin = r2_score(np.log(data["Kr"]), y2_lin)
```

We now print the result of the exponential and linear fits.
* Note how much smaller the error on the activation energy is when using a linear fit.

```python
print("----- Exponential fit -----")
print("A  parameter - Slope     = %8.3e +/- %.3e" % (popt_exp[0],pError_exp[0]))
print("Ea parameter - Intercept = %8.3f +/- %.3f" % (popt_exp[1],pError_exp[1]))
print("R squared                = %.4f" % Rsquared_exp)

print("----- Linear fit -----")
print("lnA parameter - Slope     = %8.3e +/- %.3e" % (popt_lin[0],pError_lin[0]))
print("Ea  parameter - Intercept = %8.3f +/- %.3f" % (popt_lin[1],pError_lin[1]))
print("R squared               = %.4f" % Rsquared_lin)
```

# Fitting using LMFIT

We then create two _Models_ objects using the *lmfit* library for the two independent fits.
* Note how we called the **Model** function directly, even if it is part of *lmfit* library. That is because the way we imported the library in the header. This implies that we cannot call any variable "Model".

```python
modExp = Model(arrExp)
modLin = Model(arrLin)
```

Like all fitting procedures, *lmfit* requires a starting set of parameters to operate, that can be set by the user. The closer the parameters are to the correct ones the better the library will work. Typically the frequency factor is of the order of $10^{12}-10^{15}\ Hz$ and the activation energy for a slow-ish reaction is in the $1-100\ kJ/mol$ range.

```python
paramsExp = modExp.make_params(A=1e13 , Ea=5)
paramsLin = modLin.make_params(lnA=np.log(1e13) , Ea=5)
```

We can now fit the data. The **fit** function takes three main arguments, $y$, the initial parameters and the $x$ of the data to fit.
* Note how for the linear fit we pass to the _lmfit_ function the log of the rate but not the inverse of the temperature. This is because of the way we defined the **arrLin** function.

```python
fitExp = modExp.fit(data["Kr"], paramsExp, x=data["T"])
fitLin = modLin.fit(np.log(data["Kr"]), paramsLin, x=data["T"])
```

We can now compute the $R^2$ score.

```python
expR2 = 1 - fitExp.residual.var() / np.var(data["Kr"])
linR2 = 1 - fitLin.residual.var() / np.var(data["Kr"])
```

We can then print the result of the exponential and linear fits.
Note that the results are essentially identical to those obtained from _curve_fit_ because the same minimisation algorithm has been used.

```python
print("----- Exponential fit -----")
print("A  parameter - Slope     = %8.3e +/- %.3e" % (fitExp.params["A"].value,
                                                     fitExp.params["A"].stderr))
print("Ea parameter - Intercept = %8.3f +/- %.3f" % (fitExp.params["Ea"].value,
                                                     fitExp.params["Ea"].stderr))
print("R squared                = %.4f" % expR2)

print("----- Linear fit -----")
print("lnA parameter - Slope     = %8.3e +/- %.3e" % (fitLin.params["lnA"].value,
                                                      fitLin.params["lnA"].stderr))
print("Ea  parameter - Intercept = %8.3f +/- %.3f" % (fitLin.params["Ea"].value,
                                                      fitLin.params["Ea"].stderr))
print("R squared               = %.4f" % linR2)
```

One advantage of _lmfit_ is that it also provides more extensive information of the fitting.
For example for the exponential fit we have

```python
fitExp.result
```

$\dots$ and for the linear fit.

```python
fitLin.result
```

Let's now see how the two fits compare with the input data graphically by plotting the data and fit functions.
_lmfit_ has already computed the values of best fit, so there's nothing for us to do.
Note that for the linear fit we took the exponential of the data.

```python
fig , ax = plt.subplots(1,2,figsize=(18,6))

ax[0].scatter(data["T"],data["Kr"],label="Data")
ax[0].plot(data["T"],fitExp.best_fit,label="ExpFit",color='red')
ax[0].plot(data["T"],np.exp(fitLin.best_fit),label="LinFit",color='green')
ax[0].set(xlabel="Temperature (K)")
ax[0].set(ylabel="Kr")
ax[0].legend()

ax[1].scatter(1/data["T"],np.log(data["Kr"]),label="Data")
ax[1].plot(1/data["T"],np.log(fitExp.best_fit),label="ExpFit",color='red')
ax[1].plot(1/data["T"],fitLin.best_fit,label="LinFit",color='green')
ax[1].set(xlabel="1/Temperature (1/K)")
ax[1].set(ylabel="ln[Kr]")
ax[1].legend()

plt.show()
```

Although both fits look good representations of the data when plotted in the exponential form, the linear fit appears clearly superior when the in the linear form.
This is due to fact that the least squares method minimises the residuals, so in the exponential form the tail of the function is _less_ important than the part with large numbers.