Текст распространяется на условиях лицензии Creative Commons Attribution license, CC-BY. Код под лицензией MIT.
(c)2014 Lorena A. Barba, Olivier Mesnard. Благодарности: NSF for support via CAREER award #1149784.
Перевод (c)2018 Innokentiy Kursakov
##### Версия 0.1 -- Январь 2018

# Вихрь

Давайте освежим пройденное. На первых трех занятиях курса _AeroPython_ мы рассмотрели:

1. [пару источник-сток](01_Lesson01_sourceSink.ipynb) — вы научились создавать картины линий тока (`streamplot()`) и отмечать на них особые точки при помощи функции `scatter()`. Если вы выполнили домашнее задание, то убедились, что эквипотенциальные линии перпендикулярны линиям тока.

2. [пару источник-сток в набегающем потоке](02_Lesson02_sourceSinkFreestream.ipynb) — получили _овал Рэнкина_. Мы впервые уидели, что с помощью потенциальных течений можно конструировать обтекание геометрических объектов. Вы также научились определять пользовательские функции. Они заставляют Python работать на вас, и делают его еще больше похожим на обычный английский.

3. [диполь](03_Lesson03_doublet.ipynb) — картина линий тока соответсвует двумерному обтеканию кругового цилиндра. Вы столкнулись с _парадоксом д'Аламбера_ и, надеюсь, это заставило вас задуматься. Если нет, вернитесь к этому уроку и _задумайтесь_!

Сделали вы [задание](03_Lesson03_Assignment.ipynb)? Оно показывает, как с помощью *распреленных источников* смоделировать потенциальное обтекание профиля. Это уже похоже на аэродинамику!

Какой же главный результат мы желаем получить от прикладной аэродинамики? Конечно же, полететь, опираясь не на силу мускулов, а на силу разума! А для этого нужна *подъемная сила*, превосходящая вес тела, желающего взлететь.

В этом разделе нашего курса речь пойдет о подъемной силе. Вначале мы смоделируем поток в окрестности потенциального вихря. Оказывается, циркуляция вихря и подъемная сила тесно связаны.

## Что такое вихрь?

Этот вопрос сложнее, чем кажется! Простой ответ заключается в том, что вихрь это движение по круговым линиям тока. Представьте себе линии тока в виде концентрических окружностей с центром в заданной точке — при этом _совсем не обязательно_, чтобы жидкие частицы вращались вокруг своих осей.

При безвихревом циркуляционном движении азимутальная компонента скорости постоянна вдоль круговых линии тока и обратно пропорциональна радиусу, а радиальная компонента равна нулю. В полярных координатах:

\begin{equation}
u_\theta\left(r,\theta\right) = \frac{\text{constant}}{r}  \quad \text{,} \quad  u_r\left(r,\theta\right) = 0
\end{equation}

Завихренность равна нулю во всей области за исключением точки расположения вихря, в которой производная $u_\theta$ обращается в бесконечность.

Понятие циркуляции было введено на первом занятии ([Источник и сток](01_Lesson01_sourceSink.ipynb)). Воспользуемся этим определением. Вдоль любой круговой линии тока, окружающей вихрь, учитывая правило знаков, по которому вихрь, закрученный против часовой стрелки, имеет отрицательный знак, справедливо:

\begin{equation}\Gamma = -\oint \mathbf{v}\cdot d\vec{l} = -u_\theta 2 \pi r\end{equation}

Таким образом, константа в выражении для $u_\theta$ в $(1)$ равна $\Gamma/2\pi$, и можно записать:

\begin{equation}u_\theta\left(r,\theta\right) = \frac{\Gamma}{2\pi r}\end{equation}

Функция тока можно получить, интегрируя компоненты скорости:

\begin{equation}\psi\left(r,\theta\right) = \frac{\Gamma}{2\pi}\ln r\end{equation}

В декартовых координатах функция тока записывается как

\begin{equation}\psi\left(x,y\right) = \frac{\Gamma}{4\pi}\ln\left(x^2+y^2\right)\end{equation}

а компоненты скорости:

\begin{equation}u\left(x,y\right) = \frac{\Gamma}{2\pi}\frac{y}{x^2+y^2} \qquad v\left(x,y\right) = -\frac{\Gamma}{2\pi}\frac{x}{x^2+y^2}\end{equation}

Такое циркуляционное течение является безвихревым везде, кроме центра вихря. В нем завихренность бесконечная. Интенсивность точечного вихря равна циркуляции $\Gamma$.

## Рассчитаем вихрь

Подготовительные шаги остаются прежними: импортируем любимые библиотеки и создаем сетку, в узлах которой будет рассчитываться поле скорости.

```python
import numpy
import math
from matplotlib import pyplot
# помещаем рисунки в блокнот
%matplotlib inline
```

```python
N = 50                          # Число узлов сетки в каждом направлении
x_start, x_end = -2.0, 2.0            # границы по x
y_start, y_end = -1.0, 1.0            # границы по y
x = numpy.linspace(x_start, x_end, N)    # создаем одномерный массив x
y = numpy.linspace(y_start, y_end, N)    # создаем одномерный массив y
X, Y = numpy.meshgrid(x, y)              # создает сетку
```

Зададим вихрь с интенсивностью $\Gamma=5$ и поместим его в центр расчетной области:

```python
gamma = 5.0                      # интенсивность вихря
x_vortex, y_vortex = 0.0, 0.0    # положение вихря
```

Определим две функции,

* `get_velocity_vortex()` и
* `get_stream_function_vortex()`,

для расчета компонент скорости и функции тока на декартовой сетке по заданным интенсивности и положению вихря. В последствии мы воспользуемся этими функциями и все посчитаем. Но сначала, давайте их напишем.

```python
def get_velocity_vortex(strength, xv, yv, X, Y):
    """
    Returns the velocity field generated by a vortex.

    Parameters
    ----------
    strength: float
        Strength of the vortex.
    xv: float
        x-coordinate of the vortex.
    yv: float
        y-coordinate of the vortex.
    X: 2D Numpy array of floats
        x-coordinate of the mesh points.
    Y: 2D Numpy array of floats
        y-coordinate of the mesh points.

    Returns
    -------
    u: 2D Numpy array of floats
        x-component of the velocity vector field.
    v: 2D Numpy array of floats
        y-component of the velocity vector field.
    """
    u = + strength/(2*math.pi)*(Y-yv)/((X-xv)**2+(Y-yv)**2)
    v = - strength/(2*math.pi)*(X-xv)/((X-xv)**2+(Y-yv)**2)

    return u, v
```

```python
def get_stream_function_vortex(strength, xv, yv, X, Y):
    """
    Returns the stream-function generated by a vortex.

    Parameters
    ----------
    strength: float
        Strength of the vortex.
    xv: float
        x-coordinate of the vortex.
    yv: float
        y-coordinate of the vortex.
    X: 2D Numpy array of floats
        x-coordinate of the mesh points.
    Y: 2D Numpy array of floats
        y-coordinate of the mesh points.

    Returns
    -------
    psi: 2D Numpy array of floats
        The stream-function.
    """
    psi = strength/(4*math.pi)*numpy.log((X-xv)**2+(Y-yv)**2)

    return psi
```

А теперь вызовем их, используя интенсивность и положение вихря, а также координаты узлов расчетной сетки в качестве параметров.

```python
# рассчитываем поле скорости в узлах расчетной сетки
u_vortex, v_vortex = get_velocity_vortex(gamma, x_vortex, y_vortex, X, Y)

# рассчитываем функцию тока в узлах расчетной сетки
psi_vortex = get_stream_function_vortex(gamma, x_vortex, y_vortex, X, Y)
```

Теперь мы можем визуализировать линии тока для потенциального вихря и убедиться, что они выглядят как концентрические окружности с центром в месте расположения вихря, как, впрочем, и ожидалось.

```python
# рисуем линии тока
size = 10
pyplot.figure(figsize=(size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.streamplot(X, Y, u_vortex, v_vortex, density=2, linewidth=1,\
                  arrowsize=1, arrowstyle='->')
pyplot.scatter(x_vortex, y_vortex, color='#CD2305', s=80, marker='o');
```

## Вихрь и сток

Забавы ради, применим нашу суперспособность — сурперпозицию. Добавим к вихрю сток. Для этого воспользуемся функциями, возвращающими компоненты скорости и функцию тока для стока, и добавим полученные значения к соответствующим величинам для вихря. Получим _вихресток_.

```python
strength_sink = -1.0           # интенсивность стока
x_sink, y_sink = 0.0, 0.0      # положение стока
```

```python
def get_velocity_sink(strength, xs, ys, X, Y):
    """
    Returns the velocity field generated by a sink.

    Parameters
    ----------
    strength: float
        Strength of the sink.
    xs: float
        x-coordinate of the sink.
    ys: float
        y-coordinate of the sink.
    X: 2D Numpy array of floats
        x-coordinate of the mesh points.
    Y: 2D Numpy array of floats
        y-coordinate of the mesh points.

    Returns
    -------
    u: 2D Numpy array of floats
        x-component of the velocity vector field.
    v: 2D Numpy array of floats
        y-component of the velocity vector field.
    """
    u = strength/(2*math.pi)*(X-xs)/((X-xs)**2+(Y-ys)**2)
    v = strength/(2*math.pi)*(Y-ys)/((X-xs)**2+(Y-ys)**2)

    return u, v
```

```python
def get_stream_function_sink(strength, xs, ys, X, Y):
    """
    Returns the stream-function generated by a sink.

    Parameters
    ----------
    strength: float
        Strength of the sink.
    xs: float
        x-coordinate of the sink.
    ys: float
        y-coordinate of the sink.
    X: 2D Numpy array of floats
        x-coordinate of the mesh points.
    Y: 2D Numpy array of floats
        y-coordinate of the mesh points.

    Returns
    -------
    psi: 2D Numpy array of floats
        The stream-function.
    """
    psi = strength/(2*math.pi)*numpy.arctan2((Y-ys), (X-xs))

    return psi
```

```python
# рассчитываем поле скорости в узлах расчетной сетки
u_sink, v_sink = get_velocity_sink(strength_sink, x_sink, y_sink, X, Y)

# рассчитываем функцию тока в узлах расчетной сетки
psi_sink = get_stream_function_sink(strength_sink, x_sink, y_sink, X, Y)
```

Теперь, давайте визуализируем линии тока для вихрестока, и восхитимся нашим художественным творением:

```python
# суперпозиция стока и вихря
u = u_vortex + u_sink
v = v_vortex + v_sink
psi = psi_vortex + psi_sink

# рисуем линии тока
size = 10
pyplot.figure(figsize=(size, (y_end-y_start)/(x_end-x_start)*size))
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.streamplot(X, Y, u, v, density=2, linewidth=1, arrowsize=1,\
                  arrowstyle='->')
pyplot.scatter(x_vortex, y_vortex, color='#CD2305', s=80, marker='o');
```

Получилась отличная иллюстрация в стиле [оп-арт](https://ru.wikipedia.org/wiki/%D0%9E%D0%BF%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B5_%D0%B8%D1%81%D0%BA%D1%83%D1%81%D1%81%D1%82%D0%B2%D0%BE). Кроме того, это неплохая модель слива воды в раковине. Но есть ли от этого какая-нибудь польза? Есть!

Сперва вам придется отбросить ходунки и — самостоятельно — смоделировать течение от [бесконечной цепчки вихрей](05_Lesson05_InfiniteRowOfVortices.ipynb). Ваше задание в следующем блокноте.

Затем мы узнаем, как связаны вихрь и подъемная сила. Вот увидите, это очень важное понятие в аэродинамике.

## Что такое "безвихревой" вихрь?

Остался непроясненным еще один важный вопрос.

Мы рассматриваем вихрь как потенциальное, а значит безвихревое течение. Это как? Конечно, если есть вихрь, есть вращение!

Глаза нас не обманывают. Такой ход мыслей довольно естественен. Но в случае с  потенциальным вихревым течением,  поток хоть и движется по круговым траекториям, элементарные объемы жидкости не вращаются вокруг собственных осей.

Классический эксперимент, демонстрируемый в этом видеоролике, поможет в понимании. Посмотрите 25 секунд после отметки 4:25 и увидите, что "измеритель завихренности" перемещается вокруг изолированного вихря, при этом не вращаясь.

```python
from IPython.display import YouTubeVideo
from datetime import timedelta

start=int(timedelta(hours=0, minutes=4, seconds=25).total_seconds())

YouTubeVideo("loCLkcYEWD4", start=start)
```

Помните: вихрь измеряет локальную угловую скорость каждого жидкого элемента. Если жидкие элементы движутся по кругу, но не вращаются сами по себе, то нет завихренности!

Иллюстрация из [Википедии](http://en.wikipedia.org/wiki/Vortex#Irrotational_vortices) поможет лучше понять, что происходит в изолированном вихре. Оранжевые маркеры движутся по окружностям, не меняя ориентации в пространстве.

## Далее...

В следующем блокноте курса *AeroPython* содержится задание для самостоятельной работы ["Бесконечная цепочка вихрей"](05_Lesson05_InfiniteRowOfVortices.ipynb).

---
Пожалуйста, не обращайте внимания на ячейку ниже. При её исполнении загружаются стили для отображения блокнота.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=PT+Serif' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans', sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Alegreya Sans', sans-serif;
    font-style:regular;
    font-weight: 200;
    font-size: 50pt;
    line-height: 100%;
    color:#CD2305;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'PT Serif', serif;
    font-size: 22pt;
    line-height: 100%;
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'PT Serif', serif;
    margin-top:12px;
	font-size: 16pt;
    margin-bottom: 3px;
    font-style: regular;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'PT Serif', serif;
    font-size: 2pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    color: #CD2305;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Consolas";
			font-size: 110%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


# **Least squares regression**

    Notebook version: 1.4 (Sep 26, 2019)

    Author: Jerónimo Arenas García (jarenas@tsc.uc3m.es)

    Changes: v.1.0 - First version
             v.1.1 - UTAD version
             v.1.2 - Minor corrections
             v.1.3 - Python 3 compatibility
             v.1.4 - Revised notation

    Pending changes: *

```python
# Import some libraries that will be necessary for working with data and displaying plots

# To visualize plots in the notebook
%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import scipy.io       # To read matlab files
import pylab

# For the student tests (only for python 2)
import sys
if sys.version_info.major==2:
    from test_helper import Test
```

This notebook covers the problem of fitting parametric regression models with a minimum least-squares criterion. The material presented here is based on the first lectures of this Machine Learning course. In particular, you can refer to the following presentation:  Probabilistic Regression.

## 1. A parametric approach to the regression problem

We have already presented the goal of regression. Given that we have access to a set of training points, $\{{\bf x}_k, s_k\}_{k=0}^{K-1}$, the goal is to learn a function $f({\bf x})$ that we can use to make good predictions for an arbitrary input vector.

The following plot illustrates a regression example for unidimensional input data. We have also generated three different regression curves corresponding to polynomia of degrees 1, 2, and 3 with random coefficients.

```python
K = 35
n_grid = 200
frec = 3
std_n = 0.3

# Location of the training points
X_tr = (3 * np.random.random((K, 1)) - 0.5)

# Labels are obtained from a sinusoidal function, and contaminated by noise
S_tr = np.cos(frec*X_tr) + std_n * np.random.randn(K, 1)

# Equally spaced points in the X-axis
X_grid = np.linspace(np.min(X_tr),np.max(X_tr),n_grid)

# Gererate random prediction curves
f1 = np.random.random() + np.random.random()*X_grid
f2 = np.random.random() + np.random.random()*X_grid + \
     np.random.random()*(X_grid**2)
f3 = np.random.random() + np.random.random()*X_grid + \
     np.random.random()*(X_grid**2) + np.random.random()*(X_grid**3)

plt.plot(X_tr,S_tr,'b.')
plt.plot(X_grid,f1.T,'g-',label='Arbitrary Linear function')
plt.plot(X_grid,f2.T,'r-',label='Arbitrary Quadratic function')
plt.plot(X_grid,f3.T,'m-',label='Arbitrary Cubic function')

plt.legend(loc='best')
plt.show()
```

### 1.1. Parametric model

Parametric regression models assume a parametric expression for the regression curve, adjusting the free parameters according to some criterion that measures the quality of the proposed model.

   - For a unidimensional case like the one in the previous figure, a convenient approach is to recur to polynomial expressions:

$${\hat s}(x) = f(x) = w_0 + w_1 x + w_2 x^2 + \dots + w_{m-1} x^{m-1}$$

   - For multidimensional regression, polynomial expressions can include cross-products of the variables. For instance, for a case with two input variables, the degree 2 polynomial would be given by

$${\hat s}({\bf x}) = f({\bf x}) = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_2^2 + w_5 x_1 x_2$$

   - A linear model for multidimensional regression can be expressed as

$${\hat s}({\bf x}) = f({\bf x}) = w_0 + {\bf w}^\top {\bf x}$$

When we postulate such models, the regression model is reduced to finding the most appropriate values of the parameters ${\bf w} = [w_i]$.

All the previous models have in common the fact that they are linear in the parameters, even though they can implement highly non-linear functions. All the derivations in this notebook are equally valid for other non-linear transformations of the input variables, as long as we keep linear-in-the-parameters models.

```python
## Next, we represent some random polynomial functions for degrees between 0 and 14

max_degree = 15
K = 200

#Values of X to evaluate the function
X_grid = np.linspace(-1.5, 1.5, K)

for idx in range(max_degree):
    x1 = plt.subplot(3,5, idx+1)
    x1.get_xaxis().set_ticks([])
    x1.get_yaxis().set_ticks([])

    for kk in range(5):
        #Random generation of coefficients for the model
        we = np.random.randn(idx+1, 1)

        #Evaluate the polynomial with previous coefficients at X_grid values
        fout = np.polyval(we, X_grid)
        x1.plot(X_grid,fout,'g-')
        x1.set_ylim([-5,5])
```

   - Should we choose a polynomial?

   - What degree should we use for the polynomial?

   - For a given degree, how do we choose the weights?

For now, we will find the single "best" polynomial. In a future session, we will see how we can design methods that take into account different polynomia simultaneously.

Next, we will explain how to choose optimal weights according to Least-Squares criterion.

## 2. Least squares regression

### 2.1. Problem definition

   - The goal is to learn a (possibly non-linear) regression model from a set of $K$ labeled points, $\{{\bf x}_k,s_k\}_{k=0}^{K-1}$.

   - We assume a parametric function of the form:

   $${\hat s}({\bf x}) = f({\bf x}) = w_0 z_0({\bf x}) + w_1 z_1({\bf x}) + \dots w_{m-1} z_{m-1}({\bf x})$$

   where $z_i({\bf x})$ are particular transformations of the input vector variables.

Some examples are:

   - If ${\bf z} = {\bf x}$, the model is just a linear combination of the input variables

   - If ${\bf z} = \left[\begin{array}{c}1\\{\bf x}\end{array}\right]$, we have again a linear combination with the inclusion of a constant term.

   - For unidimensional input $x$, ${\bf z} = [1, x, x^2, \dots,x^{m-1}]^\top$ would implement a polynomia of degree $m-1$.

   - Note that the variables of ${\bf z}$ could also be computed combining different variables of ${\bf x}$. E.g., if ${\bf x} = [x_1,x_2]^\top$, a degree-two polynomia would be implemented with
      $${\bf z} = \left[\begin{array}{c}1\\x_1\\x_2\\x_1^2\\x_2^2\\x_1 x_2\end{array}\right]$$

   - The above expression does not assume a polynomial model. For instance, we could consider ${\bf z} = [\log(x_1),\log(x_2)]$

Least squares (LS) regression finds the coefficients of the model with the aim of minimizing the square of the residuals. If we define ${\bf w} = [w_0,w_1,\dots,w_{m-1}]^\top$, the LS solution would be defined as

\begin{equation}
{\bf w}_{LS} = \arg \min_{\bf w} \sum_{k=0}^{K-1} [e_k]^2 = \arg \min_{\bf w} \sum_{k=0}^{K-1} \left[s_k - {\hat s}_k \right]^2
\end{equation}

### 2.2. Vector Notation

In order to solve the LS problem it is convenient to define the following vectors and matrices:

   - We can group together all available target values to form the following vector

   $${\bf s} = \left[s_0, s_1, \dots, s_{K-1} \right]^\top$$

   - The estimation of the model for a single input vector ${\bf z}_k$ (which would be computed from ${\bf x}_k$), can be expressed as the following inner product

   $${\hat s}_k = {\bf z}_k^\top {\bf w}$$

   - If we now group all input vectors into a matrix ${\bf Z}$, so that each row of ${\bf Z}$ contains the transpose of the corresponding ${\bf z}_k$, we can express

$$\hat{{\bf s}} = \left[{\hat s}_0, {\hat s}_1, \dots, {\hat s}_{K-1} \right]^\top =
    {\bf Z} {\bf w}, \;\;\;\;
    \text{with} \;\;
    {\bf Z} = \left[\begin{array}{c} {\bf z}_0^\top \\ {\bf z}_1^\top\\ \vdots \\ {\bf z}_{K-1}^\top
                    \end{array}\right]$$

### 2.3. Least-squares solution

   - Using the previous notation, the cost minimized by the LS model can be expressed as

$$
C({\bf w}) = \sum_{k=0}^{K-1} \left[s_0 - {\hat s}_{K-1} \right]^2
           = \|{\bf s} - {\hat{\bf s}}\|^2 = \|{\bf s} - {\bf Z}{\bf w}\|^2
$$

   - Since the above expression depends quadratically on ${\bf w}$ and is non-negative, we know that there is only one point where the derivative of $C({\bf w})$ becomes zero, and that point is necessarily a minimum of the cost

   $$\nabla_{\bf w} \|{\bf s} - {\bf Z}{\bf w}\|^2\Bigg|_{{\bf w} = {\bf w}_{LS}} = {\bf 0}$$

Exercise:
Solve the previous problem to show that
$${\bf w}_{LS} = \left( {\bf Z}^\top{\bf Z} \right)^{-1} {\bf Z}^\top{\bf s}$$

The next fragment of code adjusts polynomia of increasing order to randomly generated training data. To illustrate the composition of matrix ${\bf Z}$, we will avoid using functions $\mbox{np.polyfit}$ and $\mbox{np.polyval}$.

```python
n_points = 20
n_grid = 200
frec = 3
std_n = 0.2
max_degree = 20

colors = 'brgcmyk'

#Location of the training points
X_tr = (3 * np.random.random((n_points,1)) - 0.5)

#Labels are obtained from a sinusoidal function, and contaminated by noise
S_tr = np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)

#Equally spaced points in the X-axis
X_grid = np.linspace(np.min(X_tr),np.max(X_tr),n_grid)

#We start by building the Z matrix
Z = []
for el in X_tr.tolist():
    Z.append([el[0]**k for k in range(max_degree+1)])
Z = np.matrix(Z)

Z_grid = []
for el in X_grid.tolist():
    Z_grid.append([el**k for k in range(max_degree+1)])
Z_grid = np.matrix(Z_grid)

plt.plot(X_tr,S_tr,'b.')

for k in [1, 2, n_points]: # range(max_degree+1):
    Z_iter = Z[:,:k+1]

    # Least square solution
    #w_LS = (np.linalg.inv(Z_iter.T.dot(Z_iter))).dot(Z_iter.T).dot(S_tr)

    # Least squares solution, with leass numerical errors
    w_LS, resid, rank, s = np.linalg.lstsq(Z_iter, S_tr, rcond=None)
    #estimates at all grid points
    fout = Z_grid[:,:k+1].dot(w_LS)
    fout = np.array(fout).flatten()
    plt.plot(X_grid,fout,colors[k%len(colors)]+'-',label='Degree '+str(k))

plt.legend(loc='best')
plt.ylim(1.2*np.min(S_tr), 1.2*np.max(S_tr))
plt.show()
```

### 2.4. Overfitting the training data

It may seem that increasing the degree of the polynomia is always beneficial, as we can implement a more expressive function. A polynomia of degree $M$ would include all polynomia of lower degrees as particular cases. However, if we increase the number of parameters without control, the polynomia would eventually get expressive enough to adjust any given set of training points to arbitrary precision, what does not necessarily mean that the solution is obtaining a model that can be extrapolated to new data, as we show in the following example:

```python
n_points = 35
n_test = 200
n_grid = 200
frec = 3
std_n = 0.7
max_degree = 25

colors = 'brgcmyk'

#Location of the training points
X_tr = (3 * np.random.random((n_points,1)) - 0.5)
#Labels are obtained from a sinusoidal function, and contaminated by noise
S_tr = np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
#Test points to validate the generalization of the solution
X_tst = (3 * np.random.random((n_test,1)) - 0.5)
S_tst = np.cos(frec*X_tst) + std_n * np.random.randn(n_test,1)

#Equally spaced points in the X-axis
X_grid = np.linspace(np.min(X_tr),np.max(X_tr),n_grid)

#We start by building the Z matrix
def extend_matrix(X,max_degree):
    Z = []
    X = X.reshape((X.shape[0],1))
    for el in X.tolist():
        Z.append([el[0]**k for k in range(max_degree+1)])
    return np.matrix(Z)

Z = extend_matrix(X_tr,max_degree)
Z_grid = extend_matrix(X_grid,max_degree)
Z_test = extend_matrix(X_tst,max_degree)
#Variables to store the train and test errors
tr_error = []
tst_error = []

for k in range(max_degree):
    Z_iter = Z[:,:k+1]
    #Least square solution
    #w_LS = (np.linalg.inv(Z_iter.T.dot(Z_iter))).dot(Z_iter.T).dot(S_tr)

    # Least squares solution, with leass numerical errors
    w_LS, resid, rank, s = np.linalg.lstsq(Z_iter, S_tr)

    #estimates at traint and test points
    f_tr = Z_iter.dot(w_LS)
    f_tst = Z_test[:,:k+1].dot(w_LS)
    tr_error.append(np.array((S_tr-f_tr).T.dot(S_tr-f_tr)/len(S_tr))[0,0])
    tst_error.append(np.array((S_tst-f_tst).T.dot(S_tst-f_tst)/len(S_tst))[0,0])

plt.stem(range(max_degree),tr_error,'b-',label='Train error')
plt.stem(range(max_degree),tst_error,'r-o',label='Test error')
plt.legend(loc='best')
plt.show()
```

#### 2.4.1 Limitations of the LS approach. The need for assumptions

Another way to visualize the effect of overfiting is to analyze the effect of variations o a single sample. Consider a training dataset consisting of 15 points which are given, and depict the regression curves that would be obtained if adding an additional point at a fixed location, depending on the target value of that point:

(You can run this code fragment several times, to check also the changes in the regression curves between executions, and depending also on the location of the training points)

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
n_val_16 = 5
degree = 18

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-.5,2.5,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

X_16 = .3 * np.ones((n_val_16,))
S_16 = np.linspace(np.min(S_tr),np.max(S_tr),n_val_16)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)
ax.plot(X_16,S_16,'ro',markersize=6)
ax.plot(X_grid,S_grid,'r-',label='True model')

for el in zip(X_16,S_16):
    #Add point to the training set
    X_tr_iter = np.append(X_tr,el[0])
    S_tr_iter = np.append(S_tr,el[1])

    #Obtain LS regression coefficients and evaluate it at X_grid
    w_LS = np.polyfit(X_tr_iter, S_tr_iter, degree)
    S_grid_iter = np.polyval(w_LS,X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

ax.set_xlim(-.5,2.5)
ax.set_ylim(S_16[0]-2,S_16[-1]+2)
ax.legend(loc='best')
plt.show()
```

### Exercise

Analyze the performance of LS regression on the `Advertising` dataset. You can analyze:

  - The performance of linear regression when using just one variable, or using all of them together
  - The performance of different non-linear methods (e.g., polynomial or logarithmic transformations)
  - Model selection using CV strategies


```python
%matplotlib inline
import sys, platform, os
import matplotlib
from matplotlib import pyplot as plt
import numpy as np
import scipy as sci
import camb as camb
```

```python
from camb import model, initialpower
print('Using CAMB %s installed at %s'%(camb.__version__,os.path.dirname(camb.__file__)))
```

    Using CAMB 1.3.5 installed at /opt/anaconda3/lib/python3.8/site-packages/camb

```python
import classy as classy
from classy import Class
print('Using CLASS %s installed at %s'%(classy.__version__,os.path.dirname(classy.__file__)))
```

    Using CLASS v2.9.4 installed at /opt/anaconda3/lib/python3.8/site-packages

```python
from ipywidgets.widgets import *
import sympy
from sympy import cos, simplify, sin, sinh, tensorcontraction
from einsteinpy.symbolic import EinsteinTensor, MetricTensor, RicciScalar
sympy.init_printing()
```

```python
from IPython.display import Markdown, display
```

```python
def printmd(string, color='black', math=False, fmt='header2'):
    if math==True:
        mstring = string
    elif math==False:
        mstring="\\textrm{"+string+"}"
    #colorstr = "<span style='color:{}'>{}</span>".format(color, string)
    fmtstr = "${\\color{"+color+"}{"+mstring+"}}$"

    if fmt=='header2':
        fmtstr="## "+fmtstr
    if fmt=='header1':
        fmtstr="# "+fmtstr
    display(Markdown(fmtstr))
    return None
```

```python

```


* This notebook implements the SSL model in [Kingma (2014)](https://arxiv.org/abs/1406.5298).
* C.f. [this implementation](https://github.com/wiseodd/generative-models/blob/master/VAE/conditional_vae/cvae_tensorflow.py).

> What I cannot create I cannot understand.

## Conditional Variational Auto-encoder

### Motivation

* Conditional variational auto-encoder extends variational auto-encoder.

### Definitions

* Let $X$ be the observed random variable and $Y, Z$ the latent. Let

  \begin{equation}
    Y, Z \sim P_{Y, Z}
  \end{equation}

  be some given prior.

* And let

  \begin{equation}
    X \mid Y, Z \sim P_{X \mid Y, Z; \phi}
  \end{equation}

  for some distribution in a family parameterized by $\phi$.

For instance, if $X \in \mathbb{R}$, it is general to assume a multivariate Gaussian distribution, diagonalized or semi-diagonalized, then we have

\begin{equation}
  X \mid Y, Z \sim \mathcal{N} \left( \mu(Y, Z; \phi_1), \sigma(Y, Z; \phi_2) \right),
\end{equation}

for some universality functions $\mu(\cdot; \phi_2)$ and $\sigma(\cdot; \phi_2)$ and $\phi := (\phi_1, \phi_2)$.

### Variational Inference

Utilization of the dataset of $(x, y)$ forces us to employ some distribution $q(z \mid x, y)$ (to be determined) to fit $p(z \mid x, y)$ (and if the dataset is of $(y, z)$, then we are forced to $p(x \mid y, z)$ instead), then we have the bound by KL-divergence

\begin{align}
  \text{KL} \left( q(z \mid x,y) \| p(z \mid x,y) \right)
  & = \mathbb{E}_{z \sim q(z \mid x,y)} \left[
         \ln q(z \mid x,y) - \ln p(x \mid y,z) - \ln p(z \mid y) \right] \\
  & + \ln p(x \mid y) \\
  & \geq 0.
\end{align}

Thus, a loss

\begin{align}
  L(x, y)
  & := \mathbb{E}_{z \sim q(z \mid x,y)} \left[
         \ln q(z \mid x,y) - \ln p(x \mid y,z) - \ln p(z \mid y) \right] \\
  & \geq - \ln p(x \mid y).
\end{align}

The equality can be reached if and only if $q(z \mid x, y) = p(z \mid x, y)$.

### Relation with Auto-encoder

* The $\mathbb{E}_{z \sim q(z \mid x, y)} \left[ - \ln p(x \mid y, z) \right]$ part can be interpreted as a reconstruction loss.

* The $\mathbb{E}_{z \sim q(z \mid x, y)} \left[ - \ln p(z \mid y) \right]$ part serves as regularization terms.

* It seems that the $\mathbb{E}_{z \sim q(z \mid x, y)} \left[ \ln q(z \mid x, y) \right]$ part has no correspondence in auto-encoder.

### Example

In this very example on the MNIST dataset:

\begin{align}
  Y & \sim \text{Categorical uniform}; \\
  Z & \sim \mathcal{N} (0, 1); \\
  X \mid Y, Z & \sim \text{Bernoulli}\left( p(y, z; \theta) \right).
\end{align}

And inference distributions:

\begin{align}
  Z_0 \mid X, Y & \sim \mathcal{N}\left( \mu(x, y; \phi_1), \sigma(x, y; \phi_2) \right); \\
  Z \mid Z_0 & \sim T(z_0).
\end{align}

where $Z_0$ obeys the "base inference distribution", as a Gaussian; and $T$ is bijective with universality.

## Semi-supervised Learning

### Motivation

* Semi-supervised learning utilizes the conditional variational auto-encoder.

### Definitions

* Let

  \begin{equation}
    Y \mid Z \sim P_{Y \mid Z; \theta}.
  \end{equation}

  for some distribution in a family parameterized by $\theta$.

### Variational Inference (Something Wrong!)

By the conditional probability formula,

\begin{equation}
  p(y, z \mid x) = p(z \mid x, y) p(y \mid x).
\end{equation}

With this formula, we have the bound by KL-divergence,

\begin{align}
  \text{KL} \left( q(y, z \mid x) \| p(y, z \mid x) \right)
  & = \int q(y, z \mid x) \left[
        \ln q(y, z \mid x) - \ln p(y, z \mid x)
        \right] dy dz \\
  & = \int q(z \mid x, y) q(y \mid x) \left[
        \ln q(z \mid x, y) + \ln q(y \mid x) - \ln p(x \mid y, z) - \ln p(y, z)
        \right] dy dz \\
  & + \ln p(x).
\end{align}

Integrating out $z$ gives

\begin{align}
  \text{KL} \left( q(y, z \mid x) \| p(y, z \mid x) \right)
  & = \int q(y \mid x) \left[ L(x, y) + \ln q(y \mid x) \right] dy  + \ln p(x) \\
  & = \mathbb{E}_{y \sim q(y \mid x)} \left[ L(x, y) \right]
    - \mathcal{H} \left[ q(y \mid x) \right]
    + \ln p(x) \\
  & \geq 0.
\end{align}

Thus a loss

\begin{align}
  U(x) & := \mathbb{E}_{y \sim q(y \mid x)} \left[ L(x, y) \right]
          - \mathcal{H} \left[ q(y \mid x) \right] \\
       & \geq - \ln p(x).
\end{align}

The equality can be reached if and only if $q(y \mid x) = p(y \mid x)$, then $q(y, z \mid x) = p(y, z \mid x)$.

This loss can be derived in a more concise way. By approximation $-\ln p(x, y) \approx L(x, y)$,

\begin{align}
  \text{KL} \left( q(y \mid x) \| p(y \mid x) \right)
  & = \int q(y \mid x) \left[ \ln q(y \mid x) - \ln p(y \mid x) \right] dy \\
  & = \int q(y \mid x) \left[ \ln q(y \mid x) - \ln p(x, y) \right] dy + \ln p(x) \\
  & \approx \int q(y \mid x) \left[ \ln q(y \mid x) + \ln L(x, y) \right] dy
    + \ln p(x) \\
  & = \mathbb{E}_{y \sim q(y \mid x)} \left[ L(x, y) \right]
    - \mathcal{H} \left[ q(y \mid x) \right].
\end{align}

Thus, again,

\begin{align}
  U(x) & := \mathbb{E}_{y \sim q(y \mid x)} \left[ L(x, y) \right]
          - \mathcal{H} \left[ q(y \mid x) \right] \\
       & \geq - \ln p(x).
\end{align}

The equality can be reached if and only if $q(y \mid x) = p(y \mid x)$.

The trick in the training is that, the minimization of $L(x, y)$ has **completed** the training of the variables in $p(x \mid y, z)$ and $q(z \mid x, y)$, which, thus, shall
**not** be trained further in the minimization of $U(x)$.

(The strange thing is that, after training the variables in $p(x \mid y, z)$ and $q(z \mid x, y)$ by minimizing $L(x, y)$, if train them further in the minimizing of $U(x)$, then the trained variables will give a **higher** $L(x, y)$ than that after the minimization of its own. Why?)

### Example

In this very example:

\begin{equation}
  Y \mid Z \sim \text{Categorical} \left( \pi(Z; \theta) \right).
\end{equation}

## Implementation

In this implementation, we follow the spirit of TensorFlow, hiding the parameters of functions. That is, define `L(X, y; phi)` as `L(phi)(X, y)`.

```python
%matplotlib inline

import os
from IPython.display import display
import matplotlib.pyplot as plt
from PIL import Image
from tqdm import tqdm
import numpy as np
import tensorflow as tf
import tensorflow.contrib.distributions as tfd
from tensorflow.contrib.distributions.python.ops import bijectors as tfb
from tensorflow.examples.tutorials.mnist import input_data

# For reproducibility
seed = 42
np.random.seed(seed)
tf.set_random_seed(seed)
```

### Functions

```python
def get_q_z_Xy(X, y, z_dim, hidden_layers=None, bijectors=None,
               dtype='float32', name='q_z_Xy', reuse=None):
  """Returns the distribution of Z | X, Y.

  Z = bijector(Z_0), and
  Z_0 | X, Y ~ Normal(mu(X,Y;phi), sigma(X,Y;phi)).

  Args:
    X: Tensor with shape `[batch_size, X_dim]`.
    y: Tensor with shape `[batch_size, y_dim]`.
    hidden_layers: List of positive integers. Defaults to
      `[512, 256, 128]`.
    bijectors: List of `tfb.Bijector`s. Defaults to an empty
      list.
    dtype: String.
    name: String.
    reuse: Boolean.

  Returns:
    An instance of `tfd.Distribution` with batch-shape `batch_size`
    and event-shape `z_dim`.
  """
  if bijectors is None:
    bijectors = []
  if hidden_layers is None:
    hidden_layers = [512, 256, 128]

  with tf.variable_scope(name, reuse=reuse):
    # For concatenating, the dtype shall be the same
    X = tf.cast(X, dtype)
    y = tf.cast(y, dtype)
    input_ = tf.concat([X, y], axis=1)

    hidden = input_
    for hidden_layer in hidden_layers:
      hidden = tf.layers.dense(hidden, hidden_layer,
                               activation=tf.nn.relu)
    # Outputs in the fiber-bundle space
    output = tf.layers.dense(hidden, z_dim * 2, activation=None)
    # shape: [batch_size, z_dim]
    mu, log_var = tf.split(output, [z_dim, z_dim], axis=1)

    q_z0_Xy = tfd.MultivariateNormalDiag(mu, tf.exp(log_var))
    chain = tfb.Chain(bijectors)
    q_z_Xy = tfd.TransformedDistribution(distribution=q_z0_Xy,
                                         bijector=chain)
    return q_z_Xy
```

```python
def get_bijectors(bijector_layers=None, name='bjiectors', reuse=None):
  """Complexify the inference distribution by extra-bijectors like
  normalizing flows.

  Args:
    bijector_layers: List of lists of integers. Each sub-list represents
      the hidden_layers of the corresponding bijector. Defaults to ten
      `[128]` lists, i.e. ten bijectors with each having one hidden layer
      with 128 perceptrons.
    name: String.
    reuse: Boolean.

  Returns:
    List of `Bijector`s.
  """
  if bijector_layers is None:
    n_bijectors = 10
    bijector_layers = [[128] for _ in range(n_bijectors)]

  with tf.variable_scope(name, reuse=reuse):
    bijectors = []
    for bijector_layer in bijector_layers:
      shift_and_log_scale_fn = \
          tfb.masked_autoregressive_default_template(bijector_layer)
      # MAP is extremely slow in training. Use IAF instead.
      bijector = tfb.Invert(
          tfb.MaskedAutoregressiveFlow(shift_and_log_scale_fn))
      bijectors.append(bijector)
    return bijectors
```

```python
def get_p_X_yz(y, z, X_dim, hidden_layers=None,
               dtype='float32', name='p_X_yz', reuse=None):
  """Returns the distribution of X | Y, Z.

  X | Y, Z ~ Bernoulli( p(Y, Z; theta) ).

  Args:
    y: Tensor with shape `[batch_size, y_dim]` and dtype `float32`.
    z: Tensor with shape `[batch_size, z_dim]` and dtype `float32`.
    X_dim: Positive integer.
    hidden_layers: List of positive integers. Defaults to
      `[128, 256, 512]`.
    dtype: String.
    name: String.
    reuse: Boolean.

  Returns:
    An instance of `tfd.Distribution` with batch-shape `batch_size`
    and event-shape `X_dim`.
  """
  if hidden_layers is None:
    hidden_layers = [128, 256, 512]

  with tf.variable_scope(name, reuse=reuse):
    # For concatenating, the dtype shall be the same
    y = tf.cast(y, dtype)
    z = tf.cast(z, dtype)
    input_ = tf.concat([y, z], axis=1)

    hidden = input_
    for hidden_layer in hidden_layers:
      hidden = tf.layers.dense(hidden, hidden_layer,
                               activation=tf.nn.relu)
    logits = tf.layers.dense(hidden, X_dim, activation=None)

    p_X_yz = tfd.Independent(tfd.Bernoulli(logits=logits))
    return p_X_yz
```

```python
def get_loss_Xy(get_q_z_Xy, get_p_X_yz, p_z_y, reuse=None):
  """L(X,y) := E_{z ~ q(z|X,y)} [ log_q(z|X,y) - log_p(X|y,z) - log_p(z|y) ].

  Args:
    get_q_z_Xy: Callable with the signature:
      Args:
        X: Tensor with shape `[batch_size, X_dim]`.
        y: Tensor with shape `[batch_size, y_dim]`.
        reuse: Boolean.
      Returns:
        An instance of `tfd.Distribution` with batch-shape `batch_size`
        and event-shape `z_dim`.
    get_p_X_yz: Callable with the signature:
      Args:
        y: Tensor with shape `[batch_size, y_dim]` and dtype `float32`.
        z: Tensor with shape `[batch_size, z_dim]` and dtype `float32`.
        reuse: Boolean.
      Returns:
        An instance of `tfd.Distribution` with batch-shape `batch_size`
        and event-shape `X_dim`.
    p_z_y: An instance of `tfd.Distribution` with batch-shape `batch_size`
      and event-shape `z_dim`.
    reuse: If reuse the variables in `get_q_z_Xy` and `get_p_X_yz`.

  Returns:
    Callable with the signature:
      Args:
        X: Tensor of the shape `[batch_size, X_dim]`.
        y: Tensor of the shape `[batch_size, y_dim]`.
        name: String.
      Returns:
        Tensor of the shape `[batch_size]`.
  """

  def loss_Xy(X, y, name='loss_Xy'):
    """
    Args:
      X: Tensor of the shape `[batch_size, X_dim]`.
      y: Tensor of the shape `[batch_size, y_dim]`.
      name: String.

    Returns:
      Tensor of the shape `[batch_size]`.
    """
    with tf.name_scope(name):
      # Get the distribution q(z|X,y)
      q_z_Xy = get_q_z_Xy(X, y, reuse=reuse)

      # Get the distribution p(X|y,z)
      z_samples = q_z_Xy.sample()
      p_X_yz = get_p_X_yz(y, z_samples, reuse=reuse)

      # Compute the tensor of L(X,y)
      loss_Xy_tensor = tf.zeros([batch_size])  # initialize.
      # E_{z ~ q(z|X,y)} [ log_q(z|X,y) ]
      loss_Xy_tensor += q_z_Xy.log_prob(z_samples)
      # E_{z ~ q(z|X,y)} [ - log_p(z|y) ]
      loss_Xy_tensor += -1 * p_z_y.log_prob(z_samples)
      # E_{z ~ q(z|X,y)} [ - log_p(X|y,z) ]
      loss_Xy_tensor += -1 * p_X_yz.log_prob(X)
      return loss_Xy_tensor

  return loss_Xy
```

```python
def get_q_y_X(X, name='q_y_X', reuse=None):
  """Returns the distribution of Y | X.

  Y | X ~ Categorical( pi(X; alpha) ).

  Args:
    X: Tensor of the shape `[batch_size, X_dim]`.

  Returns:
    An instance of `tfd.Distribution` with batch-shape `batch_size`.
    and event-shape `()`.
  """
  with tf.variable_scope(name, reuse=reuse):
    hidden = tf.layers.dense(X, 512, activation=tf.nn.relu)
    hidden = tf.layers.dense(hidden, 256, activation=tf.nn.relu)
    hidden = tf.layers.dense(hidden, 128, activation=tf.nn.relu)
    # Outputs in the fiber-bundle space
    logits = tf.layers.dense(hidden, y_dim, activation=None)
    q_y_X = tfd.Independent(tfd.OneHotCategorical(logits=logits))
    return q_y_X
```

```python
def get_loss_X(q_y_X, loss_Xy, name='loss_X'):
  """U(X) := E_{y ~ q(y|X)} [ L(X,y) ] - H(q(y|X)).

  Args:
    q_y_X: An instance of `tfd.Distribution` with batch-shape `batch_size`
      and event-shape `()`.
    loss_Xy: Callable with the signature:
      Args:
        X: Tensor of the shape `[batch_size, X_dim]`.
        y: Tensor of the shape `[batch_size, y_dim]`.
      Returns:
        Tensor of the shape `[batch_size]`.

  Returns:
    Callable with signature:
      Args:
        X: Tensor of the shape `[batch_size, X_dim]`.
      Returns:
        Tensor of the shape `[batch_size]`.
  """
  def loss_X(X):
    """
    Args:
      X: Tensor of the shape `[batch_size, X_dim]`.

    Returns:
      Tensor of the shape `[batch_size]`.
    """
    with tf.name_scope(name):
      # shape: `[batch_size, y_dim]`
      y_samples = q_y_X.sample()
      # shape: `[batch_size]`
      loss_X_tensor = loss_Xy(X, y_samples) - q_y_X.entropy()
      return loss_X_tensor
  return loss_X
```

```python
import collections

def get_dependent_variables(tensor_or_tensors):
  """Returns all variables that the tensor `tensor` depends on.

  Forked from: https://stackoverflow.com/a/42861919/1218716

  Args:
    tensor_or_tensors: Tensor or a collection of tensors.

  Returns:
    List of variables.
  """
  if isinstance(tensor_or_tensors, (list, tuple, set)):  # a collection.
    tensors = tensor_or_tensors
    dependent_vars = []
    for tensor in tensors:
      dependent_vars += get_dependent_variables(tensor)
    return list(set(dependent_vars))
  # When not a collection,
  tensor = tensor_or_tensors

  # Initialize
  starting_op = tensor.op
  dependent_vars = []
  queue = collections.deque()
  queue.append(starting_op)
  op_to_var = {var.op: var for var in tf.trainable_variables()}
  visited = set([starting_op])

  while queue:
    op = queue.popleft()
    try:
      dependent_vars.append(op_to_var[op])
    except KeyError:
      # `op` is not a variable, so search its inputs (if any).
      for op_input in op.inputs:
        if op_input.op not in visited:
          queue.append(op_input.op)
          visited.add(op_input.op)

  return dependent_vars
```

### Loss

```python
batch_size = 128
X_dim = 28 * 28
y_dim = 10

X = tf.placeholder(shape=[batch_size, X_dim], dtype='float32', name='X')
y = tf.placeholder(shape=[batch_size, y_dim], dtype='float32', name='y')
```

```python
z_dim = 64

def _get_q_z_Xy(X, y, reuse):
  bijectors = get_bijectors(reuse=reuse)
  #bijectors = []  # for comparison
  return get_q_z_Xy(X, y, z_dim, bijectors=bijectors, reuse=reuse)

def _get_p_X_yz(y, z, reuse):
  return get_p_X_yz(y, z, X_dim=X_dim, reuse=reuse)

p_z_y = tfd.MultivariateNormalDiag(tf.zeros([batch_size, z_dim]),
                                   name='p_z_y')

loss_Xy = get_loss_Xy(_get_q_z_Xy, _get_p_X_yz, p_z_y,
                      reuse=tf.AUTO_REUSE)
loss_Xy_tensor = loss_Xy(X, y)
loss_Xy_scalar = tf.reduce_mean(loss_Xy_tensor)
```

```python
q_y_X = get_q_y_X(X, reuse=tf.AUTO_REUSE)
y_samples = q_y_X.sample()
```

```python
loss_X = get_loss_X(q_y_X, loss_Xy)
loss_X_tensor = loss_X(X)
loss_X_scalar = tf.reduce_mean(loss_X_tensor)
```

### Training

```python
# Minimize L(X,y)
optimizer = tf.train.AdamOptimizer(epsilon=1e-3)
minimize_loss_Xy = optimizer.minimize(loss_Xy_scalar)
```

```python
# Minimize U(X), while fixing the variables in L(X,y)
optimizer = tf.train.AdamOptimizer(epsilon=1e-3, learning_rate=1e-5)
loss_X_vars = [_ for _ in get_dependent_variables(loss_X_scalar)
               if _ not in get_dependent_variables(loss_Xy_scalar)]
minimize_loss_X = optimizer.minimize(loss_X_scalar, var_list=loss_X_vars)
```

### Generating

```python
# X | Y,Z
z_samples_ph = tf.placeholder(shape=[batch_size, z_dim], dtype='float32')
p_X_yz_gen = _get_p_X_yz(y, z_samples_ph, reuse=tf.AUTO_REUSE)
X_samples = p_X_yz_gen.sample()
```

```python
def get_image(array):
  """
  Args:
    array: Numpy array with shape `[28*28]`.

  Returns:
    An image.
  """
  array = 255 * array
  array = array.reshape([28, 28])
  array = array.astype(np.uint8)
  return Image.fromarray(array)
```

### Having Fun

```python
sess = tf.Session()
sess.run(tf.global_variables_initializer())
```

```python
DATA_DIR = './dat'

mnist = input_data.read_data_sets(
    os.path.join(DATA_DIR, 'MNIST'),
    one_hot=True,
    source_url='http://yann.lecun.com/exdb/mnist/')

def get_X_y_batch(source='train'):
  if source == 'train':
    data = mnist.train
  elif source == 'test':
    data = mnist.test
  else:
    raise ValueError('Argument source can either be "train" or "test".')
  X_batch, y_batch = data.next_batch(batch_size)
  # Since X | Y, Z ~ Bernoulli, the observed value of X shall
  # either be 0 or 1,
  X_batch = np.where(X_batch >= 0.5, np.ones_like(X_batch),
                     np.zeros_like(X_batch))
  return X_batch, y_batch
```

    Extracting ./dat/MNIST/train-images-idx3-ubyte.gz
    Extracting ./dat/MNIST/train-labels-idx1-ubyte.gz
    Extracting ./dat/MNIST/t10k-images-idx3-ubyte.gz
    Extracting ./dat/MNIST/t10k-labels-idx1-ubyte.gz

#### Minimizing $L(x, y)$ over batches

```python
# Training by minimizing `loss_Xy_scalar`
loss_Xy_vals = []
smeared_loss_Xy_vals = []
smear_steps = 300
for i in tqdm(range(100000)):
  X_batch, y_batch = get_X_y_batch()
  _, loss_Xy_val = sess.run([minimize_loss_Xy, loss_Xy_scalar],
                            feed_dict={X: X_batch, y: y_batch})
  if np.isnan(loss_Xy_val):
    raise ValueError('Loss has been NaN.')
  loss_Xy_vals.append(loss_Xy_val)
  smeared_loss_Xy_vals.append(np.mean(loss_Xy_vals[-smear_steps:]))
print('Mean L(X,y):', np.mean(loss_Xy_vals))
print('Final L(X,y):', smeared_loss_Xy_vals[-1])

# Visualize the training loss
plt.plot(smeared_loss_Xy_vals)
plt.xlabel('steps')
plt.ylabel('L(X,y) (smeared)')
plt.show()

# Visualize the training loss in the last several steps
last_steps = 20000
plt.plot(smeared_loss_Xy_vals[-last_steps:])
plt.xlabel('last steps')
plt.ylabel('L(X,y) (smeared)')
plt.show()
```

```python
_, y_batch = get_X_y_batch()

# Generate X
z_samples_val = np.random.normal(size=[batch_size, z_dim])
X_sample_vals = sess.run(X_samples,
    feed_dict={y: y_batch, z_samples_ph: z_samples_val})

# Display the results
n_display = 5
for i in range(n_display):
  print('Label:', np.argmax(y_batch[i]))
  print('Gnerated:')
  display(get_image(X_sample_vals[i]))
  print()
```

#### Minimizing $U(x)$ over batches

```python
'''
# Training by minimizing `loss_X_scalar`
loss_X_vals = []
smeared_loss_X_vals = []
for i in tqdm(range(100000)):
  X_batch, y_batch = get_X_y_batch()
  _, loss_X_val = sess.run([minimize_loss_X, loss_X_scalar],
                            feed_dict={X: X_batch})
  if np.isnan(loss_X_val):
    raise ValueError('Loss has been NaN.')
  loss_X_vals.append(loss_X_val)
  smeared_loss_X_vals.append(np.mean(loss_X_vals[-smear_steps:]))
print('Mean U(X):', np.mean(loss_X_vals))
print('Final U(X):', smeared_loss_X_vals[-1])

# Visualize the training loss
plt.plot(smeared_loss_X_vals)
plt.xlabel('steps')
plt.ylabel('U(X) (smeared)')
plt.show()

# Visualize the training loss in the last several steps
plt.plot(smeared_loss_X_vals[-last_steps:])
plt.xlabel('last steps')
plt.ylabel('U(X) (smeared)')
plt.show()
'''
```

    "\n# Training by minimizing `loss_X_scalar`\nloss_X_vals = []\nsmeared_loss_X_vals = []\nfor i in tqdm(range(100000)):\n  X_batch, y_batch = get_X_y_batch()\n  _, loss_X_val = sess.run([minimize_loss_X, loss_X_scalar],\n                            feed_dict={X: X_batch})\n  if np.isnan(loss_X_val):\n    raise ValueError('Loss has been NaN.')\n  loss_X_vals.append(loss_X_val)\n  smeared_loss_X_vals.append(np.mean(loss_X_vals[-smear_steps:]))\nprint('Mean U(X):', np.mean(loss_X_vals))\nprint('Final U(X):', smeared_loss_X_vals[-1])\n\n# Visualize the training loss\nplt.plot(smeared_loss_X_vals)\nplt.xlabel('steps')\nplt.ylabel('U(X) (smeared)')\nplt.show()\n\n# Visualize the training loss in the last several steps\nplt.plot(smeared_loss_X_vals[-last_steps:])\nplt.xlabel('last steps')\nplt.ylabel('U(X) (smeared)')\nplt.show()\n"

```python
'''
X_batch, y_batch = get_X_y_batch()

# Predict y
y_sample_vals = sess.run(y_samples, feed_dict={X: X_batch})

# Display the results
n_display = 5
for i in range(n_display):
  print('Label:', np.argmax(y_batch[i]))
  print('Predicted:', np.argmax(y_sample_vals[i]))
  print('Original:')
  display(get_image(X_batch[i]))
'''
```

    "\nX_batch, y_batch = get_X_y_batch()\n\n# Predict y\ny_sample_vals = sess.run(y_samples, feed_dict={X: X_batch})\n\n# Display the results\nn_display = 5\nfor i in range(n_display):\n  print('Label:', np.argmax(y_batch[i]))\n  print('Predicted:', np.argmax(y_sample_vals[i]))\n  print('Original:')\n  display(get_image(X_batch[i]))\n"

* We find that the minimization of $U(x)$ is always non-effective. Why?

### Effect of Normalizing-flow

* While not employing normalizing flow, 100000 iterations give the final L(X,y) about 70.0.
* With 10 normalizing flows, 100000 iterations give the L(X,y) loss 65.5 (improved 4.5).

```python
np.exp(4)
```

    54.598150033144236

### Discrimination (Something Wrong!)

The training by minimizing $L(x, y)$ over data-batches gains the approximation

\begin{equation}
  \ln p(x \mid y) = - L(x, y).
\end{equation}

Thus,

\begin{align}
  \ln p(y \mid x)
  & = \ln p(x \mid y) + \ln p(y) - \ln p(x) \\
  & = -L(x, y) + \ln p(y) - \ln p(x).
\end{align}

That is, given $x$, the most probable value of $y$ is the one that minimizes the $L(x, y) + \ln p(y)$.

```python
'''  # TODO
def predict(get_loss_Xy_val, X_batch, y_dim=y_dim):
  """Returns the log-probability of discrimination for each element
  in the X's batch `X_batch`.

  Args:
    get_loss_Xy_val: Callable that maps two arraies of the shapes
      `[batch_size, X_dim]` and `[batch_size, y_dim]`, respectively,
      to an array of the shape `[batch_size]`.
    X: Array of the shape `[batch_size, X_dim]`.

  Returns:
    An array of the shape `[batch_size, y_dim]`.
  """
  batch_size, _ = X_batch.shape

  loss_Xy_vals = []
  for i in range(y_dim):
    y_batch = np.zeros([batch_size, y_dim])
    y_batch[:,i] = np.ones([batch_size])
    loss_Xy_vals.append(get_loss_Xy_val(X_batch, y_batch))
  # shape: `[batch_size, y_dim]`
  log_prob_Xy = -np.stack(loss_Xy_vals, axis=1)
  # shape: `[batch_size]`
  log_prob_X = logsumexp(log_prob_Xy, axis=1)
  # shape: `[batch_size, 1]`
  log_prob_X = np.expand_dims(log_prob_X, axis=1)
  # shape: `[batch_size, y_dim]`
  log_probs = log_prob_Xy - log_prob_X
  return log_probs
'''
```

    '  # TODO\ndef predict(get_loss_Xy_val, X_batch, y_dim=y_dim):\n  """Returns the log-probability of discrimination for each element\n  in the X\'s batch `X_batch`.\n  \n  Args:\n    get_loss_Xy_val: Callable that maps two arraies of the shapes\n      `[batch_size, X_dim]` and `[batch_size, y_dim]`, respectively,\n      to an array of the shape `[batch_size]`.\n    X: Array of the shape `[batch_size, X_dim]`.\n    \n  Returns:\n    An array of the shape `[batch_size, y_dim]`.\n  """\n  batch_size, _ = X_batch.shape\n  \n  loss_Xy_vals = []\n  for i in range(y_dim):\n    y_batch = np.zeros([batch_size, y_dim])\n    y_batch[:,i] = np.ones([batch_size])\n    loss_Xy_vals.append(get_loss_Xy_val(X_batch, y_batch))\n  # shape: `[batch_size, y_dim]`\n  log_prob_Xy = -np.stack(loss_Xy_vals, axis=1)\n  # shape: `[batch_size]`\n  log_prob_X = logsumexp(log_prob_Xy, axis=1)\n  # shape: `[batch_size, 1]`\n  log_prob_X = np.expand_dims(log_prob_X, axis=1)\n  # shape: `[batch_size, y_dim]`\n  log_probs = log_prob_Xy - log_prob_X\n  return log_probs\n'

```python
def get_loss_Xy_val(X_batch, y_batch):
  return sess.run(loss_Xy_tensor, {X: X_batch, y: y_batch})
```

```python
'''  # TODO
X_batch, y_batch = get_X_y_batch(source='test')
predicted = predict(get_loss_Xy_val, X_batch)

count = 0
for i, X_val in enumerate(X_batch):
  y_val = np.argmax(y_batch, axis=1)[i]
  p_val = np.argmax(predicted, axis=1)[i]

  if y_val != p_val:
    display(get_image(X_val))
    print('Difference:', y_val, p_val)
    print('Probabilities:', ', '.join(['{0:.3f}'.format(np.exp(_))
                                       for _ in predicted[i]]))
    print()
    count += 1
print('Accuracy:', 1 - count / len(y_batch))
'''
```

    "  # TODO\nX_batch, y_batch = get_X_y_batch(source='test')\npredicted = predict(get_loss_Xy_val, X_batch)\n\ncount = 0\nfor i, X_val in enumerate(X_batch):\n  y_val = np.argmax(y_batch, axis=1)[i]\n  p_val = np.argmax(predicted, axis=1)[i]\n                 \n  if y_val != p_val:\n    display(get_image(X_val))\n    print('Difference:', y_val, p_val)\n    print('Probabilities:', ', '.join(['{0:.3f}'.format(np.exp(_))\n                                       for _ in predicted[i]]))\n    print()\n    count += 1\nprint('Accuracy:', 1 - count / len(y_batch))\n"

```python

```


  Computational Physics I | FS 2022

 Christoph Würsch

 Eastern Switzerland University of Applied Sciences OST | ICE

# How long does it take to boil an egg?

Source:
- Norwegian Centre of Excellence: [Finite difference methods for diffusion processes](http://hplgit.github.io/num-methods-for-PDEs/doc/pub/diffu/sphinx/._main_diffu001.html)
- Ashlee N. Ford, Versypt, Richard D. Braatz: [Analysis_of_finite_difference_discretization_schemes_for_diffusion_in_spheres_with_variable_diffusivity]( https://web.mit.edu/braatzgroup/analysis_of_finite_difference_discretization_schemes_for_diffusion_in_spheres_with_variable_diffusivity.pdf)

How long does it take to boil an egg? Theoretical prediction, based on a simple
adaptation of the solution to the exact thermal diffusion equation for a sphere,
is consistent with experiments. The experimental data are also used to estimate
an average value for the thermal diffusivity of an egg.

- The boiling of eggs to perfection is of delicious interest to many of us who enjoy well-cooked
hard-boiled eggs.
- Unfortunately, unlike other food, one cannot tell whether an egg is cooked  just by looking at it externally, namely at its shell. But perhaps we can be satisfied if we are able to tell by looking at a watch?
- So, how long does it take to boil an egg?

Of course, the answer depends on what is meant by a boiled egg—very much like steak—some like it soft
and some like it hard.

Boiling an egg involves **non-steady thermal diffusion** where the temperature within the
egg is dependent on both time and position, and due to its geometry, an exact solution, if
available, will likely have complicated expressions.

Based on an approximate solution, Roura
et al [1], who had asked the same question in the title, estimated that it takes about 13 min.

Here, we derive another formula for the time required to boil an egg and actually
measured the time experimentally. Our formula is a simple adaptation of the solution to the
exact thermal diffusion equation for a sphere [2].

- [1] [Roura P, Fort J and Saurina J 2000 How long does it take to boil an egg? A simple approach to the energy transfer
equation Eur. J. Phys. 21 95–100](https://iopscience.iop.org/article/10.1088/0143-0807/21/1/314)
- [2] [Unsworth J and Durate F J 1979 Heat diffusion in a solid sphere and Fourier theory: an elementary practical
example Am. J. Phys. 47 981–3](https://aapt.scitation.org/doi/10.1119/1.11601)

##  1. Transitions of the interior of an egg

What happens to the egg yolk and egg albumen, collectively known as the egg magma, as the temperature is raised?

1. According to *Powrie and Nakai* [3], an egg begins to **denature from a semi-transparent colloid to an opaque white**, at around 60 ◦C. At 75 ◦C,
2. the egg magma will attain a soft curd consistency and it toughens as the egg’s temperature rises to 87 ◦C, as the **denaturation of ovalbumin** (a major conjugated protein in the egg albumen which represents over 50% of the egg albumen) occurs from 79 ◦C to 84 ◦C.
3. When the proteins in the egg denature, they lose their specific tertiary structures due to the breakage of hydrogen bonds and the uncoiling of polypeptide chains [3]. At the same time, they coagulate through chemical and physical bonding to form a threedimensional gel network, resulting in **increasing viscosity of the egg magma as temperature rises**.

- In addition, the surface of the egg yolk will turn yellowish green as its **iron compounds react with the albumen** to form an iron–conalbumin complex.

**In view of this information, the definition for a well-cooked hard-boiled egg is the condition when the egg magma solidifies to reach a tender but firm consistency, and before its solid yolk develops a greenish yellow discolouration.**

[3] [Powrie W D and Nakai S 1985 Characteristics of edible fluids of animal origin: eggs Food Chemistry 2nd edn
ed O R Fennema (New York: Dekker) pp 830–51](https://agris.fao.org/agris-search/search.do?recordID=US201303056734)

## 2. Heat diffusion equations

Having defined the criteria of a hard-boiled egg, the next stage is to consider the heat transfer
involved in cooking an egg. In a steady state situation, the conduction of heat involves the
Fourier law, which states that the heat flux Jq is proportional to the temperature gradient:

$$ \mathbf{J}_q = -\kappa \frac{\partial T}{\partial x} \tag{1}$$

where $\kappa$ is the thermal conductivity. During the boiling process, the temperature of the interior of the egg changes with respect to time and position due to thermal diffusion. For an **unsteady heat transfer** in a uniform solid that does not have a heat source embedded, the temperature must obey the equation

$$ \frac{\partial T}{\partial t} = \alpha \nabla^2 T = \alpha \Delta T \tag{2}$$

where

$$T=T(x,y,z,t)$$

it the scalar tempeature field at position $(x,y,z)$ and time $t$,

$$ \Delta = \nabla^2 = \frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y
^2} + \frac{\partial^2}{\partial z^2} \tag{3} $$

is the so-called **Laplace differential operator** and

$$\alpha \equiv \frac{\kappa}{\rho c_p} \tag{4}$$

is the **thermal diffusivity** of the solid;
where $\rho$ is the **density** of the solid and $c_p$ is the **specific heat**.

For a solid sphere submerged in a thermal bath, equation (2) simplifies to

$$\frac{\partial T}{\partial t} = \alpha \frac{1}{r^2} \frac{\partial}{\partial r} \left ( r^2 \frac{\partial T}{\partial r} \right ) \tag{5}$$

where $\kappa$ is the **thermal conductivity**.

The solution of equation (3) for the case of cooling the sphere of radius R in a bath is given
by Unsworth and Duarte [2]. In our case of heating a sphere in a bath, their solution must be
adapted accordingly to read

$$T(r,t)=T_b - \frac{2R(T_b-T_0)}{\pi r} \sum_{n=1}^{\infty} \frac{(-1)^{n+1}}{n} \cdot \sin\left( \frac{n \pi r}{R} \right) \cdot e^{-\alpha n^2 \pi^2 t / R^2} \tag{6}$$

```python
import sympy as sym
import matplotlib.pyplot as plt

r = sym.Symbol('r')
t = sym.Symbol('t')
T = sym.Symbol('T')
a = sym.Symbol('a')
u = sym.Symbol('u_n')
v = sym.Symbol('v_n')
c = sym.Symbol('c_n')
n = sym.Symbol('n')
```

### solving be separation of variables $h(r,t)=f(r)\cdot g(t)$

```python
f = sym.Function('f')(r)
g = sym.Function('g')(t)
h= f*g
```

```python
LHS=sym.diff(h,t)
LHS
```

$\displaystyle f{\left(r \right)} \frac{d}{d t} g{\left(t \right)}$

```python
RHS=sym.simplify(1/(r**2)*sym.diff(sym.diff((r**2)*h, r),r))
RHS
```

$\displaystyle \frac{\left(r^{2} \frac{d^{2}}{d r^{2}} f{\left(r \right)} + 4 r \frac{d}{d r} f{\left(r \right)} + 2 f{\left(r \right)}\right) g{\left(t \right)}}{r^{2}}$

```python
RHS/g/f
```

$\displaystyle \frac{r^{2} \frac{d^{2}}{d r^{2}} f{\left(r \right)} + 4 r \frac{d}{d r} f{\left(r \right)} + 2 f{\left(r \right)}}{r^{2} f{\left(r \right)}}$

```python
eq1=LHS/g/f+c
eq1
```

$\displaystyle c_{n} + \frac{\frac{d}{d t} g{\left(t \right)}}{g{\left(t \right)}}$

```python
sol1=sym.dsolve(eq1)
sol1
```

$\displaystyle g{\left(t \right)} = C_{1} e^{- c_{n} t}$

```python
from sympy.solvers.ode import checkodesol
checkodesol(sol1,eq1)
```

    [(False, -C1 + g(t)*exp(c_n*t)), (False, -C1 + g(t)*exp(c_n*t))]

```python
eq2=RHS/f/g+c
sol2=sym.dsolve(eq2,f)
sol2
```

$\displaystyle f{\left(r \right)} = \frac{C_{1} J_{\frac{1}{2}}\left(\sqrt{c_{n}} r\right) + C_{2} Y_{\frac{1}{2}}\left(\sqrt{c_{n}} r\right)}{r^{\frac{3}{2}}}$

```python
from sympy.solvers.ode import checkodesol
checkodesol(sol2,eq2)
```

    [(False,
      -C1*besselj(1/2, sqrt(c_n)*r) - C2*bessely(1/2, sqrt(c_n)*r) + r**(3/2)*f(r)),
     (False,
      -C1*besselj(1/2, sqrt(c_n)*r) - C2*bessely(1/2, sqrt(c_n)*r) + r**(3/2)*f(r))]

## Exact Solution

```python
cp=2.7E3        #specific heat capacity in (J/kg/K)
kappa=3.4E-1    #heat conductivity in (W/m/K)
rho=1032        #density (kg/m^3)
alpha=kappa/rho/cp
alpha
```

    1.2202124605225383e-07

```python
import numpy as np

def CalcTemp(r,t,Tb=373, T0=293, R=0.05, alpha=alpha):
    a=2*R*(Tb-T0)/(np.pi*r)
    sn=0
    for n in range(1,1000):
        c=alpha*n**2*np.pi**2/R**2
        sn=sn+(-1)**(n+1)/n*np.sin(np.pi*n*r/R)*np.exp(-c*t)
    sn=Tb-a*sn
    return sn
```

```python
R=0.05
r=np.linspace(0.001,R,200)
t=1
T=CalcTemp(r,t,Tb=373, T0=293, R=R)
```

```python
timespan=np.linspace(1,2400,11)

plt.figure(figsize=(12,8))

for t in timespan:
    T=CalcTemp(r,t,Tb=373, T0=283, R=R)
    LabelString=('t=%i s' % int(t))
    plt.plot(r,T,label=LabelString);

plt.legend()
plt.grid(True)
plt.show()

```

## Approximate analytical solutions for the cooking time $t_E$

In this section, we outline two approximations:
- an approximation of the general solution to yield an analytical approximation of cooking time $t_E$,
- and an approximation of the shape of egg as a sphere.

Although the general solution is given in terms of the theta function which is an infinite series, terms higher than a certain order are negligible due to the damping provided by the $n^2$ dependence of the exponential factor. By neglecting the higher order terms of the series, and solving the resulting equation, the **cooking time** $t_E$ can be approximated analytically.

It is also convenient to be able to calculate it given the value of $T_c$ (temperature at the center of the egg) necessary for the egg to be cooked, although it can be obtained from equation (5) through plotting graphs such as figure 1.

$$t_E \approx \frac{R^2}{\pi^2 \alpha} \ln \left[ \frac{2\, (T_b-T_0)}{T_b-T_c} \right]$$

A similar derivation was done by Williams [7]. His result (which we have adapted for comparison purposes) was

$$t_E \approx \frac{R^2}{\pi^2 \alpha} \ln \left[ \frac{0.76\, (T_b-T_0)}{T_b-T_c} \right]$$

The difference in the results is due to the difference in the interpretation of a cooked egg:
he considered an egg is cooked *when the boundary of the yolk reaches a certain temperature*
while we consider an egg is cooked *when the centre of the yolk reaches a certain temperature*.

# Numerical Solution

## The 1D diffusion equation¶
The famous diffusion equation, also known as the heat equation,
reads

$$\frac{\partial u}{\partial t} =
{\alpha} \frac{\partial^2 u}{\partial x^2},$$

where $u(x,t)$ is the unknown function to be solved for, $x$ is a coordinate
in space, and $t$ is time. The coefficient ${\alpha}$ is the diffusion
coefficient and determines how fast $u$ changes in time. A quick
short form for the diffusion equation is $u_t = {\alpha} u_{xx}$.

- Typical diffusion problems may experience **rapid change in the very beginning**, but then the evolution of $u$ becomes slower and slower.
- The solution is usually **very smooth**, and after some time, one cannot recognize the initial shape of $u$.
- The diffusion equation converges to **a stationary solution** $\bar u(x)$ as $t\rightarrow\infty$.
- In this limit, $u_t=0$, and $\bar u$ is governed by $\bar u''(x)=0$.
- This stationary limit of the diffusion equation is called the **Laplace equation** and arises in a very wide range of applications throughout the sciences.

- It is possible to solve for $u(x,t)$ using a explicit scheme.
- Since the solution $u$ of the diffusion equation is very smooth and changes slowly, small time steps are not convenient and not required by accuracy as the diffusion process converges to a stationary state.

### The initial-boundary value problem for 1D diffusion¶

To obtain a unique solution of the diffusion equation, or equivalently,
to apply numerical methods, we need initial and boundary conditions.
The diffusion equation goes with one initial condition $u(x,0)=I(x)$, where $I$ is a prescribed function. One boundary condition is required at each point on the boundary, which in 1D means that $u$ must be known, $u_x$ must be known, or some combination of them.

- We shall start with the simplest boundary condition: $u=0$.
- The complete initial-boundary value diffusion problem in one space dimension can then be specified as

$$\tag{1}
\frac{\partial u}{\partial t} =
    {\alpha} \frac{\partial^2 u}{\partial x^2}, \quad x\in (0,L),\ t\in (0,T]$$

$$\tag{2}
u(x,0) = I(x), \quad  x\in [0,L]$$

$$\begin{align}\tag{3}
u(0,t)  = 0, \quad  t>0,\end{align}$$

$$\begin{align}\tag{4}
u(L,t)  = 0, \quad  t>0{\thinspace .}\end{align}$$

Equation (1) is known as a one-dimensional diffusion equation, also often referred to as a
**heat equation**. With only a first-order derivative in time, *only one initial condition is needed*, while the *second-order derivative in space* leads to a demand for *two boundary conditions*.
- The parameter ${\alpha}$ must be given and is referred to as the diffusion coefficient.
- Diffusion equations like (1) have a wide range of applications throughout physical, biological, and financial sciences.
- One of the most common applications is propagation of heat, where $u(x,t)$ represents the temperature of some substance at point $x$ and time $t$.

### Forward Euler scheme¶

The first step in the discretization procedure is to replace the domain $[0,L]\times [0,T]$ by a set of mesh points. Here we apply equally spaced mesh points

$$x_i=i\Delta x,\quad i=0,\ldots,N_x,$$

and

$$t_n=n\Delta t,\quad n=0,\ldots,N_t {\thinspace .}$$

Moreover, $u^n_i$ denotes the mesh function that approximates $u(x_i,t_n)$ for $i=0,\ldots,N_x$ and $n=0,\ldots,N_t$.
Requiring the PDE (1) to be fulfilled at a mesh point $(x_i,t_n)$ leads to the equation

$$\tag{5}
\frac{\partial}{\partial t} u(x_i, t_n) =
    {\alpha}\frac{\partial^2}{\partial x^2} u(x_i, t_n),$$

The next step is to replace the derivatives by finite difference approximations.
The computationally simplest method arises from
using a forward difference in time and a central difference in
space:

$$\tag{6}
[D_t^+ u = {\alpha} D_xD_x u]^n_i {\thinspace .}$$

Written out,

$$\tag{7}
\frac{u^{n+1}_i-u^n_i}{\Delta t} = {\alpha} \frac{u^{n}_{i+1} - 2u^n_i + u^n_{i-1}}{\Delta x^2} {\thinspace .}$$

We have turned the PDE into algebraic equations, also often called
discrete equations. The key property of the equations is that they
are algebraic, which makes them easy to solve.
As usual, we anticipate that $u^n_i$ is already computed such that
$u^{n+1}_i$ is the only unknown in (7).
Solving with respect to this unknown is easy:

$$\tag{8}
u^{n+1}_i = u^n_i + F\left(
    u^{n}_{i+1} - 2u^n_i + u^n_{i-1}\right){\thinspace .}$$

- $F$ is the key parameter in the discrete diffusion equation (**Courant–Friedrichs–Lewy condition**).
- Note that $F$ is a dimensionless number that lumps the key physical parameter in the problem, ${\alpha}$, and the discretization parameters $\Delta x$ and $\Delta t$ into a single parameter.
- All the  properties of the numerical method are critically dependent upon the value of $F$.

The computational algorithm then becomes

1. compute $u^0_i=I(x_i)$for $i=0,\ldots,N_x$
2. for $n=0,1,\ldots,N_t$:
3. apply (8) for all the internal spatial points $i=1,\ldots,N_x-1$
4. set the boundary values $u^{n+1}_i=0$ for $i=0$ and $i=N_x$

The algorithm is compactly fully specified in Python:

```python
import numpy as np

L=1
T=10
Nx=100
Nt=3000
a=0.01

def Initial(x):
    #definition of the initial temperature (smooth)
    return np.sin(x)
```

```python
x   = np.linspace(0, L, Nx+1)    # mesh points in space
dx  = x[1] - x[0]
t   = np.linspace(0, T, Nt+1)    # mesh points in time
dt  = t[1] - t[0]
F   = a*dt/dx**2                 # Courant–Friedrichs–Lewy condition
print('Courant–Friedrichs–Lewy condition: %f' % F)

u   = np.zeros((Nx+1,))          # unknown u at new time level
u_1 = np.zeros((Nx+1,))          # u at the previous time level

# Set initial condition u(x,0) = I(x)
for i in range(0, Nx+1):
    u_1[i] = Initial(x[i])
```

    Courant–Friedrichs–Lewy condition: 0.333333

```python
for n in range(0, Nt):
    # Compute u at inner mesh points
    for i in range(1, Nx):
        u[i] = u_1[i] + F*(u_1[i-1] - 2*u_1[i] + u_1[i+1])

    # Insert boundary conditions
    u[0] = 0;  u[Nx] = 0

    # Update u_1 before next step
    u_1[:]= u

    if n%200==1:
        plt.plot(u)
    plt.grid(True)
```

## Diffusion equation in axi-symmetric geometries¶

Suppose we have a diffusion process taking care in a straight tube
with radius $R$. We assume axi-symmetry such that $u$ is just a
function of $r$ and $t$. A model problem is

$$\tag{72}
\frac{\partial u}{\partial t} = \frac{1}{r}\frac{\partial}{\partial r}
    \left(r{\alpha}(r)\frac{\partial u}{\partial r}\right) + f(t),  r\in (0,R),\ t\in (0,T],$$

$$\tag{73}
\frac{\partial u}{\partial r}(0,t) = 0,  t\in (0,T],$$

$$\tag{74}
u(R,t) = 0,  t\in (0,T],$$

$$\tag{75}
u(r,0) = I(r),  r\in [0,R].$$

The condition (73) is a necessary symmetry condition at $r=0$, while (74) could be any **Dirichlet
or Neumann condition** (or Robin condition in case of cooling or heating).

The finite difference approximation at $r=0$ of the spatial derivative term is the only new challenge in this problem. Let us in case of constant ${\alpha}$ expand the derivative to

$$\frac{\partial^2 u}{\partial r^2} + \frac{1}{r}\frac{\partial u}{\partial r}{\thinspace .}$$

The last term faces a difficulty at $r=0$ since it becomes a $0/0$ expression
because of the symmetry condition.
L’Hosptial’s rule can be used:

$$\lim_{r\rightarrow 0} \frac{1}{r}\frac{\partial u}{\partial r}
= \lim_{r\rightarrow 0} \frac{\partial^2 u}{\partial r^2}{\thinspace .}$$

The PDE at $r=0$ therefore becomes

$$\tag{76}
\frac{\partial u}{\partial t} = 2{\alpha}\frac{\partial^2 u}{\partial r^2}
    + f(t){\thinspace .}$$

For a variable coefficient ${\alpha}(r)$ the expanded derivative reads

$${\alpha}(r)\frac{\partial^2 u}{\partial r^2} +
\frac{1}{r}({\alpha}(r) + r{\alpha}'(r))\frac{\partial u}{\partial r}{\thinspace .}$$

We have that the limit of a product is

$$\lim_{r\rightarrow 0}
\frac{1}{r}({\alpha}(r) + r{\alpha}'(r))\frac{\partial u}{\partial r} =
\lim_{r\rightarrow 0} ({\alpha}(r) + r{\alpha}'(r))\
\lim_{x\rightarrow c}
\frac{1}{r}\frac{\partial u}{\partial r}{\thinspace .}$$

The second limit becomes as above, so the PDE at $r=0$,
assuming $({\alpha}(0) + r{\alpha}'(0))\neq 0$,
looks like

$$\tag{77}
\frac{\partial u}{\partial t} = (2{\alpha} + r{\alpha}')
    \frac{\partial^2 u}{\partial r^2}
    + f(t){\thinspace .}$$

The second-order derivative is discretized in the usual way.
Consider first constant ${\alpha}$:

$$2{\alpha}\frac{\partial^2}{\partial r^2}u(r_0,t_n) \approx
[2{\alpha} 2D_rD_r u]^n_0 =
2{\alpha} \frac{u^{n}_{1} - 2u^{n}_0 + u^n_{-1}}{\Delta r^2}{\thinspace .}$$

The fictitious value $u^n_{-1}$ can be eliminated using the discrete
symmetry condition

$$[D_{2r} u =0]^n_0 \quad\Rightarrow\quad u^n_{-1} = u^n_1,$$

which then gives the modified approximation to the second-order derivative
of $u$ in $r$ at $r=0$:

$$\tag{78}
4{\alpha} \frac{u^{n}_{1} - u^{n}_0}{\Delta r^2}{\thinspace .}$$

With variable ${\alpha}$ we simply get

$$(2{\alpha} + r{\alpha}')2D_rD_r u]^n_0 =
(2{\alpha}(0) + r{\alpha}'(0)) \frac{u^{n}_{1} - 2u^{n}_0 + u^n_{-1}}{\Delta r^2}{\thinspace .}$$

The discretization of the second-order derivative in $r$ at another
internal mesh point is straightforward:

$$\left.\frac{1}{r}\frac{\partial}{\partial r}
\left(r{\alpha}\frac{\partial u}{\partial r}\right)\right\vert_{r=r_i}^{t=t_n}
\approx [r^{-1} D_r (r {\alpha} D_r u)]_i^n
= \frac{1}{\Delta r^2}\left(
r_{i+\frac{1}{2}}{\alpha}_{i+\frac{1}{2}}(u_{i+1}^n - u_i^n) - r_{i-\frac{1}{2}}{\alpha}_{i-\frac{1}{2}}(u_{i}^n - u_{i-1}^n)\right){\thinspace .}$$

$\theta$-rule in time...

## Diffusion equation in spherically-symmetric geometries¶

### Discretization in spherical coordinates¶

Let us now pose the problem from the section Diffusion equation in axi-symmetric geometries
in spherical coordinates, where $u$ only depends on the radial coordinate
$r$ and time $t$. That is, we have spherical symmetry.
For simplicity we restrict the diffusion coefficient ${\alpha}$ to be
a constant. The PDE reads

$$\tag{79}
\frac{\partial u}{\partial t} = \frac{{\alpha}}{r^\gamma}\frac{\partial}{\partial r}
    \left(r^\gamma\frac{\partial u}{\partial r}\right) + f(t),$$

for $r\in (0,R)$ and $t\in (0,T]$. The parameter $\gamma$ is 2 for
spherically-symmetric problems and 1 for axi-symmetric problems.
The boundary and initial conditions
have the same mathematical form as
in (72)-(75).

Since the PDE in spherical coordinates has the same form as the PDE
in the section Diffusion equation in axi-symmetric geometries, just with the $\gamma$ parameter
being different, we can use the same discretization approach.
At the origin $r=0$ we get problems with the term

$$\frac{\gamma}{r}\frac{\partial u}{\partial t},$$

but L’Hosptial’s rule shows that this term equals $\gamma\partial^2 u/
\partial r^2$, and the PDE at $r=0$ becomes

$$\tag{80}
\frac{\partial u}{\partial t} = (\gamma+1){\alpha}\frac{\partial^2 u}{\partial r^2}
    + f(t){\thinspace .}$$

Same discretization, write up with $\gamma$.

#### Discretization in Cartesian coordinates¶

The spherically-symmetric spatial derivative can be transformed to
the Cartesian counterpart by introducing

$$v(r,t) = ru(r,t){\thinspace .}$$

Inserting $u=v/r$ in the PDE yields

$$\frac{1}{r^2}\frac{\partial}{\partial r}
\left({\alpha}(r)r^2\frac{\partial u}{\partial t}\right),$$

and then

$$r\left(\frac{d c^2}{dr}\frac{\partial v}{\partial r} +
{\alpha}\frac{\partial^2 v}{\partial r^2}\right) - \frac{d c^2}{dr}v
{\thinspace .}$$

The two terms in the parenthesis can be combined to

$$r\frac{\partial}{\partial r}\left( {\alpha}\frac{\partial v}{\partial r}\right),$$

which is recognized as the variable-coefficient Laplace operator in
one Cartesian coordinate.

```python
import numpy as np

Tb=373
T0=293

R=0.05
T=1200
Nx=100
Nt=5000
alpha=0.0000004

```

```python
def SetInitialCondition(Nx,Tb,T0):
    #definition of the initial temperature (smooth)
    u0=np.ones(Nx+1,)*T0
    u0[Nx]=Tb
    return u0

r   = np.linspace(0, R, Nx+1)    # mesh points in space
dr  = r[1]-r[0]
t   = np.linspace(0, T, Nt+1)    # mesh points in time
dt  = t[1] - t[0]

F=dt*alpha/(dr**2)
print('Frechet number: %f' % F)

u   = np.zeros((Nx+1,))          # unknown u at new time level
u_1 = np.zeros((Nx+1,))          # u at the previous time level

```

    Frechet number: 0.384000

$$\frac{u^{n+1}_i-u^n_i}{\Delta t} = \frac{1}{\Delta r^2}\left(
r_{i+\frac{1}{2}}{\alpha}_{i+\frac{1}{2}}(u_{i+1}^n - u_i^n) - r_{i-\frac{1}{2}}{\alpha}_{i-\frac{1}{2}}(u_{i}^n - u_{i-1}^n)\right){\thinspace .}$$

```python
# Set initial condition u(x,0) = I(x)
u_1=SetInitialCondition(Nx,Tb,T0)
plt.figure(figsize=(12,8))
for n in range(0, Nt):

    # Compute u at inner mesh points
    for i in range(1, Nx):
        if i==0:
            u[i] = u_1[i]+6*F*(u_1[i+1]-u_1[i])
        else:
            a1=1/r[i]*(r[i+1]*u_1[i+1]-2*r[i]*u_1[i]+r[i-1]*u_1[i-1])
            u[i] = u_1[i] +  F*a1

    # Insert boundary conditions
    u[Nx] = Tb

    # Update u_1 before next step
    u_1[:]= u

    if n%400==0 or n==0:
        plt.plot(r[1:],u[1:],label=('t=%i s' % t[n]))
    plt.grid(True)
    plt.legend()

```

# Variable conductivity, density or specific heat capacity

```python
alpha=0.0000001
a=alpha*(1+0.02*np.sin(r/R))
A=dt*a/(dr**2)
print('Frechet number: %f' % A[0])

u   = np.zeros((Nx+1,))          # unknown u at new time level
u_1 = np.zeros((Nx+1,))          # u at the previous time level

plt.figure()
plt.plot(r,A);
```

```python
# Set initial condition u(x,0) = I(x)
u_1=SetInitialCondition(Nx,Tb,T0)
plt.figure(figsize=(12,8))
for n in range(0, Nt):

    # Compute u at inner mesh points
    for i in range(1, Nx):
        if i==0:
            u[i] = u_1[i]+6*A[0]*(u_1[i+1]-u_1[i])
        else:
            a1=A[i]/r[i]*(r[i+1]*u_1[i+1]-2*r[i]*u_1[i]+r[i-1]*u_1[i-1])
            a2=A[i+1]/4*(u_1[i+1] + u_1[i-1])
            a3=A[i-1]/4*(u_1[i-1] - u_1[i+1])
            u[i] = u_1[i] + a1 + a2 + a3

    # Insert boundary conditions
    #u[0] = 0;
    u[Nx] = Tb

    # Update u_1 before next step
    u_1[:]= u

    if n%400==0 or n==0:
        plt.plot(r[1:],u[1:],label=('t=%i s' % t[n]))
    plt.grid(True)
    plt.legend()

```

## Appendix: Backward Euler scheme (implicit)¶

We now apply a backward difference in time in (5),
but the same central difference in space:

$$\tag{9}
[D_t^- u = D_xD_x u]^n_i,$$

which written out reads

$$\tag{10}
\frac{u^{n}_i-u^{n-1}_i}{\Delta t} = {\alpha}\frac{u^{n}_{i+1} - 2u^n_i + u^n_{i-1}}{\Delta x^2} {\thinspace .}$$

Now we assume $u^{n-1}_i$ is computed, but all quantities at the “new”
time level $n$ are unknown. This time it is not possible to solve
with respect to $u_i^{n}$ because this value couples to its neighbors
in space, $u^n_{i-1}$ and $u^n_{i+1}$, which are also unknown.
Let us examine this fact for the case when $N_x=3$. Equation (10) written for $i=1,\ldots,Nx-1= 1,2$ becomes

$$\tag{11}
\frac{u^{n}_1-u^{n-1}_1}{\Delta t} = {\alpha}\frac{u^{n}_{2} - 2u^n_1 + u^n_{0}}{\Delta x^2}$$

$$\tag{12}
\frac{u^{n}_2-u^{n-1}_2}{\Delta t} = {\alpha}\frac{u^{n}_{3} - 2u^n_2 + u^n_{1}}{\Delta x^2}$$

The boundary values $u^n_0$ and $u^n_3$ are known as zero. Collecting the
unknown new values $u^n_1$ and $u^n_2$ on the left-hand side gives

$$\tag{13}
\left(1+  2F\right) u^{n}_1 - F u^{n}_{2}    = u^{n-1}_1,$$

$$\tag{14}
- F u^{n}_{1} + \left(1+  2F\right) u^{n}_2  = u^{n-1}_2{\thinspace .}$$

This is a coupled $2\times 2$ system of algebraic equations for
the unknowns $u^n_1$ and $u^n_2$. The equivalent matrix form is

$$\begin{split}\left(\begin{array}{cc}
1+  2F &   - F\\
- F    & 1+  2F
\end{array}\right)
\left(\begin{array}{c}
u^{n}_1\\
u^{n}_{2}\\
\end{array}\right)
=
\left(\begin{array}{c}
u^{n-1}_1\\
u^{n-1}_2
\end{array}\right)\end{split}$$

## Implicit vs. explicit methods

- Discretization methods that lead to a coupled system of equations for the unknown function at a new time level are said to be **implicit methods**.
- The counterpart, **explicit methods**, refers to discretization methods where there is a simple explicit formula for the values of the unknown function at each of the spatial mesh points at the new
time level.
- From an implementational point of view, implicit methods are more comprehensive to code since they require the solution of coupled equations, i.e., a matrix system, at each time level.

In the general case, (10) gives rise to a coupled $(N_x-1)\times (N_x-1)$ system of algebraic equations for all the unknown $u^n_i$ at the interior spatial points $i=1,\ldots,Nx-1$.
Collecting the unknowns on the left-hand side, (10) can be written

$$\tag{15}
- F u^n_{i-1} + \left(1+  2F \right) u^{n}_i - F u^n_{i+1} =
    u_{i-1}^{n-1},$$

for $i=1,\ldots,Nx-1$. Here,
we have introduced the mesh Fourier number

$$\tag{16}
F = {\alpha}\frac{\Delta t}{\Delta x^2}{\thinspace .}$$

One can either view these equations as a system for where the
$u^{n}_i$ values at the internal mesh points, $i=1,\ldots,N_x-1$, are
unknown, or we may append the boundary values $u^n_0$ and $u^n_{N_x}$
to the system. In the latter case, all $u^n_i$ for $i=0,\ldots,N_x$
are unknown and we must add the boundary equations to
the $N_x-1$ equations in (15):

$$\tag{17}
u_0^n = 0,$$

$$\tag{18}
u_{N_x}^n = 0{\thinspace .}$$

.

A coupled system of algebraic equations can be written on matrix form,
and this is important if we want to call up ready-made software for
solving the system.  The equations (15)
and (17)–(18)
correspond to the matrix equation

$$AU = b$$

where $U=(u^n_0,\ldots,u^n_{N_x})$, and
the matrix $A$ has the following structure:

$$\begin{equation}\tag{19}
A =
    \left(
    \begin{array}{cccccccccc}
    A_{0,0} & A_{0,1} & 0
    &\cdots &
    \cdots & \cdots & \cdots &
    \cdots & 0 \\
    A_{1,0} & A_{1,1} & 0 & \ddots &   & &  & &  \vdots \\
    0 & A_{2,1} & A_{2,2} & A_{2,3} &
    \ddots & &  &  & \vdots \\
    \vdots & \ddots &  & \ddots & \ddots & 0 &  & & \vdots \\
    \vdots &  & \ddots & \ddots & \ddots & \ddots & \ddots & & \vdots \\
    \vdots & &  & 0 & A_{i,i-1} & A_{i,i} & A_{i,i+1} & \ddots & \vdots \\
    \vdots & & &  & \ddots & \ddots & \ddots &\ddots  & 0 \\
    \vdots & & & &  &\ddots  & \ddots &\ddots  & A_{N_x-1,N_x} \\
    0 &\cdots & \cdots &\cdots & \cdots & \cdots  & 0 & A_{N_x,N_x-1} & A_{N_x,N_x}
    \end{array}
    \right)\end{equation}$$

The nonzero elements are given by

$$\tag{20}
A_{i,i-1} = -F$$

$$\tag{21}
A_{i,i} = 1+ 2F$$

$$\tag{22}
A_{i,i+1} = -F$$

for the equations for internal points, $i=1,\ldots,N_x-1$

The equations for the boundary points correspond to

$$\tag{23}
A_{0,0} = 1,$$

$$\tag{24}
A_{0,1} = 0,$$

$$\tag{25}
A_{N_x,N_x-1} = 0,$$

$$\tag{26}
A_{N_x,N_x} = 1{\thinspace .}$$

The right-hand side $b$ is written as

$$\begin{equation}\tag{27}
b = \left(\begin{array}{c}
    b_0\\
    b_1\\
    \vdots\\
    b_i\\
    \vdots\\
    b_{N_x}
    \end{array}\right)\end{equation}$$

with

$$\tag{28}
b_0 = 0,$$

$$\tag{29}
b_i = u^{n-1}_i,\quad i=1,\ldots,N_x-1,$$

$$\tag{30}
b_{N_x} = 0 {\thinspace .}$$

We observe that the matrix $A$ contains quantities that do not change
in time. Therefore, $A$ can be formed once and for all before we enter
the recursive formulas for the time evolution.
The right-hand side $b$, however, must be updated at each time step.
This leads to the following computational algorithm, here sketched
with Python code:
x = linspace(0, L, Nx+1)   # mesh points in space
dx = x[1] - x[0]
t = linspace(0, T, N+1)    # mesh points in time
u   = zeros(Nx+1)          # unknown u at new time level
u_1 = zeros(Nx+1)          # u at the previous time level

# Data structures for the linear system
A = zeros((Nx+1, Nx+1))
b = zeros(Nx+1)

for i in range(1, Nx):
    A[i,i-1] = -F
    A[i,i+1] = -F
    A[i,i] = 1 + 2*F
A[0,0] = A[Nx,Nx] = 1

# Set initial condition u(x,0) = I(x)
for i in range(0, Nx+1):
    u_1[i] = I(x[i])

import scipy.linalg

for n in range(0, Nt):
    # Compute b and solve linear system
    for i in range(1, Nx):
        b[i] = -u_1[i]
    b[0] = b[Nx] = 0
    u[:] = scipy.linalg.solve(A, b)

    # Update u_1 before next step
    u_1[:] = u


# Linear Elastic Fracture Mechanics

In the notes, we used complex functions to develop expressions for stress and displacement in the vicinity of an opening (tensile) crack.

For a general Westergaard stress function, $\psi$, we derived the stresses

\begin{equation}
\sigma_x = \text{Re}\psi -y\text{Im}\psi',\quad\quad \sigma_y = \text{Re}\psi + y\text{Im}\psi',\quad\quad \tau_{xy}=-y\text{Re}\psi'.
\end{equation}

For the case of biaxial tension, we showed that forms for $\psi$, $\psi'$ and $\bar{\psi}$ consistent with the boundary conditions are

\begin{equation}
\psi = \frac{\sigma_\infty}{\sqrt{1-a^2/z^2}}, \quad\quad \psi'=-\frac{\sigma_\infty a^2}{z^3(1-a^2/z^2)^{3/2}}, \quad\quad \bar{\psi}=\sigma_\infty z\sqrt{1-a^2/z^2}
\end{equation}

***These functions are implemented in cell below.***

```python
# run this cell by highlighting it and pressing Ctrl+Enter
%matplotlib inline
import numpy as np

# complex functions
def ipsi(z,a,s_inf):
    return s_inf*np.sqrt(z**2-a**2)
def psi(z,a,s_inf):
    return s_inf/np.sqrt(1-a**2/z**2)
def dpsi(z,a,s_inf):
    return -s_inf*a**2/((1-a**2/z**2)**1.5)/z**3

# stresses
def sx(z,a,s_inf):
    return np.real(psi(z,a,s_inf))-np.imag(z)*np.imag(dpsi(z,a,s_inf))
def sy(z,a,s_inf):
    return np.real(psi(z,a,s_inf))+np.imag(z)*np.imag(dpsi(z,a,s_inf))
def sxy(z,a,s_inf):
    return -np.imag(z)*np.real(dpsi(z,a,s_inf))

def u(z,par): return 2*np.real(ipsi(z,par))-np.imag(z)/(1-nu)*np.imag(psi(z,par))
def v(z,par): return 2*np.imag(ipsi(z,par))-np.imag(z)/(1-nu)*np.real(psi(z,par))
```

## Stress state around the crack

***Execute the cell below to plot contours of the three stresses around the open crack.***

- **Does the stress state appear consistent with the boundary conditions?**

```python
from matplotlib import pyplot as plt
import matplotlib.cm as cm
f,axs = plt.subplots(1,3)
f.set_size_inches(15,4)

# define parameter and plotting grid
a = 1                # crack length
s_inf = 1            # far field stress
x = np.linspace(-5*a,5*a,1000)    # plot grid
X,Y = np.meshgrid(x,x)
Z = X+Y*1j
cmw = cm.get_cmap('seismic')
lvls = np.linspace(-3*s_inf, 3*s_inf, 11)

for ax,s in zip(axs,[sx,sy,sxy]):
    CS = ax.contourf(X,Y,s(Z,a,s_inf),levels=lvls,cmap=cmw)
    plt.colorbar(CS,ax=ax)
    ax.set_xlim([0,2*a])
    ax.set_ylim([-a,a])
    ax.plot([-a,a],[0,0],'k-', lw=2)    # plot the fracture as a thick line
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_title(s.__name__)

plt.show()
```

## Stresses near the crack tip

By transforming to a polar coordinate system centred at the crack tip, we approximate the crack-tip stresses in the limit $|r|<<a$

\begin{equation}
\sigma_{ij} = \frac{K_I}{\sqrt{2\pi r}}f_{ij}(\theta), \quad\quad f_{ij} = \begin{cases}
      \cos\frac{\theta}{2}\left(1-\sin\frac{\theta}{2}\sin\frac{3\theta}{2}\right) & ij=xx \\
      \cos\frac{\theta}{2}\left(1+\sin\frac{\theta}{2}\sin\frac{3\theta}{2}\right) & ij=yy \\
      \cos\frac{\theta}{2}\sin\frac{\theta}{2}\cos\frac{3\theta}{2} & ij=xy
\end{cases}, \quad\quad K_I = \sigma_\infty \sqrt{\pi a}
\end{equation}

***Execute the cell below to plot the approximated crack-tip stresses alongside the true solution.***

```python
from ipywidgets import interact

def fxx(theta): return np.cos(theta/2.)*(1-np.sin(theta/2.)*np.sin(3*theta/2.))
def fyy(theta): return np.cos(theta/2.)*(1+np.sin(theta/2.)*np.sin(3*theta/2.))
def fxy(theta): return np.cos(theta/2.)*np.sin(theta/2.)*np.cos(3*theta/2.)

# define parameters
a = 1
s_inf = 1
KI = s_inf*np.sqrt(np.pi*a)

def plot_stress(theta=0):
    f,axs = plt.subplots(1,3)
    f.set_size_inches(15,4)
    # polar coords
    r = np.linspace(0.01*a,a,101)
    theta = theta/180*np.pi      # convert to radians
    for ax,fij,sij in zip(axs,[fxx,fyy,fxy],[sx,sy,sxy]):
        # plot the approximation
        s_approx = KI*fij(theta)/np.sqrt(2*np.pi*r)
        ax.plot(r, s_approx,'b-',label='approx.')
        # plot the true solution
        z = a+r*np.cos(theta)+1j*r*np.sin(theta)
        s_true = sij(z,a,s_inf)
        ax.plot(r, s_true,'r--',label='full soln.')
        # label plot
        ax.set_xlabel('r')
        ax.set_ylabel('s_'+fij.__name__[1:])
        ax.set_ylim([0,5])
    ax.set_ylim([-2.5,2.5])
    ax.legend()
    plt.show()

interact(plot_stress, theta = (0,180,10))
```

## Crack displacements

We obtained the crack-normal displacement, $v$, through integration of the $y$ strain.

\begin{equation}
v = \frac{1-\nu^2}{E}\left(2\text{Im}\bar{\psi} - \frac{1}{1-\nu}y\text{Re}\psi\right)
\end{equation}

***Execute the cell below to plot the crack displacement field.***

```python
%matplotlib inline
from ipywidgets import widgets, interact
# displacement
def v(z,a,s_inf,nu,E):
    return (1-nu**2)/E*(2*np.imag(ipsi(z,a,s_inf))-np.imag(z)/(1-nu)*np.real(psi(z,a,s_inf)))

def plot_displacement(logE):
    f,(ax1,ax2)=plt.subplots(1,2)
    f.set_size_inches([20,8])

    # plot contours of displacement
    nu = 0.25
    E = 10**logE
    lvls = np.linspace(-2,2,11)
    CS = ax1.contourf(X,Y,v(Z,a,s_inf,nu,E),levels=lvls,cmap=cmw,extend='both')
    plt.colorbar(CS,ax=ax1)
    ax1.set_xlim([0,2*a])
    ax1.set_ylim([-a,a])
    ax1.plot([-a,a],[0,0],'k-', lw=2)    # plot the fracture as a thick line
    ax1.set_xlabel('x')
    ax1.set_ylabel('y')
    ax1.set_title('crack-normal displacment')

    # plot crack opening
    x = np.linspace(0.001*a,0.9999*a,1001)
    z = x + 1j*0*x
    ax2.plot(x, v(z,a,s_inf,nu,E),'b-')
    ax2.set_xlabel('x')
    ax2.set_ylabel('v')
    ax2.set_ylim([0,6])
    ax2.set_title('Youngs modulus = {:3.2f}GPa'.format(E), size=14)
    plt.show()

interact(plot_displacement, logE = (-1,1,0.2))
```

## Tasks

***Modify the functions above to implement stress and displacement solutions for a shear crack.***


### Simple example for performing symbolic regression for a set of points

```python
from nesymres.architectures.model import Model
from nesymres.utils import load_metadata_hdf5
from nesymres.dclasses import FitParams, NNEquation, BFGSParams
from pathlib import Path
from functools import partial
import torch
from sympy import lambdify
import json
```

```python
## Load equation configuration and architecture configuration
import omegaconf
with open('100M/eq_setting.json', 'r') as json_file:
  eq_setting = json.load(json_file)

cfg = omegaconf.OmegaConf.load("100M/config.yaml")
```

```python
## Set up BFGS load rom the hydra config yaml
bfgs = BFGSParams(
        activated= cfg.inference.bfgs.activated,
        n_restarts=cfg.inference.bfgs.n_restarts,
        add_coefficients_if_not_existing=cfg.inference.bfgs.add_coefficients_if_not_existing,
        normalization_o=cfg.inference.bfgs.normalization_o,
        idx_remove=cfg.inference.bfgs.idx_remove,
        normalization_type=cfg.inference.bfgs.normalization_type,
        stop_time=cfg.inference.bfgs.stop_time,
    )

```

```python
params_fit = FitParams(word2id=eq_setting["word2id"],
                            id2word={int(k): v for k,v in eq_setting["id2word"].items()},
                            una_ops=eq_setting["una_ops"],
                            bin_ops=eq_setting["bin_ops"],
                            total_variables=list(eq_setting["total_variables"]),
                            total_coefficients=list(eq_setting["total_coefficients"]),
                            rewrite_functions=list(eq_setting["rewrite_functions"]),
                            bfgs=bfgs,
                            beam_size=cfg.inference.beam_size #This parameter is a tradeoff between accuracy and fitting time
                            )
```

```python
weights_path = "../weights/100M.ckpt"
```

```python
## Load architecture, set into eval mode, and pass the config parameters
model = Model.load_from_checkpoint(weights_path, cfg=cfg.architecture)
model.eval()
if torch.cuda.is_available():
  model.cuda()
```

```python
fitfunc = partial(model.fitfunc,cfg_params=params_fit)
```

```python
# Create points from an equation
number_of_points = 500
n_variables = 1

#To get best results make sure that your support inside the max and mix support
max_supp = cfg.dataset_train.fun_support["max"]
min_supp = cfg.dataset_train.fun_support["min"]
X = torch.rand(number_of_points,len(list(eq_setting["total_variables"])))*(max_supp-min_supp)+min_supp
X[:,n_variables:] = 0
target_eq = "x_1*sin(x_1)" #Use x_1,x_2 and x_3 as independent variables
X_dict = {x:X[:,idx].cpu() for idx, x in enumerate(eq_setting["total_variables"])}
y = lambdify(",".join(eq_setting["total_variables"]), target_eq)(**X_dict)
```

```python
print("X shape: ", X.shape)
print("y shape: ", y.shape)
```

    X shape:  torch.Size([500, 3])
    y shape:  torch.Size([500])

```python
output = fitfunc(X,y)
```

    /local/home/lbiggio/NeuralSymbolicRegressionThatScales/src/nesymres/architectures/model.py:136: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      X = torch.tensor(X,device=self.device).unsqueeze(0)
    /local/home/lbiggio/NeuralSymbolicRegressionThatScales/src/nesymres/architectures/model.py:140: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      y = torch.tensor(y,device=self.device).unsqueeze(0)

    Memory footprint of the encoder: 4.096e-05GB

    Constructing BFGS loss...
    Flag idx remove ON, Removing indeces with high values...
    checking input values range...
    Loss constructed, starting new BFGS optmization...
    Constructing BFGS loss...
    Flag idx remove ON, Removing indeces with high values...
    checking input values range...
    Loss constructed, starting new BFGS optmization...

```python
output
```

    {'all_bfgs_preds': ['((x_1)*(sin(x_1)))', '((x_1)*((cos(x_1))*(tan(x_1))))'],
     'all_bfgs_loss': [0.0, 5.970244e-14],
     'best_bfgs_preds': ['((x_1)*(sin(x_1)))'],
     'best_bfgs_loss': [0.0]}

```python

```

```python

```

```python

```