<left></left>

# Lecture 15: Deep Learning

### Applied Machine Learning

__Volodymyr Kuleshov, Jin Sun__
Cornell Tech

# Part 1: What is Deep Learning?

Deep learning is a relatively new and powerful subfield of machine learning closely tied to neural networks.

Let's find out what deep learning is, and then we will see some deep learning algorithms.

# Review: Neural Network Layers

A neural network layer is a model $f : \mathbb{R}^d \to \mathbb{R}^p$ that applies $p$ neurons in parallel to an input $x$.
$$f(x) = \sigma(W\cdot x) = \begin{bmatrix}
\sigma(w_1^\top x) \\
\sigma(w_2^\top x) \\
\vdots \\
\sigma(w_p^\top x)
\end{bmatrix},
$$
where each $w_k$ is the vector of weights for the $k$-th neuron and $W_{kj} = (w_k)_j$. We refer to $p$ as the *size* of the layer.

# Review: Neural Networks

A neural network is a model $f : \mathbb{R} \to \mathbb{R}$ that consists of a composition of $L$ neural network layers:
$$ f(x) = f_L \circ f_{L-1} \circ \ldots f_1 (x). $$
The final layer $f_L$ has size one (assuming the neural net has one ouput); intermediary layers $f_l$ can have any number of neurons.

The notation $f \circ g(x)$ denotes the composition $f(g(x))$ of functions

We can visualize this graphically as follows.

# What is Deep Learning?

In a nutshell, deep learning is a modern evolution of the field of artificial neural networks that emphasizes:
* Very deep neural networks
* Large unstructured datasets, especially images, text, and audio
* Modern computational resources, like GPUs

# Expressivity of Deep Models

Why is deep learning powerful? One reason is deep neural networks can represent complex models very compactly.
* Shallow neural networks can represent any function, but need very large hidden layers.
* Deep networks can represent very complex $\mathcal{X} \to \mathcal{Y}$ mappings with fewer parameters.

In practice, deep neural networks can learn very complex mappings such as $\text{image} \to \text{text description}$ that other algorithms cannot.

# Representation Learning

How does a deep neural network use its representational power?
* The sequential layers can learn to represent data at an increasing level of abstraction
* This can also be interpreted as learning: each layer maps input to a more abstract feature space that is *learned* from data.

This is an example of representations that a deep neural network learns from data.

# Scaling to Large Datasets

Deep learning models also scale to very large datasets.

Classical algorithms like linear regression saturate after a certain dataset size. Deep learning models keep improving as we add more data.

# Computational Scalability

Deep learning datasets benefit from large datasets because they can easily use specialzied computational hardware.
* Neural networks mostly implement linear algebra, and benefit from hardware acceleration.
* Today, they are trained on graphical processing units (GPUs), or even more specialzied hardware, resulting in order of magnitude speedups.

# Connections to Neuroscience

As artificial neurons are designed to mimic the biological ones, deep neural networks have strong connections with biological neural systems.

One of the most interesting ones is the visual processing network in brains.

There is ample evidence that deep neural networks perform computation very simiilar to that of the visual cortex ([Kravitz el al](https://plos.figshare.com/articles/_Hierarchical_representations_along_the_ventral_visual_pathway_/811315).).

# Successes of Deep Learning

Deep learning has been a real breakthrough in machine learning over the last decade.
* It has dramatically improved common technologies like speech recognition or machine translation.
* It has enabled new results that have captured the popular imagination.

One really recent impressive application is image generation.

These faces are not real! See: https://thispersondoesnotexist.com/

# Pros and Cons of Deep Neural Networks

Deep neural nets (DNNs) are among the most powerful ML models.
* DNNs set a new state-of-the-art in many application areas.
* They can be combined with many algorithms (supervised, unsupervised), and significantly improve them.
* Many specialized software and hardware platforms for DNNs.

However, DNNs also can be slow and hard to train and require a lot of data.

* OpenAI GPT-2/3 is a language model that uses deep learning to generate text, as if they were written by human.

Try it out!
https://transformer.huggingface.co/doc/gpt2-large

GPT-3 is more powerful and able to generate structured text such as HTML/CSS, tables, Python programs, food recipes and much more.

https://github.com/elyase/awesome-gpt3

# Challenges of Deep Learning

Deep learning is powerful, but it can be challenging to apply it to real-world problems.

* Training with deep networks faces the vanishing gradients problem.

  Possible solutions will be to use normalization (BatchNorm and variants), skip connections (ResNet), etc.

* Deep neural networks are prone to overfitting.

  We need proper regularization, data augmentation, and close monitoring of the training process.

* Requires a massive amount of data

  Training from scratch with limited data is difficult. Consider transfer learning from pre-trained models.

Next we are going to look into a particular type of deep neural networks that is widely used for static data such as images: Convolutional Neural Networks (CNN).

But first we need to review the foundational operations of CNNs: Convolutions and Pooling.

<left></left>
# Part 2: Convolutions and Pooling

These basic operations are the building blocks of modern convolutional neural networks.

# Review: What is Deep Learning?

In a nutshell, deep learning is a modern evolution of the field of artificial neural networks that emphasizes:
* Very deep neural networks
* Large unstructured datasets, especially images, text, and audio
* Modern computational resources, like GPUs

# Definition of Convolution

Let $f \in \mathbb{R}^n$ and $g \in \mathbb{R}^m$ be two vectors, called the *filter* and the *signal* respectively. Typically, $n < m$.

In deep learning, a convolution $(f * g ) : \mathbb{Z} \to \mathbb{R}$ is typically defined as
\begin{align}
(f * g ) [p] &\triangleq \underbrace{\sum_{t=1}^{n} f[t] g[p+t]}_\text{dot product of $f$ with part of $g$}
\end{align}
where $g[t] = 0$ when $t \leq 0$ or $t>m$.

* We may think of the convolution $$\sum_{t=1}^{n} f[t] g[p+t]$$ as a "local" dot product of `f` with `g[p:p+n]` (the subsequence of $g$ that starts at position $p$).

* $(f * g ) [p]$ is large when $f$ is similar to $g$ at $p$ and small otherwise. It measures the extent to which $f$ is "found" in $g$.

This is best understood via a picture:

The green sequence `[1,0,-1]` is the filter. The signal $g$ is in gray, and the outputs of the convolution
$$(f * g)[p] = 1 \cdot g_p + 0 \cdot g_{p+1} -1 \cdot g_{p+2}$$ are in yellow.

On a small technical note, what we have defined is called the cross-correlation in mathematics.

The convolution is technically defined as $\sum_{t=1}^{n} f[t] g[p-t]$, but in deep learning both formulas effectively give the same results and the cross-correlation is used in practice (and is called "convolution").

We can implement a convolution in Python as follows.

```python
# A naive (*slow*) implementation of convolution with loops, don't use this in practice!
def conv_naive(f, g):
    M = len(f)
    N = len(g)
    result = [0]*N
    for n in range(N):
        for m in range(M):
            if n-m>=0 and n-m<N:
                result[n] += f[m]*g[n-m]
    return result[M-1:] #only return valid results

# numpy also provides convolution function np.convolve(f,g,'valid')
```

# Example: Edge Detection

To gain more intuition about convolution, let's look at another example in 1D.

We start by defining a filter and a signal as numpy arrays.

```python
import numpy as np

# create filter
f = np.array([-1.,1.])

# create signal
g = np.ones(30)
g[5:8] = 0
g[15:18] = 0

# create convolution
conv_result = conv_naive(f,g)
```

Here, the signal has "jumps" or "edges" between 0 and 1 at a few locations.

The filter equals $\pm1$ at these edges, and zero everywhere else. If $g$ was an audio signal, then $f$ would detect boundaries between silence.

```python
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [8, 4]

ax1 = plt.subplot(221)
ax1.stem(range(len(f)), f, use_line_collection=True)
ax1.set_title('Filter f')
ax1.set_xticks([0,1])
ax2 = plt.subplot(222)
ax2.stem(range(len(g)), g, use_line_collection=True)
ax2.set_title('Signal g')
ax3 = plt.subplot(212)
ax3.stem(range(len(conv_result)), conv_result, use_line_collection=True)
ax3.set_title('Convolution f * g', y=-0.5)

plt.show()
```

# Example: Smoothing

Another application of convolutions is to smooth the signal $g$.

We again define a filter and a signal as numpy arrays.

```python
# create signal
g = np.ones(30)
g[5:8] = 0
g[15:18] = 0

# create filter
f = np.sin(np.linspace(0.1,np.pi,10))
f /= f.sum()

# create convolution
conv_result = conv_naive(f,g)
```

The filter $f$ is a "mini bell curve". When applied to the same signal, it smoothes-out the "kinks".

```python
ax1 = plt.subplot(221)
ax1.stem(range(len(f)), f, use_line_collection=True)
ax1.set_title('f')
ax2 = plt.subplot(222)
ax2.stem(range(len(g)), g, use_line_collection=True)
ax2.set_title('g')
ax3 = plt.subplot(212)
ax3.stem(range(len(conv_result)), conv_result, use_line_collection=True)
ax3.set_title('f * g', y=-0.5)

plt.show()
```

# What Are Convolutions Doing?

The convolution applies the filter $f$ onto the signal $g$.
* This can change the behavior of $g$ (e.g., smoothing it out).
* Or it can extract useful characteristics from $g$ (e.g., edge detection, or when does the signal changes drastically).

We can use convolutions in machine learning models to *learn* the filters.

# Convolutions in 2D

Convolutions can be extended to 2D by expanding dimensions of $f$ and $g$.

In this example, $f$ is a 3x3 array (gray area). $g$ is a 5x5 array (blue area). The result of the convolution is the top green area.

In the context of neural networks, $f$ are learnable weights, $g$ are inputs from previous layers.
* The signal $g$ can be an image.
* The filter $f$ can be a "feature" of the image, like an edge or a more complex object.

The results of applying the convolution are fed to the next layer.

# Review: Neural Network Layers

A neural network layer is a model $f : \mathbb{R}^d \to \mathbb{R}^p$ that applies $p$ neurons ih parallel to an input $x$.
$$f(x) = \sigma(W\cdot x) = \begin{bmatrix}
\sigma(w_1^\top x) \\
\sigma(w_2^\top x) \\
\vdots \\
\sigma(w_p^\top x)
\end{bmatrix},
$$
where each $w_k$ is the vector of weights for the $k$-th neuron and $W_{kj} = (w_k)_j$. We refer to $p$ as the *size* of the layer.

# Convolutional Layers

A convolution layer is a model $f : \mathbb{R}^d \to \mathbb{R}^p$ that applies $p$ convolutions in parallel to an input $x$.
$$f(x) = \text{conv}(W, x) = \begin{bmatrix}
\text{conv}(w_1, x) \\
\text{conv}(w_2, x) \\
\vdots \\
\text{conv}(w_p, x) \\
\end{bmatrix},
$$
where each $w_k$ is the weights for the $k$-th convolution filter (think $f$ in our 1D example).

# Understanding 2D Convolutional Layers

Each convolution filter in the conv layer produces an output map called the activation map.

A different filter produces a different activation map.

Each filter produces its own activation map.

All activation maps in the same conv layer are concatenated together to form the final multi channel output:

The activation map contains useful information about the input image.

# Attributes of Convolutional Layers

The most important attributes of a convolutional layer are:
* **# input channels**: the number of feature maps from input.
* **# output channels**: the number of feature maps for output, this is also the number of convolutional filters in this layer.
* **kernel_size**: size of the convolution filters.

# Example: Edge Detection in 2D

Let's revisit our edge detection example, but this time the signal $g$ will be a 2D image.

We will use two filters, each of which is defined below. We will also load an image as the signal $g$.

```python
from scipy import misc

# define the filters
sobel_x = np.array([[1,0,-1],[2,0,-2],[1,0,-1]])
sobel_y = np.array([[1,2,1],[0,0,0],[-1,-2,-1]])

# load and visualize the "signal"
img = misc.ascent()
plt.imshow(img, cmap='gray')
```

We can now convolve and visualize the filters with the image.

The result is an activation map with high activation around horizontal and vertical edges -- parts of the image where it changes from light to dark.

```python
from scipy import signal

# create the convolution with each filter
grad_x = signal.convolve2d(img, sobel_x, boundary='symm', mode='same')
grad_y = signal.convolve2d(img, sobel_y, boundary='symm', mode='same')

# visualize the convolution
fig, (ax_orig, ax_x, ax_y) = plt.subplots(1, 3, figsize=(15, 15))
ax_orig.imshow(img, cmap='gray')
ax_orig.set_title('Original')
ax_orig.set_axis_off()
ax_x.imshow(np.absolute(grad_x), cmap='gray')
ax_x.set_title('Vertical Edges')
ax_x.set_axis_off()
ax_y.imshow(np.absolute(grad_y), cmap='gray')
ax_y.set_title('Horizontal Edges')
ax_y.set_axis_off()
```

# When to Use Convolutions?

Convolutions work best on inputs containing "attributes" that are interesting, but whose location in the input is not important.

Images are by nature good examples of this assumption. Convolution layers can learn similar functions as fully connected layers, but with much smaller parameters.

# Pooling Layers

A pooling layer is a model $f : \mathbb{R}^d \to \mathbb{R}^p$ that applies pooling operations to an input $x$.
$$f(x) = \text{pooling}(x)
$$
where $pooling$ is a pre-defined operation applied over the input $x$.

Pooling layer does not have learnable weights.

# Pooling

Pooling is a common operation to be added after a convolution layer. It is applied to each activation map separately and reduces the spatial size of its input.

**Max Pooling**: for each region in the input, return the max value. This is the most common type.

**Average Pooling**: for each region in the input, return the average value.

## Purpose of Pooling

* Allows slight translation invariance inside the pooling window: exact location of the relevant feature is not critical. This improves robustness of feature detection.

* When used together with convolutions, pooling reduces spatial size of feature maps to capture more abstract representations.

<left></left>
# Part 3: Convolutional Neural Networks

Convolutional neural networks use convolutional layers to process signals such as images, audio, and even text.

# Review: Convolution

Convolutions apply a filter $f$ (gray) over a signal $g$ (blue). The output is an activation map (green).

# Review: Pooling

Pooling is a common operation to be added after a convolution layer. It is applied to each activation map separately and reduces the spatial size of its input.

# Convolutional Neural Networks

A convolutional neural network (CNN) is a model $f : \mathbb{R} \to \mathbb{R}$ that consists of a composition of $L$ neural network layers that contain convolutions:
$$ f(x) = f_L \circ f_{L-1} \circ \ldots f_1 (x). $$

The final $f_L$ is often a fully connected output later of size one.

Typically, CNNs are made of consecutive convolution + activation + pooling layers that form into blocks.

Next we are going to see a few famous examples.

# LeNet for MNIST Digits Recognition

LeNet successfully used CNNs for digit recognition [LeCun, Bottou, Bengio, Haffner 1998].

* LeNet contains multiple convolutional and max-pooling (called subsampling in the original paper) layers and a fully connected MLP in the end.
* Given an image of a hand-written digit, LeNet predicts it to be one of the 10 classes (0-9).
* This work inspired many subsequent CNN architectures.

# AlexNet and ImageNet

ImageNet classification with deep convolutional neural networks [Krizhevsky, Sutskever, Hinton 2012]

* AlexNet contains multiple convolutional layers followed by max pooling. The prediction is 1000 dimentional because that is the number of classes in the ImageNet classification task.

* Note that in the original implementation the image is separated into two parts that are in two different GPUs due to hardware limits.

Starting from AlexNet, the performance of image classification on ImageNet has entered a new era:

http://sqlml.azurewebsites.net/2017/09/12/convolutional-neural-network/

Convolutional neural networks with deeper layers (and other tricks) have already exceeded **human performance** on such tasks.

# Algorithm: Convolutional Neural Network

* __Type__: Supervised learning (regression and classification).
* __Model family__: Compositions of convolutional, pooling and other types of layers.
* __Objective function__: Any differentiable objective.
* __Optimizer__: Gradient descent.

# CNNs and Feature Learning

Before neural networks, computer vision algorithms used hand-crafted features to extract information from raw image inputs.

Designing such features is hard and most importantly they might not be optimal for a particular task.

One set of features that used to be popular was the Scale-Invariant Feature Transform (SIFT).

https://www.vlfeat.org/overview/sift.html

Another common set of features was the the Histogram of Oriented Gradients (HOG).

https://sarthakahuja.org/public/docs/report_ped_detection.pdf

By using CNNs, we now can extract features from input images at different level of abstraction, without the need of designing hand-craft features.

# Visualizing CNN Internals

CNNs can be seen as extracting features at low-, mid-, and high-levels of abstraction that are relevant to corresponding visual concepts.

Below, we reproduce feature visualization from the paper Visualizing and Understanding Convolutional Networks [Zeiler, Fergus 2013].

For a convolutional network with

$$ \text{Input} \rightarrow \text{Conv Layer1} \rightarrow \text{Conv Layer2} \rightarrow \cdots \rightarrow \text{Conv Layer5} \rightarrow  \cdots \rightarrow \text{Output,}$$

we can visualize what each layer is doing by finding the image patch with highest activation responses.

Layer 1 focuses on colored edges and blobs.

Layers 2-3 focus on object parts, such as the wheels of a car, or the beak of a bird.

Layers 4-5 focus on object parts and even entire objects.

# Pros and Cons of CNNs:

CNNs are powerful tools because they encode visual information efficiently.
* CNNs are now used as universal visual feature extractors. It is common to use ImageNet pretrained CNNs.
* They serve as building blocks in ML pipelines and can work with other types of networks such as a recurrent network for complex tasks.

Their main drawbacks are computational and data requirements.


# Using generalized tracers and power spectra
This example showcases how to use the generalized tracers and 2D power spectra implemented in CCL

```python
import numpy as np
import pylab as plt
import pyccl as ccl
%matplotlib inline
```

## Preliminaries
Let's just begin by setting up a cosmology and a couple of redshift distributions and bias functions

```python
# Cosmology
cosmo = ccl.Cosmology(Omega_c=0.27, Omega_b=0.045, h=0.67, A_s=2.1e-9, n_s=0.96)

# Redshift-dependent functions
z = np.linspace(0,1.2,1024)
# Redshift distributions
nz1 = np.exp(-((z-0.5)/0.05)**2/2)
nz2 = np.exp(-((z-0.65)/0.05)**2/2)
# Bias
bz = 0.95/ccl.growth_factor(cosmo,1./(1+z))
# Magnification bias
sz = np.ones_like(z)
# Intrinsic alignment amplitude
az = -0.004 * np.ones_like(z)
```

## Standard tracers
There are a set of standard tracers with built-in constructors.
First, there is number counts tracers (a.k.a. galaxy clustering). These include the standard contribution from the matter overdensity multiplied by a linear, scale-independent bias, redshift-space distortions and magnification. You can turn off any of them.

```python
# This tracer will only include the density contribution
gc_d = ccl.NumberCountsTracer(cosmo, has_rsd=False, dndz=(z,nz1), bias=(z,bz), mag_bias=None)
# If you pass bias=None, then the tracer will not have a density term. E.g. this one is an RSD-only tracer.
gc_r = ccl.NumberCountsTracer(cosmo, has_rsd=True, dndz=(z,nz1), bias=None, mag_bias=None)
# Now a magnification-only tracer. To turn on magnification just pass a non-None mag_bias.
gc_m = ccl.NumberCountsTracer(cosmo, has_rsd=False, dndz=(z,nz1), bias=None, mag_bias=(z,sz))
# You can of course create a tracer that includes all of these terms
gc_a = ccl.NumberCountsTracer(cosmo, has_rsd=True, dndz=(z,nz1), bias=(z,bz), mag_bias=(z,sz))
```

There are also standard weak lensing tracers with two terms: the cosmic shear term due to lensing and intrinsic alignments within the L-NLA model with a scale-independent IA amplitude.

```python
# This tracer will only include the lensing shear contribution (by setting the ia_bias to None)
wl_s = ccl.WeakLensingTracer(cosmo, dndz=(z,nz2), has_shear=True, ia_bias=None)
# This tracer will only include IAs
wl_i = ccl.WeakLensingTracer(cosmo, dndz=(z,nz2), has_shear=False, ia_bias=(z,az))
# And you can have the full monty
wl_a = ccl.WeakLensingTracer(cosmo, dndz=(z,nz2), has_shear=True, ia_bias=(z,az))
```

Finally, we also have CMB lensing tracers, defined by a single source redshift.

```python
# We put the source at z=1100, but you can put it at any other redshift if you want to!
# It won't be CMB lensing though.
cmbl = ccl.CMBLensingTracer(cosmo, z_source=1100.)
```

## Custom tracers

In general, tracers are things you can cross-correlate and get a power spectrum from. In the most general case, the angular power spectrum between two tracers is given by

\begin{equation}
   C^{\alpha\beta}_\ell=\frac{2}{\pi}\int d\chi_1\,d\chi_2\,dk\,k^2 P_{\alpha\beta}(k,\chi_1,\chi_2)\,\Delta^\alpha_\ell(k,\chi_1)\,\Delta^\beta_\ell(k,\chi_2).
\end{equation},

where $P_{\alpha\beta}$ is a generalized power spectrum (see below), and $\Delta^\alpha_\ell(k,\chi)$ is a sum over different contributions associated to tracer $\alpha$, where every contribution takes the form:
\begin{equation}
  \Delta^\alpha_\ell(k,\chi)=f^\alpha_\ell\,W_\alpha(\chi)\,T_\alpha(k,\chi)\,j^{(n_\alpha)}_\ell(k\chi).
\end{equation}

Here, $f^\alpha_\ell$ is an **$\ell$-dependent prefactor**, usually associated with angular derivatives, $W_\alpha(\chi)$ is the **radial kernel**, dependent only on redshift/distance, $T_\alpha(k,\chi)$ is the **transfer function**, dependent on both $k$ and $z/\chi$, and $j^{(n)}_\ell(x)$ is a generalized versions of the **spherical Bessel functions**, associated with radial derivatives or inverse Laplacians.

Generalized tracers can be created as instances of the `Tracer` class. This is done in two steps:
  1. First, generate an empty tracer:
  2. Then, add a contribution to this tracer. A contribution is defined by the 4 parameters above:
   - The **$\ell$-dependent prefactor** is defined by the argument `der_angles`. This can be 0, 1 or 2. 0 means no prefactor, 1 means a prefactor $\ell(\ell+1)$ and 2 means a prefactor $\sqrt{(\ell+2)!/(\ell-2)!}$.
   - The form of the **spherical Bessel function** is determined by the argument `der_bessel`, which can be -1, 0, 1 or 2. For 0, 1 or 2, this determines the derivative of the Bessel function that this contribution uses. For `der_bessel=-1`, $j^{(-1)}_\ell(x)=j_\ell(x)/x$.
   - The **radial kernel** is determined by two arrays corresponding to $\chi$ and $W(\chi)$, wrapped into a tuple and passed as `kernel`. If `kernel=None`, CCL assumes that the radial kernel is 1 everywhere for this tracer.
   - The **transfer function** can be passed in two different forms. If your transfer function is factorizable, then use two parameters: `transfer_k` and `transfer_a`, corresponding to the $k$-dependent and redshift-dependent factors respectively. `transfer_k` should be a tuple of two arrays $(\ln(k),K(k))$, and `transfer_a` should be $(a,A(a))$, where $a$ is the scale factor (which should be monotonically increasing!). The total transfer function is then given by $T(a,k)=K(k) A(a)$. If your transfer function is not factorizable, then use the argument `transfer_ka`. This should be a tuple of three arrays $(a,\ln(k),T(a,k))$, where $a$ is an array of scale factors in ascending order, $\ln(k)$ is an array containing the logarithms of wavenumbers, and $T(a,k)$ is a 2D array containing the values of the transfer function at the corresponding values of $a$ and $k$. If $a$ and $\ln(k)$ have sizes $n_a$ and $n_k$ respectively, then $T(a,k)$ should have shape $(n_a,n_k)$. If `transfer_ka`, `transfer_a` or `transfer_k` are `None`, CCL assumes that the corresponding quantities are 1 everywhere.

Then, keep adding contributions until you're done.

Let's try this out by building a number-counts tracer that contains a linear bias, RSDs and magnification. We could do this easily with the built-in constructors above, but it's always satisfying when you build something with your own hands.

```python
# First, initialize empty tracer
gc_custom = ccl.Tracer()
```

```python
# Now we need to generate a radial kernel and a transfer function.
# To generate the radial kernel we can use a convenience function that exists in CCL:
kernel = ccl.get_density_kernel(cosmo, (z,nz1))
# Let's plot it out of curiosity:
plt.plot(kernel[0],kernel[1])
plt.xlabel('$\\chi\\,[{\\rm Mpc}]$',fontsize=14)
plt.ylabel('$W(\\chi)$',fontsize=14)
plt.show()
```

```python
# We also need a transfer function.
# Because this tracer has a simple linear bias, the transfer function
# is only z-dependent and not k-dependent. So it's factorizable.
# The scale factor values need to be increasing, so we'll reverse
# The order of all arrays:
sf = (1./(1+z))[::-1]
transfer_a = (sf,bz[::-1])
```

```python
# Now we're ready to add the density contribution to the tracer:
gc_custom.add_tracer(cosmo, kernel=kernel, transfer_a=transfer_a)
```

```python
# Now let's add RSDs.
# In this case the transfer function is just given by the growth rate.
# Also, RSDs use the second derivative of the Bessel function
transfer_a_rsd = (sf, -ccl.growth_rate(cosmo,sf))
gc_custom.add_tracer(cosmo, kernel=kernel, transfer_a=transfer_a_rsd, der_bessel=2)
```

```python
# And now magnification.
# Again, there is a built-in function to compute the radial kernel for
# lensing observables.
# For magnification, der_bessel=-1 and der_angles=1.
# The transfer function is just 1 everywhere
chis, w = ccl.get_lensing_kernel(cosmo, (z,nz1), mag_bias=(z,sz))
kernel_m = (chis, -2*w)
gc_custom.add_tracer(cosmo, kernel=kernel_m, der_bessel=-1, der_angles=1)
```

```python
# Now let's check that both tracers, the one we just
# created and the original one created using the built-in constructors
# give exactly the same result. To do that, let's compute their
# angular power spectra
ells = np.geomspace(2,1000,20)
cl_gc_a = ccl.angular_cl(cosmo, gc_a, gc_a, ells)
cl_gc_custom = ccl.angular_cl(cosmo, gc_custom, gc_custom, ells)

# Let's plot the result
plt.plot(ells, 1E4*cl_gc_a, 'r-', label='built-in tracer')
plt.plot(ells, 1E4*cl_gc_custom, 'k--', label='custom tracer')
plt.xscale('log')
plt.xlabel('$\\ell$', fontsize=14)
plt.ylabel('$10^4\\times C_\\ell$', fontsize=14)
plt.legend(loc='upper right', fontsize=12, frameon=False)
plt.show()
```

OK, but so far we haven't done anything that we couldn't have done with the built-in tracers. Let's have some fun!

How about adding a contribution to the custom tracer corresponding to the effect of non-Gaussianity. Non-Gaussianity generates a scale-dependent bias on biased tracers that becomes relevant on large scales. Using Eq. 9 of arXiv:0710.4560, the bias receives an additive term of the form:
\begin{equation}
 \Delta b = (b-1)f_{\rm NL}\frac{3\,\delta_c\,\Omega_M\,H_0^2}{D(a)\,k^2}.
\end{equation}
So we can include this effect as an additional contribution to the custom tracer that looks like a standard galaxy clustering term with this extra bias. Since the corresponding transfer function is still factorizable, this should be pretty easy:

```python
# Let's create an array of k values and the corresponding
# k-dependent part of the transfer function
#
#  make sure you have camb instelled using pip. Anaconda may install an older version.
#
ks = np.logspace(-5,2,512)
transfer_k_fnl = (np.log(ks),1./ks**2)

# Now the z-dependent part of the transfer function
fNL = 10
delta_c = 1.686
H0 = (cosmo.cosmo.params.h/ccl.physical_constants.CLIGHT_HMPC)
Omega_M = cosmo.cosmo.params.Omega_c+cosmo.cosmo.params.Omega_b
transfer_a_fnl = (sf, (bz[::-1]-1)*3*fNL*delta_c*Omega_M*H0**2/ccl.growth_factor(cosmo,sf))

# Now let's add that contribution
gc_custom.add_tracer(cosmo, kernel=kernel, transfer_a=transfer_a_fnl,
                     transfer_k=transfer_k_fnl)

# And let's check the effect in the power spectrum:
cl_gc_custom_fnl = ccl.angular_cl(cosmo, gc_custom, gc_custom, ells)

# Let's plot the result
plt.plot(ells, 1E4*cl_gc_custom, 'r-', label='$f_{\\rm NL}=0$')
plt.plot(ells, 1E4*cl_gc_custom_fnl, 'k--', label='$f_{\\rm NL}$=%d'%(int(fNL)))
plt.xscale('log')
plt.xlabel('$\\ell$', fontsize=14)
plt.ylabel('$10^4\\times C_\\ell$', fontsize=14)
plt.legend(loc='upper right', fontsize=12, frameon=False)
plt.show()
```

## Generalized power spectra

By default, when you call `ccl.angular_cl`, CCL will use the non-linear matter power spectrum in the equation for $C_\ell$ above to compute the angular power spectrum. You can however, pass a more general `Pk2D` object. These objects encapsulate generic power spectra that depend simultaneously on $k$ and $a$. They are versatile and can be created in different ways, but for our purposes here, we will generate them in the same way that we generated the tracer transfer functions (i.e. through arrays of scale factors and wavenumbers). For more information, check out the documentation for this class.

To illustrate the use of `Pk2D` objects, let's read off the matter power spectrum from CCL, include a high-$k$ Gaussian cutoff, and generate a `Pk2D` object from that.

```python
# OK, let's first read off the matter power spectrum:
lpk_array = np.log(np.array([ccl.nonlin_matter_power(cosmo,ks,a) for a in sf]))

# Now let's impose a cutoff at a scale k_cut corresponding to roughly ~10 Mpc
k_cut = 0.4
lpk_array -= (ks/k_cut)**2

# Create a Pk2D object
pk_cut = ccl.Pk2D(a_arr=sf, lk_arr=np.log(ks), pk_arr=lpk_array, is_logp=True)

```

OK, now let's check out the impact of the cutoff on the angular power spectrum.

```python
# Compute power spectra with and without cutoff
cl_gc_ncut = ccl.angular_cl(cosmo, gc_d, gc_d, ells)
cl_gc_ycut = ccl.angular_cl(cosmo, gc_d, gc_d, ells, p_of_k_a=pk_cut)

# Plot stuff
plt.plot(ells, 1E4*cl_gc_ncut, 'r-', label='No cut-off')
plt.plot(ells, 1E4*cl_gc_ycut, 'k--', label='Cut-off at $k_* = %.1lf\,{\\rm Mpc}^{-1}$'%(k_cut))
plt.xscale('log')
plt.yscale('log')
plt.xlabel('$\\ell$', fontsize=14)
plt.ylabel('$10^4\\times C_\\ell$', fontsize=14)
plt.legend(loc='lower left', fontsize=12, frameon=False)
plt.show()
```


As every year since 2015, I'm signed up to participate in the qualifications round of [Google Hashcode](https://hashcode.withgoogle.com), a programming competition for students and professionals from Europe, the Middle-East and Africa. While trying to prepare for the competition, I tackled the suggested training problem that aims at cutting pizza slices out of a large pizza.

In this post, I'll try to outline a solution to the problem based on one of the input files, the `medium.in` input. Along the way, I plan to report on the following topics:

- visualizing your data
- formulating the problem as a greedy optimization task
- knowing your data structures
- profiling your code
- using Pythran to optimize your code execution time

Let's get started.

# Visualizing your data

The large pizza we will use as input consists of Tomatoes as well as Mushrooms, which is encode in a grid like the following:

```
TTTTT
TMMMT
TTTTT
```

Let's load the data for our example grid:

```python
import numpy as np
```

```python
def read_input_pizza(filename):
    """Reads the input of a Pizza problem.

    returns:

    R: number of Rows of pizza grid
    C: number of Cols of pizza grid
    L: Lowest number of each ingredients per slice
    H: Highest number of cells per slice
    pizza: the pizza grid (1 == tomato, 0 == mushroom)
    """
    lines = open(filename).readlines()
    R, C, L, H = [int(val) for val in lines[0].split()]
    pizza = np.array([list(map(lambda item: 1 if item == 'T' else 0, row.strip())) for row in lines[1:]])
    return R, C, L, H, pizza
```

```python
R, C, L, H, pizza = read_input_pizza('files/medium.in')
```

Let's also plot the pizza:

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
```

```python
fig, ax = plt.subplots()
ax.imshow(pizza)
ax.set_title(f'medium.in shape is {pizza.shape}, max. score {pizza.size}\nmin. ingredients is {L}, max. pizza slice is {H}');
```

As you can read in the above plot, the pizza has a 200 by 250 shape, will need slices that have each at least 4 tomatoes and 4 mushrooms on it and a maximum size of 12 cells. Also, as far as I can tell from that plot, the pizza is made of evenly distributed ingredients.

Let's now try to come up with an algorithm.

# A greedy optimization approach

While brainstorming  this problem with a friend, we came up with a greedy approach for solving the problem. It can be described by the following Python pseudo-code:

```python
slices = []

while True:
    # randomly choose an available location for the start of the slice
    location = get_random_available_location(pizza)

    # select a random shape and try to fit it at the selected location
    slice_shape = get_random_shape(shapes)
    if satisfy_constraints(location, slice_shape, pizza):
        slices.append(slice(location, slice_shape))

        # since we sliced the pizza, we need to update the available cells
        update_pizza(pizza, this_slice)

    # should we exit while loop?
    if no_more_possible_slices() or iteration_limit_reached():
        break

```

The way the problem is described above has two properties:

- it is greedy (if the slice fits, we put it there)
- it is an optimization problem (since we loop until there's nothing left to do)

I believe this is generally a good way to solve the problems that come up at Hashcode competitions, thinking about past editions...

# A first version

With that being said, we can transform the pseudo-code into real code. Below is my first version.

Note that:

- the possible shapes are hard-coded
- we only try one of the possible shapes at the location and move on if it doesn't work

```python
import tqdm
from random import randint

def get_random_available_location(slice_mask):
    """Returns a randomly selected empty location on the slice mask."""
    nonzero_elements = np.nonzero(1 - slice_mask)
    n = len(nonzero_elements[0])
    index = randint(0, n - 1)
    return nonzero_elements[0][index], nonzero_elements[1][index]

def satisfy_constraints(location, shape, slice_mask, pizza, L, H):
    """Validates whether the given location and slice shape can be applied to the pizza."""
    r, c = location
    dr, dc = shape
    if slice_mask[r:r+dr, c:c+dc].size == dr * dc:
        if np.all(slice_mask[r:r+dr, c:c+dc] == 0):
            if dr * dc <= H:
                tomatoes = np.sum(pizza[r:r+dr, c:c+dc])
                mushrooms = dr * dc - tomatoes
                if tomatoes >= L and mushrooms >= L:
                    return True
    return False

def cut_slice(location, shape, current_slices, slice_mask):
    """Adds a given slice defined by location and shape to existing slices and
    updates the slice mask."""
    r, c = location
    dr, dc = shape
    slice_mask[r:r + dr, c:c + dc] = 1
    current_slices.append((r, c, dr, dc))

def score(pizza_slices):
    """Computes score of given pizza_slices list."""
    s = 0
    for pizza_slice in pizza_slices:
        s += pizza_slice[2] * pizza_slice[3]
    return s

def write_output_pizza(filename, pizza_slices):
    """Writes an output file with the required format."""
    with open(filename, 'w') as f:
        f.write(f"{len(pizza_slices)}\n")
        for slice in pizza_slices:
            r, c, dr, dc = slice
            f.write(f"{r} {c} {r+dr-1} {c+dc-1}\n")

def greedy1(fname):
    """Greedy approach to the pizza slice problem.
    We try one random shape at one random point each turn."""
    R, C, L, H, pizza = read_input_pizza(fname)
    possible_shapes = [(4, 2), (2, 4), (3, 3), (5, 2), (2, 5), (11, 1), (1,11), (2, 6), (6, 2), (3, 4), (4, 3)]
    slice_mask = np.zeros_like(pizza) # stores the sliced cells as 1 for easy lookup
    pizza_slices = [] # stores the slices we made
    for _ in tqdm.tqdm(range(20000)):
        location = get_random_available_location(slice_mask)
        shape = possible_shapes[randint(0, len(possible_shapes) - 1)]
        if satisfy_constraints(location, shape, slice_mask, pizza, L, H):
            cut_slice(location, shape, pizza_slices, slice_mask)
    print('\n')
    print(f"score: {score(pizza_slices)}")
    write_output_pizza(fname.split('.')[0]+'.out', pizza_slices)
```

```python
greedy1('files/medium.in')
```

    100%|██████████| 20000/20000 [00:19<00:00, 1026.98it/s]

    score: 28923

Let's take a look at the output from the algorithm.

```python
fig, ax
```

    (<matplotlib.figure.Figure at 0x119ff26a0>,
     <matplotlib.axes._subplots.AxesSubplot at 0x119ff2f28>)

```python
def show_output(fname):
    """Reads an output file, assuming it's valid and plots the result."""
    input_fname = fname.split('.')[0] + '.in'
    R, C, L, H, pizza = read_input_pizza(input_fname)
    lines = open(fname).readlines()
    N = int(lines[0].strip())
    slice_mask = np.zeros_like(pizza)
    pizza_slices = []
    for i in range(1, N+1):
        r, c, dr, dc = [int(val) for val in lines[i].strip().split()]
        dr += 1 - r
        dc += 1 - c
        slice_mask[r:r + dr, c:c + dc] = i
        pizza_slices.append([r, c, dr, dc])

    fig, axes = plt.subplots(figsize=(9, 4), ncols=2, sharex=True, sharey=True)
    axes[0].imshow(slice_mask)
    axes[0].set_title(f'coloring by slice number (1 to {slice_mask.max()})')

    axes[1].imshow((slice_mask > 0).astype(np.int))
    axes[1].set_title('coloring by empty (green) / occupied (yellow)')

    plt.suptitle(f'solution score: {score(pizza_slices)}')
    plt.tight_layout(rect=[0, 0, 1, .95])

    axes[1].axis((-.5, C-.5, R-.5, -.5))
```

```python
show_output('files/medium.out')
```

As we can see in the plot on the left, a random colored pattern emerges from our algorithm, since we try to slice at random patterns every time.

But how can we improve our existing solution? One possibility is to improve its performance. Although the algorithm runs in a couple of seconds for this grid size, it will be slow on the largest grid. So let's profile it to see what takes the most time.

```python
%load_ext line_profiler
```

```python
r = %lprun -r -f greedy1 greedy1('files/medium.in')
r.print_stats()
```

    100%|██████████| 20000/20000 [00:14<00:00, 1354.32it/s]

    score: 28839
    Timer unit: 1e-06 s

    Total time: 14.6957 s
    File: <ipython-input-6-0cd214f13b94>
    Function: greedy1 at line 47

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
        47                                           def greedy1(fname):
        48                                               """Greedy approach to the pizza slice problem.
        49                                               We try one random shape at one random point each turn."""
        50         1      26742.0  26742.0      0.2      R, C, L, H, pizza = read_input_pizza(fname)
        51         1          5.0      5.0      0.0      possible_shapes = [(4, 2), (2, 4), (3, 3), (5, 2), (2, 5), (11, 1), (1,11), (2, 6), (6, 2), (3, 4), (4, 3)]
        52         1        197.0    197.0      0.0      slice_mask = np.zeros_like(pizza) # stores the sliced cells as 1 for easy lookup
        53         1          2.0      2.0      0.0      pizza_slices = [] # stores the slices we made
        54     20001     289406.0     14.5      2.0      for _ in tqdm.tqdm(range(20000)):
        55     20000   13496188.0    674.8     91.8          location = get_random_available_location(slice_mask)
        56     20000     193251.0      9.7      1.3          shape = possible_shapes[randint(0, len(possible_shapes) - 1)]
        57     20000     649720.0     32.5      4.4          if satisfy_constraints(location, shape, slice_mask, pizza, L, H):
        58      2690      17756.0      6.6      0.1              cut_slice(location, shape, pizza_slices, slice_mask)
        59         1         88.0     88.0      0.0      print('\n')
        60         1       1273.0   1273.0      0.0      print(f"score: {score(pizza_slices)}")
        61         1      21035.0  21035.0      0.1      write_output_pizza(fname.split('.')[0]+'.out', pizza_slices)

I was very surprised at this point: the function that takes the most time is the one selecting one random empty locaiton on the grid! Why is that? Well, it turns out at each iteration we are going over 200x250 = 50000 points to check whether they are zero or one. This takes a lot of time. We could do better with different data structures:

- first, a set of available locations for fast lookup
- second, a list of available locations for fast random indexing

Also:

- let's programmatically generate all possible shapes for the given grid as a function of its minimum number of ingredients and maximum slice size
- and let's loop at a given location over all of these possible slice shapes before choosing the best one

With this in mind, here's an upgraded version of the previous function.

# An upgraded version

```python
import sympy
from random import shuffle

def generate_all_shapes(L, H):
    """Generates all possible tuples that satisfy the problem's shape constraints."""
    possible_shapes = []
    for size in range(2*L, H+1):
        factors = sympy.factorint(size)
        if len(factors) == 1:
            prime = list(factors.keys())[0]
            max_exp = list(factors.values())[0]
            for exp in range(0, max_exp+1):
                factor1 = prime ** exp
                factor2 = prime ** (max_exp - exp)
                possible_shapes.append((factor1, factor2))
        elif len(factors) == 2:
            prime1, prime2 = list(factors.keys())
            max_exp1, max_exp2 = list(factors.values())
            for exp1 in range(0, max_exp1+1):
                for exp2 in range(0, max_exp2+1):
                    factor1 = prime1 ** (max_exp1 - exp1) * prime2 ** (max_exp2 - exp2)
                    factor2 = prime1 ** (exp1) * prime2 ** (exp2)
                    possible_shapes.append((factor1, factor2))
        else:
            raise NotImplementedError
    return possible_shapes

def get_random_available_location_set2(random_locations, empty_cells):
    """Selects a random location that is still in empty_cells."""
    while len(random_locations) > 0:
        candidate = random_locations.pop()
        if candidate in empty_cells:
            return candidate
    else:
        return None

def update_empty_cells(location, shape, empty_cells):
    """Updates the empty cell dictionary."""
    r, c = location
    dr, dc = shape
    for rr in range(r, r+dr):
        for cc in range(c, c+dc):
            empty_cells.discard((rr, cc))

def greedy5(fname, iters=30000):
    """Greedy approach to pizza slicing.
    Fifth iteration: randomize the locations in which we try to add stuff."""
    R, C, L, H, pizza = read_input_pizza(fname)
    possible_shapes = generate_all_shapes(L, H)
    slice_mask = np.zeros_like(pizza)  # stores the sliced cells as 1 for easy lookup
    pizza_slices = []  # stores the slices we made
    empty_cells = set(tuple(args) for args in np.transpose(np.nonzero(1-slice_mask)).tolist())
    random_locations = list(x for x in empty_cells)
    shuffle(random_locations)
    current_score = 0
    for _ in tqdm.tqdm(range(iters)):
        location = get_random_available_location_set2(random_locations, empty_cells)
        if location is None:
            break
        selected_shapes = []
        for shape in possible_shapes:
            if satisfy_constraints(location, shape, slice_mask, pizza, L, H):
                selected_shapes.append(shape)
        if len(selected_shapes) > 0:
            current_score = score(pizza_slices)
            shape = max(selected_shapes, key=lambda shp: current_score + shp[0]*shp[1])
            cut_slice(location, shape, pizza_slices, slice_mask)
            update_empty_cells(location, shape, empty_cells)
    print('\n')
    print(f"score: {score(pizza_slices)}")
    write_output_pizza(fname.split('.')[0] + '.out', pizza_slices)
```

```python
greedy5('files/medium.in')
```

     51%|█████     | 15193/30000 [00:03<00:03, 4484.51it/s]

    score: 39850

Our score is greatly improved by the new algorithm, especially since we now try every possible solution. What does the pizza tiling now look like?

```python
show_output('files/medium.out')
```

This looks a lot more compact although it's not completely yellow. Also, it runs much faster now. How much faster?

```python
r = %lprun -r -f greedy5 greedy5('files/medium.in')
r.print_stats()
```

     53%|█████▎    | 15817/30000 [00:08<00:07, 1814.87it/s]

    score: 39889
    Timer unit: 1e-06 s

    Total time: 7.95831 s
    File: <ipython-input-13-6596edaa9978>
    Function: greedy5 at line 45

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
        45                                           def greedy5(fname, iters=30000):
        46                                               """Greedy approach to pizza slicing.
        47                                               Fifth iteration: randomize the locations in which we try to add stuff."""
        48         1      48290.0  48290.0      0.6      R, C, L, H, pizza = read_input_pizza(fname)
        49         1        294.0    294.0      0.0      possible_shapes = generate_all_shapes(L, H)
        50         1         80.0     80.0      0.0      slice_mask = np.zeros_like(pizza)  # stores the sliced cells as 1 for easy lookup
        51         1          3.0      3.0      0.0      pizza_slices = []  # stores the slices we made
        52         1     101737.0 101737.0      1.3      empty_cells = set(tuple(args) for args in np.transpose(np.nonzero(1-slice_mask)).tolist())
        53         1      16380.0  16380.0      0.2      random_locations = list(x for x in empty_cells)
        54         1     159669.0 159669.0      2.0      shuffle(random_locations)
        55         1          3.0      3.0      0.0      current_score = 0
        56     15909     162496.0     10.2      2.0      for _ in tqdm.tqdm(range(iters)):
        57     15909      85501.0      5.4      1.1          location = get_random_available_location_set2(random_locations, empty_cells)
        58     15909      20380.0      1.3      0.3          if location is None:
        59         1        241.0    241.0      0.0              break
        60     15908      21161.0      1.3      0.3          selected_shapes = []
        61    318160     433763.0      1.4      5.5          for shape in possible_shapes:
        62    302252    4161306.0     13.8     52.3              if satisfy_constraints(location, shape, slice_mask, pizza, L, H):
        63     17074      29113.0      1.7      0.4                  selected_shapes.append(shape)
        64     15908      24676.0      1.6      0.3          if len(selected_shapes) > 0:
        65      3472    2565112.0    738.8     32.2              current_score = score(pizza_slices)
        66      3472      25487.0      7.3      0.3              shape = max(selected_shapes, key=lambda shp: current_score + shp[0]*shp[1])
        67      3472      21099.0      6.1      0.3              cut_slice(location, shape, pizza_slices, slice_mask)
        68      3472      67069.0     19.3      0.8              update_empty_cells(location, shape, empty_cells)
        69         1         78.0     78.0      0.0      print('\n')
        70         1       1492.0   1492.0      0.0      print(f"score: {score(pizza_slices)}")
        71         1      12884.0  12884.0      0.2      write_output_pizza(fname.split('.')[0] + '.out', pizza_slices)

# Optimization using Pythran

As you can see in the above timing, the two most intensive functions are the `score` function and the `satisfy_constraints` function.

It turns out that with a little Pythran-fu, we can optimize these to run faster. [Pythran](http://serge-sans-paille.github.io/pythran-stories/pythran-tutorial.html) is a cool C++ compiler that knows how to handle Python and a lot of Numpy array stuff. By annotating the types we want, it can compile an optimized module from our functions.

Let's demonstrate this on the two above functions.

```python
pizza_slices = [(305, 223, 7, 2), (172, 21, 14, 1), (538, 255, 14, 1), (861, 247, 7, 2), (634, 248, 2, 7), (518, 828, 14, 1)]
```

```python
%load_ext pythran.magic
```

    WARNING: Disabling color, you really want to install colorlog.
    WARNING: Pythran support disabled for module: omp

```python
%%pythran
#pythran export score_pythran((int, int, int, int) list)
def score_pythran(pizza_slices):
    """Computes score of given pizza_slices list."""
    s = 0
    for pizza_slice in pizza_slices:
        s += pizza_slice[2] * pizza_slice[3]
    return s
```

```python
%timeit score(pizza_slices)
```

    723 ns ± 4.13 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

```python
%timeit score_pythran(pizza_slices)
```

    634 ns ± 11.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

We can do the same for the constraints call:

```python
%%pythran
#pythran export satisfy_constraints_pythran((int, int), (int, int), int[][], int[][], int, int)
def satisfy_constraints_pythran(location, shape, slice_mask, pizza, L, H):
    """Validates whether the given location and slice shape can be applied to the pizza."""
    r, c = location
    dr, dc = shape
    if slice_mask[r:r+dr, c:c+dc].size == dr * dc:
        if (slice_mask[r:r+dr, c:c+dc] == 0).all():
            if dr * dc <= H:
                tomatoes = pizza[r:r+dr, c:c+dc].sum()
                mushrooms = dr * dc - tomatoes
                if tomatoes >= L and mushrooms >= L:
                    return True
    return False
```

```python
%timeit satisfy_constraints((3, 4), (4, 1), np.ones((R, C), dtype=np.int), np.ones((R, C), dtype=np.int), L, H)
```

    63.7 µs ± 295 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
%timeit satisfy_constraints_pythran((3, 4), (4, 1), np.ones((R, C), dtype=np.int), np.ones((R, C), dtype=np.int), L, H)
```

    54.2 µs ± 248 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Equipped with this, we can now rewrite our algorithm like so:

```python
def greedy5_pythran(fname, iters=30000, disable_progress_bar=False):
    """Greedy approach to pizza slicing.
    Fifth iteration: randomize the locations in which we try to add stuff."""
    R, C, L, H, pizza = read_input_pizza(fname)
    possible_shapes = generate_all_shapes(L, H)
    slice_mask = np.zeros_like(pizza)  # stores the sliced cells as 1 for easy lookup
    pizza_slices = []  # stores the slices we made
    empty_cells = set(tuple(args) for args in np.transpose(np.nonzero(1-slice_mask)).tolist())
    random_locations = list(x for x in empty_cells)
    shuffle(random_locations)
    current_score = 0
    for _ in tqdm.tqdm(range(iters), disable=disable_progress_bar):
        location = get_random_available_location_set2(random_locations, empty_cells)
        if location is None:
            break
        selected_shapes = []
        for shape in possible_shapes:
            if satisfy_constraints_pythran(location, shape, slice_mask, pizza, L, H):
                selected_shapes.append(shape)
        if len(selected_shapes) > 0:
            current_score = score_pythran(pizza_slices)
            shape = max(selected_shapes, key=lambda shp: current_score + shp[0]*shp[1])
            cut_slice(location, shape, pizza_slices, slice_mask)
            update_empty_cells(location, shape, empty_cells)
    return score(pizza_slices)
```

```python
greedy5_pythran('files/medium.in')
```

     39%|███▉      | 11779/30000 [00:00<00:00, 29440.63it/s]

    39770

This now runs almost instantly. Let's check the profiling again:

```python
r = %lprun -r -f greedy5_pythran greedy5_pythran('files/medium.in', disable_progress_bar=True)
r.print_stats()
```

    Timer unit: 1e-06 s

    Total time: 1.71979 s
    File: <ipython-input-25-3e117007c84a>
    Function: greedy5_pythran at line 1

    Line #      Hits         Time  Per Hit   % Time  Line Contents
    ==============================================================
         1                                           def greedy5_pythran(fname, iters=30000, disable_progress_bar=False):
         2                                               """Greedy approach to pizza slicing.
         3                                               Fifth iteration: randomize the locations in which we try to add stuff."""
         4         1      30197.0  30197.0      1.8      R, C, L, H, pizza = read_input_pizza(fname)
         5         1        469.0    469.0      0.0      possible_shapes = generate_all_shapes(L, H)
         6         1        758.0    758.0      0.0      slice_mask = np.zeros_like(pizza)  # stores the sliced cells as 1 for easy lookup
         7         1          2.0      2.0      0.0      pizza_slices = []  # stores the slices we made
         8         1     120025.0 120025.0      7.0      empty_cells = set(tuple(args) for args in np.transpose(np.nonzero(1-slice_mask)).tolist())
         9         1      17072.0  17072.0      1.0      random_locations = list(x for x in empty_cells)
        10         1     153681.0 153681.0      8.9      shuffle(random_locations)
        11         1          2.0      2.0      0.0      current_score = 0
        12     16045      38475.0      2.4      2.2      for _ in tqdm.tqdm(range(iters), disable=disable_progress_bar):
        13     16045      70258.0      4.4      4.1          location = get_random_available_location_set2(random_locations, empty_cells)
        14     16045      18906.0      1.2      1.1          if location is None:
        15         1         13.0     13.0      0.0              break
        16     16044      19013.0      1.2      1.1          selected_shapes = []
        17    320880     370831.0      1.2     21.6          for shape in possible_shapes:
        18    304836     562662.0      1.8     32.7              if satisfy_constraints_pythran(location, shape, slice_mask, pizza, L, H):
        19     17050      21997.0      1.3      1.3                  selected_shapes.append(shape)
        20     16044      20851.0      1.3      1.2          if len(selected_shapes) > 0:
        21      3455     183416.0     53.1     10.7              current_score = score_pythran(pizza_slices)
        22      3455      18621.0      5.4      1.1              shape = max(selected_shapes, key=lambda shp: current_score + shp[0]*shp[1])
        23      3455      15084.0      4.4      0.9              cut_slice(location, shape, pizza_slices, slice_mask)
        24      3455      56096.0     16.2      3.3              update_empty_cells(location, shape, empty_cells)
        25         1       1365.0   1365.0      0.1      return score(pizza_slices)

Now that we have such a fast function, we can run further improve our algorithm.

# Studying different ways of iterating through the pizza

The next idea here is to vary the way we sort the pizza locations before trying to slice the pizza. Instead of doing a random search as we did so far, let's explore more straightforward sortings: by row or column, decreasing as well as increasing.

```python
def greedy5_pythran(fname, iters=30000, sorting_func=lambda item: item[0], disable_progress_bar=False):
    """Greedy approach to pizza slicing.
    Fifth iteration: randomize the locations in which we try to add stuff."""
    R, C, L, H, pizza = read_input_pizza(fname)
    possible_shapes = generate_all_shapes(L, H)
    slice_mask = np.zeros_like(pizza)  # stores the sliced cells as 1 for easy lookup
    pizza_slices = []  # stores the slices we made
    empty_cells = set(tuple(args) for args in np.transpose(np.nonzero(1-slice_mask)).tolist())
    random_locations = list(x for x in empty_cells)
    random_locations = sorted(random_locations, key=sorting_func)
    current_score = 0
    for _ in tqdm.tqdm(range(iters), disable=disable_progress_bar):
        location = get_random_available_location_set2(random_locations, empty_cells)
        if location is None:
            break
        selected_shapes = []
        for shape in possible_shapes:
            if satisfy_constraints_pythran(location, shape, slice_mask, pizza, L, H):
                selected_shapes.append(shape)
        if len(selected_shapes) > 0:
            current_score = score_pythran(pizza_slices)
            shape = max(selected_shapes, key=lambda shp: current_score + shp[0]*shp[1])
            cut_slice(location, shape, pizza_slices, slice_mask)
            update_empty_cells(location, shape, empty_cells)
    write_output_pizza(fname.split('.')[0] + '.out', pizza_slices)
    return score(pizza_slices)
```

```python
greedy5_pythran('files/medium.in', sorting_func=lambda item: item[0])
show_output('files/medium.out')
```

```python
greedy5_pythran('files/medium.in', sorting_func=lambda item: -item[0])
show_output('files/medium.out')
```

```python
greedy5_pythran('files/medium.in', sorting_func=lambda item: item[1])
show_output('files/medium.out')
```

```python
greedy5_pythran('files/medium.in', sorting_func=lambda item: -item[1])
show_output('files/medium.out')
```

This is really interesting: the best scores are obtained when cutting the pizza from left to right as well as from top to bottom. Does it also work with oblique directions?

```python
greedy5_pythran('files/medium.in', sorting_func=lambda item: -item[0] -item[1])
show_output('files/medium.out')
```

This is our best score so far! Can we get even better? We should, since we see that the bottom part of the pizza is not well filled, with some border artifacts appearing.

At this point, an idea that often comes up for this sort of optimization problems is to try to improve an existing sliced pizza with some local changes, an idea known as "hill climbing" (see this Norvig Pytude about [gesture typing](https://github.com/norvig/pytudes/blob/master/ipynb/Gesture%20Typing.ipynb) for a good example).

# Improving an existing solution

Let's say we have an existing solution and we want to improve it. One possible way is to remove a set number of slices from the pizza, for example the 3 closest parts to a hole in the pizza, and try to fill it again while trying out every possible combination.

```python
from random import choice

def remove_slice(this_slice, empty_cells, pizza_slices, slice_mask):
    """Removes a slice that was sliced from the pizza."""
    pizza_slices.remove(this_slice)
    r, c, dr, dc = this_slice
    for rr in range(r, r+dr):
        for cc in range(c, c+dc):
            empty_cells.add((rr, cc))

    slice_mask[r:r + dr, c:c + dc] = 0

def improve_pizza(fname, slices_to_remove=3, iters=1000):
    """Reads an output of pizza slices and tries to improve it by removing slices."""
    input_fname = fname.split('.')[0] + '.in'
    R, C, L, H, pizza = read_input_pizza(input_fname)
    possible_shapes = generate_all_shapes(L, H)
    lines = open(fname).readlines()
    N = int(lines[0].strip())
    slice_mask = np.zeros_like(pizza)
    pizza_slices = []
    for i in range(1, N + 1):
        r, c, dr, dc = [int(val) for val in lines[i].strip().split()]
        dr += 1 - r
        dc += 1 - c
        slice_mask[r:r + dr, c:c + dc] = i
        pizza_slices.append((r, c, dr, dc))
    # find empty pizza cells and make a set out of it
    empty_cells_list = np.transpose(np.nonzero(slice_mask == 0)).tolist()
    empty_cells = set(tuple(args) for args in empty_cells_list)
    print(f"starting score {score_pythran(pizza_slices)}")
    for _ in tqdm.tqdm(range(iters), disable=True):
        # copy params
        empty_cells_original = empty_cells.copy()
        pizza_slices_original = pizza_slices.copy()
        slice_mask_original = slice_mask.copy()

        # select a random location on the pizza grid from one of the already vacant spots
        removal_location = choice(empty_cells_list)
        # remove existing slices closest to the grid
        sorted_slices = sorted(pizza_slices, key=lambda item: abs(item[0] + item[2] / 2 - removal_location[0]) + \
                                                              abs(item[1] + item[3] / 2 - removal_location[1]))
        for this_slice in sorted_slices[:slices_to_remove]:
            remove_slice(this_slice, empty_cells, pizza_slices, slice_mask)
        # try to place new slices by trying out all possible solutions
        candidates = list(empty_cells_original)
        shuffle(candidates)
        for location in candidates:
            selected_shapes = []
            for shape in possible_shapes:
                if satisfy_constraints_pythran(location, shape, slice_mask, pizza, L, H):
                    selected_shapes.append(shape)
            if len(selected_shapes) > 0:
                current_score = score_pythran(pizza_slices)
                shape = max(selected_shapes, key=lambda shp: current_score + shp[0] * shp[1])
                cut_slice(location, shape, pizza_slices, slice_mask)
                update_empty_cells(location, shape, empty_cells)

        if score_pythran(pizza_slices) <= score_pythran(pizza_slices_original):
            empty_cells = empty_cells_original
            pizza_slices = pizza_slices_original
            slice_mask = slice_mask_original
        else:
            print(f"improved score: {score_pythran(pizza_slices)}")
    write_output_pizza(fname.split('.')[0] + '.out', pizza_slices)
```

```python
improve_pizza('files/medium.out', slices_to_remove=3, iters=1000)
```

    starting score 48433
    improved score: 48434
    improved score: 48442
    improved score: 48446
    improved score: 48449
    improved score: 48452
    improved score: 48458
    improved score: 48464
    improved score: 48468
    improved score: 48470
    improved score: 48471
    improved score: 48473
    improved score: 48483
    improved score: 48484
    improved score: 48494
    improved score: 48496

```python
show_output('files/medium.out')
```

We can observe the result of our hill-climbing algorithm here: it added a few optimisations to some of the pizza slices at the bottom of the field, shown in the left plot as patches of yellow color (slices added last). If we let it run for a long time, maybe it would add some more of these patches.

# Conclusions

In this blog post, we went from an idea to the implementation of a fast greedy solution to the training problem of the 2018 Google Hashcode. We then improved it using an iterative hill-climbing algorithm. I think this nicely shows how all the moving parts of a Hashcode problem fit together.

To conclude, here's a high level summary of the tools that I found useful while tackling the pizza cutting problem:

- know your data structures and their speeds: when I realized that I needed [sets](https://docs.python.org/3.6/library/stdtypes.html?highlight=set#set) (for fast membership testing) and lists (for fast lookup of index $n$) I got a 10x speedup
- use a debugger for development: during the writing of the above code (something that is not shown in this Jupyter notebook), the Pycharm debugger proved invaluable
- profile your functions using line_profiler and troubleshoot your hotspots
- use Jupyter Notebook for interactive exploration and plotting
- use pythran or numba if you need to make things faster
- use tqdm to display progress bars! this gives you great feedback about how your algorithm is doing

*This post was entirely written using the IPython notebook. Its content is BSD-licensed. You can see a static view or download this notebook with the help of nbviewer at [20180207_PreparingHashcode2018.ipynb](http://nbviewer.ipython.org/urls/raw.github.com/flothesof/posts/master/20180207_PreparingHashcode2018.ipynb).*


```python
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import pandas as pd
from scipy.stats import norm, uniform, expon
from scipy.integrate import quad
from sympy.solvers import solve
from sympy import Symbol
import numpy as np
from pandas import Series, DataFrame
```

```python
x = st.norm.rvs(size=10)
st.probplot(x,plot=plt)
```

```python

```


# ENV / ATM 415: Climate Laboratory

# Radiative- and Radiative-Convective Equilibrium with `climlab`

Thursday March 31, 2016

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import netCDF4 as nc
import climlab
```

____________

## Using `climlab` to implement the two-layer leaky greenhouse model
____________

One of the things that ``climlab`` is set up to do is the grey-radiation modeling we have already been discussing.

Since we already derived a complete analytical solution to the two-layer leaky greenhouse model, we will use this to validate the `climlab` code.

### Validation

We want to verify that the model reproduces the observed OLR given observed temperatures, and the absorptivity that we tuned in the analytical model. The target numbers are:

\begin{align}
T_s &= 288 \text{ K} \\
T_0 &= 275 \text{ K} \\
T_1 &= 230 \text{ K} \\
\end{align}

$$ \epsilon = 0.58377  $$

$$ OLR = 239 \text{ W m}^{-2} $$

### Initialize a model in `climlab`
The first thing we do is create a new model.

The following example code is sparsely commented but will hopefully orient you on the basics of defining and working with a `climlab Process` object.

```python
#  Test in a 2-layer atmosphere
col = climlab.GreyRadiationModel(num_lev=2)
print col
```

    climlab Process of type <class 'climlab.model.column.GreyRadiationModel'>.
    State variables and domain shapes:
      Tatm: (2,)
      Ts: (1,)
    The subprocess tree:
    top: <class 'climlab.model.column.GreyRadiationModel'>
       LW: <class 'climlab.radiation.greygas.GreyGas'>
       SW: <class 'climlab.radiation.greygas.GreyGasSW'>
       insolation: <class 'climlab.radiation.insolation.FixedInsolation'>

```python
col.state
```

    {'Tatm': Field([ 200.,  278.]), 'Ts': Field([ 288.])}

```python
col.Ts
```

    Field([ 288.])

```python
#  Set the temperatures to our observed values
col.Ts[:] = 288.
col.Tatm[:] = np.array([230., 275.])
col.state
```

    {'Tatm': Field([ 230.,  275.]), 'Ts': Field([ 288.])}

```python
LW = col.subprocess['LW']
print LW
```

    climlab Process of type <class 'climlab.radiation.greygas.GreyGas'>.
    State variables and domain shapes:
      Tatm: (2,)
      Ts: (1,)
    The subprocess tree:
    top: <class 'climlab.radiation.greygas.GreyGas'>

```python
LW.absorptivity
```

    Field([ 0.47737425,  0.47737425])

```python
#  Set the absorptivity to our tuned value
LW.absorptivity = 0.58377
LW.absorptivity
```

    Field([ 0.58377,  0.58377])

```python
#  This does all the calculations that would be performed at each time step,
#  but doesn't actually update the temperatures
col.compute_diagnostics()
#  Let's see what's in the diagnostics dictionary
col.diagnostics
```

    {'ASR': array([ 239.2513]),
     'LW_absorbed_atm': array([ 20.03935568, -96.82138041]),
     'LW_absorbed_sfc': 0.0,
     'LW_down_sfc': array([ 227.87116061]),
     'LW_emission': Field([  92.63278385,  189.31461699]),
     'LW_up_sfc': 0.0,
     'OLR': array([ 239.01589408]),
     'SW_absorbed_atm': array([ 0.,  0.]),
     'SW_absorbed_sfc': 0.0,
     'SW_down_TOA': array([ 341.3]),
     'SW_up_TOA': array([ 102.0487]),
     'SW_up_sfc': Field([ 102.0487]),
     'absorbed': array([ 0.,  0.]),
     'absorbed_total': 0.0,
     'emission': Field([ 0.,  0.]),
     'emission_sfc': Field([ 0.]),
     'flux_from_sfc': Field([ 102.0487]),
     'flux_reflected_up': array([   0.    ,    0.    ,  102.0487]),
     'flux_to_sfc': array([ 341.3]),
     'flux_to_space': array([ 102.0487]),
     'insolation': array([ 341.3]),
     'planetary_albedo': array([ 0.299])}

```python
#  Check OLR against our analytical solution
col.OLR
```

    array([ 239.01589408])

```python
col.state
```

    {'Tatm': Field([ 230.,  275.]), 'Ts': Field([ 288.])}

```python
# perform a single time step
col.step_forward()
```

```python
col.state
```

    {'Tatm': Field([ 230.33800245,  273.36692033]), 'Ts': Field([ 289.59144429])}

```python
# integrate out to radiative equilibrium
col.integrate_years(2.)
```

    Integrating for 730 steps, 730.4844 days, or 2.0 years.
    Total elapsed time is 2.00141166601 years.

```python
# Check for equilibrium
col.ASR - col.OLR
```

    array([ -2.78206130e-07])

```python
#  The temperatures at radiative equilibrium
col.state
```

    {'Tatm': Field([ 233.62925791,  262.08988335]), 'Ts': Field([ 296.20384534])}

Compare these to the analytical solutions for radiative equilibrium with $\epsilon = 0.58$:

\begin{align}
T_1 &= 234 \text{ K} \\
T_0 &= 262 \text{ K} \\
T_s &= 296 \text{ K} \\
\end{align}

So it looks like `climlab` agrees with our analytical results. That's good.

____________

## The observed annual, global mean temperature profile
____________

We want to model the OLR in a column whose temperatures match observations. We'll calculate the global, annual mean air temperature from the NCEP Reanalysis data.

```python
# This will try to read the data over the internet.
ncep_filename = 'air.mon.1981-2010.ltm.nc'
#ncep_url = "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/"
#ncep_air = nc.Dataset( ncep_url + 'pressure/' + ncep_filename )
# Or to read from local disk
ncep_air = nc.Dataset( ncep_filename )

level = ncep_air.variables['level'][:]
lat = ncep_air.variables['lat'][:]
# A log-pressure height coordinate
zstar = -np.log(level/1000)
```

```python
#  Take averages of the temperature data
Tzon = np.mean(ncep_air.variables['air'][:],axis=(0,3))
Tglobal = np.average( Tzon , weights=np.cos(np.deg2rad(lat)), axis=1) + climlab.constants.tempCtoK
#  Note the useful conversion factor. climlab.constants has lots of commonly used constant pre-defined
```

```python
#  Here we are plotting with respect to log(pressure) but labeling the axis in pressure units
fig = plt.figure( figsize=(8,6) )
ax = fig.add_subplot(111)
ax.plot( Tglobal , zstar )
yticks = np.array([1000., 750., 500., 250., 100., 50., 20., 10.])
ax.set_yticks(-np.log(yticks/1000.))
ax.set_yticklabels(yticks)
ax.set_xlabel('Temperature (K)', fontsize=16)
ax.set_ylabel('Pressure (hPa)', fontsize=16 )
ax.set_title('Global, annual mean sounding from NCEP Reanalysis', fontsize = 24)
ax.grid()
```

____________

## A 30-layer model using the observed temperatures
____________

```python
#  initialize a grey radiation model with 30 levels
col = climlab.GreyRadiationModel()
print col
```

    climlab Process of type <class 'climlab.model.column.GreyRadiationModel'>.
    State variables and domain shapes:
      Tatm: (30,)
      Ts: (1,)
    The subprocess tree:
    top: <class 'climlab.model.column.GreyRadiationModel'>
       LW: <class 'climlab.radiation.greygas.GreyGas'>
       SW: <class 'climlab.radiation.greygas.GreyGasSW'>
       insolation: <class 'climlab.radiation.insolation.FixedInsolation'>

```python
# interpolate to 30 evenly spaced pressure levels
lev = col.lev
Tinterp = np.interp(lev, np.flipud(level), np.flipud(Tglobal))

print Tinterp
#  Need to 'flipud' because the interpolation routine needs the pressure data to be in increasing order
```

    [ 224.34736633  211.66334534  206.96233453  208.29142761  212.58644104
      217.19397481  221.78252157  226.3520813   231.30422974  236.08017476
      240.67991638  245.279658    249.35979716  252.92033386  256.48087056
      259.66789246  262.48139954  265.29490662  267.81303914  270.03579712
      272.25855509  274.21642049  275.90939331  277.60236613  279.29533895
      280.98831177  282.48550415  283.98269653  285.6810201   287.4463874 ]

```python
# Initialize model with observed temperatures
col.Ts[:] = Tglobal[0]
col.Tatm[:] = Tinterp
```

```python
#  A handy re-usable routine for making a plot of the temperature profiles
#  We will plot temperatures with respect to log(pressure) to get a height-like coordinate
def plot_sounding(collist):
    color_cycle=['r', 'g', 'b', 'y']
    # col is either a column model object or a list of column model objects
    if isinstance(collist, climlab.Process):
        # make a list with a single item
        collist = [collist]
    fig = plt.figure()
    ax = fig.add_subplot(111)
    for i, col in enumerate(collist):
        zstar = -np.log(col.lev/climlab.constants.ps)
        ax.plot(col.Tatm, zstar, color=color_cycle[i])
        ax.plot(col.Ts, 0, 'o', markersize=12, color=color_cycle[i])
    #ax.invert_yaxis()
    yticks = np.array([1000., 750., 500., 250., 100., 50., 20., 10.])
    ax.set_yticks(-np.log(yticks/1000.))
    ax.set_yticklabels(yticks)
    ax.set_xlabel('Temperature (K)')
    ax.set_ylabel('Pressure (hPa)')
    ax.grid()
    return ax
```

```python
# This should look just like the observations
plot_sounding(col)
```

### Tune absorptivity to get observed OLR

```python
col.compute_diagnostics()
col.OLR
```

    array([ 263.15000167])

```python
# Need to tune absorptivity to get OLR = 239
epsarray = np.linspace(0.01, 0.1, 100)
OLRarray = np.zeros_like(epsarray)
```

```python
for i in range(epsarray.size):
    col.subprocess['LW'].absorptivity = epsarray[i]
    col.compute_diagnostics()
    OLRarray[i] = col.OLR

plt.plot(epsarray, OLRarray)
plt.grid()
```

The necessary value seems to lie near 0.055 or so.

A precise numerical search gives $\epsilon = 0.0534$

```python
eps = 0.0534

col.subprocess.LW.absorptivity = eps
col.subprocess.LW.absorptivity
```

    Field([ 0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,
            0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,
            0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,
            0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,  0.0534,
            0.0534,  0.0534])

```python
#  Double check to make sure this gives the right OLR

col.compute_diagnostics()
col.OLR
```

    array([ 239.00554514])

```python
col.ASR - col.OLR
```

    array([ 0.24575486])

____________

## Radiative equilibrium in the 30-layer model
____________

```python
#  Make a clone of our first model
re = climlab.process_like(col)
```

```python
#  To get to equilibrium, we just time-step the model forward long enough
re.integrate_years(2.)
```

    Integrating for 730 steps, 730.4844 days, or 2.0 years.
    Total elapsed time is 1.99867375676 years.

```python
#  Check for energy balance
re.ASR - re.OLR
```

    array([ -4.63536253e-07])

Make a plot that compares the **observed temperatures** to the **radiative equilibrium** temperature profile.

```python
plot_sounding([col, re])
```

Some properties of the **radiative equilibrium** temperature profile:

- The surface is warmer than observed.
- The lower troposphere is colder than observed.
- Very cold air is sitting immediately above the warm surface.
- There is no tropopause, no stratosphere.

____________

## Radiative-Convective Equilibrium in the 30-layer model
____________

We recognize that the large drop in temperature just above the surface is unphysical. Parcels of air in direct contact with the ground will be warmed by mechansisms other than radiative transfer.

These warm air parcels will then become buoyant, and will convect upward, mixing their heat content with the environment.

We **parameterize** the statistical effects of this mixing through a **convective adjustment**.

At each timestep, our model checks for any locations at which the **lapse rate** exceeds some threshold. Unstable layers are removed through an energy-conserving mixing formula.

This process is assumed to be fast relative to radiative heating. In the model, it is instantaneous.

```python
rce = climlab.RadiativeConvectiveModel(adj_lapse_rate=6.)
print rce
```

    climlab Process of type <class 'climlab.model.column.RadiativeConvectiveModel'>.
    State variables and domain shapes:
      Tatm: (30,)
      Ts: (1,)
    The subprocess tree:
    top: <class 'climlab.model.column.RadiativeConvectiveModel'>
       convective adjustment: <class 'climlab.convection.convadj.ConvectiveAdjustment'>
       LW: <class 'climlab.radiation.greygas.GreyGas'>
       SW: <class 'climlab.radiation.greygas.GreyGasSW'>
       insolation: <class 'climlab.radiation.insolation.FixedInsolation'>

This model is exactly like our previous models, except for one additional subprocess called ``convective adjustment``.

We passed a parameter ``adj_lapse_rate`` (in K / km) that sets the neutrally stable lapse rate -- in this case, 6 K / km.

This number is chosed to very loosely represent the net effect of **moist convection**.

```python
# Set our tuned absorptivity value
rce.subprocess.LW.absorptivity = eps
```

```python
#  Run out to equilibrium
rce.integrate_years(2.)
```

    Integrating for 730 steps, 730.4844 days, or 2.0 years.
    Total elapsed time is 1.99867375676 years.

```python
#  Check for energy balance
rce.ASR - rce.OLR
```

    array([  1.96076689e-06])

```python
#  Make a plot to compare observations, Radiative Equilibrium, and Radiative-Convective Equilibrium
plot_sounding([col, re, rce])
```

Introducing convective adjustment into the model cools the surface quite a bit (compared to Radiative Equilibrium, in green here) -- and warms the lower troposphere. It gives us a MUCH better fit to observations.

But of course we still have no stratosphere.

The missing ingredient is **absorption of shortwave UV radiation by ozone**.

## The heating rates due to radiation and convection

```python
#  Here we plotting the heating rates in K / day
degrees_per_day_atm = {'LW': (rce.subprocess['LW'].heating_rate['Tatm'] /
                                 rce.Tatm.domain.heat_capacity *
                                 climlab.constants.seconds_per_day),
                       'SW': (rce.subprocess['SW'].heating_rate['Tatm'] /
                                 rce.Tatm.domain.heat_capacity *
                                 climlab.constants.seconds_per_day),
                   'Convection': (rce.subprocess['convective adjustment'].adjustment['Tatm'] /
                                  rce.timestep *
                                  climlab.constants.seconds_per_day)}
degrees_per_day_sfc = {'LW': (rce.subprocess['LW'].heating_rate['Ts'] /
                                      rce.Ts.domain.heat_capacity *
                                      climlab.constants.seconds_per_day),
                        'SW': (rce.subprocess['SW'].heating_rate['Ts'] /
                                      rce.Ts.domain.heat_capacity *
                                      climlab.constants.seconds_per_day),
                        'Convection': (rce.subprocess['convective adjustment'].adjustment['Ts'] /
                                       rce.timestep *
                                       climlab.constants.seconds_per_day)}
```

```python
color_cycle=['r', 'g', 'b', 'y']

fig = plt.figure( figsize=(8,6) )
ax = fig.add_subplot(111)
zstar = -np.log(rce.lev/climlab.constants.ps)
for i, item in enumerate(degrees_per_day_atm):
    ax.plot(degrees_per_day_atm[item], zstar, color=color_cycle[i], label=item)
for i, item in enumerate(degrees_per_day_sfc):
    ax.plot(degrees_per_day_sfc[item], 0, 'o', markersize=12, color=color_cycle[i])
yticks = np.array([1000., 750., 500., 250., 100., 50., 20., 10.])
ax.set_yticks(-np.log(yticks/1000.))
ax.set_yticklabels(yticks)
ax.set_xlabel('Heating rates (degrees per day)', fontsize=16)
ax.set_ylabel('Pressure (hPa)', fontsize=16 )
ax.legend()
ax.grid()
```

What do we see here?

- SW radiation is acting to heat the surface strongly
- No SW heating in the atmosphere (assumed to be transparent)
- At the surface, there is a **balance** between **heat gain from SW radiation** and **heat loss from both LW radiation and convection**
- LW radiation is acting to **cool the troposphere**
- Convection is acting to **warm the troposphere**

Collectively the LW and SW radiation are trying to push the temperatures towards **radiative equilibrium** while the convection is moving heat from the surface to the troposphere.

```python

```