```python
%matplotlib inline
%load_ext iminizinc
import asyncio
from IPython.display import HTML
import ipywidgets as widgets
from ipywidgets import interact, interactive

from problems import nqueens, us_map_coloring
from draw_utils import draw_nqueens, draw_us_map
from slide_utils import SlideController
from backtracking_search import nqueens_backtracking
from utils import autoupdate_cells
```

    The iminizinc extension is already loaded. To reload it, use:
      %reload_ext iminizinc

```python
HTML('')
```

# Constraint Programming

## Introduction

Constraints naturally arise in a variety of interactions and fields of study such as game theory, social studies, operations research, engineering, and artificial intelligence. A constraint refers to the relationship between the state of objects, such as the constraint that the three angles of a triangle must sum to 180 degrees. Note that this constraint has not precisely stated each angle's value and still allows some flexibility. Said another way, the triangle constraint restricts the values that the three variables (each angle) can take, thus providing information that will be useful in finding values for the three angles.

Another example of a constrained problem comes from the recently-aired hit TV series *Buddies*, where a group of five (mostly mutual) friends would like to sit at a table with three chairs in specific arrangements at different times, but have requirements as to who they will and will not sit with.

Another example comes from scheduling: at the university level, there is a large number of classes that must be scheduled in various classrooms such that no professor or classroom is double booked. Further, there are some constraints on which classes can be scheduled for the same time, as some students will need to be registered for both.

Computers can be employed to solve these types of problems, but in general these tasks are computationally intractable and cannot be solved efficiently in all cases with a single algorithm \cite{Dechter2003}. However, by formalizing these types of problems in a constraint processing framework, we can identify classes of problems that can be solved using efficient algorithms.

Below, we discuss generally the three core concepts in constraint programming: **modeling**, **inference**, and **search**. Modeling is an important step that can greatly affect the ability to efficiently solve constrained problems and inference (e.g., constraint propagation) and search are solution methods. Basic constraint propagation and state-space search are building blocks that state of the art solvers incorporate.

### Modeling

A **constraint satisfaction problem** (CSP) is formalized by a *constraint network*, which is the triple $\mathcal{R} = \langle X,D,C\rangle$, where
- $X = \{x_i\}_{i=1}^n$ is the set of $n$ variables
- $D = \{D_i\}_{i=1}^n$ is the set of variable domains, where the domain of variable $x_k$ is $D_k$
- $C = \{C_i\}_{i=1}^m$ is the set of constraints on the values that each $x_i$ can take on. Specifically,
    - Each constraint $C_i = \langle S_i,R_i\rangle$ specifies allowed variable assignments.
    - $S_i \subset X$ contains the variables involved in the constraint, called the *scope* of the constraint.
    - $R_i$ is the constraint's *relation* and represents the simultaneous legal value assignments of variables in the associated scope.
        - For example, if the scope of the first constraint is $S_1 = \{x_3, x_8\}$, then the relation $R_1$ is a subset of the Cartesian product of those variables' domains: $R_1 \subset D_3 \times D_8$, and an element of the relation $R_1$ could be written as a 2-tuple $(a,b)\in R_1$.

Each variable in a CSP can be assigned a value from its domain. A **complete assignment** is one in which every variable is assigned and a **solution** to a CSP is a consistent (or legal w.r.t. the constraints) complete assignment.

Note that for a CSP model, *any* consistent complete assignment of the variables (i.e., where all constraints are satisfied) constitutes a valid solution; however, this assignment may not be the "best" solution. Notions of optimality can be captured by introducing an objective function which is used to find a valid solution with the lowest cost. This is referred to as a **constraint *optimization* problem** (COP). We will refer generally to CSPs with the understanding that a CSP can easily become a COP by introducing a heuristic.

In this notebook, we will restrict ourselves to CSPs that can be modeled as having **discrete, finite domains**. This helps us to manage the complexity of the constraints so that we can clearly discuss the different aspects of CSPs. Other variations exist such as having discrete but *infinite* domains, where constraints can no longer be enumerated as combinations of values but must be expressed as either linear or nonlinear inequality constraints, such as $T_1 + d_1 \leq T_2$. Therefore, infinite domains require a different constraint language and special algorithms only exist for linear constraints. Additionally, the domain of a CSP may be continuous. With this change, CSPs become mathematical programming problems which are often studied in operations research or optimization theory, for example.

### Modeling as a Graph

In a general CSP, the *arity* of each constraint (i.e., the number of variables involved) is arbitrary. We can have unary constraints on a single variable, binary constraints between two variables, or $n$-ary constraints between $n$ variables. However, having more than binary constraints adds complexity to the algorithms for solving CSPs. It can be shown that every finite-domain constraint can be reduced to a set of binary constraints by adding enough auxiliary variables \cite{AIMA}. Therefore, since we are only discussing CSPs with finite domains, we will assume that the CSPs we are working with have only unary and binary constraints, meaning that each constraint scope has at most two variables.

An important view of a binary constraint network that defines a CSP is as a graph, $\langle\mathcal{V},\mathcal{E}\rangle$. In particular, each vertex corresponds to a variable, $\mathcal{V} = X$, and the edges of the graph $\mathcal{E}$ correspond to various constraints between variables. Since we are only working with binary and unary constraint networks, it is easy to visualize a graph corresponding to a CSP. For constraint networks with more than binary constraints, the constraints must be represented with a hypergraph, where hypernodes are inserted that connect three or more variables together in a constraint.

For example, consider a CSP $\mathcal{R}$ with the following definition
\begin{align}
X &= \{x_1, x_2, x_3\} \\
D &= \{D_1, D_2, D_3\},\  \text{where}\; D_1 = \{0,5\},\ D_2 = \{1,2,3\},\ D_3 = \{7\} \\
C &= \{C_1, C_2, C_3\},
\end{align}
where
\begin{align}
C_1 &= \langle S_1, R_1 \rangle = \langle \{x_1\}, \{5\} \rangle \\
C_2 &= \langle S_2, R_2 \rangle = \langle \{x_1, x_2\}, \{(0, 1), (0,3), (5,1)\} \rangle \\
C_3 &= \langle S_3, R_3 \rangle = \langle \{x_2, x_3\}, \{(1, 7), (2, 7)\} \rangle.
\end{align}
The graphical model of this CSP is shown below.

### Solving

The goal of formalizing a CSP as a constraint network model is to efficiently solve it using computational algorithms and tools. **Constraint programming** (CP) is a powerful tool to solve combinatorial constraint problems and is the study of computational systems based on constraints. Once the problem has been modeled as a formal CSP, a variety of computable algorithms could be used to find a solution that satisfies all constraints.

In general, there are two methods used to solve a CSP: search or inference. In previous 16.410/413 problems, **state-space search** was used to find the best path through some sort of graph or tree structure. Likewise, state-space search could be used to find a valid "path" through the CSP that satisfies each of the local constraints and is therefore a valid global solution. However, this approach would quickly become intractable as the number of variables and the size of each of their domains increase.

In light of this, the second solution method becomes more attractive. **Constraint propagation**, a specific type of inference, is used to reduce the number of legal values from a variable's domain by pruning values that would violate the constraints of the given variable. By making a variable locally consistent with its constraints, the domain of adjacent variables may potentially be further reduced as a result of missing values in the pairwise constraint of the two variables. In this way, by making the first variable consistent with its constraints, the constraints of neighboring variables can be re-evaluated, causing a further reduction of domains through the propagation of constraints. These ideas will later be formalized as $k$-consistency.

Constraint propagation may be combined with search, using the pros of both methods simultaneously. Alternatively, constraint propagation may be performed as a pre-processing pruning step so that search has a smaller state space to search over. Sometimes, constraint propagation is all that is required and a solution can be found without a search step at all.

After giving examples of modeling CSPs, this notebook will explore a variety of solution methods based on constraint propagation and search.

---

## Problem Models

Given a constrained problem, it is desirable to identify an appropriate constraint network model $\mathcal{R} = \langle X,D,C\rangle$ that can be used to find its solution. Modeling for CSPs is an important step that can dramatically affect the difficulty in enumerating the associated constraints or efficiency of finding a solution.

Using the general ideas and formalisms from the previous section, we consider two puzzle problems and model them as CSPs in the following sections.

### N-Queens

The N-Queens problem (depicted below for 4 queens) is a well-know puzzle among computer scientists and will be used as a recurring example throughout this notebook. The problem statement is as follows: given any integer $N$, the goal is to place $N$ queens on an $N\times N$ chessboard satisfying the constraint that no two queens threaten each other. A queen can threaten any other queen that is on the same row, column, or diagonal.

```python
# Example n-queens
draw_nqueens(nqueens(4))
```

Now let's try to understand the problem formally.

#### Attempt 1

To illustrate the effect of modeling, we first consider a (poor) model for the N-Queens constraint problem, given by the following definitions:
\begin{align}
X &= \{x_i\}_{i=1}^{N^2}                      && \text{(Chessboard positions)} \\
D &= \{D_i\}_{i=1}^{N^2},\ \text{where}\; D_i = \{0, 1,2,\dots,N\}  && \text{(Empty or the $k^\text{th}$ queen)}
\end{align}

Without considering constraints, the size of the state space (i.e., the number of assignments) is an enormous $(N+1)^{N^2}$. For only $N=4$ queens, this becomes $5^{16} \approx 153$ billion states that could potentially be searched.

Expressing the constraints of this problem in terms of the variables and their domains also poses a challenge. Because of the way we have modeled this problem, there are six primary constraints to satisfy:
1. Exactly $N$ chess squares shall be filled (i.e., there are only $N$ queens and all of them must be used)
1. The $k^\text{th}$ queen, ($1\le k\le N$) shall only be used once.
1. No queens share a column
1. No queens share a row
1. No queens share a positive diagonal (i.e., a diagonal from bottom left to top right)
1. No queens share a negative diagonal (i.e., a diagonal from top left to bottom right)

To express these constraints mathematically, we first let $Y\triangleq\{1\le i\le N^2|x_i\in X,x_i\ne 0\}$ be the set of chess square numbers that are non-empty and $Z \triangleq \{x\in X|x\ne 0\}$ be the set of queens in those chess squares (unordered). With pointers back to which constraint they satisfy, the expressions are:
\begin{align}
|Z| = |Y| &= N && (C1) \\
z_i-z_j &\ne 0 && (C2) \\
|y_i-y_j| &\ne N && (C3) \\
\left\lfloor\frac{y_i-1}{N}\right\rfloor &\ne \left\lfloor\frac{y_j-1}{N}\right\rfloor && (C4) \\
|y_i-y_j| &\ne (N-1) && (C5) \\
|y_i-y_j| &\ne (N+1), && (C6)
\end{align}
where $z_i, z_j\in Z$ and $y_i,y_j\in Y, \forall i\ne j$, and applying $|\cdot|$ to a set is the set's cardinality (i.e., size) and applied to a scalar is the absolute value. Additionally, we use $\lfloor\cdot\rfloor$ as the floor operator. Notice how we are able to express all the constraints as pairwise (binary).

We can count the number of constraints in this model as a function of $N$. In each pairwise constraint (C2)-(C6), there are $N$ choose $2$ pairs. Since we have 5 different types of pairwise constraints, we have that the number of constraints, $\Gamma$, is
\begin{equation}
\Gamma(N) = 5 {N \choose 2} + 1 = \frac{5N!}{2!(N-2)!} + 1,
\end{equation}
where the plus one comes from the single constraint for (C1). Thus, $\Gamma(N=4) = 31$.

Examining the size of the state space in this model, we see the infeasibility of simply performing a state-space search and then performing a goal test that encodes the problem constraints. This motivates the idea of efficiently using constraints either before or during our solution search, which we will explore in the following sections.

#### Attempt 2

Motivated by the desire to do less work in searching and writing constraints, we consider another model of the N-Queens problem. We wish to decrease the size of the state space and number and difficulty of writing the constraints. Good modeling involves cleverly choosing variables and their semantics so that constraints are implicitly encoded, requiring less explicit constraints.

We can achieve this by encoding the following assumptions:
1. assume one queen per column;
1. an assignment determines which row the $i^\text{th}$ queen should be in.

With this understanding, we can write the constraint network as
\begin{align}
X &= \{x_i\}_{i=1}^{N}                      && \text{(Queen $i$ in the $i^\text{th}$ column)} \\
D &= \{D_i\}_{i=1}^{N},\ \text{where}\; D_i = \{1,2,\dots,N\}  && \text{(The row in which the $i^\text{th}$ queen should be placed)}.
\end{align}

Now considering the size of the state space without constraints, we see that this intelligent encoding reduces the size to only $N^N$ assignments.

Writing down the constraints is also easier for this model. In fact, we only need to address constraints (C4)-(C6) from above, as (C1)-(C3) are taken care of by intelligently choosing our variables and their domains. The expressions, $\forall x_i,x_j\in X, i\ne j$, are
\begin{align}
x_i &\ne x_j && \text{(C4)} \\
|x_i-x_j| &\ne |i-j|. && \text{(C5, C6)}
\end{align}

With this reformulation, the number of constraints is
\begin{equation}
\Gamma(N) = 2 {N \choose 2} = \frac{N!}{(N-2)!}.
\end{equation}
Thus $\Gamma(N=4) = 12$.

We have successfully modeled the N-Queens problem with a reduced state space and with only two pairwise constraints. Both of these properties will allow the solvers discussed next to more efficiently find solutions to this CSP.

### Map Coloring

Map coloring is another classic example of a CSP. Consider the map of Australia shown below (from \cite{AIMA}). The goal is to assign a color to Australia's seven territories such that no neighboring regions share the same color. We are further constrained by only being able to use three colors (e.g., R, G, B). Next to the map is the constraint graph representation of this specific map-coloring problem.

The constraint network model $\mathcal{R}=\langle X,D,C \rangle$ for the general map-coloring problem with $N$ regions and $M$ colors is defined as:
\begin{align}
X &= \{x_i\}_{i=1}^N && \text{(Each region)} \\
D &= \{D_i\}_{i=1}^N,\ \text{where}\; D_i = \{c_j\}_{j=1}^M,   && \text{(Available colors)}
\end{align}
and the constraints are encoded as
\begin{align}
\forall x_i\in X: x_i &\ne n_j,\ \forall n_j\in\mathcal{N}(x_i), && \text{(Each region cannot have the same color as any of its neighbors)}
\end{align}
where the neighborhood of the region $x_i$ is defined as the set $\mathcal{N}(x_i) = \{x_j\in X| A_{ij}=1,i\ne j, \forall j\}$. The matrix $A\in\mathbb{Z}_{\ge 0}^{N\times N}$ is called the *adjacency matrix* of a graph with $N$ vertices and represents the variables that a given variable is connected to by constraints (i.e., edges). The notation $A_{mn}$ indexes into the matrix by row $m$ and column $n$.

We will use the map coloring problem as a COP example later on.

### First MiniZinc model

We are now ready to solve our first CPS problem! Let us now introduce [MiniZinc](https://www.minizinc.org/), a **high-level**, **solver-independent** language to express constraint programming problems and solve them. It has a large library of constraints already encoded that we can exploit to encode our problem.

A very useful constraint is `alldifferent(array[int] of var int: x)`, which is one of the most studied and used constraint in constraint programming. As the name suggest it takes an array of variables and constrains them to take different values.

Let's focus on the N-Queens problem as formulated in attempt 2. The reader can notice that we can write (C1), (C2) and (C3) leveraging the `alldifferent` constraint. As result we get the following model.

```python
%%minizinc
include "globals.mzn";

int: n = 4;
array[1..n] of var 1..n: queens;

constraint all_different(queens);
constraint all_different([queens[i]+i | i in 1..n]);
constraint all_different([queens[i]-i | i in 1..n]);

solve satisfy;
```

Here we are asking MiniZinc to solve find any feasible solution (`solve satisfy`) given the constraints.

With high-level languages is easy to describe and solve a CSP. The solver at the same time abstract away the complexity of the search process. Let's now focus on how a CSP is actually solved.

---

## Constraint Propagation Methods

As previously mentioned, the domain size of a CSP can be dramatically reduced by removing values from variable domains that would violate the relevant constraints. This idea is called **local consistency**. By representing a CSP as a binary constraint graph, making a graph locally consistent amounts to visiting the $i^\text{th}$ node and for each of the values in the domain $D_i$, removing the values of neighboring domains that would cause an illegal assignment.

A great example of the power of constraint propagation is seen in Sudoku puzzles. Simple puzzles are designed to be solved by constraint propagation alone. By enforcing local consistency throughout simple formulations of Sudoku common in newspapers, the unique solution is found without the need for search.

While there are multiple forms of consistency, we will forgo a discussion of node consistency (single node), path consistency (3 nodes), and generally **$k$-consistency** ($k$ nodes) to focus on arc consistency.

### Arc Consistency

The most well-known notion of local consistency is **arc consistency**, where the key idea is to remove values of variable domains that can never satisfy a specified constraint. The arc $\langle x_i, x_j \rangle$ between two variables $x_i$ and $x_j$ is said to be arc consistent if $\langle x_i, x_j \rangle$ and $\langle x_j, x_i \rangle$ are *directed* arc consistent.

The arc $\langle x_i, x_j \rangle$ is **directed arc consistent** (from $x_i$ to $x_j$) if $\forall a_i \in D_i \;
\exists a_j \in D_j$ s.t. $\langle a_i, a_j \rangle \in C_{ij}$. The notation $C_{ij}$ represents a constraint between variables $x_i$ and $x_j$ with a relation on their domains $D_i, D_j$. In other words, we write a constraint $\langle \{x_i, x_j\}, R \rangle$ as $C_{ij} = R$, where $R\subset D_i\times D_j$.

As an example, consider the following simple constraint network:
\begin{align}
X &= \{x_1, x_2\} \\
D &= \{D_1, D_2\},\ \text{where}\; D_1=\{1,3,5,7\}, D_2=\{2,4,6,8\} \\
C &= \{C_{12}\},
\end{align}
where $C_{12} = \{(1,2),(3,8),(7,4)\}$ lists legal assignment relationships between $x_1$ and $x_2$.

To make $\langle x_1, x_2 \rangle$ directed arc consistent, we would remove the values from $D_1$ that could never satisfy the constraint $C_{12}$. The original domains are shown on the left, while the directed arc consistent graph is shown on the right. Note that 6 is not removed from $D_2$ because directed arc consistency only considers consistency in one direction.

Similarly, we can make $\langle x_2, x_1 \rangle$ directed arc consistent by removing 6 from $D_2$. This results in an arc consistent graph, shown below.

#### Sound but Incomplete
By making a CSP arc consistent, we are guaranteed that solutions to the CSP will be found in the reduced domain of the arc consistent CSP. However, we are not guaranteed that any arbitrary assignment of variables from the reduced domain will offer a valid CSP solution. In other words, arc consistency is sound (all solutions are arc-consistent solutions) but incomplete (not all arc-consistent solutions are valid solutions).

### Algorithms

To achieve arc consistency in a graph, we can formalize the ideas that we discussed above about removing values from domains that will never participate in a legal constraint. Two widespread algorithms are considered, known `AC-1` and `AC-3`, which are the first and third versions described by Mackworth in \cite{Mackworth1977}.

In this section, we give the pseudocode for these algorithms and a discussion of their complexities and trade offs.

#### The `REVISE` Algorithm

First, we formalize the procedure of achieving local consistency via the `REVISE` procedure, which is an algorithm that enforces directed arc consistency on a subnetwork. This is the algorithm that we used in the toy example above with $x_1$ and $x_2$.

```vhdl
1  procedure REVISE(xi,xj)
2  for each ai in Di
3    if there is no aj in Dj such that (ai,aj) is consistent,
4       delete ai from Di
5    end if
6  end for
7  end
```

##### Complexity Analysis

The complexity of `REVISE` is $O(k^2)$, where $k$ bounds the domain size, i.e., $k=\max_i|D_i|$. The $k^2$ comes from the fact that there is a double `for loop`---the outer loop is on line 2 and the inner loop is on line 3.

#### The `AC-1` Algorithm

A first pass of enforcing arc consistency on an entire constraint network would be to revise each variable domain in a brute-force manner. This is the objective of the following `AC-1` procedure, which takes a CSP definition $\mathcal{R}=\langle X, D, C\rangle$ as input.

```vhdl
1  procedure AC1(csp)
2    loop
3      for each cij in C
4        REVISE(xi, xj)
5        REVISE(xj, xi)
6      end for
7    until no domain is changed
8  end
```

If after the `AC-1` procedure is run any of the variable domains are empty, then we conclude that the network has no solution. Otherwise, we are guaranteed an arc-consistent network.

##### Complexity Analysis

Let $k$ bound the domain size as before and let $n=|X|$ be the number of variables and $e=|C|$ be the number of constraints. One cycle through all of the constraints (lines 3-6) takes $O(2\,e\,O_\text{REVISE}) = O(ek^2)$. In the worst case, only a single domain is changed in one cycle. In this case, the maximum number of repeats (line 7) will be the total number of values, $nk$. Therefore, the worst-case complexity of the `AC-1` procedure is $O(enk^3)$.

#### The `AC-3` Algorithm

Clearly, `AC-1` is straightforward to implement and generates an arc-consistent network, but at great expense. The question we must ask ourselves when using any brute-force method is: Can we do better?

A key observation about `AC-1` is that it processes all constraints even if only a single domain was reduced. This is unnecessary because changes in a domain typically only affect a local subgraph around the node in question.

The `AC-3` procedure is an improved version that maintains a queue of ordered pairs of variables that participate in a constraint (see lines 2-4). Each arc that is processed is removed from the queue (line 6). If the domain of the arc tail $x_i$ is revised, arcs that have $x_i$ as the head will need to be re-evaluated and are added back to the queue (lines 8-10).

```vhdl
 1  procedure AC3(csp)
 2    for each cij in C do
 3      Q ← Q ∪ {<xi,xj>, <xj,xi>};
 4    end for
 5    while Q is not empty
 6      select and delete any arc (xi,xj) from Q
 7      REVISE(xi,xj)
 8      if REVISE(xi,xj) caused a change in Di
 9        Q ← Q ∪ {<xk,xi> | k ≠ i, k ≠ j, ∀k }
10      end if
11    end while
12  end
```

##### Complexity Analysis

Using the same notation as before, the time complexity of `AC-3` is computed as follows. Building the initial `Q` is $O(e)$. We know that `REVISE` is $O(k^2)$ (line 7). This algorithm processes constraints at most $2k$ times since each time it is reintroduced into the queue (line 9), the domain of one of its associated variables has just been revised by at least one value, and there are at most $2k$ values. Therefore, the total time complexity of `AC-3` is $O(ek^3)$.

Note that the optimal algorithm has complexity $O(ek^2)$ since the worst case of merely verifying the arc consistency of a network requires $ek^2$ operations. There is an `AC-4` algorithm that achieves this performance by not using `REVISE` as a block box, but by exploiting the structures at the constraint level \cite{Dechter2003}.

### Example

Using our efficient CSP model (Attempt 2) from the previous section, consider the following 4-Queens problem, with the chessboard shown to the left and the corresponding constraint graph representation to the right. We have already placed the first queen in the first row, $x_1=1$.

We would like to use the `AC-3` algorithm to propagate constraints and eliminate inconsistent values in the domains of variables $x_2$, $x_3$ and $x_4$. Intuitively, we already know which values are inconsistent with our constraints (shown with $\times$ in the chessboard above). Follow the slides below to walk through the `AC-3` algorithm.

```python
ac3_slides = SlideController('images/4queens_slide%02d.png', 8)
```

Note how in this example, the efficiencies of `AC-3` were unnecessary. In fact, a single pass of `AC-2` would have achieved the same result. Although this was the case for this specific instance, by adding arcs back to the queue to by examined, `AC-3` is more computationally efficient in general.

---

## Search Methods

In the previous 4-Queens example, constraint propagation via `AC-3` was not enough to find a satisfying complete assignment to the CSP. In fact, if `AC-3` had been applied to the empty 4-Queens chessboard, no domains would have been pruned because all variables were already arc consistent. In these cases, we must assign the next variable a value by *guessing and testing*.

This trial and error method of guessing a variable and testing if it is consistent is formalized in **search methods** for solving CSPs. As mentioned previously, a simple state-space search would be intractable as the number of variables and their domains increase. However, we will first examine state-space search in more detail and then move to a more clever search algorithm called backtrack search (BT) that checks consistency along the way.

### Generic Search for CSPs

As we have studied before, a generic search problem can be specified by the following four elements: (1) state space, (2) initial states, (3) operator, and (4) goal test. In a CSP, consider the following definitions of these elements:
- state space
    - partial assignment to variables at the current iteration of the search
- initial state
    - no assignment
- operator
    - add a new assignment to any unassigned variable, e.g., $x_i = a$, where $a\in D_i$.
    - child extends parent assignments with new
- goal test
    - all variables are assigned
    - all constraints are satisfied

### Making Search More Efficient for CSPs

The inefficiency of using the generic state-space search approaches we have previously employed is caused by the size of the state space. Recall that a simple state-space search (using either breadth-first search or depth-first search) has worst case performance of $O(b^d)$, where $b$ is the branching factor and $d$ is the search depth, as illustrated below (from 16.410/413, Lecture 3).

In the above formulation of generic state-space search of CSPs, note that the branching factor is calculated as the sum of the maximum domain size $k$ for all variables $n$, i.e., $b = nk$. The search depth of a CSP is exactly $n$, because all variables must be assigned to be considered a solution. Therefore, the performance is exponential in the number of variables, $O([nk]^n)$.

This analysis fails to recognize that there are only $k^n$ possible complete assignments of the CSP. That is because the property of **commutativity** is ignored in the above formulation of CSP state-space search. CSPs are commutative because the order in which partial assignments are made do not affect the outcome. Therefore, by restricting the choice of assignment to a single variable at each node in the search tree, the runtime performance becomes only $O(k^n)$.

By combining this property with the idea that **extensions to inconsistent partial assignments are always inconsistent**, backtracking search shows how checking consistency after each assignment enables a more efficient CSP search.

### Backtracking Search

Backtracking (BT) search is based on depth-first search to choose values for one variable at a time, but it backtracks whenever there are no legal values left to assign. The state space is searched by extending the current partial solution with an assignment to unassigned variables. Starting with the first variable, the algorithm assigns a provisional value to each subsequent variable, checking value consistency along the way. If the algorithm encounters a variable for which no domain value is consistent with the previous assignments, a *dead-end* occurs. At this point, the search *backtracks* and the variable preceding the dead-end assignment is changed and the search continues. The algorithm returns when a solution is found, or when the search is exhausted with no solution.

#### Algorithm

The following recursive algorithm performs a backtracking search on a given CSP. The recursion base case occurs on line 3, which indicates the halting condition of the algorithm.

```vhdl
1  procedure backtrack(csp)
2    if csp.assignment is complete and feasible then
3      return assignment                                    ; recursion base case
4    end if
5    var ← csp.get_unassigned_var()
6    for next value in csp.var_domain(var)
7      original_domain = csp.assign(var, value)
8      if csp.assignment is feasible then
9        result ← backtrack(csp)
10       if result ≠ failure then
11         return result
12       end if
13     csp.restore_domain(original_domain)
14     end if
15    csp.unassign(var, value)
16    return failure
17  end
```

#### Example

We can apply the backtrack search algorithm to the N-Queens problem. Note that this simple version of the algorithm makes finding a solution tractable for a handful of queens, but there are other improvements that can be made that are discussed in the following section.

```python
queens, exec_time = nqueens_backtracking(4)
draw_nqueens([queens.assignment])
print("Solution found in %0.4f seconds" % exec_time)
```

### Branch and Bound

Suppose we would like to find the *best* solution (in some sense) to the CSP. This amounts to solving the associated constraint optimization problem (COP), where our constraint network is now a 4-tuple, $\langle X, D_X, C, f \rangle$, where $X\in D_X$, $C: D_X \to \{\operatorname{True},\operatorname{False}\}$ and $f: D_x\to\mathbb{R}$ is a cost function. We would like to find the variable assignments $X$ that solve
\begin{array}{ll@{}ll}
\text{minimize}  & f(X) &\\
\text{subject to}& C(X) &
\end{array}

By adding a cost function $f(X)$, we turn a CSP into a COP, and we can use the **branch and bound algorithm** to find the solution with the lowest cost.

To find a solution of a COP we could surely explore the whole tree and then pick the leaves with the smallest cost value. However, one may want to integrate the optimization process into the search process allowing to **prune** even if no inconsistency has been detected yet.

The main idea behind branch and bound is the following: if the best solution so far has cost $c$, this is a _lower bound_ for all other possible solutions. So, if a partial solution has led to costs of $x$ (cost so far) and the best we can achieve for all other cost components is $y$ with $x + y < c$, then we do not need to continue in this branch.

Of course every time we prune a subtree we are implicitly making the search faster compared with full exploration. Therefore with a small overhead in the algorithm, we can improve (in the average case) the runtime.

#### Algorithm

```vhdl
 1  procedure BranchAndBound(cop)
 2    i ← 1; ai ← {}                                 ; initialize variable counter and assignments
 3    a_inc ← {}; f_inc ← ∞                          ; initialize incumbent assignment and cost
 4    Di´ ← Di                                       ; copy domain of first variable
 5    while 1 ≤ i ≤ n+1
 6      if i = n+1                                   ; "unfathomed" consistent assignment
 7          f_inc ← f(ai) and a_inc ← ai             ; updated incumbent
 8          i ← i - 2
 9      else
10          instantiate xi ← SelectValueBB(f_inc)    ; Add to assignments ai; update Di
11          if xi is null                            ; if no value was returned,
12              i ← i - 1                            ;    then backtrack
13          else
14              i ← i + 1                            ; else step forward and
15              Di´ ← Di                             ;    copy domain of next variable.
16          end if
17      end if
18    end while
19    return incumbent X_inc and f_inc               ; Assignments exhausted, return incumbent
20  end
```

```vhdl
 1  procedure SelectValueBB(f_inc)
 2    while Di´ ≠ ∅
 3      select an arbitrary element a ∈ Di´ and remove a from Di´
 4      ai ← ai ∪ {xi = a}
 5      if consistent(ai) and b(ai) < f_inc
 6        return a;
 7      end if
 8    end while                                      ; no consistent value
 9    return null
10  end
```

#### Example

Now let's revive our discussion on the map coloring problem. Imagine that we work at a company that wishes to print a colored map of the United States, so they need to choose a color for each state. Let's also imagine that the available colors are:

```python
colors = [
  'red',
  'green',
  'blue',
  '#6f2da8', #Grape
  '#ffbf00', #Amber
  '#01796f', #Pine
  '#813f0b', #Clay
  '#ff2000', #yellow
  '#ff66cc', #pink
  '#d21f3c'  #raspberry
  ]
```

The CEO asks the engineering department (they have one of course) to find a color assignment that satisfies the constraints as specified above in _Map Coloring_ and they arrive at the following solution:

```python
map_colors, num_colors = us_map_coloring(colors)
draw_us_map(map_colors)
```

Unfortunately, management is never happy and they complain that {{ num_colors }} colors are really too many. Can we do better? Yes, by adding an objective function $f$ that gives a cost proportional to the number of used colors, we can minimize $f$. This results in the following solution:

```python
map_colors, opt_num_colors = us_map_coloring(colors, optimize=True)
draw_us_map(map_colors)
```

Fortunately we saved {{ num_colors - opt_num_colors }} color, well done!

---

## Extended Methods

The methods discussed in this section arise from viewing a CSP from different perspectives and from a combination of constraint propagation and search methods.

### BT Search with Forward Checking (BT-FC)

By interleaving inference from constraint propagation and search, we can obtain much more efficient solutions. A well-known way of doing this is by adding an arc consistency step to the backtracking algorithm. The result is called **forward checking**, which allows us to run search on graphs that have not already been pre-processed into arc consistent CSPs.

#### Algorithm

**Main Idea**: Maintain n domain copies for resetting, one for each search level i.

```vhdl
 1  procedure BTwithFC(csp)
 2    Di´ ← Di for 1 ≤ i ≤ n                  ; copy all domains
 3    i ← 1; ai = {}                          ; init variable counter, assignments
 4    while 1 ≤ i ≤ n
 5      instantiate xi ← SelectValueFC()      ; add to assignments, making ai
 6      if xi is null                         ; if no value was returned
 7        reset each Dk´ for k>i to
 8                its value before xi
 9                was last instantiated
10        i ← i - 1                           ; backtrack
11      else
12        i ← i + 1                           ; step forward
13      end if
14    end while
15    if i = 0
16      return "inconsistent"
17    else
18      return ai                             ; the instantiated values of {xi, ..., xn}
19  end
```

```vhdl
 1  procedure SelectValueFC()
 2    while Di´ ≠ ∅
 3      select an arbitrary element a ∈ Di´ and remove a from Di´
 4      for all k, i < k ≤ n
 5        for all values b ∈ Dk´
 6          if not consistent(a_{i-1}, xi=a, xk=b)
 7            remove b from Dk´
 8          end if
 9        end for
10        if Dk´ = ∅                          ; xi=a leads to a dead-end: do not select a
11          reset each Dk´, i<k≤n to its value before a was selected
12        else
13          return a
14        end if
15      end for
16    end while
17    return null
18  end
```
#### Example

The example code below runs a backtracking search with forward checking on the N-Queens problem. For the same value of $N$, note how a solution can be found much faster than without forward checking.

```python
queens, exec_time = nqueens_backtracking(4, with_forward_checking=True)
draw_nqueens([queens.assignment])
print("Solution found in %0.4f seconds" % exec_time)
```

### BT-FC with Dynamic Variable and Value Ordering

Traditional backtracking as it was introduced above uses a fixed ordering over variables and values. However, it is often better to choose ordering dynamically as the search proceeds. The idea is as follows. At each node during the search, choose:
- the most constrained variable; picking the variable with the fewest legal variables in its domain will minimize the branching factor,
- the least constraining value; choosing a value that rules out the smallest number of values of variables connected to the chosen variable via constraints will leave most options for finding a satisfying assignment.

These two ordering heuristics cause the algorithm to choose the variable that fails first and the value that fails last. This helps minimize the search space by pruning larger parts of the tree early on.

#### Example

The example code below demonstrates BT-FC with dynamic variable ordering using the most-constrained-variable heurestic. The run time cost of finding a solution to the N-Queens problem is lower than both BT and BT-FC, allowing the problem to be solved for even higher $N$.

```python
queens, exec_time = nqueens_backtracking(4, with_forward_checking=True, var_ordering='smallest_domain')
draw_nqueens([queens.assignment])
print("Solution found in %0.4f seconds" % exec_time)
```

### Adaptive Consistency: Bucket Elimination

Another method of solving constraint problems entails eliminating constraints through bucket elimination. This method can be understood through the lens of Gaussian elimination, where equations (i.e., constraints) are added and then extra variables are eliminated. More formally, these operations can be thought of from the perspective of relations as **join** and **project** operations.

Bucket elimination uses the join and projection operations on the set of constraints in order to transform a constraint graph into a single variable. After solving for that variable, other constraints are solved for by back substitution just as you would in an algebraic system in Gaussian elimination.

Using the map coloring problem where an `AllDiff` constraint exists between each neighboring variables, the join and project operators are explained. The constraint graph for the map coloring problem is shown below.

#### The Join Operator

The map coloring CSP can be trivially solved using the join operation on the constraints, which is defined as the consistent Cartesian product of the constraint relations.

Written as tables, the relations of each constraint $C_{12}$, $C_{23}$, and $C_{13}$ are

$C_{12}$	$C_{23}$	$C_{13}$

|$V_1$|$V_2$|
|-----|-----|
|  R  |  G  |
|  G  |  R  |
|  B  |  R  |
|  B  |  G  |

|$V_2$|$V_3$|
|-----|-----|
|  R  |  G  |

|$V_1$|$V_3$|
|-----|-----|
|  R  |  G  |
|  B  |  G  |

These constraint relation tables are then joined together as

$C_{12}\Join C_{23}$	$C_{13}$

|$V_1$|$V_2$|$V_3$|
|-----|-----|-----|
|  G  |  R  |  G  |
|  B  |  R  |  G  |

|$V_1$|$V_3$|
|-----|-----|
|  R  |  G  |
|  B  |  G  |

$C_{12}\Join C_{23}\Join C_{13}$

|$V_1$|$V_2$|$V_3$|
|-----|-----|-----|
|  B  |  R  |  G  |

#### The Projection Operator

The projection operator is akin to the elimination step in Gaussian elimination and is useful for shrinking the size of the constraints. After joining all the constraints in the above example, we can project out all constraints except for one to obtain the value of that variable.

For example, the projection of $C_{12}\Join C_{23}\Join C_{13}$ onto $C_1$ is

$C_2 = \Pi_2 (C_{12}\Join C_{23}\Join C_{13})$

|$V_1$|
|-----|
|  B  |

---

# Symmetries

 M. C. Escher

## Introduction

A CSP often exhibits some symmetries, which are mappings that preserve satisfiability of the CSP. Symmetries are particularly disadvantageous when we are looking for **all possible solutions** of a CSP, since search can revisit equivalent states over and over again.

\begin{definition} \label{def:symmetry}
(Symmetry). For any CSP instance $P = \langle X, D, C \rangle$, a solution symmetry of $P$ is a permutation of the set $X\times D$ that preserves the set of solutions to $P$.
\end{definition}

In other words, a solution symmetry is a bijective mapping defined on the set of possible variable-value pairs of a CSP that maps solutions to solutions.

### Why is symmetry important?

A principal reason for identifying CSP symmetries is to **reduce search efforts** by not exploring assignments that are symmetrically equivalent to assignments considered elsewhere in the search. In other words, if a problem has a lot of symmetric solutions of a small subset of non-symmetric solutions, the search tree is bigger and if we are looking for all those solutions, the search process is forced to visit all the symmetric solutions of the big search tree. Alternatively, if we can prune-out the subtree containing symmetric solutions, the search effort will reduce drastically.

### Case Study: symmetries in N-Queens problem

We have already seen the N-Queens problem. Let us see all the solutions of a $4 \times 4$ chessboard.

```python
queens = nqueens(4)
draw_nqueens(queens, all_solutions=True)
```

There are exactly 2 solutions.

It's easy to notice the two are the same solution if we flip (or rotate) the chessboard.

### Interactive examples

All the following code snippets are a refinement of the original N-Queens problem where we modify the problem to reduce the number of symmetries. Feel free to explore how the number of solutions to the N-Queens problem changes when we change symmetry breaking strategy and $N$.

You can use the following slider to change $N$, than press the button `Update cells...` to quickly update the results of the models.

```python
n = 5
def update_n(x):
    global n
    n = x
interact(update_n , x=widgets.IntSlider(value=n, min=1,max=12,step=1, description='Queens:'));
```

```python
## Update all cells dependent from the slider with the following button
button = widgets.Button(description="Update cells...")
display(button)
button.on_click(autoupdate_cells)
```

## Avoid symmetries

### Adding Constraints Before Search

In practice, symmetry in CSPs is usually identified by applying human insight: the programmer sees that some transformation would translate a hypothetical solution into another hypothetical solution. Then, the programmer can try to formalize some constraint that preserves solutions but removes some of the symmetries.

For $N$ = {{n}} the N-Queens problem has {{ len(nqueens(n)) }} solutions. One naive way to remove some of the symmetric solutions is to restrict the position for some of the queens, for example, we can say that the first queen should be on the top half of the chess board by imposing an additional constraint like

```
constraint queens[0] <= n div 2;
```

This constraint should remove approximately half of the symmetries. Let's try the new model!

```python
%%minizinc --all-solutions --statistics -m bind
include "globals.mzn";
int: n;
array[0..n-1] of var 0..n-1: queens;

constraint all_different(queens);
constraint all_different([queens[i]+i | i in 0..n-1]);
constraint all_different([queens[i]-i | i in 0..n-1]);

constraint queens[0] <= n div 2;

solve satisfy;
```

If you play with $N$ you will notice that for $N=4$ all solutions are retained. However, For $N>4$ symmetric solutions will begin to be pruned out.

This approach is fine and if done correctly it can greatly reduce the search space. However, this additional constraint can lose solutions if done incorrectly.

To address the problem in a better way we need some formal tool.

### Chessboard symmetries

Looking at the chessboard, we notice that it has eight geometric symmetries---one for each geometric transformation. In particular they are:

- identity (no-reflections) $id$ (we always include the identity)
- horizontal reflection $r_x$
- vertical reflection $r_y$
- reflections along the two diagonal axes ($r_{d_1}$ and $r_{d_2}$)
- rotations through $90$°, $180$° and $270$° ($r_{90}$, $r_{180}$, $r_{270}$)

If we label the sixteen squares of a $4 \times 4$ chessboard with the numbers 1 to 16, we can graphically see how symmetries move cells.

Now it's easy to see that a symmetry is a **permutation** that acts on a point: for example, if a queen is at $(2,1)$ (which correspondes to element $2$ in $id$), under the mapping $r_{90}$, it moves to $(4,2)$.

One useful form to write a permutation is in _Cauchy form_, for example for $r_{90}$

\begin{equation}
r_{90} : \left( \begin{array} { c c c c c c c c c }
1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 & 12 & 13 & 14 & 15 & 16\\
13 & 9 & 5 & 1 & 14 & 10 & 6 & 2 & 15 & 11 & 7 & 3 & 16 & 12 & 8 & 4
\end{array} \right)
\end{equation}

What this notation says is that an element in position $i$ in the top row, is moved to the corresponding position of the bottom row. For example $1$ → $13$, $2$ → $9$, $3$ → $5$ and so on.

This form will help us compactly write constraints to remove unwanted permutations.

### The Lex-Leader Method

Puget proved that whenever a CSP has symmetry that can be expressed as permutations of the variables, it is possible to find a _reduced form_ with the symmetries eliminated by adding constraints to the original problem  \cite{Puget2003}. Puget found such a reduction for three simple constraint problems and showed that this reduced CSP could be solved more efficiently than in its original form.

The intuition is rather simple: for each equivalence class of solutions (permutation), we predefine one to be the **canonical solution**. We achieve this by choosing a static variable ordering and imposing the **lexicographic order** for each permutation. This method is called **lex-leader**.

For example, let us consider a problem where we have three variables $x_1$, $x_2$, and $x_3$ subject to the `alldifferent` constraint and domain {A,B,C}. This problem has $3!$ solutions, where $3!-1$ are symmetric solutions. Let's say that our canonical solution is `ABC`, and we want to prevent `ACB` from being a solution, the lex-leader method would impose the following additional constraint:

$$ x_1\,x_2\,x_3 \preceq_{\text{lex}} x_1\,x_3\,x_2. $$

In fact, if $x = (\text{A},\text{C},\text{B})$ the constraint is not satisfied, written as

$$ \text{A}\text{C}\text{B}\,\, \npreceq_{\text{lex}} \text{A}\text{B}\text{C}. $$

Adding constraints like this for all $3!$ permutations will remove all symmetric solutions, leaving exactly one solution (`ABC`). All other solutions can be recovered by applying each symmetry.

In general, if we have a permutation $\pi$ that generates a symmetric solution that we wish to remove, we would impose an additional constraint, usually expressed as

$$ x_1 \ldots x_k \preceq_{\text{lex}} x_{\pi (1)} \ldots x_{\pi (k)},  $$

where $\pi(i)$ is the index of the variable after the permutation.

Unfortunately, for the N-Queens problem formulated as we have seen, this technique does not immediately apply, because some of its symmetries cannot be described as permutations of the `queens` array.

The trick to overcoming this limitation is to express the N-Queens problem in terms of Boolean variables for each square of the chessboard that model whether it contains a queen or not (i.e., Attempt 1 from above). Now all the symmetries can be modeled as permutations of this array using Cauchy form.

Since the main constraints of the N-Queens problem are much easier to express with the integer `queens` array, we use both models together connecting them using _channeling constraints_.

```python
%%minizinc --all-solutions --statistics -m bind
include "globals.mzn";

int: n;
array[0..n-1,0..n-1] of var bool: qb;
array[0..n-1] of var 0..n-1: q;

constraint all_different(q);
constraint all_different([q[i]+i | i in 0..n-1]);
constraint all_different([q[i]-i | i in 0..n-1]);

constraint % Channeling constraint
    forall (i,j in 0..n-1) ( qb[i,j] <-> (q[i]=j) );

constraint % Lexicographic symmetry breaking constraints
    lex_lesseq(array1d(qb), [ qb[j,i] | i in reverse(0..n-1), j in 0..n-1 ]) /\ % r_{90}
    lex_lesseq(array1d(qb), [ qb[i,j] | i,j in reverse(0..n-1) ]) /\            % r_{180}
    lex_lesseq(array1d(qb), [ qb[j,i] | i in 0..n-1, j in reverse(0..n-1) ]) /\ % r_{270}
    lex_lesseq(array1d(qb), [ qb[i,j] | i in reverse(0..n-1), j in 0..n-1 ]) /\ % r_{x}
    lex_lesseq(array1d(qb), [ qb[i,j] | i in 0..n-1, j in reverse(0..n-1) ]) /\ % r_{y}
    lex_lesseq(array1d(qb), [ qb[j,i] | i,j in 0..n-1 ]) /\                     % r_{d_1}
    lex_lesseq(array1d(qb), [ qb[j,i] | i,j in reverse(0..n-1) ]);              % r_{d_2}

solve satisfy;
```

In this model the constraint `lex_lesseq(array_1, array_2)` implements the lexicographic operator $\preceq_{\text{lex }}$ between `array_1` and `array_2`. Notice that `array_2` represent the permutation fo each of the geometric symmetry of the chessboard (except the identity).

Using the lex-leader method we reduced the number of solutions but we also added a lot of constraints...

### Double-Lex

When dealing with a matrix of decision variables, we often have that any permutation of the rows (or columns) of a solution is also a solution. This class of symmetries is called _row and column symmetries_.

We can certainly use the lex-leader method to break all symmetries, but in an $n \times m$ matrix with rows and column symmetry we would add $n!m!$ constraints. Adding so many constraints can be counter-productive.

When breaking all symmetries proves too difficult, it is often possible to achieve good results by breaking a smaller
set of symmetries. One method to do this for row and column symmetries is **double-lex** (Flener et al. 2002). The idea is to impose the ordering on the rows and on the columns **independently**. This produces only $n + m − 2$ symmetry breaking constraints.

One example where the double-lex can be applied is the problem seen during the course assignments: _Buddies_. In that problem, we could permute each element on each row (i.e., seat assignment) independently while preserving the same solution. Similarly, we could also permute each column independently (i.e., swap the 20-minute segments). This is a typical case where the double-lex is effective and cheap to implement.

The double-lex method is not applicable to the N-Queens problem, because not all column (or row) permutations preserve the solution.

## Symmetry breaking constraints

### Soundness and completeness

Two important properties of symmetry breaking constraints are **soundness** and **completeness**, a set of symmetry breaking constraints
- is **sound** if and only if it leaves at least one solution in each symmetry class
- is **complete** if and only if it leaves at most one solution in each symmetry class

All the approaches we used so far in the N-Queens problem are sound and complete since they leave at least one solution in the only symmetry class available (geometric symmetries). Other problems might have different symmetry classes and it is very important that constraints added to remove a given symmetry don't remove desirable solutions from the problem.

### Intractability of Breaking Symmetry

It is worth mentioning that lex-leader requires one constraint for each element of the group.
In the case of a matrix with $m$ rows and $n$ columns, rows and column symmetry, this is $m!n!$, which is impractical in general. Therefore there are many cases where lex-leader is applicable but impractical.

\begin{theorem} \label{theo:simple_ordering_NP}
(Walsh 2011) Given any _simple_ ordering, there exists a symmetry group such that deciding if an assignment is smallest in its symmetry class according to this ordering is NP-hard.
\end{theorem}

In other words, Walsh proved that breaking symmetry completely by adding constraints to eliminate symmetric solutions is computationally intractable in general. More specifically, he proves that given any simple variable ordering, deciding if an assignment is the smallest in its symmetry class is NP-hard.

An alternative to full symmetry breaking is to break some symmetry by using just a subset of the lex-leader constraints, for example, the double-lex.

\begin{theorem} \label{theo:lex2_NP}
(Katsirelos, Narodytska, and Walsh 2010) Propagating the double-lex constraint is NP-hard.
\end{theorem}

In other words, this theorem states that there is no efficient algorithm to restrict variable domain after an assignment.

Since symmetry breaking appears intractable in general, a major research direction is to identify special cases where the symmetry group is more tractable in practice.

## Reducing the Set of Symmetry Breaking Constraints

Lex-leader constraints can be simplified to remove redundancies. For example, imagine having the following lexicographic constraint:

$$ x_1\,x_2\,x_3\,x_4\,x_5,\,x_6 \preceq_{\text{lex}}x_1\,x_3\,x_2\,x_4\,x_6,x_5. $$

We can remove the first and fourth variables from each tuple, since clearly $x_1 = x_1$ and $x_4 = x_4$, obtaining

$$ x_2\,x_3\,x_5,\,x_6 \preceq_{\text{lex}}x_3\,x_2\,x_6,x_5. $$

But we can also notice that if $x_2 < x_3$ the constraint is satisfied no matter the other values, and otherwise we have $x_2 = x_3$ to satisfy the constraint. In other words, if the second variables in the tuples are relevant, they must be equal. Similarly for $x_5$ and $x_6$. Thus, the constraint is equivalent to

$$ x_2\,x_5\preceq_{\text{lex}}x_3\,x_5. $$

Since $\preceq_{\text{lex}}$ is transitive we can go further, treating the constraints as a set and not just individually. This would help us to reduce the size of the constraint even more.

Unfortunately, the approach outlined here does not get around the fundamental problem of the exponential number of symmetries. However, the approach does illustrate how the set of constraints can be simplified, and we will see in the next section a particular case where the results are quite dramatic.

### Lex constraint decomposition

We can decompose the lex constraint of the form $x_1\ldots x_2 \preceq_{\text{lex }} y_1\ldots y_2$ to a conjunction of clauses like
$$ \left(x_1 = y_1 \right),\ldots,\left(x_k = y_k\right)\rightarrow x_{k+1}\leq y_{k+1} $$
We call clauses of this form lex implications.

In practice many lex-implications are redundant \cite{Codish2018}. Given this observation, we can ask ourselves the direct question: how many lex implications are required to express a complete symmetry break?

### Reduce implications

In a recent paper \cite{Codish2018}, the authors develop a method to find a complete and compact set of symmetry breaking constraints. Basically, the algorithm iterates over the set of lex implications, and checks for each of them if they are redundant.

Define $\phi$ to be the set of constraints expressed as a Boolean formula, and $\psi$ to be the set of lex implications used to break the symmetries in the solution space defined by $\phi$.

Given these two sets, the idea behind the reduction is quite intuitive: remove one clause from the formula, and check if there is a solution which would be forbidden by this clause. If this is not the case, the clause is redundant and can be removed.

We note that the number of clauses which can actually be removed depends on the order in which clauses are checked, thus the reduction splits in two phases:

- The first phase is shown in Algorithm 1. We rank the clauses by checking if a clause $c\in\phi$ is redundant. If so, we compute a subset $\psi\subseteq\varphi^\prime$ of clauses which makes $c$ redundant, and increase the ranking of all clauses within this set.

The rationale is that removing these clauses is more likely make other clauses no longer redundant, and so increase the size of the final symmetry break.

- The second stage is shown in Algorithm 2. We sort the clauses by ranking, so clauses which were frequently the cause of redundancy appear as late as possible. Then remove the clauses if there is not a solution which would be forbidden.

This approach can be applied to any lexicographic-based ordering.

### Example

In their paper \cite{Codish2018} show some result of their method applied to a generic matrix model

    n
    Double-Lex
    All Permutations

    Original
    Reduced
    Original
    Reduced

    3
    12
    12
    48
    13

    4
    24
    24
    312
    32

    5
    40
    40
    2440
    71

    6
    60
    60
    21660
    148

    7
    84
    84
    211764
    310

It is interesting to note that with the complete symmetry break it is possible to reduce the number of implications quite drastically but double-lex has no redundant implications.

### Conclusion

Removing redundant clauses can be costly due to the use of a SAT solver. However, if an instance of the same size needs to be solved several times but with different values it can greatly improve the search time since the reduction can be done offline just once.

# References

(Dechter, 2003) Rina Dechter, ``_Constraint Processing_'',  2003.

(Russell and Norvig, 2003) Stuart J. Russell and Peter Norvig, ``_Artificial Intelligence: A Modern Approach_'',  2003.

(Mackworth, 1977) Mackworth Alan K., ``_Consistency in Networks of Relations_'', Artif. Intell., vol. 8, number 1, pp. 99--118,  1977.  [online](http://dx.doi.org/10.1016/0004-3702(77)90007-8)

(Puget, 1993) J.F. Puget, ``_On the satisfiability of symmetrical constrained satisfaction problems_'', Methodologies for Intelligent Systems,  1993.

(Codish, Ehlers et al., 2018) M. Codish, T. Ehlers, G. Gange et al., ``_Breaking Symmetries with Lex Implications_'', FLOPS,  2018.


```python
# Notebook imports, do not care about them!
import jdc
```

# Introdução

Redes Neurais de Múltiplas Camadas (RNM) são modelos computacionais inspirados no sistema nervoso humano, compostas por estruturas matemáticas que simulam os neurônios e suas conexões. Os usos mais comuns desses modelos consistem em tarefas de classificação e regressão,
o que favorece a aplicação em diversas áreas de pesquisa, sendo uma delas
a de Visão Computacional.

O objetivo deste *notebook* é apresentar os fundamentos das RNM *feedforward fully connected*, também conhecidas como Multilayer Perceptron (MLP), dos pontos de vista matemático e algorítmico, gerando uma implementação
básica de uma MLP em Python. O tratamento matemático foi baseado no excelente livro de Michael Nielsen [1] sobre este tema. As implementações foram adaptadas do também excelente livro de Adrian Rosebrock [2], porém com severas alterações para que houvesse clara correspondência entre a notação matemática e o código. Certamente, a implementação não foi otimizada como poderia (e como o é em bibliotecas especializadas), por propósitos didáticos.

Para compreender as RNM, é importante primeiramente conhecer os seus dois principais componentes: os **neurônios** e a **arquitetura em camadas**. Depois, trataremos sobre o aprendizado utilizando o método do *Gradiente Descendente Estocástico* com o algoritmo *Backpropagation*.

## Neurônios
Um neurônio, no contexto das redes neurais, é uma estrutura que aceita como argumento um vetor $\mathbf{x} = \langle x_1, x_2, \ldots, x_n \rangle \in \mathbb{R}^n$
de entrada e **responde com um valor real** de saída, comumente chamado de **valor de ativação**. Cada neurônio está relacionado a um valor $b \in \mathbb{R}$, chamado **bias**, e a um vetor de pesos $\mathbf{w} = \langle w_1, w_2, \ldots, w_n \rangle \in \mathbb{R}^n$, de forma que cada componente $x_i$ da entrada é associada ao peso $w_i$. Ao receber a entrada,
o neurônio responde com o valor $f(\mathbf{x}\cdot\mathbf{w} + b)$, onde $f$ é chamada de
**função de ativação** e $\mathbf{x}\cdot\mathbf{w} = \sum_i x_i \cdot w_i$, ou seja,
o produto interno entre $\mathbf{x}$ e $\mathbf{w}$, também chamado de $net$ do neurônio.

A contante $b$, o *bias*, existe para permitir um ajuste fino
do valor de saída por meio de um deslocamento horizontal da função de ativação. Por essa afirmação,
é de se esperar que $b$ seja ajustado na preparação do modelo, o que de fato acontece:
o *bias* é aprendido pelas redes neurais, juntamente com os pesos.

Uma função comum de ativação é a **sigmoide**, expressa pela equação:
$$f(x) = \frac{1}{1 + e^{-x}}.$$

Ela possui a importante propriedade de ser contínua e derivável em $(-\infty, +\infty)$,
o que favorece o algoritmo de treinamento a ser em breve explicado, apesar de existirem algumas desvantagens em sua utilização.

Abaixo, estão os gráficos sem e com *bias* da função sigmoide. Perceba o deslocamento horizontal causado
pelo *bias*:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# The sigmoid function
def f(X):
    return 1/(1+np.exp(-X))

# Values for the X axis
X = np.arange(-10,10,.1)
# Bias value
b = -2.5

# Plotting without bias
plt.subplot(211)
plt.plot(X,f(X))
plt.title("Unbiased")
plt.grid(True)

# Plotting with bias
plt.subplot(212)
plt.plot(X,f(X+b))
plt.title("Biased")
plt.grid(True)

plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.5,
                    wspace=0.35)
plt.show()
```

## Arquitetura em camadas

Uma RNM é organizada em uma sequência de $C > 2$ camadas de neurônios, denotadas aqui por $L_0, L_1, \ldots, L_{C-1}$. O tamanho de uma camada $L_i$, denotado aqui por $tam\;L_i$, é a quantidade de neurônios que ela possui. A camada $L_0$ é dita de entrada, na qual existe um **neurônio de entrada** para cada componente do vetor de entrada da rede. Esses neurônios se diferenciam dos demais porque,
ao invés de aceitarem todo o vetor, recebem apenas uma componente e simplesmente a **transmitem** para o interior da rede, sem
a aplicação da função de ativação. Já a camada $L_{C-1}$ é dita de saída,
e a sua resposta é a resposta da rede. As demais camadas são chamadas de **camadas escondidas**.

O que viabiliza o funcionamento do modelo são as conexões entre essas camadas, e elas podem ocorrer de diversas maneiras na rede. Aqui,
o objeto de estudo são as **redes densas *feedforward***, em que cada neurônio da camada $L_i$ se conecta com
todos os neurônios da camada $L_{i+1}$, $i = 0,\ldots, C-2$, de maneira que o vetor de entrada dos neurônios da camada $L_{i+1}$ é o vetor composto pelas
respostas dos neurônios da camada $L_{i}$. Assim, um neurônio na camada $L_{i+1}$ recebe, como entrada, um vetor
de dimensão $tam\;L_i$, $i=0,1, \ldots, C-2$.

Cada conexão entre neurônios pode ser vista como associada ao peso que o neurônio
atribui ao valor por ela provido. Essa visão possibilita a utilização de uma
notação muito útil para se trabalhar com pesos: denota-se por $w_{kj}^l$ o peso
da conexão entre o neurônio $j$ da camada $l-1$ com o neurônio $k$ da camada $l$,
$l = 1, \ldots, C-1$. Mais ainda, é possível representar todas as conexões entre
duas camadas por uma matriz de pesos $\mathbf{W}^l = (w_{kj}^l)$ de dimensão $tam\;L_{l} \times tam\;L_{l-1}$.
Para facilitar a identificação dos *biases* de cada neurônio em uma camada, denota-se por
$\mathbf{b}^l = \langle b^l_1, b^l_2, \ldots, b^l_{tam\;L_l} \rangle$ os *biases*
dos neurônios da camada $L_l$, sendo $b^l_i$ o *bias* do neurônio $i$ na camada $L_l$.

Sabendo disso, já é possível iniciar a implementação em Python da RNM!

### A classe `MultilayerNeuralNetwork`

Como o objetivo é
produzir um módulo reutilizável, os recursos de orientação a objetos da linguagem Python
serão aplicados. Além disso, o pacote `numpy` será utilizado para a manipulação de matrizes.

A programação tem início com a importação do `numpy` (ele foi importado antes, para o gráfico da sigmoide, mas repetimos aqui por questões didáticas) e a declaração da classe `MultilayerNeuralNetwork`, a qual encapsulará todos os dados e métodos necessários ao
funcionamento e à utilização da rede, como a matriz de pesos, a arquitetura e as rotinas de treinamento:

```python
import numpy as np
import random

class MultilayerNeuralNetwork:
    '''A Multilayer Dense Feedforward Neural Network implementation.'''
```

Essa é apenas uma classe, e nada tem a ver com redes neurais além do nome simplesmente. O próximo passo é criar um **construtor**, método que sempre será chamado quando alguém
criar (ou, mais tecnicamente, instanciar) a rede neural. Nele, é importante garantir
a criação e inicialização da arquitetura, o que implica na configuração das camadas (quantas e quantos neurônios devem possuir), das matrizes de pesos e dos vetores de *bias*.  Estas serão inicializadas com valores aleatórios, seguindo
a distribuição normal, e, em seguida, normalizadas pela raiz quadrada da quantidade de neurônios da camada correspondente. A inicialização dos pesos é um tópico importante, para o qual existem diversas propostas, mas que será abstraído neste momento por economia de tempo.
Além disso, um parâmetro `alpha` será acrescentado,
cuja existência será justificada mais adiante. Portanto, o construtor receberá do
usuário a **arquitetura da rede**, no formato $[tam\;L_0, tam\;L_1, \ldots, tam\;L_{k-1}]$, e o tal `alpha`:

```python
    %%add_to MultilayerNeuralNetwork
    def __init__(self, arch = [1,2,1], alpha = 0.1):
        '''Initialize the network.'''

        # A dictionary to store the weight matrices
        self.W = {}
        # A dictionary to store the biases vectors
        self.B = {}
        # The learning rate (to be explained later)
        self.alpha = alpha
        # The arch answers the questions: how many layers and how many neurons in each layer.
        self.arch = arch

        # Initialize the weight matrix and biases with normalized random values
        for i in np.arange(1,len(self.arch)):
            # Weights
            w = np.random.randn(self.arch[i], self.arch[i-1])
            self.W[i] = w/np.sqrt(self.arch[i])
            # Biases
            b = np.random.randn(self.arch[i],1)
            self.B[i] = b/np.sqrt(self.arch[i])
```

Como será utilizada a função sigmoide para a ativação dos neurônios, é importante
que ela e sua derivada estejam implementadas na classe:

```python
    %%add_to MultilayerNeuralNetwork
    def sigmoid(self, x):
        '''Sigmoid function.'''
        return 1.0/(1 + np.exp(-x))

    def sigmoid_deriv(self, x):
        '''Derivative of the sigmoid function, considering that x is the result
        of applying the sigmoid function to the net.
        '''
        return x * (1 - x)
```

Com isso, é possível instanciar uma rede, mesmo que não se possa fazer nada com ela ainda, apenas para checar a sua estrutura:

```python
# Creating an architecture with 2 neurons in the input layer, 3 in the unique hidden layer
# and 1 in the output layer
neuralnet = MultilayerNeuralNetwork(arch = [2,3,1], alpha = 0.5)
print(neuralnet.W)
print(neuralnet.B)
```

### Respostas das camadas

Note que cada camada produz um vetor de resposta composto pelas respostas individuais
de cada um de seus neurônios. Denote por $\mathbf{a^{l-1}} = \langle a^{l-1}_1, a^{l-1}_2, \ldots, a^{l-1}_{tam\; L_{l-1}} \rangle$ esse vetor para a camada $L_{l-1}$. Como visto, ele deverá ser a entrada para a camada seguinte, $L_l$, mas os seus valores serão combinados de forma particular
pela matriz de pesos $\mathbf{W}^{l}$. A expressão

$$\mathbf{z}^l = \mathbf{W}^{l}\cdot\mathbf{a}^{l-1} + \mathbf{b}^l$$

produz um vetor cujos componentes são os valores $net$ de cada neurônio da camada
$L_{l+1}$. Sabendo disso, qual seria a expressão para o vetor de saída da camada $L_{l+1}$, ou seja, para $\mathbf{a}^{l+1}$?
Basta aplicar a função de ativação a cada componente do vetor obtido! Considerando-se a aplicação da função ponto a ponto, ou seja, $f(\langle x_1, x_2, \ldots, x_n \rangle) = \langle f(x_1), f(x_2), \ldots, f(x_n) \rangle$, tem-se que:

$$\mathbf{a}^{l} = f(\mathbf{W}^{l}\cdot\mathbf{a}^{l-1} + \mathbf{b}^l) = f(\mathbf{z}^l).$$

O conhecimento adquirido até este ponto é suficiente para se entender a arquitetura de uma rede neural e como computar as saídas das suas camadas de neurônios a partir
de um vetor de entrada. Já podemos implementar uma função muito importante da nossa rede neural: dado um vetor de entrada, qual é o vetor de resposta da rede, ou seja, qual o vetor na camada de saída? Essa função é comumente chamada de `predict`, pois é utilizada para gerar a previsão (resposta) da rede a partir de uma entrada. Note como é simples e advém diretamente da definição:

```python
    %%add_to MultilayerNeuralNetwork
    def predict(self, X):
        ''' Given a matrix X of input row vectors, compute the output of the network. '''
        # Set the result of the input layer to be the column vectors from X, transposed
        a = np.atleast_2d(X).T
        # Compute the output, passing through each layer
        for layer in np.arange(1, len(self.arch)):
            a = self.sigmoid(np.dot(self.W[layer], a) + self.B[layer])
        return a
```

Ocorre que apenas saber como obter a saída da rede não resolve problema algum: as matrizes de pesos aleatórios
tornam o modelo inútil. Ele necessita se ajustar (lê-se "aprender")
para solucionar os problemas com os quais é confrontado, e é disso que
trataremos a partir de agora!

# Aprendizado em uma RNM

O modelo de aprendizagem de redes neurais é geralmente o **supervisionado**, pois faz
uso de exemplos representantes da verdade (*ground truth*) para fazer
com que a rede gere as respostas desejadas após uma **etapa de treinamento**. É como se um professor
apresentasse a entrada, a rede respondesse e ele informasse qual foi o
erro cometido. A rede, com base nisso, modifica sua estrutura (suas matrizes
 de pesos) para que o erro, da próxima vez que o professor mostrar
 o exemplo, seja garantidamente menor. Esse processo termina quando algum critério de parada é atingido. Os mais comuns são o erro máximo e o número de ciclos ou *epochs* (percorrimentos do conjunto inteiro de treino).

 Um problema a ser resolvido com aprendizado de máquinas geralmente provê um conjunto com $N$ exemplos

 $$\mathcal{D} = \{\langle \mathbf{x_1}, \mathbf{y_1} \rangle\, \langle \mathbf{x_2}, \mathbf{y_2} \rangle\, \ldots, \langle \mathbf{x_N}, \mathbf{y_N} \rangle\}$$

 sendo $\mathbf{x_i} \in \mathbb{R}^d$ os vetores de entrada da rede e $\mathbf{y_i} \in \mathbb{R}^e$ a
 resposta esperada para a rede, também chamada de ***target*** ou **classe**.

 É prática comum
 particionar esse conjunto em dois outros: um para o treinamento da rede, denotado por $\mathcal{T}$,
 e outro para o teste do modelo treinado, denotado por $\mathcal{T^*}$. Tamanhos comuns para o conjunto de treino são $65\%$, $70\%$ ou $75\%$ de $|\mathcal{D}|$. A biblioteca `sklearn`, em Python,
 provê a função `train_test_split`, que realiza esse particionamento.

 O erro cometido pela rede, ao responder para dado exemplo $\langle \mathbf{x}, \mathbf{y} \rangle \in \mathcal{D}$, é modelado matematicamente por uma **função de custo**
 $C_x:\mathbb{R}^{\omega+\beta} \to \mathbb{R}$, onde $\omega$ é a quantidade de pesos (como calcular $\omega$ a partir da arquitetura?) e $\beta$, a de *biases*. As funções que podem desempenhar esse papel devem cumprir alguns requisitos
 básicos e dependem dos valores dos pesos e dos *biases*. Uma das mais comuns é a do **erro quadrático**, de equação:

 $$C_x = \frac 1 2 ||\mathbf{y}- \mathbf{a}^{k-1}(\mathbf{x})||^2 = \frac 1 2 \sum_i (y_i - a^{k-1}_i(\mathbf{x}))^2,$$
  onde $a^{k-1}(\mathbf{x})$ é a saída da última camada da rede para o exemplo $\langle \mathbf{x}, \mathbf{y} \rangle$ (que nossa rede já sabe computar). Note que,
 quanto maior a diferença entre a verdade e a saída da rede, maior o valor dessa
 função, o que mostra sua adequação para representar o erro.

 A fim de representar o erro com respeito a todos os exemplos, define-se também a função
 de custo $\bar{C}:\mathbb{R}^{\omega+\beta} \to \mathbb{R}$, dada pela média dos erros para cada exemplo, ou seja:

 $$
     \bar{C} = \frac 1 n \sum_{\langle \mathbf{x}, \mathbf{y} \rangle \in \mathcal{T}} C_x = \frac{1}{2n} \sum_{\langle \mathbf{x}, \mathbf{y} \rangle \in \mathcal{T}} ||\mathbf{y}- \mathbf{a}^{k-1}(\mathbf{x})||^2.
 $$

 Precisaremos, em nossa implementação, do cálculo da função de custo quadrática. Façamos, pois:

```python
    %%add_to MultilayerNeuralNetwork
    def quadratic_loss(self, X, Y):
        ''' Compute the total quadratic loss, given a matrix with row vectors. '''
        # Transform all target line vectors in column vectors
        Y = np.atleast_2d(Y).T
        # Generate predictions
        predictions = self.predict(X)
        # Compute loss
        n = X.shape[0]
        loss = (1/n) * 0.5 * np.sum((predictions - Y) ** 2)
        return loss
```

 O objetivo do treinamento é fazer com que o valor de $C_x$ (e $\bar{C}$, consequentemente), para cada exemplo $\langle \mathbf{x}, \mathbf{y} \rangle$,
 seja menor a cada ciclo de treino. Como $C_x$ é função dos pesos e dos *biases*, nada melhor que buscar ajustá-los a fim de alcançar esse objetivo. Esse ajuste ocorre por meio de algum
 **método de otimização**, cujo objetivo é encontrar um conjunto de valores para pesos e *biases* tal que o valor de $\bar{C}$ seja tão menor quanto se queira. Um método
 muito utilizado para tanto chama-se **gradiente descendente**, do qual a próxima seção tratará.

 ## Gradiente descendente

 Fujamos do escopo das redes neurais, e tratemos do problema geral de minimizar uma função
 de múltiplas variáveis $C:\mathbb{R}^n \to \mathbb{R}$. Queremos encontrar o vetor $\mathbf{v} = \langle v_1, v_2, \ldots, v_n \rangle$ tal que $C(\mathbf{v}) = C(v_1, v_2, \ldots, v_n)$ seja um mínimo global de $C$.

 Para conseguir a intuição sobre o método, vale reduzir a dimensão de entrada de $C$ para
 duas variáveis, $v_1$ e $v_2$, e imaginar que seu gráfico assuma a forma de um vale. Por exemplo:

```python
from mpl_toolkits import mplot3d

def quad(x,y):
    return x**2 + y**2

x = np.linspace(-10,10,30)
y = np.linspace(-10,10,30)

X,Y = np.meshgrid(x,y)

Z = quad(X,Y)

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.plot_surface(X,Y,Z,rstride=1, cstride=1, cmap='viridis', edgecolor='none')
ax.set_title('Quadratic plot')
```

A função acima é bem simples, devemos concordar, e, apenas observando seu gráfico, é possível encontrar o mínimo. Também, utilizando técnicas primárias de cálculo, é possível encontrar o mínimo global sem muito esforço nesse caso. Porém, quando lidamos com aprendizado, geralmente estamos interessados em minimizar funções de dezenas, centenas, milhares de variáveis! É por isso que as técnicas tradicionais de cálculo se tornam inviáveis, e outros métodos entram em cena.

O gradiente descendente, sendo um deles, não busca encontrar um mínimo global, mas sim **algum mínimo para a função**. Isso pode ser um problema em alguns casos, mas não se mostra um empecilho grave para o seu uso em muitas aplicações de redes neurais. Imagine uma bola colocada em algum ponto arbitrário do vale mostrado no gráfico acima. Nossa experiência diz que ela começará a rolar até o ponto mais baixo do vale, e esse fato nos dá uma heurística para encontrar um mínimo da função: encontremos uma maneira de mover a bola nas direções
$v_1$ e $v_2$ de forma que ela sempre esteja descendo o vale!

Já que queremos mover a bola, vamos representar o deslocamento por um vetor $\Delta v = \langle \Delta v_1, \Delta v_2 \rangle$. Assumindo a bola inicialmente na posição $v = \langle v_1, v_2 \rangle$, qual será a mudança no valor de $C$ causada por esse deslocamento? Ou seja, como podemos estimar o valor de $\Delta C = C(v+\Delta v)-C(v)$?
Da fórmula de Taylor para o caso Multivariável, temos que

$$\Delta C \approx \nabla C(v) \cdot \Delta v = \left \langle \frac{\partial C}{\partial v_1}, \frac{\partial C}{\partial v_2} \right \rangle \cdot \langle \Delta v_1, \Delta v_2 \rangle ^T = \frac{\partial C}{\partial v_1}\Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2.$$

Queremos reduzir o valor de $C$, correto? Para isso, devemos escolher um valor de $\Delta v$ apropriado, tal que $\Delta C < 0$. Pela equação acima, isso se torna muito fácil! Vamos tomar $\Delta v$ como

$$\Delta v = -\alpha \nabla C^T,$$
onde $\alpha \in \mathbb{R}_+$. Ainda não convencido de que o $\Delta C$ causado por essa variação $\Delta v$ seja negativo? Vamos substituir:

$$\Delta C \approx \nabla C \cdot \Delta v = \nabla C \cdot (-\alpha \nabla C^T) = -\alpha (\nabla C \cdot \nabla C^T ) = -\alpha ||\nabla C||^2 \leq 0,$$
já que $||\nabla C||^2 \geq 0$ e $\alpha > 0$. Isso nos garante que, sempre que escolhermos o $\Delta v$ daquela forma, o valor do $C$ (o valor do erro!) sempre vai diminuir (claro, dentro dos limites da aproximação).

Observe que utilizamos, aqui, uma função de duas variáveis, como forma de simplificação. Porém, note que, nas equações vetoriais acima, esse fato não se mostrou nada relevante! De fato, não importa a dimensão: a regra do gradiente descendente funcionará, e é por isso que podemos utilizá-la em nossa rede neural! Assim, para reduzirmos o valor da função de custo $C$, alteraremos os pesos $\mathbf w$ e os biases $\mathbf b$ (as variáveis de $C$) pela regra abaixo, considerando a aplicação em cada variável:

$$w_k \gets w_k + (-\alpha \frac{\partial C}{\partial w_k})$$
$$b_l \gets b_l + (-\alpha \frac{\partial C}{\partial b_l})$$

No contexto das redes, o parâmetro $\alpha$, chamado **taxa de aprendizado** (*learning rate*), permite diminuir o risco de provocarmos um $\Delta C$ muito elevado, o que poderia representar uma fuga de um ponto mínimo, ou um muito pequeno, o que atrasaria o processo de otimização.

### O gradiente descendente estocástico

Suponha que nosso conjunto de treino $\mathcal{T}$ possua muitas instâncias. Como
$C = \sum_{x \in \mathcal{T}} C_x,$
computar $\nabla C$ seria muito demorado, pois envolveria computar todos os valores
$\nabla C_x$ e depois fazer $\nabla C = \frac{1}{n} \sum_{x \in \mathcal{T}} \nabla C_x$.
Isso poderia atrasar muito o aprendizado, pois haveria uma atualização dos pesos e *biases* por epoch!

O que comumente se faz é adaptar o método do gradiente descendente para sua versão
estocástica. Isso significa particionar aleatoriamente o conjunto $\mathcal T$ em uma família de conjuntos
$\{\mathcal{T_i}\}_i$ de mesma cardinalidade (tamanho) $m$. Seja $\mathcal{T_j} = \{x^j_1, x^j_2, \ldots, x^j_m\}$ algum desses conjuntos e suponha $m$ suficientemente grande.
Consideramos que a média entre os $\nabla C_{x^j_k}$ se aproxime da média dos $\nabla C_{x}$, ou seja:

$$\frac{\sum_{k = 1}^m \nabla C_{x^j_k}}{m} \approx \frac{\sum_{x \in \mathcal{T}} \nabla C_x}{n} = \nabla C.$$

Fantástico! Agora não precisamos esperar a análise do conjunto inteiro e potencialmente gigantesco $\mathcal{T}$ para que a rede aprenda! Podemos pegar porções menores (chamadas *mini-batches*), estimar $\nabla C$ utilizando a aproximação acima, e atualizar os pesos e *biases*! Na prática, tomamos um conjunto com $m$  pontos arbitrários, $\mathcal{T_j} = \{x^j_1, x^j_2, \ldots, x^j_m\}$, e executamos as atualizações:

$$w_k \gets w_k + (-\frac{\alpha}{m} \sum_{p = 1}^m \frac{\partial C_{x^j_p}}{\partial w_k})$$
$$b_l \gets b_l + (-\frac{\alpha}{m} \sum_{p = 1}^m \frac{\partial C_{x^j_p}}{\partial b_l})$$

Depois, tomamos outro conjunto de tamanho $m$,
e executamos as atualizações. Isso se repete até que o conjunto $\mathcal{T}$
se exaura, marcando o fim de uma *epoch* do treino. As próximas *epochs* repetirão esse processo.

Que tal implementarmos nosso *stochastic gradient descent* em nossa rede neural?

```python
    %%add_to MultilayerNeuralNetwork
    def stochastic_gradient_descent(self, D, batch_size):
        ''' Update the weights and bias through stochastic gradient
        descent optimization. '''
        # Get the amount of data points
        train_size = len(D)
        # Shuffle the data
        random.shuffle(D)
        # Obtain mini-batches
        mini_batches = [D[k : k + batch_size]
            for k in range(0, train_size, batch_size)]
        # Update weights and biases
        for mini_batch in mini_batches:
            self.optimize_mini_batch(mini_batch)

    def optimize_mini_batch(self, mini_batch):
        ''' Given a minibatch {<x1,y1>,<x2,y2>,...,<xm,ym>}, optimize the cost function. '''
        # Get the mini-batch size
        m = len(mini_batch)
        # Dictionaries to store the gradients of C
        GCw = {}
        GCb = {}
        # Initialize gradient matrices with zeroes
        for i in np.arange(1,len(self.arch)):
            GCw[i] = np.zeros(self.W[i].shape)
            GCb[i] = np.zeros(self.B[i].shape)
        # Use the SGD update rules for teaching the neural network for each batch point
        for x, y in mini_batch:
            # Compute gradients: see next section to understand how!
            Gw, Gb = self.compute_gradient(x, y)
            for i in np.arange(1,len(self.arch)):
                GCw[i] += Gw[i]
                GCb[i] += Gb[i]
        # Apply rules for updating weights and biases
        for l in np.arange(1,len(self.arch)):
            self.W[l] = self.W[l] - (self.alpha/m)*(GCw[l])
            self.B[l] = self.B[l] - (self.alpha/m)*(GCb[l])
```

Esta seção nos deu um método muito poderoso para nossa rede neural aprender
com a experiência! Existe, porém, um ponto ainda obscuro... **Como calcular o gradiente?** Ou melhor, como calcular de forma eficiente? O algoritmo da próxima seção tratará exatamente disso! Vamos aproveitar a estrutura em camadas da nossa rede para computar rapidamente gradientes de forma brilhante!

 ## O algoritmo Backpropagation

 Por questões de simplicidade, consideraremos um exemplo fixo $\langle \mathbf{x}, \mathbf{y} \rangle$, e chamaremos $C_x$ de $C$.

O ajuste dos pesos, pelo método do gradiente descendente, como visto, demanda o conhecimento sobre como variações nas matrizes
 de pesos - de camadas diferentes da última, inclusive - e dos *biases* afetam o resultado da função de custo. Em outras palavras, queremos conhecer as derivadas

 $$\frac{\partial C}{\partial w_{ij}^l} \qquad \frac{\partial C}{\partial b_{i}^l},$$
 para todos os $w_{ij}^l$ e $b_{i}^l$, o que nos permitirá utilizar as regras de atualização do gradiente descendente e fazer a rede neural aprender.

 Computá-los de forma explícita seria dispendioso. Existe, porém, um caminho implícito. Imagine que o neurônio $j$ da camada $L_l$
 deva produzir o valor $net$ denotado por $z_j^l$. Porém, suponha que esse valor
 seja alterado por uma quantidade pequena $\Delta z_j^l$, de tal forma que a resposta dele
 seja $f(z_j^l + \Delta z_j^l)$. Como a função de custo é afetada por essa mudança?
 Da fórmula de Taylor para o Cálculo Multivariável, chega-se que

 $$\Delta C = \frac{\partial C}{\partial z_{j}^l} \Delta z_{j}^l.$$

 Note que, se $\frac{\partial C}{\partial z_{j}^l}$
 possuir um valor alto, é necessário um $\Delta z_{j}^l$ de sinal oposto
 para reduzir o custo. Se possuir um valor próximo de zero,
 como $\Delta z_j^l$ é pequeno, não se pode fazer muito para reduzir o custo, e o neurônio é dito estar
 em estado próximo do ótimo. Assim, faz sentido definir o erro nesse neurônio,
 denotado por $\delta_j^l$, como

 \begin{equation}
 \delta_j^l = \frac{\partial C}{\partial z_{j}^l}.
 \end{equation}

 Além disso, define-se o vetor de erros dos neurônios da camada $L_l$ por
 $\delta^l = \langle \delta^l_1, \delta^l_2, \ldots, \delta^l_{tam\;L_l} \rangle$. O algoritmo Backpropagation está fundamentado em quatro equações, das quais se pode obter
 o gradiente da função de custo, de interesse para as regras de atualização do gradiente descendente. Pode-se dizer que este é o grande viabilizador das redes neurais modernas, incluindo
as utilizadas para *Deep Learning*. Para simplificar a notação, considere a camada $L_{k-1}$ (de saída) denotada pelo índice $L$.

Resta-nos compreender cada equação.

### Uma equação para o $\delta^L$

Esta equação fornece meios para se calcular o erro na camada de saída. A função de custo pode ser vista da forma

$$C=C(X_1, X_2, \ldots, X_{tam\,L}),$$

tal que $X_1 = a_1^L(z^L_1, z^L_2 \ldots, z^L_{tam\,L}), X_2 = a_2^L(z^L_1, z^L_2 \ldots, z^L_{tam\,L}), \ldots, X_{tam\,L} = a_{tam\, L}^L(z^L_1, z^L_2 \ldots, z^L_{tam\,L})$. Pela definição anterior, sabemos que:

$$\delta_j^L = \frac{\partial C}{\partial z_{j}^L}.$$

Utilizando a **regra da cadeia** do Cálculo Multivariável, tem-se que

$$\frac{\partial C}{\partial z_{j}^L} = \sum_k \frac{\partial C}{\partial a^L_j}\frac{\partial a^L_k}{\partial z^L_j}.$$

Como apenas $a^L_j$ depende de $z^L_j$, todas as $\frac{\partial a^L_k}{\partial z^L_j}$, com $k \neq j$, serão anuladas, restando que:

$$\frac{\partial C}{\partial z_{j}^L} =\frac{\partial C}{\partial a^L_j}\frac{\partial a^L_j}{\partial z^L_j}.$$

Como $a^L_j = f(z^L_j)$,

$$\frac{\partial a^L_j}{\partial z^L_j} = f'(z^L_j).$$

Finalmente, chega-se à equação

$$\delta^L_j = \frac{\partial C}{\partial a^L_j}f'(z_j^L).$$

Na forma vetorial, temos:

$$\delta^L = \nabla C_a \odot f'(\mathbf{z}^L),$$
onde $\odot$ é o **produto de Hadamard**, o qual tem o seguinte efeito, para dois vetores:

$$\langle a_1, a_2, \ldots, a_n \rangle^T \odot \langle b_1, b_2, \ldots, b_n \rangle^T = \langle a_1b_1, a_2b_2,\ldots,a_nb_n \rangle^T.$$

### Uma equação para o $\delta^l$ em função de $\delta^{l+1}$

Na equação passada, conseguimos uma forma de calcular os erros da última camada, entretanto, queremos também um modo
de se calcularem os erros nas demais camadas. A estratégia é tentar escrever $\delta^{l}_j = \frac{\partial C}{\partial z^l_j}$ em termos de $\delta^{l+1}_{j} = \frac{\partial C}{\partial z^{l+1}_j}$. Como? Primeiro, é válido notar que $z_j^{l+1}$, por definição,
depende $z_j^l$:

$$z_j^{l+1} = \mathbf{W}^{l+1}_ja^l = \mathbf{W}^{l+1}_jf(z^l) = \sum_i w^{l+1}_{ij}f(z^l_i).$$

Além disso, $C$ depende de $z_j^{l+1}$, devido à definição recursiva de $a^n$. Com isso, basta aplicar a regra da cadeia novamente, e temos que:

$$\delta_j^l = \sum_k \frac{\partial C}{\partial z^{l+1}_k}\frac{\partial z^{l+1}_k}{\partial z^l_j} = \sum_k \frac{\partial z^{l+1}_k}{\partial z^l_k}\delta^{l+1}_k.$$

Diferenciando a expressão para $z_j^{l+1}$, temos que:
$$\frac{\partial z_k^{l+1}}{\partial z_j^l} = w^{l+1}_{jk}f'(z^l_j).$$

Assim, substituindo, chegamos a:
$$\delta_j^l = \sum_k w^{l+1}_{jk}\delta^{l+1}_kf'(z^l_j).$$

Podemos obter a versão vetorial da seguinte forma:

\begin{align}
\delta_j^l = & (\sum_k w^{l+1}_{kj}\delta^{l+1}_k)f'(z^l_j) \\
           = & ((w^{l+1}_{j})^T \cdot \delta^{l+1})f'(z^l_j)
\end{align}

Essa é a forma componente a componente; para obtermos o vetor $\delta_j$, basta
utilizarmos o produto de Hadamard:

$$\delta^l = ((w^{l+1})^T \cdot \delta^{l+1}) \odot f'(z^l)$$

### Uma equação para $\frac{\partial C}{\partial b^l_{j}}$

Chegou a hora de obtermos as nossas derivadas de interesse! Começaremos por aquelas
com respeito aos *biases*. Da regra da cadeira para o cálculo de muitas variáveis, temos que:

$$\frac{\partial C}{\partial b^l_{j}}=\sum_m \frac{\partial C}{\partial z_m^l} \frac{\partial z_m^l}{\partial b_{j}^l}$$

Note agora que $\frac{\partial z_m^l}{\partial b_{j}^l}$, quando $m \neq j$,
é zero, pois $z_m^l$ não depende de $b_{j}^l$ nessas condições. Assim, a equação acima
se reduz a:

$$\frac{\partial C}{\partial b^l_{j}}=\frac{\partial C}{\partial z_j^l} \frac{\partial z_j^l}{\partial b_{j}^l}$$

Vamos lembrar a forma de $z_j^l$:

$$z_j^l = \sum_k a^{l-1}_k w^l_{kj} + b^l_j$$

Daí, derivando $z_j^l$ com respeito a $b^l_j$, temos:

$$\frac{\partial z_j^l}{\partial b_{j}^l} = 1.$$

Substituindo na equação mais acima, chegamos que, simplesmente:

$$\frac{\partial C}{\partial b^l_{j}}=\frac{\partial C}{\partial z_j^l} = \delta^l_j.$$

### Uma equação para $\frac{\partial C}{\partial w^l_{jk}}$

Sabemos que $C$ está em função de $z^l$ e que $z^l$ está em função dos pesos,
o que inclui $w_{jk}^l$. Por essa razão, podemos novamente aplicar a regra
da cadeia, da seguinte forma:
$$\frac{\partial C}{\partial w^l_{jk}}=\sum_m \frac{\partial C}{\partial z_m^l} \frac{\partial z_m^l}{\partial w_{jk}^l}$$

Note que $w^l_{jk}$ apenas influencia no cálculo de $z^l_j$, logo
$$\frac{\partial C}{\partial w^l_{jk}}=\frac{\partial C}{\partial z_j^l} \frac{\partial z_j^l}{\partial w_{jk}^l}=\delta^l_j\frac{\partial z_j^l}{\partial w_{jk}^l}.$$

Lembremo-nos de que:
$$z_j^{l} = \mathbf{W^{l}}_j \cdot \mathbf{a}^{l-1} = \sum_i w^{l}_{ji}a^{l-1}_i,$$
onde $\mathbf{W^{l}}_j$ é a $j$-ésima linha de $\mathbf{W}^l$.

Com isso, diferenciando, temos:
$$\frac{\partial z_j^{l}}{\partial w_{jk}^l} =  a^{l-1}_k.$$

Finalmente, substituindo:
$$\frac{\partial C}{\partial w^l_{jk}}= \delta^l_j a^{l-1}_k.$$

Temos, assim, todas as equações fundamentais que constituem o algoritmo
Backpropagation, e elas nos dão todas as derivadas parciais de que precisamos para treinar nossa rede com o gradiente descendente! Note que o algoritmo Backpropagation é um método geral de computar derivadas parciais em grafos como os utilizados para representar MLPs, e, portanto, pode ser utilizado em outros métodos de otimização!

Agora, resta implementarmos. Antes de tudo, é preciso observarmos que, nas equações acima, são utilizados
os valores dos outputs de cada camada, ou seja, os $\mathbf{a}^l, l=1,\ldots,k-1$, e, portanto,
precisamos calculá-los. Esse processo é comumente referenciado como **forward pass**, e funciona como
a função `predict`, porém trabalha apenas sobre uma instância e retorna os *outputs* intermediários
da rede:

```python
    %%add_to MultilayerNeuralNetwork
    def forward_pass(self, x):
        ''' Given a line vector x, compute the output of each layers. '''
        # Set the result of the input layer to be column vectors
        a = np.atleast_2d(x).T
        # Compute and store the output of each layer, following the definition
        A = {}
        A[0] = a
        for layer in np.arange(1, len(self.arch)):
            a = self.sigmoid(np.dot(self.W[layer], a) + self.B[layer])
            A[layer] = a
        return A
```

Com isso, podemos calcular os gradientes executando primeiro o `forward_pass` e, em seguida, utilizando
as quatro equações discutidas acima:

```python
    %%add_to MultilayerNeuralNetwork
    def compute_gradient(self, x, y):
        ''' Use backpropagation equations to compute the gradient of the cost function. '''
        # The index of the output layer, L
        L = len(self.arch) - 1
        # Dicts for gradients
        Gw = {}
        Gb = {}
        # Compute the network output for x input sample
        A = self.forward_pass(x)
        # Declare what will be set of delta vectors
        D = {}
        # Compute the error for the last neurons, using Equation 1
        y = np.atleast_2d(y).T
        deltaL = (A[L] - y) * self.sigmoid_deriv(A[L])
        # Include in the D matrix
        D[L] = deltaL
        # Compute each delta{l} = g(delta{l+1}), using Equation 2
        for l in np.arange(L-1, 0, -1):
            D[l] = (self.W[l+1].T.dot(D[l+1])) * self.sigmoid_deriv(A[l])
        for l in np.arange(L, 0, -1):
            # Compute derivatives with respect to biases, using Equation 3
            Gb[l] = D[l]
            # Compute derivatives with respect to weights, using Equation 4
            Gw[l] = D[l].dot(A[l-1].T)
        # Return all derivatives
        return (Gw, Gb)
```

Nós temos agora todos os componentes para construir uma função de treino, que chamaremos de `fit`. Ela, basicamente, executará todos os métodos criados acima, informando ao usuário dados importantes sobre o processo:

```python
    %%add_to MultilayerNeuralNetwork
    def fit(self, X, Y, batch_size = 20, epochs = 1000, max_loss = 0.001, displayUpdate = 100):
        ''' Train the neural network '''
        # Assemble T dataset
        D = list(zip(X,Y))
        # Store losses for returning
        losses = [self.quadratic_loss(X,Y)]
        # Loop over epochs
        current_epoch = 0
        while current_epoch < epochs and losses[current_epoch] > max_loss:
            # Optimize using gradient descent
            self.stochastic_gradient_descent(D, batch_size)
            # Compute and store loss
            losses.append(self.quadratic_loss(X,Y))
            # Print information
            if current_epoch % displayUpdate == 0:
                print("Epoch: {}, Loss: {:.7f}".format(current_epoch + 1, losses[current_epoch]))
            current_epoch += 1
        # Print about stop reason
        if losses[current_epoch] < max_loss:
            print("Stopped by max loss criterion, loss: {:.7f}.", losses[current_epoch])
        elif current_epoch == epochs:
            print("Stopped by epochs criterion.")
        # Return training information
        return (losses, current_epoch)
```

# Utilizando a rede neural

Agora que temos nossa rede neural implementada, podemos resolver alguns problemas bastante interessantes. Aqui serão mostrados dois exemplos: o XOR e o reconhecimento de caracteres utilizando o *dataset* MNIST.

## XOR

XOR é uma operação bem conhecida da lógica proposicional que possui a seguinte tabela verdade:

|$x_1$|$x_2$|$y$
|---|---|---|
|0|0|0|
|1|0|1|
|0|1|1|
|1|1|0|

A partir disso, tem-se o *dataset* XOR, composto pelos vetores:
$\langle 0,0 \rangle, \langle 1,0 \rangle, \langle 0,1 \rangle, \langle 1,1 \rangle$,
com os respectivos *labels* $0, 1 , 1, 0$, correspondendo a cada linha da tabela acima. O que acontece se plotarmos
$x_1$ no eixo $x$, $x_2$ no eixo $y$ e atribuirmos uma cor para cada um dos dois valores possíveis para $y$? Note:

```python
import matplotlib.pyplot as plt

x1 = np.array([[0],[1],[0],[1]])
x2 = np.array([[0],[0],[1],[1]])

plt.scatter(x1,x2,c=['green','red','red','green'])
plt.title('The XOR dataset')
plt.show()
```

O que há de importante nesse *dataset*? **Não é possível separá-lo utilizando uma reta**! Ou seja, não corresponde a um problema de classificação com classes linearmente separáveis. Um neurônio somente não conseguiria resolvê-lo, mas nossa rede pode! Vamos?

```python
# Let's have our dataset D
X = np.array([[1,1],[1,0],[0,1],[0,0]])
# Let's have our targets, following the truth table presented above
y = np.array([[0],[1],[1],[0]])

# Now, let's create and train the neural network!
xorNetwork = MultilayerNeuralNetwork(arch=[2,2,1], alpha=0.5)
(losses, last_epoch) = xorNetwork.fit(X, y, batch_size=1, epochs = 20000, displayUpdate=100, max_loss=0.001)
```

Podemos checar o aprendizado por meio de um gráfico, visto que a função `fit` retorna uma lista com os valores dos custos em cada *epoch*:

```python
# List epochs (x-axis)
epochs = np.arange(0, last_epoch + 1)
# Plot losses
plt.title("Loss per epoch - XOR dataset")
plt.plot(epochs, losses)
plt.show()
```

Com nosso modelo treinado, podemos checar se ele, de fato, resolveu o problema do XOR, ou seja, se ele consegue classificar (separar) corretamente os pontos segundo a tabela verdade:

```python
# Obtain predictions
pred = xorNetwork.predict(X)
print("Network output: " + str(pred))
# Round values for getting the targets we expect
print("In target format: " + str(np.round(pred)))
```

## MNIST

O MNIST é um *dataset* massivamente utilizado pelos estudiosos de Machine Learning composto
por imagens de dígitos de 0 a 9. Ele possui 60000 exemplos de treino e 10000 de testes.
Os vetores de características são 784-dimensionais ($28 \times 28$ pixels por imagem),
com componentes assumindo valores em $[0,255]$. O propósito é corretamente classificar
os dígitos desse *dataset*!

A biblioteca `sklearn` oferece uma amostra do MNIST por meio de comandos simples.
Para concluir este *notebook*, vamos utilizá-los para obter uma parte desse *dataset* e, com ela,
treinar uma rede neural especializada em classificar
imagens de dígitos de 0 a 9! Repare que o código é praticamente o mesmo para o treinamento que fizemos
com o *dataset* XOR:

```python
# Import sklearn tools
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn import datasets

# Load a sample of MNIST through simple sklearn commands
print("[INFO] loading MNIST dataset")
digits = datasets.load_digits()
data = digits.data.astype("float")
data = (data - data.min()) / (data.max() - data.min())    # normalize data
print("[INFO] samples: {}, dim: {}".format(data.shape[0], data.shape[1]))

# Split the dataset into train and test
(trainX, testX, trainY, testY) = train_test_split(data, digits.target, test_size=0.25)

# Binarize labels, such that, for example, the target 5 becomes the row vector (0,0,0,0,0,1,0,0,0,0)
trainY = LabelBinarizer().fit_transform(trainY)
testY = LabelBinarizer().fit_transform(testY)

# Train the neural network
print("[INFO] training network...")
mnistNN = MultilayerNeuralNetwork(arch=[trainX.shape[1], 32, 16, 10], alpha=0.1)
(losses, last_epoch) = mnistNN.fit(trainX, trainY, epochs=100, batch_size=1, max_loss=0.00001, displayUpdate=10)
```

Observemos também o gráfico do custo por *epoch*:

```python
epochs = np.arange(0, last_epoch + 1)

plt.plot(epochs, losses)
plt.show()
```

Uma vez treinada a rede, podemos checar seu desempenho no conjunto de testes, por meio da função `classification_report`, que fornece índices de desempenho:

```python
print("[INFO] evaluating network...")
# Test the model
print(testX.shape)
# Obtain predictions
predictions = mnistNN.predict(testX)
# Transform binary vectors in digits 0-9
predictions = predictions.T.argmax(axis=1)
# Print classification report
print(classification_report(testY.argmax(axis=1), predictions))
```

# Referências

[1] Michael Nielsen. *Neural Networks and Deep Learning*. Determination Press. Disponível em: http://neuralnetworksanddeeplearning.com/

[2] Adrian Rosebrock. *Deep Learning for Computer Vision with Python*. 1st ed. https://www.pyimagesearch.com: PyImageSearch.