  NEST Topology Module: A Case-Based Tutorial
 ===========================================

 #### **Author:** Hans Ekkehard Plesser
 #### **Institution:** Norwegian University of Life Sciences
 #### **Version:** 0.4
 #### **Date:** 21 November 2012
 #### **Copyright:** The NEST Initiative (2009-2012)
 #### **License:** Creative Commons Attribution License
 **NOTE:** The network generated by this script does generate
 dynamics in which the activity of the entire system, especially
 Rp and Vp oscillates with approx 5 Hz. This is different from
 the full model. Deviations are due to the different model type
 and the elimination of a number of connections, with no changes
 to the weights.

 Introduction
 ============

 This tutorial shows you how to implement a simplified version of the
 Hill-Tononi model of the early visual pathway using the NEST Topology
 module.  The model is described in the paper

   S. L. Hill and G. Tononi.
   Modeling Sleep and Wakefulness in the Thalamocortical System.
   J Neurophysiology **93**:1671-1698 (2005).
   Freely available via `doi 10.1152/jn.00915.2004
   <http://dx.doi.org/10.1152/jn.00915.2004>

 We simplify the model somewhat both to keep this tutorial a bit
 shorter, and because some details of the Hill-Tononi model are not
 currently supported by NEST. Simplifications include:

 1. We use the `iaf_cond_alpha` neuron model, which is
    simpler than the Hill-Tononi model.

 2. As the `iaf_cond_alpha` neuron model only supports two
    synapses (labeled "ex" and "in"), we only include AMPA and
    GABA_A synapses.

 3. We ignore the secondary pathway (Ts, Rs, Vs), since it adds just
    more of the same from a technical point of view.

 4. Synaptic delays follow a Gaussian distribution in the HT
    model. This implies actually a Gaussian distributions clipped at
    some small, non-zero delay, since delays must be
    positive. Currently, there is a bug in the Topology module when
    using clipped Gaussian distribution. We therefore draw delays from a
    uniform distribution.

 5. Some further adaptations are given at the appropriate locations in
    the script.

 This tutorial is divided in the following sections:

 - **Philosophy**
    Discusses the philosophy applied to model implementation in this
    tutorial

 - **Preparations**
    Neccessary steps to use NEST and the Topology Module

 - **Configurable Parameters**
    Define adjustable network parameters

 - **Neuron Models**
    Define the neuron models needed by the network model

 - **Populations**
    Create Populations

 - **Synapse models**
    Define the synapse models used in the network model

 - **Connections**
    Create Connections

 - **Example simulation**
    Perform a small simulation for illustration. This
    section also discusses the setup for recording.

 Philosophy
 ==========

 A network models has two essential components: *populations* and
 *projections*.  We first use NEST's ``CopyModel()`` mechanism to
 create specific models for all populations and subpopulations in
 the network, and then create the populations using the Topology
 modules ``CreateLayer()`` function.

 We use a two-stage process to create the connections, mainly
 because the same configurations are required for a number of
 projections: we first define dictionaries specifying the
 connections, then apply these dictionaries later.

 The way in which we declare the network model here is an
 example. You should not consider it the last word: we expect to see
 a significant development in strategies and tools for network
 descriptions in the future. The following contributions to CNS\*09
 seem particularly interesting

 - Ralf Ansorg & Lars Schwabe. Declarative model description and
   code generation for hybrid individual- and population-based
   simulations of the early visual system (P57);
 - Sharon Crook, R. Angus Silver, & Padraig Gleeson. Describing
   and exchanging models of neurons and neuronal networks with
   NeuroML (F1);

 as well as the following paper which will apply in PLoS
 Computational Biology shortly:

 - Eilen Nordlie, Marc-Oliver Gewaltig, & Hans Ekkehard Plesser.
   Towards reproducible descriptions of neuronal network models.

 Preparations
 ============

 Please make sure that your ``PYTHONPATH`` is set correctly, so
 that Python can find the NEST Python module.

 **Note:** By default, the script does not show any graphics.
 Set ``SHOW_FIGURES`` to ``True`` to activate graphics.

```
import sys
sys.path.append('/opt/lib/python2.7/site-packages/')
```

```
SHOW_FIGURES = True

import pylab
if not SHOW_FIGURES:
    pylab_show = pylab.show
    def nop(s=None): pass
    pylab.show = nop
else:
    pylab.ion()
```

 Introduction
=============
 This tutorial gives a brief introduction to the ConnPlotter
 toolbox.  It is by no means complete.

```
# Load pynest
import nest

# Load NEST Topoplogy module (NEST version>2.2)
import nest.topology as topo

# Make sure we start with a clean slate, even if we re-run the script
# in the same Python session.
nest.ResetKernel()

# Import math, we need Pi
import math
```

 Configurable Parameters
 =======================

 Here we define those parameters that we take to be
 configurable. The choice of configurable parameters is obviously
 arbitrary, and in practice one would have far more configurable
 parameters. We restrict ourselves to:

 - Network size in neurons ``N``, each layer is ``N x N``.
 - Network size in subtended visual angle ``visSize``, in degree.
 - Temporal frequency of drifting grating input ``f_dg``, in Hz.
 - Spatial wavelength and direction of drifting grating input,
   ``lambda_dg`` and ``phi_dg``, in degree/radian.
 - Background firing rate of retinal nodes and modulation amplitude,
   ``retDC`` and ``retAC``, in Hz.
 - Simulation duration ``simtime``; actual simulation is split into
   intervals of ``sim_interval`` length, so that the network state
   can be visualized in those intervals. Times are in ms.

```
Params = {'N'           :     40,
          'visSize'     :    8.0,
          'f_dg'        :    2.0,
          'lambda_dg'   :    2.0,
          'phi_dg'      :    0.0,
          'retDC'       :   30.0,
          'retAC'       :   30.0,
          'simtime'     :  100.0,
          'sim_interval':    5.0
          }
```

 Neuron Models
 =============

 We declare models in two steps:

 1. We define a dictionary specifying the NEST neuron model to use
    as well as the parameters for that model.
 2. We create three copies of this dictionary with parameters
    adjusted to the three model variants specified in Table~2 of
    Hill & Tononi (2005) (cortical excitatory, cortical inhibitory,
    thalamic)

 In addition, we declare the models for the stimulation and
 recording devices.

 The general neuron model
 ------------------------

 We use the `iaf_cond_alpha` neuron, which is an
 integrate-and-fire neuron with two conductance-based synapses which
 have alpha-function time course.  Any input with positive weights
 will automatically directed to the synapse labeled `_ex`, any
 with negative weights to the synapes labeled `_in`.  We define
 **all** parameters explicitly here, so that no information is
 hidden in the model definition in NEST. `V_m` is the membrane
 potential to which the model neurons will be initialized.
 The model equations and parameters for the Hill-Tononi neuron model
 are given on pp. 1677f and Tables 2 and 3 in that paper. Note some
 peculiarities and adjustments:

 - Hill & Tononi specify their model in terms of the membrane time
   constant, while the `iaf_cond_alpha` model is based on the
   membrane capcitance. Interestingly, conducantces are unitless in
   the H&T model. We thus can use the time constant directly as
   membrane capacitance.
 - The model includes sodium and potassium leak conductances. We
   combine these into a single one as follows:
$$   \begin{equation}-g_{NaL}(V-E_{Na}) - g_{KL}(V-E_K)
      =
   -(g_{NaL}+g_{KL})\left(V-\frac{g_{NaL}E_{NaL}+g_{KL}E_K}{g_{NaL}g_{KL}}\right)
   \end{equation}
$$
 - We write the resulting expressions for g_L and E_L explicitly
   below, to avoid errors in copying from our pocket calculator.
 - The paper gives a range of 1.0-1.85 for g_{KL}, we choose 1.5
   here.
 - The Hill-Tononi model has no explicit reset or refractory
   time. We arbitrarily set V_reset and t_ref.
 - The paper uses double exponential time courses for the synaptic
   conductances, with separate time constants for the rising and
   fallings flanks. Alpha functions have only a single time
   constant: we use twice the rising time constant given by Hill and
   Tononi.
 - In the general model below, we use the values for the cortical
   excitatory cells as defaults. Values will then be adapted below.

```
nest.CopyModel('iaf_cond_alpha', 'NeuronModel',
               params = {'C_m'       :  16.0,
                         'E_L'       : (0.2 * 30.0 + 1.5 * -90.0)/(0.2 + 1.5),
                         'g_L'       : 0.2 + 1.5,
                         'E_ex'      :   0.0,
                         'E_in'      : -70.0,
                         'V_reset'   : -60.0,
                         'V_th'      : -51.0,
                         't_ref'     :   2.0,
                         'tau_syn_ex':   1.0,
                         'tau_syn_in':   2.0,
                         'I_e'       :   0.0,
                         'V_m'       : -70.0})
```

 Adaptation of models for different populations
 ----------------------------------------------

 We must copy the `NeuronModel` dictionary explicitly, otherwise
 Python would just create a reference.

```
# Cortical excitatory cells
# .........................
# Parameters are the same as above, so we need not adapt anything
nest.CopyModel('NeuronModel', 'CtxExNeuron')
```

```
# Cortical inhibitory cells
# .........................
nest.CopyModel('NeuronModel', 'CtxInNeuron',
               params = {'C_m'  :   8.0,
                         'V_th' : -53.0,
                         't_ref':   1.0})
```

```
# Thalamic cells
# ..............
nest.CopyModel('NeuronModel', 'ThalamicNeuron',
               params = {'C_m'  :   8.0,
                         'V_th' : -53.0,
                         't_ref':   1.0,
                         'E_in' : -80.0})
```

 Input generating nodes
 ----------------------

 Input is generated by sinusoidally modulating Poisson generators,
 organized in a square layer of retina nodes. These nodes require a
 slightly more complicated initialization than all other elements of
 the network:

 - Average firing rate ``DC``, firing rate modulation depth ``AC``, and
   temporal modulation frequency ``Freq`` are the same for all retinal
   nodes and are set directly below.
 - The temporal phase ``Phi`` of each node depends on its position in
   the grating and can only be assigned after the retinal layer has
   been created. We therefore specify a function for initalizing the
   phase ``Phi``. This function will be called for each node.

```
def phiInit(pos, lam, alpha):
    '''Initializer function for phase of drifting grating nodes.

       pos  : position (x,y) of node, in degree
       lam  : wavelength of grating, in degree
       alpha: angle of grating in radian, zero is horizontal

       Returns number to be used as phase of AC Poisson generator.
    '''
    return 2.0 * math.pi / lam * (math.cos(alpha) * pos[0] + math.sin(alpha) * pos[1])

nest.CopyModel('sinusoidal_poisson_generator', 'RetinaNode',
               params = {'ac'    : Params['retAC'],
                         'dc'    : Params['retDC'],
                         'freq'  : Params['f_dg'],
                         'phi'   : 0.0,
                         'individual_spike_trains': False})
```

 Recording nodes
 ---------------

 We use the new `multimeter` device for recording from the model
 neurons. At present, `iaf_cond_alpha` is one of few models
 supporting `multimeter` recording.  Support for more models will
 be added soon; until then, you need to use `voltmeter` to record
 from other models.

 We configure multimeter to record membrane potential to membrane
 potential at certain intervals to memory only. We record the GID of
 the recorded neurons, but not the time.

```
nest.CopyModel('multimeter', 'RecordingNode',
               params = {'interval'   : Params['sim_interval'],
                         'record_from': ['V_m'],
                         'record_to'  : ['memory'],
                         'withgid'    : True,
                         'withtime'   : False})
```

 Populations
 ===========

 We now create the neuron populations in the model, again in the
 form of Python dictionaries. We define them in order from eye via
 thalamus to cortex.

 We first define a dictionary defining common properties for all
 populations

```
layerProps = {'rows'     : Params['N'],
              'columns'  : Params['N'],
              'extent'   : [Params['visSize'], Params['visSize']],
              'edge_wrap': True}
# This dictionary does not yet specify the elements to put into the
# layer, since they will differ from layer to layer. We will add them
# below by updating the `'elements'` dictionary entry for each
# population.
```

Retina
------

```
layerProps.update({'elements': 'RetinaNode'})
retina = topo.CreateLayer(layerProps)

# Now set phases of retinal oscillators; we use a list comprehension instead
# of a loop.
[nest.SetStatus([n], {"phi": phiInit(topo.GetPosition([n])[0],
                                      Params["lambda_dg"],
                                      Params["phi_dg"])})
 for n in nest.GetLeaves(retina)[0]];
```

 Thalamus
 --------

  We first introduce specific neuron models for the thalamic relay
 cells and interneurons. These have identical properties, but by
 treating them as different models, we can address them specifically
 when building connections.

```
# We use a list comprehension to do the model copies.
[nest.CopyModel('ThalamicNeuron', SpecificModel) for SpecificModel in ('TpRelay', 'TpInter')]

# Now we can create the layer, with one relay cell and one
# interneuron per location:
layerProps.update({'elements': ['TpRelay', 'TpInter']})
Tp = topo.CreateLayer(layerProps)
```

 Reticular nucleus
 -----------------
 We follow the same approach as above, even though we have only a
 single neuron in each location.

```
[nest.CopyModel('ThalamicNeuron', SpecificModel) for SpecificModel in ('RpNeuron',)]
layerProps.update({'elements': 'RpNeuron'})
Rp = topo.CreateLayer(layerProps)
```

 Primary visual cortex
 ---------------------

 We follow again the same approach. We differentiate neuron types
 between layers and between pyramidal cells and interneurons. At
 each location, there are two pyramidal cells and one interneuron in
 each of layers 2-3, 4, and 5-6. Finally, we need to differentiate
 between vertically and horizontally tuned populations. When creating
 the populations, we create the vertically and the horizontally
 tuned populations as separate populations.

```
# We use list comprehesions to create all neuron types:
[nest.CopyModel('CtxExNeuron', layer+'pyr') for layer in ('L23','L4','L56')]
[nest.CopyModel('CtxInNeuron', layer+'in' ) for layer in ('L23','L4','L56')]

# Now we can create the populations, suffixes h and v indicate tuning
layerProps.update({'elements': ['L23pyr', 2, 'L23in', 1,
                                'L4pyr' , 2, 'L4in' , 1,
                                'L56pyr', 2, 'L56in', 1]})
Vp_h = topo.CreateLayer(layerProps)
Vp_v = topo.CreateLayer(layerProps)
```

 Collect all populations
 -----------------------

```
# For reference purposes, e.g., printing, we collect all populations
# in a tuple:
populations = (retina, Tp, Rp, Vp_h, Vp_v)
```

Inspection
----------

```
# We can now look at the network using `PrintNetwork`:
nest.PrintNetwork()

# We can also try to plot a single layer in a network. For
# simplicity, we use Rp, which has only a single neuron per position.
topo.PlotLayer(Rp)
pylab.title('Layer Rp')
pylab.show()
```

   Synapse models
 ==============

 Actual synapse dynamics, e.g., properties such as the synaptic time
 course, time constants, reversal potentials, are properties of
 neuron models in NEST and we set them in section `Neuron models`_
 above. When we refer to *synapse models* in NEST, we actually mean
 connectors which store information about connection weights and
 delays, as well as port numbers at the target neuron (`rport`)
 and implement synaptic plasticity. The latter two aspects are not
 relevant here.

 We just use NEST's `static_synapse` connector but copy it to
 synapse models `AMPA` and `GABA_A` for the sake of
 explicitness. Weights and delays are set as needed in section
 `Connections`_ below, as they are different from projection to
 projection. De facto, the sign of the synaptic weight decides
 whether input via a connection is handle by the `_ex` or the
 `_in` synapse.

```
nest.CopyModel('static_synapse', 'AMPA')
nest.CopyModel('static_synapse', 'GABA_A')
```

 Connections
 ====================

 Building connections is the most complex part of network
 construction. Connections are specified in Table 1 in the
 Hill-Tononi paper. As pointed out above, we only consider AMPA and
 GABA_A synapses here.  Adding other synapses is tedious work, but
 should pose no new principal challenges. We also use a uniform in
 stead of a Gaussian distribution for the weights.

 The model has two identical primary visual cortex populations,
 `Vp_v` and `Vp_h`, tuned to vertical and horizonal gratings,
 respectively. The *only* difference in the connection patterns
 between the two populations is the thalamocortical input to layers
 L4 and L5-6 is from a population of 8x2 and 2x8 grid locations,
 respectively. Furthermore, inhibitory connection in cortex go to
 the opposing orientation population as to the own.

 To save us a lot of code doubling, we thus defined properties
 dictionaries for all connections first and then use this to connect
 both populations. We follow the subdivision of connections as in
 the Hill & Tononi paper.

 **Note:** Hill & Tononi state that their model spans 8 degrees of
 visual angle and stimuli are specified according to this. On the
 other hand, all connection patterns are defined in terms of cell
 grid positions. Since the NEST Topology Module defines connection
 patterns in terms of the extent given in degrees, we need to apply
 the following scaling factor to all lengths in connections:

```
dpc = Params['visSize'] / (Params['N'] - 1)

# We will collect all same-orientation cortico-cortical connections in
ccConnections = []
# the cross-orientation cortico-cortical connections in
ccxConnections = []
# and all cortico-thalamic connections in
ctConnections = []
```

 Horizontal intralaminar
 -----------------------
 *Note:* "Horizontal" means "within the same cortical layer" in this
 case.

```
# We first define a dictionary with the (most) common properties for
# horizontal intralaminar connection. We then create copies in which
# we adapt those values that need adapting, and
horIntraBase = {"connection_type": "divergent",
                "synapse_model": "AMPA",
                "mask": {"circular": {"radius": 12.0 * dpc}},
                "kernel": {"gaussian": {"p_center": 0.05, "sigma": 7.5 * dpc}},
                "weights": 1.0,
                "delays": {"uniform": {"min": 1.75, "max": 2.25}}}
```

```
# We use a loop to do the for for us. The loop runs over a list of
# dictionaries with all values that need updating
for conn in [{"sources": {"model": "L23pyr"}, "targets": {"model": "L23pyr"}},
             {"sources": {"model": "L23pyr"}, "targets": {"model": "L23in" }},
             {"sources": {"model": "L4pyr" }, "targets": {"model": "L4pyr" },
              "mask"   : {"circular": {"radius": 7.0 * dpc}}},
             {"sources": {"model": "L4pyr" }, "targets": {"model": "L4in"  },
              "mask"   : {"circular": {"radius": 7.0 * dpc}}},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "L56pyr" }},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "L56in"  }}]:
    ndict = horIntraBase.copy()
    ndict.update(conn)
    ccConnections.append(ndict)
```

 Vertical intralaminar
 -----------------------
 *Note:* "Vertical" means "between cortical layers" in this
 case.

```
# We proceed as above.
verIntraBase = {"connection_type": "divergent",
                "synapse_model": "AMPA",
                "mask": {"circular": {"radius": 2.0 * dpc}},
                "kernel": {"gaussian": {"p_center": 1.0, "sigma": 7.5 * dpc}},
                "weights": 2.0,
                "delays": {"uniform": {"min": 1.75, "max": 2.25}}}
```

```
for conn in [{"sources": {"model": "L23pyr"}, "targets": {"model": "L56pyr"}, "weights": 1.0},
             {"sources": {"model": "L23pyr"}, "targets": {"model": "L23in" }, "weights": 1.0},
             {"sources": {"model": "L4pyr" }, "targets": {"model": "L23pyr"}},
             {"sources": {"model": "L4pyr" }, "targets": {"model": "L23in" }},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "L23pyr"}},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "L23in" }},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "L4pyr" }},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "L4in"  }}]:
    ndict = verIntraBase.copy()
    ndict.update(conn)
    ccConnections.append(ndict)
```

 Intracortical inhibitory
 ------------------------

 We proceed as above, with the following difference: each connection
 is added to the same-orientation and the cross-orientation list of
 connections.

 **Note:** Weights increased from -1.0 to -2.0, to make up for missing GabaB

```
# Note that we have to specify the **weight with negative sign** to make
# the connections inhibitory.
intraInhBase = {"connection_type": "divergent",
                "synapse_model": "GABA_A",
                "mask": {"circular": {"radius": 7.0 * dpc}},
                "kernel": {"gaussian": {"p_center": 0.25, "sigma": 7.5 * dpc}},
                "weights": -2.0,
                "delays": {"uniform": {"min": 1.75, "max": 2.25}}}
```

```
# We use a loop to do the for for us. The loop runs over a list of
# dictionaries with all values that need updating
for conn in [{"sources": {"model": "L23in"}, "targets": {"model": "L23pyr"}},
             {"sources": {"model": "L23in"}, "targets": {"model": "L23in" }},
             {"sources": {"model": "L4in" }, "targets": {"model": "L4pyr" }},
             {"sources": {"model": "L4in" }, "targets": {"model": "L4in"  }},
             {"sources": {"model": "L56in"}, "targets": {"model": "L56pyr"}},
             {"sources": {"model": "L56in"}, "targets": {"model": "L56in" }}]:
    ndict = intraInhBase.copy()
    ndict.update(conn)
    ccConnections.append(ndict)
    ccxConnections.append(ndict)
```

 Corticothalamic
 ---------------

```
corThalBase = {"connection_type": "divergent",
               "synapse_model": "AMPA",
               "mask": {"circular": {"radius": 5.0 * dpc}},
               "kernel": {"gaussian": {"p_center": 0.5, "sigma": 7.5 * dpc}},
               "weights": 1.0,
               "delays": {"uniform": {"min": 7.5, "max": 8.5}}}
```

```
# We use a loop to do the for for us. The loop runs over a list of
# dictionaries with all values that need updating
for conn in [{"sources": {"model": "L56pyr"}, "targets": {"model": "TpRelay" }},
             {"sources": {"model": "L56pyr"}, "targets": {"model": "TpInter" }}]:
    ndict = intraInhBase.copy()
    ndict.update(conn)
    ctConnections.append(ndict)
```

 Corticoreticular
 ----------------

```
# In this case, there is only a single connection, so we write the
# dictionary itself; it is very similar to the corThalBase, and to
# show that, we copy first, then update. We need no `targets` entry,
# since Rp has only one neuron per location.
corRet = corThalBase.copy()
corRet.update({"sources": {"model": "L56pyr"}, "weights": 2.5})
```

 Build all connections beginning in cortex
 -----------------------------------------

```
# Cortico-cortical, same orientation
print("Connecting: cortico-cortical, same orientation")
[topo.ConnectLayers(Vp_h, Vp_h, conn) for conn in ccConnections]
[topo.ConnectLayers(Vp_v, Vp_v, conn) for conn in ccConnections]

# Cortico-cortical, cross-orientation
print("Connecting: cortico-cortical, other orientation")
[topo.ConnectLayers(Vp_h, Vp_v, conn) for conn in ccxConnections]
[topo.ConnectLayers(Vp_v, Vp_h, conn) for conn in ccxConnections]

# Cortico-thalamic connections
print("Connecting: cortico-thalamic")
[topo.ConnectLayers(Vp_h, Tp, conn) for conn in ctConnections]
[topo.ConnectLayers(Vp_v, Tp, conn) for conn in ctConnections]
topo.ConnectLayers(Vp_h, Rp, corRet)
topo.ConnectLayers(Vp_v, Rp, corRet)
```

    Connecting: cortico-cortical, same orientation
    Connecting: cortico-cortical, other orientation
    Connecting: cortico-thalamic

 Thalamo-cortical connections
 ----------------------------
  **Note:** According to the text on p. 1674, bottom right, of
 the Hill & Tononi paper, thalamocortical connections are
 created by selecting from the thalamic population for each
 L4 pyramidal cell, ie, are *convergent* connections.

```
thalCorRect = {"connection_type": "convergent",
               "sources": {"model": "TpRelay"},
               "synapse_model": "AMPA",
               "weights": 5.0,
               "delays": {"uniform": {"min": 2.75, "max": 3.25}}}

print("Connecting: thalamo-cortical")
```

    Connecting: thalamo-cortical

```
# Horizontally tuned
thalCorRect.update({"mask": {"rectangular": {"lower_left" : [-4.0*dpc, -1.0*dpc],
                                             "upper_right": [ 4.0*dpc,  1.0*dpc]}}})
for conn in [{"targets": {"model": "L4pyr" }, "kernel": 0.5},
             {"targets": {"model": "L56pyr"}, "kernel": 0.3}]:
    thalCorRect.update(conn)
    topo.ConnectLayers(Tp, Vp_h, thalCorRect)
```

```
# Vertically tuned
thalCorRect.update({"mask": {"rectangular": {"lower_left" : [-1.0*dpc, -4.0*dpc],
                                             "upper_right": [ 1.0*dpc,  4.0*dpc]}}})
for conn in [{"targets": {"model": "L4pyr" }, "kernel": 0.5},
             {"targets": {"model": "L56pyr"}, "kernel": 0.3}]:
    thalCorRect.update(conn)
    topo.ConnectLayers(Tp, Vp_v, thalCorRect)
```

```
# Diffuse connections
thalCorDiff = {"connection_type": "convergent",
               "sources": {"model": "TpRelay"},
               "synapse_model": "AMPA",
               "weights": 5.0,
               "mask": {"circular": {"radius": 5.0 * dpc}},
               "kernel": {"gaussian": {"p_center": 0.1, "sigma": 7.5 * dpc}},
               "delays": {"uniform": {"min": 2.75, "max": 3.25}}}
```

```
for conn in [{"targets": {"model": "L4pyr" }},
             {"targets": {"model": "L56pyr"}}]:
    thalCorDiff.update(conn)
    topo.ConnectLayers(Tp, Vp_h, thalCorDiff)
    topo.ConnectLayers(Tp, Vp_v, thalCorDiff)
```

 Thalamic connections
 --------------------

 Connections inside thalamus, including Rp

 *Note:* In Hill & Tononi, the inhibition between Rp cells is mediated by
 GABA_B receptors. We use GABA_A receptors here to provide some self-dampening
 of Rp.

 **Note:** The following code had a serious bug in v. 0.1: During the first
 iteration of the loop, "synapse_model" and "weights" were set to "AMPA" and "0.1",
 respectively and remained unchanged, so that all connections were created as
 excitatory connections, even though they should have been inhibitory. We now
 specify synapse_model and weight explicitly for each connection to avoid this.

```
thalBase = {"connection_type": "divergent",
            "delays": {"uniform": {"min": 1.75, "max": 2.25}}}

print("Connecting: intra-thalamic")

for src, tgt, conn in [(Tp, Rp, {"sources": {"model": "TpRelay"},
                                 "synapse_model": "AMPA",
                                 "mask": {"circular": {"radius": 2.0 * dpc}},
                                 "kernel": {"gaussian": {"p_center": 1.0, "sigma": 7.5 * dpc}},
                                 "weights": 2.0}),
                       (Tp, Tp, {"sources": {"model": "TpInter"},
                                 "targets": {"model": "TpRelay"},
                                 "synapse_model": "GABA_A",
                                 "weights": -1.0,
                                 "mask": {"circular": {"radius": 2.0 * dpc}},
                                 "kernel": {"gaussian": {"p_center": 0.25, "sigma": 7.5 * dpc}}}),
                       (Tp, Tp, {"sources": {"model": "TpInter"},
                                 "targets": {"model": "TpInter"},
                                 "synapse_model": "GABA_A",
                                 "weights": -1.0,
                                 "mask": {"circular": {"radius": 2.0 * dpc}},
                                 "kernel": {"gaussian": {"p_center": 0.25, "sigma": 7.5 * dpc}}}),
                       (Rp, Tp, {"targets": {"model": "TpRelay"},
                                 "synapse_model": "GABA_A",
                                 "weights": -1.0,
                                 "mask": {"circular": {"radius": 12.0 * dpc}},
                                 "kernel": {"gaussian": {"p_center": 0.15, "sigma": 7.5 * dpc}}}),
                       (Rp, Tp, {"targets": {"model": "TpInter"},
                                 "synapse_model": "GABA_A",
                                 "weights": -1.0,
                                 "mask": {"circular": {"radius": 12.0 * dpc}},
                                 "kernel": {"gaussian": {"p_center": 0.15, "sigma": 7.5 * dpc}}}),
                       (Rp, Rp, {"targets": {"model": "RpNeuron"},
                                 "synapse_model": "GABA_A",
                                 "weights": -1.0,
                                 "mask": {"circular": {"radius": 12.0 * dpc}},
                                 "kernel": {"gaussian": {"p_center": 0.5, "sigma": 7.5 * dpc}}})]:
    thalBase.update(conn)
    topo.ConnectLayers(src, tgt, thalBase)
```

    Connecting: intra-thalamic

 Thalamic input
 --------------

 Input to the thalamus from the retina.

 **Note:** Hill & Tononi specify a delay of 0 ms for this connection.

```
# We use 1 ms here.
retThal = {"connection_type": "divergent",
           "synapse_model": "AMPA",
           "mask": {"circular": {"radius": 1.0 * dpc}},
           "kernel": {"gaussian": {"p_center": 0.75, "sigma": 2.5 * dpc}},
           "weights": 10.0,
           "delays": 1.0}

print("Connecting: retino-thalamic")

for conn in [{"targets": {"model": "TpRelay"}},
             {"targets": {"model": "TpInter"}}]:
    retThal.update(conn)
    topo.ConnectLayers(retina, Tp, retThal)
```

    Connecting: retino-thalamic

 Checks on connections
 ---------------------

 As a very simple check on the connections created, we inspect
 the connections from the central node of various layers.

```
# Connections from Retina to TpRelay
topo.PlotTargets(topo.FindCenterElement(retina), Tp, 'TpRelay', 'AMPA')
pylab.title('Connections Retina -> TpRelay')
pylab.show()

# Connections from TpRelay to L4pyr in Vp (horizontally tuned)
topo.PlotTargets(topo.FindCenterElement(Tp), Vp_h, 'L4pyr', 'AMPA')
pylab.title('Connections TpRelay -> Vp(h) L4pyr')
pylab.show()

# Connections from TpRelay to L4pyr in Vp (vertically tuned)
topo.PlotTargets(topo.FindCenterElement(Tp), Vp_v, 'L4pyr', 'AMPA')
pylab.title('Connections TpRelay -> Vp(v) L4pyr')
pylab.show()
```

 Recording devices
 =================

 This recording device setup is a bit makeshift. For each population
 we want to record from, we create one `multimeter`, then select
 all nodes of the right model from the target population and
 connect. `loc` is the subplot location for the layer.

```
print("Connecting: Recording devices")
recorders = {}
for name, loc, population, model in [('TpRelay'   , 1, Tp  , 'TpRelay'),
                                     ('Rp'        , 2, Rp  , 'RpNeuron'),
                                     ('Vp_v L4pyr', 3, Vp_v, 'L4pyr'),
                                     ('Vp_h L4pyr', 4, Vp_h, 'L4pyr')]:
    recorders[name] = (nest.Create('RecordingNode'), loc)
    tgts = [nd for nd in nest.GetLeaves(population)[0]
            if nest.GetStatus([nd], 'model')[0]==model]
    nest.Connect(recorders[name][0], tgts)   # one recorder to all targets
```

    Connecting: Recording devices

 Example simulation
 ====================

 This simulation is set up to create a step-wise visualization of
 the membrane potential. To do so, we simulate `sim_interval`
 milliseconds at a time, then read out data from the multimeters,
 clear data from the multimeters and plot the data as pseudocolor
 plots.

```
# show time during simulation
nest.SetStatus([0],{'print_time': True})

# lower and upper limits for color scale, for each of the four
# populations recorded.
vmn=[-80,-80,-80,-80]
vmx=[-50,-50,-50,-50]

nest.Simulate(Params['sim_interval'])

# loop over simulation intervals
for t in pylab.arange(Params['sim_interval'], Params['simtime'], Params['sim_interval']):

    # do the simulation
    nest.Simulate(Params['sim_interval'])

    # clear figure and choose colormap
    pylab.clf()
    pylab.jet()

    # now plot data from each recorder in turn, assume four recorders
    for name, r in recorders.items():
        rec = r[0]
        sp = r[1]
        pylab.subplot(2,2,sp)
        d = nest.GetStatus(rec)[0]['events']['V_m']

        if len(d) != Params['N']**2:
            # cortical layer with two neurons in each location, take average
            d = 0.5 * ( d[::2] + d[1::2] )

        # clear data from multimeter
        nest.SetStatus(rec, {'n_events': 0})
        pylab.imshow(pylab.reshape(d, (Params['N'],Params['N'])),
                     aspect='equal', interpolation='nearest',
                     extent=(0,Params['N']+1,0,Params['N']+1),
                     vmin=vmn[sp-1], vmax=vmx[sp-1])
        pylab.colorbar()
        pylab.title(name + ', t = %6.1f ms' % nest.GetKernelStatus()['time'])

    pylab.draw()  # force drawing inside loop
    pylab.show()  # required by `pyreport`

# just for some information at the end
print(nest.GetKernelStatus())
```

```

```


```python
import os
import numpy as np
import pickle
```

```python
from __future__ import print_function
import mxnet as mx
from mxnet import nd, autograd
from mxnet import gluon
mx.random.seed(1)
```

%matplotlib inline
import matplotlib.pyplot as plt

```python
root_path = os.getcwd()
```

## 1. Prepare Dataset

Example:

```python
!python prepare_data.py --data-path data/omniglot --save-path data/omniglot
```

### 1.1 Build Self-defined Iterator

```python
train_data = os.path.join(root_path, "omniglot", "train.pickle")
val_data = os.path.join(root_path, "omniglot", "val.pickle")
```

```python
class Data_Loader:
    """
    batch: (batch_size, channel, height, width)
    """
    def __init__(self, path):
        self.data = {}
        self.chars = {}
        self.n_chars = {}
        self.h = 105
        self.w = 105
        self.n_images = 20
        with open(os.path.join(path,"train.pickle"),"rb") as f:
            (tr_imgs, tr_char) = pickle.load(f)
            self.data['train'] = tr_imgs
            self.chars["train"] = tr_char
            self.n_chars["train"] = len(tr_char.keys())

        with open(os.path.join(path,"val.pickle"),"rb") as f:
            (va_imgs, va_char) = pickle.load(f)
            self.data['val'] = va_imgs
            self.chars["val"] = va_char
            self.n_chars["val"] = len(va_char.keys())

    def get_batch(self, batch_size, case="train"):
        """
        return #batch_size pairs and corresponding labels
        """
        cur_data = self.data[case]
        cur_chars = self.chars[case]
        cur_labels = cur_chars.keys()

        img0 = mx.nd.empty([batch_size, 1, self.h, self.w])
        img1 = mx.nd.empty([batch_size, 1, self.h, self.w])
        tmp = mx.nd.uniform(low=0, high=1, shape=(batch_size,))
        targets = mx.nd.rint(tmp)

        for i in range(batch_size):
            if targets[i].asscalar() != 1:
                # pairs are different chars
                labels = np.random.choice(cur_labels, size=2, replace=False)
                idxes = np.random.choice(self.n_images, size=2)
                img0[i, 0, :, :] = cur_data[labels[0]][idxes[0]].T
                img1[i, 0, :, :] = cur_data[labels[1]][idxes[1]].T
            else:
                # pairs are from same char
                label = np.random.choice(cur_labels)
                idxes = np.random.choice(self.n_images, size=2, replace=False)
                img0[i, 0, :, :] = cur_data[label][idxes[0]].T
                img1[i, 0, :, :] = cur_data[label][idxes[1]].T

        return img0, img1, targets

    def get_test_batch(self, support_size, case="val", gt=None):
        """
        return pairs and corresponding labels
        batch: (support_size, 1, height, width)
        """
        cur_data = self.data[case]
        cur_chars = self.chars[case]
        cur_labels = cur_chars.keys()

        img0 = mx.nd.empty([support_size, 1, self.h, self.w])
        img1 = mx.nd.empty([support_size, 1, self.h, self.w])
        targets = mx.nd.zeros((support_size,))

        labels = np.random.choice(cur_labels, size=support_size, replace=False)

        gt_idx = np.random.randint(support_size)
        true_label = labels[gt_idx]
        targets[gt_idx] = 1

        if gt:
            true_label = gt
            labels[gt_idx] = gt

        idxes = np.random.choice(self.n_images, size=support_size)
        for i in range(support_size):
            img0[i, 0, :, :] = cur_data[true_label][idxes[gt_idx]].T
            img1[i, 0, :, :] = cur_data[labels[i]][idxes[i]].T

        return img0, img1, targets

    def evaluate_accuracy(self, net, eval_size, support_size, ctx, top_k=0, case="val"):
        """
        calculate accuracy over #eval_size test task, each task with a supporting set of #support_size
        """
        n_correct = 0
        n_correct_topk = 0

        for i in range(eval_size):
            img0, img1, targets = self.get_test_batch(support_size, case)
            img0 = img0.as_in_context(ctx)
            img1 = img1.as_in_context(ctx)
            targets = targets.as_in_context(ctx)

            score = net(img0, img1)

            targets_np = targets.asnumpy()
            score_np = score.asnumpy().flatten()
            true_idx = np.argmax(targets_np)
            pred_idx = np.argmax(score_np)

            if (targets[pred_idx]).asscalar() > 0:
                n_correct += 1
                n_correct_topk += 1
            else:
                if top_k > 0:
                    topk_inds = np.argpartition(score_np, -top_k)[-top_k:]
                    if true_idx in topk_inds:
                        n_correct_topk += 1

        acc = 100.0 * (n_correct*1.0 / eval_size)
        acc_topk = 100.0 * (n_correct_topk*1.0 / eval_size)
        return acc, acc_topk
```

```python
pkl_path = os.path.join(root_path, "omniglot")
loader = Data_Loader(pkl_path)
```

```python
# check one generated batch
img0, img1, targets = loader.get_batch(batch_size=4)
#img0, img1, targets = loader.get_test_batch(support_size=5)
```

```python
print(targets)
for j in range(4):
    plt.subplot(1,2,1)
    plt.imshow(img0[j,0,:,:].T.asnumpy(), cmap='gray');
    plt.subplot(1,2,2)
    plt.imshow(img1[j,0,:,:].T.asnumpy(), cmap='gray');
    plt.show()
```

## 2. Build Network

```python
input_shape = (105, 105, 1)
num_fc = 4096
num_outputs = 1
```

```python
class Siamese(gluon.Block):
    def __init__(self, **kwargs):
        super(Siamese, self).__init__(**kwargs)
        # use name_scope to give child Blocks appropriate names.
        # It also allows sharing Parameters between Blocks recursively.
        with self.name_scope():
            self.wing = gluon.nn.Sequential()
            with self.wing.name_scope():
                self.wing.add(gluon.nn.Conv2D(channels=64, kernel_size=10, activation='relu'))
                self.wing.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
                self.wing.add(gluon.nn.Conv2D(channels=128, kernel_size=7, activation='relu'))
                self.wing.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
                self.wing.add(gluon.nn.Conv2D(channels=128, kernel_size=4, activation='relu'))
                self.wing.add(gluon.nn.MaxPool2D(pool_size=2, strides=2))
                self.wing.add(gluon.nn.Conv2D(channels=256, kernel_size=4, activation='relu'))
                # The Flatten layer collapses all axis, except the first one, into one axis.
                self.wing.add(gluon.nn.Flatten())
                self.wing.add(gluon.nn.Dense(4096, activation="sigmoid"))

            self.fc = gluon.nn.Dense(1, activation="sigmoid")

    def forward(self, input0, input1):
        out0 = self.wing(input0)
        out1 = self.wing(input1)
        dist = mx.ndarray.abs(out0 - out1)
        score = self.fc(dist)
        return score
```

```python
def count_params(params):
    cnt = 0
    for k in params.keys():
        cur_size = params[k].shape
        cur_cnt = reduce(lambda x,y: x*y, cur_size)
        print("Params: {}: {}".format(k, cur_cnt))
        cnt += cur_cnt
    return cnt
```

## 3. Define Loss
### Regularized Cross-Entropy

\begin{equation}
L(x_1, x_2, t) = t* \log({\bf{p}} (x_1,x_2)) + (1-t)* \log(1-{\bf{p}}(x_1,x_2)) + \lambda \|w \|_2
\end{equation}

```python
ce_loss = gluon.loss.SigmoidBinaryCrossEntropyLoss(from_sigmoid=True)
```

## 4. Training

```python
ctx = mx.gpu()
n_iters = 10
batch_size = 16

eval_size = 500
support_size = 16
top_k = 3

from_scratch = True
model_path = os.path.join(root_path, "siamese_checkpoints")
```

```python
net = Siamese()
```

```python
if from_scratch:
    net.collect_params().initialize(mx.init.Xavier(),ctx=ctx)
else:
    ckpt_name = os.path.join(model_path, "siamese_net_80.params")
    net.load_params(ckpt_name, ctx=ctx)
```

```python
trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.0003, 'wd': 0.001})
```

```python
loss_every = 5
eval_every = 5000
cnt = 0
best = 99

loss_train = []
acc_eval = []
acc_topk_eval = []

for i in range(n_iters):
    img0, img1, targets = loader.get_batch(batch_size=batch_size)
    img0 = img0.as_in_context(ctx)
    img1 = img1.as_in_context(ctx)
    targets = targets.as_in_context(ctx)
    with autograd.record():
        preds = net(img0, img1)
        loss = ce_loss(preds, targets)
    loss.backward()
    trainer.step(batch_size)

    tr_loss = nd.mean(loss).asscalar()
    loss_train.append(tr_loss)

    if i % loss_every == 0:
        print("iteration {}, training loss: {:.4f},".format(i, tr_loss))

    if i % eval_every == 0:
        val_acc, val_acc_topk = loader.evaluate_accuracy(net,eval_size, support_size, ctx, top_k=top_k)
        print("iteration {}, top-1 accuracy: {:.4f}%, top-{} accuracy: {:.4f}%".format(i, val_acc, top_k, val_acc_topk))
        acc_eval.append(val_acc)
        acc_topk_eval.append(val_acc_topk)
        cnt += 1
        if val_acc >= best:
            print("saving model ---------->")
            best=val_acc
            save_name = "siamese_net_" + str(cnt) + ".params"
            file_path = os.path.join(model_path, save_name)
            net.save_params(file_path)
            break
```

    iteration 0, training loss: 0.6743,
    iteration 0, top-1 accuracy: 14.6000%, top-3 accuracy: 44.8000%
    iteration 5, training loss: 0.8142,

```python
params = net.collect_params()
count_params(params)
```

    Params: siamese0_sequential0_conv0_weight: 6400
    Params: siamese0_sequential0_conv0_bias: 64
    Params: siamese0_sequential0_conv1_weight: 401408
    Params: siamese0_sequential0_conv1_bias: 128
    Params: siamese0_sequential0_conv2_weight: 262144
    Params: siamese0_sequential0_conv2_bias: 128
    Params: siamese0_sequential0_conv3_weight: 524288
    Params: siamese0_sequential0_conv3_bias: 256
    Params: siamese0_sequential0_dense0_weight: 37748736
    Params: siamese0_sequential0_dense0_bias: 4096
    Params: siamese0_dense0_weight: 4096
    Params: siamese0_dense0_bias: 1

    38951745L

## 5. Sample Test

```python
img0, img1, targets = loader.get_test_batch(support_size)
img0 = img0.as_in_context(ctx)
img1 = img1.as_in_context(ctx)
targets = targets.as_in_context(ctx)
```

```python
score = net(img0, img1)
```

```python
targets_np = targets.asnumpy()
score_np = score.asnumpy().flatten()
```

```python
pred_idx = np.argmax(score_np)
topk_inds = np.argpartition(score_np, -3)[-3:]
preds = np.zeros(support_size,)
preds[pred_idx] = 1
```

```python
# plot the characters which has scores in top-3
for j in range(3):
    plt.subplot(1,2,1)
    plt.imshow(img0[topk_inds[j],0,:,:].T.asnumpy(), cmap='gray');
    plt.subplot(1,2,2)
    plt.imshow(img1[topk_inds[j],0,:,:].T.asnumpy(), cmap='gray');
    plt.show()
```

```python
targets_np
```

    array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,
            0.,  0.,  0.], dtype=float32)

```python
preds
```

    array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,
            0.,  0.,  0.])

```python

```


```python
import sys
# install ipython dependencies
dep = ['pip', 'scipy', 'numpy', 'matplotlib', 'sympy', 'utm', 'netcdf4']
for lib in dep:
    print(lib)
    !{sys.executable} -m pip install {lib}

```

    pip
    Requirement already satisfied: pip in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (21.1.2)
    scipy
    Requirement already satisfied: scipy in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (1.6.3)
    Requirement already satisfied: numpy<1.23.0,>=1.16.5 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from scipy) (1.20.3)
    numpy
    Requirement already satisfied: numpy in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (1.20.3)
    matplotlib
    Requirement already satisfied: matplotlib in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (3.4.2)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from matplotlib) (1.3.1)
    Requirement already satisfied: numpy>=1.16 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from matplotlib) (1.20.3)
    Requirement already satisfied: cycler>=0.10 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from matplotlib) (0.10.0)
    Requirement already satisfied: python-dateutil>=2.7 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from matplotlib) (2.8.1)
    Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from matplotlib) (2.4.7)
    Requirement already satisfied: pillow>=6.2.0 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from matplotlib) (8.2.0)
    Requirement already satisfied: six in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from cycler>=0.10->matplotlib) (1.16.0)
    sympy
    Requirement already satisfied: sympy in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (1.8)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from sympy) (1.2.1)
    utm
    Requirement already satisfied: utm in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (0.7.0)
    netcdf4
    Requirement already satisfied: netcdf4 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (1.5.6)
    Requirement already satisfied: numpy>=1.9 in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from netcdf4) (1.20.3)
    Requirement already satisfied: cftime in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (from netcdf4) (1.5.0)

```python
!{sys.executable} -m pip install --upgrade pip
```

    Requirement already satisfied: pip in /usr/local/Cellar/jupyterlab/3.0.16/libexec/lib/python3.9/site-packages (21.1.1)
    Collecting pip
      Downloading pip-21.1.2-py3-none-any.whl (1.5 MB)
         |████████████████████████████████| 1.5 MB 4.0 MB/s eta 0:00:01
    Installing collected packages: pip
      Attempting uninstall: pip
        Found existing installation: pip 21.1.1
        Uninstalling pip-21.1.1:
          Successfully uninstalled pip-21.1.1
    Successfully installed pip-21.1.2

```python

```


```
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive

```
current_path = '/content/drive/My Drive/Colab Notebooks/Project_GANder/gankook_cuda/semantic-segmentation-pytorch/pytorch_room_layout/'

import os
os.chdir(current_path)
!ls
```

     build		        info.json	        segment_st.py
     cv2_requirements.txt   lsun.pth.tar	       'sunrgbd (1).zip'
     datasets	        lu_vp_detect.egg-info   sunrgbd.pth.tar
     data_transforms.py     pics		        test_result
     dist		        __pycache__	        Vanish_Point_Detection
     drn_d_105_024_val      README.md	        visualize.m
     drn_d_105_024_val_ms   requirements.txt        XiaohuLuVPDetection
     drn.py		        segment_lsun.py
     features	        segment_rl.py

### Get Vanishing point, inliers

```
# !python XiaohuLuVPDetection/setup.py build
# !python XiaohuLuVPDetection/setup.py install
!pip install -r cv2_requirements.txt
```

    Collecting opencv-python==4.0.0.21
      Downloading https://files.pythonhosted.org/packages/37/49/874d119948a5a084a7ebe98308214098ef3471d76ab74200f9800efeef15/opencv_python-4.0.0.21-cp36-cp36m-manylinux1_x86_64.whl (25.4MB)
         |████████████████████████████████| 25.4MB 1.6MB/s
    Collecting opencv-contrib-python==4.0.0.21
      Downloading https://files.pythonhosted.org/packages/1f/c0/0463d91f297521b2e15e3d682d7077557fe773db72a03a0d2dba899ab8a5/opencv_contrib_python-4.0.0.21-cp36-cp36m-manylinux1_x86_64.whl (31.2MB)
         |████████████████████████████████| 31.2MB 190kB/s
    Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python==4.0.0.21->-r cv2_requirements.txt (line 1)) (1.19.5)
    ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
    Installing collected packages: opencv-python, opencv-contrib-python
      Found existing installation: opencv-python 4.1.2.30
        Uninstalling opencv-python-4.1.2.30:
          Successfully uninstalled opencv-python-4.1.2.30
      Found existing installation: opencv-contrib-python 4.1.2.30
        Uninstalling opencv-contrib-python-4.1.2.30:
          Successfully uninstalled opencv-contrib-python-4.1.2.30
    Successfully installed opencv-contrib-python-4.0.0.21 opencv-python-4.0.0.21

```
import cv2
cv2.__version__
```

    '4.0.0'

### Functions

```
from PIL import Image
import os
import matplotlib.pyplot as plt
import numpy as np
from XiaohuLuVPDetection.lu_vp_detect.vp_detection import VPDetection
import time
# import os
import cv2
# import pylab as pl
from skimage import morphology as mp
import sys
```

```
#             Detect Intersection             #
import math
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.cluster import KMeans
from sympy import Line
from sympy.geometry import Ray

def drawLines(img, lines, color=(255,255,255)):
    """
    Draw lines on an image
    """
    for line in lines:
        for rho,theta in line:
          a = np.cos(theta)
          b = np.sin(theta)
          x0 = a*rho
          y0 = b*rho
          x1 = int(x0 + 1000*(-b))
          y1 = int(y0 + 1000*(a))
          x2 = int(x0 - 1000*(-b))
          y2 = int(y0 - 1000*(a))

          slope = (y1 - y0) / float(x1 - x0)
          angle = math.degrees(math.atan(slope))
          # if abs(angle) < 70:
          cv2.line(img, (x1,y1), (x2,y2), color, 1)

def line(p1, p2):
    A = (p1[1] - p2[1])
    B = (p2[0] - p1[0])
    C = (p1[0]*p2[1] - p2[0]*p1[1])
    return A, B, -C

def intersection2(L1, L2):
    D  = L1[0] * L2[1] - L1[1] * L2[0]
    Dx = L1[2] * L2[1] - L1[1] * L2[2]
    Dy = L1[0] * L2[2] - L1[2] * L2[0]
    if D != 0:
        x = Dx / D
        y = Dy / D
        return x,y
    else:
        return False

def regression(img, x, y, color=(255, 0, 0), axis=1):

  if axis == 1:
    y_at_border = np.array([0, img.shape[0]])
    p = np.polyfit(y, x, deg=1)
    x_at_border = np.poly1d(p)(y_at_border)
  else:
    x_at_border = np.array([0, img.shape[1]])
    p = np.polyfit(x, y, deg=1)
    y_at_border = np.poly1d(p)(x_at_border)

  cv2.line(img, (int(x_at_border[0]), int(y_at_border[0])), (int(x_at_border[1]), int(y_at_border[1])), color, 2)

  return x_at_border, y_at_border

def find_reg_points(img, lines, color=(255,0,0), drawlines=False):

  centroids = list()
  r_xs = list()
  r_ys = list()
  for line_ in lines:
      for rho,theta in line_:
          a = np.cos(theta)
          b = np.sin(theta)
          x0 = a*rho
          y0 = b*rho
          x1 = int(x0 + 1000*(-b))
          y1 = int(y0 + 1000*(a))
          x2 = int(x0 - 1000*(-b))
          y2 = int(y0 - 1000*(a))

          slope = (y1 - y0) / float(x1 - x0)
          angle = math.degrees(math.atan(slope))
          if abs(angle) > 80:
            # print(img.shape[1])
            h_layout = line((0, 0), (img.shape[1], 0))
            h_layout_lower = line((0, img.shape[0]), (img.shape[1], img.shape[0]))
            r = intersection2(h_layout, line((x1, y1), (x2, y2)))
            r_lower = intersection2(h_layout_lower, line((x1, y1), (x2, y2)))
            # cv2.line(img, (int(x1), int(y1)), (int(x2), int(y2)), color, 2)

            center_p = (int((r[0] + r_lower[0]) / 2), int((r[1] + r_lower[1])/ 2))
            centroids.append(center_p)

            r_xs.append((r[0], r_lower[0]))
            r_ys.append((r[1], r_lower[1]))

            if drawlines:
              cv2.line(img, (int(r[0]), int(r[1])), (int(r_lower[0]), int(r_lower[1])), color, 2)
              cv2.circle(img, center_p, 10, (255, 0, 255), -1)

  if drawlines:
    cv2.line(img, (int(0), int(0)), (int(0), int(img.shape[0])), color, 2)
    cv2.line(img, (int(img.shape[1]), int(0)), (int(img.shape[1]), int(img.shape[0])), color, 2)
    cv2.circle(img, (0, int(img.shape[0] / 2)), 10, (255, 0, 255), -1)
    cv2.circle(img, (img.shape[1], int(img.shape[0] / 2)), 10, (255, 0, 255), -1)
  centroids.append((0, int(img.shape[0] / 2)))
  centroids.append((img.shape[1], int(img.shape[0] / 2)))

  return r_xs, r_ys, centroids

from scipy.spatial import distance as sci_dist

def order_points(pts):

  xSorted = pts[np.argsort(pts[:, 0]), :]
  leftMost = xSorted[:2, :]
  rightMost = xSorted[2:, :]
  leftMost = leftMost[np.argsort(leftMost[:, 1]), :]
  (tl, bl) = leftMost
  D = sci_dist.cdist(tl[np.newaxis], rightMost, "euclidean")[0]
  (br, tr) = rightMost[np.argsort(D)[::-1], :]

  return np.array([tl, tr, br, bl], dtype="float32")

def get_vline_points_inborder(vpd, img, show_vline=False):

  vpd.find_vps(img)
  # vps = vpd.vps_2D
  vl_img, vl_list = vpd.create_debug_VP_image(show_vl=True)

  vl_list_for_vline = vl_list
  vl_img_for_vline = vl_img.copy()

  # print('#       Used vl_list for vline work       #')
  # print('vl_list')
  # print()

  #           Extract Vertical line            #
  h_border = Line((0, 0), (img.shape[1], 0))
  h_border_lower = Line((0, img.shape[0]), (img.shape[1], img.shape[0]))

  reg_xs = list()
  reg_ys = list()
  v_lines = list()
  # h_lines_xrange = list()
  for vl in vl_list_for_vline:
      x0, y0, x1, y1 = vl
      slope = (y1 - y0) / float(x1 - x0)
      angle = math.degrees(math.atan(slope))
      # print(angle)
      if abs(angle) > 80:
        # cv2.line(skl_copy, (int(x1), int(y1)), (int(x0), int(y0)), (255, 0, 0), 3,
        #                  cv2.LINE_AA)
        v_line = Line((x0, y0), (x1, y1))
        [top_point] = v_line.intersection(h_border)
        [bot_point] = v_line.intersection(h_border_lower)
        # v_lines.append()
        reg_xs.append((float(top_point[0]), float(bot_point[0])))
        reg_ys.append((float(top_point[1]), float(bot_point[1])))

      # if abs(angle) < 70:
      #   h_lines_xrange.append((min(x0, x1), max(x0, x1)))

  # print('#        Before Sorting      #')
  # print('reg_xs :', reg_xs)

  center_xs = list()
  for reg_x, reg_y in zip(reg_xs, reg_ys):
    center_xs.append(np.mean(reg_x))
    #           Draw Line         #
    regression(vl_img_for_vline, reg_x, reg_y)

  if show_vline:
    plt.imshow(vl_img_for_vline)
    plt.title('Divided Session')
    plt.show()

  sorted_index = list()
  sorted_center_xs = sorted(center_xs)

  for center_x in sorted_center_xs:
    # print('center_x :', center_x)
    sorted_index.append(center_xs.index(center_x))
  print('sorted_index :', sorted_index)   #

  sorted_reg_xs = list()
  sorted_reg_ys = list()
  for s_index in sorted_index:
    sorted_reg_xs.append(reg_xs[s_index])
    sorted_reg_ys.append(reg_ys[s_index])
  reg_xs = sorted_reg_xs
  reg_ys = sorted_reg_ys

  print()
  # print("#        After Sorting       #")
  # print('reg_xs :', reg_xs)
  # print()

  return reg_xs

def remove_outlier_angle(vl_list, copy_vl_list, angle_list, limit_angle=15):

    if len(vl_list) != 0:
      copy_angle_list = angle_list.copy()
      mean_angle_list = np.array(angle_list).mean()
      # print('mean_angle_list :', mean_angle_list)
      # print('angle_list :', angle_list)
      print('max angle_list gap : ', max(angle_list) - mean_angle_list)

      for angle_index, angle in enumerate(copy_angle_list):
        if abs(angle - mean_angle_list) > limit_angle:
          vl_list.remove(copy_vl_list[angle_index])
          # angle_list.remove(copy_angle_list[angle_index])

      return

def get_hline_points_inborder(img, left_border, right_border, vl_list, top_vl, bot_vl, top_down_ratio=0.66):

  ex_top_vl = list()
  ex_bot_vl = list()

  for vl in vl_list:

    x0, y0, x1, y1 = vl
    l1 = Line((x0, y0), (x1, y1))
    #     v_border line     #
    [left_point] = l1.intersection(left_border)
    # print('left_point :', left_point)
    [right_point] = l1.intersection(right_border)

    #      Figure out current vl is top / bottom vl     #
    if vl in top_vl:
      if (left_point[1] < img.shape[0] * top_down_ratio and right_point[1] < img.shape[0] * top_down_ratio):
          ex_top_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))

    elif vl in bot_vl:
      if (left_point[1] > img.shape[0] * (1 - top_down_ratio) and right_point[1] > img.shape[0] * (1 - top_down_ratio)):
          ex_bot_vl.append((left_point[0], left_point[1], right_point[0], right_point[1]))

  return ex_top_vl, ex_bot_vl

def toppest_bottest_vl(img, top_vl, bot_vl, direction):

    max_y = 0
    min_y = img.shape[0]
    toppest_vl = None
    bottest_vl = None
    index = 1 if direction == 'right' else 3

    for vl in top_vl:
      if vl[index] < min_y:
        toppest_vl = vl
        min_y = vl[index]

    for vl in bot_vl:
      if vl[index] > max_y:
        bottest_vl = vl
        max_y = vl[index]

    return toppest_vl, bottest_vl

def line_mirroring(src_vl, src_point, left_border, right_border):

    x0, y0, x1, y1 = src_vl
    slope = (y1 - y0) / float(x1 - x0)
    angle = - math.degrees(math.atan(slope))

    #       white_max_y 를 - toppest_vl angle 로 지나는 직선      #
    print('src_point :', src_point)
    print('angle :', angle)
    endy = 100 * math.sin(math.radians(angle))
    endx = 100 * math.cos(math.radians(angle))
    temp_vl = Line(src_point, (src_point[0] + endx, src_point[1] + endy))
    [left_point] = temp_vl.intersection(left_border)
    [right_point] = temp_vl.intersection(right_border)
    # print('left_point :', left_point)
    mirrored_vl = (float(left_point[0]), float(left_point[1]), float(right_point[0]), float(right_point[1]))

    return mirrored_vl

def choose_4points(toppest_vl, bottest_vl, src_point, left_border, right_border, parallel):

  x1, y1, x0, y0 = toppest_vl
  if parallel:
    top_line = Line((x1, y1), (x0, y0))
    print('top_line.slope :', float(top_line.slope))
    if abs(top_line.slope) < 0.05:
        src_point = (0, 0)
    parallel_tl = top_line.parallel_line(src_point)

    [tl] = parallel_tl.intersection(left_border)
    [tr] = parallel_tl.intersection(right_border)
    br, bl = bottest_vl[2:], bottest_vl[:2]
  else:
    tl, tr, br, bl = toppest_vl[:2], toppest_vl[2:], bottest_vl[2:], bottest_vl[:2]

  return tl, tr, br, bl

def top_bot_mask(mask_img, threshold=1/10):

    top_white_cnt = 0
    bot_white_cnt = 0
    white_min_x = mask_img.shape[1]
    white_max_x = 0
    white_min_y = mask_img.shape[0]
    white_max_y = 0
    temp_min_y = white_min_y
    temp_max_y = white_max_y
    min_x_coord, max_x_coord, min_y_coord, max_y_coord = None, None, None, None
    top_parallel = False
    bot_parallel = False
    for i in range(mask_img.shape[1]):

      #     Check Top Condition     #
      for j in range(int(mask_img.shape[0] * threshold)):
        # if mask_img[j][i] == 255 and j < thr_min_y:
          # thr_min_y = j
          # thr_min_y_coord = (i, j)
        if mask_img[j][i] == 255:
          top_white_cnt += 1
          if i > white_max_x:
            white_max_x = i
            max_x_coord = (i, j)
          if i < white_min_x:
            white_min_x = i
            min_x_coord = (i, j)

          if j < temp_min_y:
            temp_min_y = j
            temp_min_y_coord = (i, j)

      #           Middle Condition        #
      for j in range(int(mask_img.shape[0] * threshold), int(mask_img.shape[0] * (1 - threshold))):
        if mask_img[j][i] == 255:
          if j > white_max_y:
            white_max_y = j
            max_y_coord = (i, j)
          if j < white_min_y:
            white_min_y = j
            min_y_coord = (i, j)

      #     Check Bot Condition   #
      for j in range(int(mask_img.shape[0] * (1 - threshold)), mask_img.shape[0]):
        # if mask_img[j][i] == 255 and j < thr_min_y:
          # thr_min_y = j
          # thr_min_y_coord = (i, j)
        if mask_img[j][i] == 255:
          bot_white_cnt += 1
          if j > temp_max_y:
            temp_max_y = j
            temp_max_y_coord = (i, j)

    if top_white_cnt / (mask_img.shape[1] * int(mask_img.shape[0] * threshold)) > 0.5:
      top_parallel = True
      min_y_coord = temp_min_y_coord

    if bot_white_cnt / (mask_img.shape[1] * int(mask_img.shape[0] * threshold)) > 0.5:
      #     바닥은 평행 이동 사용하지 않는다.    #
      # bot_parallel = True
      max_y_coord = temp_max_y_coord

    return min_x_coord, max_x_coord, min_y_coord, max_y_coord, top_parallel, bot_parallel

def crop_and_warp(refer, vl_img, src_4p, dst_4p):

  #       Crop      #
  refered = np.asarray(refer)[:vl_img.shape[0], :vl_img.shape[1]]

  # plt.imshow(refered)
  # plt.title('refered')
  # plt.show()

  # compute the perspective transform matrix and then apply it
  matrix = cv2.getPerspectiveTransform(src_4p, dst_4p)
  refered = cv2.warpPerspective(refered, matrix, (refered.shape[1], refered.shape[0]))

  return refered
```

```
#     Refering for one image data   #
def synthesize(org_image, refer_image, vpd, object_, scale_factor=None, show_img=False):

  #       Layout Part       #
  # img = Image.open(layout_path + image)
  # # print(type(img))
  # img_np = np.invert(np.asarray(img))
  # # print(img_np.max(), img_np.min())
  # ret, thr = cv2.threshold(img_np, 254, 255, cv2.THRESH_BINARY_INV)

  #       Mask Part         #
  org_color_np, org_np = np.split(np.asarray(org_image), 2, axis=1)

  # plt.subplot(131)
  # plt.imshow(org)
  # plt.show()
  img_size = (org_np.shape[1], org_np.shape[0])

  # refer = Image.open(refer_path)
  refer = np.asarray(refer_image)

  #     refer_size와 img_size가 동일하거나 refer_size가 작은 경우를 고려해야한다.     #
  print('scale_factor :', scale_factor)
  refer = np.tile(refer, (scale_factor, scale_factor, 1))
  print('refer.shape :', refer.shape)
  # size_ratio = math.floor(min((refer.shape[0] / (org_np.shape[0] * 1.5)), (refer.shape[1] / (org_np.shape[1] * 1.5))))
  size_ratio = min((refer.shape[0] / (org_np.shape[0] * 1.5)), (refer.shape[1] / (org_np.shape[1] * 1.5)))
  print('size_ratio :', size_ratio)
  refer = Image.fromarray(refer).resize((int(refer.shape[1] / size_ratio), int(refer.shape[0] / size_ratio)))

  org_color_np2 = org_color_np.copy()
  org_np = org_np.astype(np.uint8)
  org_np2 = org_np.copy()
  org_np3 = org_np.copy()

  start = time.time()

  if object_ == 'wall':

    #                   Find Best Vline_list                #
    kernel = np.ones((10, 10), np.uint8)

    org_np2_morp = cv2.morphologyEx(org_np2, cv2.MORPH_CLOSE, kernel)
    org_np2_copy = org_np2_morp.copy()
    # org_np2_copy_gray = cv2.cvtColor(org_np2_copy, cv2.COLOR_RGB2GRAY)

    ret, thr_org_np2 = cv2.threshold(org_np2_copy, 127, 255, cv2.THRESH_BINARY)
    thr_org_np2 = cv2.morphologyEx(thr_org_np2, cv2.MORPH_CLOSE, kernel)
    thr_org_np2_copy = thr_org_np2.copy()

    edge_org_np2 = cv2.Canny(org_np2_morp, 20, 100)
    edge_org_np2_copy = edge_org_np2.copy()
    edge_org_np2_copy2 = cv2.Canny((org_np2_copy / 255. * org_np2_copy).astype(np.uint8), 20, 60)

    reg_xs = get_vline_points_inborder(vpd, edge_org_np2_copy2)
    print('#            Used vl_list for vline work          #')
    print('edge_org_np2_copy2')
    print()

    #                                     Find Best Hline                                      #
    #     vline 이 존재하지 않는 경우도 고려해야한다.     #
    if len(reg_xs) == 0:
      reg_xs.append([0, 0])
      # reg_ys.append([0, org_np2.shape[0]])

    #         Divide Session by vline       #
    for reg_index, reg_x in enumerate(reg_xs):

      #     In a Session     #
      #     1.  find vanishing point    #
      #     2.  Find 3 points (vp, top & bottom points)     #
      #     3.  Do warfine and attach to the black plane      #

      print("#      Session Status      #")
      print('reg_index, reg_x :', reg_index, reg_x)

      #     0.  crop by vline     #
            #   Find Max_x, min_x, (Max_y, min_y = org.shape[0], 0)
            #   1.    우편에 한해서 max_x = 우편 vline max_x & min_x = 현재 vline min_x
            #   2.    좌편에 한해서 max_x = 현재 Max_x & min_x = 좌편 vline min_x

      #      vline 별로 양옆으로 작업을 하면 len(vline) = 1의 작업을 반복할 필요가 없어진다.    #
      iter = False
      while True:

        #                   We need Max, min x & y                #
        # four_inters = list()
        find_pair = True
        # centroid_inters = all_centroid_inters[inters_i]

        if not iter:

          #       오른쪽 끝 vline 이면        #
          if reg_index == len(reg_xs) - 1:
            print('rightest vline')
            #   1.    우편에 한해서 max_x = 우편 vline max_x & min_x = 현재 vline min_x
            max_x = org_np.shape[1]
            min_x = np.min(reg_x)

          else:
            print('middle vline')
            next_reg_x = reg_xs[reg_index + 1]
            max_x = np.max(next_reg_x)
            if np.min(reg_x) < 0:
              min_x = 0
            else:
              min_x = np.min(reg_x)

        #     i = 0 에 한해서만 왼쪽으로도 refering 진행, 나머지는 오른쪽으로만     #
        else:
            #   2.    좌편에 한해서 max_x = 현재 Max_x & min_x = 좌편 vline min_x
            max_x = np.max(reg_x)
            min_x = 0

        print('min_x, max_x :', min_x, max_x)
        # max_y = org_np.shape[0]
        # min_y = 0

        #     Make Session      #
        session = org_np2[:, int(min_x):int(max_x)]
        # plt.imshow(session)
        # plt.show()

        if (max_x - min_x) > 50:
          length_thresh = 50
        else:
          length_thresh = 20
          if max_x - min_x < 20:
              #     i != 0 인 경우 break     #
            if reg_index == 0 and np.sum(reg_xs) != 0 and not iter:
              iter = True
              print('iter :', iter)
              continue
            else:
              break

        principal_point = None
        focal_length = 1300 # 1102.79
        seed = None
        vpd = VPDetection(length_thresh, principal_point, focal_length, seed)

        try:
          vpd.find_vps(session)
          # vps = vpd.vps_2D
          vl_img, vl_list = vpd.create_debug_VP_image(show_vl=True)

          #       Closing Morphing by OpenCV      #
          kernel = np.ones((10, 10), np.uint8)
          session = cv2.morphologyEx(session, cv2.MORPH_CLOSE, kernel)

          #     Before Canny    #
          session_prob = session / 255.

          #     Multiply mask map with Edge => Erase Unnecessary vl line    #
          #   multiply one time   #
          session2 = session_prob * (session)
          edge_session2 = cv2.Canny(session2.astype(np.uint8), 20, 60)
          #       two time      #
          # session2 = session_prob**2 * (session)
          # edge_session2 = cv2.Canny(session2.astype(np.uint8), 20, 60)

          vpd.find_vps(edge_session2)
          # vps = vpd.vps_2D
          vl_edge_img2, vl_edge_list2 = vpd.create_debug_VP_image(show_vl=True)

        except Exception as e:
          print("Error in vpd Sessions Zone :", e)
          print()
            #     i != 0 인 경우 break     #
          if reg_index == 0 and np.sum(reg_xs) != 0 and not iter:
            iter = True
            print('iter :', iter)
            continue

          else:
            break

        #         What is best vl_list      #
        vl_list = vl_edge_list2
        print('#            Used vl_list for hline work          #')
        print('vl_edge_list2')
        print()

        h_lines = list()
        # print('vps :', vps)

        top_vl = list()
        bot_vl = list()

        left_vl_list = list()
        right_vl_list = list()

        left_angle = list()
        right_angle = list()

        for vl in vl_list:
          x0, y0, x1, y1 = vl
          slope = (y1 - y0) / float(x1 - x0)
          angle = math.degrees(math.atan(slope))

          if abs(angle) < 70:
            if (y0 + y1) / 2 < vl_img.shape[0] / 2:
              # print(y0, y1)
              top_vl.append(vl)
              if angle < 0:
                left_vl_list.append(vl)
                left_angle.append(abs(angle))
              else:
                right_vl_list.append(vl)
                right_angle.append(abs(angle))
            else:
              bot_vl.append(vl)
              if angle > 0:
                left_vl_list.append(vl)
                left_angle.append(abs(angle))
              else:
                right_vl_list.append(vl)
                right_angle.append(abs(angle))
            # cv2.line(skl_copy, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 2,
            #                  cv2.LINE_AA)
            h_lines.append(vl)
          #   regression(vl_img, (x0, x1), (y0, y1), color=(0,255,255),axis=1)

        print('#        Original vl list        #')
        print('len(left_vl_list) :', len(left_vl_list))
        print('len(right_vl_list) :', len(right_vl_list))
        print()

        #             소실점 방향 선택            #
        #       original vl_list 는 remove 가 존재하기 때문에 copy_ version 사용한다.       #
        copy_left_vl_list = left_vl_list.copy()
        copy_right_vl_list = right_vl_list.copy()
        print('len(copy_left_vl_list) :', len(copy_left_vl_list))
        print('len(copy_right_vl_list) :', len(copy_right_vl_list))

        if len(copy_left_vl_list) >= len(copy_right_vl_list):
          direction = 'left'
          vl_list, copy_vl_list, angle = left_vl_list, copy_left_vl_list, left_angle
        else:
          direction = 'right'
          vl_list, copy_vl_list, angle = right_vl_list, copy_right_vl_list, right_angle

        print("#            Remove Outliered Angle in Hlines           #")
        remove_outlier_angle(vl_list, copy_vl_list, angle)
        # remove_outlier_angle(right_vl_list, copy_right_vl_list, right_angle)
        print()

        print('len(left_vl_list) :', len(left_vl_list))
        print('len(right_vl_list) :', len(right_vl_list))
        print()

        # if len(left_vl_list) == 0 and len(right_vl_list) == 0:
        if len(vl_list) == 0:
          #     i != 0 인 경우 break     #
          if reg_index == 0 and not iter:
            iter = True
            print('iter :', iter)
            continue

          else:
            break

        #                       Find External vn_line                     #
        l2 = Line((0, 0), (0, vl_img.shape[0]))
        l3 = Line((vl_img.shape[1], 0), (vl_img.shape[1], vl_img.shape[0]))

        ex_top_vl, ex_bot_vl = get_hline_points_inborder(vl_img, l2, l3, vl_list, top_vl, bot_vl)

        print()
        print('#          Extended vl list left / right  TB Condition        #')
        print('len(ex_top_vl) :', len(ex_top_vl))
        print('len(ex_bot_vl) :', len(ex_bot_vl))
        print()

        #       find min max Line     #
        toppest_vl, bottest_vl = toppest_bottest_vl(vl_img, ex_top_vl, ex_bot_vl, direction)

        #       Figure out min_y & max_y of thr_session       #
        # print('thr_session.shape :', thr_session.shape)
        gray = cv2.cvtColor(session, cv2.COLOR_RGB2GRAY)
        ret, thr_session = cv2.threshold(session, 127, 255, cv2.THRESH_BINARY)
        thr_session = cv2.morphologyEx(thr_session, cv2.MORPH_CLOSE, kernel)
        thr_session_gray = cv2.cvtColor(thr_session, cv2.COLOR_RGB2GRAY)
        print('thr_session_gray.shape :', thr_session_gray.shape)

        min_x_coord, max_x_coord, min_y_coord, max_y_coord, top_parallel, _ = top_bot_mask(thr_session_gray)

        print('max_x_coord, min_x_coord :', max_x_coord, min_x_coord)
        print('max_y_coord, min_y_coord :', max_y_coord, min_y_coord)

        try:

          #     None 이라면 대칭이동을 통해 만들어주어야한다.     #
          if len(ex_bot_vl) == 0:
            if toppest_vl is not None:
              bottest_vl = line_mirroring(toppest_vl, max_y_coord, l2, l3)
              print('bottest_vl :', bottest_vl)

          elif len(ex_top_vl) == 0:
            if bottest_vl is not None:
              toppest_vl = line_mirroring(bottest_vl, min_y_coord, l2, l3)
              print('toppest_vl :', toppest_vl)

          for (x1, y1, x0, y0) in [toppest_vl, bottest_vl]:
            cv2.line(vl_img, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 3, cv2.LINE_AA)

        except Exception as e:
          print('Error in drawing top, bot -est lines :', e)

          #     i != 0 인 경우 break     #
          if reg_index == 0 and not iter:
            iter = True
            print('iter :', iter)
            continue

          else:
            break

        #               Choose 4 points             #
        tl, tr, br, bl = choose_4points(toppest_vl, bottest_vl, min_x_coord, l2, l3, top_parallel)

        #       Refering      #
        #   tl, tr, br, bl    #
        #     refer를 위해 src_x => 0 으로 맞춰준다.    #
        src = np.array([
            [0, 0],
            [vl_img.shape[1], 0],
            [vl_img.shape[1], vl_img.shape[0]],
            [0, vl_img.shape[0]]], dtype = "float32")
        dst = np.array([list(tl),
                        list(tr),
                        list(br),
                        list(bl)], dtype = "float32")

        print()
        print("#          Warping Points        #")
        print('src :', src)
        print('dst :', dst)

        refered = crop_and_warp(refer, vl_img, src, dst)

        org_color_np2[:, int(min_x):int(max_x)] = refered

        #     i != 0 인 경우 break     #
        if reg_index == 0 and np.sum(reg_xs) != 0 and not iter:
          iter = True
          print('iter :', iter)

        else:
          break

  elif object_ == 'floor':

      # print(thr_org_np2.dtype)
      thr_org_np2 = cv2.cvtColor(thr_org_np2, cv2.COLOR_RGB2GRAY)
      contours, _ = cv2.findContours(thr_org_np2, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
      min_height_point = np.array([[thr_org_np2.shape[0], thr_org_np2.shape[1]]])
      # print(min_height_point)
      for contour in contours:
        # print(contour.shape)
        for point in contour:
          # print(point[0][1], min_height_point[0][1])
          if 0 < point[0][1] <= min_height_point[0][1]:
            # print(min_height_point[0][1])
            min_height_point = point

      dst = np.float32([min_height_point[0], [0, thr_org_np2.shape[1]], [thr_org_np2.shape[0], thr_org_np2.shape[1]]])
      src = np.float32([[min_height_point[0][0], 0], [0, thr_org_np2.shape[1]], [thr_org_np2.shape[0], thr_org_np2.shape[1]]])
      matrix = cv2.getAffineTransform(src, dst)

      refer = np.asarray(refer)
      refer = cv2.warpAffine(refer, matrix, (refer.shape[1], refer.shape[0]))
      row, col, _ = org_color_np2.shape
      org_color_np2 = refer[:row, :col]

  #         Refer의 검은 부분은 original image로 채운다.       #
  if object_ is None:
      # print('org_color_np2.shape :', org_color_np2.shape)
      row, col, _ = org_color_np2.shape
      org_color_np2 = np.array(refer)[:row, :col]

  else:
    org_color_np2 = np.where(org_color_np2 == 0, org_color_np, org_color_np2)

  print('elapsed time :', time.time() - start)
  print()

  # print('np.max(org_np) :', np.max(org_np))
  org_np = org_np / 255.
  refer = org_color_np2
  # plt.show()

  #     1.wall_mask * reference color image + (1 - wall_mask) * original_image)    #
  syn_ = org_np * (refer) + (1 - org_np) * org_color_np
  syn = org_np * syn_ + (1 - org_np) * org_color_np

  #       Brightness Preservation      #
  org_hsv = cv2.cvtColor(np.uint8(org_color_np), cv2.COLOR_RGB2HSV)
  syn_hsv = cv2.cvtColor(np.uint8(syn), cv2.COLOR_RGB2HSV)

  org_h, org_s, org_v = cv2.split(org_hsv)
  syn_h, syn_s, syn_v = cv2.split(syn_hsv)

  mask_map = org_np[:, :, 0]
  # print(mask_map.shape)

  w = org_v / 255.
  w = w / 1.87
  syn_v2 = mask_map * syn_v + (1 - mask_map) * org_v
  syn_v2 = (1 - w) * syn_v2 + w * org_v
  syn_v2 = syn_v2.astype(np.uint8)

  final_syn_hsv = cv2.merge([syn_h, syn_s, syn_v2])
  final_syn = cv2.cvtColor(final_syn_hsv, cv2.COLOR_HSV2RGB)

  if show_img:

    plt.figure(figsize=(15, 10))

    plt.subplot(141)
    plt.imshow(org_np3)
    plt.axis('off')

    plt.subplot(142)
    plt.imshow(org_color_np2)
    plt.axis('off')

    plt.subplot(143)
    plt.imshow(final_syn)
    plt.axis('off')

    plt.subplot(144)
    plt.imshow(org_color_np)
    plt.axis('off')
    plt.show()

  return final_syn
```

```
if __name__ == '__main__':

  object_ = 'wall'
  # mask_path = '../previous_test_result/' + object_ + '/'
  mask_path = '../test_result/' + object_ + '/'
  # layout_path = '../test_layout_result/' + object_ + '/'
  # layout_path = '../mask_layout_data/' + object_ + '/'
  # layout_list = os.listdir(layout_path)

  refer_path = '../refer_data/' + object_ + '/방_벽지/'
  # save_path = mask_path.replace('previous_test_result', 'syn_result')
  save_path = mask_path.replace('test_result', 'syn_result')

  length_thresh = 70
  principal_point = None
  focal_length = 1300 # 1102.79
  seed = 1300
  vpd = VPDetection(length_thresh, principal_point, focal_length, seed)

  for image in os.listdir(mask_path):

    # image = '720x480_20190315154538242_cqtFKEk4r5.png'
    image_name = image.split('.')[0]
    try:
      os.makedirs(os.path.join(save_path, image_name))

    except Exception as e:
      print('Error in makedirs :', e)

    for refer in os.listdir(refer_path):

      try:

        print('image :', image)
        print('refer :', refer)

        org = Image.open(mask_path + image)
        refer_img = Image.open(refer_path + refer)
        # result_img = synthesize(org, refer_img, vpd, object_=object_, scale_factor=1, show_img=True)
        result_img = synthesize(org, refer_img, vpd, object_=None, scale_factor=1, show_img=True)

        Image.fromarray(result_img).save(os.path.join(save_path, image_name, refer))

      except Exception as e:
        print('Error in load images :', e)

    # break

```

    Output hidden; open in https://colab.research.google.com to view.

```
6.486232280731201
elapsed time : 6.074836015701294
elapsed time : 6.072072744369507
elapsed time : 6.072662591934204
elapsed time : 6.140763998031616
---
elapsed time : 4.42145848274231
elapsed time : 4.432448148727417
elapsed time : 4.47941255569458
---
elapsed time : 2.750857353210449
elapsed time : 2.7681655883789062
elapsed time : 2.7948927879333496
elapsed time : 2.882319927215576

```

```

```

Layout Estimation Func Final without AI_latest.ipynb


# Hello world!

I'm a test Markdown document.

**Here's some bold text** and *here's some italic text*. `This is inline code`.

# Markdown Tables

First Name  |  Last Name  |  Location           |  Allegiance
------------|-------------|---------------------|-----------------
Mance       |  Rayder     |  North of the Wall  |  Wildlings
Margaery    |  Tyrell     |  The Reach          |  House Tyrell
Danerys     |  Targaryen  |  Meereen            |  House Targaryen
Tyrion      |  Lannister  |  King's Landing     |  House Lannister

# Code blocks

    This code
    is in
    a code block.

Here's a syntax-highlighted code block:

```python
#!/usr/bin/env python3

import sys

if __name__ == '__main__':
    print('This is highlighted Python code!')
    sys.exit(0)
```

# Page Layout with \LaTeX\ Commands

Here's a forced page break.

\pagebreak

# LaTeX support

This document supports inline \LaTeX!

Here's the proof: $\frac{n!}{k!(n-k)!} = \binom{n}{k}$

Creating a footnote is easy.\footnote{An example footnote.}

Here's an equation:

$$
  x = a_0 + \cfrac{1}{a_1
          + \cfrac{1}{a_2
          + \cfrac{1}{a_3 + \cfrac{1}{a_4} } } }
$$

Here are some numbered equations:

\begin{equation}
 f(x)=(x+a)(x+b)
\end{equation}

\begin{equation}
5^2 - 5 = 20
\end{equation}

\begin{equation}
a = bq + r
\end{equation}

Here's some multi-line math stuff:

$$
 u(x) =
  \begin{cases}
   \exp{x} & \text{if } x \geq 0 \\
   1       & \text{if } x < 0
  \end{cases}
$$

\begin{align*}
 f(x) &= (x+a)(x+b) \\
 &= x^2 + (a+b)x + ab
\end{align*}

```python
import ROOT
```

    Welcome to JupyROOT 6.08/06

```python
h = ROOT.TH1F("gauss","Example histogram",100,-4,4)
h.FillRandom("gaus")
```

```python
c = ROOT.TCanvas("myCanvasName","The Canvas Title",800,600)
h.Draw()
```

```python
c.Draw()
```

```python

```