# Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.  You may obtain a copy of the License at:
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.

# ***********************************************  RUN MODULE **********************************************************

"""

Overview
--------

.. _Run_Overview:

The :keyword:`run` function is used for executing a Mechanism, Process or System.  It can be called directly, however
it is typically invoked by calling the :keyword:`run` method of the Component to be run.  It  executes a Component by
calling the Component's :keyword:`execute` method.  While a Component's :keyword:`execute` method can be called
directly, using its :keyword:`run` method is easier because it:

    * allows multiple rounds of execution to be run in sequence, whereas the :keyword:`execute` method of a Component
      runs only a single execution of the object;
    ..
    * uses simpler formats for specifying `inputs <Run_Inputs>` and `targets <Run_Targets>`;
    ..
    * automatically aggregates results across executions and stores them in the results attribute of the object.

Understanding a few basic concepts about how the :keyword:`run` function operates will make it easier to use the
:keyword:`execute` and :keyword:`run` methods of PsyNeuLink Components.  These are discussed below.

.. _Run_Scope_of_Execution:

Scope of Execution
~~~~~~~~~~~~~~~~~~

When the :keyword:`run` method of a Component is called, it executes that Component and all others within its scope of
execution.  For a `Mechanism <Mechanism>`, the scope of execution is simply the Mechanism itself.  For a `Process`,
the scope of
execution is all of the Mechanisms specified in its `pathway` attribute.  For a `System`, the scope of execution is
all of the Mechanisms in the Processes specified in the System's `processes <System.processes>` attribute.

.. _Run_Timing:

Timing
~~~~~~

When :keyword:`run` is called by a Component, it calls that Component's :keyword:`execute` method once for each
`input <Run_Inputs>`  (or set of inputs) specified in the call to :keyword:`run`, which constitutes a `TRIAL` of
execution.  For each `TRIAL`, the Component makes repeated `calls to its Scheduler <Scheduler_Execution>`,
executing the Components it specifies in each `TIME_STEP`, until every Component has been executed at least once or
another `termination condition <Scheduler_Termination_Conditions>` is met.  The `Scheduler` can be used in combination
with `Condition` specifications for individual Components to execute different Components at different time scales.

.. _Run_Inputs:

Inputs
~~~~~~

The :keyword:`run` function presents the inputs for each `TRIAL` to the input_states of the relevant Mechanisms in
the `scope of execution <Run_Scope_of_Execution>`. These are specified in the **inputs** argument of a Component's
:keyword:`execute` or :keyword:`run` method.

Inputs are specified in a Python dictionary where the keys are `ORIGIN` Mechanisms, and the values are lists in which
the i-th element represents the input value to the mechanism on trial i. Each input value must be compatible with the
shape of the mechanism's variable. This means that the inputs to an origin mechanism are usually specified by a
list of 2d lists/arrays, though `some shorthand notations are allowed <Input_Specification_Examples>`.

::

        >>> import psyneulink as pnl

        >>> a = pnl.TransferMechanism(name='a',
        ...                          default_variable=[[0.0, 0.0]])
        >>> b = pnl.TransferMechanism(name='b',
        ...                          default_variable=[[0.0], [0.0]])
        >>> c = pnl.TransferMechanism(name='c')

        >>> p1 = pnl.Process(pathway=[a, c],
        ...                 name='p1')
        >>> p2 = pnl.Process(pathway=[b, c],
        ...                 name='p2')

        >>> s = pnl.System(processes=[p1, p2])

        >>> input_dictionary = {a: [[[1.0, 1.0]], [[1.0, 1.0]]],
        ...                    b: [[[2.0], [3.0]], [[2.0], [3.0]]]}

        >>> s.run(inputs=input_dictionary)

.. _Run_Inputs_Fig:

.. figure:: _static/input_spec_variables.svg
   :alt: Example input specifications with variable

.. note::
    Keep in mind that a mechanism's variable is the concatenation of its input states. In other words, a fully specified
    mechanism variable is a 2d list/array in which the i-th element is the variable of the mechanism's i-th input state.
    Because of this `relationship between a mechanism's variable and its input states <Mechanism_Figure>`, it is also
    valid to think about the input specification for a given origin mechanism as a nested list of values for each input
    state on each trial.

    .. _Run_Inputs_Fig_States:

    .. figure:: _static/input_spec_states.svg
       :alt: Example input specifications with input states

The number of inputs specified **must** be the same for all origin mechanisms in the system. In other words, all of the
values in the input dictionary must have the same length.

If num_trials is not in use, the number of inputs provided determines the number of trials in the run. For example, if
five inputs are provided for each origin mechanism, and num_trials is not specified, the system will execute five times.

+----------------------+-------+------+------+------+------+
| Trial #              |1      |2     |3     |4     |5     |
+----------------------+-------+------+------+------+------+
| Input to Mechanism a |1.0    |2.0   |3.0   |4.0   |5.0   |
+----------------------+-------+------+------+------+------+

::

        >>> import psyneulink as pnl

        >>> a = pnl.TransferMechanism(name='a')
        >>> b = pnl.TransferMechanism(name='b')

        >>> p1 = pnl.Process(pathway=[a, b])

        >>> s = pnl.System(processes=[p1])

        >>> input_dictionary = {a: [[[1.0]], [[2.0]], [[3.0]], [[4.0]], [[5.0]]]}

        >>> s.run(inputs=input_dictionary)

If num_trials is in use, `run` will iterate over the inputs until num_trials is reached. For example, if five inputs
are provided for each `ORIGIN` mechanism, and num_trials = 7, the system will execute seven times. The first two
items in the list of inputs will be used on the 6th and 7th trials, respectively.

+----------------------+-------+------+------+------+------+------+------+
| Trial #              |1      |2     |3     |4     |5     |6     |7     |
+----------------------+-------+------+------+------+------+------+------+
| Input to Mechanism a |1.0    |2.0   |3.0   |4.0   |5.0   |1.0   |2.0   |
+----------------------+-------+------+------+------+------+------+------+

::

        import psyneulink as pnl

        a = pnl.TransferMechanism(name='a')
        b = pnl.TransferMechanism(name='b')

        p1 = pnl.Process(pathway=[a, b])

        s = pnl.System(processes=[p1])

        input_dictionary = {a: [[[1.0]], [[2.0]], [[3.0]], [[4.0]], [[5.0]]]}

        s.run(inputs=input_dictionary,
              num_trials=7)

.. _Input_Specification_Examples:

For convenience, condensed versions of the input specification described above are also accepted in the following
situations:

* **Case 1: Origin mechanism has only one input state**
+--------------------------+-------+------+------+------+------+
| Trial #                  |1      |2     |3     |4     |5     |
+--------------------------+-------+------+------+------+------+
| Input to **Mechanism a** |1.0    |2.0   |3.0   |4.0   |5.0   |
+--------------------------+-------+------+------+------+------+

Complete input specification:

::

        import psyneulink as pnl

        a = pnl.TransferMechanism(name='a')
        b = pnl.TransferMechanism(name='b')

        p1 = pnl.Process(pathway=[a, b])

        s = pnl.System(processes=[p1])

        input_dictionary = {a: [[[1.0]], [[2.0]], [[3.0]], [[4.0]], [[5.0]]]}

        s.run(inputs=input_dictionary)
..

Shorthand - drop the outer list on each input because **Mechanism a** only has one input state:

::

        input_dictionary = {a: [[1.0], [2.0], [3.0], [4.0], [5.0]]}

        s.run(inputs=input_dictionary)
..

Shorthand - drop the remaining list on each input because **Mechanism a**'s variable is length 1:

::

        input_dictionary = {a: [1.0, 2.0, 3.0, 4.0, 5.0]}

        s.run(inputs=input_dictionary)
..

* **Case 2: Only one input is provided for the mechanism**

+--------------------------+------------------+
| Trial #                  |1                 |
+--------------------------+------------------+
| Input to **Mechanism a** |[[1.0], [2.0]]    |
+--------------------------+------------------+

Complete input specification:

::

        import psyneulink as pnl

        a = pnl.TransferMechanism(name='a',
                                  default_variable=[[0.0], [0.0]])
        b = pnl.TransferMechanism(name='b')

        p1 = pnl.Process(pathway=[a, b])

        s = pnl.System(processes=[p1])

        input_dictionary = {a: [[[1.0], [2.0]]]}

        s.run(inputs=input_dictionary)
..

Shorthand - drop the outer list on **Mechanism a**'s input specification because there is only one trial:

::

        input_dictionary = {a: [[1.0], [2.0]]}

        s.run(inputs=input_dictionary)
..

* **Case 3: The same input is used on all trials**

+--------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
| Trial #                  |1                  |2                  |3                  |4                  |5                  |
+--------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+
| Input to **Mechanism a** | [[1.0], [2.0]]    | [[1.0], [2.0]]    | [[1.0], [2.0]]    | [[1.0], [2.0]]    | [[1.0], [2.0]]    |
+--------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+

Complete input specification:

::

        import psyneulink as pnl

        a = pnl.TransferMechanism(name='a',
                                  default_variable=[[0.0], [0.0]])
        b = pnl.TransferMechanism(name='b')

        p1 = pnl.Process(pathway=[a, b])

        s = pnl.System(processes=[p1])

        input_dictionary = {a: [[[1.0], [2.0]], [[1.0], [2.0]], [[1.0], [2.0]], [[1.0], [2.0]], [[1.0], [2.0]]]}

        s.run(inputs=input_dictionary)
..

Shorthand - drop the outer list on **Mechanism a**'s input specification and use `num_trials` to repeat the input value

::

        input_dictionary = {a: [[1.0], [2.0]]}

        s.run(inputs=input_dictionary,
              num_trials=5)
..

* **Case 4: There is only one origin mechanism**

+--------------------------+-------------------+-------------------+
| Trial #                  |1                  |2                  |
+--------------------------+-------------------+-------------------+
| Input to **Mechanism a** | [1.0, 2.0, 3.0]   |  [1.0, 2.0, 3.0]  |
+--------------------------+-------------------+-------------------+

Complete input specification:

::

        import psyneulink as pnl

        a = pnl.TransferMechanism(name='a',
                                  default_variable=[[1.0, 2.0, 3.0]])
        b = pnl.TransferMechanism(name='b')

        p1 = pnl.Process(pathway=[a, b])

        s = pnl.System(processes=[p1])

        input_dictionary = input_dictionary = {a: [[1.0, 2.0, 3.0], [1.0, 2.0, 3.0]]}

        s.run(inputs=input_dictionary)
..

Shorthand - specify **Mechanism a**'s inputs in a list because it is the only origin mechanism

::

        input_list = [[1.0, 2.0, 3.0], [1.0, 2.0, 3.0]]

        s.run(inputs=input_list)
..

COMMENT:
.. _Run_Initial_Values:

Initial Values
~~~~~~~~~~~~~~

Any Mechanism that is the `sender <Projection.Projection.sender>` of a Projection that closes a loop in a Process or
System, and that is not an `ORIGIN` Mechanism, is designated as `INITIALIZE_CYCLE`. An initial value can be assigned
to such Mechanisms, that will be used to initialize them when the Process or System is first run.  These values are
specified in the **initial_values** argument of :keyword:`run`, as a dictionary. The key for each entry must
be a Mechanism designated as `INITIALIZE_CYCLE`, and its value an input for the Mechanism to be used as its initial
value.  The size of the input (length of the outermost level if it is a list, or axis 0 if it is an np.ndarray),
must equal the number of InputStates of the Mechanism, and the size of each value must match (in number and type of
elements) that of the `variable <InputState.InputState.variable>` for the corresponding InputState.
COMMENT

.. _Run_Targets:

Targets
~~~~~~~

If learning is specified for a `Process <Process_Learning_Sequence>` or `System <System_Execution_Learning>`, then
target values for each `TRIAL` must be provided for each `TARGET` Mechanism in the Process or System being run.  These
are specified in the **targets** argument of the :keyword:`execute` or :keyword:`run` method.

Recall that the `TARGET`, or `ComparatorMechanism`, of a learning sequence receives a TARGET, which is provided by the
user at run time, and a SAMPLE, which is received from a projection sent by the last mechanism of the learning sequence.
The TARGET and SAMPLE values for a particular `TARGET` Mechanism must have the same shape. See `learning sequence
<Process_Learning_Sequence>` for more details on how these components relate to each other.

The standard format for specifying targets is a Python dictionary where the keys are the last mechanism of each learning
sequence, and the values are lists in which the i-th element represents the target value for that learning sequence on
trial i. There must be the same number of keys in the target specification dictionary as there are `TARGET` Mechanisms
in the system. Each target value must be compatible with the shape of the `TARGET` mechanism's TARGET `input state
<ComparatorMechanism.input_states>`. This means that for a given key (which is always the last mechanism of the
learning sequence) in the target specification dictionary, the value is usually a list of 1d lists/arrays.

The number of targets specified for each Mechanism must equal the number specified for the **inputs** argument;  as
with **inputs**, if the number of `TRIAL` \\s specified is greater than the number of inputs (and targets), then the
list will be cycled until the number of `TRIAL` \\s specified is completed.

+------------------------------------------+--------------+--------------+
| Trial #                                  |1             |   2          |
+------------------------------------------+--------------+--------------+
| Target value for the learning sequence   | [1.0, 1.0]   |   [2.0, 2.0] |
| containing **Mechanism b**               |              |              |
+------------------------------------------+--------------+--------------+
| Target value for the learning sequence   |  [1.0]       |   [2.0]      |
| containing **Mechanism c**               |              |              |
+------------------------------------------+--------------+--------------+

::

        >>> import psyneulink as pnl

        >>> a = pnl.TransferMechanism(name="a")
        >>> b = pnl.TransferMechanism(name="b",
        ...                           default_variable=np.array([[0.0, 0.0]]))
        >>> c = pnl.TransferMechanism(name="c")

        >>> learning_sequence_1 = pnl.Process(name="learning-sequence-1",
        ...                                   pathway=[a, b],
        ...                                   learning=pnl.ENABLED)
        >>> learning_sequence_2 = pnl.Process(name="learning-sequence-2",
        ...                                   pathway=[a, c],
        ...                                   learning=pnl.ENABLED)

        >>> s = pnl.System(name="learning-system",
        ...                processes=[learning_sequence_1, learning_sequence_2])

        >>> input_dictionary = {a: [[[0.1]], [[0.2]]]}

        >>> target_dictionary = {b: [[1.0, 1.0], [2.0, 2.0]],
        ...                      c: [[1.0], [2.0]]}

        >>> s.run(inputs=input_dictionary,
        ...       targets=target_dictionary)

.. _Run_Targets_Fig:

.. figure:: _static/target_spec_dictionary.svg
   :alt: Example of dictionary format of target specification

Alternatively, the value for a given key (last mechanism in the learning sequence) in the target specification
dictionary may be a function. The output of that function must be compatible with the shape of the `TARGET` mechanism's
TARGET `input state <ComparatorMechanism.input_states>`. The function will be executed at the start of the learning
portion of each trial. This format allows targets to be constructed programmatically, in response
to computations made during the run.

::

        >>> a = TransferMechanism(name="a")
        >>> b = TransferMechanism(name="b",
        ...                       default_variable=np.array([[0.0, 0.0]]))

        >>> learning_sequence = Process(name="learning-sequence",
        ...                             pathway=[A, B],
        ...                             learning=ENABLED)

        >>> s = System(name="learning-system",
        ...            processes=[LP])

        >>> def target_function():
        ...     val_1 = NormalDist(mean=3.0).function()
        ...     val_2 = NormalDist(mean=3.0).function()
        ...     target_value = np.array([val_1, val_2])
        ...     return target_value

        >>> s.run(inputs={A: [[[1.0]], [[2.0]], [[3.0]]]},
        ...       targets={B: target_function})

.. note::

    Target specification dictionaries that provide values for multiple learning sequences may contain functions for some
    learning sequences and lists of values for others.

Finally, for convenience, if there is only one learning sequence in a system, the targets may be specified in a list,
rather than a dictionary.

+------------------------------------------+-------+------+------+------+------+
| Trial #                                  |1      |2     |3     |4     |5     |
+------------------------------------------+-------+------+------+------+------+
| Target corresponding to  **Mechanism b** |1.0    |2.0   |3.0   |4.0   |5.0   |
+------------------------------------------+-------+------+------+------+------+

Complete input specification:

::

        >>> import psyneulink as pnl

        >>> a = pnl.TransferMechanism(name='a')
        >>> b = pnl.TransferMechanism(name='b')

        >>> p1 = pnl.Process(pathway=[a, b])

        >>> s = pnl.System(processes=[p1])

        >>> input_dictionary = {a: [[[1.0]], [[2.0]], [[3.0]], [[4.0]], [[5.0]]]}
        >>> target_dictionary = {b: [[1.0], [2.0], [3.0], [4.0], [5.0]]}

        >>> s.run(inputs=input_dictionary,
        ...       targets=target_dictionary)

Shorthand - specify the targets in a list because there is only one learning sequence:

::

        >>> target_list = [[1.0], [2.0], [3.0], [4.0], [5.0]]

        >>> s.run(inputs=input_dictionary,
        ...       targets=target_list)

.. _Run_Class_Reference:

Class Reference
---------------

"""

import datetime
import warnings
from collections import Iterable
from numbers import Number

import numpy as np
import typecheck as tc

from psyneulink.components.component import function_type
from psyneulink.components.shellclasses import Mechanism, Process_Base, System_Base
from psyneulink.globals.context import ContextFlags
from psyneulink.globals.keywords import INPUT_LABELS_DICT, MECHANISM, \
    PROCESS, RUN, SAMPLE, SYSTEM, TARGET
from psyneulink.globals.log import LogCondition
from psyneulink.scheduling.time import TimeScale

__all__ = [
    'EXECUTION_SET_DIM', 'MECHANISM_DIM', 'RunError', 'STATE_DIM', 'run'
]

EXECUTION_SET_DIM = 0
MECHANISM_DIM = 2
STATE_DIM = 3  # Note: only meaningful if mechanisms are homnogenous (i.e., all have the same number of states -- see chart below):

class RunError(Exception):
     def __init__(object, error_value):
         object.error_value = error_value

     def __str__(object):
         return repr(object.error_value)

@tc.typecheck
def run(object,
        inputs,
        num_trials:tc.optional(int)=None,
        initialize:bool=False,
        initial_values:tc.optional(tc.any(list, dict, np.ndarray))=None,
        targets=None,
        learning:tc.optional(bool)=None,
        call_before_trial:tc.optional(callable)=None,
        call_after_trial:tc.optional(callable)=None,
        call_before_time_step:tc.optional(callable)=None,
        call_after_time_step:tc.optional(callable)=None,
        termination_processing=None,
        termination_learning=None,
        context=ContextFlags.COMMAND_LINE):
    """run(                      \
    inputs,                      \
    num_trials=None,             \
    initialize=False,            \
    intial_values=None,          \
    targets=None,                \
    learning=None,               \
    call_before_trial=None,      \
    call_after_trial=None,       \
    call_before_time_step=None,  \
    call_after_time_step=None,   \)

    Run a sequence of executions for a `Process` or `System`.

    COMMENT:
        First, validate inputs (and targets, if learning is enabled).  Then, for each `TRIAL`:
            * call call_before_trial if specified;
            * for each time_step in the trial:
                * call call_before_time_step if specified;
                * call ``object.execute`` with inputs, and append result to ``object.results``;
                * call call_after_time_step if specified;
            * call call_after_trial if specified.
        Return ``object.results``.

        The inputs argument must be a list or an np.ndarray array of the appropriate dimensionality:
            * the inner-most dimension must equal the length of object.instance_defaults.variable (i.e., the input to the object);
            * for Mechanism format, the length of the value of all entries must be equal (== number of executions);
            * the outer-most dimension is the number of input sets (num_input_sets) specified (one per execution)
                Note: num_input_sets need not equal num_trials (the number of executions to actually run)
                      if num_trials > num_input_sets:
                          executions will cycle through input_sets, with the final one being only a partial cycle
                      if num_trials < num_input_sets:
                          the executions will only partially sample the input sets
    COMMENT

   Arguments
   ---------

    inputs : List[input] or ndarray(input) : default default_variable for a single `TRIAL`
        the input for each `TRIAL` in a sequence (see `Run_Inputs` for detailed description of formatting
        requirements and options).

    num_trials : int : default None
        the number of `TRIAL` \\s to run.  If it is `None` (the default), then a number of `TRIAL` \\s run will be equal
        equal to the number of items specified in the **inputs** argument.  If **num_trials** exceeds the number of
        inputs, then the inputs will be cycled until the number of `TRIAL` \\s specified have been run.

    initialize : bool default False
        calls the `initialize <System.initialize>` method of the System prior to the first `TRIAL`.

    initial_values : Dict[Mechanism:List[input]], List[input] or np.ndarray(input) : default None
        the initial values assigned to Mechanisms designated as `INITIALIZE_CYCLE`.

    targets : dict : default None
        the target values assigned to the `ComparatorMechanism` of each learning sequence on each `TRIAL`.

    learning : bool :  default None
        enables or disables learning during execution for a `Process <Process_Execution_Learning>` or
        `System <System_Execution_Learning>`.  If it is not specified, the current state of learning is left intact.
        If it is `True`, learning is forced on; if it is `False`, learning is forced off.

    call_before_trial : Function : default `None`
        called before each `TRIAL` in the sequence is run.

    call_after_trial : Function : default `None`
        called after each `TRIAL` in the sequence is run.

    call_before_time_step : Function : default ``None`
        called before each `TIME_STEP` is executed.

    call_after_time_step : Function : default `None`
        called after each `TIME_STEP` is executed.

    termination_processing : Dict[TimeScale: Condition]
        a dictionary containing `Condition`\\ s that signal the end of the associated `TimeScale` within the :ref:`processing
        phase of execution <System_Execution_Processing>`

    termination_learning : Dict[TimeScale: Condition]
        a dictionary containing `Condition`\\ s that signal the end of the associated `TimeScale` within the :ref:`learning
        phase of execution <System_Execution_Learning>`

   Returns
   -------

    <object>.results : List[OutputState.value]
        list of the values, for each `TRIAL`, of the OutputStates for a Mechanism run directly,
        or of the OutputStates of the `TERMINAL` Mechanisms for the Process or System run.
    """
    from psyneulink.globals.context import ContextFlags

    # small version of 'sequence' format in the once case where it was still working (single origin mechanism)
    if isinstance(inputs, (list, np.ndarray)):
        if len(object.origin_mechanisms) == 1:
            inputs = {object.origin_mechanisms[0]: inputs}
        else:
            raise RunError("Inputs to {} must be specified in a dictionary with a key for each of its {} origin "
                           "mechanisms.".format(object.name, len(object.origin_mechanisms)))
    elif not isinstance(inputs, dict):
        if len(object.origin_mechanisms) == 1:
            raise RunError("Inputs to {} must be specified in a list or in a dictionary with the origin mechanism({}) "
                           "as its only key".format(object.name, object.origin_mechanisms[0].name))
        else:
            raise RunError("Inputs to {} must be specified in a dictionary with a key for each of its {} origin "
                           "mechanisms.".format(object.name, len(object.origin_mechanisms)))

    inputs, num_inputs_sets = _adjust_stimulus_dict(object, inputs)

    if num_trials is not None:
        num_trials = num_trials
    else:
        num_trials = num_inputs_sets

    # num_trials = num_trials or num_inputs_sets  # num_trials may be provided by user, otherwise = # of input sets

    if targets is not None:

        if isinstance(targets, dict):
            targets, num_targets = _adjust_target_dict(object, targets)

        elif isinstance(targets, (list, np.ndarray)):
            # small version of former 'sequence' format -- only allowed if there is a single Target mechanism
            if len(object.target_mechanisms) == 1:
                targets = {object.target_mechanisms[0].input_states[SAMPLE].path_afferents[0].sender.owner: targets}
                targets, num_targets = _adjust_target_dict(object, targets)
            else:
                raise RunError("Target values for {} must be specified in a dictionary.".format(object.name))

        elif isinstance(targets, function_type):
            if len(object.target_mechanisms) == 1:
                targets = {object.target_mechanisms[0].input_states[SAMPLE].path_afferents[0].sender.owner: targets}
                targets, num_targets = _adjust_target_dict(object, targets)
            else:
                raise RunError("Target values for {} must be specified in a dictionary.".format(object.name))
        else:
            raise RunError("Target values for {} must be specified in a dictionary.".format(object.name))

        # if num_targets = -1, all targets were specified as functions
        if num_targets != num_inputs_sets and num_targets != -1:
            raise RunError("Number of target values specified ({}) for each learning sequence in {} must equal the "
                           "number of input values specified ({}) for each origin mechanism in {}."
                           .format(num_targets, object.name, num_inputs_sets, object.name))

    object_type = _get_object_type(object)

    object.targets = targets

    # SET LEARNING (if relevant)
    # FIX: THIS NEEDS TO BE DONE FOR EACH PROCESS IF THIS CALL TO run() IS FOR SYSTEM
    #      IMPLEMENT learning_enabled FOR SYSTEM, WHICH FORCES LEARNING OF PROCESSES WHEN SYSTEM EXECUTES?
    #      OR MAKE LEARNING A PARAM THAT IS PASSED IN execute
    # If learning is specified, buffer current state and set to specified state
    if learning is not None:
        try:
            learning_state_buffer = object._learning_enabled
        except AttributeError:
            if object.verbosePref:
                warnings.warn("WARNING: learning not enabled for {}".format(object.name))
        else:
            if learning is True:
                object._learning_enabled = True

            elif learning is False:
                object._learning_enabled = False

    # SET LEARNING_RATE, if specified, for all learningProjections in process or system
    if object.learning_rate is not None:
        from psyneulink.components.projections.modulatory.learningprojection import LearningProjection
        for learning_mech in object.learning_mechanisms.mechanisms:
            for projection in learning_mech.output_state.efferents:
                if isinstance(projection, LearningProjection):
                    projection.function_object.learning_rate = object.learning_rate

    # Class-specific validation:
    if not object.context.flags:
        object.context.initialization_status = ContextFlags.VALIDATING
        object.context.string = RUN + "validating " + object.name

    # INITIALIZATION
    if initialize:
        object.initialize()

    # SET UP TIMING
    if object_type == MECHANISM:
        time_steps = 1
    else:
        time_steps = object.numPhases

    # EXECUTE
    execution_inputs = {}
    execution_targets = {}
    for execution in range(num_trials):

        execution_id = _get_unique_id()

        if call_before_trial:
            call_before_trial()

        for time_step in range(time_steps):

            if call_before_time_step:
                call_before_time_step()

            input_num = execution%num_inputs_sets

            for mech in inputs:
                execution_inputs[mech] = inputs[mech][input_num]
            if object_type == SYSTEM:
                object.inputs = execution_inputs

            # Assign targets:
            if targets is not None:

                if isinstance(targets, function_type):
                    object.target = targets
                else:
                    for mech in targets:
                        if callable(targets[mech]):
                            execution_targets[mech] = targets[mech]
                        else:
                            execution_targets[mech] = targets[mech][input_num]
                    if object_type is SYSTEM:
                        object.target = execution_targets
                        object.current_targets = execution_targets

            if context == ContextFlags.COMMAND_LINE and not object.context.execution_phase == ContextFlags.SIMULATION:
                object.context.execution_phase = ContextFlags.PROCESSING
                object.context.string = RUN + ": EXECUTING " + object_type.upper() + " " + object.name

            result = object.execute(
                input=execution_inputs,
                execution_id=execution_id,
                termination_processing=termination_processing,
                termination_learning=termination_learning,
                context=context
            )

            if call_after_time_step:
                call_after_time_step()

        # object.results.append(result)
        if isinstance(result, Iterable):
            result_copy = result.copy()
        else:
            result_copy = result
        object.results.append(result_copy)

        if call_after_trial:
            call_after_trial()

        from psyneulink.globals.log import _log_trials_and_runs, ContextFlags
        _log_trials_and_runs(composition=object,
                             curr_condition=LogCondition.TRIAL,
                             context=context)

    try:
        object.scheduler_processing.date_last_run_end = datetime.datetime.now()
        object.scheduler_learning.date_last_run_end = datetime.datetime.now()

        for sched in [object.scheduler_processing, object.scheduler_learning]:
            sched.clock._increment_time(TimeScale.RUN)
    except AttributeError:
        # this will fail on processes, which do not have schedulers
        pass

    # Restore learning state
    try:
        learning_state_buffer
    except UnboundLocalError:
        pass
    else:
        object._learning_enabled = learning_state_buffer

    from psyneulink.globals.log import _log_trials_and_runs
    _log_trials_and_runs(composition=object,
                         curr_condition=LogCondition.RUN,
                         context=context)

    return object.results

@tc.typecheck

def _input_matches_variable(input, var):
    # input states are uniform
    if np.shape(np.atleast_2d(input)) == np.shape(var):
        return "homogeneous"
    # input states have different lengths
    elif len(np.shape(var)) == 1 and isinstance(var[0], (list, np.ndarray)):
        for i in range(len(input)):
            if len(input[i]) != len(var[i]):
                return False
        return "heterogeneous"
    return False

def _target_matches_input_state_variable(target, input_state_variable):
    if np.shape(np.atleast_1d(target)) == np.shape(input_state_variable):
        return True
    return False

def _adjust_stimulus_dict(obj, stimuli):

    #  STEP 0:  parse any labels into array entries
    if any(mech.input_labels_dict for mech in obj.origin_mechanisms):
        _parse_input_labels(obj, stimuli)

    # STEP 1: validate that there is a one-to-one mapping of input entries to origin mechanisms

    # Check that all of the mechanisms listed in the inputs dict are ORIGIN mechanisms in the object
    for mech in stimuli.keys():
        if not mech in obj.origin_mechanisms.mechanisms:
            raise RunError("{} in inputs dict for {} is not one of its ORIGIN mechanisms".
                           format(mech.name, obj.name))
    # Check that all of the ORIGIN mechanisms in the obj are represented by entries in the inputs dict
    for mech in obj.origin_mechanisms:
        if not mech in stimuli:
            raise RunError("Entry for ORIGIN Mechanism {} is missing from the inputs dict for {}".
                           format(mech.name, obj.name))

    # STEP 2: Loop over all dictionary entries to validate their content and adjust any convenience notations:

    # (1) Replace any user provided convenience notations with values that match the following specs:
    # a - all dictionary values are lists containing and input value on each trial (even if only one trial)
    # b - each input value is a 2d array that matches variable
    # example: { Mech1: [Fully_specified_input_for_mech1_on_trial_1, Fully_specified_input_for_mech1_on_trial_2 … ],
    #            Mech2: [Fully_specified_input_for_mech2_on_trial_1, Fully_specified_input_for_mech2_on_trial_2 … ]}
    # (2) Verify that all mechanism values provide the same number of inputs (check length of each dictionary value)

    adjusted_stimuli = {}
    num_input_sets = -1

    for mech, stim_list in stimuli.items():

        check_spec_type = _input_matches_variable(stim_list, mech.instance_defaults.variable)
        # If a mechanism provided a single input, wrap it in one more list in order to represent trials
        if check_spec_type == "homogeneous" or check_spec_type == "heterogeneous":
            if check_spec_type == "homogeneous":
                # np.atleast_2d will catch any single-input states specified without an outer list
                # e.g. [2.0, 2.0] --> [[2.0, 2.0]]
                adjusted_stimuli[mech] = [np.atleast_2d(stim_list)]
            else:
                adjusted_stimuli[mech] = [stim_list]

            # verify that all mechanisms have provided the same number of inputs
            if num_input_sets == -1:
                num_input_sets = 1
            elif num_input_sets != 1:
                raise RunError("Input specification for {} is not valid. The number of inputs (1) provided for {}"
                               "conflicts with at least one other mechanism's input specification.".format(obj.name,
                                                                                mech.name))
        else:
            adjusted_stimuli[mech] = []
            for stim in stimuli[mech]:
                check_spec_type = _input_matches_variable(stim, mech.instance_defaults.variable)
                # loop over each input to verify that it matches variable
                if check_spec_type == False:
                    err_msg = "Input stimulus ({}) for {} is incompatible with its variable ({}).".\
                        format(stim, mech.name, mech.instance_defaults.variable)
                    # 8/3/17 CW: I admit the error message implementation here is very hacky; but it's at least not a hack
                    # for "functionality" but rather a hack for user clarity
                    if "KWTA" in str(type(mech)):
                        err_msg = err_msg + " For KWTA mechanisms, remember to append an array of zeros (or other values)" \
                                            " to represent the outside stimulus for the inhibition input state, and " \
                                            "for systems, put your inputs"
                    raise RunError(err_msg)
                elif check_spec_type == "homogeneous":
                    # np.atleast_2d will catch any single-input states specified without an outer list
                    # e.g. [2.0, 2.0] --> [[2.0, 2.0]]
                    adjusted_stimuli[mech].append(np.atleast_2d(stim))
                else:
                    adjusted_stimuli[mech].append(stim)

            # verify that all mechanisms have provided the same number of inputs
            if num_input_sets == -1:
                num_input_sets = len(stimuli[mech])
            elif num_input_sets != len(stimuli[mech]):
                raise RunError("Input specification for {} is not valid. The number of inputs ({}) provided for {}"
                               "conflicts with at least one other mechanism's input specification."
                               .format(obj.name, (stimuli[mech]), mech.name))

    return adjusted_stimuli, num_input_sets

def _adjust_target_dict(component, target_dict):

    #  STEP 0:  parse any labels into array entries
    if any(mech.input_labels_dict for mech in component.target_mechanisms):
        _parse_input_labels(component, target_dict)

    # STEP 1: validate that there is a one-to-one mapping of target entries and target mechanisms
    for target_mechanism in component.target_mechanisms:
        # If any projection to a target does not have a sender in the stimulus dict, raise an exception
        if not any(mech is projection.sender.owner for
                   projection in target_mechanism.input_states[SAMPLE].path_afferents
                   for mech in target_dict.keys()):
                raise RunError("Entry for {} is missing from specification of targets for run of {}".
                               format(target_mechanism.input_states[SAMPLE].path_afferents[0].sender.owner.name,
                                      component.name))

    for mech in target_dict:
        # If any mechanism in the target dict does not have a projection to a target, raise an error
        if not any(target is projection.receiver.owner for
                   projection in mech.output_state.efferents
                   for target in component.target_mechanisms):
            raise RunError("{} does not project to a target Mechanism in {}".format(mech.name, component.name))

        # STEP 2: Loop over all dictionary entries to validate their content and adjust any convenience notations:

        # (1) Replace any user provided convenience notations with values that match the following specs:
        # a - all dictionary values are lists containing a target value on each trial (even if only one trial)
        # b - each input value is at least a 1d array that matches the variable of the TARGET input state

        # (2) Verify that all mechanism values provide the same number of inputs (check length of each dictionary value)

    adjusted_targets = {}
    num_targets = -1
    for mech, target_list in target_dict.items():
        if isinstance(target_list, (float, list, np.ndarray)):
            input_state_variable = mech.output_state.efferents[0].receiver.owner.input_states[TARGET].instance_defaults.variable
            num_targets = -1

            # first check if only one target was provided:
            if np.shape(np.atleast_1d(target_list)) == np.shape(input_state_variable):
                adjusted_targets[mech] = [np.atleast_1d(target_list)]
                if num_targets == -1:
                    num_targets = 1
                elif num_targets != 1:
                    raise RunError("Target specification for {} is not valid. The number of targets (1) provided for {}"
                                   "conflicts with at least one other mechanism's target specification."
                                   .format(component.name, mech.name))

            # iterate over list and check that each candidate target is compatible with corresponding TARGET input state
            elif isinstance(target_list, (list, np.ndarray)):
                adjusted_targets[mech] = []
                for target_value in target_list:
                    if np.shape(np.atleast_1d(target_value)) == np.shape(input_state_variable):
                        adjusted_targets[mech].append(np.atleast_1d(target_value))
                    else:
                        raise RunError("Target specification ({}) for {} is not valid. The shape of {} is not compatible "
                                       "with the TARGET input state of the corresponding ComparatorMechanism ({})"
                                       .format(target_list, mech.name, target_value,
                                               mech.output_state.efferents[0].receiver.owner.name))
                current_num_targets = len(adjusted_targets[mech])
                # verify that all mechanisms have provided the same number of inputs
                if num_targets == -1:
                    num_targets = current_num_targets
                elif num_targets != current_num_targets:
                    raise RunError("Target specification for {} is not valid. The number of targets ({}) provided for {}"
                                   "conflicts with at least one other mechanism's target specification."
                                   .format(component.name, current_num_targets, mech.name))

        elif callable(target_list):
            _validate_target_function(target_list, mech.output_state.efferents[0].receiver.owner, mech)
            adjusted_targets[mech] = target_list
    return adjusted_targets, num_targets

@tc.typecheck
def _parse_input_labels(obj, stimuli:dict):
    from psyneulink.components.states.inputstate import InputState

    # def get_input_for_label(mech, key, input_array=None):
    def get_input_for_label(mech, key, subdicts, input_array=None):
        """check mech.input_labels_dict for key
        If input_array is passed, need to check for subdicts (should be one for each InputState of mech)"""

        # FIX: FOR SOME REASON dict IN TEST BELOW IS TREATED AS AN UNBOUND LOCAL VARIABLE
        # subdicts = isinstance(list(mech.input_labels_dict.keys())[0], dict)

        if input_array is None:
            if subdicts:
                raise RunError("Attempt to reference a label for a stimulus at top level of {} for {},"
                               "which contains subdictionaries for each of its {}s".
                               format(INPUT_LABELS_DICT, mech.name, InputState))
            try:
                return mech.input_labels_dict[key]
            except KeyError:
                raise RunError("No entry \'{}\' found for input to {} in {} for mech.name".
                               format(key, obj.name, INPUT_LABELS_DICT, mech.name))
        else:
            if not subdicts:
                try:
                    return mech.input_labels_dict[key]
                except KeyError:
                    raise RunError("No entry \'{}\' found for input to {} in {} for mech.name".
                                   format(key, obj.name, INPUT_LABELS_DICT, mech.name))
            else:
                # if subdicts, look exhaustively for any instances of the label in keys of all subdicts
                name_value_pairs = []
                for name, dict in mech.input_labels.items():
                    if key in dict:
                        name_value_pairs.append((name,dict[key]))
                if len(name_value_pairs)==1:
                    # if only one found, use its value
                    return name_value_pairs[0][1]
                else:
                    # if more than one is found, now know that "convenience notation" has not been used
                    #     check that number of items in input_array == number of states
                    if len(input_array) != len(mech.input_states):
                        raise RunError("Number of items in input for {} of {} ({}) "
                                       "does not match the number of its {}s ({})".
                                       format(mech.name, obj.name, len(input_array),
                                              InputState, len(mech.input_states)))
                    # use index of item in outer array and key (int or name of state) to determine which subdict to use
                    input_index = input_array.index(key)

                    # try to match input_index against index in name_value_pairs[0];
                    value = [item[1] for item in name_value_pairs if item[0]==input_index]
                    if value:
                        return value[0]
                    else:
                        # otherwise, match against index associated with name of state in name_value_pairs
                        value = [item[1] for item in name_value_pairs if mech.input_states.index(item[0])==input_index]
                        if value:
                            return value[0]
                        else:
                            raise RunError("Unable to find value for label ({}) in {} for {} of {}".
                                           format(key, INPUT_LABELS_DICT, mech.name, obj.name))

    for mech, inputs in stimuli.items():

        subdicts = isinstance(list(mech.input_labels_dict.keys())[0], dict)

        if any(isinstance(input, str) for input in inputs) and not mech.input_labels_dict:
            raise RunError("Labels can not be used to specify the inputs to {} since it does not have an {}".
                           format(mech.name, INPUT_LABELS_DICT))
        for i, stim in enumerate(inputs):
            # "Burrow" down to determine whether there's a number at the "bottom";
            #     if so, leave as is; otherwise, check if its a string and, if so, get value for label
            if isinstance(stim, (list, np.ndarray)): # format of stimuli dict is at least: [[???]...?]
                for j, item in enumerate(stim):
                    if isinstance(item, (Number, list, np.ndarray)): # format of stimuli dict is [[int or []...?]]
                        continue # leave input item as is
                    elif isinstance(item, str): # format of stimuli dict is [[label]...]
                        # inputs[i][j] = get_input_for_label(mech, item, stim)
                        inputs[i][j] = get_input_for_label(mech, item, subdicts, stim)
                    else:
                        raise RunError("Unrecognized specification ({}) in stimulus {} of entry "
                                       "for {} in inputs dictionary specified for {}".
                                       format(item, i, mech.name, obj.name))
            elif isinstance(stim, str):
                # Don't pass input_array as no need to check for subdicts
                # inputs[i] = get_input_for_label(mech, stim)
                inputs[i] = get_input_for_label(mech, stim, subdicts)
            else:
                raise RunError("Unrecognized specification ({}) for stimulus {} in entry "
                               "for {} of inputs dictionary specified for {}".
                               format(stim, i, mech.name, obj.name))

def _validate_target_function(target_function, target_mechanism, sample_mechanism):

    generated_targets = np.atleast_1d(target_function())
    expected_shape = target_mechanism.input_states[TARGET].instance_defaults.variable
    if np.shape(generated_targets) != np.shape(expected_shape):
            raise RunError("Target values generated by target function ({}) are not compatible with TARGET input state "
                           "of {} ({}). See {} entry in target specification dictionary. "
                           .format(generated_targets, target_mechanism.name, expected_shape, sample_mechanism.name))

def _get_object_type(object):
    if isinstance(object, Mechanism):
        return MECHANISM
    elif isinstance(object, Process_Base):
        return PROCESS
    elif isinstance(object, System_Base):
        return SYSTEM
    else:
        raise RunError("{} type not supported by Run module".format(object.__class__.__name__))

import uuid
def _get_unique_id():
    return uuid.uuid4()


# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations

class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        migrations.CreateModel(
            name='Investigation',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, verbose_name='ID', serialize=False)),
            ],
        ),
        migrations.CreateModel(
            name='Score',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, verbose_name='ID', serialize=False)),
                ('investigation', models.ForeignKey(default=None, to='scales.Investigation')),
            ],
        ),
    ]


from bs4 import BeautifulSoup
import requests
import shutil

urls = []
query = 'puppies'

class data_miner:

    def __init__(self, base_url):
        self.base_url = base_url

    def get_links(self, query):
        html=self.base_url
        payload={'q': query}
        r=requests.get(html, params=payload)
        soup=BeautifulSoup(r.content, "html5lib")
        for image in soup.find_all('img'):
            urls.append(image.attrs['src'])

    def download_files(self, query):
        number = 1
        for item in urls:
            with open('static/happy-images/' + query + str(number) + '.jpg', 'wb') as out_file:
                req = requests.get(item, stream=True)
                shutil.copyfileobj(req.raw, out_file)
                number += 1

x = data_miner('http://www.google.com/images')
x.get_links(query)
x.download_files(query)


"""
CDICorporation spider created on the top of ATSSpider

scrapy crawl cdicorporation -a mining_job_id=9999 -a iteration=1 -a extract=1 -a url="https://cdicorporation.bullhorncloud.com/portal/jobBoardPublic/jobListPublic"

Sample URL:
    https://cdicorporation.bullhorncloud.com/portal/jobBoardPublic/jobListPublic
"""

from json import loads
from math import ceil
from scrapy.http import FormRequest, Request
from scrapy.selector import Selector
from urlparse import urljoin

from brightcorp.base.atsspiders import ATSSpider
from brightcorp.items import BrightcorpItemLoader
from brightcorp.processors import HtmlFormatter, NormalizedJoin, Prefix, Replace

class CDICorporation(ATSSpider):

    name = 'cdicorporation'
    page = 1
    total_pages = 0
    form_data = {
        'bRegex': 'false',
        'bRegex_0': 'false',
        'bRegex_1': 'false',
        'bRegex_2': 'false',
        'bRegex_3': 'false',
        'bRegex_4': 'false',
        'bRegex_5': 'false',
        'bSearchable_0': 'true',
        'bSearchable_1': 'true',
        'bSearchable_2': 'true',
        'bSearchable_3': 'true',
        'bSearchable_4': 'true',
        'bSearchable_5': 'true',
        'bSortable_0': 'true',
        'bSortable_1': 'true',
        'bSortable_2': 'true',
        'bSortable_3': 'true',
        'bSortable_4': 'true',
        'bSortable_5': 'false',
        'iColumns': '6',
        'iDisplayLength': '10',
        'iDisplayStart': '0',
        'iSortCol_0': '0',
        'iSortingCols': '1',
        'mDataProp_0': '0',
        'mDataProp_1': '1',
        'mDataProp_2': '2',
        'mDataProp_3': '3',
        'mDataProp_4': '4',
        'mDataProp_5': '5',
        'sColumns': '',
        'sEcho': '1',
        'sSearch': '',
        'sSearch_0': '',
        'sSearch_1': '',
        'sSearch_2': '',
        'sSearch_3': '',
        'sSearch_4': '',
        'sSearch_5': '',
        'sSortDir_0': 'asc',
    }

    def parse(self, response):
        yield FormRequest(
            callback=self.parse_jobs_list,
            formdata=self.form_data,
            url=urljoin(
                response.url,
                '/portal/jobBoardPublic/data?queryEntity='
            )
        )

    def parse_jobs_list(self, response):
        jsonResponse = loads(response.body.decode('unicode_escape'))
        if jsonResponse:
            # set expected job count
            if not self.expected_job_count_set:
                total_records = jsonResponse.get('iTotalRecords')
                self.total_pages = int(ceil(int(total_records) / 10.0))
                self.expected_job_count = total_records

            results = jsonResponse.get('aaData')
            if results:
                for item in results:
                    job_id = item[0]
                    if job_id:
                        href = '/portal/jobBoardPublic/jobDetailsPublic?jobOrderID=%s' % job_id
                        yield Request(
                            callback=self.parse_job_callback(),
                            meta={
                                'jobid': job_id,
                                'title': item[1],
                                'city': item[2],
                                'state': item[3],
                                'country': item[4],
                            },
                            url=urljoin(response.url, href)
                        )
            if self.page <= self.total_pages:
                self.page += 1
                count = (self.page - 1) * 10
                self.form_data.update({
                    'sEcho': str(self.page),
                    'iDisplayStart': str(count),
                })
                yield FormRequest(
                    callback=self.parse_jobs_list,
                    formdata=self.form_data,
                    url=urljoin(
                        response.url,
                        '/portal/jobBoardPublic/data?queryEntity='
                    )
                )

    def parse_job(self, response):
        """
        Extract all required information.
        """
        sel = Selector(response)

        loader = BrightcorpItemLoader(selector=sel)
        loader.add_value(
            'title', response.meta.get('title')
        )
        loader.add_value(
            'location',
            [
                response.meta.get('city'),
                response.meta.get('state'),
                response.meta.get('country'),
            ],
            NormalizedJoin(', '),
            Replace('Not Listed')
        )
        loader.add_value(
            'referencenumber',
            response.meta.get('jobid'),
            Prefix('%s-' % self.name)
        )
        loader.add_value('url', response.url)
        loader.add_xpath(
            'description',
            '//tr[@id="publicDescription"]',
            HtmlFormatter()
        )
        loader.add_value('apply_url', response.url)

        yield loader.load_item()


import unittest
import logging
import time, datetime
import thespian.test.helpers
from thespian.actors import *
from thespian.test import TestSystem

class EchoActor(Actor):
    def receiveMessage(self, msg, sender):
        logging.info('EchoActor got %s (%s) from %s', msg, type(msg), sender)
        self.send(sender, msg)

class Kill_The_Messenger(Actor):
    def receiveMessage(self, message, sender):
        self.send(sender, ActorExitRequest())

class FakeSystemMessage(ActorSystemMessage):
    pass

smallwait = datetime.timedelta(milliseconds=50)

class TestASimpleSystem(unittest.TestCase):
    testbase='Simple'
    scope='func'

    sysbase = 'simpleSystemBase'
    portbase = 17100

    def testCreateActorSystem(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+1}) as asys:
            pass

    def testSimpleActor(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+2}) as asys:
            echo = asys.createActor(EchoActor)

    def testSimpleMessageTell(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+3}) as asys:
            echo = asys.createActor(EchoActor)
            asys.tell(echo, 'hello')
            time.sleep(0.02)  # allow tell to work before ActorSystem shutdown

    def testSystemMessageTell(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+4}) as asys:
            echo = asys.createActor(EchoActor)
            asys.tell(echo, FakeSystemMessage())
            time.sleep(0.02)  # allow tell to work before ActorSystem shutdown

    def testKillMessageTell(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+5}) as asys:
            echo = asys.createActor(EchoActor)
            asys.tell(echo, ActorExitRequest())
            time.sleep(0.02)  # allow tell to work before ActorSystem shutdown

    def testKillMessageTellKiller(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+6}) as asys:
            ktm = asys.createActor(Kill_The_Messenger)
            asys.tell(ktm, 'hello')
            asys.tell(ktm, ActorExitRequest())
            time.sleep(0.02)  # allow tell to work before ActorSystem shutdown

    def testSimpleMessageAsk(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+7}) as asys:
            echo = asys.createActor(EchoActor)
            self.assertEqual(asys.ask(echo, 'hello', smallwait), 'hello')

    def testSystemMessageAsk(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+8}) as asys:
            echo = asys.createActor(EchoActor)
            # SystemMessages are explicitly filtered from being returned
            # via Ask() or Tell(), with the exception of PoisonMessage.
            self.assertIsNone(asys.ask(echo, FakeSystemMessage(), smallwait))

    def testKillMessageAsk(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+9}) as asys:
            echo = asys.createActor(EchoActor)
            # SystemMessages are explicitly filtered from being returned
            # via Ask() or Tell(), with the exception of PoisonMessage.
            self.assertIsNone(asys.ask(echo, ActorExitRequest(), smallwait))

    def testKillMessageAskKiller(self):
        with TestSystem(self.sysbase, {'Admin Port': self.portbase+10}) as asys:
            ktm = asys.createActor(Kill_The_Messenger)
            self.assertIsNone(asys.ask(ktm, 'hello', smallwait))
            self.assertIsNone(asys.ask(ktm, ActorExitRequest(), smallwait))

class TestMultiprocUDPSystem(TestASimpleSystem):
    testbase='MultiprocUDP'
    sysbase = 'multiprocUDPBase'
    portbase = 17120

class TestMultiprocTCPSystem(TestASimpleSystem):
    testbase='MultiprocTCP'
    sysbase = 'multiprocTCPBase'
    portbase = 17140

class TestMultiprocQueueSystem(TestASimpleSystem):
    testbase='MultiprocQueue'
    sysbase = 'multiprocQueueBase'
    portbase = 17160


"""Input/output operations for Sentera imagery."""

import logging
import os
import re
import shutil
from glob import glob

import numpy as np
import pandas as pd
import tifffile as tf

from imgcorrect import detect_panel
from imgcorrect.sensor_defs import sensor_defs

logger = logging.getLogger(__name__)

def apply_sensor_settings(image_df):
    """Rebuild image dataframe with settings based on sensor model."""
    rows = []

    for index, row in image_df.iterrows():
        for s in sensor_defs:
            # verify image metadata matches that of a supported sensor
            meets_criteria = True
            for key, val in s["criteria"].items():
                if key not in row["EXIF"] or val not in str(row["EXIF"][key]):
                    meets_criteria = False
            if meets_criteria:
                # ignore images that meet ignore_criteria
                if "ignore_criteria" in s:
                    ignore = False
                    for key, val in s["ignore_criteria"].items():
                        if key in row["EXIF"] and val in str(row["EXIF"][key]):
                            logger.info("Ignoring %s", row["image_path"])
                            ignore = True
                    if ignore:
                        break

                # apply settings for that sensor
                for key, val in s["settings"].items():
                    row[key] = val

                # if each image contains data for multiple bands, configure accordingly
                if "bands" in s:
                    for band in s["bands"]:
                        band_row = row.copy()
                        band_row["band"] = band[0]
                        band_row["band_math"] = band[1]
                        band_row["XMP_index"] = band[2]
                        band_row["reduce_xmp"] = True
                        band_row["output_path"] = add_band_to_path(
                            row.output_path, band[0]
                        ).replace(".jpg", ".tif")
                        rows.append(band_row)
                # otherwise, assume band is indicated in root folder name
                else:
                    row["band"] = re.search(
                        r"[A-Za-z]+", os.path.basename(row.image_root)
                    ).group(0)
                    row["XMP_index"] = 0
                    row["reduce_xmp"] = False
                    rows.append(row)

                break
        else:
            logger.error("Sensor not supported")
            raise Exception("Sensor not supported")

    return pd.DataFrame(rows)

def create_image_df(input_path, output_path):
    """Build image dataframe."""
    if not output_path:
        output_path = input_path

    image_df = pd.DataFrame()

    image_df["image_path"] = glob(input_path + "/**/*.tif", recursive=True) + glob(
        input_path + "/**/*.jpg", recursive=True
    )
    image_df["image_root"] = image_df.image_path.apply(os.path.dirname)
    image_df["output_path"] = image_df.image_path.str.replace(
        input_path, output_path, regex=False
    )

    return image_df

def reflectance_if_panel(row):
    """If reflectance panel images are not identifiable by filename, try computing panel reflectance for all images."""
    if not row["cal_in_path"]:
        row["mean_reflectance"], row["aruco_id"] = detect_panel.get_reflectance(row)
    return row

def detect_cal(row, calibration_id):
    """If reflectance panel images are identifiable by filename, identify them. Otherwise, refer to results of reflectance_if_panel()."""
    if row["cal_in_path"]:
        return calibration_id in row["image_path"]
    return not np.isnan(row["mean_reflectance"])

def create_cal_df(image_df, calibration_id):
    """Build calibration image dataframe."""
    image_df = image_df.apply(reflectance_if_panel, axis=1)
    is_cal_image = image_df.apply(lambda row: detect_cal(row, calibration_id), axis=1)
    return image_df.loc[is_cal_image], image_df.loc[~is_cal_image]

def delete_all_originals(image_df):
    """Delete all input images."""
    image_df.image_path.apply(os.remove)

def add_band_to_path(path, band):
    """Add band directory to path just before filename."""
    dirname, base = os.path.split(path)
    return os.path.join(dirname, band, base)

def move_images(image_df_row):
    """Move corrected image to final destination."""
    shutil.move(image_df_row.temp_path, image_df_row.output_path)

def move_corrected_images(image_df):
    """Create output directories if necessary, then move corrected images to final destination."""
    for folder in image_df.output_path.apply(os.path.dirname).unique():
        os.makedirs(folder, exist_ok=True)
    image_df.apply(lambda row: move_images(row), axis=1)

def write_image(image_arr_corrected, image_df_row, temp_dir):
    """Write corrected image to temporary location and record maximum value in case normalization is required."""
    path_list = os.path.normpath(image_df_row.image_path).split(os.path.sep)
    path_list[0] = temp_dir
    temp_path = os.path.join(*path_list)
    if "band_math" in image_df_row.index:
        temp_path = add_band_to_path(temp_path, image_df_row.band).replace(
            ".jpg", ".tif"
        )
    os.makedirs(os.path.dirname(temp_path), exist_ok=True)
    # noinspection PyTypeChecker
    tf.imwrite(temp_path, image_arr_corrected)

    image_df_row["max_val"] = np.max(image_arr_corrected)
    image_df_row["temp_path"] = temp_path
    return image_df_row

def write_corrections_csv(image_df, file):
    """Write vital correction data from the dataframe to the given csv file."""
    columns = [
        "image_path",
        "independent_ils",
        "band",
        "autoexposure",
        "ILS_ratio",
        "slope_coefficient",
        "correction_coefficient",
    ]
    csv_df = image_df[columns].copy()
    base_dir = os.path.dirname(file)

    # Get the path relative to the output folder
    csv_df["image_path"] = csv_df["image_path"].apply(
        (lambda x: os.path.relpath(x, base_dir))
    )
    csv_df.to_csv(file, index=False)


#!/usr/bin/env python

from os import path

from setuptools import find_packages
from distutils.core import setup

here = path.abspath(path.dirname(__file__))

with open(path.join(here, 'README.md'), encoding='utf-8') as f:
    long_description = f.read()

requires = [
    'gspread'
]

classifiers = [
    'Development Status :: 4 - Beta',
    'License :: OSI Approved :: MIT License',
    'Operating System :: OS Independent',
    'Intended Audience :: Developers',
    'Intended Audience :: Science/Research',
    'Programming Language :: Python',
    'Programming Language :: Python :: Implementation :: PyPy',
]

setup(
    name='labgsheet',
    version='0.1.1',
    packages=find_packages(exclude=['contrib', 'docs', 'tests']),
    license='MIT',
    description='A python library to note ml experiments on google sheet',
    long_description=long_description,
    long_description_content_type='text/markdown',
    url='https://github.com/shotarok/labgsheet',
    author='Shotaro Kohama',
    author_email='khmshtr28@gmail.com',
    install_requires=requires,
    classifiers=classifiers
)


import os

from MainWindow import MainWindow
from NewProjectWizard import NewProjectWizard
from Windows import StarterWindow

from PyQt5.QtCore import Qt, QDateTime, QStringListModel, QDate
from PyQt5.QtWidgets import qApp, QMessageBox, QFileDialog
from PyQt5 import QtGui
import pandas as pd
import json

DIR_NAME = os.path.abspath(os.path.dirname(__file__))
SETTINGS_DIR = os.path.join(DIR_NAME, 'Settings')
SETTINGS_PATH = os.path.join(SETTINGS_DIR, 'settings.json')
PROJECTS_DIR = os.path.join(DIR_NAME, 'Projects')
RESOURCES_DIR = os.path.join(DIR_NAME, 'Resources')

class ProjectController():
    def __init__(self):
        icon_path = os.path.join(RESOURCES_DIR, 'icons', 'app_icon.png')
        qApp.setWindowIcon(QtGui.QIcon(icon_path))

        if not os.path.exists(PROJECTS_DIR):
            os.mkdir(PROJECTS_DIR)
        if not os.path.exists(SETTINGS_DIR):
            self.createSettingsFile()

        self.loadSettings()
        self.sw = StarterWindow(self)
        self.mw = None

    def loadSettings(self):
        '''
        Load default user settings
        '''
        with open(SETTINGS_PATH, 'rt') as f:
            self.settings = json.loads(f.read())

        self.loadTheme(self.settings.get('Theme'))

    def saveSettings(self):
        '''
        Save updated settings data
        '''
        try:
            with open(SETTINGS_PATH, 'w+') as f:
                f.write(json.dumps(self.settings))
        except:
            return False
        else:
            return True

    def loadTheme(self, theme=None):
        '''
        Load given theme and save to settings file
        '''
        if theme:
            theme_path = os.path.join(RESOURCES_DIR, f'stylesheet_{theme}.css')
            with open(theme_path, 'rt') as f:
                qApp.setStyleSheet(f.read())
        else:
            qApp.setStyleSheet(None)

        self.settings['Theme'] = theme
        self.saveSettings()

    def setAPI(self, api_key):
        '''
        Save api key to settings file
        '''
        self.settings['API'] = api_key
        return self.saveSettings()

    def createSettingsFile(self):
        '''
        Create settings directory and settings.json file if it doesn't exist
        this function should only be called on first launch of application
        '''
        default_data = {
            'Theme': None,
            'API': None
        }
        os.mkdir(SETTINGS_DIR)
        with open(SETTINGS_PATH, 'w+') as f:
            f.write(json.dumps(default_data))

    def newProject(self, window_ref):
        '''
        window_ref: (QDialog) reference to the window that function is called from
        '''
        self.npw = NewProjectWizard()

        if self.npw.exec_():
            project_name = self.npw.dataPage.getProjectName()
            ref = self.npw.dataPage.getReferencePoint()
            if project_name and ref:
                self.createProject(window_ref, project_name, ref)

    def createProject(self, window_ref, project_name, ref):
        '''
        window_ref: (QDialog) reference to the window that function is called from
        project_name: (str) name of new project
        ref: (tuple) reference point
        '''
        project_file = os.path.join(PROJECTS_DIR, project_name, 'project_data.json')
        created_date = QDateTime().currentDateTime().toString('MM-dd-yyyy hh:mm:ss ap')
        default_data = {
                'ProjectName': project_name,
                'Created': created_date,
                'LastAccessed': created_date,
                'Reference': ref,
                'Scale': 0,
                'Units': '',
                'Points': []
        }
        try:
            os.makedirs(os.path.join(PROJECTS_DIR, project_name, 'Reports'))
            with open(project_file, 'w+') as f:
                f.write(json.dumps(default_data))

        except FileExistsError:
            QMessageBox.critical(
                window_ref,
                'Project Creation Error',
                f'Invalid project name: {project_name}'
            )
        else:
            window_ref.close()
            self.mw = MainWindow(project_name, self, openExisting=True, api=self.settings.get('API'))

    def closeProject(self):
        '''
        Close the currently open projects and redisplay starter window
        '''
        self.mw.close()
        self.sw = StarterWindow(self)

    def browseProjectsDir(self, window_ref):
        fileDialog = QFileDialog(window_ref, 'Projects', PROJECTS_DIR)
        fileDialog.setFileMode(QFileDialog.DirectoryOnly)
        fileDialog.setAttribute(Qt.WA_QuitOnClose, False)

        #If a valid path is returned from file dialog screen
        if fileDialog.exec_():
            selected_path = fileDialog.selectedFiles()[0]
            #Check if json data file is in selected folder
            projectName = selected_path.split('/')[-1]
            self.openProject(projectName, window_ref)

    def openProject(self, project_name, window_ref):
        '''
        window_ref: (QDialog) a reference to the window that called this function
        '''
        path = os.path.join(PROJECTS_DIR, project_name, 'project_data.json')

        if os.path.exists(path):
            window_ref.close()
            self.mw = MainWindow(project_name, self, openExisting=True, api=self.settings.get('API'))
            #alert for invalid project and return to main window or starter screen
        else:
            QMessageBox.critical(
                window_ref,
                'Export File',
                f'{project_name} file failed to be created'
            )

    def saveProject(self, project_name, project_data):
        '''
        Saves the project data in json format and writes to a file
        '''
        project_path = os.path.join(PROJECTS_DIR, project_name, 'project_data.json')

        try:
            with open(project_path, 'w+') as f:
                f.write(json.dumps(project_data, indent=2))
        except:
            return False
        else:
            return True

    def setProjectName(self, old_name, new_name):
        '''
        Rename project
        '''
        old_path = os.path.join(PROJECTS_DIR, old_name)
        new_path = os.path.join(PROJECTS_DIR, new_name)

        try:
            os.rename(old_path, new_path)
        except:
            return False
        else:
            return True

    def exportProjectData(self, project_name, data, file_type):
        '''
        '''
        df = pd.DataFrame(data)
        path = os.path.join(PROJECTS_DIR, project_name, 'Reports', QDate.currentDate().toString("MM-dd-yy") + f'_Report.{file_type}')

        try:
            if file_type == 'csv':
                df.to_csv(path, index=False)
            elif file_type == 'json':
                df.to_json(path)
            elif file_type == 'xlsx':
                df.to_excel(path, index=False)
            elif file_type == 'html':
                df.to_html(path, index=False)
            else:
                raise ValueError
        except:
            return False
        else:
            return True

    def getProjectData(self, project_name):

        path = os.path.join(PROJECTS_DIR, project_name, 'project_data.json')

        try:
            with open(path, 'r') as f:
                data = json.loads(f.read())
        except:
            return False
        else:
            return data

    def getAllProjectData(self):

        project_names = self.getProjects()
        data = []

        for name in project_names:
            data.append(self.getProjectData(name))

        return data

    def getProjects(self):
        '''
        Return list of project names within Projects directory
        '''
        return [f.name for f in os.scandir(PROJECTS_DIR) if f.is_dir()]


from django.db import models
from django.contrib.auth import get_user_model

MEMBERSHIP_PLANS = (
        ('Free', 'Free'),
        ('Basic Plan', 'Basic Plan'),
        ('Standard Plan', 'Standard Plan'),
        ('Plus Plan', 'Plus Plan'),
    )

User = get_user_model()

# Create your models here.
class Account(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE)
    membership = models.CharField(max_length=100, choices=MEMBERSHIP_PLANS, default='Free')
    payments_method = models.CharField(max_length=100, null=False, blank=False)
    number_transactions = models.IntegerField(null=False, blank=False, default=0)


import bpy, bmesh
import random

startFrame = 1
endFrame = 601
step = 5
scene = bpy.context.scene
fp = scene.render.filepath # get existing output path
scene.render.image_settings.file_format = 'PNG' # set output format to .png

obj = bpy.context.object                  # Reference to selected object
all_materials   = obj.data.materials      # All the materials of the selected object
no_of_materials = len(all_materials) # Number of materials on the sel. object
print("number of mat :" + str(no_of_materials))
obj.active_material_index = 0

for frame_nr in range(startFrame,endFrame):

    # set current frame
    scene.frame_set(frame_nr)
    print(frame_nr)
    # set a render step for changing materials
    if frame_nr % step == 0:

        print("changing mat")
        bpy.ops.object.mode_set(mode = 'EDIT')
        bpy.context.object.active_material_index += 1
        bpy.ops.object.material_slot_assign()
        bpy.ops.object.mode_set(mode = 'OBJECT')

    obj.data.update()

       # ==========  set output path so render won't get overwritten
    scene.render.filepath = fp + str(frame_nr)
    bpy.ops.render.render(write_still=True) # render still

    # print("changing material")
    # bpy.ops.object.mode_set(mode = 'EDIT')  # Go to edit mode to create bmesh
    # ob = bpy.context.object                 # Reference to selected object

    # bm = bmesh.from_edit_mesh(ob.data)      # Create bmesh object from object mesh

    # bm.select_mode = {'FACE'}               # Go to face selection mode
    # # bm.faces[0].select_set(True)            # Select   face 0
    # # bm.faces[1].select_set(False)           # Deselect face 1

    #         # Create bmesh object from object mesh

    # for face in bm.faces:        # Iterate over all of the object's faces
    #     face.material_index = random.randint(0, no_of_materials - 1)  # Assing random material to face

    # ob.data.update()                            # Update the mesh from the bmesh data
    # bpy.ops.object.mode_set(mode = 'OBJECT')    # Return to object mode</pre>

    # # end if

    # # ==========  set output path so render won't get overwritten
    # scene.render.filepath = fp + str(frame_nr)
    # bpy.ops.render.render(write_still=True) # render still


from src import modules as m
import numpy as np
from src import ofdm
from scipy.linalg import dft
from simulations.common import graph
from simulations.common import settings
import matplotlib.pyplot as plt

graph.init_graph()
dirname = "../results/ofdm/test"
settings.init_output(dirname)

params = {
    "block": 1000,
    "subcarrier": 10,
    "CP": 5,
    "chanel_len": 2,
    "SNR_MIN": 0,
    "SNR_MAX": 25,
    "SNR_NUM": 6,
    "SNR_AVERAGE": 100,
    "equalizer": "ZF",
}

settings.dump_params(params, dirname)

ofdm_zero = np.hstack((np.zeros((params["subcarrier"], params["CP"])), np.eye(params["subcarrier"])))

F = dft(params['subcarrier'], "sqrtn")
FH = F.conj().T

snrs_db = m.snr_db(params['SNR_MIN'], params['SNR_MAX'], params['SNR_NUM'])
sigmas = m.sigmas(snrs_db)
errors = np.zeros((params['SNR_NUM'], params['SNR_AVERAGE']))
q_errors = np.zeros((params['SNR_NUM'], params['SNR_AVERAGE']))

for trials_index in range(params['SNR_AVERAGE']):
    h_si = m.channel(1, params['chanel_len'])
    H = ofdm.toeplitz_channel(h_si.T, params['chanel_len'], params['subcarrier'], params['CP'])
    z_H = np.hstack((np.zeros((H.shape[0], params["subcarrier"] + params["CP"] - 3)), H))
    Hc = ofdm.circulant_channel(h_si.T, params['chanel_len'], params['subcarrier'])

    D = F @ Hc @ FH
    D_1 = np.linalg.inv(D)

    for sigma_index, sigma in enumerate(sigmas):
        d = np.random.choice([0, 1], (params['subcarrier'] * 2 * params['block'], 1))
        s_n = m.modulate_qpsk(d)
        s = s_n.reshape(params['subcarrier'], params['block'])
        x = np.matmul(FH, s)
        x_cp = ofdm.add_cp(x, params['CP'])

        x_receive = x_cp
        if params["chanel_len"] > 1:
            x_receive = np.zeros((params['chanel_len'] - 1 + x_cp.shape[0], x_cp.shape[1]), dtype=complex)
            x_receive[:(params["chanel_len"] - 1), 1:] = x_cp[-(params["chanel_len"] - 1):, :-1]
            x_receive[(params["chanel_len"] - 1):, :] = x_cp

        noise = m.awgn((params['subcarrier'] + params['CP'], params['block']), sigma)
        r = np.matmul(H, x_receive) + noise

        r_s = r.flatten()

        y_p = r_s.reshape((params['subcarrier'] + params['CP'], params['block']))
        y_remove_cp = np.matmul(ofdm_zero, y_p)
        y = np.matmul(F, y_remove_cp)

        s_hat = np.matmul(D_1, y)
        s_n_hat = s_hat.reshape(params['subcarrier'] * params['block'])
        d_hat = m.demodulate_qpsk(s_n_hat).reshape((params['subcarrier'] * 2 * params['block'], 1))
        error = np.sum(d != d_hat)

        errors[sigma_index][trials_index] = error

        ### QPSK
        # q_r = (h_si * s_n) + m.awgn((params['subcarrier'] * params['block'], 1), sigma)
        # q_r = q_r * h_si.conj() / (np.abs(h_si) ** 2)
        # q_d_hat = m.demodulate_qpsk(q_r.squeeze()).reshape((params['subcarrier'] * 2 * params['block'], 1))
        # q_error = np.sum(d != q_d_hat)
        #
        # q_errors[sigma_index][trials_index] = q_error

ber_fig, ber_ax = graph.new_snr_ber_canvas(params['SNR_MIN'], params['SNR_MAX'], -4, 0)
n_sum = params['subcarrier'] * 2 * params['block'] * params['SNR_AVERAGE']
errors_sum = np.sum(errors, axis=1)
bers = errors_sum / n_sum
ber_ax.plot(snrs_db, bers, color="k", marker='o', linestyle='--', label="OFDM(QPSK)")

errors_sum = np.sum(q_errors, axis=1)
bers = errors_sum / n_sum
ber_ax.plot(snrs_db, bers, color="b", marker='x', linestyle=':', label="QPSK")
ber_ax.legend()

plt.tight_layout()
# plt.savefig(dirname + '/SNR_BER.pdf', bbox_inches='tight')
plt.show()


# Note: These views are public with the exception of the response_view
# which has "secure" parts to it in the template.

import json
from datetime import datetime, timedelta

from django.http import HttpResponse
from django.shortcuts import get_object_or_404, render

from elasticutils.contrib.django import F, es_required_or_50x
from funfactory.urlresolvers import reverse
from mobility.decorators import mobile_template
from tower import ugettext as _

from fjord.analytics.tools import (
    JSONDatetimeEncoder,
    generate_query_parsed,
    counts_to_options,
    zero_fill)
from fjord.base.helpers import locale_name
from fjord.base.util import (
    check_new_user,
    smart_int,
    smart_date,
    Atom1FeedWithRelatedLinks)
from fjord.feedback.models import Response, ResponseMappingType

@check_new_user
@mobile_template('analytics/{mobile/}response.html')
def response_view(request, responseid, template):
    response = get_object_or_404(Response, id=responseid)

    # We don't pass the response directly to the template and instead
    # do some data tweaks here to make it more palatable for viewing.
    return render(request, template, {
        'response': response,
    })

def generate_json_feed(request, search):
    """Generates JSON feed for first 100 results"""
    search_query = request.GET.get('q', None)
    responses = search.values_dict()[:100]
    json_data = {
        'total': len(responses),
        'results': list(responses),
        'query': search_query
    }
    return HttpResponse(
        json.dumps(json_data, cls=JSONDatetimeEncoder),
        mimetype='application/json')

def generate_atom_feed(request, search):
    """Generates ATOM feed for first 100 results"""
    search_query = request.GET.get('q', None)

    if search_query:
        title = _(u'Firefox Input: {query}').format(query=search_query)
    else:
        title = _(u'Firefox Input')

    # Build the non-atom dashboard url and maintain all the
    # querystring stuff we have
    dashboard_url = request.build_absolute_uri()
    dashboard_url = dashboard_url.replace('format=atom', '')
    dashboard_url = dashboard_url.replace('&&', '&')
    if dashboard_url.endswith(('?', '&')):
        dashboard_url = dashboard_url[:-1]

    feed = Atom1FeedWithRelatedLinks(
        title=title,
        link=dashboard_url,
        description=_('Search Results From Firefox Input'),
        author_name=_('Firefox Input'),
    )
    for response in search[:100]:
        categories = {
            'sentiment': _('Happy') if response.happy else _('Sad'),
            'platform': response.platform,
            'locale': response.locale
        }
        categories = (':'.join(item) for item in categories.items())

        link_url = reverse('response_view', args=(response.id,))
        link_url = request.build_absolute_uri(link_url)

        feed.add_item(
            title=_('Response id: {id}').format(id=response.id),
            description=response.description,
            link=link_url,
            pubdate=response.created,
            categories=categories,
            link_related=response.url_domain,
        )
    return HttpResponse(
        feed.writeString('utf-8'), mimetype='application/atom+xml')

def generate_dashboard_url(request, output_format='atom',
                                viewname='dashboard'):
    """For a given request, generates the dashboard url for the given format"""
    qd = request.GET.copy()

    # Remove anything from the querystring that isn't good for a feed:
    # page, start_date, end_date, etc.
    for mem in qd.keys():
        if mem not in ('happy', 'locale', 'platform', 'product',
                       'version', 'q'):
            del qd[mem]

    qd['format'] = output_format

    return reverse(viewname) + '?' + qd.urlencode()

@check_new_user
@es_required_or_50x(error_template='analytics/es_down.html')
def dashboard(request):
    template = 'analytics/dashboard.html'

    output_format = request.GET.get('format', None)
    page = smart_int(request.GET.get('page', 1), 1)

    # Note: If we add additional querystring fields, we need to add
    # them to generate_dashboard_url.
    search_happy = request.GET.get('happy', None)
    search_platform = request.GET.get('platform', None)
    search_locale = request.GET.get('locale', None)
    search_product = request.GET.get('product', None)
    search_version = request.GET.get('version', None)
    search_query = request.GET.get('q', None)
    search_date_start = smart_date(
        request.GET.get('date_start', None), fallback=None)
    search_date_end = smart_date(
        request.GET.get('date_end', None), fallback=None)
    search_bigram = request.GET.get('bigram', None)
    selected = request.GET.get('selected', None)

    filter_data = []
    current_search = {'page': page}

    search = ResponseMappingType.search()
    f = F()
    # If search happy is '0' or '1', set it to False or True, respectively.
    search_happy = {'0': False, '1': True}.get(search_happy, None)
    if search_happy in [False, True]:
        f &= F(happy=search_happy)
        current_search['happy'] = int(search_happy)

    def unknown_to_empty(text):
        """Convert "Unknown" to "" to support old links"""
        return u'' if text.lower() == u'unknown' else text

    if search_platform is not None:
        f &= F(platform=unknown_to_empty(search_platform))
        current_search['platform'] = search_platform
    if search_locale is not None:
        f &= F(locale=unknown_to_empty(search_locale))
        current_search['locale'] = search_locale
    if search_product is not None:
        f &= F(product=unknown_to_empty(search_product))
        current_search['product'] = search_product

        if search_version is not None:
            # Note: We only filter on version if we're filtering on
            # product.
            f &= F(version=unknown_to_empty(search_version))
            current_search['version'] = search_version

    if search_date_start is None and search_date_end is None:
        selected = '7d'

    if search_date_end is None:
        search_date_end = datetime.now()
    if search_date_start is None:
        search_date_start = search_date_end - timedelta(days=7)

    current_search['date_end'] = search_date_end.strftime('%Y-%m-%d')
    # Add one day, so that the search range includes the entire day.
    end = search_date_end + timedelta(days=1)
    # Note 'less than', not 'less than or equal', because of the added
    # day above.
    f &= F(created__lt=end)

    current_search['date_start'] = search_date_start.strftime('%Y-%m-%d')
    f &= F(created__gte=search_date_start)

    if search_query:
        current_search['q'] = search_query
        es_query = generate_query_parsed('description', search_query)
        search = search.query_raw(es_query)

    if search_bigram is not None:
        f &= F(description_bigrams=search_bigram)
        filter_data.append({
            'display': _('Bigram'),
            'name': 'bigram',
            'options': [{
                'count': 'all',
                'name': search_bigram,
                'display': search_bigram,
                'value': search_bigram,
                'checked': True
            }]
        })

    search = search.filter(f).order_by('-created')

    # If the user asked for a feed, give him/her a feed!
    if output_format == 'atom':
        return generate_atom_feed(request, search)

    elif output_format == 'json':
        return generate_json_feed(request, search)

    # Search results and pagination
    if page < 1:
        page = 1
    page_count = 20
    start = page_count * (page - 1)
    end = start + page_count

    search_count = search.count()
    opinion_page = search[start:end]

    # Navigation facet data
    facets = search.facet(
        'happy', 'platform', 'locale', 'product', 'version',
        filtered=bool(search._process_filters(f.filters)))

    # This loop does two things. First it maps 'T' -> True and 'F' ->
    # False.  This is probably something EU should be doing for
    # us. Second, it restructures the data into a more convenient
    # form.
    counts = {
        'happy': {},
        'platform': {},
        'locale': {},
        'product': {},
        'version': {}
    }
    for param, terms in facets.facet_counts().items():
        for term in terms:
            name = term['term']
            if name == 'T':
                name = True
            elif name == 'F':
                name = False

            counts[param][name] = term['count']

    def empty_to_unknown(text):
        return _('Unknown') if text == u'' else text

    filter_data.extend([
        counts_to_options(
            counts['happy'].items(),
            name='happy',
            display=_('Sentiment'),
            display_map={True: _('Happy'), False: _('Sad')},
            value_map={True: 1, False: 0},
            checked=search_happy),
        counts_to_options(
            counts['product'].items(),
            name='product',
            display=_('Product'),
            display_map=empty_to_unknown,
            checked=search_product)
    ])
    # Only show the version if we're showing a specific
    # product.
    if search_product:
        filter_data.append(
            counts_to_options(
                counts['version'].items(),
                name='version',
                display=_('Version'),
                display_map=empty_to_unknown,
                checked=search_version)
        )

    filter_data.extend(
        [
            counts_to_options(
                counts['platform'].items(),
                name='platform',
                display=_('Platform'),
                display_map=empty_to_unknown,
                checked=search_platform),
            counts_to_options(
                counts['locale'].items(),
                name='locale',
                display=_('Locale'),
                checked=search_locale,
                display_map=locale_name),
        ]
    )

    # Histogram data
    happy_data = []
    sad_data = []

    happy_f = f & F(happy=True)
    sad_f = f & F(happy=False)
    histograms = search.facet_raw(
        happy={
            'date_histogram': {'interval': 'day', 'field': 'created'},
            'facet_filter': search._process_filters(happy_f.filters)
        },
        sad={
            'date_histogram': {'interval': 'day', 'field': 'created'},
            'facet_filter': search._process_filters(sad_f.filters)
        },
    ).facet_counts()

    # p['time'] is number of milliseconds since the epoch. Which is
    # convenient, because that is what the front end wants.
    happy_data = dict((p['time'], p['count']) for p in histograms['happy'])
    sad_data = dict((p['time'], p['count']) for p in histograms['sad'])

    zero_fill(search_date_start, search_date_end, [happy_data, sad_data])
    histogram = [
        {'label': _('Happy'), 'name': 'happy',
         'data': sorted(happy_data.items())},
        {'label': _('Sad'), 'name': 'sad',
         'data': sorted(sad_data.items())},
    ]

    return render(request, template, {
        'opinions': opinion_page,
        'opinion_count': search_count,
        'filter_data': filter_data,
        'histogram': histogram,
        'page': page,
        'prev_page': page - 1 if start > 0 else None,
        'next_page': page + 1 if end < search_count else None,
        'current_search': current_search,
        'selected': selected,
        'atom_url': generate_dashboard_url(request),
    })