text:
# Copyright 2022 The Kubric Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# pylint: disable=line-too-long, unexpected-keyword-arg
import dataclasses
import json
import logging

import imageio
import numpy as np
import png

import tensorflow as tf
import tensorflow_datasets.public_api as tfds
from typing import List, Dict, Union

_DESCRIPTION = """
A simple rigid-body simulation based on the CLEVR dataset.
The scene consists of a randomly colored floor, four light sources, a camera, and between
3 and 10 random objects.
The camera position is sampled randomly in a half-sphere shell around the scene
and always points at the origin.

The objects are randomly chosen from:
 - one of eleven shapes ("cube", "cylinder", "sphere", "cone", "torus", "gear",
                         "torus_knot", "sponge", "spot", "teapot", "suzanne")
 - scaled uniformly between 0.7 and 1.4,
 - have one of two materials [rubber, metal],
 - a uniformly random hue

They are spawned without overlap in the region [(-5, -5, 1), (5, 5, 5)], and
initialized with a random velocity from the range [(-4, -4, 0), (4, 4, 0)]
minus the position of the object to bias their trajectory towards the center of
the scene.

The scene is simulated for 2 seconds, with the physical properties of the
objects depending on the material:
 - metal: friction=0.4, restitution=0.3, density=2.7
 - rubber: friction=0.8, restitution=0.7, density=1.1

The dataset contains approx 10k videos rendered at 256x256 pixels and 12fps.

Each sample contains the following video-format data:
(s: sequence length, h: height, w: width)

- "video": (s, h, w, 3) [uint8]
  The RGB frames.
- "segmentations": (s, h, w, 1) [uint8]
  Instance segmentation as per-pixel object-id with background=0.
  Note: because of this the instance IDs used here are one higher than their
  corresponding index in sample["instances"].
- "depth": (s, h, w, 1) [uint16]
  Distance of each pixel from the center of the camera.
  (Note this is different from the z-value sometimes used, which measures the
  distance to the camera *plane*.)
  The values are stored as uint16 and span the range specified in
  sample["metadata"]["depth_range"]. To convert them back to world-units
  use:
    minv, maxv = sample["metadata"]["depth_range"]
    depth = sample["depth"] / 65535 * (maxv - minv) + minv
- "forward_flow": (s, h, w, 2) [uint16]
  Forward optical flow in the form (delta_row, delta_column).
  The values are stored as uint16 and span the range specified in
  sample["metadata"]["forward_flow_range"]. To convert them back to pixels use:
    minv, maxv = sample["metadata"]["forward_flow_range"]
    depth = sample["forward_flow"] / 65535 * (maxv - minv) + minv
- "backward_flow": (s, h, w, 2) [uint16]
  Backward optical flow in the form (delta_row, delta_column).
  The values are stored as uint16 and span the range specified in
  sample["metadata"]["backward_flow_range"]. To convert them back to pixels use:
    minv, maxv = sample["metadata"]["backward_flow_range"]
    depth = sample["backward_flow"] / 65535 * (maxv - minv) + minv
- "normal": (s, h, w, 3) [uint16]
  Surface normals for each pixel in world coordinates.
- "object_coordinates": (s, h, w, 3) [uint16]
  Object coordinates encode the position of each point relative to the objects
  bounding box (i.e. back-left-top (X=Y=Z=1) corner is white,
  while front-right-bottom (X=Y=Z=0) corner is black.)

Additionally there is rich instance-level information in sample["instances"]:
- "mass": [float32]
  Mass of the object used for simulation.
- "friction": [float32]
  Friction coefficient used for simulation.
- "restitution": [float32]
  Restitution coefficient (bounciness) used for simulation.
- "positions": (s, 3) [float32]
  Position of the object for each frame in world-coordinates.
- "quaternions": (s, 4) [float32]
  Rotation of the object for each frame as quaternions.
- "velocities": (s, 3) [float32]
  Velocity of the object for each frame.
- "angular_velocities": (s, 3) [float32]
  Angular velocity of the object for each frame.
- "bboxes_3d": (s, 8, 3) [float32]
  World-space corners of the 3D bounding box around the object.
- "image_positions": (s, 2) [float32]
  Normalized (0, 1) image-space (2D) coordinates of the center of mass of the
  object for each frame.
- "bboxes": (None, 4) [float32]
   The normalized image-space (2D) coordinates of the bounding box
   [ymin, xmin, ymax, xmax] for all the frames in which the object is visible
   (as specified in bbox_frames).
- "bbox_frames": (None,) [int]
   A list of all the frames the object is visible.
- "visibility": (s,) [uint16]
  Visibility of the object in number of pixels for each frame (can be 0).
- "shape_label": ["cube", "cylinder", "sphere", "cone", "torus", "gear",
                  "torus_knot", "sponge", "spot", "teapot", "suzanne"]
- "scale": float between 0.7 and 1.4
- "color": (3,) [float32]
  Color of the object in RGB.
- "material_label": ["metal", "rubber"]

Information about the camera in sample["camera"]
(given for each frame eventhough the camera is static, so as to stay
consistent with other variants of the dataset):

- "focal_length": [float32]
- "sensor_width": [float32]
- "field_of_view": [float32]
- "positions": (s, 3) [float32]
- "quaternions": (s, 4) [float32]

A single entry about the background:
- "background_color": (3,) [float32]

And finally information about collision events in sample["events"]["collisions"]:

- "instances": (2,)[uint16]
  Indices of the two instance between which the collision happened.
  Note that collisions with the floor/background objects are marked with 65535
- "frame": tf.int32,
  Frame in which the collision happenend.
- "force": tf.float32,
  The force (strength) of the collision.
- "position": tfds.features.Tensor(shape=(3,), dtype=tf.float32),
  Position of the collision event in 3D world coordinates.
- "image_position": tfds.features.Tensor(shape=(2,), dtype=tf.float32),
  Position of the collision event projected onto normalized 2D image coordinates.
- "contact_normal": tfds.features.Tensor(shape=(3,), dtype=tf.float32),
  The normal-vector of the contact (direction of the force).
"""

_CITATION = """\
@inproceedings{greff2022kubric,
title = {Kubric: a scalable dataset generator},
    author = {Klaus Greff and Francois Belletti and Lucas Beyer and Carl Doersch and
              Yilun Du and Daniel Duckworth and David J Fleet and Dan Gnanapragasam and
              Florian Golemo and Charles Herrmann and Thomas Kipf and Abhijit Kundu and
              Dmitry Lagun and Issam Laradji and Hsueh-Ti (Derek) Liu and Henning Meyer and
              Yishu Miao and Derek Nowrouzezahrai and Cengiz Oztireli and Etienne Pot and
              Noha Radwan and Daniel Rebain and Sara Sabour and Mehdi S. M. Sajjadi and Matan Sela and
              Vincent Sitzmann and Austin Stone and Deqing Sun and Suhani Vora and Ziyu Wang and
              Tianhao Wu and Kwang Moo Yi and Fangcheng Zhong and Andrea Tagliasacchi},
    booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition, {CVPR}},
    year = {2022},
    publisher = {Computer Vision Foundation / {IEEE}},
}"""

@dataclasses.dataclass
class MoviBConfig(tfds.core.BuilderConfig):
  """"Configuration for Multi-Object Video (MOVi) dataset."""
  height: int = 256
  width: int = 256
  num_frames: int = 24
  validation_ratio: float = 0.1
  train_val_path: str = None
  test_split_paths: Dict[str, str] = dataclasses.field(default_factory=dict)

class MoviB(tfds.core.BeamBasedBuilder):
  """DatasetBuilder for Movi-B dataset."""
  VERSION = tfds.core.Version("1.0.0")
  RELEASE_NOTES = {
      "1.0.0": "initial release",
  }

  BUILDER_CONFIGS = [
      MoviBConfig(
          name="256x256",
          description="Full resolution of 256x256",
          height=256,
          width=256,
          validation_ratio=0.025,
          # train_val_path="/usr/local/google/home/klausg/movi_tmp",
          train_val_path="gs://research-brain-kubric-xgcp/jobs/movi_b_regen_10k/",
          test_split_paths={
          }
      ),
      MoviBConfig(
          name="128x128",
          description="Downscaled to 128x128",
          height=128,
          width=128,
          validation_ratio=0.025,
          # train_val_path="/usr/local/google/home/klausg/movi_tmp",
          train_val_path="gs://research-brain-kubric-xgcp/jobs/movi_b_regen_10k/",
          test_split_paths={
          }
      ),
  ]

  def _info(self) -> tfds.core.DatasetInfo:
    """Returns the dataset metadata."""

    h = self.builder_config.height
    w = self.builder_config.width
    s = self.builder_config.num_frames

    def get_movi_b_instance_features(seq_length: int):
      features = get_instance_features(seq_length)
      features.update({
          "shape_label": tfds.features.ClassLabel(
              names=["cube", "cylinder", "sphere", "cone", "torus", "gear",
                      "torus_knot", "sponge", "spot", "teapot", "suzanne"]),
          "color": tfds.features.Tensor(shape=(3,), dtype=tf.float32),
          "material_label": tfds.features.ClassLabel(
              names=["metal", "rubber"]),
      })
      return features

    return tfds.core.DatasetInfo(
        builder=self,
        description=_DESCRIPTION,
        features=tfds.features.FeaturesDict({
            "metadata": {
                "video_name": tfds.features.Text(),
                "width": tf.int32,
                "height": tf.int32,
                "num_frames": tf.int32,
                "num_instances": tf.uint16,

                "depth_range": tfds.features.Tensor(shape=(2,),
                                                    dtype=tf.float32),
                "forward_flow_range": tfds.features.Tensor(shape=(2,),
                                                           dtype=tf.float32),
                "backward_flow_range": tfds.features.Tensor(shape=(2,),
                                                            dtype=tf.float32),
            },
            "background_color": tfds.features.Tensor(shape=(3,),
                                                     dtype=tf.float32),
            "instances": tfds.features.Sequence(
                feature=get_movi_b_instance_features(seq_length=s)),
            "camera": get_camera_features(s),
            "events": get_events_features(),
            # -----
            "video":  tfds.features.Video(shape=(s, h, w, 3)),
            "segmentations": tfds.features.Sequence(
                tfds.features.Image(shape=(h, w, 1), dtype=tf.uint8),
                length=s),
            "forward_flow": tfds.features.Sequence(
                tfds.features.Tensor(shape=(h, w, 2), dtype=tf.uint16),
                length=s),
            "backward_flow": tfds.features.Sequence(
                tfds.features.Tensor(shape=(h, w, 2), dtype=tf.uint16),
                length=s),
            "depth": tfds.features.Sequence(
                tfds.features.Image(shape=(h, w, 1), dtype=tf.uint16),
                length=s),
            "normal": tfds.features.Video(shape=(s, h, w, 3), dtype=tf.uint16),
            "object_coordinates": tfds.features.Video(shape=(s, h, w, 3),
                                                      dtype=tf.uint16),
        }),
        supervised_keys=None,
        homepage="https://github.com/google-research/kubric",
        citation=_CITATION)

  def _split_generators(self, unused_dl_manager: tfds.download.DownloadManager):
    """Returns SplitGenerators."""
    del unused_dl_manager
    path = tfds.core.as_path(self.builder_config.train_val_path)
    all_subdirs = [str(d) for d in path.iterdir()]
    logging.info("Found %d sub-folders in master path: %s",
                 len(all_subdirs), path)

    # shuffle
    rng = np.random.RandomState(seed=42)
    rng.shuffle(all_subdirs)

    validation_ratio = self.builder_config.validation_ratio
    validation_examples = max(1, round(len(all_subdirs) * validation_ratio))
    training_examples = len(all_subdirs) - validation_examples
    logging.info("Using %f of examples for validation for a total of %d",
                 validation_ratio, validation_examples)
    logging.info("Using the other %d examples for training", training_examples)

    splits = {
        tfds.Split.TRAIN: self._generate_examples(all_subdirs[:training_examples]),
        tfds.Split.VALIDATION: self._generate_examples(all_subdirs[training_examples:]),
    }

    for key, path in self.builder_config.test_split_paths.items():
      path = tfds.core.as_path(path)
      split_dirs = [d for d in path.iterdir()]
      # sort the directories by their integer number
      split_dirs = sorted(split_dirs, key=lambda x: int(x.name))
      logging.info("Found %d sub-folders in '%s' path: %s",
                   len(split_dirs), key, path)
      splits[key] = self._generate_examples([str(d) for d in split_dirs])

    return splits

  def _generate_examples(self, directories: List[str]):
    """Yields examples."""

    target_size = (self.builder_config.height, self.builder_config.width)

    def _process_example(video_dir):
      key, result, metadata = load_scene_directory(video_dir, target_size)

      # add Movi-B specific instance information:
      for i, obj in enumerate(result["instances"]):
        obj["shape_label"] = metadata["instances"][i]["shape"]
        obj["material_label"] = metadata["instances"][i]["material"]
        obj["color"] = np.array(metadata["instances"][i]["color"],
                                dtype=np.float32)
      return key, result

    beam = tfds.core.lazy_imports.apache_beam
    return (beam.Create(directories) |
            beam.Filter(is_complete_dir) |
            beam.Map(_process_example))

DEFAULT_LAYERS = ("rgba", "segmentation", "forward_flow", "backward_flow",
                  "depth", "normal", "object_coordinates")

def load_scene_directory(scene_dir, target_size, layers=DEFAULT_LAYERS):
  scene_dir = tfds.core.as_path(scene_dir)
  example_key = f"{scene_dir.name}"

  with tf.io.gfile.GFile(str(scene_dir / "data_ranges.json"), "r") as fp:
    data_ranges = json.load(fp)

  with tf.io.gfile.GFile(str(scene_dir / "metadata.json"), "r") as fp:
    metadata = json.load(fp)

  with tf.io.gfile.GFile(str(scene_dir / "events.json"), "r") as fp:
    events = json.load(fp)

  num_frames = metadata["metadata"]["num_frames"]

  result = {
      "metadata": {
          "video_name": example_key,
          "width": target_size[1],
          "height": target_size[0],
          "num_frames": num_frames,
          "num_instances": metadata["metadata"]["num_instances"],
      },
      "background_color": rgb_from_hexstr(metadata["metadata"]["background"]),
      "instances": [format_instance_information(obj)
                    for obj in metadata["instances"]],
      "camera": format_camera_information(metadata),
      "events": format_events_information(events),
  }

  resolution = metadata["metadata"]["resolution"]

  assert resolution[0] / target_size[0] == resolution[1] / target_size[1]
  scale = resolution[0] / target_size[0]
  assert scale == resolution[0] // target_size[0]

  paths = {
      key: [scene_dir / f"{key}_{f:05d}.png" for f in range(num_frames)]
      for key in layers if key != "depth"
  }

  if "depth" in layers:
    depth_paths = [scene_dir / f"depth_{f:05d}.tiff" for f in range(num_frames)]
    depth_frames = np.array([
        subsample_nearest_neighbor(read_tiff(frame_path), target_size)
        for frame_path in depth_paths])
    depth_min, depth_max = np.min(depth_frames), np.max(depth_frames)
    result["depth"] = convert_float_to_uint16(depth_frames, depth_min, depth_max)
    result["metadata"]["depth_range"] = [depth_min, depth_max]

  if "forward_flow" in layers:
    result["metadata"]["forward_flow_range"] = [
        data_ranges["forward_flow"]["min"] / scale,
        data_ranges["forward_flow"]["max"] / scale]
    result["forward_flow"] = [
        subsample_nearest_neighbor(read_png(frame_path)[..., :2],
                                   target_size)
        for frame_path in paths["forward_flow"]]

  if "backward_flow" in layers:
    result["metadata"]["backward_flow_range"] = [
        data_ranges["backward_flow"]["min"] / scale,
        data_ranges["backward_flow"]["max"] / scale]
    result["backward_flow"] = [
        subsample_nearest_neighbor(read_png(frame_path)[..., :2],
                                   target_size)
        for frame_path in paths["backward_flow"]]

  for key in ["normal", "object_coordinates", "uv"]:
    if key in layers:
      result[key] = [
          subsample_nearest_neighbor(read_png(frame_path),
                                     target_size)
          for frame_path in paths[key]]

  if "segmentation" in layers:
    # somehow we ended up calling this "segmentations" in TFDS and
    # "segmentation" in kubric. So we have to treat it separately.
    result["segmentations"] = [
        subsample_nearest_neighbor(read_png(frame_path),
                                   target_size)
        for frame_path in paths["segmentation"]]

  if "rgba" in layers:
    result["video"] = [
        subsample_avg(read_png(frame_path), target_size)[..., :3]
        for frame_path in paths["rgba"]]

  return example_key, result, metadata

def get_camera_features(seq_length):
  return {
      "focal_length": tf.float32,
      "sensor_width": tf.float32,
      "field_of_view": tf.float32,
      "positions": tfds.features.Tensor(shape=(seq_length, 3),
                                        dtype=tf.float32),
      "quaternions": tfds.features.Tensor(shape=(seq_length, 4),
                                          dtype=tf.float32),
  }

def format_camera_information(metadata):
  return {
      "focal_length": metadata["camera"]["focal_length"],
      "sensor_width": metadata["camera"]["sensor_width"],
      "field_of_view": metadata["camera"]["field_of_view"],
      "positions": np.array(metadata["camera"]["positions"], np.float32),
      "quaternions": np.array(metadata["camera"]["quaternions"], np.float32),
  }

def get_events_features():
  return {
      "collisions": tfds.features.Sequence({
          "instances": tfds.features.Tensor(shape=(2,), dtype=tf.uint16),
          "frame": tf.int32,
          "force": tf.float32,
          "position": tfds.features.Tensor(shape=(3,), dtype=tf.float32),
          "image_position": tfds.features.Tensor(shape=(2,), dtype=tf.float32),
          "contact_normal": tfds.features.Tensor(shape=(3,), dtype=tf.float32),
      })
  }

def format_events_information(events):
  return {
      "collisions": [{
          "instances": np.array(c["instances"], dtype=np.uint16),
          "frame": c["frame"],
          "force": c["force"],
          "position": np.array(c["position"], dtype=np.float32),
          "image_position": np.array(c["image_position"], dtype=np.float32),
          "contact_normal": np.array(c["contact_normal"], dtype=np.float32),
      } for c in events["collisions"]],
  }

def get_instance_features(seq_length: int):
  return {
      "mass": tf.float32,
      "friction": tf.float32,
      "restitution": tf.float32,

      "positions": tfds.features.Tensor(shape=(seq_length, 3),
                                        dtype=tf.float32),
      "quaternions": tfds.features.Tensor(shape=(seq_length, 4),
                                          dtype=tf.float32),
      "velocities": tfds.features.Tensor(shape=(seq_length, 3),
                                         dtype=tf.float32),
      "angular_velocities": tfds.features.Tensor(shape=(seq_length, 3),
                                                 dtype=tf.float32),
      "bboxes_3d": tfds.features.Tensor(shape=(seq_length, 8, 3),
                                        dtype=tf.float32),

      "image_positions": tfds.features.Tensor(shape=(seq_length, 2),
                                              dtype=tf.float32),
      "bboxes": tfds.features.Sequence(
          tfds.features.BBoxFeature()),
      "bbox_frames": tfds.features.Sequence(
          tfds.features.Tensor(shape=(), dtype=tf.int32)),
      "visibility": tfds.features.Tensor(shape=(seq_length,), dtype=tf.uint16),
  }

def format_instance_information(obj):
  return {
      "mass": obj["mass"],
      "friction": obj["friction"],
      "restitution": obj["restitution"],
      "positions": np.array(obj["positions"], np.float32),
      "quaternions": np.array(obj["quaternions"], np.float32),
      "velocities": np.array(obj["velocities"], np.float32),
      "angular_velocities": np.array(obj["angular_velocities"], np.float32),
      "bboxes_3d": np.array(obj["bboxes_3d"], np.float32),
      "image_positions": np.array(obj["image_positions"], np.float32),
      "bboxes": [tfds.features.BBox(*bbox) for bbox in obj["bboxes"]],
      "bbox_frames": np.array(obj["bbox_frames"], dtype=np.uint16),
      "visibility": np.array(obj["visibility"], dtype=np.uint16),
  }

def subsample_nearest_neighbor(arr, size):
  src_height, src_width, _ = arr.shape
  dst_height, dst_width = size
  height_step = src_height // dst_height
  width_step = src_width // dst_width
  assert height_step * dst_height == src_height
  assert width_step * dst_width == src_width

  height_offset = int(np.floor((height_step-1)/2))
  width_offset = int(np.floor((width_step-1)/2))
  subsampled = arr[height_offset::height_step, width_offset::width_step, :]
  return subsampled

def convert_float_to_uint16(array, min_val, max_val):
  return np.round((array - min_val) / (max_val - min_val) * 65535
                  ).astype(np.uint16)

def subsample_avg(arr, size):
  src_height, src_width, channels = arr.shape
  dst_height, dst_width = size
  height_bin = src_height // dst_height
  width_bin = src_width // dst_width
  return np.round(arr.reshape((dst_height, height_bin,
                               dst_width, width_bin,
                               channels)).mean(axis=(1, 3))).astype(np.uint8)

def is_complete_dir(video_dir, layers=DEFAULT_LAYERS):
  video_dir = tfds.core.as_path(video_dir)
  filenames = [d.name for d in video_dir.iterdir()]
  if not ("data_ranges.json" in filenames and
          "metadata.json" in filenames and
          "events.json" in filenames):
    return False
  nr_frames_per_category = {
      key: len([fn for fn in filenames if fn.startswith(key)])
      for key in layers}

  nr_expected_frames = nr_frames_per_category["rgba"]
  if nr_expected_frames == 0:
    return False
  if not all(nr_frames == nr_expected_frames
             for nr_frames in nr_frames_per_category.values()):
    return False

  return True

PathLike = Union[str, tfds.core.ReadWritePath]

def as_path(path: PathLike) -> tfds.core.ReadWritePath:
  """Convert str or pathlike object to tfds.core.ReadWritePath.

  Instead of pathlib.Paths, we use the TFDS path because they transparently
  support paths to GCS buckets such as "gs://kubric-public/GSO".
  """
  return tfds.core.as_path(path)

def read_png(filename, rescale_range=None) -> np.ndarray:
  filename = as_path(filename)
  png_reader = png.Reader(bytes=filename.read_bytes())
  width, height, pngdata, info = png_reader.read()
  del png_reader

  bitdepth = info["bitdepth"]
  if bitdepth == 8:
    dtype = np.uint8
  elif bitdepth == 16:
    dtype = np.uint16
  else:
    raise NotImplementedError(f"Unsupported bitdepth: {bitdepth}")

  plane_count = info["planes"]
  pngdata = np.vstack(list(map(dtype, pngdata)))
  if rescale_range is not None:
    minv, maxv = rescale_range
    pngdata = pngdata / 2**bitdepth * (maxv - minv) + minv

  return pngdata.reshape((height, width, plane_count))

def write_tiff(data: np.ndarray, filename: PathLike):
  """Save data as as tif image (which natively supports float values)."""
  assert data.ndim == 3, data.shape
  assert data.shape[2] in [1, 3, 4], "Must be grayscale, RGB, or RGBA"

  img_as_bytes = imageio.imwrite("<bytes>", data, format="tiff")
  filename = as_path(filename)
  filename.write_bytes(img_as_bytes)

def read_tiff(filename: PathLike) -> np.ndarray:
  filename = as_path(filename)
  img = imageio.imread(filename.read_bytes(), format="tiff")
  if img.ndim == 2:
    img = img[:, :, None]
  return img

def rgb_from_hexstr(hexstr: str):
  """Create a Color instance from a hex string like #ffaa22 or #11aa88ff.

  Supports both long and short form (i.e. #ffffff is the same as #fff), and also an optional
  alpha value (e.g. #112233ff or #123f).
  """
  if hexstr[0] == "#":  # get rid of leading #
    hexstr = hexstr[1:]
  if len(hexstr) == 3:
    r = int(hexstr[0], 16) / 15.
    g = int(hexstr[1], 16) / 15.
    b = int(hexstr[2], 16) / 15.
    return r, g, b
  elif len(hexstr) == 4:
    r = int(hexstr[0], 16) / 15.
    g = int(hexstr[1], 16) / 15.
    b = int(hexstr[2], 16) / 15.
    a = int(hexstr[3], 16) / 15.
    return r, g, b
  elif len(hexstr) == 6:
    r = int(hexstr[0:2], 16) / 255.0
    g = int(hexstr[2:4], 16) / 255.0
    b = int(hexstr[4:6], 16) / 255.0
    return r, g, b
  elif len(hexstr) == 8:
    r = int(hexstr[0:2], 16) / 255.0
    g = int(hexstr[2:4], 16) / 255.0
    b = int(hexstr[4:6], 16) / 255.0
    a = int(hexstr[6:8], 16) / 255.0
    return r, g, b
  else:
    raise ValueError("invalid color hex string")

max_stars_repo_name: ziyuw/kubric

max_issues_repo_name: ziyuw/kubric

max_forks_repo_name: ziyuw/kubric


text:
"""Tests for `narrow_down.minhash`."""
import numpy as np
import pytest

from narrow_down import _minhash, storage
from narrow_down.storage import StorageLevel, StoredDocument, TooLowStorageLevel

def test_minhash_lsh_config__json():
    config1 = _minhash.MinhashLshConfig(0, 1, 2)
    assert _minhash.MinhashLshConfig.from_json(config1.to_json()) == config1

def test_minhash():
    """Check minhashing of a document with hand-checked result."""
    mh = _minhash.MinHasher(2, 42)
    minhashes = mh.minhash(["abc", "def", "g"])

    assert minhashes.shape == (2,)
    assert minhashes.dtype == np.uint32
    assert (minhashes == np.array([2048153058, 2194504465], dtype=np.uint32)).all()

def test_minhash_benchmark(benchmark, sample_byte_strings):
    sample_strings = [s.decode("utf-8") for s in sample_byte_strings]

    def f():
        mh = _minhash.MinHasher(64, 42)
        return list(map(mh.minhash, sample_strings))[-1]

    minhashes = benchmark(f)

    assert minhashes.dtype == np.uint32
    assert minhashes.shape == (64,)

@pytest.mark.asyncio
async def test_lsh__basic_lookup_without_exact_part():
    """Minimal check if an LSH can be constructed."""
    test_doc = StoredDocument(
        exact_part="exact:part",
        fingerprint=storage.Fingerprint(np.array([2 * i for i in range(1, 22)])),
    )
    unrelated_doc = StoredDocument(
        exact_part="exact:part",
        fingerprint=storage.Fingerprint(np.array([3 * i for i in range(0, 21)])),
    )
    lsh = _minhash.LSH(
        _minhash.MinhashLshConfig(n_hashes=2, n_bands=2, rows_per_band=1),
        storage=await storage.InMemoryStore().initialize(),
    )
    await lsh.insert(test_doc.without("exact_part"))
    await lsh.insert(unrelated_doc.without("exact_part"))
    result = await lsh.query(test_doc.fingerprint)
    print(result)
    assert len(result) == 1
    assert (list(result)[0].fingerprint == test_doc.fingerprint).all

@pytest.mark.asyncio
async def test_lsh__basic_lookup_with_exact_part():
    """Minimal check if an LSH can be constructed."""
    test_doc = StoredDocument(
        # document="abc def",
        exact_part="exact:part",
        # data="some custom payload",
        fingerprint=storage.Fingerprint(np.array([2, 4, 6])),
    )
    lsh = _minhash.LSH(
        _minhash.MinhashLshConfig(n_hashes=2, n_bands=2, rows_per_band=1),
        storage=await storage.InMemoryStore().initialize(),
    )
    await lsh.insert(test_doc)
    result = await lsh.query(test_doc.fingerprint, exact_part=test_doc.exact_part)
    print(result)
    assert len(result) == 1
    assert (list(result)[0].fingerprint == test_doc.fingerprint).all
    assert list(result)[0].exact_part == test_doc.exact_part

@pytest.mark.asyncio
async def test_lsh__top_n():
    """Minimal check if an LSH can be constructed."""
    lsh = _minhash.LSH(
        _minhash.MinhashLshConfig(n_hashes=2, n_bands=2, rows_per_band=1),
        storage=await storage.InMemoryStore().initialize(),
    )
    await lsh.insert(
        StoredDocument(document="1", fingerprint=storage.Fingerprint(np.array([2, 4, 6]))),
    )
    await lsh.insert(
        StoredDocument(document="2", fingerprint=storage.Fingerprint(np.array([2, 4, 7]))),
    )
    await lsh.insert(
        StoredDocument(document="3", fingerprint=storage.Fingerprint(np.array([2, 5, 7]))),
    )
    await lsh.insert(
        StoredDocument(document="4", fingerprint=storage.Fingerprint(np.array([3, 5, 7]))),
    )

    result = await lsh.query_top_n(n=1, fingerprint=storage.Fingerprint(np.array([2, 4, 6])))
    print(result)
    assert sorted([r.document for r in result]) == ["1"]

    result = await lsh.query_top_n(n=2, fingerprint=storage.Fingerprint(np.array([2, 4, 6])))
    print(result)
    assert sorted([r.document for r in result]) == ["1", "2"]

    result = await lsh.query_top_n(n=3, fingerprint=storage.Fingerprint(np.array([2, 4, 6])))
    print(result)
    assert sorted([r.document for r in result]) == ["1", "2", "3"]

    result = await lsh.query_top_n(n=4, fingerprint=storage.Fingerprint(np.array([2, 4, 6])))
    print(result)
    assert sorted([r.document for r in result]) == ["1", "2", "3"]

@pytest.mark.asyncio
async def test_lsh__insert_invalid_document():
    """Test error handling of insert()."""
    lsh = _minhash.LSH(_minhash.MinhashLshConfig(1, 1, 1), None)
    with pytest.raises(ValueError):
        await lsh.insert(StoredDocument())

@pytest.mark.asyncio
async def test_lsh__remove_by_id():
    """Try to remove a document from the index."""
    f = storage.Fingerprint(np.array([2, 4, 6]))
    test_doc = StoredDocument(fingerprint=f)
    lsh = _minhash.LSH(
        _minhash.MinhashLshConfig(n_hashes=2, n_bands=2, rows_per_band=1),
        storage=await storage.InMemoryStore().initialize(),
    )
    index = await lsh.insert(test_doc)
    await lsh.remove_by_id(index, check_if_exists=True)
    assert list(await lsh.query(f)) == []

@pytest.mark.asyncio
async def test_lsh__remove_by_id__missing_fingerprint():
    """Try to remove a document from the index when the storage level is too low."""
    f = storage.Fingerprint(np.array([2, 4, 6]))
    test_doc = StoredDocument(fingerprint=f)
    lsh = _minhash.LSH(
        _minhash.MinhashLshConfig(n_hashes=2, n_bands=2, rows_per_band=1),
        storage=await storage.InMemoryStore().initialize(),
    )
    index = await lsh.insert(test_doc, storage_level=StorageLevel.Minimal)
    with pytest.raises(TooLowStorageLevel):
        await lsh.remove_by_id(index, check_if_exists=True)

@pytest.mark.parametrize(
    "j, fn, fp, expected",
    [
        (0.5, 0.05, 0.05, _minhash.MinhashLshConfig(n_hashes=128, n_bands=22, rows_per_band=5)),
        (0.5, 1, 0.1, _minhash.MinhashLshConfig(n_hashes=16, n_bands=1, rows_per_band=16)),
        (
            0.5,
            -1,
            0.2,
            _minhash.MinhashLshConfig(n_hashes=16384, n_bands=16384, rows_per_band=1),
        ),
        (
            0.5,
            1,
            -1,
            _minhash.MinhashLshConfig(n_hashes=16384, n_bands=1, rows_per_band=16384),
        ),
    ],
)
def test_find_optimal_config(j, fn, fp, expected) -> None:
    """Test the parameter optimization.

    Expectations: As long as the proba thresholds can't be reached the number of hashes should be
    increased. If the false negative threshold isn't reachable, the number of bands should be
    increased because that increases the matching chances. If the false positive threshold can't
    be reached the rows_per_band should be increased instead to reduce the matching chances.

    Args:
        j: Jaccard similarity threshold
        fn: Max false negative probability
        fp: Max false positive probabiblity
        expected: Expected output parameters
    """
    cfg = _minhash.find_optimal_config(
        jaccard_threshold=j, max_false_negative_proba=fn, max_false_positive_proba=fp
    )
    print(
        "False negative proba:",
        _minhash._false_negative_probability(j, b=cfg.n_bands, r=cfg.rows_per_band),
    )
    print(
        "False positive proba:",
        _minhash._false_positive_probability(j, b=cfg.n_bands, r=cfg.rows_per_band),
    )
    assert cfg == expected

ext: py

reason: import numpy


text:
from collections import OrderedDict
import torch
from torch import nn, optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models
from PIL import Image
import argparse
import numpy as np
import json

import utility # utility.py
import ai_model # ai_model.py

# Command Line Arguments
parser = argparse.ArgumentParser(description='predict.py')
parser.add_argument('--topk', default = 3, dest='topk', help='Return top K most likely classes', type=int)
parser.add_argument('--category_names', default = 'cat_to_name.json', dest='flower_names', help='Use a mapping of categories to real names')
parser.add_argument('--gpu', default='gpu', dest='gpu', help='Use GPU for inference')
parser.add_argument('checkpoint', default='/home/workspace/aipnd-project/checkpoint.pth', nargs='*', type = str)
parser.add_argument('input', default='./flowers/test/5/image_05159.jpg', nargs='*', type = str)

args = parser.parse_args()
img_path = args.input
topk = args.topk
power = args.gpu
filepath = args.checkpoint

trainloader, validloader, testloader, train_data = utility.load_data()
model, class_to_idx = ai_model.load_checkpoint(filepath)

# Label mapping

with open('cat_to_name.json', 'r') as f:
    cat_to_name = json.load(f)

pil_image = utility.process_image(img_path)

result = ai_model.predict(pil_image, model, topk, power)
probs = F.softmax(result[0].data, dim=1).cpu().numpy()[0]
classes = result[1].data.cpu().numpy()[0]

idx_to_class =  {v : k for k,v in class_to_idx.items()}
classes = [idx_to_class[x] for x in classes]
print(probs)
print(classes)
labels = [cat_to_name[str(i)] for i in classes]

i = 0
while i < topk:
    print ("It's {} percent likely that the flower is {}.".format(probs[i], labels[i]))
    i += 1

print("---End---")


text:
import matplotlib.animation as animation
import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import KMeans
import inferpy as inf

from datareduction.variational_gaussian_mixture_DR import VariationalGaussianMixture_DR
from datareduction.variational_gaussian_mixture_SVI import VariationalGaussianMixture_SVI
from prml.rv import VariationalGaussianMixture
from prml.features import PolynomialFeatures
from prml.linear import (
    VariationalLinearRegressor,
    VariationalLogisticRegressor
)

np.random.seed(1234)

N=10000
K=5
M=10
D=10

x_train = inf.models.Normal(0,1, dim = D).sample(int(N/K))
x_test = inf.models.Normal(0,1, dim = D).sample(int(N/K))

for i in range(1,K):
    x_train=np.append(x_train, inf.models.Normal(i,1, dim = D).sample(int(N/K)),axis=0)
    x_test=np.append(x_test, inf.models.Normal(i,1, dim = D).sample(int(N/K)),axis=0)

np.take(x_train,np.random.permutation(x_train.shape[0]),axis=0,out=x_train)
np.take(x_test,np.random.permutation(x_test.shape[0]),axis=0,out=x_test)

######################################################
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/")

#data = data[np.random.choice(np.where(target == 3)[0], 10000)]
np.take(mnist.train.images,np.random.permutation(mnist.train.images.shape[0]),axis=0,out=mnist.train.images)
np.take(mnist.test.images,np.random.permutation(mnist.test.images.shape[0]),axis=0,out=mnist.test.images)

D=mnist.train.images.shape[1]

x_train = mnist.train.images[0:100,:]
x_test = mnist.test.images[100:200,:]
######################################################

# np.random.seed(1234)
# vgmm = VariationalGaussianMixture(n_components=K)
# vgmm.fit(x_train)
# print(np.sum(vgmm.logpdf(x_test)))

np.random.seed(1234)
vgmm_svi = VariationalGaussianMixture_SVI(n_components=K)
vgmm_svi.fit(x_train, iter_max=1, batch_size=10)
print(np.sum(vgmm_svi.logpdf(x_test)))

max_stars_repo_name: andresmasegosa/PRML-CoreSets

max_issues_repo_name: andresmasegosa/PRML-CoreSets

max_forks_repo_name: andresmasegosa/PRML-CoreSets


text:
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
import torchvision.transforms as transforms

import densetorch as dt
from config import *

# model_options
## encoder
pretrained = True
return_idx = [1, 20]
config = 8  # output stride 8

# set seeds
dt.misc.set_seed(seed)

# data setup
transform_common = [dt.data.Normalise(*normalise_params), dt.data.ToTensor()]
transform_train = transforms.Compose(
    [dt.data.RandomMirror(), dt.data.RandomCrop(crop_size)] + transform_common
)
transform_val = transforms.Compose(transform_common)
trainloader = DataLoader(
    dt.data.MMDataset(
        data_file, data_dir, line_to_paths_fn, masks_names, transform=transform_train,
    ),
    batch_size=batch_size,
    shuffle=True,
    num_workers=4,
    pin_memory=True,
    drop_last=True,
)
valloader = DataLoader(
    dt.data.MMDataset(
        val_file, data_val_dir, line_to_paths_fn, masks_names, transform=transform_val,
    ),
    batch_size=val_batch_size,
    shuffle=False,
    num_workers=4,
    pin_memory=True,
    drop_last=False,
)

# model setup
enc = dt.nn.xception65(pretrained=pretrained, return_idx=return_idx)
dec = dt.nn.DLv3plus(enc._out_c, num_classes, rates=enc.rates)
model1 = nn.DataParallel(nn.Sequential(enc, dec).cuda())
print("Model has {} parameters".format(dt.misc.compute_params(model1)))
start_epoch, _, state_dict = saver.maybe_load(
    ckpt_path=ckpt_path, keys_to_load=["epoch", "best_val", "state_dict"],
)
dt.misc.load_state_dict(model1, state_dict)
if start_epoch is None:
    start_epoch = 0

# optim setup
optims = [
    dt.misc.create_optim(
        optim_enc, enc.parameters(), lr=lr_enc, momentum=mom_enc, weight_decay=wd_enc
    ),
    dt.misc.create_optim(
        optim_dec, dec.parameters(), lr=lr_dec, momentum=mom_dec, weight_decay=wd_dec
    ),
]

# schedulers
opt_scheds = []
for opt in optims:
    opt_scheds.append(
        torch.optim.lr_scheduler.MultiStepLR(
            opt, np.arange(start_epoch + 1, n_epochs, 100), gamma=0.1
        )
    )

for i in range(start_epoch, n_epochs):
    for sched in opt_scheds:
        sched.step(i)
    model1.train()
    print("Epoch {:d}".format(i))
    dt.engine.train(model1, optims, [crit_segm], trainloader, loss_coeffs)
    if i % val_every == 0:
        metrics = [dt.engine.MeanIoU(num_classes)]
        model1.eval()
        with torch.no_grad():
            vals = dt.engine.validate(model1, metrics, valloader)
        saver.maybe_save(
            new_val=vals, dict_to_save={"state_dict": model1.state_dict(), "epoch": i}
        )

max_stars_repo_name: DrSleep/DenseTorch

max_issues_repo_name: DrSleep/DenseTorch

max_forks_repo_name: DrSleep/DenseTorch


text:
"""
This script is used to generate values that are plotted in the right panel of Figure 5.5.
Here, we calculate numerically the average fraction of slow genes present in the steady
state densities for the dimorphic fast-dimer competition (with Hölder exponent
$\theta =1$) for a range of strengths of between-protocell competition $\lambda$.
"""

import numpy as np
import matplotlib.pyplot as plt
import scipy.integrate as spi

x_min = 0.
x_max = 1.
x_step = 0.0001
partial_x_range = np.arange(x_min + x_step, x_max + x_step, x_step)
full_x_range = np.arange(x_min, x_max + x_step, x_step)

lamb_min = 0.
lamb_max = 250.
lamb_step = 20.
lamb_range = np.arange(lamb_min,lamb_max + lamb_step, lamb_step)

eta = 1.
s = 1.
theta = 1.

"""
Formulas for the threshold selection strength needed to achieve coexistence, as well as
the formula for steady state densities on the fast-dimer edge and the corresponding
integrand used to compute the mean fraction of slow genes.
"""

def lamb_tilde(lamb,s):
	b_F = 1 + s
	b_D = (1.0+s) / (2.0 + s)
	lamb_tilde = lamb / (2.0 * (b_F - b_D))
	return lamb_tilde

def steadyFD(lamb, eta, s, z, theta):
	z_exp = 0.5 * lamb_tilde(lamb,s) * (1.0 - 0.5 * eta) - theta - 1.
	if z_exp > -1.:
		return (z ** z_exp) * ((1.0 - z) ** (theta - 1.)) * np.exp(-0.25 *lamb_tilde(lamb,s) * eta * z)
	else:
		return 0.

def steadyFD_times_z(lamb, eta, s, z, theta):
	z_exp = 0.5 * lamb_tilde(lamb,s) * (1.0 - 0.5 * eta) - theta
	if z_exp > 0.:
		return (z ** z_exp) * ((1.0 - z) ** (theta - 1.)) * np.exp(-0.25 *lamb_tilde(lamb,s) * eta * z)
	else:
		return 0.

steadyFD_vec = np.vectorize(steadyFD)
steadyFD_times_x_vec = np.vectorize(steadyFD_times_z)

"""
Calculating the integrals for the mass of groups in the unnormalized steady state density
and the mean fraction of slow replicators for the normalized steady state densities.
"""
def int_near_zero(lamb,eta,s,z0,theta):
	exp = 0.5 * lamb_tilde(lamb,s) * (1.0 - 0.5 * eta) - theta
	if exp > 0:
		return (1.0 / exp) * (z0 ** (exp))
	else:
		return 0.

def mass(lamb,eta,s,z0,theta,x_range):
	near_zero = int_near_zero(lamb,eta,s,x_min,theta)
	rest = spi.simps(steadyFD_vec(lamb,eta,s,partial_x_range,theta),partial_x_range)
	return near_zero + rest

def mean(lamb,eta,s,z_0,theta,x_range_1,x_range_2):
	moment_mass = spi.simps(steadyFD_times_x_vec(lamb,eta,s,x_range_2,theta),x_range_2)
	normalizer = mass(lamb,eta,s,x_min,theta,x_range_1)
	if normalizer > 0.:
		return moment_mass / normalizer
	else:
		return 0.

print mass(200.,1.,1.,x_min,1.,partial_x_range)
print mean(7200.,1.,1.,x_min,1.,partial_x_range,full_x_range)

"""
Extracting $\lambda$ values used for studying the mean fraction of slow genes under
the trimorphic dynamics.
"""
script_folder = os.getcwd()
protocell_folder = os.path.dirname(script_folder)
file = protocell_folder + "/Simulation_Outputs/meanplottrimorphiceta1.txt"

def process(list):
	list = list.split(',')
	list = [float(a) for a in list]
	return list

f = open(file, 'r+')
lamb_list = f.readline()
#print lamb_list
lamb_list = process(lamb_list)

f.close()

mean_holder = [mean(lamb,eta,s,x_min,theta,partial_x_range,full_x_range) for lamb in lamb_list]
plt.plot(lamb_list,mean_holder, lw = 5., color = 'r', ls = '--')

"""
Saving the numerically computed means of the steady state density for the fast-dimer
competition.
"""
file2 = file_path + "FDmeanploteta1.txt"

f2 = open(file2, 'w+')

f2.write(str(lamb_list)[1:-1])
f2.write('\n')
f2.write(str(mean_holder)[1:-1])
f2.write('\n')

f2.close()

plt.show()

reason: import numpy,import scipy


text:
import pyredner
import redner
import torch
import scipy
import scipy.ndimage
import numpy as np

# Test Quasi Monte Carlo rendering.
# We optimize for the materials of a Cornell box scene

scene = pyredner.load_mitsuba('scenes/cbox/cbox.xml')
scene_args = pyredner.RenderFunction.serialize_scene(\
    scene = scene,
    num_samples = 512,
    max_bounces = 5, # Set max_bounces = 5 for global illumination
    sampler_type = redner.SamplerType.sobol) # Use Sobol sampler
render = pyredner.RenderFunction.apply
img = render(0, *scene_args)
pyredner.imwrite(img.cpu(), 'results/test_qmc/target.exr')
pyredner.imwrite(img.cpu(), 'results/test_qmc/target.png')
target = pyredner.imread('results/test_qmc/target.exr')
if pyredner.get_use_gpu():
    target = target.cuda()

# Now let's generate an initial guess by perturbing the reference.
# Let's set all the diffuse color to gray by manipulating material.diffuse_reflectance.
# We also store all the material variables to optimize in a list.
material_vars = []
for mi, m in enumerate(scene.materials):
    var = torch.tensor([0.5, 0.5, 0.5],
                       device = pyredner.get_device(),
                       requires_grad = True)
    material_vars.append(var)
    m.diffuse_reflectance = pyredner.Texture(var)

# Serialize the scene and render the initial guess
scene_args = pyredner.RenderFunction.serialize_scene(\
    scene = scene,
    num_samples = 512,
    max_bounces = 5,
    sampler_type = redner.SamplerType.sobol)
img = render(1, *scene_args)
pyredner.imwrite(img.cpu(), 'results/test_qmc/init.png')

# Optimize for parameters.
optimizer = torch.optim.Adam(material_vars, lr=1e-2)
for t in range(200):
    print('iteration:', t)
    optimizer.zero_grad()
    # Forward pass: serialize the scene and render the image
    # Need to redefine the camera
    scene_args = pyredner.RenderFunction.serialize_scene(\
        scene = scene,
        num_samples = 4,
        max_bounces = 5,
        sampler_type = redner.SamplerType.sobol)
    # Important to use a different seed every iteration, otherwise the result
    # would be biased.
    img = render(t+1, *scene_args)
    pyredner.imwrite(img.cpu(), 'results/test_qmc/iter_{}.png'.format(t))

    # Compute the loss function.
    # We clamp the difference between -1 and 1 to prevent
    # light source from being dominating the loss function
    loss = (img - target).clamp(-1.0, 1.0).pow(2).sum()
    print('loss:', loss.item())

    # Backpropagate the gradients.
    loss.backward()

    # Take a gradient descent step.
    optimizer.step()

    # Important: the material parameters has hard constraints: the
    # reflectance and roughness cannot be negative. We enforce them here
    # by projecting the values to the boundaries.
    for var in material_vars:
        var.data = var.data.clamp(1e-5, 1.0)
        print(var)

# Render the final result.
scene_args = pyredner.RenderFunction.serialize_scene(\
    scene = scene,
    num_samples = 512,
    max_bounces = 5,
    sampler_type = redner.SamplerType.sobol)
img = render(202, *scene_args)
# Save the images and differences.
pyredner.imwrite(img.cpu(), 'results/test_qmc/final.exr')
pyredner.imwrite(img.cpu(), 'results/test_qmc/final.png')
pyredner.imwrite(torch.abs(target - img).cpu(), 'results/test_qmc/final_diff.png')

# Convert the intermediate renderings to a video.
from subprocess import call
call(["ffmpeg", "-framerate", "24", "-i",
    "results/test_qmc/iter_%d.png", "-vb", "20M",
    "results/test_qmc/out.mp4"])

max_stars_repo_name: aferrall/redner

max_issues_repo_name: Awcrr/redner

max_forks_repo_name: Awcrr/redner


text:
#     Copyright 2016-present CERN – European Organization for Nuclear Research
#
#     Licensed under the Apache License, Version 2.0 (the "License");
#     you may not use this file except in compliance with the License.
#     You may obtain a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#     Unless required by applicable law or agreed to in writing, software
#     distributed under the License is distributed on an "AS IS" BASIS,
#     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#     See the License for the specific language governing permissions and
#     limitations under the License.

from unittest import TestCase

from numpy import sqrt, array
from pandas import date_range, Timedelta, DatetimeIndex

from qf_lib.common.enums.frequency import Frequency
from qf_lib.common.utils.dateutils.get_values_common_dates import get_values_for_common_dates
from qf_lib.common.utils.dateutils.to_days import to_days
from qf_lib.common.utils.miscellaneous.annualise_with_sqrt import annualise_with_sqrt
from qf_lib.common.utils.miscellaneous.kelly import kelly_binary, kelly
from qf_lib.common.utils.miscellaneous.volume_weighted_average_price import volume_weighted_average_price
from qf_lib.common.utils.ratios.gain_to_pain_ratio import gain_to_pain_ratio
from qf_lib.common.utils.ratios.sharpe_ratio import sharpe_ratio
from qf_lib.common.utils.volatility.volatility_manager import VolatilityManager
from qf_lib.containers.dataframe.qf_dataframe import QFDataFrame
from qf_lib.containers.series.prices_series import PricesSeries
from qf_lib.containers.series.qf_series import QFSeries
from qf_lib.containers.series.simple_returns_series import SimpleReturnsSeries
from qf_lib_tests.helpers.testing_tools.containers_comparison import assert_series_equal, assert_dataframes_equal

class TestMiscellaneousUtils(TestCase):
    def setUp(self):
        self.return_dates = date_range('2015-01-01', periods=20, freq='D')
        self.test_returns = [0.01, 0.02, 0.03, 0.02, 0.01, 0.00, -0.01, -0.02, 0.01, 0.03, 0.05, 0.04, 0.03, 0.02, 0.01,
                             0.00, 0.01, 0.03, 0.02, 0.04]
        self.test_returns_tms = SimpleReturnsSeries(data=self.test_returns, index=self.return_dates, dtype=float)

        monthly_returns = [0.05, 0.03, -0.1, 0.2, -0.01, -0.01]
        monthly_return_dates = date_range('2015-01-01', periods=6, freq='M')
        self.monthly_ret_series = SimpleReturnsSeries(data=monthly_returns, index=monthly_return_dates, dtype=float)

    def test_annualise_with_sqrt(self):
        actual_annualised_values = annualise_with_sqrt(self.test_returns, frequency=Frequency.DAILY)
        expected_annualised_values = [i * sqrt(252) for i in self.test_returns]

        self.assertEqual(expected_annualised_values, actual_annualised_values)

    def test_exponential_average(self):
        actual_smoothed_series = self.test_returns_tms.exponential_average()

        expected_smoothed_values = [0.01, 0.0194, 0.029364, 0.02056184, 0.0106337104, 0.000638022624, -0.00936171864256,
                                    -0.019361703118554, 0.0082382978128868, 0.028694297868773, 0.048721657872126,
                                    0.040523299472328, 0.03063139796834, 0.0206378838781, 0.010638273032686,
                                    0.00063829638196116, 0.0094382977829177, 0.028766297866975, 0.020525977872019,
                                    0.038831558672321]
        expected_smoothed_series = SimpleReturnsSeries(data=expected_smoothed_values, index=self.return_dates.copy())

        assert_series_equal(expected_smoothed_series, actual_smoothed_series)

        actual_smoothed_series = self.test_returns_tms.exponential_average(lambda_coeff=1)
        expected_smoothed_series = self.test_returns_tms

        assert_series_equal(expected_smoothed_series, actual_smoothed_series)

    def test_to_days(self):
        expected = 3
        actual = to_days(Timedelta('3 days'))
        self.assertEqual(expected, actual)

    def test_get_volume_weighted_average_price(self):
        original_datetimes = date_range('2015-01-01', periods=12, freq='10Min')
        prices_tms = PricesSeries(data=[1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4], index=original_datetimes)
        volumes_tms = QFSeries(data=[1, 2, 3, 4, 1, 2, 4, 3, 2, 1, 3, 4], index=original_datetimes)
        interval = Timedelta('40 min')

        expected_datetimes = date_range('2015-01-01 00:40', periods=2, freq=interval)
        expected_avg_weighted_prices = [3, 2.9]
        expected_prices_tms = PricesSeries(data=expected_avg_weighted_prices, index=expected_datetimes)
        actual_prices_tms = volume_weighted_average_price(prices_tms, volumes_tms, interval)
        assert_series_equal(expected_prices_tms, actual_prices_tms, absolute_tolerance=0.0)

    def test_kelly(self):
        expected_kelly_criterion_value = 57.575757575758
        actual_kelly_criterion_value = kelly(self.test_returns_tms)
        self.assertAlmostEqual(expected_kelly_criterion_value, actual_kelly_criterion_value)

    def kelly_binary(self):
        expected_kelly_criterion_value = 2.2987
        p = 0.5801749
        a = 0.0265798
        b = -0.0320652
        actual_kelly_criterion_value = kelly_binary(p, a, b)
        self.assertAlmostEqual(expected_kelly_criterion_value, actual_kelly_criterion_value, places=4)

    def test_min_max_normalized(self):
        actual_normalized_tms = self.test_returns_tms.min_max_normalized()

        expected_normalized_values = array([3, 4, 5, 4, 3, 2, 1, 0, 3, 5, 7, 6, 5, 4, 3, 2, 3, 5, 4, 6]) / 7
        expected_normalized_tms = SimpleReturnsSeries(data=expected_normalized_values,
                                                      index=self.test_returns_tms.index)

        assert_series_equal(expected_normalized_tms, actual_normalized_tms)

    def test_sharpe_ratio(self):
        actual_sharpe_ratio = sharpe_ratio(self.test_returns_tms, frequency=Frequency.DAILY)
        expected_sharpe_ratio = 23.059159631739007
        self.assertAlmostEqual(expected_sharpe_ratio, actual_sharpe_ratio, places=10)

    def test_gain_to_pain_ratio(self):
        gain_to_pain = gain_to_pain_ratio(self.monthly_ret_series)
        expected_gain_to_pain = 1.33333333333
        self.assertAlmostEqual(expected_gain_to_pain, gain_to_pain, places=10)

    def test_get_values_for_common_dates(self):
        data = range(6)
        dates1 = DatetimeIndex(['2014-12-31', '2015-01-02', '2015-01-04', '2015-01-05', '2015-01-09', '2015-01-10'])
        dates2 = DatetimeIndex(['2015-02-01', '2015-01-02', '2015-01-03', '2015-01-04', '2015-01-05', '2015-01-10'])
        series1 = QFSeries(data=data, index=dates1, name='Series 1')
        series2 = QFSeries(data=data, index=dates2, name='Series 2')
        data_2d = array([data, data]).transpose()
        dataframe1 = QFDataFrame(data=data_2d, index=dates2, columns=['DataFrame Col. A', 'DataFrame Col. B'])

        expected_index = DatetimeIndex(['2015-01-02', '2015-01-04', '2015-01-05', '2015-01-10'])
        expected_data1 = [1, 2, 3, 5]
        expected_series1 = QFSeries(data=expected_data1, index=expected_index, name='Series 1')
        expected_data2 = [1, 3, 4, 5]
        expected_series2 = QFSeries(data=expected_data2, index=expected_index, name='Series 2')
        expected_dataframe = QFDataFrame(data=array([expected_data2, expected_data2]).transpose(), index=expected_index,
                                         columns=['DataFrame Col. A', 'DataFrame Col. B'])

        actual_series1, actual_series2, actual_dataframe = get_values_for_common_dates(series1, series2, dataframe1)

        assert_series_equal(expected_series1, actual_series1)
        assert_series_equal(expected_series2, actual_series2)
        assert_dataframes_equal(expected_dataframe, actual_dataframe)

    def test_volatility_manager(self):
        periods = 20
        returns = []
        for i in range(periods):
            ret = 0.01
            if i >= 10:
                ret = 0.02

            returns.append(ret * pow(-1, i))  # 1, -1, 1 ... -2, 2 -2 ... in %

        dates = date_range('2015-01-01', periods=periods, freq='D')
        series = SimpleReturnsSeries(data=returns, index=dates, name='Series 1')

        vol_manager = VolatilityManager(series)
        window_size = 5
        managed_series, weights_series = vol_manager.get_managed_series(vol_level=0.1, window_size=window_size, lag=1)

        ret = 0.005750
        self.assertAlmostEqual(abs(managed_series[window_size]), ret, places=5)
        self.assertAlmostEqual(abs(managed_series[window_size + 1]), ret, places=5)
        self.assertAlmostEqual(abs(managed_series[-2]), ret, places=5)
        self.assertAlmostEqual(abs(managed_series[-1]), ret, places=5)

    def test_volatility_manager_min_max_leverage(self):
        periods = 20
        returns = []
        for i in range(periods):
            ret = 0.005
            if i >= 10:
                ret = 0.1
            returns.append(ret * pow(-1, i))  # 0.5, -0.5, 0.5 ... -10, 10, -10 ... in %

        dates = date_range('2015-01-01', periods=periods, freq='D')
        series = SimpleReturnsSeries(data=returns, index=dates, name='Series 1')

        min_lev = 0.2
        max_lev = 2
        window_size = 5
        vol_manager = VolatilityManager(series)
        managed_series, weights_series = vol_manager.get_managed_series(vol_level=0.2, window_size=window_size, lag=1,
                                                                        min_leverage=min_lev, max_leverage=max_lev)

        self.assertAlmostEqual(abs(managed_series[window_size]), abs(max_lev * series[window_size]), places=5)
        self.assertAlmostEqual(abs(managed_series[-1]), abs(min_lev * series[-1]), places=5)

        self.assertAlmostEqual(abs(weights_series[window_size]), abs(max_lev), places=5)
        self.assertAlmostEqual(abs(weights_series[-1]), abs(min_lev), places=5)

reason: from numpy


text:
import numpy
numpy.set_printoptions(legacy='1.13')
import ast
N=int(input())

inp=[]
for i in range(N):
    n=[float(x) for x in input().split(" ")]
    inp.append(n)

arr=numpy.array(inp)
print(round(numpy.linalg.det(arr),2))

max_stars_repo_name: mahimaraghu/Hackerrank-Practice-Code

max_issues_repo_name: mahimaraghu/Hackerrank-Practice-Code

max_forks_repo_name: mahimaraghu/Hackerrank-Practice-Code


text:
#! /usr/local/bin/python3
# Must have `numpy`, `tensorflow`, `keras`, and `zeromq` pip installed
from keras import optimizers
from keras.layers.advanced_activations import LeakyReLU, PReLU
from keras import losses
from keras.models import Sequential
from keras.layers import Dense
from keras import backend as K
import numpy as np
import random
import zmq

numEpochs = 10000; epochsPerIteration = 50; curEpoch = 0
context = zmq.Context()
socket = context.socket(zmq.PAIR)
socket.bind("tcp://*:12345")
socket.SNDTIMEO = 1000
socket.RCVTIMEO = 1000
socket.LINGER = 0

# fix random seed for reproducibility
np.random.seed(7)

# create and compile model
XY = [[], []]
model = Sequential()
model.add(Dense(16, input_dim=10, activation="linear"))
model.add(LeakyReLU(alpha=0.3))
model.add(Dense(16, activation="linear"))
model.add(LeakyReLU(alpha=0.3))
model.add(Dense(16, activation="linear"))
model.add(LeakyReLU(alpha=0.3))
model.add(Dense(10, activation="linear"))

model.compile(loss="mse", optimizer="adam", metrics=['accuracy'])

# the main training/simulation loop
while (curEpoch < numEpochs):
    #trigger Unity to generate a set of training samples
    socket.send_string("s")

    #split/parse received data into matrices and fit the network to it (skip if only received prediction query)
    data = str(socket.recv())[2:-1]
    dataStrings = data.split(";")
    if(len(dataStrings) > 1):
        XY[0] = []; XY[1] = []
        for i in range(len(dataStrings)-1):
            sampleStrings = dataStrings[i].split(",")
            XY[0].append([float(sampleStrings[0]),  float(sampleStrings[1]),  float(sampleStrings[2]),  float(sampleStrings[3]),  float(sampleStrings[4]),  float(sampleStrings[5]),  float(sampleStrings[6]),  float(sampleStrings[7]),  float(sampleStrings[8]),  float(sampleStrings[9])])
            XY[1].append([float(sampleStrings[10]), float(sampleStrings[11]), float(sampleStrings[12]), float(sampleStrings[13]), float(sampleStrings[14]), float(sampleStrings[15]), float(sampleStrings[16]), float(sampleStrings[17]), float(sampleStrings[18]), float(sampleStrings[19])])

        #fit the model for a few epochs
        model.fit(np.asarray(XY[0]), np.asarray(XY[1]), epochs=epochsPerIteration, batch_size=128, verbose=1)# initial_epoch = curEpoch)
        curEpoch += epochsPerIteration

    #visualize the prediction from the last sample
    visualizationStrings = dataStrings[-1].split(",")
    prediction = model.predict(np.asarray([[float(visualizationStrings[0]), float(visualizationStrings[1]), float(visualizationStrings[2]), float(visualizationStrings[3]), float(visualizationStrings[4]), float(visualizationStrings[5]), float(visualizationStrings[6]), float(visualizationStrings[7]), float(visualizationStrings[8]), float(visualizationStrings[9])]]))
    socket.send_string("v " + str(prediction[0][0]) + "," + str(prediction[0][1]) + "," + str(prediction[0][2]) + "," + str(prediction[0][3]) + "," + str(prediction[0][4]) + "," + str(prediction[0][5]) + "," + str(prediction[0][6]) + "," + str(prediction[0][7]) + "," + str(prediction[0][8]) + "," + str(prediction[0][9]))

max_stars_repo_name: swordlegend/MathUtilities

max_issues_repo_name: swordlegend/MathUtilities

max_forks_repo_name: swordlegend/MathUtilities


text:
import numpy as np

# t axis in seconds
(t, deltat) = np.linspace(-10, 10, 5001, retstep=True)

# Elementary signals
gauss     = lambda t: np.exp(-(t)**2)
unitstep  = lambda t: np.where(t>=0, 1, 0)
rect      = lambda t: unitstep(t+0.5) - unitstep(t-0.5)
tri       = lambda t: rect(t/2)*(1-abs(t))
si        = lambda t: np.sinc(t/np.pi) # English notation sinc(t) = sin(pi t)/(pi t)
rc_tp     = lambda f: 1 / (1 + 1j * 2 * np.pi * f) # RC = 1
rl_hp     = lambda f: (1j * 2 * np.pi * f) / (1 + 1j * 2 * np.pi * f) # RL = 1

eps = np.finfo(float).eps # floating point accuracy

def findIndOfLeastDiff(x, x0):
    """
    Find Index of the value that's nearest to x0

    Parameters
    ----------
    x : array_like
        Array to be inspected

    x0 : digit
        Target value to be searched for

    Returns
    -------
    index : int or list
        index(indices) of value(s) that is (are) nearest to x0
    """
    if isinstance(x0, list) or isinstance(x0, np.ndarray):
        return list(map(lambda t0: np.abs(x - t0).argmin(), x0))
    else:
        return np.abs(x - x0).argmin()


text:
from __future__ import print_function
import sys
sys.path.append('..')
from Game import Game
from .CheckersLogic import Board
import numpy as np

class CheckersGame(Game):
    square_content = {
        -2: "V",
        -1: "X",
        +0: "-",
        +1: "O",
        +2: "D"
    }

    @staticmethod
    def getSquarePiece(piece):
        return CheckersGame.square_content[piece]

    def __init__(self, n):
        self.n = n
        self.count = 0

    def getInitBoard(self):
        # return initial board (numpy board)
        b = Board(self.n)
        return (np.array(b.pieces), b.count)

    def getBoardSize(self):
        # (a,b) tuple
        return (self.n, self.n)

    def getActionSize(self):
        # return number of actions
        # 2 possible moves per each of the 4 directions, starting at n*n/2 possible positions
        # (i,j) position ->
        return self.n*self.n*4+1

    def getNextState(self, board, player, action):
        # if player takes action on board, return next (board,player)
        # action must be a valid move
        if action == self.n*self.n*4:   # When you don't move
            return ((board[0], board[1]+1), -player)

        b = Board(self.n)
        b.pieces = np.copy(board[0])
        b.count = board[1]

        move = self.action2move(action)
        if player == -1:
            (x,y),(z,w) = move
            move = ((self.n-1-x, self.n-1-y),(-z, -w))
        b.execute_move(move, player)
        return ((b.pieces, board[1]+1), -player)

    '''
    Move direction represented by action%8:
    4   5
     0 1
      X
     2 3
    6   7
    Coordinate of X: (i,j) -> action = (n//2*i + j//2) * 8 + (action%8)
    '''

    def getValidMoves(self, board, player):
        # return a fixed size binary vector
        valids = [0]*self.getActionSize()
        b = Board(self.n)
        b.pieces = np.copy(board[0])
        b.count = board[1]
        legalMoves =  b.get_legal_moves(player)
        if len(legalMoves)==0:
            valids[-1]=1
            return np.array(valids)
        for ((x,y),(z,w)) in legalMoves:
            valids[self.move2action(x,y,z,w)]=1
        return np.array(valids)

    def getGameEnded(self, board, player):
        # return 0 if not ended, 1 if player 1 won, -1 if player 1 lost
        # player = 1
        b = Board(self.n)
        b.pieces = np.copy(board[0])
        b.count = board[1]
        if board[1] > 150:
            return 0.01
        return b.game_over()

    def getCanonicalForm(self, board, player):
        # return state if player==1, else return -state if player==-1
        if player == 1:
            return board
        else:
            newB = np.copy(board[0])
            newB = -np.flip(newB, [0,1])
            return (newB, board[1])

    def getSymmetries(self, board, pi):
        # LR mirror only
        assert(len(pi) == self.getActionSize())  # 1 for pass
        l = [(CheckersGame.encodeBoard(board), pi)]

        newB = np.flip(np.copy(board[0]), 1)
        newPi = [0]*self.getActionSize()
        for i in range(self.getActionSize()-1):
            if i%2 == 0:
              newPi[i] = pi[i+1]
            else:
              newPi[i] = pi[i-1]
        newPi[self.getActionSize()-1] = pi[self.getActionSize()-1]
        l += [(CheckersGame.encodeBoard((newB, board[1])), newPi)]
        return l

    def stringRepresentation(self, board):
        return CheckersGame.encodeBoard(board).tostring()
        #return ','.join(str(item) for innerlist in board for item in innerlist)

    def stringRepresentationReadable(self, board):
        board_s = "".join(self.square_content[square] for row in board[0] for square in row)
        return board_s

    def getScore(self, board, player):
        b = Board(self.n)
        b.pieces = np.copy(board[0])
        b.count = board[1]
        return b.countScore(player)

    @staticmethod
    def display(board):
        n = board[0].shape[0]
        print("-----------------")
        for y in range(n):
            print("|"+str(n-1-y)+" |", end="")    # print the row #
            for x in range(n):
                piece = board[0][y][n-1-x]    # get the piece to print
                print(CheckersGame.square_content[piece], end=" ")
            print("|")

        print("-----------------")
        print("   |", end="")
        for y in range(n):
            print(y, end=" ")
        print("|")

        print("   --------------")

    @staticmethod
    def encodeBoard(board):
        b, move_cnt = board
        b1 = np.copy(b) == 1
        b2 = np.copy(b) == 2
        b3 = np.copy(b) == -1
        b4 = np.copy(b) == -2

        n = b.shape[0]

        mc = [None]*n
        for i in range(n):
            mc[i] = [board[1]]*n
        mc = np.array(mc)

        res = np.concatenate((b1, b2, b3, b4, mc), 0)

        return res

    def action2move(self, action):
        multiplier = (action//4)%2+1
        xval = (action//8)//(self.n//2)
        yval = (action//8)%(self.n//2)*2 + xval%2
        return ((xval, yval), ((-1+2*(action%2))*multiplier, (-1+2*((action%4)//2))*multiplier))

    def move2action(self, x, y, z, w):
        direction = (z+2*w+3)//2 if abs(z) == 1 else (z//2+w+3)//2+4
        return (self.n//2*x + y//2) * 8 + direction

max_stars_repo_name: totolacky/alpha-zero-general

max_issues_repo_name: totolacky/alpha-zero-general

max_forks_repo_name: totolacky/alpha-zero-general


text:
import sys
import os
import csv
import pathlib
import numpy as np
import pandas as pd
import pyqtgraph as pg
from PyQt5 import QtWidgets as qtw
from PyQt5 import QtCore as qtc
from PyQt5 import QtGui as qtg
from PyQt5 import uic
import pyqtgraph.exporters

import matplotlib
from matplotlib.figure import Figure
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg

from about import Ui_Form
from layout3 import Ui_MainWindow
matplotlib.use('Qt5Agg')

class About(qtw.QWidget):
    def __init__(self):
        super().__init__()
        self.ui = Ui_Form()
        self.ui.setupUi(self)

        self.show()

class MplCanvas(FigureCanvasQTAgg):

    def __init__(self, parent=None, width=5, height=4, dpi=100):
        fig = Figure(figsize=(width, height), dpi=dpi)
        self.axes = fig.add_subplot(111)
        super(MplCanvas, self).__init__(fig)

class MainWindow(qtw.QMainWindow):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.ui = Ui_MainWindow()
        self.ui.setupUi(self)

        self.filenames = ['', '', '']
        self.graphChannels = [self.ui.signal1Graph,
                              self.ui.signal2Graph, self.ui.signal3Graph]
        self.spectrogramChannels = [
            self.ui.spectrogram1Graph, self.ui.spectrogram2Graph, self.ui.spectrogram3Graph]
        self.timers = [self.ui.timer1, self.ui.timer2, self.ui.timer3]
        self.pen = [pg.mkPen(color=(255, 0, 0), width=2), pg.mkPen(
            color=(0, 255, 0), width=2), pg.mkPen(color=(0, 0, 255), width=2)]

        self.y = [[], [], []]
        self.x = [[], [], []]

        self.data_line = [[], [], []]
        self.data = [[], [], []]
        self.pointsToAppend = [0, 0, 0]
        self.isResumed = [False, False, False]
        self.setChannelChecked = [self.ui.showChannel1,
                                  self.ui.showChannel2, self.ui.showChannel3]
        self.channelComponents = [self.ui.channel1,
                                  self.ui.channel2, self.ui.channel3]

        self.CHANNEL1 = 0
        self.CHANNEL2 = 1
        self.CHANNEL3 = 2

        self.ui.showChannel1.setChecked(True)
        self.ui.channel1.show()
        self.ui.channel2.hide()
        self.ui.channel3.hide()

        self.ui.showChannel1.stateChanged.connect(
            lambda: self.hide(self.CHANNEL1))
        self.ui.showChannel2.stateChanged.connect(
            lambda: self.hide(self.CHANNEL2))
        self.ui.showChannel3.stateChanged.connect(
            lambda: self.hide(self.CHANNEL3))

        self.ui.actionAbout.triggered.connect(lambda: self.showAbout())
        self.ui.actionExit.triggered.connect(lambda: self.close())
        self.ui.actionNew.triggered.connect(lambda: self.create_new_window())

        self.ui.actionOpenChannel1.triggered.connect(
            lambda: self.browse(self.CHANNEL1))
        self.ui.playBtn1.clicked.connect(lambda: self.play(self.CHANNEL1))
        self.ui.pauseBtn1.clicked.connect(lambda: self.pause(self.CHANNEL1))
        self.ui.focusBtn1.clicked.connect(
            lambda: self.graphChannels[self.CHANNEL1].getPlotItem().enableAutoRange())
        self.ui.zoomInBtn1.clicked.connect(lambda: self.zoomin(self.CHANNEL1))
        self.ui.zoomOutBtn1.clicked.connect(
            lambda: self.zoomout(self.CHANNEL1))
        self.ui.clearBtn1.clicked.connect(lambda: self.clear(self.CHANNEL1))

        self.ui.actionOpenChannel2.triggered.connect(
            lambda: self.browse(self.CHANNEL2))
        self.ui.playBtn2.clicked.connect(lambda: self.play(self.CHANNEL2))
        self.ui.pauseBtn2.clicked.connect(lambda: self.pause(self.CHANNEL2))
        self.ui.focusBtn2.clicked.connect(
            lambda: self.graphChannels[self.CHANNEL2].getPlotItem().enableAutoRange())
        self.ui.zoomInBtn2.clicked.connect(lambda: self.zoomin(self.CHANNEL2))
        self.ui.zoomOutBtn2.clicked.connect(
            lambda: self.zoomout(self.CHANNEL2))
        self.ui.clearBtn2.clicked.connect(lambda: self.clear(self.CHANNEL2))

        self.ui.actionOpenChannel3.triggered.connect(
            lambda: self.browse(self.CHANNEL3))
        self.ui.playBtn3.clicked.connect(lambda: self.play(self.CHANNEL3))
        self.ui.pauseBtn3.clicked.connect(lambda: self.pause(self.CHANNEL3))
        self.ui.focusBtn3.clicked.connect(
            lambda: self.graphChannels[self.CHANNEL3].getPlotItem().enableAutoRange())
        self.ui.zoomInBtn3.clicked.connect(lambda: self.zoomin(self.CHANNEL3))
        self.ui.zoomOutBtn3.clicked.connect(
            lambda: self.zoomout(self.CHANNEL3))
        self.ui.clearBtn3.clicked.connect(lambda: self.clear(self.CHANNEL3))

        self.show()

    def showAbout(self) -> None:
        self.about = About()
        self.about.show()

    def play(self, channel: int) -> None:
        if not self.y[channel]:
            self.browse(channel)
        if not self.isResumed[channel]:
            self.timers[channel].start()
            self.isResumed[channel] = True

    def pause(self, channel: int) -> None:
        if not self.y[channel]:
            self.browse(channel)
        if self.isResumed[channel]:
            self.timers[channel].stop()
            self.isResumed[channel] = False

    def clear(self, channel: int) -> None:
        if self.y[channel]:
            self.graphChannels[channel].removeItem(self.data_line[channel])
            self.timers[channel].stop()
            self.isResumed[channel] = False
            self.y[channel] = []
            self.x[channel] = []
            self.data_line[channel] = []

    def hide(self, channel: int) -> None:
        if(self.channelComponents[channel].isVisible()):
            self.channelComponents[channel].hide()
            self.setChannelChecked[channel].setChecked(False)
            self.clear(channel)
        else:
            self.setChannelChecked[channel].setChecked(True)
            self.channelComponents[channel].show()

    def create_new_window(self):
        self.newWindow = MainWindow()
        self.newWindow.show()

    def browse(self, channel: int) -> None:
        self.hide(channel)

        self.clear(channel)
        self.filenames[channel] = qtw.QFileDialog.getOpenFileName(
            None, 'Load Signal', './', "Raw Data(*.csv *.txt *.xls)")
        path = self.filenames[channel][0]
        self.openFile(path, channel)

    def openFile(self, path: str, channel: int) -> None:
        with open(path, 'r') as csv_file:
            csv_reader = csv.reader(csv_file, delimiter=',')
            for line in csv_reader:
                self.y[channel].append(float(line[1]))
                self.x[channel].append(float(line[0]))
        self.isResumed[channel] = True

        self.plotGraph(channel)
        # self.plotSpectrogram(channel)

    def plotGraph(self, channel: int) -> None:
        self.data_line[channel] = self.graphChannels[channel].plot(
            self.x[channel], self.y[channel], name='CH1', pen=self.pen[channel])
        self.graphChannels[channel].plotItem.setLimits(
            xMin=0, xMax=6, yMin=-3, yMax=3)

        # self.exporter = pg.exporters.ImageExporter(
        #     self.data_line[channel].plotItem)
        # self.exporter.parameters()['width'] = 100
        # self.exporter.export('filename.png')

        self.pointsToAppend[channel] = 0
        self.timers[channel].setInterval(150)
        self.timers[channel].timeout.connect(lambda: self.updatePlot(channel))
        self.timers[channel].start()

    def updatePlot(self, channel: int) -> None:
        xaxis = self.x[channel][:self.pointsToAppend[channel]]
        yaxis = self.y[channel][:self.pointsToAppend[channel]]
        self.pointsToAppend[channel] += 20
        if self.pointsToAppend[channel] > len(self.y[channel]):
            self.timers[channel].stop()
            return
        else:
            if self.x[channel][self.pointsToAppend[channel]] > 1.0:
                self.graphChannels[channel].setLimits(xMin=min(xaxis, default=0), xMax=max(
                    xaxis, default=0), yMin=-3, yMax=3)
            self.graphChannels[channel].plotItem.setXRange(
                max(xaxis, default=0)-1.0, max(xaxis, default=0))

            self.data_line[channel].setData(xaxis, yaxis)

    # def plotSpectrogram(self, channel: int) -> None:
    #     sc = MplCanvas(
    #         self, width=5, height=4, dpi=100)
    #     sc.axes.plot([0, 1, 2, 3, 4], [10, 1, 20, 3, 40])
    #     self.setCentralWidget(sc)
    #     self.data_line[channel] = self.graphChannels[channel].plot(
    #         list(range(self.y[channel].size)), self.y[channel], name='CH1', pen=self.pen[channel])

    def zoomin(self, channel: int) -> None:
        self.graphChannels[channel].plotItem.getViewBox().scaleBy((0.75, 0.75))

    def zoomout(self, channel: int) -> None:
        self.graphChannels[channel].plotItem.getViewBox().scaleBy((1.25, 1.25))

if __name__ == '__main__':
    app = qtw.QApplication(sys.argv)
    mw = MainWindow()
    sys.exit(app.exec_())


text:
import shutil, subprocess
import numpy as np
import re
import sys, os

def parseConfigFindPath(stringFind,configFile):
    """findPath will find path of associated specified string or info from config file"""
    for line in configFile:
        if stringFind in line: # if find string specified, return pathname
            configFile.seek(0)
            return line.split()[-1].strip('\n')
configFile = open('configCNSAnalysis.txt','r')
rootfolder = parseConfigFindPath('root_folder',configFile) # grab root folder for analysis
pathPython = parseConfigFindPath('pathPython',configFile) # grab python path
configFile.close()

sys.path.append(pathPython) # add python modules

# conditionals, will take conditionals in paths, and intersect and differentiate them in different ways to find following combos
# this can find one or two copy (NK) sequences for intra vs inter genus conservation (DS00 or DS11)
outFname = ['NKHDS11111.txt','NKHDS01111.txt','NKHDS10111.txt','NKHDS11100.txt','NKHDS01100.txt','NKHDS10100.txt','NKH111.txt','NKH011.txt','NKH101.txt','HDS111.txt']
paths = ['%s/CS/NKH111/'%rootfolder,
         '%s/CS/NKH011/'%rootfolder,
         '%s/CS/NKH101/'%rootfolder,
         '%s/CS/HDS111/'%rootfolder]

def bed2dict(bedfname): # create a dictionary that has MA Segments as keys
    bedDict={}
    for line in open(bedfname,'r'):
        if line:
            MASeg = line.split()[-1].split(';')[2].replace('SegmentID=','')
            if bedDict.has_key(MASeg) and tuple(line.split()[0:3]) not in bedDict[MASeg]:
                bedDict[MASeg].append(tuple(line.split()[0:3]))
            else:
                bedDict[MASeg] = [tuple(line.split()[0:3])]
    return bedDict

def grabPartMAFSeq(MAFSeq,x_coords): # irrelevant script now...
    coordtransform = np.array(range(len(MAFSeq)))
    coordtransform = np.delete(coordtransform,[val.start(0) for val in re.finditer('-',MAFSeq)])
    print coordtransform, coordtransform[x_coords[1]-1], x_coords[1]
    return MAFSeq[coordtransform[x_coords[0]]:coordtransform[x_coords[1]-1]]

listALLFiles = str(subprocess.Popen('ls', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                   .stdout.read()).split('\n')

ultimateBedDicts = {'Conserved_CDS.bed':{},'CS_Minus_CDS.bed':{}} # split into ~CDS or ~CNS

for file in listALLFiles:
    if file.endswith('Conserved_CDS.bed') or file.endswith('CS_Minus_CDS.bed'):
        ultimateBedDicts[file[file.find('_')+1:]][file[:file.find('_')]] = bed2dict(file) # a dictionary with keys [species] and inside a key of MASEG
        # hierarchy is ultimate[CDS or CNS][species][MASeg]

def concatFastas(fname,HDSPath,NKHPath,FinalPath): # if intergenus conservation, concatenate fastas together and add it to relevant path
    NKHFileStringList = open(NKHPath+fname,'r').read().split('>')
    HDSFileStringList = open(HDSPath+fname,'r').read().split('>')
    outputStringList = list(set(NKHFileStringList+HDSFileStringList))
    outputFile = open(FinalPath+fname,'w')
    outputFile.write('>'.join(seqs for seqs in outputStringList))
    outputFile.close()

def grabConservedSeq(PathAndfname,conservedSeqs): # build a list of conserved sequences that are either Conserved sequence,
    # protein coding, nonprotein, or mixed and will join list together
    inputFastaReadSegments = open(PathAndfname,'r').read().split('>')
    for sequence in inputFastaReadSegments:
        if sequence:
            speciesName = sequence.split('.')[0]
            if 'K' == sequence.split('_')[0][-1]: #FIXME cahnge _ to -
                speciesName += 'K'
            if 'N' == sequence.split('_')[0][-1]:
                speciesName += 'N'
            if conservedSeqs.has_key(speciesName):
                conservedSeqs[speciesName].append(sequence.split('\n')[1].strip('\n')) # add sequence to main list to generate main fasta file output
            else:
                conservedSeqs[speciesName] = [sequence.split('\n')[1].strip('\n')]
    return conservedSeqs

for analysisType in ['CS','ProteinCoding_CS','Mixed_CS','NonProteinCodingN_CS']: # run analysis on these types and sort them into relevant conditionals and output fastas for PhyML analysis and conservation ratios
    # see more details about ratio conservation in various documents...
    TwoCopyStatNumbers = {}
    analysisPath = paths[0][:-1][:paths[0][:-1].rfind('/')+1]#.replace('CS',analysisType)
    outFname = ['NKHDS11111.txt','NKHDS01111.txt','NKHDS10111.txt','NKHDS11100.txt','NKHDS01100.txt','NKHDS10100.txt','NKH111.txt','NKH011.txt','NKH101.txt','HDS111.txt']
    if analysisType == 'CS':
        analysisType2 = ''
        arrayFiles = []
        for path in paths:
            arrayFiles.append(str(subprocess.Popen(['ls', '%s' % path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                    .stdout.read()).split('\n'))#.replace('CS',analysisType)
        #print analysisPath
        for i in range(3):
            np.savetxt(analysisPath+outFname[i].replace('.txt','_%s.txt'%analysisType),np.intersect1d(arrayFiles[3],arrayFiles[i]),fmt='%s',comments='')
            np.savetxt(analysisPath+outFname[i+3].replace('.txt','_%s.txt'%analysisType),np.setdiff1d(arrayFiles[i],arrayFiles[3]),fmt='%s',comments='')
            np.savetxt(analysisPath+outFname[i+6].replace('.txt','_%s.txt'%analysisType),arrayFiles[i],fmt='%s',comments='')

        np.savetxt(analysisPath+outFname[9].replace('.txt','_%s.txt'%analysisType),arrayFiles[3],fmt='%s',comments='')
        # the above script finds the intersection and differences
    else:
        analysisType2 = analysisType
        for fname in outFname[0:6]:
            outFile = open(analysisPath+fname.replace('.txt','_%s.txt'%analysisType),'w')
            outFile.write(str(subprocess.Popen('ls %s/%s/%s | grep "%s"'%(rootfolder,'CS',fname.replace('.txt',''),
                    analysisType),shell=True , stdout=subprocess.PIPE).stdout.read()))
            outFile.close() # the CS analysis separates NHK files into various NKHDS folders, now they are named according to type of analysis

    if analysisType == 'CS':
        # if in CS, sort NKH files into various NKHDS folders according to conditional statements and which files fall under each conditional
        for fname in outFname[0:6]:
            inputFile = open(analysisPath+fname.replace('.txt','_%s.txt'%analysisType),'r')
            conditionalState = fname.split('.')[0].strip('NKHDS')
            if len(conditionalState) == 5 and conditionalState.endswith('11'):
                HDSpath = '%s/%s/HDS111/'%(rootfolder,'CS')
                NKHpath = '%s/%s/NKH%s/'%(rootfolder,'CS',conditionalState[0:3])
                finalPath = '%s/%s/NKHDS%s/'%(rootfolder,'CS',conditionalState)
                for line in inputFile:
                    if line and line != '\n' and ':' not in line and '/' not in line:
                        inputNKHDSfname = line.strip('\n')
                        concatFastas(inputNKHDSfname,HDSpath,NKHpath,finalPath)
            elif len(conditionalState) == 5 and conditionalState.endswith('00'):
                NKHpath = '%s/%s/NKH%s/'%(rootfolder,'CS',conditionalState[0:3])
                finalPath = '%s/%s/NKHDS%s/'%(rootfolder,'CS',conditionalState)
                for line in inputFile:
                    if line and line != '\n' and ':' not in line and '/' not in line:
                        inputNKHDSfname = line.strip('\n')
                        #print line, fname.replace('.txt','_%s.txt'%analysisType)
                        #print NKHpath, inputNKHDSfname, finalPath
                        shutil.copy(NKHpath+inputNKHDSfname,finalPath)
            inputFile.seek(0)
            inputFile.close()
    for fname in outFname[0:6]: # compute twocopy statistics (ratio) between NKHDS, see output text
        # output final fasta for NKHDS conditional type and analysis type seen above, which generate trees for each
        fileArray2 = open(analysisPath+fname.replace('.txt','_%s.txt'%analysisType),'r').read().splitlines()
        speciesRatios = fname.replace('.txt','')
        TwoCopyStatNumbers[speciesRatios] = float(len(fileArray2))
        conservedSeqs = {}
        for file in fileArray2:
            if file and ':' not in file and '/' not in file:
                try:
                    conservedSeqs = grabConservedSeq(analysisPath+speciesRatios+'/'+file,conservedSeqs)
                except:
                    print analysisType
        outputFinalFasta = fname.replace('.txt','_%s.fasta'%analysisType)
        open(analysisPath+outputFinalFasta,'w').close()
        outputFinalFastaFile = open(analysisPath+outputFinalFasta, 'w')
        for species in conservedSeqs.keys():
            outputFinalFastaFile.write('>%s\n%s\n'%(species,''.join(conservedSeq for conservedSeq in conservedSeqs[species])))
        outputFinalFastaFile.close()

    ratio2copyStat = [TwoCopyStatNumbers['NKHDS11111']/(TwoCopyStatNumbers['NKHDS10111']+TwoCopyStatNumbers['NKHDS01111']+TwoCopyStatNumbers['NKHDS11111']),
                      TwoCopyStatNumbers['NKHDS11100'] / (TwoCopyStatNumbers['NKHDS10100'] + TwoCopyStatNumbers['NKHDS01100']+TwoCopyStatNumbers['NKHDS11100'])]
    open(analysisPath+'Twocopyratios_%s.txt'%analysisType,'w').close()
    twoCopyFile = open(analysisPath+'Twocopyratios_%s.txt'%analysisType,'w')
    twoCopyFile.write('Intergenus Conservation [NKHDS11111/(NKHDS11111+NKHDS10111+NKHDS01111)]: %f\n'
                      'Intragenus Conservation [NKHDS11100/(NKHDS11100+NKHDS10100+NKHDS01100)]: %f\n'%(tuple(ratio2copyStat)))
    twoCopyFile.close()
    if analysisType == 'CS':
        for path in ['%s/%s/%s/'%(rootfolder,'CS',fname.replace('.txt','')) for fname in outFname[0:6]]:
            for file in str(subprocess.Popen(['ls', path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                                    .stdout.read()).split('\n'):
                if file and file.endswith('.fasta'):
                    """>Pvirgatum.Pvirgatum.Chr01K_14565140_14836634_328_23_+
                    AGACAGCTAACAGCTTACTTTCT
                    >Phallii.Phallii.Chr03_5677168_5837565_290_23_+
                    AGACAGCTAACAGCTTACTTTCT
                    """
                    inputFasta = open(path + file, 'r')  # 011
                    MASeg = file.split('.')[0]
                    speciesList = []
                    for line in inputFasta:
                        if '>' in line:
                            lineList = line[1:].split('.')
                            speciesName = lineList[0]
                            lineList2 = lineList[-1].split('_')
                            scaffold_name = lineList[-1][:lineList[-1].find(lineList2[-5]) - 1]
                            # orientation = lineList2[-1].strip('\n')
                            if 'K' == scaffold_name[-1]:
                                speciesName += 'K'
                            if 'N' == scaffold_name[-1]:
                                speciesName += 'N'
                            """
                            if orientation == '+':
                                x = (int(lineList2[-5])+int(lineList2[-3]),int(lineList2[-5])+int(lineList2[-3])+int(lineList2[-2]))
                                print (scaffold_name,),x
                            else:
                                x = (int(lineList2[-4])-int(lineList2[-3])-int(lineList2[-2]),int(lineList2[-4])-int(lineList2[-3]))
                                print (scaffold_name,),x
                            """
                        else:
                            speciesList.append(speciesName)
                            # for key in ultimateBedDicts.keys():
                            #    if ultimateBedDicts[key][speciesName].has_key(MASeg):
                            #        for CNSCDS in ultimateBedDicts[key][speciesName][MASeg]:
                            # xRel = tuple(np.vectorize(lambda y: int(y))(CNSCDS[1:])-x[0])
                            # print ultimateBedDicts[key][speciesName][MASeg],conservedSeg, len(conservedSeg), xRel,grabPartMAFSeq(conservedSeg,xRel)
                    CDS_conditional = np.vectorize(
                        lambda speciesName: ultimateBedDicts['Conserved_CDS.bed'][speciesName].has_key(MASeg))(speciesList)
                    # in each folder for the NKHDS conditionals, change file names to match relevant analysis, eg. whether
                    # sequence intersects a CDS, is not, or some do and some don't between different species
                    if np.all(CDS_conditional):
                        os.rename(path + file, path + file.replace('.fasta', '_ProteinCoding_CS.fasta'))
                    elif np.any(CDS_conditional) and not np.all(CDS_conditional):
                        os.rename(path + file, path + file.replace('.fasta', '_Mixed_CS.fasta'))
                    elif not np.any(CDS_conditional):
                        os.rename(path + file, path + file.replace('.fasta', '_NonProteinCodingN_CS.fasta'))
                    inputFasta.close()


text:
import argparse
import random
from pathlib import Path

import chainer
import chainer.functions as F
import cupy
import numpy as np
import pandas as pd
from chainer import function, iterators, serializers
from chainer.training import extensions
from tqdm import tqdm, trange

from dataloader import KMNIST
from models.modified_classifier import ModifiedClassifier
from models.se_res2net import SERes2Net50
from models.se_resnext import SEResNeXt50, SEResNeXt101

def main():
    parser = argparse.ArgumentParser(description='training mnist')
    parser.add_argument('--gpu', '-g', default=-1, type=int,
                        help='GPU ID (negative value indicates CPU)')
    parser.add_argument('--batchsize', '-b', type=int, default=100,
                        help='Number of images in each mini-batch')
    parser.add_argument('--n_tta', type=int, default=1,
                        help='Number of test time augmentations')
    parser.add_argument('--seed', '-s', type=int, default=0,
                        help='Random seed')
    parser.add_argument('--save_prob', action='store_true',
                        help='Save model prediction probabilities')
    args = parser.parse_args()

    random.seed(args.seed)
    np.random.seed(args.seed)
    cupy.random.seed(args.seed)
    chainer.config.cudnn_deterministic = True

    resize = [('Resize', {'p': 1.0, 'height': 32, 'width': 32})]

    models = [
        ('SEResNeXt101', None,
         ['output/se_resnext101_28_fold_1/best_model.npz',
          'output/se_resnext101_28_fold_2/best_model.npz',
          'output/se_resnext101_28_fold_3/best_model.npz',
          'output/se_resnext101_28_fold_4/best_model.npz',
          'output/se_resnext101_28_fold_5/best_model.npz']),
        ('SEResNeXt101', resize,
         ['output/se_resnext101_32_fold_1/best_model.npz',
          'output/se_resnext101_32_fold_2/best_model.npz',
          'output/se_resnext101_32_fold_3/best_model.npz',
          'output/se_resnext101_32_fold_4/best_model.npz',
          'output/se_resnext101_32_fold_5/best_model.npz']),
    ]

    xp = cupy if args.gpu >= 0 else np
    batchsize = args.batchsize
    n_tta = args.n_tta
    preds = []
    pbar1 = tqdm(models, ncols=100, leave=False)

    for model_name, augmentation, model_pathes in pbar1:
        pbar1.set_postfix(model_name=model_name)
        test_data = KMNIST(augmentation=augmentation, train=False)
        test_iter = iterators.SerialIterator(test_data, batchsize,
                                             repeat=False, shuffle=False)
        n_data = len(test_data)
        model = ModifiedClassifier(eval(model_name)())

        pbar2 = tqdm(model_pathes, ncols=100, leave=False)
        for model_path in pbar2:
            pbar2.set_postfix(model_path=model_path)
            serializers.load_npz(model_path, model)
            if args.gpu >= 0:
                chainer.backends.cuda.get_device_from_id(args.gpu).use()
                model.to_gpu()

            pbar3 = trange(n_tta, ncols=100, leave=False)
            for i in pbar3:
                pbar3.set_postfix(tta=i+1)
                test_iter.reset()

                pred = []
                with chainer.no_backprop_mode(), chainer.using_config(
                        'train', False):
                    pbar4 = trange(
                        n_data // batchsize + (n_data % batchsize != 0),
                        ncols=100, leave=False)
                    for j in pbar4:
                        batch = test_iter.next()
                        pbar4.set_postfix(data=batchsize * (j + 1))
                        x = xp.array(batch)
                        y = F.softmax(model.predictor(x))
                        pred.extend(chainer.backends.cuda.to_cpu(y.data))
                    pbar4.close()
                preds.append(np.array(pred))
            pbar3.close()
        pbar2.close()
    pbar1.close()

    preds = np.array(preds)
    if preds.ndim > 2:
        preds = np.mean(preds, axis=0)

    result = pd.DataFrame(columns=['ImageId', 'Label'])
    result.ImageId = np.arange(len(test_data)) + 1
    result.Label = np.argmax(preds, axis=1)
    if args.save_prob:
        result['Prob'] = np.max(preds, axis=1)
    result.to_csv('results.csv', index=False)

if __name__ == '__main__':
    main()

max_stars_repo_name: diceroll/kmnist

max_issues_repo_name: diceroll/kmnist

max_forks_repo_name: diceroll/kmnist

reason: import numpy,import cupy