{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab Part II: RNN Sentiment Classifier\n",
    "\n",
    "In the previous lab, you built a tweet sentiment classifier with a simple feedforward neural network. Now we ask you to improve this model by representing it as a *sequence* of words, with a recurrent neural network.\n",
    "\n",
    "First import some things:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import pickle as p\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import utils\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model will be like this:\n",
    "\n",
    "![alt-text](lab2-diagram.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We feed words one by one into LSTM layers.  After feeding in all the words, we take the final state of the LSTM and run it thorugh one fully connected layer to multiply it by a final set of weights. We specificy that this fully connected layer should have a single output, which, one sigmoid-ed, is the probability that the tweet is positive!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Set up our Model Parameters\n",
    "\n",
    "Similarly to the last lab, we'll be training using batches. Our hidden layer will have 100 units, and we have 7597 words in the vocabulary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set variables\n",
    "tweet_size = 20\n",
    "hidden_size = 100\n",
    "vocab_size = 7597\n",
    "batch_size = 64\n",
    "\n",
    "# this just makes sure that all our following operations will be placed in the right graph.\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# create a session variable that we can run later.\n",
    "session = tf.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create Placeholders\n",
    "\n",
    "We need to create placeholders for variable data that we will feed in ourselves (aka our tweets). Placeholders allow us to incorporate this data into the graph even though we don't know what it is yet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-3-fca0d63c6f9b>, line 12)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-3-fca0d63c6f9b>\"\u001b[0;36m, line \u001b[0;32m12\u001b[0m\n\u001b[0;31m    labels = #todo\u001b[0m\n\u001b[0m                  ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# the placeholder for tweets has first dimension batch_size for each tweet in a batch,\n",
    "# second dimension tweet_size for each word in the tweet, and third dimension vocab_size\n",
    "# since each word itself is represented by a one-hot vector of size vocab_size.\n",
    "# Note that we use 'None' instead of batch_size for the first dimsension.  This allows us \n",
    "# to deal with variable batch sizes\n",
    "tweets = tf.placeholder(tf.float32, [None, tweet_size, vocab_size])\n",
    "\n",
    "'''TODO: create a placeholder for the labels (our predictions).  \n",
    "   This should be a 1D vector with size = None, \n",
    "   since we are predicting one value for each tweet in the batch,\n",
    "   but we want to be able to deal with variable batch sizes.''';\n",
    "labels = #todo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build the LSTM Layers\n",
    "\n",
    "We want to feed the input sequence, word by word, into an LSTM layer, or multiple LSTM layers (we could also call this an LSTM **encoder**).  At each \"timestep\", we feed in the next word, and the LSTM updates its cell state. The final LSTM cell state can then be fed through a final classification layer(s) to get our sentiment prediction.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make our LSTM layer. The steps for this are: \n",
    "1. Create a LSTM Cell using [tf.contrib.rnn.LSTMCell](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMCell)\n",
    "\n",
    "2. Wrap a couple of these cells in [tf.nn.rnn_cell.MultiRNNCell](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/MultiRNNCell) to create a multiple LSTM layers.\n",
    "\n",
    "2. Define the operation to run these layers with [dynamic_rnn](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "'''TODO: create an LSTM Cell using BasicLSTMCell.  Note that this creates a *layer* of LSTM\n",
    "   cells, not just a single one.''';\n",
    "lstm_cell = #todo\n",
    "\n",
    "'''TODO: create three LSTM layers by wrapping three instances of \n",
    "   lstm_cell from above in tf.contrib.rnn_cell.MultiRNNCell. Note that\n",
    "   you can create multiple cells by doing [lstm_cell] * 2. Also note\n",
    "   that you should use state_is_tuple=True as an argument.  This will allow\n",
    "   us to access the part of the cell state that we need later on.''';\n",
    "multi_lstm_cells = #todo\n",
    "\n",
    "'''TODO: define the operation to create the RNN graph across time.  \n",
    "   tf.nn.dynamic_rnn dynamically constructs the graph when it is executed,\n",
    "   and returns the final cell state.''';\n",
    "_, final_state = #todo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Classification Layer\n",
    "\n",
    "Now we have the final state of the LSTM layers after feeding in the tweet word by word. We can take this final state and feed it into a simple classfication layer that takes the cell state, multiplies it by some weight matrix (with bias) and outputs a single value corresponding to whether it thinks the tweet is overall positive or not.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## We define this function that creates a weight matrix + bias parameter\n",
    "## and uses them to do a matrix multiplication.\n",
    "def linear(input_, output_size, name, init_bias=0.0):\n",
    "    shape = input_.get_shape().as_list()\n",
    "    with tf.variable_scope(name):\n",
    "        W = tf.get_variable(\"weights\", [shape[-1], output_size], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(shape[-1])))\n",
    "    if init_bias is None:\n",
    "        return tf.matmul(input_, W)\n",
    "    with tf.variable_scope(name):\n",
    "        b = tf.get_variable(\"bias\", [output_size], initializer=tf.constant_initializer(init_bias))\n",
    "    return tf.matmul(input_, W) + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''TODO: pass the final state into this linear function to multiply it \n",
    "   by the weights and add bias to get our output.\n",
    "   \n",
    "   {Quick note that we need to feed in final_state[-1][-1] into linear since \n",
    "   final_state is actually a tuple consisting of the cell state \n",
    "   (used internally for the cell to keep track of things) \n",
    "   as well as the hidden state (the output of the cell), and one of these \n",
    "   tuples for each layer. We want the hidden state for the last layer, so we use \n",
    "   final_state[-1][-1]}''';\n",
    "\n",
    "sentiment = #todo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Define Loss\n",
    "\n",
    "Now we define a loss function that we'll use to determine the difference between what we predicted and what's actually correct.  We'll want to use cross entropy, since we can take into account what probability the model gave to the a tweet being positive.\n",
    "\n",
    "The output we just got from the linear classification layer is called a 'logit' -- the raw value before transforming it into a probability between 0 and 1.  We can feed these logits to  [`tf.nn.sigmoid_cross_entropy_with_logits`](https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits), which will take the sigmoid of these logits (making them between 0 and 1) and then calculate the cross-entropy with the ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sentiment = tf.squeeze(sentiment, [1])\n",
    "\n",
    "'''TODO: define our loss function.  \n",
    "   We will use tf.nn.sigmoid_cross_entropy_with_logits, which will compare our \n",
    "   sigmoid-ed prediction (sentiment from above) to the ground truth (labels).''';\n",
    "\n",
    "loss = #todo\n",
    "\n",
    "# our loss with sigmoid_cross_entropy_with_logits gives us a loss for each \n",
    "# example in the batch.  We take the mean of all these losses.\n",
    "loss = tf.reduce_mean(loss)\n",
    "\n",
    "# to get actual results like 'positive' or 'negative' , \n",
    "# we round the prediction probability to 0 or 1.\n",
    "prediction = tf.to_float(tf.greater_equal(sentiment, 0.5))\n",
    "\n",
    "# calculate the error based on which predictions were actually correct.\n",
    "pred_err = tf.to_float(tf.not_equal(prediction, labels))\n",
    "pred_err = tf.reduce_sum(pred_err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Train\n",
    "\n",
    "Now we define the operation that actually changes the weights by minimizing the loss.  \n",
    "\n",
    "[`tf.train.AdamOptimizer`](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer) is just a gradient descent algorithm that uses a variable learning rate to converge faster and more effectively.\n",
    "\n",
    "We want to specify this optimizer and then call the [minimize](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer#minimize) function, the optimizer knows it wants to minimize the loss we defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''Define the operation that specifies the AdamOptimizer and tells\n",
    "   it to minimize the loss.''';\n",
    "optimizer = #todo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Run Session!\n",
    "\n",
    "Now that we've made all the variable and operations in our graph, we can load the data, feed it in, and run the model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# initialize any variables\n",
    "tf.global_variables_initializer().run(session=session)\n",
    "\n",
    "# load our data and separate it into tweets and labels\n",
    "train_data = json.load(open('data/trainTweets_preprocessed.json', 'r'))\n",
    "train_data = list(map(lambda row:(np.array(row[0],dtype=np.int32),str(row[1])),train_data))\n",
    "train_tweets = np.array([t[0] for t in train_data])\n",
    "train_labels = np.array([int(t[1]) for t in train_data])\n",
    "\n",
    "test_data = json.load(open('data/testTweets_preprocessed.json', 'r'))\n",
    "test_data = list(map(lambda row:(np.array(row[0],dtype=np.int32),str(row[1])),test_data))\n",
    "# we are just taking the first 1000 things from the test set for faster evaluation\n",
    "test_data = test_data[0:1000] \n",
    "test_tweets = np.array([t[0] for t in test_data])\n",
    "one_hot_test_tweets = utils.one_hot(test_tweets, vocab_size)\n",
    "test_labels = np.array([int(t[1]) for t in test_data])\n",
    "\n",
    "# we'll train with batches of size 128.  This means that we run \n",
    "# our model on 128 examples and then do gradient descent based on the loss\n",
    "# over those 128 examples.\n",
    "num_steps = 1000\n",
    "\n",
    "for step in range(num_steps):\n",
    "    # get data for a batch\n",
    "    offset = (step * batch_size) % (len(train_data) - batch_size)\n",
    "    batch_tweets = utils.one_hot(train_tweets[offset : (offset + batch_size)], vocab_size)\n",
    "    batch_labels = train_labels[offset : (offset + batch_size)]\n",
    "    \n",
    "    # put this data into a dictionary that we feed in when we run \n",
    "    # the graph.  this data fills in the placeholders we made in the graph.\n",
    "    data = {tweets: batch_tweets, labels: batch_labels}\n",
    "    \n",
    "    # run the 'optimizer', 'loss', and 'pred_err' operations in the graph\n",
    "    _, loss_value_train, error_value_train = session.run(\n",
    "      [optimizer, loss, pred_err], feed_dict=data)\n",
    "    \n",
    "    # print stuff every 50 steps to see how we are doing\n",
    "    if (step % 50 == 0):\n",
    "        print(\"Minibatch train loss at step\", step, \":\", loss_value_train)\n",
    "        print(\"Minibatch train error: %.3f%%\" % error_value_train)\n",
    "        \n",
    "        # get test evaluation\n",
    "        test_loss = []\n",
    "        test_error = []\n",
    "        for batch_num in range(int(len(test_data)/batch_size)):\n",
    "            test_offset = (batch_num * batch_size) % (len(test_data) - batch_size)\n",
    "            test_batch_tweets = one_hot_test_tweets[test_offset : (test_offset + batch_size)]\n",
    "            test_batch_labels = test_labels[test_offset : (test_offset + batch_size)]\n",
    "            data_testing = {tweets: test_batch_tweets, labels: test_batch_labels}\n",
    "            loss_value_test, error_value_test = session.run([loss, pred_err], feed_dict=data_testing)\n",
    "            test_loss.append(loss_value_test)\n",
    "            test_error.append(error_value_test)\n",
    "        \n",
    "        print(\"Test loss: %.3f\" % np.mean(test_loss))\n",
    "        print(\"Test error: %.3f%%\" % np.mean(test_error))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solutions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Minibatch train loss at step', 0, ':', 0.6933049)\n",
      "Minibatch train error: 40.000%\n",
      "Test loss: 0.693\n",
      "Test error: 28.400%\n",
      "('Minibatch train loss at step', 50, ':', 0.53027713)\n",
      "Minibatch train error: 15.000%\n",
      "Test loss: 0.629\n",
      "Test error: 22.800%\n",
      "('Minibatch train loss at step', 100, ':', 0.57602787)\n",
      "Minibatch train error: 23.000%\n",
      "Test loss: 0.589\n",
      "Test error: 20.667%\n",
      "('Minibatch train loss at step', 150, ':', 0.5539102)\n",
      "Minibatch train error: 14.000%\n",
      "Test loss: 0.581\n",
      "Test error: 18.733%\n",
      "('Minibatch train loss at step', 200, ':', 0.50935715)\n",
      "Minibatch train error: 16.000%\n",
      "Test loss: 0.548\n",
      "Test error: 17.400%\n",
      "('Minibatch train loss at step', 250, ':', 0.44617847)\n",
      "Minibatch train error: 16.000%\n",
      "Test loss: 0.532\n",
      "Test error: 16.467%\n",
      "('Minibatch train loss at step', 300, ':', 0.587883)\n",
      "Minibatch train error: 17.000%\n",
      "Test loss: 0.533\n",
      "Test error: 16.267%\n",
      "('Minibatch train loss at step', 350, ':', 0.54082739)\n",
      "Minibatch train error: 20.000%\n",
      "Test loss: 0.545\n",
      "Test error: 17.200%\n",
      "('Minibatch train loss at step', 400, ':', 0.56833553)\n",
      "Minibatch train error: 18.000%\n",
      "Test loss: 0.522\n",
      "Test error: 15.733%\n",
      "('Minibatch train loss at step', 450, ':', 0.57482541)\n",
      "Minibatch train error: 19.000%\n",
      "Test loss: 0.533\n",
      "Test error: 16.400%\n",
      "('Minibatch train loss at step', 500, ':', 0.55153328)\n",
      "Minibatch train error: 18.000%\n",
      "Test loss: 0.516\n",
      "Test error: 16.133%\n",
      "('Minibatch train loss at step', 550, ':', 0.57234359)\n",
      "Minibatch train error: 21.000%\n",
      "Test loss: 0.513\n",
      "Test error: 16.133%\n",
      "('Minibatch train loss at step', 600, ':', 0.41946551)\n",
      "Minibatch train error: 11.000%\n",
      "Test loss: 0.520\n",
      "Test error: 16.867%\n",
      "('Minibatch train loss at step', 650, ':', 0.54832065)\n",
      "Minibatch train error: 16.000%\n",
      "Test loss: 0.516\n",
      "Test error: 15.467%\n",
      "('Minibatch train loss at step', 700, ':', 0.46384442)\n",
      "Minibatch train error: 15.000%\n",
      "Test loss: 0.505\n",
      "Test error: 15.267%\n",
      "('Minibatch train loss at step', 750, ':', 0.58556581)\n",
      "Minibatch train error: 16.000%\n",
      "Test loss: 0.506\n",
      "Test error: 15.733%\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "import pickle as p\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import utils \n",
    "\n",
    "# set variables\n",
    "tweet_size = 20\n",
    "hidden_size = 100\n",
    "vocab_size = 7597\n",
    "batch_size = 64\n",
    "\n",
    "# this just makes sure that all our following operations will be placed in the right graph.\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# create a session variable that we can run later.\n",
    "session = tf.Session()\n",
    "\n",
    "# make placeholders for data we'll feed in\n",
    "tweets = tf.placeholder(tf.float32, [None, tweet_size, vocab_size])\n",
    "labels = tf.placeholder(tf.float32, [None])\n",
    "\n",
    "# make the lstm cells, and wrap them in MultiRNNCell for multiple layers\n",
    "lstm_cell = tf.contrib.rnn.LSTMCell(hidden_size)\n",
    "multi_lstm_cells = tf.contrib.rnn.MultiRNNCell(cells=[lstm_cell] * 2, state_is_tuple=True)\n",
    "\n",
    "# define the op that runs the LSTM, across time, on the data\n",
    "_, final_state = tf.nn.dynamic_rnn(multi_lstm_cells, tweets, dtype=tf.float32)\n",
    "\n",
    "# a useful function that takes an input and what size we want the output \n",
    "# to be, and multiples the input by a weight matrix plus bias (also creating\n",
    "# these variables)\n",
    "def linear(input_, output_size, name, init_bias=0.0):\n",
    "    shape = input_.get_shape().as_list()\n",
    "    with tf.variable_scope(name):\n",
    "        W = tf.get_variable(\"weight_matrix\", [shape[-1], output_size], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(shape[-1])))\n",
    "    if init_bias is None:\n",
    "        return tf.matmul(input_, W)\n",
    "    with tf.variable_scope(name):\n",
    "        b = tf.get_variable(\"bias\", [output_size], initializer=tf.constant_initializer(init_bias))\n",
    "    return tf.matmul(input_, W) + b\n",
    "\n",
    "# define that our final sentiment logit is a linear function of the final state \n",
    "# of the LSTM\n",
    "sentiment = linear(final_state[-1][-1], 1, name=\"output\")\n",
    "\n",
    "\n",
    "sentiment = tf.squeeze(sentiment, [1])\n",
    "\n",
    "# define cross entropy loss function\n",
    "loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=sentiment, labels=labels)\n",
    "loss = tf.reduce_mean(loss)\n",
    "\n",
    "# round our actual probabilities to compute error\n",
    "prob = tf.nn.sigmoid(sentiment)\n",
    "prediction = tf.to_float(tf.greater_equal(prob, 0.5))\n",
    "pred_err = tf.to_float(tf.not_equal(prediction, labels))\n",
    "pred_err = tf.reduce_sum(pred_err)\n",
    "\n",
    "# define our optimizer to minimize the loss\n",
    "optimizer = tf.train.AdamOptimizer().minimize(loss)\n",
    "\n",
    "# initialize any variables\n",
    "tf.global_variables_initializer().run(session=session)\n",
    "\n",
    "# load our data and separate it into tweets and labels\n",
    "train_data = json.load(open('data/trainTweets_preprocessed.json', 'r'))\n",
    "train_data = list(map(lambda row:(np.array(row[0],dtype=np.int32),str(row[1])),train_data))\n",
    "\n",
    "train_tweets = np.array([t[0] for t in train_data])\n",
    "train_labels = np.array([int(t[1]) for t in train_data])\n",
    "\n",
    "test_data = json.load(open('data/testTweets_preprocessed.json', 'r'))\n",
    "test_data = map(lambda row:(np.array(row[0],dtype=np.int32),str(row[1])),test_data)\n",
    "# we are just taking the first 1000 things from the test set for faster evaluation\n",
    "test_data = test_data[0:1000] \n",
    "test_tweets = np.array([t[0] for t in test_data])\n",
    "one_hot_test_tweets = utils.one_hot(test_tweets, vocab_size)\n",
    "test_labels = np.array([int(t[1]) for t in test_data])\n",
    "\n",
    "# we'll train with batches of size 128.  This means that we run \n",
    "# our model on 128 examples and then do gradient descent based on the loss\n",
    "# over those 128 examples.\n",
    "num_steps = 1000\n",
    "\n",
    "for step in range(num_steps):\n",
    "    # get data for a batch\n",
    "    offset = (step * batch_size) % (len(train_data) - batch_size)\n",
    "    batch_tweets = utils.one_hot(train_tweets[offset : (offset + batch_size)], vocab_size)\n",
    "    batch_labels = train_labels[offset : (offset + batch_size)]\n",
    "    \n",
    "    # put this data into a dictionary that we feed in when we run \n",
    "    # the graph.  this data fills in the placeholders we made in the graph.\n",
    "    data = {tweets: batch_tweets, labels: batch_labels}\n",
    "    \n",
    "    # run the 'optimizer', 'loss', and 'pred_err' operations in the graph\n",
    "    _, loss_value_train, error_value_train = session.run(\n",
    "      [optimizer, loss, pred_err], feed_dict=data)\n",
    "    \n",
    "    # print stuff every 50 steps to see how we are doing\n",
    "    if (step % 50 == 0):\n",
    "        print(\"Minibatch train loss at step\", step, \":\", loss_value_train)\n",
    "        print(\"Minibatch train error: %.3f%%\" % error_value_train)\n",
    "        \n",
    "        # get test evaluation\n",
    "        test_loss = []\n",
    "        test_error = []\n",
    "        for batch_num in range(int(len(test_data)/batch_size)):\n",
    "            test_offset = (batch_num * batch_size) % (len(test_data) - batch_size)\n",
    "            test_batch_tweets = one_hot_test_tweets[test_offset : (test_offset + batch_size)]\n",
    "            test_batch_labels = test_labels[test_offset : (test_offset + batch_size)]\n",
    "            data_testing = {tweets: test_batch_tweets, labels: test_batch_labels}\n",
    "            loss_value_test, error_value_test = session.run([loss, pred_err], feed_dict=data_testing)\n",
    "            test_loss.append(loss_value_test)\n",
    "            test_error.append(error_value_test)\n",
    "        \n",
    "        print(\"Test loss: %.3f\" % np.mean(test_loss))\n",
    "        print(\"Test error: %.3f%%\" % np.mean(test_error))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "name": "_merged"
 },
 "nbformat": 4,
 "nbformat_minor": 1
}