{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RNN Sentiment Classifier\n",
    "\n",
    "In the previous lab, you built a tweet sentiment classifier based on Bag-Of-Words features. Now we ask you to improve this model by representing it as a sequence of words. \n",
    "\n",
    "\n",
    "## Step 1: Input Preprocessing\n",
    "\n",
    "Run `read_data()` below to read training data, normalizing the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('finally back twitterberry messed up my phone lets try this one out', '0'), ('so board', '0'), ('early night woo', '1'), ('cyktrussell runningtodisney ill bet you both have hot legs', '1'), ('god bless', '0'), ('how wonderful am not able to get to sleep cause my bladders so uncomfortable gtlti think it may be time to just go to the er', '0'), ('clear httpplurkcompz510j', '1'), ('glitterbubbles hello there lt3', '1'), ('because i need a fresh start aaaah that feels nice', '1'), ('juzmcmuz yeah thanks', '1')]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import cPickle as pickle\n",
    "from collections import defaultdict\n",
    "import re, random\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "#Read data and do preprocessing\n",
    "def read_data(fn):\n",
    "    with open(fn) as f:\n",
    "        data = pickle.load(f)\n",
    "    \n",
    "    #Clean the text\n",
    "    new_data = []\n",
    "    pattern = re.compile('[\\W_]+')\n",
    "    for text,label in data:\n",
    "        text = text.strip(\"\\r\\n \").split()\n",
    "        x = []\n",
    "        for word in text:\n",
    "            word = pattern.sub('', word)\n",
    "            word = word.lower()\n",
    "            if 0 < len(word) < 20:\n",
    "                x.append(word)\n",
    "        new_data.append((' '.join(x),label))\n",
    "    return new_data \n",
    "\n",
    "train = read_data(\"data/train.p\")\n",
    "print train[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Build a Vocabulary\n",
    "\n",
    "Here we will use sklearn's CountVectorizer to automatically build a vocabulary over the training set. Infrequent words are pruned to make our life easier. Here we have two special tokens: `UNK_ID` for unknown words and `PAD_ID` for special token `<PAD>` that is used to pad sentences to the same length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[([279, 70, 981, 981, 866, 559, 620, 471, 848, 814, 592, 602], '0'), ([736, 981], '0'), ([227, 572, 940], '1'), ([981, 981, 419, 88, 972, 104, 368, 400, 981], '1'), ([327, 981], '0'), ([405, 937, 35, 577, 9, 830, 317, 830, 729, 141, 559, 981, 736, 981, 981, 811, 433, 516, 77, 827, 830, 444, 326, 830, 800, 981], '0'), ([981, 981], '1'), ([981, 380, 804, 503], '1'), ([80, 981, 564, 981, 981, 756, 981, 798, 271, 571], '1'), ([981, 966, 797], '1')]\n"
     ]
    }
   ],
   "source": [
    "train_x, train_y = zip(*train)\n",
    "vectorizer = CountVectorizer(train_x, min_df=0.001) \n",
    "vectorizer.fit(train_x)\n",
    "vocab = vectorizer.vocabulary_\n",
    "\n",
    "UNK_ID = len(vocab)\n",
    "PAD_ID = len(vocab) + 1\n",
    "word2id = lambda w:vocab[w] if w in vocab else UNK_ID\n",
    "train_x = [[word2id(w) for w in x.split()] for x in train_x]\n",
    "train_data = zip(train_x, train_y)\n",
    "print train_data[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build an LSTM Encoder\n",
    "\n",
    "A classifier requires the input feature vector to be of fixed size, while sentences are of different lengths. Thus, we need a model (called as **encoder**) to transform a sentence to a fixed size vector. This could be done by a recurrent neural net (RNN), by taking the last hidden state of LSTM encoder as the feature vector. We could then build a linear (or a multi-layer) network upon it to perform a classifier.\n",
    "\n",
    "### Step 3.1 Embedding Lookup Layer\n",
    "\n",
    "At input layer, words are represented by their ID (one-hot vector). Before feeding words to LSTM cell, we need an embedding lookup layer to map words to their word vector, given their ID. You should write a function to perform this operation.\n",
    "\n",
    "```def lookup_table(input_, vocab_size, output_size)```\n",
    "\n",
    "where `input_` is a matrix of sentences (**sentences are padded to the same length in a batch**), `vocab_size` is the size of vocabulary, `output_size` the size of word vector. You could use the tensorflow API function [embedding-lookup](https://www.tensorflow.org/api_docs/python/nn/embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.2 LSTM Layer\n",
    "\n",
    "Now we have the embedding layer, we can build LSTM layer upon it. It requires 4 steps:\n",
    "1. Create a LSTM Cell using [BasicLSTMCell](https://www.tensorflow.org/api_docs/python/rnn_cell/rnn_cells_for_use_with_tensorflow_s_core_rnn_methods)\n",
    "2. Let's say you have a `lstm_cell` object, declare initial state vector by calling `lstm_cell.zero_state()`.\n",
    "3. Create a RNN Layer using [dynamic-rnn](https://www.tensorflow.org/api_docs/python/nn/recurrent_neural_networks), get the final state of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.3 Classification Layer\n",
    "\n",
    "Now you have a fixed-size vector for sentences, build a classification layer the same as previous. Declare the cross-entropy loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.4 Training\n",
    "\n",
    "Now you need to feed the network with training data, and optimize it. Note that you have to **pad** sentences in the batch to the same length. To do this, you should put some `PAD_ID` tokens before each sentences so that they are in the same length. (Don't put them in the back because it would be harder to get the final hidden state of each sentence!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Full Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'vocab' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m----------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m    Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-0d86a5a72728>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mbatch_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mhidden_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m100\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mvocab_size\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mlookup_table\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvocab_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'vocab' is not defined"
     ]
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "#build RNN model\n",
    "batch_size = 20\n",
    "hidden_size = 100\n",
    "vocab_size = len(vocab) + 2\n",
    "\n",
    "def lookup_table(input_, vocab_size, output_size, name):\n",
    "    with tf.variable_scope(name):\n",
    "        embedding = tf.get_variable(\"embedding\", [vocab_size, output_size], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(output_size)))\n",
    "    return tf.nn.embedding_lookup(embedding, input_)\n",
    "\n",
    "def linear(input_, output_size, name, init_bias=0.0):\n",
    "    shape = input_.get_shape().as_list()\n",
    "    with tf.variable_scope(name):\n",
    "        W = tf.get_variable(\"Matrix\", [shape[-1], output_size], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(shape[-1])))\n",
    "    if init_bias is None:\n",
    "        return tf.matmul(input_, W)\n",
    "    with tf.variable_scope(name):\n",
    "        b = tf.get_variable(\"bias\", [output_size], initializer=tf.constant_initializer(init_bias))\n",
    "    return tf.matmul(input_, W) + b\n",
    "\n",
    "session = tf.Session()\n",
    "\n",
    "tweets = tf.placeholder(tf.int32, [batch_size, None])\n",
    "labels = tf.placeholder(tf.float32, [batch_size])\n",
    "\n",
    "embedding = lookup_table(tweets, vocab_size, hidden_size, name=\"word_embedding\")\n",
    "lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(hidden_size)\n",
    "init_state = lstm_cell.zero_state(batch_size, tf.float32)\n",
    "_, final_state = tf.nn.dynamic_rnn(lstm_cell, embedding, initial_state=init_state)\n",
    "sentiment = linear(final_state[1], 1, name=\"output\")\n",
    "\n",
    "sentiment = tf.squeeze(sentiment, [1])\n",
    "loss = tf.nn.sigmoid_cross_entropy_with_logits(sentiment, labels)\n",
    "loss = tf.reduce_mean(loss)\n",
    "prediction = tf.to_float(tf.greater_equal(sentiment, 0.5))\n",
    "pred_err = tf.to_float(tf.not_equal(prediction, labels))\n",
    "pred_err = tf.reduce_sum(pred_err)\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer().minimize(loss)\n",
    "\n",
    "tf.global_variables_initializer().run(session=session)\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "random.shuffle(train_data)\n",
    "\n",
    "err_rate = 0.0\n",
    "for step in xrange(0, len(train_data), batch_size):\n",
    "    batch = train_data[step:step+batch_size]\n",
    "    batch_x, batch_y = zip(*batch)\n",
    "    batch_x = list(batch_x)\n",
    "    if len(batch_x) != batch_size:\n",
    "        continue\n",
    "    max_len = max([len(x) for x in batch_x])\n",
    "    for i in xrange(batch_size):\n",
    "        len_x = len(batch_x[i])\n",
    "        batch_x[i] = [PAD_ID] * (max_len - len_x) + batch_x[i]\n",
    "    batch_x = np.array(batch_x, dtype=np.int32)\n",
    "    batch_y = np.array(batch_y, dtype=np.float32)\n",
    "    feed_map = {tweets:batch_x, labels:batch_y}\n",
    "    _, batch_err = session.run([optimizer, pred_err], feed_dict=feed_map)\n",
    "    err_rate += batch_err\n",
    "    if step % 1000 == 0 and step > 0:\n",
    "        print err_rate / step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}