{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[],"collapsed_sections":["bvDPONxkp6cc","qt_suTxbFCzH","cH1ZaptYFOZS","TCI_Ev1hF0Wr","2lDehY6DF4MQ","V2IdFhOML_Mk","nk6mayRPINDm","S2QTH4B4KfxZ","fz1inb00JqKf","vcaONOHfINf-","xUFHMT0hPt2T","NO-tepevOTBZ","YSSMJYdEO5eg","JWNJJauKO0sN","M0HaKxbyPm52","8q7KEuZAagP8"]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["**Goal**: Learn how to construct an autograd graph and implementing a basic engine to get the intuition of the autograd engine framework.\n","\n","**Lab workflow**:\n","- Go to **[Piazza post @74](https://piazza.com/class/m4cvt38ufqk155/post/74)**, the autolab .\n","- Follow the instruction in this notebook and complete **checkpoints???**.\n","- By the end of the Lab, respond in **[Piazza post @365](https://piazza.com/class/m4cvt38ufqk155/post/365)** how many checkpoints you completed. It's not graded and designed to help you with your HWs.\n","\n","**Prepared by**:\n","- John Liu, IDL TA S25, johnliu@andrew.cmu.edu\n","- Michael Kireeff, IDL TA S25, mkireeff@andrew.cmu.edu\n","\n","**Extra resources**: This Lab builds upon lectures 4 and 5."],"metadata":{"id":"Sb-AEda-Qv1C"}},{"cell_type":"markdown","source":["**Plan**:\n","1. How to\n"," - Initialization\n"," - Basic Operations\n"," - Matrix Multiplication\n"," - Backward Function\n"," - SoftPlus\n"," - Leaky Relu\n"," - Tanh"],"metadata":{"id":"A68x1ufpNK2C"}},{"cell_type":"markdown","source":["#**Basic Node Class**\n","- We will suggest you do this in numpy (it is way easier).\n","- When initializing some of these arrays look into the Numpy library: https://numpy.org/doc/2.1/reference/generated/numpy.array.html\n","and understand there are some other interesting things (Understanding functions will be crucial to understand for HW2)."],"metadata":{"id":"Tn3mrmXXQsiU"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"MdRwzRrZXFPX"},"outputs":[],"source":["import numpy as np\n","\n","class Node():\n"," def __init__(self, data, _parents=()):\n"," \"\"\"\n"," We want to initialize data, grad, and backward.\n"," where:\n"," data is an array, do we have data?\n"," grad is an array, do we have gradients?\n"," _prev is a set, which one of our inputs holds the previous nodes?\n"," \"\"\"\n"," #TODO: Hold data in a np.array(), initialize grad to the same shape as data with values of 0, and keep track of parents in a list or set\n"," self.data = np.array(data) # Store data as NumPy array\n"," self.grad = np.zeros_like(self.data) # Gradient initialized to zero\n"," self._parents = list(_parents) # Parent nodes in the computation graph\n","\n"," def __repr__(self):\n"," return f\"Node(data={self.data}, grad={self.grad})\"\n"]},{"cell_type":"markdown","source":["## **Checkin \\#1**"],"metadata":{"id":"bvDPONxkp6cc"}},{"cell_type":"markdown","source":[" Question: What is the point of autograd?\n"," \n"," Answer: To incrementally add operations so that it can all be automatically differentiated,\n"," "],"metadata":{"id":"5nKgxnf1-gvJ"}},{"cell_type":"markdown","source":["## Base Node Tests"],"metadata":{"id":"qt_suTxbFCzH"}},{"cell_type":"code","source":["print(\"Testing Node Initialization...\")\n","a_test = Node(3.0)\n","b_test = Node(2.0)\n","\n","assert a_test.data == 3.0, \"Error: a_test.data not set correctly\"\n","assert b_test.data == 2.0, \"Error: b_test.data not set correctly\"\n","assert a_test.grad == 0.0, \"Error: a_test.grad not set correctly\"\n","assert b_test.grad == 0.0, \"Error: b_test.grad not set correctly\"\n","\n","print(\"Passed!\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"PbW4lz2p9tNe","executionInfo":{"status":"ok","timestamp":1738992259667,"user_tz":300,"elapsed":36,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"808fa976-22ba-4420-a985-8a7bcd00dd5a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Testing Node Initialization...\n","Passed!\n"]}]},{"cell_type":"markdown","source":["#**Elementary functions/ combinations**\n"],"metadata":{"id":"eqLMWVCipcEA"}},{"cell_type":"markdown","source":["## Add Operation"],"metadata":{"id":"Y1BNeOf1FqgE"}},{"cell_type":"code","source":["def add_node(a_node, b_node):\n"," \"\"\"\n"," Element-wise addition: A + B\n"," \"\"\"\n"," # TODO:\n"," # Produce a new node:\n"," # Add the input data together to produce the output for the out node.\n"," # Make sure the output knows what its parents are\n"," out = Node(a_node.data + b_node.data, (a_node, b_node))\n","\n"," # Note:\n"," # - out.grad will be populated later\n"," # - for now, assume out.grad has its proper value for this node.\n"," #\n"," # TODO:\n"," # define a backward function to update a_node's gradients and b_node's gradients\n"," # based on out.grad\n"," #\n"," def add_backward():\n"," a_node.grad += out.grad # dL/dA = dL/dC\n"," b_node.grad += out.grad # dL/dB = dL/dC\n","\n"," out.backward_fn = add_backward\n","\n"," return out\n"],"metadata":{"id":"ZgeW09zL93Mx"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Add Operation Tests"],"metadata":{"id":"cH1ZaptYFOZS"}},{"cell_type":"code","source":["# Test your add operation\n","a_test = Node(3.0)\n","b_test = Node(2.0)\n","\n","print(\"Testing Node Addition...\")\n","c_test = add_node(a_test, b_test)\n","assert c_test.data == 5.0, \"Error: add_node DATA not evaluated correctly\"\n","assert c_test.grad == 0.0, \"Error: add_node GRAD not initialized correctly\"\n","print(\"Passed!\")\n","\n","print(\"Testing Node Addition Backward...\")\n","c_test.grad = 1.0\n","c_test.backward_fn()\n","assert a_test.grad == 1.0, \"Error: add_node BACKWARD does not update parent gradients correctly\"\n","assert b_test.grad == 1.0, \"Error: add_node BACKWARD does not update parent gradients correctly\"\n","\n","print(\"Passed!\")\n","\n","print(\"All tests passed!\")"],"metadata":{"id":"mTvZIznG95Xx","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1738992259687,"user_tz":300,"elapsed":4,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"94f01e20-963f-447e-ba23-d503fc76c6b3"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Testing Node Addition...\n","Passed!\n","Testing Node Addition Backward...\n","Passed!\n","All tests passed!\n"]}]},{"cell_type":"markdown","source":["## Mul Operation"],"metadata":{"id":"SZJ4Ft6GFxqi"}},{"cell_type":"code","source":["def mul_node(a_node, b_node):\n"," \"\"\"\n"," Element-wise multiplication: A * B\n"," \"\"\"\n"," # TODO:\n"," # Produce a new node:\n"," # Make sure the output's data is correct\n"," # Make sure the output knows what its parents are\n"," out = Node(a_node.data * b_node.data, (a_node, b_node))\n","\n"," # Note:\n"," # - out.grad will be populated later\n"," # - for now, assume out.grad has its proper value for this node.\n"," #\n"," # TODO:\n"," # define a backward function to update a_node's gradients and b_node's gradients\n"," # based on out.grad\n"," #\n"," def mul_backward():\n"," a_node.grad += b_node.data * out.grad # dL/dA = B * dL/dC\n"," b_node.grad += a_node.data * out.grad # dL/dB = A * dL/dC\n","\n"," out.backward_fn = mul_backward\n","\n"," return out"],"metadata":{"id":"HL6hGIcBrhNN"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Mul Operation Tests"],"metadata":{"id":"TCI_Ev1hF0Wr"}},{"cell_type":"code","source":["# Test your multiplication operation\n","a_test = Node(3.0)\n","b_test = Node(2.0)\n","\n","print(\"Testing Node Multiplication...\")\n","d_test = mul_node(a_test, b_test)\n","assert d_test.data == 6.0, \"Error: mul_node DATA not evaluated correctly\"\n","assert d_test.grad == 0.0, \"Error: mul_node GRAD not initialized correctly\"\n","\n","print(\"Passed!\")\n","\n","\n","print(\"Testing Node Multiplication Backward...\")\n","d_test.grad = 1.0\n","d_test.backward_fn()\n","assert a_test.grad == 2.0, \"Error: mul_node BACKWARD does not update parent gradients correctly\"\n","assert b_test.grad == 3.0, \"Error: mul_node BACKWARD does not update parent gradients correctly\"\n","\n","print(\"Passed!\")\n","\n","print(\"All tests passed!\")\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"pxchDyBl9zq2","executionInfo":{"status":"ok","timestamp":1738992259771,"user_tz":300,"elapsed":81,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"41a3649d-9e56-4249-fffc-747dbcd355f1"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Testing Node Multiplication...\n","Passed!\n","Testing Node Multiplication Backward...\n","Passed!\n","All tests passed!\n"]}]},{"cell_type":"markdown","source":["## **Checkin \\#2**"],"metadata":{"id":"2lDehY6DF4MQ"}},{"cell_type":"markdown","source":["Question: Why is it important to have these elementary functions before going to more complex functions?\n","\n","Answer: All complex functions require basic elementary functions to work. So instead of having to code backward each function, we can just call backward of these passed modules.\n"," "],"metadata":{"id":"ZpALS62zsYm5"}},{"cell_type":"markdown","source":["# **Meme \\#1**"],"metadata":{"id":"V2IdFhOML_Mk"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"lup88-A0MDlf"}},{"cell_type":"markdown","source":["#Matrix Multiplication"],"metadata":{"id":"x0HdTm_kLZOf"}},{"cell_type":"code","source":["def matmul_node(a_node, b_node):\n"," \"\"\"Matrix multiplication: A @ B\"\"\"\n"," out = Node((a_node.data @ b_node.data), (a_node, b_node))\n","\n"," def matmul_backward():\n"," a_node.grad += np.dot(out.grad, b_node.data.T) # Gradient wrt a\n"," b_node.grad += np.dot(a_node.data.T, out.grad) # Gradient wrt b\n","\n"," out.backward_fn = matmul_backward\n","\n"," return out"],"metadata":{"id":"cGTyXsBwPWYF"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Tests matmul function"],"metadata":{"id":"nk6mayRPINDm"}},{"cell_type":"code","source":["a_test = Node(np.array([[1, 2], [3, 4]]))\n","b_test = Node(np.array([[2, 0], [1, 2]]))\n","\n","print(\"Testing Matrix Multiplication...\")\n","c_test = matmul_node(a_test, b_test)\n","\n","assert np.allclose(c_test.data, np.array([[4, 4], [10, 8]])), \"Error: matmul DATA not evaluated correctly\"\n","print(\"Passed!\")\n","\n","print(\"Testing Matrix Multiplication Backward...\")\n","c_test.grad = np.array([[1, 1], [1, 1]])\n","c_test.backward_fn()\n","\n","assert np.allclose(a_test.grad, np.array([[2, 3], [2, 3]])), \"Error: matmul BACKWARD does not update parent gradients correctly\"\n","assert np.allclose(b_test.grad, np.array([[4, 4], [6, 6]])), \"Error: matmul BACKWARD does not update parent gradients correctly\"\n","print(\"Passed!\")\n","\n","print(\"All tests passed!\")"],"metadata":{"id":"ONyRhCS9rlAH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1738992259784,"user_tz":300,"elapsed":4,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"99229500-1fb7-4567-dff3-ef99c2b35c77"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Testing Matrix Multiplication...\n","Passed!\n","Testing Matrix Multiplication Backward...\n","Passed!\n","All tests passed!\n"]}]},{"cell_type":"markdown","source":["#**Meme #2**"],"metadata":{"id":"S2QTH4B4KfxZ"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"FNYpUSt-KupZ"}},{"cell_type":"markdown","source":["# **Backward Method**\n"],"metadata":{"id":"2_dxaRC2MT3d"}},{"cell_type":"code","source":["class Node(Node):\n"," def backward(self):\n"," backward_list = []\n","\n"," self.grad = np.ones_like(self.data) # Placeholder for the gradient of the loss\n","\n"," #TODO: Define a recursive function to build the backward list\n"," def build_backward_list(n):\n"," # TODO: Check if the node is in the backward list\n"," if n not in backward_list:\n"," # TODO: Add the node to the backward list\n"," backward_list.append(n)\n","\n"," # TODO: iterate through the parents of the node and call the function recursively\n"," for parent in n._parents:\n"," build_backward_list(parent)\n","\n","\n"," build_backward_list(self)\n"," # TODO: print the output backward list if you want to see it!\n","\n"," # TODO: Iterate through the backward list and call the backward functions (if it has parents)\n"," input_nodes = []\n"," for node in backward_list:\n"," if node._parents:\n"," node.backward_fn()\n"," else:\n"," print(\"input node\")\n"," input_nodes.append(node)\n"," print(node)\n"," print(\"Input nodes: \", input_nodes)\n","\n"],"metadata":{"id":"jPF1pTIPrju4"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["###Backward Test"],"metadata":{"id":"fz1inb00JqKf"}},{"cell_type":"code","source":["a_test = Node(3)\n","b_test = Node(2)\n","\n","c_test = add_node(a_test, b_test)\n","f_test = mul_node(c_test, b_test)\n","\n","\n","print(\"Testing Node BACKWARD...\")\n","f_test.grad = 1.0\n","f_test.backward()\n","\n","print(f\"a_grad: {a_test}\")\n","print(f\"b_grad: {b_test}\")\n","print(f\"c_grad: {c_test}\")\n","print(f\"f_grad: {f_test}\")\n","\n","assert a_test.grad == 2.0, \"Error: BACKWARD does not update parent gradients correctly\"\n","assert b_test.grad == 7.0, \"Error: BACKWARD does not update parent gradients correctly\"\n","assert c_test.grad == 2.0, \"Error: BACKWARD does not update parent gradients correctly\"\n","assert f_test.grad == 1.0, \"Error: BACKWARD does not update parent gradients correctly\"\n","\n","print(\"All tests passed!\")"],"metadata":{"id":"7H552k5NJpxk","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1738992259791,"user_tz":300,"elapsed":5,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"50d27135-272e-4f54-b655-5bccd0b8f427"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Testing Node BACKWARD...\n","Node(_ | data=10, grad=1)\n","Input nodes: []\n","Node(_ | data=5, grad=2)\n","Input nodes: []\n","input node\n","Node(_ | data=3, grad=2)\n","Input nodes: [Node(_ | data=3, grad=2)]\n","input node\n","Node(_ | data=2, grad=7)\n","Input nodes: [Node(_ | data=3, grad=2), Node(_ | data=2, grad=7)]\n","a_grad: Node(_ | data=3, grad=2)\n","b_grad: Node(_ | data=2, grad=7)\n","c_grad: Node(_ | data=5, grad=2)\n","f_grad: Node(_ | data=10, grad=1)\n","All tests passed!\n"]}]},{"cell_type":"markdown","source":["#**Checkin \\#3**\n"],"metadata":{"id":"wwLhpY5OdIYE"}},{"cell_type":"markdown","source":["Question: Why do we need a backward pass? \\\\\n","**Hint:**\n","Look at Meme #2\n","\n","Answer: Because if we have many layers, than we need loop through them all.\n"," \n"],"metadata":{"id":"PUm9ET_NudoJ"}},{"cell_type":"markdown","source":["#**Meme \\#3**"],"metadata":{"id":"vcaONOHfINf-"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"MC2hzAB-MngX"}},{"cell_type":"markdown","source":["# Writing the computational Graph on the Lecture Slides (slide 35)"],"metadata":{"id":"VXY9XTTGwTqG"}},{"cell_type":"code","source":["# TODO\n","# Declare two nodes: A and B with values 7 and 8, then create the function f = b * (a + b)\n","# Set F.grad to 1.0\n","# Call backward, and print all your nodes!\n","\n","\n","# TODO"],"metadata":{"id":"5w6jj074wPZG"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["# **Activation Functions**\n"],"metadata":{"id":"eZuuISYrMs_z"}},{"cell_type":"markdown","source":["* For these, you will want to use your primitives (mul_node, add_node, as well as some others we're about to give you).\n","* This is so that in the final output, all you simply have to do is call out.backward() and all the derivatives will be computed for you!"],"metadata":{"id":"gKCvM021NNpS"}},{"cell_type":"markdown","source":["##**Softplus**"],"metadata":{"id":"ePV0KrN1PS_u"}},{"cell_type":"code","source":["# Here's a natural log node for you :)\n","def log_node(a_node):\n"," out = Node(np.log(a_node.data), (a_node,))\n"," def backward_fn():\n"," a_node.grad += out.grad / a_node.data\n"," out.backward_fn = backward_fn\n"," return out\n","\n","def exp_node(a_node):\n"," out = Node(np.exp(a_node.data), (a_node,))\n"," def backward_fn():\n"," a_node.grad += out.grad * out.data\n"," out.backward_fn = backward_fn\n"," return out"],"metadata":{"id":"cm3gCCF9HdUN"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["\n","def softplus_node(x):\n"," \"\"\"\n"," softplus(x) = ln(1 + e^x)\n"," \"\"\"\n"," exp_x = exp_node(x) # e^x\n","\n"," one_exp_x = add_node(Node(np.ones_like(x.data)), exp_x) # 1 + e^x\n","\n"," out = log_node(one_exp_x) # ln(1 + e^x)\n","\n"," return out"],"metadata":{"id":"mGL-rCmFcIm1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Tests for SoftPlus"],"metadata":{"id":"xUFHMT0hPt2T"}},{"cell_type":"code","source":["# Create a test input node (use a range of values for variety)\n","x = Node(np.array([-2.0, -1.0, 0.0, 1.0, 2.0])) # A mix of negative and positive values\n","out = softplus_node(x)\n","expected_forward = np.log(1.0 + np.exp(x.data))\n","\n","# Print results for forward pass\n","print(\"Forward Output (softplus(x)):\", out.data)\n","print(\"Expected:\", expected_forward)\n","print(\"Match:\", np.allclose(out.data, expected_forward))\n","\n","\n","out.grad = np.ones_like(out.data)\n","out.backward()\n","expected_grad_x = 1.0 / (1.0 + np.exp(-x.data)) # Sigmoid function\n","\n","# Print results for backward pass (gradients)\n","print(\"\\nGradient wrt Input:\", x.grad)\n","print(\"Expected Gradient:\", expected_grad_x)\n","print(\"Match:\", np.allclose(x.grad, expected_grad_x))\n"],"metadata":{"id":"4A9diT8ocfJj","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1738992302343,"user_tz":300,"elapsed":8,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"54a4b7ef-0653-4f37-9c67-2bf5855dd8e2"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Forward Output (softplus(x)): [0.12692801 0.31326169 0.69314718 1.31326169 2.12692801]\n","Expected: [0.12692801 0.31326169 0.69314718 1.31326169 2.12692801]\n","Match: True\n","Node(sp_out | data=[0.12692801 0.31326169 0.69314718 1.31326169 2.12692801], grad=[1. 1. 1. 1. 1.])\n","Input nodes: []\n","Node(sp_one_exp_x | data=[1.13533528 1.36787944 2. 3.71828183 8.3890561 ], grad=[0.88079708 0.73105858 0.5 0.26894142 0.11920292])\n","Input nodes: []\n","input node\n","Node(_ | data=[1. 1. 1. 1. 1.], grad=[0.88079708 0.73105858 0.5 0.26894142 0.11920292])\n","Input nodes: [Node(_ | data=[1. 1. 1. 1. 1.], grad=[0.88079708 0.73105858 0.5 0.26894142 0.11920292])]\n","Node(sp_exp_x | data=[0.13533528 0.36787944 1. 2.71828183 7.3890561 ], grad=[0.88079708 0.73105858 0.5 0.26894142 0.11920292])\n","Input nodes: [Node(_ | data=[1. 1. 1. 1. 1.], grad=[0.88079708 0.73105858 0.5 0.26894142 0.11920292])]\n","input node\n","Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.11920292 0.26894142 0.5 0.73105858 0.88079708])\n","Input nodes: [Node(_ | data=[1. 1. 1. 1. 1.], grad=[0.88079708 0.73105858 0.5 0.26894142 0.11920292]), Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.11920292 0.26894142 0.5 0.73105858 0.88079708])]\n","\n","Gradient wrt Input: [0.11920292 0.26894142 0.5 0.73105858 0.88079708]\n","Expected Gradient: [0.11920292 0.26894142 0.5 0.73105858 0.88079708]\n","Match: True\n"]}]},{"cell_type":"markdown","source":["#**Checkin \\#4**\n","Question: What are the memory implications for storing the Softplus forward-pass values in autograd?\n","\n","Autograd systems often store forward-pass values for backprop. For Softplus, the output itself $ln(1+e^x)$ may be used to compute or simplify the gradient. So even functions like softplus, can use sigmoid to help simplify memory allocation.\n"," "],"metadata":{"id":"_3vdPm8vuL7Q"}},{"cell_type":"markdown","source":["#**Meme \\#4**"],"metadata":{"id":"NO-tepevOTBZ"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"8RnyTvwiOUvX"}},{"cell_type":"markdown","source":["# **Leaky ReLU**"],"metadata":{"id":"1L7-WmbFOI8G"}},{"cell_type":"code","source":["def leaky_relu_node(input, alpha=0.01):\n"," \"\"\"\n"," Leaky ReLU activation:\n"," f(x) = x, if x > 0\n"," alpha * x, otherwise\n"," \"\"\"\n"," # Forward pass\n"," alpha_node = Node(np.ones_like(input.data) * alpha) # Create a node filled with alpha\n","\n"," positives_mask_node = Node(np.where(input.data >= 0., 1., 0.)) # 1 where x >= 0\n"," negatives_mask_node = Node(np.where(input.data < 0., 1., 0.)) # 1 where x < 0\n","\n"," alpha_mask_node = mul_node(alpha_node, negatives_mask_node) # alpha * (x < 0)\n","\n"," positive_part = mul_node(input, positives_mask_node) # x * (x >= 0)\n"," negative_part = mul_node(input, alpha_mask_node) # alpha * x * (x < 0)\n","\n"," out = add_node(positive_part, negative_part)\n","\n"," return out\n"],"metadata":{"id":"QKjcok8qcBb2"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Tests for Leaky ReLU"],"metadata":{"id":"YSSMJYdEO5eg"}},{"cell_type":"code","source":["# Create a test input node\n","x = Node(np.array([-2.0, -1.0, 0.0, 1.0, 2.0])) # A mix of negative and positive values\n","alpha = 0.01 # Leaky ReLU slope for negatives\n","\n","# Compute Leaky ReLU\n","out = leaky_relu_node(x, alpha)\n","\n","# Set the output gradient (simulating dL/dout = 1 for all elements)\n","out.grad = np.ones_like(out.data)\n","\n","# Perform backpropagation\n","out.backward()\n","\n","# Expected values\n","expected_forward = np.array([-0.02, -0.01, 0.0, 1.0, 2.0]) # Leaky ReLU applied\n","expected_grad_x = np.array([alpha, alpha, 1.0, 1.0, 1.0]) # Gradients from backprop\n","\n","# Print results\n","print(\"Forward Output:\", out.data)\n","print(\"Expected:\", expected_forward)\n","print(\"Match:\", np.allclose(out.data, expected_forward))\n","\n","print(\"\\nGradient wrt Input:\", x.grad)\n","print(\"Expected Gradient:\", expected_grad_x)\n","print(\"Match:\", np.allclose(x.grad, expected_grad_x))\n"],"metadata":{"id":"TNYJTcZ2cBhR","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1738992302424,"user_tz":300,"elapsed":48,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"86f3c199-65d3-4313-eeeb-421587dd5acc"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Node(_ | data=[-0.02 -0.01 0. 1. 2. ], grad=[1. 1. 1. 1. 1.])\n","Input nodes: []\n","Node(_ | data=[-0. -0. 0. 1. 2.], grad=[1. 1. 1. 1. 1.])\n","Input nodes: []\n","input node\n","Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0. 0. 1. 1. 1.])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0. 0. 1. 1. 1.])]\n","input node\n","Node(_ | data=[0. 0. 1. 1. 1.], grad=[-2. -1. 0. 1. 2.])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0. 0. 1. 1. 1.]), Node(_ | data=[0. 0. 1. 1. 1.], grad=[-2. -1. 0. 1. 2.])]\n","Node(_ | data=[-0.02 -0.01 0. 0. 0. ], grad=[1. 1. 1. 1. 1.])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01 0.01 1. 1. 1. ]), Node(_ | data=[0. 0. 1. 1. 1.], grad=[-2. -1. 0. 1. 2.])]\n","Node(_ | data=[0.01 0.01 0. 0. 0. ], grad=[-2. -1. 0. 1. 2.])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01 0.01 1. 1. 1. ]), Node(_ | data=[0. 0. 1. 1. 1.], grad=[-2. -1. 0. 1. 2.])]\n","input node\n","Node(_ | data=[0.01 0.01 0.01 0.01 0.01], grad=[-2. -1. 0. 0. 0.])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01 0.01 1. 1. 1. ]), Node(_ | data=[0. 0. 1. 1. 1.], grad=[-2. -1. 0. 1. 2.]), Node(_ | data=[0.01 0.01 0.01 0.01 0.01], grad=[-2. -1. 0. 0. 0.])]\n","input node\n","Node(_ | data=[1. 1. 0. 0. 0.], grad=[-0.02 -0.01 0. 0.01 0.02])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01 0.01 1. 1. 1. ]), Node(_ | data=[0. 0. 1. 1. 1.], grad=[-2. -1. 0. 1. 2.]), Node(_ | data=[0.01 0.01 0.01 0.01 0.01], grad=[-2. -1. 0. 0. 0.]), Node(_ | data=[1. 1. 0. 0. 0.], grad=[-0.02 -0.01 0. 0.01 0.02])]\n","Forward Output: [-0.02 -0.01 0. 1. 2. ]\n","Expected: [-0.02 -0.01 0. 1. 2. ]\n","Match: True\n","\n","Gradient wrt Input: [0.01 0.01 1. 1. 1. ]\n","Expected Gradient: [0.01 0.01 1. 1. 1. ]\n","Match: True\n"]}]},{"cell_type":"markdown","source":["#**Checkin \\#5**\n"],"metadata":{"id":"sMNeITGQu_60"}},{"cell_type":"markdown","source":["Question: How does autograd handle piecewise-defined functions (like Leaky ReLU)?\n","\n","Answer: Activation functions such as ReLU are piecewise (i.e., different definitions in different ranges of input).\n"," \n"],"metadata":{"id":"4ifQiMKIvAGC"}},{"cell_type":"markdown","source":["## **Meme \\#5**"],"metadata":{"id":"foNAIb0KOac3"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"dRkyGziMPNsM"}},{"cell_type":"markdown","source":["##**Tanh**"],"metadata":{"id":"KEwU_D-6OY2m"}},{"cell_type":"code","source":["# We'll be nice and give you a div_nodes and exp_nodes primitives :)\n","def div_node(a_node, b_node):\n"," out = Node(a_node.data / b_node.data, (a_node, b_node))\n"," def backward_fn():\n"," a_node.grad += out.grad / b_node.data\n"," b_node.grad += -out.grad * a_node.data / (b_node.data**2)\n"," out.backward_fn = backward_fn\n"," return out\n","\n","def exp_node(a_node):\n"," out = Node(np.exp(a_node.data), (a_node,))\n"," def backward_fn():\n"," a_node.grad += out.grad * out.data\n"," out.backward_fn = backward_fn\n"," return out"],"metadata":{"id":"-Qyf6kNnHOAg"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["\n","def tanh_node(x):\n"," \"\"\"\n"," tanh(x) = (e^x - e^-x) / (e^x + e^-x)\n"," \"\"\"\n"," #Negative One declaration\n"," neg_one = Node(-1.0 * np.ones_like(x.data)) # -1\n","\n"," neg_x = mul_node(neg_one, x) # -x\n","\n"," exp_x = exp_node(x) # e^x\n"," exp_neg_x = exp_node(neg_x) # e^-x\n"," neg_exp_neg_x = mul_node(exp_neg_x, neg_one) # -e^-x\n","\n"," # Numerator calculation\n"," num = add_node(exp_x, neg_exp_neg_x) # e^x - e^-x\n","\n","\n"," # Denominator calculation\n"," neg_input2 = mul_node(neg_one, x) # -x\n","\n"," exp_x2 = exp_node(x) # e^x\n"," exp_neg_x2 = exp_node(neg_input2) # e^-x\n","\n"," den = add_node(exp_x2, exp_neg_x2) # e^x + e^-x\n","\n","\n"," out = div_node(num, den) # (e^x - e^-x) / (e^x + e^-x)\n","\n"," return out\n"],"metadata":{"id":"DKwFL3hYb9mx"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["### Tests for Tanh "],"metadata":{"id":"JWNJJauKO0sN"}},{"cell_type":"code","source":["# Create a test input node\n","x = Node(np.array([-2.0, -1.0, 0.0, 1.0, 2.0])) # A mix of negative and positive values\n","\n","# Compute tanh using our function\n","out = tanh_node(x)\n","\n","# Expected forward values (using NumPy for verification)\n","expected_forward = np.tanh(x.data)\n","\n","# Set the output gradient (simulating dL/dout = 1 for all elements)\n","out.grad = np.ones_like(out.data)\n","\n","# Perform backpropagation\n","out.backward()\n","\n","# # Expected gradients: d/dx tanh(x) = 1 - tanh^2(x)\n","expected_grad_x = 1.0 - expected_forward**2 # (1 - tanh^2(x))\n","print()\n","\n","# Print results\n","print(\"Forward Output:\", out.data)\n","print(\"Expected:\", expected_forward)\n","print(\"Match:\", np.allclose(out.data, expected_forward))\n","\n","print(\"\\nGradient wrt Input:\", x.grad)\n","print(\"Expected Gradient:\", expected_grad_x)\n","print(\"Match:\", np.allclose(x.grad, expected_grad_x))\n"],"metadata":{"id":"iRa84bVocemb","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1738992302516,"user_tz":300,"elapsed":49,"user":{"displayName":"Michael Kireeff","userId":"05753936332644561694"}},"outputId":"00a2cec6-b0df-47b1-852e-20a8b6b7133a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["Node(out | data=[-0.96402758 -0.76159416 0. 0.76159416 0.96402758], grad=[1. 1. 1. 1. 1.])\n","Input nodes: []\n","Node(num | data=[-7.25372082 -2.35040239 0. 2.35040239 7.25372082], grad=[0.13290111 0.32402714 0.5 0.32402714 0.13290111])\n","Input nodes: []\n","Node(e^x | data=[0.13533528 0.36787944 1. 2.71828183 7.3890561 ], grad=[0.13290111 0.32402714 0.5 0.32402714 0.13290111])\n","Input nodes: []\n","input node\n","Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01798621 0.11920292 0.5 0.88079708 0.98201379])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01798621 0.11920292 0.5 0.88079708 0.98201379])]\n","Node(-e^-x | data=[-7.3890561 -2.71828183 -1. -0.36787944 -0.13533528], grad=[0.13290111 0.32402714 0.5 0.32402714 0.13290111])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01798621 0.11920292 0.5 0.88079708 0.98201379])]\n","Node(e^-x | data=[7.3890561 2.71828183 1. 0.36787944 0.13533528], grad=[-0.13290111 -0.32402714 -0.5 -0.32402714 -0.13290111])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.01798621 0.11920292 0.5 0.88079708 0.98201379])]\n","Node(-x | data=[ 2. 1. -0. -1. -2.], grad=[-0.98201379 -0.88079708 -0.5 -0.11920292 -0.01798621])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[1. 1. 1. 1. 1.])]\n","input node\n","Node(-1Node | data=[-1. -1. -1. -1. -1.], grad=[ 2.94604137 1.76159416 0.5 0. -0.01798621])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[1. 1. 1. 1. 1.]), Node(-1Node | data=[-1. -1. -1. -1. -1.], grad=[ 2.94604137 1.76159416 0.5 0. -0.01798621])]\n","Node(den | data=[7.52439138 3.08616127 2. 3.08616127 7.52439138], grad=[ 0.12812034 0.24677717 0. -0.24677717 -0.12812034])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[1. 1. 1. 1. 1.]), Node(-1Node | data=[-1. -1. -1. -1. -1.], grad=[ 2.94604137 1.76159416 0.5 0. -0.01798621])]\n","Node(e^x | data=[0.13533528 0.36787944 1. 2.71828183 7.3890561 ], grad=[ 0.12812034 0.24677717 0. -0.24677717 -0.12812034])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[1.0173392 1.09078425 1. 0.32919009 0.05331162]), Node(-1Node | data=[-1. -1. -1. -1. -1.], grad=[ 2.94604137 1.76159416 0.5 0. -0.01798621])]\n","Node(e^-x | data=[7.3890561 2.71828183 1. 0.36787944 0.13533528], grad=[ 0.12812034 0.24677717 0. -0.24677717 -0.12812034])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[1.0173392 1.09078425 1. 0.32919009 0.05331162]), Node(-1Node | data=[-1. -1. -1. -1. -1.], grad=[ 2.94604137 1.76159416 0.5 0. -0.01798621])]\n","Node(-x | data=[ 2. 1. -0. -1. -2.], grad=[ 0.94668838 0.67080991 0. -0.09078425 -0.0173392 ])\n","Input nodes: [Node(_ | data=[-2. -1. 0. 1. 2.], grad=[0.07065082 0.41997434 1. 0.41997434 0.07065082]), Node(-1Node | data=[-1. -1. -1. -1. -1.], grad=[ 1.05266461 1.09078425 0.5 -0.09078425 -0.05266461])]\n","\n","Forward Output: [-0.96402758 -0.76159416 0. 0.76159416 0.96402758]\n","Expected: [-0.96402758 -0.76159416 0. 0.76159416 0.96402758]\n","Match: True\n","\n","Gradient wrt Input: [0.07065082 0.41997434 1. 0.41997434 0.07065082]\n","Expected Gradient: [0.07065082 0.41997434 1. 0.41997434 0.07065082]\n","Match: True\n"]}]},{"cell_type":"markdown","source":["#**Checkin \\#6**\n"],"metadata":{"id":"b4tVd84pvKzF"}},{"cell_type":"markdown","source":["Question: What intermediate values do autograd frameworks store for\n","tanh?\n","\n","Answer: we can do $y = tanh(x)$ as the derivative of tanh is $1 - y^2$.\n"," "],"metadata":{"id":"05f1XsrJvMBR"}},{"cell_type":"markdown","source":["#**Meme \\#6**"],"metadata":{"id":"M0HaKxbyPm52"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"f_RHNrwfPpa0"}},{"cell_type":"markdown","source":["#**Meme \\#7**"],"metadata":{"id":"8q7KEuZAagP8"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"a6RxzBCIaggT"}}]}