RNN Model and Input Sequence

RNN (Recurren Neural Networks) are special type of neural networks designed for processing sequential data. Above diagram shows the basic structure of RNN with inputs x(t-1), x(t), x(t+1) at time t-1, t, t+1 and output O(t-1), O(t), O(t+1) at the corresponding time. Here input and output are sequenced data at different times.

Input at different timestamps can be a features extracted from a data which is provided at different timestamps. For example, in case of Natural Language processing, input will be feature matrix of words from sentence. Suppose we have a sentence, “RNN are great at processing sequence data” Then the input will feature representation of words, “RNN”, “are”, “great”, “at”, “processing”, “sequence”, “data”. Feature is nothing but a vector maping of those words or the input at different timestamps.

• x[0] => RNN => [1.5, 2.3, 7.8]
• x[1] => are => [8.2, 9.4, 17.0]
• x[2] => great => [8.1, 6.7, 89.0]
• x[3] => at => [6.8, 8.45, 9.0]
• x[4] => processing => [6.7, 8.2, 7.8]
• x[5] => sequence => [5.7, 7.2, 9.8]
• x[6] => data => [2.7, 0.2, 6.8]

Mapping of those words to vector is called as Embeding. There are various techniques by which a word can be converted into those feature mapping (Embedings). One of such model is Word2Vect which is a neutal network designed to generate a embeding values from a word. So given a word from a sentence, it mapps each word to a vector representation.

Computing Activation Value And For Forward Propagation

So far, we have seen how input and output of RNN is modeled at different timestamps. Now, lets see how outputs O(t) and h(t) are calculated.

Here,

• h(t-1) represents state of hidden unit at time t-1
• h(t) represents sttae of hidden unit at time t
• Whh => Connection weight from hidden unit state at t-1 to t.
• Whx => Connection Weight from input to hidden layer.
• Who => Connection Weight from hidden to output layer.

Weight represents a strength of connection from different components (input, output, previous state). These values represents how much weight does each connection carries. Those values are initially initialized with a random values and as model evolves through learning, its values are optimized or basically corrected to represent a mapping from input to output. To calculate different values of i.e. h(t) and x(t) we can refer the model below.

Hidden State Value

To calculate the value of hidden state at time t, we look at the left side of the image.

Given different weights, previous hidden state and current input, hidden unit bias, we can calculate next hidden state at time t as below:

Given, current state of hidden unit, now we can compute the current output o(t) at time t as:

Thus given weights, input at time t, biases of different unit. We have a mathematical formulation below to calculate output and hidden state values at current time t.