[theano-users] Implementing RNN Layers with LSTM

[Sunjeet Jena]

*So this is my code for creating the RNN with LSTM:*

class RNN(object):

def __init__(self, INPUT,input_units, output_units):

self.input=INPUT #Input
parameters/features
self.input_layer_units= input_units
#Number of Input units
self.output_layer_units=output_units #Number
of Output Units



W_XH=(np.array(np.random.rand(self.input_layer_units,self.output_layer_units),
dtype=theano.config.floatX)) #Weights for
Connection from Previous Layer to Current Layer

self.W_XH=shared(value=W_XH,name='Weight of Previous Hidden Layer
to Current Hidden in RNN',borrow=True)

B_XH=(np.array(np.zeros(self.output_layer_units),
dtype=theano.config.floatX))

self.B_XH=shared(value=B_XH,name='Bias of Previous Hidden Layer to
Current Hidden in RNN', borrow=True)
#Initializing the Bias
Units

W_HH= (np.array(np.random.rand(self.output_layer_units,
self.output_layer_units), dtype=theano.config.floatX))

self.W_HH=shared(value=W_HH,name='Weight of Previous Time Frame of
the Hidden Layer to Current Time frame hidden layer in RNN',borrow=True)

#Weights for Connectiof=n through current time unit and
previous time unit

B_HH=(np.array(np.zeros(self.output_layer_units),
dtype=theano.config.floatX))

self.B_HH=shared(value=B_HH,name='Bias of Previous Time Frame of
the Hidden Layer to Current Time frame hidden layer in RNN',
borrow=True)
#Initializing the Bias Units

hidden_layers_output=(np.array(np.zeros(self.output_layer_units,),
dtype=theano.config.floatX))

self.hidden_layers_output=shared(value=hidden_layers_output,
name='hidden_layers_output', borrow=True)

c_t_1=0.0

self.c_t_1=shared(value=c_t_1, name='c_t_1', borrow=True)

self.z= (tensor.dot(self.input, self.W_XH) +
tensor.dot(self.hidden_layers_output, self.W_HH) + self.B_XH +
self.B_HH) #Output of Hidden Layers Before Going Through LSTM

""" Passing Through LSTM Cell """

self.I=tensor.nnet.sigmoid(self.z)
self.F=tensor.nnet.sigmoid(self.z)
self.O=tensor.nnet.sigmoid(self.z)
self.G=tensor.tanh(self.z)

self.c_t= (self.F*self.c_t_1) + (self.I*self.G)
#Cell output
self.h_t= self.O*(tensor.tanh(self.c_t))
#Output of Hidden Layer after passing through LSTM

self.output=self.h_t


self.parameters=tensor.concatenate([self.W_XH.flatten(1),self.B_XH.flatten(1),self.W_HH.flatten(1),self.B_HH.flatten(1)])


self.c_t_1= self.c_t

self.hidden_layers_output=self.h_t

*After creating the class I am calling the Class for 'N' times (where 'N'
is the number of Hidden Layers) and storing then parameters in another
class and later on taking the gradient of the cost w.r.t. parameters with
the option disconnected_inputs='warn'. Is this the right implementation ?
Please help.*

[Ramana Subramanyam]

Hi,
There is a verbose tutorial on LSTM here ,
http://deeplearning.net/tutorial/lstm.html . And code is here,
http://deeplearning.net/tutorial/code/lstm.py . Please refer to them

Cheers,
Ramana

Post by Sunjeet Jena
*So this is my code for creating the RNN with LSTM:*
self.input=INPUT #Input
parameters/features
self.input_layer_units= input_units
#Number of Input units
self.output_layer_units=output_units
#Number of Output Units
W_XH=(np.array(np.random.rand(self.input_layer_units,self.output_layer_units),
dtype=theano.config.floatX)) #Weights for
Connection from Previous Layer to Current Layer
self.W_XH=shared(value=W_XH,name='Weight of Previous Hidden Layer
to Current Hidden in RNN',borrow=True)
B_XH=(np.array(np.zeros(self.output_layer_units),
dtype=theano.config.floatX))
self.B_XH=shared(value=B_XH,name='Bias of Previous Hidden Layer to
Current Hidden in RNN', borrow=True)
#Initializing the Bias Units
W_HH= (np.array(np.random.rand(self.output_layer_units,
self.output_layer_units), dtype=theano.config.floatX))
self.W_HH=shared(value=W_HH,name='Weight of Previous Time Frame of
the Hidden Layer to Current Time frame hidden layer in RNN',borrow=True)
#Weights for Connectiof=n through current time unit and
previous time unit
B_HH=(np.array(np.zeros(self.output_layer_units),
dtype=theano.config.floatX))
self.B_HH=shared(value=B_HH,name='Bias of Previous Time Frame of
the Hidden Layer to Current Time frame hidden layer in RNN',
borrow=True)
#Initializing the Bias Units
hidden_layers_output=(np.array(np.zeros(self.output_layer_units,),
dtype=theano.config.floatX))
self.hidden_layers_output=shared(value=hidden_layers_output,
name='hidden_layers_output', borrow=True)
c_t_1=0.0
self.c_t_1=shared(value=c_t_1, name='c_t_1', borrow=True)
self.z= (tensor.dot(self.input, self.W_XH) +
tensor.dot(self.hidden_layers_output, self.W_HH) + self.B_XH +
self.B_HH) #Output of Hidden Layers Before Going Through LSTM
""" Passing Through LSTM Cell """
self.I=tensor.nnet.sigmoid(self.z)
self.F=tensor.nnet.sigmoid(self.z)
self.O=tensor.nnet.sigmoid(self.z)
self.G=tensor.tanh(self.z)
self.c_t= (self.F*self.c_t_1) + (self.I*self.G)
#Cell output
self.h_t= self.O*(tensor.tanh(self.c_t))
#Output of Hidden Layer after passing through LSTM
self.output=self.h_t
self.parameters=tensor.concatenate([self.W_XH.flatten(1),self.B_XH.flatten(1),self.W_HH.flatten(1),self.B_HH.flatten(1)])
self.c_t_1= self.c_t
self.hidden_layers_output=self.h_t
*After creating the class I am calling the Class for 'N' times (where 'N'
is the number of Hidden Layers) and storing then parameters in another
class and later on taking the gradient of the cost w.r.t. parameters with
the option disconnected_inputs='warn'. Is this the right implementation ?
Please help.*