[theano-users] theano ctc loss underflow

[Fei Tao]

Hi, dear all,

Recently, I ran experiment with CTC loss, which is based on theano-ctc
(Theano binding of Baidu's warp-ctc library). I use the default SGD
optimizer. My data timestep is about 3500, the truth length is not longer
than 639. All the input sequences are zero padded to the timestep, while
the label are padded with -1. Based on my understanding, the theano-ctc
will not compute loss for negative value, while the keras masking will
filter out zero-padding input. My batch size is 32, inupt dimension is 30,
output dimension is 63.

In the first epoch, everything is fine. However when it goes into second
epoch, I observe underflow (loss is very very very small displaying as 0).
Baidu claims that they have optimize the stability by using log value for
loss. However, I still have this problem. Does anyone know how to solve
this problem?

Thank you very much!