[theano-users] Low GPU utilization for larger batches

[Ameretat Reith]

Hi,

I'm training Baidu's Deepspeech
<https://github.com/baidu-research/ba-dls-deepspeech> on a Nvidia 1080 Ti
and I realized on larger (46 at-most) batches GPU utilization is zero
nearly half of the time, but training time per input, continuously becomes
better by increasing batch size. It looks counter-intuitive to me and I
doubt whether I could do better with hardware or not.

To give a context about model:

- It's five GRU layers on top of a 1D convolution.
- It uses batch normalizations after each RNN and convolution
- Loss function is CTC implemented by Baidu's warp-ctc
<https://github.com/baidu-research/warp-ctc>

It seems GPU utilization is zero at start of each batch and during
`theano.function` execution. There is no CPU computation node in graph
computation as reported by `assert_no_cpu_op`. Mode is `FAST_RUN`. I
tweaked optimizers, SGD with nesterov and momentum had same overhead as
Adam.

Cuda version is 8.0, CuDNN is 5.1.10, libgpuarray is 0.6.5 and Theano is
0.9.0.

Thanks in advance, any hint is welcome

[Ameretat Reith]

Nevermind, turn out warp-ctc binding I'm using, just works with CPU

[Frédéric Bastien]

We merged in Theano dev the CTC wrapper. I would recommend that you update
to it.

Post by Ameretat Reith
Nevermind, turn out warp-ctc binding I'm using, just works with CPU
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.