[theano-users] Need help with Debugging Memory Issues

[Abbhinav Venkat]

Hi,

I've been facing memory issues for the past few weeks with my code. I have
a few general doubts/requests regarding the working of Theano -

1. How do I find the number of parameters for my model? It is a model with
custom designed layers, and built on top of the 3D-R2N2
(https://github.com/chrischoy/3D-R2N2/)
In these some pre-existing function (like model.count_params() in Keras),
or could someone help me out in writing it? I have seen the "pydotprint"
function, but, I need something that is easier to understand like the
count_params() or model.summary() in Keras.

2. I need to find out the GPU usage of my model. I read somewhere that
Theano allocates the whole of the GPU memory for the process, at first, and
then scales down depending on usage. How does it exactly work?

3. Based on the tests I ran on the R2N2 code, it "appears" as though my
WHOLE dataset gets loaded into the GPU memory and then it splits into
batches for training and validation. Since my dataset is huge, this is
highly inefficient and leads to an error requesting for more memory.
Although it my intuition that this is what is happening, I am unable to
prove it. Based on the documentation, it is only the "theano.shared",
"gpuArray" and certain frequently accessed variables that go into the GPU
memory. The size of my weights file is about 250 MB, and I am using a 11GB
1080Ti to train the model. I shouldn't be getting memory allocation errors.

If I can print the params, and find out how much of the data is occupied by
them, I can prove that it is indeed my data that is getting loaded into the
GPU memory and is causing issues.

4. Moreover, can someone tell me that if it is customary for theano to load
ALL of the the data (from my dataset) into the GPU memory? If so, how do I
avoid it? I don't mind slowing it down by making multiple disk I/Os, as
long as it fits in the memory and runs.

5. In Theano, specifically, and in any library, in general, what goes into
the system RAM, and what into the GPU RAM? My system RAM usage for the
above seems to be close to 50GB. While that is not a bottleneck now, I'd
like to reduce it, all the same.

I know I'm asking a lot. But, I couldn't find any resource that explains
all of the above.

Thanks,

Abbhinav


P.S: I'm posting a snapshot of my memory error
here: https://pastebin.com/FJy11p3s