[theano-users] theano0.9 and cuda-8

[n***@interactions.com]

I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
I have:
theano version 0.9

CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...

Essentially I am using some parts of cuda-8 and some of cuda-7.5.

With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.

Thanks
--
*******************************************************************************

This e-mail and any of its attachments may contain Interactions Corporation
proprietary information, which is privileged, confidential, or subject to
copyright belonging to the Interactions Corporation. This e-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.

*******************************************************************************

[Frédéric Bastien]

You should not mix cuda version...

Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?

Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[n***@interactions.com]

in my .theanorc file I have device = gpu

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
*******************************************************************************

This e-mail and any of its attachments may contain Interactions Corporation
proprietary information, which is privileged, confidential, or subject to
copyright belonging to the Interactions Corporation. This e-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.

*******************************************************************************

[n***@interactions.com]

I changed it to cuda*, thanks, everything is OK now.

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
*******************************************************************************

This e-mail and any of its attachments may contain Interactions Corporation
proprietary information, which is privileged, confidential, or subject to
copyright belonging to the Interactions Corporation. This e-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.

*******************************************************************************

[n***@interactions.com]

Changing it to cuda* results is CPU usage and not GPU

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
*******************************************************************************

This e-mail and any of its attachments may contain Interactions Corporation
proprietary information, which is privileged, confidential, or subject to
copyright belonging to the Interactions Corporation. This e-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.

*******************************************************************************

[Pascal Lamblin]

How did you determine it is using the CPU?

Post by n***@interactions.com
Changing it to cuda* results is CPU usage and not GPU

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

[n***@interactions.com]

I found out that it is use CPU by running the program on a machine where
all GUPs are already in use.
On our setup if all GPUs are already in use the 'import theano' causes the
program to crash.
When I remove the * from cuda* it does use GPU, but then the theano
functions
do not compile and dumps long header files and error descriptions on the
screen.
My suspicion is that this may be due to the fact I am using Lasagne
libarary that
seems not to be compatible with the new backend.
I am OK using the old backend but it does not work with cuda-8. import
theano
gives me the following warning
-------------------------------
Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is enabled with initial
size: 95.0% of memory, cuDNN 6021)
/var/local/miniconda2/lib/python2.7/site-packages/theano/sandbox/cuda/__init__.py:631:
UserWarning: Your cuDNN version is more recent than the one Theano
officially supports. If you see any problems, try updating Theano or
downgrading cuDNN to version 5.1.
--------------------------------
And once again theano functions do not compile. Long error messages with
some header files are dumped as part of error.
However when I the following settings

CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...

Thing work but after certain number of training epochs (reducing the error
nicely)
all training parameters suddenly become nan. I have a complex network using
4 Bi-GRUs, Conv1DLayer,
MaxPool1DLayer from lasagne and some attention layers I implemented.
I have spent numerous hours trying to make sure gradients remain bounded,
by using 'theano.gradient.grad_clip'
at various stages of computation and by using
'lasagne.updates.norm_constraint' and still have
not been able to pin down the cause of parameters suddenly becoming 'nan'.

I just would like to be sure that this is not happening because of my usage
of cuda-8 and cuda-7.5 at
the same time.
I appreciate your help in resolving this.

Post by Pascal Lamblin
How did you determine it is using the CPU?

Post by n***@interactions.com
Changing it to cuda* results is CPU usage and not GPU

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
*******************************************************************************

This e-mail and any of its attachments may contain Interactions Corporation
proprietary information, which is privileged, confidential, or subject to
copyright belonging to the Interactions Corporation. This e-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.

*******************************************************************************

[Frédéric Bastien]

You are mixing cuda 7.5 and 8 in your end variable. Never do this. Delete
your Theano cache. It is not safe to reuse it.

The new backend support cuda 8. Your warning is about cudnn version. This
is not the same.

If in a clean environment,you still have problems, give the error you have.

Fred

Post by n***@interactions.com
I found out that it is use CPU by running the program on a machine where
all GUPs are already in use.
On our setup if all GPUs are already in use the 'import theano' causes the
program to crash.
When I remove the * from cuda* it does use GPU, but then the theano
functions
do not compile and dumps long header files and error descriptions on the
screen.
My suspicion is that this may be due to the fact I am using Lasagne
libarary that
seems not to be compatible with the new backend.
I am OK using the old backend but it does not work with cuda-8. import
theano
gives me the following warning
-------------------------------
Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is enabled with initial
size: 95.0% of memory, cuDNN 6021)
UserWarning: Your cuDNN version is more recent than the one Theano
officially supports. If you see any problems, try updating Theano or
downgrading cuDNN to version 5.1.
--------------------------------
And once again theano functions do not compile. Long error messages with
some header files are dumped as part of error.
However when I the following settings
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Thing work but after certain number of training epochs (reducing the error
nicely)
all training parameters suddenly become nan. I have a complex network
using 4 Bi-GRUs, Conv1DLayer,
MaxPool1DLayer from lasagne and some attention layers I implemented.
I have spent numerous hours trying to make sure gradients remain bounded,
by using 'theano.gradient.grad_clip'
at various stages of computation and by using
'lasagne.updates.norm_constraint' and still have
not been able to pin down the cause of parameters suddenly becoming 'nan'.
I just would like to be sure that this is not happening because of my
usage of cuda-8 and cuda-7.5 at
the same time.
I appreciate your help in resolving this.

Post by Pascal Lamblin
How did you determine it is using the CPU?

Post by n***@interactions.com
Changing it to cuda* results is CPU usage and not GPU

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new back-end
(device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[n***@interactions.com]

I set
CUDA_ROOT =/usr/local/cuda-8
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
delete .theano dir
and ran my program. I get the errors as shown in the attached file.
The error takes place when theano.function() is called.
My .theanorc file looks like
--------------------
[global]
floatX = float32
device = gpu
exception_verbosity = high

[nvcc]
optimizer_including=alloc_empty_to_zeros
use_fast_math=False

[lib]
cnmem = 1

#gpuarray.preallocate = 1
------------------------
I am using old backend, because it seems Lasagne is not compatible with new
backend.

What am I doing wrong?

Post by Frédéric Bastien
You are mixing cuda 7.5 and 8 in your end variable. Never do this. Delete
your Theano cache. It is not safe to reuse it.
The new backend support cuda 8. Your warning is about cudnn version. This
is not the same.
If in a clean environment,you still have problems, give the error you have.
Fred

Post by n***@interactions.com
I found out that it is use CPU by running the program on a machine where
all GUPs are already in use.
On our setup if all GPUs are already in use the 'import theano' causes
the program to crash.
When I remove the * from cuda* it does use GPU, but then the theano
functions
do not compile and dumps long header files and error descriptions on the
screen.
My suspicion is that this may be due to the fact I am using Lasagne
libarary that
seems not to be compatible with the new backend.
I am OK using the old backend but it does not work with cuda-8. import
theano
gives me the following warning
-------------------------------
Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is enabled with initial
size: 95.0% of memory, cuDNN 6021)
UserWarning: Your cuDNN version is more recent than the one Theano
officially supports. If you see any problems, try updating Theano or
downgrading cuDNN to version 5.1.
--------------------------------
And once again theano functions do not compile. Long error messages with
some header files are dumped as part of error.
However when I the following settings
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Thing work but after certain number of training epochs (reducing the
error nicely)
all training parameters suddenly become nan. I have a complex network
using 4 Bi-GRUs, Conv1DLayer,
MaxPool1DLayer from lasagne and some attention layers I implemented.
I have spent numerous hours trying to make sure gradients remain bounded,
by using 'theano.gradient.grad_clip'
at various stages of computation and by using
'lasagne.updates.norm_constraint' and still have
not been able to pin down the cause of parameters suddenly becoming 'nan'.
I just would like to be sure that this is not happening because of my
usage of cuda-8 and cuda-7.5 at
the same time.
I appreciate your help in resolving this.

Post by Pascal Lamblin
How did you determine it is using the CPU?

Post by n***@interactions.com
Changing it to cuda* results is CPU usage and not GPU

Post by Frédéric Bastien
You should not mix cuda version...
Do you still use the old gpu back-end (device=gpu*) or the new
back-end (device=cuda*)?
Fred

Post by n***@interactions.com
I trying to understand some unexplained behavior of my code.
To be sure that the problem is with my code and not with software
incompatibility I would like to sure about the correctness of my setup
theano version 0.9
CUDA_ROOT =/usr/local/cuda-7.5
LD_PATH=/usr/local/cuda-8/lib64:...
PATH=/usr/local/cuda-8/bin: ...
Essentially I am using some parts of cuda-8 and some of cuda-7.5.
With CUDA_ROOT =/usr/local/cuda-8, I cannot compile the theano functions.
Thanks
*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it,
For more options, visit https://groups.google.com/d/optout.

*******************************************************************************
This e-mail and any of its attachments may contain Interactions
Corporation proprietary information, which is privileged, confidential, or
subject to copyright belonging to the Interactions Corporation. This e-mail
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.
*******************************************************************************
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
*******************************************************************************

This e-mail and any of its attachments may contain Interactions Corporation
proprietary information, which is privileged, confidential, or subject to
copyright belonging to the Interactions Corporation. This e-mail is
intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient of this e-mail, you are
hereby notified that any dissemination, distribution, copying, or action
taken in relation to the contents of and attachments to this e-mail is
strictly prohibited and may be unlawful. If you have received this e-mail
in error, please notify the sender immediately and permanently delete the
original and any copy of this e-mail and any printout. Thank You.

*******************************************************************************