[theano-users] SUDDENLY: CUDA is installed, but device gpu0 is not available

Discussion:

Kin

2014-12-09 02:42:19 UTC

Hello all,

A few months ago Theano was working fine on my machine. However, this
morning I woke up to theano giving me these errors when I try to run some
codes (they used to work without issues!):

ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc

return status', 1, 'for cmd', 'nvcc -shared -O3 -use_fast_math -m64
-Xcompiler -DCUDA_NDARRAY_CUH=f6d896860391dd5093b721c63a0b2b33,-D
NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC -Xlinker
-rpath,/home/microway/.theano/compiledir_Linux-3.13.0-40-generic-x86_64-with-debian-jessie-sid-x86_64-2.7.8-64/cuda_ndarray
-I/home/microway/Theano/theano/sandbox/cuda
-I/home/microway/anaconda/lib/python2.7/site-packages/numpy/core/include
-I/home/microway/anaconda/include/python2.7 -o
/home/microway/.theano/compiledir_Linux-3.13.0-40-generic-x86_64-with-debian-jessie-sid-x86_64-2.7.8-64/cuda_ndarray/cuda_ndarray.so
mod.cu -L/home/microway/anaconda/lib -lpython2.7 -lcublas -lcudart')
*WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not
available (error: cuda unavilable)*

I think I might have accidentally updated something on Ubuntu (as Ubuntu
kept nagging me to update softwares and things like that) and that broke my
Cuda or Theano. Here are a few points I want to mention:

1) nvidia-smi is working properly and giving me correct outputs.
2) I have ran .deviceQuery from the CUDA samples directory and the system
is able to see my two GPUs.
3) I tried uninstalling all NVIDIA-related things and reinstalling the
drivers/Cuda 6.5, but no luck.
4) I also tried to reinstall Anaconda because that was how I used to
install Theano in the first place.

Any ideas? I am using Ubuntu 14.04, NVCC version 6.5, Python version 2.7.8,
Anaconda 2.1.0, Cuda 6.5, and NVIDIA driver version 340.29.

Thanks!

Kin

--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Frédéric Bastien

2014-12-09 19:12:25 UTC

Permalink

Hi,

This is strange. My first guess is that this updated the cuda version and
for some unknow reason this cause problem with theano cache.

Can you try to clear theano cache?

theano-cache purge

thean-cache is under Theano/bin.

Fred

Post by Kin
Hello all,
A few months ago Theano was working fine on my machine. However, this
morning I woke up to theano giving me these errors when I try to run some
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc

I think I might have accidentally updated something on Ubuntu (as Ubuntu
kept nagging me to update softwares and things like that) and that broke my
1) nvidia-smi is working properly and giving me correct outputs.
2) I have ran .deviceQuery from the CUDA samples directory and the system
is able to see my two GPUs.
3) I tried uninstalling all NVIDIA-related things and reinstalling the
drivers/Cuda 6.5, but no luck.
4) I also tried to reinstall Anaconda because that was how I used to
install Theano in the first place.
Any ideas? I am using Ubuntu 14.04, NVCC version 6.5, Python version
2.7.8, Anaconda 2.1.0, Cuda 6.5, and NVIDIA driver version 340.29.
Thanks!
Kin
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

Kin

2014-12-09 19:46:59 UTC

Permalink

Hello, thanks for your response. If the updated Cuda version was causing
problems, it might affect other theano users as well doesn't it?

I did try to clear the cache, and the output is the same. It displayed a
long string of codes in the terminal before eventually arriving at:

"*WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not

available (error: cuda unavilable)"*

It seems like Theano wasn't able to use my GPU for some reason. I was able
to run most CUDA samples which utilizes my GPU.

I have an old Anaconda directory. If I instruct python to use that
directory and when theano runs a code, I was able to get the message
similar to "using gpu0..." at the beginning. Eventually it will fail when
computations need to be done. So I renamed the old anaconda directory to
"anaconda_old", then reinstalled anaconda. This time, whenever Theano runs,
I will see the aforementioned error message which implies that it couldn't
use the GPU.

I am running out of ideas now... I've even tried the bleeding edge version
of Theano...

Kin

Hi,
This is strange. My first guess is that this updated the cuda version and
for some unknow reason this cause problem with theano cache.
Can you try to clear theano cache?
theano-cache purge
thean-cache is under Theano/bin.
Fred

Kin

2014-12-09 23:20:53 UTC

Permalink

*UPDATE: Issue resolved!*

Ubuntu updated my g++ to 4.9 whereas the nvcc does not work with g++ 4.9.
Fixed it by changing gcc to link to version 4.8 instead.

Disabling system updates now!

Kin

Frédéric Bastien

2014-12-14 03:53:20 UTC

Permalink

Thanks for the solution. It will probably help other people.

Fred

Post by Kin
*UPDATE: Issue resolved!*
Ubuntu updated my g++ to 4.9 whereas the nvcc does not work with g++ 4.9.
Fixed it by changing gcc to link to version 4.8 instead.
Disabling system updates now!
Kin
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

Ramana Subramanyam

2016-05-25 20:15:15 UTC

Permalink

Hey there. I was also stuck with the same problem this evening. I have
installed gcc-4.8 and my gcc -v shows the new compiler's path. Do i need to
update nvcc with the new gcc compiler ? Even after downgrading to gcc 4.8,
I am getting that error. Would be glad for your support.

Thanks,
Ramana

Frédéric Bastien

2016-05-26 12:33:23 UTC

Permalink

nvcc have a parameter to tell him to use a different g++ then the default
one. You can have it passed to nvcc with this Theano flag:

nvcc.compiler_bindir

Give it the directory where g++/gcc is installed.

Fred

Post by Ramana Subramanyam
Hey there. I was also stuck with the same problem this evening. I have
installed gcc-4.8 and my gcc -v shows the new compiler's path. Do i need to
update nvcc with the new gcc compiler ? Even after downgrading to gcc 4.8,
I am getting that error. Would be glad for your support.
Thanks,
Ramana

--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

Xingxing Zhang

2015-04-21 20:38:02 UTC

Permalink

I also went into this error today and it was solved by using
theano-cache clear
I think theano-cache purge should also work

åš 2014å¹Ž12æ9æ¥ææäº UTCäžå7:12:47ïŒnouizåéïŒ

Post by FrÃ©dÃ©ric Bastien
Hi,
This is strange. My first guess is that this updated the cuda version and
for some unknow reason this cause problem with theano cache.
Can you try to clear theano cache?
theano-cache purge
thean-cache is under Theano/bin.
Fred

Jan Vojt

2016-04-06 20:29:43 UTC

Permalink

Theano stopped working for me as well after upgrade to gcc 5. It is not the
fault of Theano though, but NVIDIA not supporting the new version. Theano
is picking up the new gcc version, which is the default in my system. So
you just need to point it to the older version. I resolved with the below:

sudo ln -s /usr/bin/gcc-4.4 /usr/local/cuda/bin/gcc

Post by Xingxing Zhang
I also went into this error today and it was solved by using
theano-cache clear
I think theano-cache purge should also work
åš 2014å¹Ž12æ9æ¥ææäº UTCäžå7:12:47ïŒnouizåéïŒ

I think I might have accidentally updated something on Ubuntu (as Ubuntu
kept nagging me to update softwares and things like that) and that broke my
1) nvidia-smi is working properly and giving me correct outputs.
2) I have ran .deviceQuery from the CUDA samples directory and the
system is able to see my two GPUs.
3) I tried uninstalling all NVIDIA-related things and reinstalling the
drivers/Cuda 6.5, but no luck.
4) I also tried to reinstall Anaconda because that was how I used to
install Theano in the first place.
Any ideas? I am using Ubuntu 14.04, NVCC version 6.5, Python version
2.7.8, Anaconda 2.1.0, Cuda 6.5, and NVIDIA driver version 340.29.
Thanks!
Kin
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.