Discussion:
[theano-users] GpuCorrMM encountered a CUBLAS error
Beatriz G.
2017-12-13 12:50:44 UTC
Permalink
Hi everyone.

I used to work with Theano and it works perfectly, but after installing
tensorflow with conda, and some dependencies to work with it, my Theano has
stopped to work.

I obtain the following error:

Using gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled, cuDNN not
available)
... loading data
... building the model
... training
training @ iter = 0
Traceback (most recent call last):
File "LENET.py", line 394, in <module>
evaluate_lenet5()
File "LENET.py", line 301, in evaluate_lenet5
cost_ij = train_model(minibatch_index)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line
314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 859, in __call__
outputs = self.fn()
RuntimeError: GpuCorrMM encountered a CUBLAS error: the library was not
initialized
This could be a known bug in CUDA, please see the GpuCorrMM() documentation.

Apply node that caused the error: GpuCorrMM_gradWeights{valid, (1,
1)}(GpuContiguous.0, GpuContiguous.0)
Toposort index: 28
Inputs types: [CudaNdarrayType(float32, (True, False, False, False)),
CudaNdarrayType(float32, 4D)]
Inputs shapes: [(1, 500, 28, 28), (1, 20, 5, 5)]
Inputs strides: [(0, 784, 28, 1), (0, 25, 5, 1)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[GpuDimShuffle{1,0,2,3}(GpuCorrMM_gradWeights{valid, (1,
1)}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a
back-trace of when this node was created. This can be done with by setting
the Theano flag 'optimizer=fast_compile'. If that does not work, Theano
optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and
storage map footprint of this apply node.

I have tried to install cuda and cudnn, but it does not work (apart from
the toolkit that i had already installed)

My theanorc file looks like:
[global]
device = gpu
floatX = float32


[blas]
ldflags = -lopenblas


[nvcc]
flags=-D_FORCE_INLINES


I would appreciate any advice or help.

Regards.

Beatriz.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Beatriz G.
2017-12-13 14:20:02 UTC
Permalink
After trying a lot of thigns, I have decided to uninstall and install
theano, and a new version has installed, the new version requires cuda, so
my theanorc file is now like:

[global]
device = cuda
floatX = float32


[blas]
ldflags = -lopenblas


[nvcc]
# flags=-D_FORCE_INLINES
optimizer_including=cudnn

[cuda]
root=/usr/local/cuda-9.1


And I get the following output after trying Lenet:


Using cuDNN version 7005 on context None
Mapped name None to device cuda: GeForce GTX 750 Ti (0000:06:00.0)
... loading data
... building the model
LENET.py:108: UserWarning: DEPRECATION: the 'ds' parameter is not going to
exist anymore as it is going to be replaced by the parameter 'ws'.
ignore_border=True
Traceback (most recent call last):
File "LENET.py", line 394, in <module>
evaluate_lenet5()
File "LENET.py", line 228, in evaluate_lenet5
y: test_set_y[index * batch_size: (index + 1) * batch_size]
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function.py",
line 317, in function
output_keys=output_keys)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py",
line 486, in pfunc
output_keys=output_keys)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 1841, in orig_function
fn = m.create(defaults)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 1715, in create
input_storage=input_storage_lists, storage_map=storage_map)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line
699, in make_thunk
storage_map=storage_map)[:3]
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/vm.py",
line 1084, in make_all
impl=impl))
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/op.py",
line 955, in make_thunk
no_recycling)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/op.py",
line 858, in make_c_thunk
output_storage=node_output_storage)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1217, in make_thunk
keep_lock=keep_lock)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1157, in __compile__
keep_lock=keep_lock)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1620, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py",
line 1174, in module_from_key
module = lnk.compile_cmodule(location)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1523, in compile_cmodule
preargs=preargs)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py",
line 2368, in compile_str
return dlimport(lib_filename)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py",
line 302, in dlimport
rval = __import__(module_name, {}, {}, [module_name])
ImportError: ('The following error happened while compiling the node',
GpuDnnConv{algo='small', inplace=True, num_groups=1}(GpuContiguous.0,
GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0,
GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), dilation=(1, 1),
conv_mode='conv', precision='float32', num_groups=1}.0, Constant{1.0},
Constant{0.0}), '\n',
'/home/bea/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.12-64/tmpPD9sEN/97ac95f817846a3cb0867215657bdc2150272dcddf165864039b936dd3b77309.so:
undefined symbol: cudnnGetConvolutionGroupCount',
"[GpuDnnConv{algo='small', inplace=True,
num_groups=1}(<GpuArrayType<None>(float32, (False, True, False, False))>,
<GpuArrayType<None>(float32, 4D)>, <GpuArrayType<None>(float32, 4D)>,
<CDataType{cudnnConvolutionDescriptor_t}>, Constant{1.0}, Constant{0.0})]")


Regards.

El miércoles, 13 de diciembre de 2017, 13:50:44 (UTC+1), Beatriz G.
Post by Beatriz G.
Hi everyone.
I used to work with Theano and it works perfectly, but after installing
tensorflow with conda, and some dependencies to work with it, my Theano has
stopped to work.
Using gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled, cuDNN not
available)
... loading data
... building the model
... training
File "LENET.py", line 394, in <module>
evaluate_lenet5()
File "LENET.py", line 301, in evaluate_lenet5
cost_ij = train_model(minibatch_index)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line
314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 859, in __call__
outputs = self.fn()
RuntimeError: GpuCorrMM encountered a CUBLAS error: the library was not
initialized
This could be a known bug in CUDA, please see the GpuCorrMM()
documentation.
Apply node that caused the error: GpuCorrMM_gradWeights{valid, (1,
1)}(GpuContiguous.0, GpuContiguous.0)
Toposort index: 28
Inputs types: [CudaNdarrayType(float32, (True, False, False, False)),
CudaNdarrayType(float32, 4D)]
Inputs shapes: [(1, 500, 28, 28), (1, 20, 5, 5)]
Inputs strides: [(0, 784, 28, 1), (0, 25, 5, 1)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[GpuDimShuffle{1,0,2,3}(GpuCorrMM_gradWeights{valid, (1,
1)}.0)]]
HINT: Re-running with most Theano optimization disabled could give you a
back-trace of when this node was created. This can be done with by setting
the Theano flag 'optimizer=fast_compile'. If that does not work, Theano
optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and
storage map footprint of this apply node.
I have tried to install cuda and cudnn, but it does not work (apart from
the toolkit that i had already installed)
[global]
device = gpu
floatX = float32
[blas]
ldflags = -lopenblas
[nvcc]
flags=-D_FORCE_INLINES
I would appreciate any advice or help.
Regards.
Beatriz.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Frédéric Bastien
2018-01-10 21:34:22 UTC
Permalink
Do you have multiple cudnn version installed? I have the impression Theano
is in an environment with multiple cudnn version available.

Can you delete your Theano cache? This could also help.

theano-cache purge
Post by Beatriz G.
After trying a lot of thigns, I have decided to uninstall and install
theano, and a new version has installed, the new version requires cuda, so
[global]
device = cuda
floatX = float32
[blas]
ldflags = -lopenblas
[nvcc]
# flags=-D_FORCE_INLINES
optimizer_including=cudnn
[cuda]
root=/usr/local/cuda-9.1
Using cuDNN version 7005 on context None
Mapped name None to device cuda: GeForce GTX 750 Ti (0000:06:00.0)
... loading data
... building the model
LENET.py:108: UserWarning: DEPRECATION: the 'ds' parameter is not going to
exist anymore as it is going to be replaced by the parameter 'ws'.
ignore_border=True
File "LENET.py", line 394, in <module>
evaluate_lenet5()
File "LENET.py", line 228, in evaluate_lenet5
y: test_set_y[index * batch_size: (index + 1) * batch_size]
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function.py",
line 317, in function
output_keys=output_keys)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py",
line 486, in pfunc
output_keys=output_keys)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 1841, in orig_function
fn = m.create(defaults)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 1715, in create
input_storage=input_storage_lists, storage_map=storage_map)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line
699, in make_thunk
storage_map=storage_map)[:3]
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/vm.py",
line 1084, in make_all
impl=impl))
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/op.py",
line 955, in make_thunk
no_recycling)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/op.py",
line 858, in make_c_thunk
output_storage=node_output_storage)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1217, in make_thunk
keep_lock=keep_lock)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1157, in __compile__
keep_lock=keep_lock)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1620, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py",
line 1174, in module_from_key
module = lnk.compile_cmodule(location)
File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py",
line 1523, in compile_cmodule
preargs=preargs)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py",
line 2368, in compile_str
return dlimport(lib_filename)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py",
line 302, in dlimport
rval = __import__(module_name, {}, {}, [module_name])
ImportError: ('The following error happened while compiling the node',
GpuDnnConv{algo='small', inplace=True, num_groups=1}(GpuContiguous.0,
GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0,
GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), dilation=(1, 1),
conv_mode='conv', precision='float32', num_groups=1}.0, Constant{1.0},
Constant{0.0}), '\n',
undefined symbol: cudnnGetConvolutionGroupCount',
"[GpuDnnConv{algo='small', inplace=True,
num_groups=1}(<GpuArrayType<None>(float32, (False, True, False, False))>,
<GpuArrayType<None>(float32, 4D)>, <GpuArrayType<None>(float32, 4D)>,
<CDataType{cudnnConvolutionDescriptor_t}>, Constant{1.0}, Constant{0.0})]")
Regards.
El miércoles, 13 de diciembre de 2017, 13:50:44 (UTC+1), Beatriz G.
Post by Beatriz G.
Hi everyone.
I used to work with Theano and it works perfectly, but after installing
tensorflow with conda, and some dependencies to work with it, my Theano has
stopped to work.
Using gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled, cuDNN not
available)
... loading data
... building the model
... training
File "LENET.py", line 394, in <module>
evaluate_lenet5()
File "LENET.py", line 301, in evaluate_lenet5
cost_ij = train_model(minibatch_index)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line
314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File
"/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py",
line 859, in __call__
outputs = self.fn()
RuntimeError: GpuCorrMM encountered a CUBLAS error: the library was not
initialized
This could be a known bug in CUDA, please see the GpuCorrMM()
documentation.
Apply node that caused the error: GpuCorrMM_gradWeights{valid, (1,
1)}(GpuContiguous.0, GpuContiguous.0)
Toposort index: 28
Inputs types: [CudaNdarrayType(float32, (True, False, False, False)),
CudaNdarrayType(float32, 4D)]
Inputs shapes: [(1, 500, 28, 28), (1, 20, 5, 5)]
Inputs strides: [(0, 784, 28, 1), (0, 25, 5, 1)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[GpuDimShuffle{1,0,2,3}(GpuCorrMM_gradWeights{valid,
(1, 1)}.0)]]
HINT: Re-running with most Theano optimization disabled could give you a
back-trace of when this node was created. This can be done with by setting
the Theano flag 'optimizer=fast_compile'. If that does not work, Theano
optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and
storage map footprint of this apply node.
I have tried to install cuda and cudnn, but it does not work (apart from
the toolkit that i had already installed)
[global]
device = gpu
floatX = float32
[blas]
ldflags = -lopenblas
[nvcc]
flags=-D_FORCE_INLINES
I would appreciate any advice or help.
Regards.
Beatriz.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...