Discussion:
[theano-users] ValueError: dimension mismatch in x,y_idx arguments
ephi5757 via theano-users
2017-08-16 21:49:37 UTC
Permalink
I'm retraining my implementation of the neural network model AlexNet in
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note that
the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.


C:\SciSoft\Git\theano_alexnet>python train.py THEANO_FLAGS=mode=FAST_RUN,
floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training

______________________________________________________________________________.
Traceback (most recent call last):
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line 871,
in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line 859,
in __call__
outputs = self.fn()

ValueError: dimension mismatch in x,y_idx arguments
Apply node that caused the error:
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32,
vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
Outputs clients:
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Frédéric Bastien
2017-08-17 02:00:26 UTC
Permalink
I think the problem are the values in the index vector. Double check that.

Frédéric

On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users <
Post by ephi5757 via theano-users
I'm retraining my implementation of the neural network model AlexNet in
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note that
the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.
C:\SciSoft\Git\theano_alexnet>python train.py THEANO_FLAGS=mode=FAST_RUN,
floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training
______________________________________________________________________________.
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
859, in __call__
outputs = self.fn()
ValueError: dimension mismatch in x,y_idx arguments
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32,
vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ephi5757 via theano-users
2017-08-17 21:09:28 UTC
Permalink
Hi Frederic,
Good to hear from you. Thanks for your rapid response. I searched my
code for the index vector as you suggested and found it in
\theano\compile\function_module.py and in \theano\gof\link.py. I haven't
found anything out of line, however. Please advise.
Arnold
Post by Frédéric Bastien
I think the problem are the values in the index vector. Double check that.
Frédéric
On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users <
Post by ephi5757 via theano-users
I'm retraining my implementation of the neural network model AlexNet in
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note
that the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.
C:\SciSoft\Git\theano_alexnet>python train.py THEANO_FLAGS=mode=FAST_RUN,
floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training
______________________________________________________________________________.
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
859, in __call__
outputs = self.fn()
ValueError: dimension mismatch in x,y_idx arguments
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32,
vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ephi5757 via theano-users
2017-08-17 22:24:13 UTC
Permalink
I added a print statement before the line where the code crashes, i.e.,

t0_fn = time.time()
print(self.fn())
try:
outputs = self.fn()

and the error is the same, i.e., ValueError: dimension mismatch in x,y_idx
arguments.
Please recommend where I can find and print out the x, y_idx arguments to
see what's going on.
Thank you,
Arnold
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ephi5757 via theano-users
2017-08-17 22:26:09 UTC
Permalink
the extracted code above is in \theano\compile\function_module.py
Post by ephi5757 via theano-users
I added a print statement before the line where the code crashes, i.e.,
t0_fn = time.time()
print(self.fn())
outputs = self.fn()
and the error is the same, i.e., ValueError: dimension mismatch in
x,y_idx arguments.
Please recommend where I can find and print out the x, y_idx arguments to
see what's going on.
Thank you,
Arnold
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ephi5757 via theano-users
2017-08-18 15:02:11 UTC
Permalink
Here are the print results before the code crashes:
======================================.
[(3007, 3007)]
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
[array(5001.4716796875, dtype=float32)]
.
[(3007, 3007)]
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
[array(5058.68408203125, dtype=float32)]
.
===========================================
Post by ephi5757 via theano-users
the extracted code above is in \theano\compile\function_module.py
Post by ephi5757 via theano-users
I added a print statement before the line where the code crashes, i.e.,
t0_fn = time.time()
print(self.fn())
outputs = self.fn()
and the error is the same, i.e., ValueError: dimension mismatch in
x,y_idx arguments.
Please recommend where I can find and print out the x, y_idx arguments to
see what's going on.
Thank you,
Arnold
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ephi5757 via theano-users
2017-08-21 23:15:06 UTC
Permalink
Hi Frederic,
I am pre-processing the image data again to regenerate the training
and validation .hkl image files. I found in my code (alexnet\train.py) that
the program crashes before it completes the first iteration, i.e., as it
looks at the first of 5003 minibatches. In order to make room on my
external solid state hard drive, I deleted the training and validation file
folders named train_(or val_)hkl_b256_b_128, which I don't think are used
but take up 237GB of space... and kept the folders named train_(or
val_)hkl_b256_b_256. Perhaps in another day or two when the 1.2 M images
are reshaped into 5003 files each containing 256 images that are size (256
x 256)... then I can try to run the train.py again and see if the errors
correct themselves.
This may have been my mistake for wanting to save space for my neural
net model output (weights and biases).
Best,
Arnold
Post by Frédéric Bastien
I think the problem are the values in the index vector. Double check that.
Frédéric
On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users <
Post by ephi5757 via theano-users
I'm retraining my implementation of the neural network model AlexNet in
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note
that the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.
C:\SciSoft\Git\theano_alexnet>python train.py THEANO_FLAGS=mode=FAST_RUN,
floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training
______________________________________________________________________________.
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
859, in __call__
outputs = self.fn()
ValueError: dimension mismatch in x,y_idx arguments
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32,
vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Frédéric Bastien
2017-08-23 01:15:48 UTC
Permalink
To get better error message from Theano, disable the GPU and use this flag.
optimizer=fast_compile

In this way, Theano will probably give you a stack trace where you created
the computation that cause problem.

On lun. 21 août 2017 19:15 ephi5757 via theano-users <
Post by ephi5757 via theano-users
Hi Frederic,
I am pre-processing the image data again to regenerate the training
and validation .hkl image files. I found in my code (alexnet\train.py) that
the program crashes before it completes the first iteration, i.e., as it
looks at the first of 5003 minibatches. In order to make room on my
external solid state hard drive, I deleted the training and validation file
folders named train_(or val_)hkl_b256_b_128, which I don't think are used
but take up 237GB of space... and kept the folders named train_(or
val_)hkl_b256_b_256. Perhaps in another day or two when the 1.2 M images
are reshaped into 5003 files each containing 256 images that are size (256
x 256)... then I can try to run the train.py again and see if the errors
correct themselves.
This may have been my mistake for wanting to save space for my neural
net model output (weights and biases).
Best,
Arnold
Post by Frédéric Bastien
I think the problem are the values in the index vector. Double check that.
Frédéric
On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users <
I'm retraining my implementation of the neural network model AlexNet in
Post by Frédéric Bastien
Post by ephi5757 via theano-users
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note
that the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.
C:\SciSoft\Git\theano_alexnet>python train.py
THEANO_FLAGS=mode=FAST_RUN, floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training
______________________________________________________________________________.
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
859, in __call__
outputs = self.fn()
ValueError: dimension mismatch in x,y_idx arguments
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix),
CudaNdarrayType(float32, vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
Post by ephi5757 via theano-users
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
ephi5757 via theano-users
2017-08-23 17:08:42 UTC
Permalink
Problem solved. In one of my configuration files (spec_1gpu.yaml) I forgot
to reset my batch_size parameter back to 256. I had assigned batch_size = 1
when I was doing some analyses on single image files. Fortunately, several
debugging print statements before and after crash points identified this
mislabeled parameter. I am still grateful to the theano users group. It is
a generous resource for the community.
Best,
Arnold
Post by Frédéric Bastien
To get better error message from Theano, disable the GPU and use this
flag. optimizer=fast_compile
In this way, Theano will probably give you a stack trace where you created
the computation that cause problem.
On lun. 21 août 2017 19:15 ephi5757 via theano-users <
Post by ephi5757 via theano-users
Hi Frederic,
I am pre-processing the image data again to regenerate the training
and validation .hkl image files. I found in my code (alexnet\train.py) that
the program crashes before it completes the first iteration, i.e., as it
looks at the first of 5003 minibatches. In order to make room on my
external solid state hard drive, I deleted the training and validation file
folders named train_(or val_)hkl_b256_b_128, which I don't think are used
but take up 237GB of space... and kept the folders named train_(or
val_)hkl_b256_b_256. Perhaps in another day or two when the 1.2 M images
are reshaped into 5003 files each containing 256 images that are size (256
x 256)... then I can try to run the train.py again and see if the errors
correct themselves.
This may have been my mistake for wanting to save space for my
neural net model output (weights and biases).
Best,
Arnold
Post by Frédéric Bastien
I think the problem are the values in the index vector. Double check that.
Frédéric
On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users <
I'm retraining my implementation of the neural network model AlexNet in
Post by Frédéric Bastien
Post by ephi5757 via theano-users
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note
that the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.
C:\SciSoft\Git\theano_alexnet>python train.py
THEANO_FLAGS=mode=FAST_RUN, floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training
______________________________________________________________________________.
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
859, in __call__
outputs = self.fn()
ValueError: dimension mismatch in x,y_idx arguments
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix),
CudaNdarrayType(float32, vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
Post by ephi5757 via theano-users
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
j***@gmail.com
2017-08-23 03:58:36 UTC
Permalink
Post by ephi5757 via theano-users
I'm retraining my implementation of the neural network model AlexNet in
Theano and not long after it initializes the program crashes with the error
"ValueError: dimension mismatch in x,y_idx arguments." see traceback below.
Any comments or suggestions that you may offer would be helpful. Note that
the only discernible difference in this training in comparison to the
previous one is that I am using 5003 .hkl training image data files instead
of 5004. Nevertheless, I don't think this value needs to be fixed.
Looking forward to your reply.
Arnold
_______________________________________________________________.
C:\SciSoft\Git\theano_alexnet>python train.py THEANO_FLAGS=mode=FAST_RUN,
floatX=float32
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007)
... building the model
conv (cudnn) layer with shape_in: (3, 227, 227, 1)
conv (cudnn) layer with shape_in: (96, 27, 27, 1)
conv (cudnn) layer with shape_in: (256, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
conv (cudnn) layer with shape_in: (384, 13, 13, 1)
fc layer with num_in: 9216 num_out: 4096
dropout layer with P_drop: 0.5
fc layer with num_in: 4096 num_out: 4096
dropout layer with P_drop: 0.5
softmax layer with num_in: 4096 num_out: 1000
... training
______________________________________________________________________________.
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 266, in _bootstrap
self.run()
File
"C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py",
line 120, in run
self._target(*self._args, **self._kwargs)
File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net
recv_queue=load_recv_queue)
File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in
train_model_wrap
cost_ij = train_model()
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in
raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "c:\scisoft\git\theano\theano\compile\function_module.py", line
859, in __call__
outputs = self.fn()
ValueError: dimension mismatch in x,y_idx arguments
GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0,
<CudaNdarrayType(float32, vector)>, GpuFromHost.0)
Toposort index: 298
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32,
vector), CudaNdarrayType(float32, vector)]
Inputs shapes: [(256, 1000), (1000,), (1,)]
Inputs strides: [(1000, 1), (1,), (0,)]
Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])]
[[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)],
[GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0,
GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []]
.
_____________________________________________________________________.
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...