[theano-users] Compilation issue in Theano 1.0.1

Discussion:

i***@gmail.com

2018-05-03 13:12:29 UTC

Hi,

I have a windows machine and the content of the .theanorc file is:

[global]
floatX = float32
device = cpu

[nvcc]
fastmath = True
flags = -LC:\Temp\Anaconda3\libs
compiler_bindir = "C:\Temp\Anaconda3\Library\mingw-w64\bin"

[blas]
ldflags = -LC:\Temp\Anaconda3\Library\bin -lmkl_rt

[cuda]
root = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0

[dnn]
include_path = C:\Program Files\NVIDIA GPU Computing
Toolkit\CUDA\v8.0\include
library_path = C:\Program Files\NVIDIA GPU Computing
Toolkit\CUDA\v8.0\lib\x64
enabled = True

[lib]
cudnn = 1.0

I used Theano 0.9.0 and was able to set up the configuration option
"base_compilerdir" inside a script as follows:

import itertools

import numpy as np

from sklearn import metrics, preprocessing

import matplotlib.pyplot as plt

import time

import os

os.environ["THEANO_FLAGS"]= "base_compiledir=\Model_1"

from functools import partial

from hyperopt import hp, Trials, fmin, tpe, STATUS_OK

from neupy import algorithms, layers, init#, environment, plots

from neupy.exceptions import StopTraining

import random

import pandas as pd

#import shutil

import theano

print(theano.config.base_compiledir)

plt.style.use('ggplot')

theano.config.floatX= 'float32'

Theano was able to create a folder named "Model_1" in the path were the
script was located. Then, Theano was able to compile and run the script as
expected.

Now, I replaced the old Theano by its newest version 1.0.1. When I run the
script, I got this message at the end:

Exception: ('The following error happened while compiling the node',
InplaceDimShuffle{x,0}(layer:tanh-1/bias), '\n', 'Compilation failed
(return status=1):
C:/Temp/Anaconda3/Library/mingw-w64/bin/../lib/gcc/x86_64-w64-mingw32/5.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot open output file
C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1\\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64\\tmphss2exfn\\m885ff006a95d626dac547a7bdfdb471bbf058622ece2b4435e42316c4012ea56.pyd:
No such file or directory\r. collect2.exe: error: ld returned 1 exit
status\r. ', '[InplaceDimShuffle{x,0}(layer:tanh-1/bias)]')

I noticed the following:

a) the folder "Model_1" got created

b) the sub-folder
"compiler_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64"
got created. Inside this sub-folder, I only have another folder named
"lazylinker.ext" and a file name "__init__"

Then, I comment out these lines from the header in the script:

import os

os.environ["THEANO_FLAGS"]= "base_compiledir=\Model_1"

and run the script. Theano uses the default path:
C:\Users\IMarroquin\AppData\Local\Theano. Additionally, Theano creates a
whole series of "tmp...." sub-folders right under the folder
"compiler_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64"

What other configuration variable should I have in my Python script? Any
other suggestions?

Many thanks,

Ivan

--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pascal Lamblin

2018-05-03 15:39:15 UTC

Permalink

If you add "nocleanup=True" to the config options (for instance in the
.theanorc), can you see more things in Model_1\compiledir...?

I see a backslash before "Model_1" in the config option, is that
intended, or an artifact of the email? Is it necessary?
What happens if you put the whole path instead?

I see you mention once
compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64,
and then
compiler_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64

with "compiler" instead of "compiledir", is that just a copy/paste error?

Post by i***@gmail.com
Hi,
[global]
floatX = float32
device = cpu
[nvcc]
fastmath = True
flags = -LC:\Temp\Anaconda3\libs
compiler_bindir = "C:\Temp\Anaconda3\Library\mingw-w64\bin"
[blas]
ldflags = -LC:\Temp\Anaconda3\Library\bin -lmkl_rt
[cuda]
root = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
[dnn]
include_path = C:\Program Files\NVIDIA GPU Computing
Toolkit\CUDA\v8.0\include
library_path = C:\Program Files\NVIDIA GPU Computing
Toolkit\CUDA\v8.0\lib\x64
enabled = True
[lib]
cudnn = 1.0
I used Theano 0.9.0 and was able to set up the configuration option
import itertools
import numpy as np
from sklearn import metrics, preprocessing
import matplotlib.pyplot as plt
import time
import os
os.environ["THEANO_FLAGS"]= "base_compiledir=\Model_1"
from functools import partial
from hyperopt import hp, Trials, fmin, tpe, STATUS_OK
from neupy import algorithms, layers, init#, environment, plots
from neupy.exceptions import StopTraining
import random
import pandas as pd
#import shutil
import theano
print(theano.config.base_compiledir)
plt.style.use('ggplot')
theano.config.floatX= 'float32'
Theano was able to create a folder named "Model_1" in the path were the
script was located. Then, Theano was able to compile and run the script
as expected.
Now, I replaced the old Theano by its newest version 1.0.1. When I run
Exception: ('The following error happened while compiling the node',
InplaceDimShuffle{x,0}(layer:tanh-1/bias), '\n', 'Compilation failed
cannot open output file
No such file or directory\r. collect2.exe: error: ld returned 1 exit
status\r. ', '[InplaceDimShuffle{x,0}(layer:tanh-1/bias)]')
a) the folder "Model_1" got created
b) the sub-folder
"compiler_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64"
got created. Inside this sub-folder, I only have another folder named
"lazylinker.ext" and a file name "__init__"
import os
os.environ["THEANO_FLAGS"]= "base_compiledir=\Model_1"
C:\Users\IMarroquin\AppData\Local\Theano. Additionally, Theano creates a
whole series of "tmp...." sub-folders right under the folder
"compiler_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64"
What other configuration variable should I have in my Python script? Any
other suggestions?
Many thanks,
Ivan
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
Pascal Lamblin
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

i***@gmail.com

2018-05-03 18:00:28 UTC

Permalink

Thanks Pascal for the prompt reply.

To answer your questions:

A) Added the option nocleanup=True to the .theanorc file

B) The backslash before "Model 1". In my previous attempts in Theano 0.9.0,
this was the way that I found to make it work.

Following your suggestion, I edited the header of the script to
this: os.environ["THEANO_FLAGS"]= "base_compiledir=Model_1"

and ran the script. This time, I noticed an extra sub-folder under
"Model_1\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64".
The sub-folder is named "tmpgqmnjspd", which only contains a mod.c file

The compilation ended with this error:

Exception: ('The following error happened while compiling the node',
InplaceDimShuffle{x,0}(layer:tanh-1/bias), '\n', 'Compilation failed
(return status=1): C:/Temp/Anaconda3/Library/mingw-w64/bin/../
lib/gcc/x86_64-w64-mingw32/5.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot open output file
C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1\
\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64\\tmpgqmnjspd\\m885ff006a95d626dac547a7bdfdb471bbf058622ece2b4435e42316c4012ea56.pyd:
No such file or directory\r. collect2.exe: error: ld returned 1 exit
status\r. ', '[InplaceDimShuffle{x,0}(layer:tanh-1/bias)]')

It seems that the source of the problem is a missing file with extension
*.pyd

C) "compiler" instead of "compiledir". Sorry for the confusion, I made a
mistake during the copy/paste

Many thanks,
Ivan

Pascal Lamblin

2018-05-03 20:01:24 UTC

Permalink

On 2018-05-03 02:00 PM, ***@gmail.com wrote:> A) Added the
option nocleanup=True to the .theanorc file

Thanks. This makes sure the "mod.c" file you mention later is not deleted.

Post by i***@gmail.com
B) The backslash before "Model 1". In my previous attempts in Theano
0.9.0, this was the way that I found to make it work.

We have added quotation marks and such in order to better handle paths
on Windows since 0.9, so I'm not surprised.

Post by i***@gmail.com
Following your suggestion, I edited the header of the script to
this: os.environ["THEANO_FLAGS"]= "base_compiledir=Model_1"
and ran the script. This time, I noticed an extra sub-folder under
"Model_1\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64".
The sub-folder is named "tmpgqmnjspd", which only contains a mod.c file

That makes sense.

Post by i***@gmail.com
Exception: ('The following error happened while compiling the node',
InplaceDimShuffle{x,0}(layer:tanh-1/bias), '\n', 'Compilation failed
(return status=1): C:/Temp/Anaconda3/Library/mingw-w64/bin/../
cannot open output file
C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1\
No such file or directory\r. collect2.exe: error: ld returned 1 exit
status\r. ', '[InplaceDimShuffle{x,0}(layer:tanh-1/bias)]')
It seems that the source of the problem is a missing file with extension
*.pyd

From my understanding, the process is not able to open the *output*
file (the .pyd) in order to *write* to it, and so the compilation cannot
proceed.

I find that surprising, since it was able to create the mod.c file.
Maybe it's a quota problem, or not enough free disk space?

Can you specify a full path, for instance:

os.environ["THEANO_FLAGS"] =
"base_compiledir=C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1"

Another thing to try would be to put a breakpoint or try to print the
compilation command line that is executed, and try to execute it
directly from a shell (and the same working directory), rather than
inside Theano.

Post by i***@gmail.com
C) "compiler" instead of "compiledir". Sorry for the confusion, I made a
mistake during the copy/paste
Many thanks,
Ivan
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

i***@gmail.com

2018-05-03 20:50:21 UTC

Permalink

Thanks for the suggestion.

I checked the disk space and have ~700Gbytes. Then, I tested with
"base_compiledir" using the entire path.

This time, a sub-folder with this name was created:
UsersIMarroquinDocumentsMy_Python_ScriptsMLPSEG_contestIndependent_ScriptsModel_1

Because the name is way too long, windows complained about the filename and
the compilation failed again.

I am running the command: python <script.py> from a conda window. I
attached the compilation error by Theano and the problem occurs when it
tries to open the *.pyd file.

Also, I tried with this command os.environ["THEANO_FLAGS"]=
'base_compiledir=Model_1' and unfortunately, I ended with the same problem
(see attached file).

It is interesting that it used to work on Theano 0.9.0

I am hoping that you have other suggestions.

Thanks for all,
Ivan

Post by i***@gmail.com
Added the
option nocleanup=True to the .theanorc file
Thanks. This makes sure the "mod.c" file you mention later is not deleted.

Post by i***@gmail.com
B) The backslash before "Model 1". In my previous attempts in Theano
0.9.0, this was the way that I found to make it work.

We have added quotation marks and such in order to better handle paths
on Windows since 0.9, so I'm not surprised.

"Model_1\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64".

Post by i***@gmail.com
The sub-folder is named "tmpgqmnjspd", which only contains a mod.c file

That makes sense.

C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1\

Post by i***@gmail.com
No such file or directory\r. collect2.exe: error: ld returned 1 exit
status\r. ', '[InplaceDimShuffle{x,0}(layer:tanh-1/bias)]')
It seems that the source of the problem is a missing file with extension
*.pyd

From my understanding, the process is not able to open the *output*
file (the .pyd) in order to *write* to it, and so the compilation cannot
proceed.
I find that surprising, since it was able to create the mod.c file.
Maybe it's a quota problem, or not enough free disk space?
os.environ["THEANO_FLAGS"] =
"base_compiledir=C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1"
Another thing to try would be to put a breakpoint or try to print the
compilation command line that is executed, and try to execute it
directly from a shell (and the same working directory), rather than
inside Theano.

--
Pascal Lamblin

Pascal Lamblin

2018-05-03 22:32:13 UTC

Permalink

Post by i***@gmail.com
Thanks for the suggestion.
I checked the disk space and have ~700Gbytes. Then, I tested with
"base_compiledir" using the entire path.
UsersIMarroquinDocumentsMy_Python_ScriptsMLPSEG_contestIndependent_ScriptsModel_1

backslashes may have not been escaped enough, maybe with
os.environ["THEANO_FLAGS"] = r"basecompiledir=C:\\Users\\ ..."

Post by i***@gmail.com
I am running the command: python <script.py> from a conda window. I
attached the compilation error by Theano and the problem occurs when it
tries to open the *.pyd file.
"C:\Temp\Anaconda3\Library\mingw-w64\bin\g++.exe" -shared -g -O3

-fno-math-errno -Wno-unused-label -Wno-unused-variable
-Wno-write-strings -march=haswell -mmmx -mno-3dnow -msse -msse2 -msse3
-mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -mno-sha -mpclmul
-mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm
-mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c
-mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt
-mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1
-mno-clflushopt -mno-xsavec -mno-xsaves -mno-avx512dq -mno-avx512bw
-mno-avx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-pcommit
-mno-mwaitx --param l1-cache-size=32 --param l1-cache-line-size=64
--param l2-cache-size=25600 -mtune=haswell
-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -DMS_WIN64
-I"C:\Temp\Anaconda3\lib\site-packages\numpy\core\include"
-I"C:\Temp\Anaconda3\include"
-I"C:\Temp\Anaconda3\lib\site-packages\theano\gof\c_code"
-L"C:\Temp\Anaconda3\libs" -L"C:\Temp\Anaconda3" -o
"C:\Users\IMarroquin\Documents\My_Python_Scripts\MLP\SEG_contest\Independent_Scripts\Model_1\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64\tmp1vufnfg1\m885ff006a95d626dac547a7bdfdb471bbf058622ece2b4435e42316c4012ea56.pyd"
"C:\Users\IMarroquin\Documents\My_Python_Scripts\MLP\SEG_contest\Independent_Scripts\Model_1\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64\tmp1vufnfg1\mod.cpp"
-lpython36

The name of the "tmp..." directory changed, but that is expected (it is
randomly generated each time).
The file name for the .pyd is the same as before, since it only depends
on the file's content.

Can you try running that command line from the conda window?

Post by i***@gmail.com
B) The backslash before "Model 1". In my previous attempts in Theano
0.9.0, this was the way that I found to make it work.

We have added quotation marks and such in order to better handle paths
on Windows since 0.9, so I'm not surprised.

"Model_1\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_63_Stepping_2_GenuineIntel-3.6.1-64".

Post by i***@gmail.com
The sub-folder is named "tmpgqmnjspd", which only contains a

mod.c file
That makes sense.

C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1\

extension

Post by i***@gmail.com
*.pyd

From my understanding, the process is not able to open the *output*
file (the .pyd) in order to *write* to it, and so the compilation cannot
proceed.
I find that surprising, since it was able to create the mod.c file.
Maybe it's a quota problem, or not enough free disk space?
os.environ["THEANO_FLAGS"] =
"base_compiledir=C:\\Users\\IMarroquin\\Documents\\My_Python_Scripts\\MLP\\SEG_contest\\Independent_Scripts\\Model_1"
Another thing to try would be to put a breakpoint or try to print the
compilation command line that is executed, and try to execute it
directly from a shell (and the same working directory), rather than
inside Theano.

Post by i***@gmail.com
C) "compiler" instead of "compiledir". Sorry for the confusion, I

made a

Post by i***@gmail.com
mistake during the copy/paste
Many thanks,
Ivan
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it,

send

Post by i***@gmail.com
For more options, visit https://groups.google.com/d/optout

<https://groups.google.com/d/optout>.
--
Pascal Lamblin
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

Pascal Lamblin

2018-05-04 01:38:49 UTC

Permalink

Also, according to this [1], it may be your antivirus preventing the
creation of the file.

[1]
https://stackoverflow.com/questions/7655471/ld-exe-cannot-open-output-file-permission-denied