[theano-users] Need help with running simultaneously three neural network scripts

[i***@gmail.com]

Hi all,

I have a windows machine with a kepler card. On this machine, I installed
Conda 4.3.30 and theano 0.9.0 (along with its required packages).

I used neupy package as a front end to theano for implementing the neural
network to train.

I would like to run simultaneously three scripts - each of them trains the
same neural network architecture on three different data sets. My first
approach was to open three conda terminals to run the scripts.

So, I let the first script to work for a short moment and then, I ran the
second script. While I am waiting to run the third script, I noticed that
the second script stopped and there was the following message on the conda
terminal used to run the first script:

WARNING (theano.gof.compilelock): Overriding existing lock by dead process
'13168' (I am process '11140')

The first script just continued working.

any suggestion on how to solve this? or, on how to run several scripts
simultaneously?

many thanks,

Ivan

[i***@gmail.com]

Hi all,

After reviewing the Theano configuration guide, I think that I found a way
to run three simultaneously jobs. For that, I opted to define a specific
"base compiler directory" for each job with the hope this will avoid the
lock failure reported earlier.

Below you have a copy of the header of one the scripts:

import itertools
import numpy as np
from sklearn import metrics, preprocessing
import matplotlib.pyplot as plt
import time
import os

os.environ["THEANO_FLAGS"]= "base_compiledir=\Model_1"

from functools import partial
from hyperopt import hp, Trials, fmin, tpe, STATUS_OK
from neupy import algorithms, layers, init, environment # Note that neupy
is a front end for Theano
from neupy.exceptions import StopTraining
import theano
import random
import pandas as pd

By changing the definition of "base_compiledir", Theano created a
subdirectory named Model_1 (see zip file attached) in the same folder where
the script file exists. I believed this is the desired result.

From the Conda console window, there is a long output (see pdf file
attached) pointing to errors during the compilation. It seems that both
errors point to mod.cpp and lazylinker_ext.pyd files don't exist. I checked
the content of the folder created by Theano (see attached file). I found
the mod.cpp file but the lazylinker_ext.pyd is not present.

Thanks for your help,

Ivan

[i***@gmail.com]

In case this may help. If I run one script with the default definition of
"base compile dir", the script works just fine.

In addtion to Theano 0.9.0, I also have these packages:

pygpu 0.6.9 py36_0

mingw 4.7 1

libgpuarray 0.6.9 vc14_0 [vc14]

m2w64-binutils 2.25.1 5
m2w64-bzip2 1.0.6 6
m2w64-crt-git 5.0.0.4636.2595836 2
m2w64-gcc 5.3.0 6
m2w64-gcc-ada 5.3.0 6
m2w64-gcc-fortran 5.3.0 6
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gcc-objc 5.3.0 6
m2w64-gmp 6.1.0 2
m2w64-headers-git 5.0.0.4636.c0ad18a 2
m2w64-isl 0.16.1 2
m2w64-libiconv 1.14 6
m2w64-libmangle-git 5.0.0.4509.2e5a9a2 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
m2w64-make 4.1.2351.a80a8b8 2
m2w64-mpc 1.0.3 3
m2w64-mpfr 3.1.4 4
m2w64-pkg-config 0.29.1 2
m2w64-toolchain 5.3.0 7
m2w64-tools-git 5.0.0.4592.90b8472 2
m2w64-windows-default-manifest 6.4 3
m2w64-winpthreads-git 5.0.0.4634.697f757 2

[Pascal Lamblin]

Hi,

This is just an informative warning, because several scripts are trying
to compile and cache binaries at the same time.
The second time the script is run, the cache should be full and this
should not happen any more.
In any case, there should be no consequence on the computation or its speed.

If you specify a different cache for each process, you will get rid of
the messages, but each script will compile its own versions of the
binaries, so work will be duplicated and it is not clear that the
computation will actually start sooner.

Post by i***@gmail.com
Hi all,
I have a windows machine with a kepler card. On this machine, I
installed Conda 4.3.30 and theano 0.9.0 (along with its required packages).
I used neupy package as a front end to theano for implementing the
neural network to train.
I would like to run simultaneously three scripts - each of them trains
the same neural network architecture on three different data sets. My
first approach was to open three conda terminals to run the scripts.
So, I let the first script to work for a short moment and then, I ran
the second script. While I am waiting to run the third script, I noticed
that the second script stopped and there was the following message on
 WARNING (theano.gof.compilelock): Overriding existing lock by dead
process '13168' (I am process '11140')
The first script just continued working.
any suggestion on how to solve this? or, on how to run several scripts
simultaneously?
many thanks,
Ivan
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

[i***@gmail.com]

Hi Pascal,

Thanks for your comments. Could you show me how specify a different cache
for each process? I would like to learn more on how to "manipulate"
Theano's configuration.

On the other hand. I found the solution to the compilation problem as well
to be able to run simultaneously three scripts.

The original path were the scripts are located
was: C:\Users\IMarroquin\Documents\My Python
Scripts\MLP\Well_8_to_five_wells\Independent_Scripts.

Because of the two blank spaces in "My Python Scripts", the cmodule.py was
not able to correctly parse the entire path. Once, I replaced "My Python
Scripts" by "My_Python_Scripts" the compilation error went away and I was
able to run three jobs.

Although it is not a bug, it is still a problem. It is very likely to have
folders in windows with blank spaces in them. Perhaps, Theano developers
will come with a solution to handle directory trees in windows.

I found this related post: https://github.com/Theano/Theano/issues/5348


Thanks to all,

Post by Pascal Lamblin
Hi,
This is just an informative warning, because several scripts are trying
to compile and cache binaries at the same time.
The second time the script is run, the cache should be full and this
should not happen any more.
In any case, there should be no consequence on the computation or its speed.
If you specify a different cache for each process, you will get rid of
the messages, but each script will compile its own versions of the
binaries, so work will be duplicated and it is not clear that the
computation will actually start sooner.

Post by i***@gmail.com
Hi all,
I have a windows machine with a kepler card. On this machine, I
installed Conda 4.3.30 and theano 0.9.0 (along with its required

packages).

Post by i***@gmail.com
I used neupy package as a front end to theano for implementing the
neural network to train.
I would like to run simultaneously three scripts - each of them trains
the same neural network architecture on three different data sets. My
first approach was to open three conda terminals to run the scripts.
So, I let the first script to work for a short moment and then, I ran
the second script. While I am waiting to run the third script, I noticed
that the second script stopped and there was the following message on
WARNING (theano.gof.compilelock): Overriding existing lock by dead
process '13168' (I am process '11140')
The first script just continued working.
any suggestion on how to solve this? or, on how to run several scripts
simultaneously?
many thanks,
Ivan
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
Pascal Lamblin