How to change the learning rate dynamically

[s***@public.gmane.org]

Hi,

I would to change the learning rate in learning procedure. A large learning
rate is used in the initial stage and small is used in later. How can I do.

Thanks
Jiancheng

[Frédéric Bastien]

Make a theano variable that is the learning rate and pass it as an input to
your theano function.

You could also use a shared variable is you don't want to pass it
explicitly each time, but only change it from time to time:

http://deeplearning.net/software/theano/tutorial/examples.html#using-shared-variables

Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in later. How
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Ofir Levy]

for the CNN example we currently have the following code:

learning_rate = 0.1

updates = []for param_i, grad_i in zip(params, grads):
updates.append((param_i, param_i - learning_rate * grad_i))train_model = theano.function([index], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})

and in the training loop:

cost_ij = train_model(minibatch_index)


can you kindly tell me how to change it to have a adaptive learning rate?

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an input
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in later. How
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Ofir Levy]

ok I think I got it

learning_rate = 0.1

l_r = T.scalar('l_r', dtype=theano.config.floatX)

updates = []for param_i, grad_i in zip(params, grads):
updates.append((param_i, param_i - l_r * grad_i))

train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})

and in the training loop:

cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate * grad_i))train_model = theano.function([index], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning rate?

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an input
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in later. How
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

[Frédéric Bastien]

Exacte.

Fred

Post by Ofir Levy
ok I think I got it
learning_rate = 0.1
l_r = T.scalar('l_r', dtype=theano.config.floatX)
updates.append((param_i, param_i - l_r * grad_i))
train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate * grad_i))train_model = theano.function([index], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning rate?

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an input
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/
examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in later. How
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Pascal Lamblin]

X-Received: by 10.152.5.4 with SMTP id o4mr2625lao.40.1412613619750;
Mon, 06 Oct 2014 09:40:19 -0700 (PDT)
X-BeenThere: theano-users-/***@public.gmane.org
Received: by 10.152.21.193 with SMTP id x1ls626838lae.53.gmail; Mon, 06 Oct
2014 09:40:16 -0700 (PDT)
X-Received: by 10.112.220.8 with SMTP id ps8mr930623lbc.5.1412613616749;
Mon, 06 Oct 2014 09:40:16 -0700 (PDT)
Received: from ks356264.kimsufi.com (ks356264.kimsufi.com. [91.121.141.22])
by gmr-mx.google.com with ESMTP id hv5si563555wib.1.2014.10.06.09.40.16
for <theano-users-/***@public.gmane.org>;
Mon, 06 Oct 2014 09:40:16 -0700 (PDT)
Received-SPF: none (google.com: blip-***@public.gmane.org does not designate permitted sender hosts) client-ip‘.121.141.22;
Received: by ks356264.kimsufi.com (Postfix, from userid 1000)
id 66708573; Mon, 6 Oct 2014 18:40:16 +0200 (CEST)
In-Reply-To: <CADKKbti71TSoPJn=kpz15i7b81sngC5EuskiyQMU-TEH+zrEMg-JsoAwUIsXosN+***@public.gmane.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Original-Sender: lamblinp-***@public.gmane.org
X-Original-Authentication-Results: gmr-mx.google.com; spf=neutral
(google.com: blip-***@public.gmane.org does not designate permitted sender hosts) smtp.mail=blip-***@public.gmane.org
Precedence: list
Mailing-list: list theano-users-/***@public.gmane.org; contact theano-users+owners-/***@public.gmane.org
List-ID: <theano-users.googlegroups.com>
X-Google-Group-Id: 255958038354
List-Post: <http://groups.google.com/group/theano-users/post>, <mailto:theano-users-/***@public.gmane.org>
List-Help: <http://groups.google.com/support/>, <mailto:theano-users+help-/***@public.gmane.org>
List-Archive: <http://groups.google.com/group/theano-users
Sender: theano-users-/***@public.gmane.org
List-Subscribe: <http://groups.google.com/group/theano-users/subscribe>, <mailto:theano-users+subscribe-/***@public.gmane.org>
List-Unsubscribe: <mailto:googlegroups-manage+255958038354+unsubscribe-/***@public.gmane.org>,
<http://groups.google.com/group/theano-users/subscribe>
Content-Disposition: inline
Archived-At: <http://permalink.gmane.org/gmane.comp.mathematics.theano.user/3067>

Post by Frédéric Bastien
Exacte.

Also, you can make l_r a shared variable, and update it via a Theano
expression, if it is convenient to do so. For instance:

l_r = theano.shared(np.array(0.1, dtype=theano.config.floatX))
...
updates.append((l_r, 0.9 * l_r)) # Or however you want to change your learning rate

train_model = theano.function([index], cost, updates=updates, givens=...)

Post by Frédéric Bastien
Fred

Post by Ofir Levy
ok I think I got it
learning_rate = 0.1
l_r = T.scalar('l_r', dtype=theano.config.floatX)
updates.append((param_i, param_i - l_r * grad_i))
train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate * grad_i))train_model = theano.function([index], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning rate?

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an input
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/
examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in later. How
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
For more options, visit https://groups.google.com/d/optout.

[Beatriz G.]

Hi.

I have tried to apply your code, but it does not work for me, I got "nan"
at training cost.

l_r = theano.shared(np.array(learning_rate, dtype=theano.config.floatX))
....
cost = layer8.negative_log_likelihood(y)
....

validate_model = theano.function(
[index],
layer8.errors(y),
givens={
x: valid_set_x[index * batch_size: (index + 1) * batch_size],
y: valid_set_y[index * batch_size: (index + 1) * batch_size],
is_train: numpy.cast['int32'](0)

}
)

# create a list of all model parameters to be fit by gradient descent
params = layer0.params + layer1.params + layer2.params + layer3.params
+ layer4.params + layer5.params + layer6.params + layer7.params +
layer8.params

# create a list of gradients for all model parameters
grads = T.grad(cost, params)

## Learning rate update
updates = [
(param_i, param_i - l_r * grad_i)
for param_i, grad_i in zip(params, grads)
]
updates.append((l_r,0.95*l_r))

train_model = theano.function([index], cost, updates=updates,
givens={
x: train_set_x[index * batch_size:
(index + 1) * batch_size],
y: train_set_y[index * batch_size:
(index + 1) * batch_size],
is_train: np.cast['int32'](1)})
in the while loop:
cost_ij = train_model(minibatch_index)
when it is time to validate:
validation_losses = [validate_model(i) for i in
range(n_valid_batches)]


The learning rate updates its value, the validation error is 90%
continuously and the training cost is "nan"

Thank you in advance.

Regards.

Post by Pascal Lamblin

Post by Frédéric Bastien
Exacte.

Also, you can make l_r a shared variable, and update it via a Theano
l_r = theano.shared(np.array(0.1, dtype=theano.config.floatX))
...
updates.append((l_r, 0.9 * l_r)) # Or however you want to change your learning rate
train_model = theano.function([index], cost, updates=updates, givens=...)

Post by Frédéric Bastien
Fred

Post by Ofir Levy
ok I think I got it
learning_rate = 0.1
l_r = T.scalar('l_r', dtype=theano.config.floatX)
updates.append((param_i, param_i - l_r * grad_i))
train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) *

batch_size],

Post by Frédéric Bastien

Post by Ofir Levy
y: train_set_y[index * batch_size: (index + 1) *

batch_size]})

Post by Frédéric Bastien

Post by Ofir Levy
cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate *

grad_i))train_model = theano.function([index], cost, updates = updates,

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
givens={
x: train_set_x[index * batch_size: (index + 1) *

batch_size],

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
y: train_set_y[index * batch_size: (index + 1) *

batch_size]})

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning

rate?

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an

input

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/
examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in

later. How

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it,

send

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google

Groups

Post by Frédéric Bastien

Post by Ofir Levy
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send

an

Post by Frédéric Bastien

Post by Ofir Levy
For more options, visit https://groups.google.com/d/optout.

--
---
You received this message because you are subscribed to the Google

Groups "theano-users" group.

Post by Frédéric Bastien
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
Pascal

[Michael Klachko]

The learning rate schedule is defined in this line:
updates.append((l_r,0.95*l_r)), and is used in this line: train_model =
theano.function([index], cost, updates=updates ...
If you don't understand what's going on, read about theano functions.

Post by Beatriz G.
Hi.
I have tried to apply your code, but it does not work for me, I got "nan"
at training cost.
l_r = theano.shared(np.array(learning_rate,
dtype=theano.config.floatX))
....
cost = layer8.negative_log_likelihood(y)
....
validate_model = theano.function(
[index],
layer8.errors(y),
givens={
x: valid_set_x[index * batch_size: (index + 1) * batch_size],
y: valid_set_y[index * batch_size: (index + 1) * batch_size],
is_train: numpy.cast['int32'](0)
}
)
# create a list of all model parameters to be fit by gradient descent
params = layer0.params + layer1.params + layer2.params + layer3.params
+ layer4.params + layer5.params + layer6.params + layer7.params +
layer8.params
# create a list of gradients for all model parameters
grads = T.grad(cost, params)
## Learning rate update
updates = [
(param_i, param_i - l_r * grad_i)
for param_i, grad_i in zip(params, grads)
]
updates.append((l_r,0.95*l_r))
train_model = theano.function([index], cost, updates=updates,
givens={
(index + 1) * batch_size],
(index + 1) * batch_size],
is_train: np.cast['int32'](1)})
cost_ij = train_model(minibatch_index)
validation_losses = [validate_model(i) for i in
range(n_valid_batches)]
The learning rate updates its value, the validation error is 90%
continuously and the training cost is "nan"
Thank you in advance.
Regards.

Post by Pascal Lamblin

Post by Frédéric Bastien
Exacte.

Also, you can make l_r a shared variable, and update it via a Theano
l_r = theano.shared(np.array(0.1, dtype=theano.config.floatX))
...
updates.append((l_r, 0.9 * l_r)) # Or however you want to change your learning rate
train_model = theano.function([index], cost, updates=updates, givens=...)

Post by Frédéric Bastien
Fred

Post by Ofir Levy
ok I think I got it
learning_rate = 0.1
l_r = T.scalar('l_r', dtype=theano.config.floatX)
updates.append((param_i, param_i - l_r * grad_i))
train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) *

batch_size],

Post by Frédéric Bastien

Post by Ofir Levy
y: train_set_y[index * batch_size: (index + 1) *

batch_size]})

Post by Frédéric Bastien

Post by Ofir Levy
cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate *

grad_i))train_model = theano.function([index], cost, updates = updates,

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
givens={
x: train_set_x[index * batch_size: (index + 1) *

batch_size],

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
y: train_set_y[index * batch_size: (index + 1) *

batch_size]})

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning

rate?

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an

input

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/
examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in

later. How

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it,

send

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google

Groups

Post by Frédéric Bastien

Post by Ofir Levy
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it,

send an

Post by Frédéric Bastien

Post by Ofir Levy
For more options, visit https://groups.google.com/d/optout.

--
---
You received this message because you are subscribed to the Google

Groups "theano-users" group.

Post by Frédéric Bastien
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
Pascal

[Beatriz G.]

I have undestood it, and It works for me, I have modified the code so after
training each epocf and after validate, the learning rate is changed.

The cost took nan because the learning rate value was too high, I am not
pretty sure why.

Post by Michael Klachko
updates.append((l_r,0.95*l_r)), and is used in this line: train_model =
theano.function([index], cost, updates=updates ...
If you don't understand what's going on, read about theano functions.

Post by Beatriz G.
Hi.
I have tried to apply your code, but it does not work for me, I got "nan"
at training cost.
l_r = theano.shared(np.array(learning_rate,
dtype=theano.config.floatX))
....
cost = layer8.negative_log_likelihood(y)
....
validate_model = theano.function(
[index],
layer8.errors(y),
givens={
x: valid_set_x[index * batch_size: (index + 1) * batch_size],
y: valid_set_y[index * batch_size: (index + 1) * batch_size],
is_train: numpy.cast['int32'](0)
}
)
# create a list of all model parameters to be fit by gradient descent
params = layer0.params + layer1.params + layer2.params +
layer3.params + layer4.params + layer5.params + layer6.params +
layer7.params + layer8.params
# create a list of gradients for all model parameters
grads = T.grad(cost, params)
## Learning rate update
updates = [
(param_i, param_i - l_r * grad_i)
for param_i, grad_i in zip(params, grads)
]
updates.append((l_r,0.95*l_r))
train_model = theano.function([index], cost, updates=updates,
givens={
(index + 1) * batch_size],
(index + 1) * batch_size],
is_train: np.cast['int32'](1)})
cost_ij = train_model(minibatch_index)
validation_losses = [validate_model(i) for i in
range(n_valid_batches)]
The learning rate updates its value, the validation error is 90%
continuously and the training cost is "nan"
Thank you in advance.
Regards.

Post by Pascal Lamblin

Post by Frédéric Bastien
Exacte.

Also, you can make l_r a shared variable, and update it via a Theano
l_r = theano.shared(np.array(0.1, dtype=theano.config.floatX))
...
updates.append((l_r, 0.9 * l_r)) # Or however you want to change your learning rate
train_model = theano.function([index], cost, updates=updates, givens=...)

Post by Frédéric Bastien
Fred

Post by Ofir Levy
ok I think I got it
learning_rate = 0.1
l_r = T.scalar('l_r', dtype=theano.config.floatX)
updates.append((param_i, param_i - l_r * grad_i))
train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) *

batch_size],

Post by Frédéric Bastien

Post by Ofir Levy
y: train_set_y[index * batch_size: (index + 1) *

batch_size]})

Post by Frédéric Bastien

Post by Ofir Levy
cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate *

grad_i))train_model = theano.function([index], cost, updates = updates,

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
givens={
x: train_set_x[index * batch_size: (index + 1) *

batch_size],

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
y: train_set_y[index * batch_size: (index + 1) *

batch_size]})

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning

rate?

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy
On Thursday, July 17, 2014 9:48:24 PM UTC+3, Frédéric Bastien

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an

input

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/
examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A

large

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
learning rate is used in the initial stage and small is used in

later. How

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the

Google

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it,

send

Post by Frédéric Bastien

Post by Ofir Levy

Post by Ofir Levy

Post by Frédéric Bastien

Post by s***@public.gmane.org
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google

Groups

Post by Frédéric Bastien

Post by Ofir Levy
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it,

send an

Post by Frédéric Bastien

Post by Ofir Levy
For more options, visit https://groups.google.com/d/optout.

--
---
You received this message because you are subscribed to the Google

Groups "theano-users" group.

Post by Frédéric Bastien
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.

--
Pascal

[Beatriz G.]

Hi,

I am trying to implement a dynamic learning rate, but I do not understand
how the learning rate is updated in the code that you have shared. I can't
see where is updated the learning rate in the way: learning rate = learning
rate * 0.8

In addition, I have tried to implement the code but it does not works, but
I would like to understand the code in order to realize what is wrong in my
code.

Thank you in advance.

Beatriz.

Post by Ofir Levy
ok I think I got it
learning_rate = 0.1
l_r = T.scalar('l_r', dtype=theano.config.floatX)
updates.append((param_i, param_i - l_r * grad_i))
train_model = theano.function([index,l_r], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index, learning_rate)

Post by Ofir Levy
learning_rate = 0.1
updates.append((param_i, param_i - learning_rate * grad_i))train_model = theano.function([index], cost, updates = updates,
givens={
x: train_set_x[index * batch_size: (index + 1) * batch_size],
y: train_set_y[index * batch_size: (index + 1) * batch_size]})
cost_ij = train_model(minibatch_index)
can you kindly tell me how to change it to have a adaptive learning rate?

Post by Frédéric Bastien
Make a theano variable that is the learning rate and pass it as an input
to your theano function.
You could also use a shared variable is you don't want to pass it
http://deeplearning.net/software/theano/tutorial/examples.html#using-shared-variables
Fred

Post by s***@public.gmane.org
Hi,
I would to change the learning rate in learning procedure. A large
learning rate is used in the initial stage and small is used in later. How
can I do.
Thanks
Jiancheng
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
For more options, visit https://groups.google.com/d/optout.