[theano-users] Question on Theano's L1 penalty

[t***@rams.colostate.edu]

Hello Theano Group - My name is Tomojit Ghosh and I'm doing my PhD in
Machine Learning here in Colorado State University,USA. I'm looking to
use Theano's Symbolic Differentiation.

I've a question on calculating the partial derivatives of a l1-pernalty
function. The absolute value function is not differentiable at the point
0. The common approach is to take the sign of the individual elements to
compute the partial derivative, in other words: partial of w_i=-1 if w_i<0,
partial of w_i=+1 if w_i>0, partial of w_i=0 if w_i=0. Although I'm quite
aware of the fact that when w_i=0, it's not differentiable.

I'm not sure how symbolic differentiotion is computing the gradient of w_i
when the value is zero.

It'll be really helpful if you can shed some light onto this.Thank you very
much for your time.

Regards,
Tomojit