Learning Activation Functions to Improve Deep Neural Networks

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piece-wise linear activation function that is learned independently for each neuron using standard gradient descent. With this learned activation function we are able to get significant improvements over static rectified linear units, and performance that is competitive with the state of the art on benchmark learning tasks from computer vision and high-energy physics.