This week the artificial intelligence academic world is in absolute shock, and the shockwave is reaching technical forums and sites like Hacker News. A couple of months ago, at the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) competition, a team from the University of Toronto consisting of Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton presented something that has destroyed the competition.

Until now, computer vision worked through "feature engineering". Mathematical experts designed algorithms (such as SIFT or HOG) to manually detect edges, textures, or shapes in an image, and then passed it to a classifier. The best systems in the world had an error rate of 26%.

This team's algorithm, informally dubbed "AlexNet", has dropped the error rate to 15.3%. It is a technical humiliation to the rest of the researchers. And they have done it by reviving a technology that many considered dead: neural networks.

The brute force of GPUs and convolution

The paper "ImageNet Classification with Deep Convolutional Neural Networks" is fascinating to read. For decades, deep neural networks were computational suicide. They took months to train and suffered from the "vanishing gradient" problem (neurons stopped learning as you added layers).

What has changed? Two key things on a technical level.

First, the hardware. Instead of using the CPU, researchers have ported massive matrix calculations to the graphics card (GPU). Specifically, they trained their giant 60-million parameter network on two 3GB NVIDIA GTX 580 cards for six days straight. GPUs, designed to calculate pixels in video games simultaneously, turn out to be the perfect mathematical engines for neural networks.

Second, the activation function. They discarded the classic Sigmoid or Hyperbolic Tangent functions and used something very simple called ReLU (Rectified Linear Unit). At the code level, the math of a neuron is now as dumb as this:

# The ReLU activation function that saved Deep Learning
def relu(x):
    # If the value is negative, returns 0. If positive, returns the value.
    return max(0.0, x)

# Applied to a list of values resulting from the weighted sum
valores_salida = [-1.5, 2.3, 0.0, -0.8, 5.1]
valores_activados = [relu(x) for x in valores_salida]
# Result: [0.0, 2.3, 0.0, 0.0, 5.1]

It seems incredible that a simple function that converts negative numbers into zero has solved the mathematical problem that prevented training networks with many hidden layers. Besides, it is computationally extremely cheap to calculate on a GPU.

Reflection: The end of manual algorithm design?

What blows my mind about AlexNet is the paradigm shift. You no longer have to program an algorithm to look for a cat's ears in a photo. If you set up a Convolutional Neural Network (CNN) architecture and throw millions of labeled images at it, the first mathematical layers of the network "learn" on their own to detect edges, the middle layers learn to detect textures, and the final layers identify the cat.

The algorithm learns to extract its own features. If this is true, and GPUs continue to scale in raw power... What will be next?