Synapses are cheap, experiences are expensive

Here is a simple back-of-the envelope calculation based on Hinton’s talk on deep learning:

You live about 102 years. A year is π x 107 seconds, so your life is ~109 seconds. Let’s say you receive 10-100 “experiences” (impressions) per second (brain activity is between few tens to few hundreds of Hz).

Thus, your life is about 1010-11 experiences. Blink an eye and it’s gone.

Your brain has 1011 neurons, with average connectivity of almost 104. That is a total of 1015 synapses.

You have thus about ~104-5 synapses per experience. There is no way the brain could fit a proper model (in the statistical sense), since Nparameters >> Ndata. Instead it has to strongly rely on regularization and sparsity.

Experiences are exceedingly expansive, a synapse is very cheap.

A synapse. Ugh, look at it… so cheap!

This puts all the tired blank slate arguments to grave – brains have to come somewhat pre-trained/regularized via genetics.

Also relevant – the argument is actually just a reformulation and generalization of the poverty of the stimulus argument by Noam Chomsky in the field of linguistics (a field now incidentally totally dominated by deep learning).

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s