On the Weakness of Artificial Neural Networks: Energy Efficiency

In 2016, I asked on ai.stackexchange: [...] can $1000.00 buy enough operations per second to be approximately equal to the computational power of a human brain? This led to some interesting followups about the Landauer limit, the relationship between information and energy, and quantum computers. The full thread went as follows:

DJG:

In his 1999 book "The Age of Spiritual Machines", Ray Kurzweil predicted that in 2009, a $1000 computing device would be able to perform a trillion operations per second. Additionally, he claimed that in 2019, a $1000 computing device would be approximately equal to the computational ability of the human brain (due to Moore's Law and exponential growth.)

Did Kurzweil's first prediction come true? Are we on pace for his second prediction to come true? If not, how many years off are we?

BlindKungFuMaster:

"The development of CPUs didn't quite keep up with Kurzweil's predictions. But if you also allow for GPUs, his prediction for 2009 is pretty accurate.

I think Moore's law slowed down recently and has now been pretty much abandoned by the industry. How much that will affect the 2019 prediction remains to be seen. Maybe the industry will hit its stride again with non-silicon based chips, maybe not.

And of course whether hitting Kurzweil's estimate of the computing power of the human brain will make an appreciable difference for the development of [Artificial General Intelligence] is another question altogether."

Ankur:

1) Yes we do have computing systems that does fall in to teraFLOPS range.

2) The human brain is a biological system and saying it has some sort of FLOPS ability is just plain dumb because there is no way to take a human brain and measure it's FLOPS. You could say "hey by looking at the neurons activity using fMRI we can reach some sort of approximation" but comparing the result of this approach with the way FLOPS are measured in computers will be comparing apples with oranges, which again is dumb.

DJG: Why don't we measure it in energy consumed instead, with some sort of efficiency factor that denotes how much of the heat is being generated by useful computation (as opposed to supportive biological processes?)

Ankur: Heat is just another factor to optimise. You want to maximise the FLOPS and minimise heat generation (aka energy consumption of the system). People, in high performance computing, first focus to maximise FLOPS generally as they want their algorithms to run fast and later focus on heat depending on the requirements.

Peteris: it's not a currently useful measure, because the "heat being generated by useful computation" (e.g. Landauer limit) is many orders of magnitude smaller than the waste heat of even the most efficient computing devices that we can build. For both modern electronic computers and biological neurons, despite their enormous efficiency differences, they still effectively are 100% waste heat, spending millions of times more power than theoretically necessary for that computation.

DJG: Perhaps this indicates a flaw in or incompletion of the theory.

Ankur: Given a list of 100 numbers, who would be fast[er]to compute their sum, a computer or a human? This simple example shows that what human brain does and what computers does are completely different things. We built computers to perform a given sequence of arithmetic and logic operations as fast as possible because human brains are very very slow to do this task.

DJG: The average person would be slower than a computer. Certain autistic individuals might be faster.

This piqued my curiosity about the Landauer limit, so I asked about it in another thread:

https://physics.stackexchange.com/questions/357998/is-there-a-trade-off-between-computation-speed-and-heat-creation-in-relation-to

DJG: In this article, an experiment is referenced in which information was converted into energy via erasure. It is said that the slower the erasure took place, the less energy was released, and that the Landauer limit was approached as the length of the erasure approached infinity.

Is this trade-off inevitable, or did it have to do with how the experiment was performed? Has the trade-off been quantified?

That question has since been seemingly answered with a reference to adiabaticity.

Sadly, the article (which is on nature.com) starts with the statement "forgetting always takes a little energy", which is essentially the opposite of what the described results really say, which is that resetting a bit actually RELEASES (gives off) energy.

Nevertheless, it does reference many interesting results, including this one, involving the conversion of information to energy:

"In 2010, physicists in Japan showed that information can indeed be converted to energy by selectively exploiting random thermal fluctuations, just as Maxwell’s demon uses its ‘knowledge’ of molecular motions to build up a reservoir of heat² (see Demonic device converts information to energy). But Jarzynski points out that the work also demonstrated that selectivity requires the information about fluctuations to be stored."

The article ends on an interesting note about quantum computers and the Landauer limit:

"Meanwhile, in fledgling quantum computers, which exploit the rules of quantum physics to achieve greater processing power, this limitation is already being confronted. “Logic processing in quantum computers already is well within the Landauer regime,” says physicist Seth Lloyd of the Massachusetts Institute of Technology in Cambridge. “One has to worry about Landauer's principle all the time.”

---

Today, I ask another question: Yes, we may be able to rival the raw computational output of the human brain today -- but can we do it efficiently?

The human brain uses only about 40 Watts of power when it's awake. That's comparable to the power consumed by a lightbulb. Meanwhile, my newly-built $1500 custom desktop PC's power supply can supply about 750W. A typical microwave oven uses 900W or more; a refrigerator uses in the same ballpark.