On the Weakness of Artificial Neural Networks: Energy Efficiency

In 2016, I asked on ai.stackexchange: [...] can $1000.00 buy enough operations per second to be approximately equal to the computational power of a human brain? This led to some interesting followups about the Landauer limit, the relationship between information and energy, and quantum computers. The full thread went as follows:

DJG:

In his 1999 book "The Age of Spiritual Machines", Ray Kurzweil predicted that in 2009, a $1000 computing device would be able to perform a trillion operations per second. Additionally, he claimed that in 2019, a $1000 computing device would be approximately equal to the computational ability of the human brain (due to Moore's Law and exponential growth.)

Did Kurzweil's first prediction come true? Are we on pace for his second prediction to come true? If not, how many years off are we?


BlindKungFuMaster:

"The development of CPUs didn't quite keep up with Kurzweil's predictions. But if you also allow for GPUs, his prediction for 2009 is pretty accurate.

I think Moore's law slowed down recently and has now been pretty much abandoned by the industry. How much that will affect the 2019 prediction remains to be seen. Maybe the industry will hit its stride again with non-silicon based chips, maybe not.

And of course whether hitting Kurzweil's estimate of the computing power of the human brain will make an appreciable difference for the development of [Artificial General Intelligence] is another question altogether."


Ankur:

1) Yes we do have computing systems that does fall in to teraFLOPS range.

2) The human brain is a biological system and saying it has some sort of FLOPS ability is just plain dumb because there is no way to take a human brain and measure it's FLOPS. You could say "hey by looking at the neurons activity using fMRI we can reach some sort of approximation" but comparing the result of this approach with the way FLOPS are measured in computers will be comparing apples with oranges, which again is dumb.


DJG: Why don't we measure it in energy consumed instead, with some sort of efficiency factor that denotes how much of the heat is being generated by useful computation (as opposed to supportive biological processes?)

Ankur: Heat is just another factor to optimise. You want to maximise the FLOPS and minimise heat generation (aka energy consumption of the system). People, in high performance computing, first focus to maximise FLOPS generally as they want their algorithms to run fast and later focus on heat depending on the requirements.

Peteris: it's not a currently useful measure, because the "heat being generated by useful computation" (e.g. Landauer limit) is many orders of magnitude smaller than the waste heat of even the most efficient computing devices that we can build. For both modern electronic computers and biological neurons, despite their enormous efficiency differences, they still effectively are 100% waste heat, spending millions of times more power than theoretically necessary for that computation.

DJG: Perhaps this indicates a flaw in or incompletion of the theory.

Ankur: Given a list of 100 numbers, who would be fast[er]to compute their sum, a computer or a human? This simple example shows that what human brain does and what computers does are completely different things. We built computers to perform a given sequence of arithmetic and logic operations as fast as possible because human brains are very very slow to do this task.

DJG: The average person would be slower than a computer. Certain autistic individuals might be faster.


This piqued my curiosity about the Landauer limit, so I asked about it in another thread:

https://physics.stackexchange.com/questions/357998/is-there-a-trade-off-between-computation-speed-and-heat-creation-in-relation-to

DJG: In this article, an experiment is referenced in which information was converted into energy via erasure. It is said that the slower the erasure took place, the less energy was released, and that the Landauer limit was approached as the length of the erasure approached infinity.

Is this trade-off inevitable, or did it have to do with how the experiment was performed? Has the trade-off been quantified?


That question has since been seemingly answered with a reference to adiabaticity.

Sadly, the article (which is on nature.com) starts with the statement "forgetting always takes a little energy", which is essentially the opposite of what the described results really say, which is that resetting a bit actually RELEASES (gives off) energy.

Nevertheless, it does reference many interesting results, including this one, involving the conversion of information to energy:

"In 2010, physicists in Japan showed that information can indeed be converted to energy by selectively exploiting random thermal fluctuations, just as Maxwell’s demon uses its ‘knowledge’ of molecular motions to build up a reservoir of heat2 (see Demonic device converts information to energy). But Jarzynski points out that the work also demonstrated that selectivity requires the information about fluctuations to be stored."

The article ends on an interesting note about quantum computers and the Landauer limit:

"Meanwhile, in fledgling quantum computers, which exploit the rules of quantum physics to achieve greater processing power, this limitation is already being confronted. “Logic processing in quantum computers already is well within the Landauer regime,” says physicist Seth Lloyd of the Massachusetts Institute of Technology in Cambridge. “One has to worry about Landauer's principle all the time.”

---

Today, I ask another question: Yes, we may be able to rival the raw computational output of the human brain today -- but can we do it efficiently?

The human brain uses only about 40 Watts of power when it's awake. That's comparable to the power consumed by a lightbulb. Meanwhile, my newly-built $1500 custom desktop PC's power supply can supply about 750W. A typical microwave oven uses 900W or more; a refrigerator uses in the same ballpark. 

The Moon or Mars?

Foreword

Below is a paper I wrote in my first semester of college (Fall of 2009). Influenced significantly by Robert Zubrin's book "The Case for Mars," it compares the prospects of missions to the Moon and missions to Mars and, along the way, attempts to describe the general state of affairs at that point in time.

I no longer have quite the same values or fears as I did when I wrote this. I am less concerned with the prospect of humanity being extinguished by some unforeseeable event. While this is a valid motivation for the proliferation of our species in the long run, it does not seem so likely to occur within the 50 years. Nevertheless, the exploration of space, the utilization of the resources that exist there, and the eventual colonization of other celestial bodies remain lofty goals that will continue to be pursued for centuries to come.

---

Through the decades since the Apollo 11 moon landing of 1969 and the subsequent Apollo missions through 1972, humanity has never again set foot on a celestial body. As we look forward to the future of manned space exploration, the question is, where should we go next, and what should we do when we get there? Two primary options have emerged: we can return to the Moon, or we can press outward to Mars. The debate between these two options holds implications of cosmic magnitude for what course the future of humanity will take—and how quickly it will take it. Some favor a return to the Moon to continue research there and perhaps develop a permanent human presence, a sustainable lunar base: they want to understand and colonize the Moon. Others argue for the undertaking of an entirely new endeavor: to send humans to Mars, both for temporary missions and to begin the long process of colonization. There are powerful reasons for both options, but considering the lack of widespread and committed public and global interest, it would be entirely infeasible to push for both—and so the proponents of each plan are forced to battle it out and fight for their own side to the [current] exclusion of the other.

One difference between the Moon and Mars is immediately obvious and has tremendous importance in terms of mission control and adaptivity: the distance. The moon orbits at an average of about 385,000 km from Earth,  never reaching a distance of more than 410,000 km. In comparison, the closest that Mars ever gets to earth is about 54.6 million km, and it can retreat as far away 401 million km. As one blog commenter pointed out, this means that if something goes wrong on a mission to the Moon, help is only three days away, whereas if a problem were to arise after a mission to Mars got underway, it could be months before the astronauts could return home. So, in theory, if undertaking a Mars run as opposed to one to the Moon, we need to be even more sure that the mission and all the equipment utilized in it is very reliable. But some unique aspects of travel in space—such as low gravity and cosmic radiation—cannot truly be replicated on Earth, and so some argue that Moon missions ought to be used as a proving grounds of sorts to ascertain the reliability of the instruments and technologies in question before sending them on a Mars mission.

Another facet of the distance difference is that light and other electromagnetic waves, including radio signals, take only about 1.3 seconds to reach the Moon, while they can take anywhere between three and twenty-three minutes to reach Mars (depending on its current position in relation to Earth.) This makes communications much more delayed for trips to Mars than for trips to the Moon—a fact that can be used to argue for either side. It means that robots on the Moon can be controlled from Earth’s surface in something close to real-time, making robotic missions to the Moon more similar to human missions in terms of their capacity to adapt quickly to unexpected obstacles. As a result, it may be very feasible to use robots to build a base for human habitation on the Moon prior to any future human landings, as reported by Marcel Williams in his New Papyrus article. On the other hand, the very fact that robotic missions to Mars have much less ability to react to unforeseen challenges enhances the benefits of having humans present on the mission. So we see that different aspects of the distance difference might be used to promote either cause, sending men back to the Moon or sending them to Mars.

While the issue of distance mainly brings up points of deterrence to the two options, mainly to the Mars prospect, perhaps a larger amount of the debate lies in what there is to gain from each destination. Donald Rapp posits in his book that the primary motivation to go to Mars expressed by supporters and skeptics alike is to search for life there, but Robert Zubrin—an avid and prominent Mars supporter—puts forth a number of other reasons. Zubrin, in his book The Case For Mars, works towards the point that the final goal of travel to Mars is not just research, but colonization. If one considers Zubrin’s a valid proposition, then the question of sustainability independent of Earth becomes very important in deciding which body to colonize most intensively. One of the strongest advantages that Mars has over the Moon in this sense is the Red Planet’s carbon dioxide atmosphere, which provides in theory an almost unlimited supply of carbon and oxygen to use for chemical manufacturing of rocket propellant, negating the need to bring those two elements along as cargo on missions to Mars. There is also a significant amount of water in the form of permafrost in the Martian regolith, and perhaps a much larger amount of ice deeper in the crust. The combination of this ice and the carbon dioxide in the atmosphere and in the polar caps means that all three of the essential organic elements (carbon, hydrogen, and oxygen) are present on Mars, giving it a much greater potential for self-sufficiency than the Moon. Noting the length of the Martian day to be only very slightly longer than Earth’s, at 24 hours, 39 minutes, and 35 seconds, it is even considered a possibility to terraform Mars in the future, making it into a world with a habitable atmosphere, climate, and biosphere, like Earth. The Moon has absolutely no foreseeable capacity for this, due to its much smaller surface gravity (about 16.5 percent of Earth’s, compared to Mars’ 37.6 percent,) which is linked to its lack of an atmosphere or the potential to maintain one of practical significance.

However, all the talk of terraforming is still considered to be on the verge of science fiction by those comparing the missions’ worth in the more immediate future. Going back to the topic of the search for past or present life on Mars, the argument is that a human mission’s advantages over a robotic one are small compared to the extra cost and risk involved in sending humans to Mars. So the idea now is to get humans back on the moon, at first for transient missions and then to establish a permanent human presence there. Its low gravity makes it an optimal option for a launch pad for future satellites in all sorts of commercial industries, since much less fuel would be required to escape the Moon’s gravitational pull and nonexistent atmosphere than the very large amounts of fuel currently necessary to put any payload in space.

This possible future in using the Moon as the primary base of operations for satellite launches illuminates a major potential paradigm shift in the future of space operations: the privatization of the space sector. Throughout its recent history, NASA has been left wanting for funding, and things are not looking towards improvement in today’s uncertain economy. The Human Space Flight Committee said recently that NASA’s current goals for human spaceflight are impossible without a budget increase of two to three billion dollars per year. Whether this funding increase will occur is unclear… so there’s a lot of talk among space-supporters about a shift from almost purely government-funded programs to a lot more privately-funded operations, motivated of course by the prospect of profit. 

In addition to the likely advent of Moon-based launches for commercial satellites, there may be other commercial options waiting to be explored in space when the time is right. Space mining will almost certainly become a major business in the future; it’s a matter of when, and the first companies to lay claims (once the United Nations develops the guiding principles of property law in space) will have a leg up. A surveying business will inevitably accompany future space business sectors of many kinds, including perhaps a space tourism sector, if the market is right for it, and a residential sector, once colonization methods are proven and established to be safe for the sake of the general populace. All of these apply to both the Moon and Mars, but probably more so to the Moon in the more proximal future, due to the distance difference.

The reader may be wondering, then: What exactly does NASA plan to do? With which side of the debate does current policy lie? It sides with the Moon supporters. The current objective of NASA is to land humans back on the Moon by the year 2020, “in preparation for human exploration of Mars and other destinations,” quoted from NASA’s Vision for Space Exploration. Beyond that goal, no firm plans have been established for what path the space program will follow, leaving further decisions pending, presumably until the success of the planned human returns to the Moon. 

Of course, policies are always subject to change, especially with a change of the presidential regime. This is one of the reasons that Zubrin cites when he argues that an essential element for the success of any large undertaking of the space program is a shortened time frame within which the project is to be completed or the goal is to be attained. He harks back to the Apollo missions of the 1960’s, when on May 25, 1961, President John F. Kennedy set the goal of a United States manned moon landing and subsequent safe return to earth before the end of the decade. The goal was ambitious, to say the very least, and yet because of the concerted, unified effort, driven in part by the Cold War and the desire to catch up with and overtake the Russian space program, we reached that goal. It was a gigantic triumph not just for the United States, but for the entire world, capturing the imaginations of an entire generation. That project employed 400,000 people and involved over 20,000 companies and universities, spurring an unprecedented burst in technological advancements required to complete the objective. Through all the technical obstacles that had to be overcome over the course of the program, the goal was ultimately achieved. In light of this, it honestly seems somewhat pathetic to me that we would set a goal in 2005, forty-four years after the President’s original one in 1961, to simply return to the Moon by 2020—essentially to attempt to repeat a mission that could literally be defined as antique, in a time frame half-again as long as the very first time we ever did it, with the technology of forty years ago. To use blunt simplicity: This idea does not similarly capture the imagination.

So what is my stance on how manned space exploration should progress? I favor that same head-over-heels, get-it-done-and-fast kind of approach that got us to the Moon and back in 1969, and that doesn’t mean aiming for a repeat feat fifteen years from now. Whether it’s to the Moon or to Mars, I think we must aim for the stars with the intent of landing there to stay, and that means that the establishment of a permanent extraterrestrial human presence should be a central goal, if not the singular defining objective, of future programs and missions. Some argue that we should fix our economy before looking seriously to space once again, but who’s to say that the push towards colonization would not stimulate the economy as well as any other plan might? Funding is an issue currently for NASA—but isn’t that a foreseeable result of the lack of truly inspiring objectives? What if we set a goal to have a human population of at least fifty established on the Moon before 2030—or we decided that we would land the first humans on Mars before 2020? Both Zubrin and Rapp agree that such a goal has the potential to inspire a new generation of young minds, perhaps helping to give the United States the boost it’s looking for in science education.


To me, though, it is none of these reasons that takes true precedence. To me, the reason why we have to colonize and expand as soon as possible is to combat the chance, however small, that some tremendous, unforeseen disaster will wipe out the whole of humanity. There are many conceivable threats to our species’ existence—a supervirus pandemic, an asteroid or other kind of cosmic impact, radical climate change, and nuclear war, to name a few—and to me, even the smallest chance that one of these might occur is sufficient motivation to make immediate, focused efforts to ensure the survival of our species as best we can. That means that we have to proliferate—to plant our seed in as many places as we can, first in our own solar system, and eventually throughout others, light years apart, so that we can be sure that we can continue the legacy of life, which, to the best of our knowledge, is our unique legacy in the universe. It seems to me one of the noblest of all possible causes, one that I would certainly be willing to contribute my tax dollars to, and perhaps my career and life as well. And so although I am in favor of all exploratory and colonization efforts, I more strongly support the Mars position, as it more honestly captures the spirit of my cause.

Many consider the two separate sides in different lights, categorizing them to each contain certain kinds of people. The Moon camp is perceived to hold more down-to-earth, realistic types—they want to establish a solid base and take progress slowly but surely. The Mars people, on the other hand, are seen as idealists and visionaries who want to make one grand push to the stars, a push that is “impossible” or “unrealistic” with current technologies and knowledge. But couldn’t it be said that it is exactly this concept—the concept of the conquering the impossible, of taking what has been said cannot be done and doing it anyway—that has always captivated the human imagination and the heart? It drives athletes to perform ever greater feats of precision, endurance, and strength; it drives researchers and theorists to challenge old ways of thinking and come up with new models that come to be accepted as fact; it drives explorers, like Christopher Columbus, to voyage to new worlds and continue there the legacy of the frontier. It is this legacy that Zubrin identifies at the heart of the American spirit, beginning with the first colonists from Europe and continuing as they eventually spread west across North America until the nation spread from coast to coast. But now that frontier is gone, and so we hear: “Space, the Final Frontier.” This it is, but NASA’s uninspiring goals in the current day have made us lose sight of this inspiring thought and caused us to relegate the Space Program to a status and importance somehow downgraded from the one we knew in the ‘60s. It is a vicious cycle: less ambitious goals have led to less perceived importance, which has led to less funding, which has led to less ambitious goals, and so on. 

The difficulty lies in cutting to the root of the problem and reestablishing, in the eyes and minds of the people, the importance of the space program to our future as a culture and a species. To communicate any message to the mass public is an objective that advertisement and propaganda have tried to attain for years, to varying degrees of success. Without going into details, I will propose that to influence the public at large thinks, one must control or modify the underlying, basic premises in the media—and where there are none, as may be the case for space (outside the realm of science fiction,) one must create them. One must be aiming to change outlooks, not minds. So the key to rallying public support for the space program, I think, would be to turn space exploration into an assumed, essential part of the immediate future, perhaps through TV shows, movies, and even commercials.

As for how the reader can take action: for representatives in Congress to support a certain stance, they have to know that it is popular with the people whom they represent. So if you support the continuation and expansion of the space program, it would certainly be wise to write a letter of that effect to your representatives. Joining space societies like the Mars Society or the Planetary Society can also help further the cause as the numbers swell and their potential political influence grows. And you can spread the word—remind people of the magic of Apollo 11 and help them understand how it can and must happen again, for the sake of all mankind.

Terraforming Earth

In the year 2200 -- 185 years from now -- will Earth as we know it look the same?

Or will we build giant space mirrors, melt the ice caps, abandon our coastal cities, and enjoy a global paradise?

What if Alaskans and Brazilians alike could enjoy a temperate climate -- 70 degrees every day with mild humidity and plenty of sunlight to go around?

Sounds good to me! But how?

Simple enough in concept: We divert some sunlight from the equator and redirect it towards the higher latitudes.



In fact, the same concept serves as a potential solution for global warming and climate change. Too much sunlight, too much energy being pumped into the system? No problem -- reduce the input.

But before any of this can happen, we have to develop space. If we can't manufacture the mirrors, goodbye global paradise.

Of course, climate science will have to make some advances too. It might take us a while to decide that it's actually an idea worth pursuing. Some will insist that we don't know the consequences of our actions. But in the end, we'll probably try it and see, because doers run the world.

Information Is King

The space-time continuum is no longer cutting-edge.

Conservation of mass and energy is old news. We are realizing more and more that information matters--that, in a sense, matter is information.


Matter exists only in the extent to which it has the potential to interact with other matter. This interaction can be described as an information exchange between particles -- Particle A tells Particle B, "I'm here!" and imparts a force on Particle B (momentum is exchanged.)

In more than just physics, information is of the utmost importance. The Google brand is worth over one hundred billion dollars, and this since its founding in 1998, less than twenty years ago. A company younger than I am is one of the most valuable in the world, all due to its creative use of data--of information.

To this point, human history has been marked primarily by the increasing availability of fundamental quantities: Energy and material resources. Now more than ever, it is clear that information is one of these fundamental quantities.

It is not energy or matter that determine what happens next. The second law of thermodynamics is that entropy always increases (when the universe is considered as a whole.) The amount of information in the universe is constantly increasing--and our access to that information, and the ways in which we can examine it, are changing as well.

Intangible though it may seem, information may prove to be the most fundamental way of looking at the universe. The idea that the universe we live in is in some sense a simulation -- that our existence is as digitized as the pixels in an Atari game -- has not been discredited, and sometimes seems the only credible explanation for seemingly arbitrary laws of physics.

What we know is but a fraction of what can be known -- and what can be known is only a fraction of what is. The universe is great and wonderful, but our only guarantee is that it will forever be mysterious to us as individuals. Summarize and simplify all you want -- but realize that your compression is not lossless, and there will always be more data.

A Moment For the Past

Fantasies of the future are an integral part of our culture. We dedicate movies to imagined worlds. We write books about what one day might be. We imagine our lives next week, next month, next year, ten years from now. But amidst all of this imagining, we must leave just a little room for remembering.


We live in an incredible world. Things are possible for us that our ancestors could only dream of. There is and always will be progress to be made, but while keeping in mind what remains to be done, we must appreciate what we have.

Our technology enables us to fly, in a single day, across a distance that would have taken our great-great-grandfather's great-great-grandfather years to traverse.

We can communicate with people on the other side of the world at the touch of a button.

The music that pours through our radios on a daily basis would sound, to the Greeks, either alien or magic.

It is easy to compare ourselves only to our modern peers. And it is good that we do--it drives us to climb ever higher. But in the instance where that comparison leads to pain, depression, or feelings of worthlessness, it is time to remember those who have gone before. What was their lot? Would we trade places with them, given the chance? Maybe as a vacation -- but few or none of us would choose to permanently, irreversibly swap lives with the average human of ten thousand, one thousand, or even one hundred years ago.

Stay angry at the obstacles ahead -- but stay happy with what you have.

In-Space Manufacturing is Key

Edited 5/16/20

Postulate: While launch costs remain high, in-space manufacturing is paramount to development.


Right now, our process for creating structures in space goes like:


Gather --> Construct --> Launch --> Assemble

Of these, launch is the most costly by far. [5/20/20: Actually not sure about the math on that.] So, it seems like we should aim to cut it out of the process as much as possible. To do this, we need to move the "Gather" and "Construct" phases into space. Can we modularize our current industrial complex to the extent that it can be launched off-planet, or simply reconstructed in space using matter and energy harvested there?

Some relevant sub-threads:
  • How far are we from a real space elevator? (How long will launch costs remain high? If not too long, then maybe developing our in-space manufacturing capabilities isn't so important after all--but I doubt that this is the case.)
  • Where do we get the materials for in-space manufacturing? (Near-Earth Asteroids come to mind; so does the Moon.)
  • What will mining-in-space look like? What about construction? How do we accomplish them most efficiently? To what extent can they be roboticized?

The Internet and I

---

I am not a specialist. I am not even an amateur. But still, I am an arbiter.

Like you, I float above the world, on the wings of the Internet. Like you, I hang from satellites by invisible wires -- suspended, I can look down and see everything in the world at a glance.

Every website is a portal. At my fingertips are collections of data, served to me on a platter, freely and readily available -- an intellectual soup kitchen. I process so much data each day -- the physical world can hardly compare. I become simply a brain and a set of fingers. I no longer have legs; the physical world around me no longer matters. It is simply a platform from which I can observe the abstract world, the world of news feeds, headlines, technology, opinions, trends, shared videos, blogs, libraries, code.


What does it mean to be alive in 2015?

We are no longer the cavemen of the ancient age. We are hybrids. We are cyborgs. Augmented by data, bound to our flesh. We have needs, but so easily are they fulfilled that most of our lives are spent seeking out entertainment and finding ways to self-express. Our clothes, our news feeds, our blogs, our songs, our paintings, our projects -- all modes of self-expression, all with a digital element. We are Wired, and in the digital forum, our Vices congregate and bid for our attention in the Book of Faces; the power of suggestion lies as much now in the Search Bar as in billboards and TV advertisements. Emotion and Aesthetic are distilled into shows and served at the salad bar known as Netflix. Meanwhile, we munch on popcorn, or ice cream, or strawberries, or Sour Patch Kids. This is everyday life.

To a caveman, it was only a dream. It wasn't even a dream. It was a glimmer of hope, it was a desire to conquer, to improve -- conquer the natural world, improve our selves and our systems. Well, here we are! But paradox is eternal: you know someone who knows Prozac, and Prozac is dying to know you. We prescribe pills for ADHD but still provide "Articles You Might Like." More is still more. More will always be more.

It's fine. I love it. Life is a roller coaster, and it was meant to be. This is the best of all possible worlds. You have no idea what I'm talking about, but our genetic material is a 99% match. I love you, but I don't know you, or even know who you are. Our only interaction is through words on a screen, and you're choosing that above everything else, right this second. 

I love you for it. But more importantly: we both love the Internet. 

@akaDavidGarrett