A mind is like a parachute
It might save your life,
but you have to know how to use it first.

Sunday, May 12, 2013

Wheel of Fortune, The Mill Pond, Black Swans and Entropy

That's not just a whacky title.  All those things are related in a pretty simple way that will help us unlock some insights about information and why Black Swans are so transformative.

Let's start with the gameshow Wheel of Fortune.  The puzzle is solved by filling letters into the blanks.  As the puzzle gets more filled in the puzzle gets easier to solve.  Take this puzzle for example:

Historical Figure

_ _ _ _ _ _    _ _ _ _ _ _ _ _ _ _

Do you know who it is?

Maybe this will help.  I'll give you a 't':

_ _ _ _ _ _    _ _ _ _ _ _ _ T _ _

Know it yet?

How about if we buy a vowel.  An 'e':

_ E _ _ _ E    _ _ _ _ _ _ _ T _ _ 

Maybe take an 'r'"

_ E _ R _ E    _ _ _ _ _ _ _ T _ _ 

Still nothing?  The most common selection of consonants on "wheel" can be remembered by the word "Translate" or in other words:  TRNSL, so let's go ahead and add the 'n':

_ E _ R _ E    _ _ _ _ _ N _ T _ N

and the 's':

_ E _ R _ E    _ _ S _ _ N _ T _ N

If you are a good player, you probably got this long ago, since you know what letter combos are suggested when certain letters appear in certain places, but I'll just go ahead and throw a 'g' out there for fun:

G E _ RG E    _ _ _ _ _ N G T _ N

Ready to solve now?  Of course you are.  But wait a minute.  There are still seven blanks in the name. What does it say about these remaining letters if we do not need them in order to decode the name?  It says, according to Claude Shannon, that they do not contain as much information as the other letters do.  The fact that we know the name is "George Washington" without it being spelled out entirely has to do with how much information is carried by the letters we do have.  Once we "see where this is going" we don't need the other letters, except maybe as some sort of fail safe verification of the message.  The first few letters have done all the heavy lifting -- they have done all the work.  That is a very important phrase that we will get back to.  The first few pieces of data have done all the work.

Working at the Mill

Hey, speaking of something that does a lot of work, have you seen that waterwheel?  No seriously, the Mill Pond and Waterwheel have a place in this conversation.  Let's find out how.

We need to start with how the waterwheel at a mill works.  The basic model is that moving water causes the wheel to turn.  The shaft of the moving wheel is connected inside the mill to some machine using gears.  In a very simple example, the moving shaft turns a millstone (or grindstone) in order to crush grain into flour.  Other more complicated variants exist, but each uses the movement of the wheel to accomplish some work.

The classic example of a mill is the "overshot waterwheel" where water from above the wheel spills onto it and causes it to turn before settling in the mill pond.

So what does the actual work here?  The water does the work.  But how? By falling down on the wheel and causing it to turn.  You could make a sensible case that gravity is doing the work.  But that's part of the answer isn't it?  Work is done by the water because it starts off at a place that is higher and ends up at a place that is lower.  All we are really doing is capturing the power of the water flow, and that flow is caused by gravity.  In physics terms, the water is moving from a state of higher potential energy to a state of lower potential energy.  The energy it releases in this process is energy of motion, or kinetic energy.  So the water is all full of potential at the top, releases its energy on the way down, and comes to rest in the mill pond at a lower state of energy.  Now in the real world, unless the mill pond is at sea level, there is still a bunch of potential energy in that water -- it could flow from the mill pond into another brook and continue to move down the mountain to another water wheel and mill pond if it happened to be located there.  But all that's important for our model is to realize what takes place at this one mill.  The water starts high, drives the wheel as it moves to a lower state of energy, and comes to rest having lost some of its energy but having done work in the process.

How much work the water can do is determined by the difference between its potential energy at the start of the process and its potential energy at the end of the process.  And that is determined by how high the water starts and how low it ends up.

Entropy is the Same Thing, Only with Heat instead of Water

The concept of entropy is the same thing we see with the mill pond and water wheel.  Substances which are hot contain a lot of ability to do work, but only if they can transfer their heat to something which is cold.  This is a lot like the water starting out high (hot) and moving into something lower (cold).  The ability of heat energy to do work is measured by the temperature difference between it and the thing it is flowing into -- the same way the ability of the water to do work is measured by how far it falls from start to end.  

For a simple example, let's look at a steam engine.  The water is heated and it is turned to steam.  The expansion of the steam gas (based on how hot it is) drives a piston which turns a shaft and we are right where we were with the mill.  The shaft can be connected to anything.  The turning of the shaft is the work accomplished by the steam.  The rest of the engine is devoted to condensing the steam back to water as it cools and pumping it back to the furnace so it can be re-heated.

There are No Steam Engines in Hell

Here's the funny thing about steam engines, though.  They achieve the greatest efficiency when the steam is as hot as can be (because it expands rapidly) and it is vented to a chamber which is as cold as possible (so the steam can be quickly condensed back to water).  If the condensation chamber is at or near the boiling point of water, the steam will not condense and the engine will not work or will hardly work at all.  This would be like having a water wheel which is driven by water that was only falling an inch or two -- there would not be much chance to release any energy.   In both cases the release of energy is tied to the difference between the two energy states.  Hot water (steam) moving into cold water accomplishes a great deal more work than if the temperature difference is small.

So What Does Entropy Measure?

As you may have guessed, entropy says something about the ability of the heat energy to do work.  The higher the entropy the greater the "spent fuel".   It might be easier for us if higher entropy meant greater ability to do work, but it was not set up that way.  The tendency of a thermodynamic (heat energy) system towards greater entropy simply means that heat tends toward equilibrium, and we already know that if everything is equal not much work can be done.  Low entropy means greater differences between the heat and cold and so greater potential work.  The law of entropy suggests that the universe is gradually mixing together its cold and its hot and will someday be nothing but a big bowl of lukewarm.

Fewer Missing Letters Means Higher Entropy

So let's get back to our Wheel of Fortune game and revisit what Shannon had to say about information and entropy.  In a nutshell what he is saying is that the first piece of data in a blank puzzle has the greatest ability to do work.  In a water wheel we measured the ability to do work by the difference between the high point and the low point.  In a steam engine we measured the ability to do work by the difference between the hottest point and the coldest point.  Well in an information system, we measure potential work by the difference between the blank puzzle and the answer.  The greatest ability to do work is when we have no information at all and we need to arrive at a message.  That first letter accomplishes so much in telling us what the answer is -- far more than the last letter does.  So by the time we have the answer nearly solved:

G E _ RG E    _ _ _ _ _ N G T _ N

Each new letter does hardly any work at all.  The work is measured by the distance we still have to travel to get to the answer and frankly that difference here is not very large.  Sorry, 'o', but you are just not doing much for us right now.  We don't even need you to get the job done.  'G', on the other hand, good job.  Your work was excellent.

One Last Step

Now if we look at the message as being "reality" and information as being that data that reveals reality to us, we need to see the blank puzzle as a state of cluelessness about reality, a literal blank slate.  We only learn about the "Truth" by receiving the information that explains the world to us.  The bigger difference between our blank slate and reality, the more work each piece of information can do.  And that is why Shannon viewed the measure of information as the level of surprise contained in each new piece of data.  If the new data is a 'W' to start the second word, well that is pretty much what we expected.  That information is not doing much work.  If the answer was actually "George Dashington", however, the appearance of a 'D' would rock our world and set our previous assumption about reality on its ear.  That 'D' would be doing a hell of a lot of work.

And as you can see here, George Dashington appears in the 1940 census, so he is in some sense a "historical figure".

So now the question is, was 'D' a Black Swan?  After all who was expecting a 'D' for the beginning of the second name!

Maybe that's pushing it, but there is something interesting we can say about Black Swans in relation to this conversation.

The Black Swan Carries a Lot of Information

If, as we have said before, the "potential energy" of information has to do with the difference between the Answer ("reality") and the Puzzle as we have it filled in so far, then a Black Swan does indeed contain a great deal of potential energy.  Or viewed another way, the Black Swan carries a lot of information because the Black Swan event reveals the disconnect between what we think is possible and the world as it really is.  A Black Swan really is the moment we see the 'D' and not the 'W', only with matters far more important than a silly word game.

To cite one of our earlier examples, Columbus' voyage did not create the Western Hemisphere.  It was there the whole time.  And the difference between the world as it really was versus the world that the Europeans thought existed was huge.  Therefore there was a great deal of work that would one day be done by the information that flowed into this gap.  From an information standpoint, the work done by Columbus' journey was huge.  As Claude Shannon says, the information is the surprise carried by the data.  Well what a surprise it was to discover that there was this huge extra piece of the Earth.  It is no wonder it is treated as such a substantial event in history.  It released a great deal of information energy when it opened the spigot between the two worlds.

Now it is often rightly cited that Columbus did not "discover" the new world.  In the first place there were already people living there, thank you very much.  But even from the European perspective, he did not discover the Americas.  The first Europeans to visit the Western Hemisphere were the Vikings.  But they did not release the potential energy of the reality gap because their discovery was not shared with the rest of the world.  It is not even clear that they knew they had discovered anything at all.  For that matter, neither did Columbus at first, which is why we have the term "West Indies".  He thought he was in the Indian Ocean.  He thought the people watched him from the shore were "Indians".   But eventually they understood where they really were, and the Black Swan event released a great deal of information energy (the difference between what is and what is thought to be).

So where does this leave us?  We now know that the power or potential energy of information is determined by how much surprise it carries, and this sort of informational entropy is measured by the difference between what is expected (smooth sailing to India for example) and what actually is (such as this huge land mass in the way).   Black Swans, by dint of their being so different from what was expected, bring us a huge amount of information.  How we handle that information and put it into context, or form new context to accommodate the new information will be explored later.

No comments:

Post a Comment