Zipf's Law: From the Voynich Manuscript to Alien Civilizations

Quote: Is that a lie?

The questions in this article can also be addressedStatistics and Truth: How to use the accident (Statistics and Truth)、Anscom Quartet: Visualized power and statistical illusionHow the concept of a relatively close read together is developed in different contexts.

In 1912, the Polish bookkeeper Wilfrid Voynich bought a mysterious manuscript in Italy. The book is full of strange plant illustrations, astrometric charts and naked bathing women, and, more importantly, it is written in a text that has never been seen on Earth.

That's famous. Voynich Manuscript。

For a century, top coders, including experts in the Enigma code, have tried to untangle it, but have found nothing. Some start to wonder: Is this a medieval charade that some medieval liar drew to cheat money? It doesn't mean anything?

Until statisticians get involved. Instead of trying to read every word, they counted the frequency of the appearance of these symbols. They found that these seemingly obscurantized symbols, which are distributed in a word-frequency manner that is consistent with the statistical patterns common in human languages, are not only the most important but also the most important ones.Zipf Law's Law)I'm sorry. This means that whatever it is written, it is likely to be a real, logical language, not random graffiti.

Today, we come to talk about this truly odious law that dominates the universe from the human language to the size of the city, and possibly even the signals of alien civilization.

Simple math, amazing universality.

In 1949, George Kingsley Zipf, a linguist, discovered an amazing phenomenon:

In any English book, if you have all words in order of frequency of occurrence, then:

Number 1 word (usually) "the"The frequency of occurrence is about the word of the second ranking (in the second place)."of"It's a "sweet" Two times.。
About the third word in the ranking (in the first place)"and"It's a "sweet" Three times.。
...
About the top of the ranking. $n$ The word. $n$Double。

In mathematical terms, it's:

$$ f(r) \propto \frac{1}{r^\alpha} $$

of which $f(r)$ It's a frequency.$r$ The blog is a blog that has been published by the blog of the blog @Rank.$\alpha$ Usually close to 1.

Visual magic.

If you draw this distribution on the normal axis, it's a steep drop. "L" . But if you take both the cross-axis (rank) and the vertical axis (frequency) log-Log Plot, the miracle happens:It became a straight, straight, last line, leaning down.- The slope is -1.

How is the world even even a logarithmic?

Interesting application

The law of Chippov is not just a toy of linguists, but it can also work in many unexpected areas.

Looking for aliens.

How do we know if the incoming cosmic radio waves are signals of alien civilization or the noise from a dead neutron star?

The SETI scientist, Zipulance Doyle, has proposed a method based on the law of Zipov:

Pure random noise: If we sort the pattern of the signal by frequency, it is usually flat (Flat) or it shows an exponential decline, and the slope is flat.
Extremely simple signal.(e.g. pulsed): Only repeats the same frequency, lacking information entropy.
Complex language: Must be between “total random” and “total repetition”. The logarithmic frequency slope of human languages is just about -1.

If we capture a waveform in the universe, which is perfectly distributed at a frequency that is in line with the law of Zipov, it is likely that it carries information of some kind of intellectual exchange.

Music's "good" password

Why is Mozart's music so good and cats' piano sound so bad?

It was found that the frequency of the music leap in the beautiful melody was also consistent with the Chippov law. $1/f$ Noise or Pink Noise）。

White Noise: Totally random, too noisy, like a TV snow spot.
Brown Noise: Randomly swimming, too dreary and solphisticated.
1/f Noise: right in between. It's got enough.Predictability"and enough to make you familiar and comfortable."Accident(to surprise you) That's the mathematical essence of art beauty.

The Internet's “long tail effect”

Why would Amazon challenge Wal-Mart? The law of Zipov gives an angle.

In traditional physical bookstores, because of limited shelf space, the merchants are able to sell the most popular “hot books” (head) ranking. The millions of hard-core books that are ranked behind are too low to be worth the price.

But the law of Zipov tells us that although each element of the long tail is low, it's very long.$r$ The blogger says that the government is not a party to the law.The area of the long tail is huge.。

The Amazon ate this part of the long-term market, using virtual shelves that are almost unlimited. This is the long-tail theory that Chris Anderson has put forward — a business model that has moved from being dominated by a few hot commodities to being supported by an endless array of small commodities.

The truth against intuition: is it a miracle or a coincidence?

Since the law of Zipov is so universal, is it hiding some kind of uniform mechanism behind it?

The answer may be disappointing or more interesting.

Monkey type paradox.

The early linguists believe that the law of Zipov represents a human communication strategy: In order to improve efficiency, we prefer to use short words to convey the meaning of high frequency (H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-H-Prince of Last Effort）。

But the mathematician Benoit Mandelbrot poured a cold water bowl. He testified:If you let a monkey knock on a typewriter randomly, the frequency distribution of random words can also be consistent with the law of Chippov!

This suggests that the law of Zipov may not need "smart." It may be just a statistical necessity that is naturally emerging in a combination system, as is the case with normal distribution, as a sort of “default setting” in nature.

The rich get richer.

Another interpretation is much more cruel. Herbert Simon has submitted Preferential Connection The mechanism, which we used to say in sociology. Matthew effect：

"Anything, and he shall be given more than that."

The more a word goes, the easier you'll use it again, even if you don't have a brain.
The larger the city, the easier it is to attract new immigrants.
The more a website is linked, the more easily it is linked to new pages.

Such mechanisms magnify inequalities. In the Zipov distribution, the first element is a huge share of resources. That explains why 1% of people have huge wealth, why Internet traffic is concentrated in the hands of several giants, Google and Netflix.

80/20: Another face of the law of Tsipf.

You must have heard of inequality. Paretto Law (Pareto Prince)It's called. 80/20 Code80% of the wealth is in the hands of 20% of the people.

The Paretto Law and the Tsipov Law describe the same structure, but they see the world differently.。

Zipov's law.Answer: Number one. $r$ How much money is there?Individual）
The Paretto Law.The answer is: how many of these people have more wealth than they have. $x$♪ I'm not sure ♪ "Concerned by theCumulative total）

It can be shown mathematically that if a system is in line with the law of Zipov, then it is also in line with the Paretto distribution. They're like two sides of a coin, one with a long tail, telling us how long it is; the other with a head, telling us how many heads there are.

When we say "the first 20 percent of the vocabulary covers 80 percent of the daily conversation," we're describing the Zipov phenomenon in the Paretto language.

King's effects and defined traps

The law is beautiful, but it often fails. The most famous is... The King's Effect: The first data point (the King) is often unruly, either out of scale or smaller than expected.

Many times, it's because the border is painted wrong. If you count the population of Shanghai City District, you may find it inconsistent with the law of Zipov. But if you count the population of Shanghai Metropolitan Circles (which includes the surrounding regions of Kunshan, Suzhou and so on, which are closely linked to their economies), the pattern is amazing.

This reminds us:The law of Zipov describes the natural boundaries of the organic system, not the administrative boundaries that are divided by humans.

Summary

The law of Zipov is like a fingerprint from a complex system. It appears in books, cities, wealth, and possibly even in radio waves of distant galaxies.

It may represent the efficient (saving) of systems, the cruel Matteo effect (inequity) or simply the random result of maximization of entropy (mix).

But anyway, next time you see that kind of thing,A very small number of them have a very large share, and the vast majority of them have a very small share.And when it happened, it was probably the Chippew Law that worked.$\frac{1}{r^\alpha}$ And it's the frequency of the universe's breathing.