Compression for AGI: Compression as Intelligence

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Theme: Compression is intelligence: Why is ChatGPT intelligent? This paper is based on the theme-sharing of OpenAI researcher Jack Rae, "Community for AGI." The main lines of discussion were:Basic model training can be understood as compressing effective information as much as possible without loss.

The questions in this article can also be addressedNeal Scaling Laws: From Kaplan to Chinchilla、What's a big model of Los Landscape?How the concept of a relatively close read together is developed in different contexts.

About General Artificial Intelligence

Before explaining why “compression” is a path to universal intelligence, we briefly look back at a classic thought experiment: the Chinese room (John Seal, 1980).

A man who knows nothing about Chinese and speaks only English is locked in a closed room with only one small window. There was a manual with a Chinese and English translation, with sufficient paper and pencils. Paper pieces written in Chinese were sent to the room through a small window. The room will be accessible to those who can translate Chinese into English and then to those who can translate English back into Chinese. Although he was totally unChinese, the outsiders would think he could speak Chinese fluently.

Such a “big manual” clearly corresponds to a low level of intelligence: once it is encountered with input that is not covered, it cannot be responded to.

If we can extract grammar and rules from a large amount of data, the manual can be streamlined; at the same time, the system is more intelligent (more broadly oriented).

The thicker the manual, the weaker the intelligence; the thinr the manual, the stronger the intelligence. Just like companies employ one person: the stronger the ability, the less you need to explain; the weaker the capacity, the more you need to explain.

The example above explains in a visual way why “compression is intelligence”: to get a smaller description (the shortest “handbook”) the system is closer to intelligence.

Generate models and compression

Set to give $D$We can use the generation model. $f$ Compress it:

$$ \lvert D \rvert = -\log P_f(D) + \lvert f \rvert $$

of which $\lvert D \rvert$ The size of the data set, which is no-loss compression, is equal to the sum of the losses projected for the next token, plus the minimum description of the estimated function (here) $\lvert f \rvert$ is an abstract item that describes the minimum length/coding cost and is not equivalent to the parameter amount. At this point, the process of compressing data is the process of training to generate models.

A more compressed expression is available:

$$ r_n = 1 - \frac{S_1}{S_0}

1 - \frac{\lvert f_1 \rvert + n + \sum_{t=1}^{n} -\log P(x_{t+1} \mid x_{1:t}, f_1)}{\lvert f_0 \rvert + n \log m} $$

This explains why larger models often show greater generalization: larger models, usually means lower ross, and therefore higher compression rates; in the context of “compression is intelligence”, this should be a shorter and more effective description.

Next Token Prediction Although it may seem simple, it can be justified by the compression theory: this is one of the reasons why teams like OpenAI have long insisted on Next Token Protection. In relative terms, the BERT “predict intermediate” is often difficult to align directly to a strong generating capacity in terms of the end application effect.

Limitations and summary

It is not realistic to compress everything: for example, the cost of pixel-level image modelling is enormous. In reality, it is often necessary to identify information fragments that want to be preserved and modelled before finding a way to filter out unwanted irrelevant calculations and information clips, thereby reducing the subset of data being processed before it is compressed without loss.

From the perspective of “compressed intelligence”, the current single-state model aims to continue to improve the effective information compression capability, while the multi-modular model is tasked with finding a compressed modelling method for complex modulate information. BPE (Byte-Pair Encoding) can process the word-formulation of the text, and the method based on statistical frequency is not only efficient, but also ultimately significant, with the greater disadvantage that it may be unfriendly to small languages; however, the sound and video-like mosaics still need more appropriate dissemination and expression.

Many of the data in reality may not be directly observed and cannot simply be expected to be achieved by compressing “all observations”. A more conservative understanding is that compression provides a path to explain the capability of the underlying model to be generalized, but it also needs to be discussed with data selection, modelling, interactive environments, etc.