Can Closed-Source Models Be Distilled? Knowledge Distillation for Generative Language Models

Recently, distillation has become more than a technical word. Anthropic describes the unauthorized, bulk-based use of its model output to train competition models as “distillation attacks”I'm sorry. In Chinese, the question is often answered: how can the closed-source model be distilled without the public weight?

The questions in this article can also be addressedPost-linguistic training and fine-tuning of practice: from SFT, LoRA to human alignment、Data synthesis is becoming a project: from Terminal-CorpusHow the concept of a relatively close read together is developed in different contexts.

It looks like a yes-or-no problem, actually mixing up different levels. Neither the direct answer to “can” nor “can” is sufficient.

The teacher model provides a signal.
What exactly is the student training plan?
Do you have the right to use models and data like this?

If these three things are not removed, it is easy to slip the discussion to the slogan: while all model aids are called distillation, the other side, because the model has no public weight, asserts that it cannot be distilled.

Focus on the training chain, whether the model is open or closed.

Classic KD: Students come in direct to teachers

The core of the classic Knowledge Division (KD) is simple: teachers have learned to behave and students are trained to reduce their differences directly from teachers.

The typical thing is...Output distillationI'm sorry. The question raised by Hinton and others about the soft classes not only tells students what the correct categories are, but also retains information on how close the teachers think the other categories are to the correct answers. In classification tasks, students are drawn up by the probability distribution given by the teacher ' s output layer. In the generation model, this distribution becomes the next probability distribution for token; it is the teacher who is more inclined to say what he/she will say at every step.

In addition, there are two types of classic practices:

Characteristic distillation: Alignment hide layer, attitudinal or intermediate. FitNets' instinct is that students should not only copy the final teacher response card, but also use the teacher ' s intermediary advice.
Relationship distillationInstead of forcing the two models to look the same at each level, students are left to the teacher to express the distance, angle, similarity, etc. in space.

These three methods are all doing the same thing:The goal of student excellence is more like a teacher. Characteristic distillation and relationship distillation usually require access to the hidden layer or weight, so it is more appropriate for the white box scene; the classical token-level KD has the lowest threshold, but still requires access to the final logits and needs to ensure that the vocabulary of students and teachers is aligned.

For LLM, the black box model, although not complete logits, can give a string of outputs tokens. Students target this token and train themselves to continue to produce with cross-critics. This form of loss is the same as self-repatriation pre-training, and the difference is simply that the monitoring signal is replaced by the original language into the teacher's output. The final answer, long-debate text, JSON, functional call and tool call track, as long as it is used to train students, is an output or sequence-level behavioural imitation. The training signal has not changed: students are still reproducing a sequence that has been given. This is what Kim and Rush call a security-level KD; from the data line, it's also like using teachers' answers as a false label for SFT.

This border need not be too categorical. The more useful question is:When students are updated, are they learning directly from teachers or in a redesigned data and feedback system?

It's another training link.

To involve stronger models in training does not mean that the entire output is moved into the training set.

Models can also be just a tool in the data production and assessment chain: scaling up seed tasks, constructing counter-scenes and dilemmas, generating candidate answers, helping to mark preferences, or acting as filters. The trainers then put it together with the search for evidence, rule-checking, unit testing, manual auditing or incentive models to generate data required for SFT, preference optimization or RL.

Self-Instract is here: the command can expand input reporting data without requiring that each sample be written manually from zero. But model generation is not the same as natural high quality. The choice of the distribution of tasks, which samples are retained, what is correct, which acts should be rejected, what to reward, and how to validate, still depends on the trainers to judge.

That is why I want to separate the two links.

The goal of the pure KD is to reduce the distance between students and teachers. As a technical tool, it can certainly be used in home-grown models, clearly mandated models, or in the training chain for teachers-students within the team. But if the context becomes "Closed-source-Power Model Output" and small models, as much as possible, it looks more like a behavioral reproduction: Trainers are hardly part of their judgment, and the goal is to repeat what the teacher said. It's a lack of taste, you're not training a model, but is replicating, not deciding what to teach, what not to learn.

Model-aided SFT or RL are different things. The value here is not only what teachers say, but also how people design curricula, constrain data sources, join certification machines, organize difficulties, define rewards and fail borders. It does not have to faithfully replicate every word of the teacher, and it can even clearly filter out the teachers ' bad answers. It is a technology of data synthesis, not a mere reproduction.

Nor can it be called as a substitute for content. A large number of teachers were arrested, the same text was inserted into the training set, and no new mission design and quality control was available, even if it was called synthetic data, which was another output. Conversely, the same COT or tool tracks can also be teaching materials, provided they are validated, reorganized and placed in a new task and assessment chain.The difference is not what the text looks like, but what the training links look like.

On-policy KD: Why do students write first?

The above distinguishes between training objectives. Even if the target remains distilled for the self-regression model, there is a problem in the training process: students see different prefixes when they are trained and deployed.

Normal offline KDs are often this way:

Real data or teachers continue to copy students imitating.

But when deployed, students do not face ideal prefixes provided by teachers, but rather those they have just written. It may have misled an entity, missed a condition, or picked the wrong parameter in the tool call. It can only continue to go down in this distorted context.

On-policy KD:

The student teacher gives a distribution or feedback on the student's own trajectory.

It deals with a mismatch in the distribution of the state between training and deployment, rather than with the creation of a further data set. MiniLM discusses LLM distillation from the point of view of on-policy and reverse-KL; GKD directly studies teacher feedback on student generation.

Therefore, the fact that the data are not the third type of data synthesis is not equal to the fact that the teachers produce the data and then make RL. It also targets teacher behaviour, but only allows teachers to give feedback on the trajectory that students actually reach. This can be achieved by using tools close to the tactical optimization, but it is still discussed in distillation.

Black and white boxes: only limited to signals, not conclusions

On-policy KD asks teachers on which tracks to give feedback. And then, what we're going to ask is what kind of signal the teacher can give. The difference between open and closed sources is mainly here, rather than deciding directly whether or not to distill.

The white box teacher.: access logits, hide layers, attention and relationship structure. The three types of KD that are output, feature, relationship can be established and can be more easily on-policy KD.
Only the final text of the black box teacher.: usually provides answers, reasoning texts, tool tracks or preferred judgements, but cannot be distilled from the hidden layer or the full vocabulary of logits KD.
Interface with logprobs: In the middle. Even if weights are not disclosed, KD may still be limited if sufficient token probabilities are obtained; this is not the white box feature distillation or relationship distillation.

This also means that closed source models cannot be distilled too full. More specifically:

Closed-source models do not necessarily support white box distillation; only text-output interfaces do not necessarily support classic token-level KD; however, they may still provide information that can be recalculated or processed into SFT/RL data.

The question of whether the closed source model can be distilled has reduced several layers to a single sentence and is therefore not suitable for a one-size-fits-all answer. The same paragraph, which is exported by the teacher, is a whole-student training package, allowing the student to reset the teacher as much as possible, is a replica of behaviour; when the trainee re-decides the task, screens, validates and rewards, the teacher is only one source of material. These choices are what I say is about: what is worth teaching, what is credible, what mistakes must be rejected. Whether the closed-source model is used is only a surface layer, and the real gap is whether the trainers put these judgments in the training chain.

References

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, Distilling the Knowledge in a Neural Network
Adriana Romero et al., FitNets: Hints for Thin Deep Nets
Wonpyo Park et al., Relational Knowledge Distillation
Yoon Kim, Alexander M. Rush, Sequence-Level Knowledge Distillation
Yizhong Wang et al., Self-Instruct: Aligning Language Models with Self-Generated Instructions
Zuyang Gu et al., MiniLLM: Knowledge Distillation of Large Language Models
Rishabh Agarwal et al., GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models
Florian Tramèr et al., Stealing Machine Learning Models via Prediction APIs
Anthropic, Detecting and preventing distillation attacks