Text Embedding: From Bag-of-Words to Qwen3 Embedding

This English version was machine-translated from the Chinese original. Technical terms may require verification.

From Bag of Words to Qwen3 Embedding describes the development and application of embedded text models.

The purpose of this paper is to provide a comprehensive overview of the core development nexus of text retrieval and expression techniques, and to map a clear evolutionary path from classic statistical models to modern large-scale neuronet models.

We will understand its narrow search paradigm based on word-frequency statistics, starting with the word bag and BM25 algorithms, the cornerstone of information retrieval. It then entered the age of deep learning, reading Word2Vec, which created static words embedded in the river.

Then focus on the two revolutions brought about by Transformer: the deep two-way understanding model represented by BERT and the powerful self-regression generation model represented by GPT, exploring how they can generate revolutionary dynamic contextual embedding.

Finally, taking the example of the recently released Qwen3 Embedding, an in-depth analysis of how the new generation of SOTA (State-of-the-Art) models, based on the ideas of the heirs, is moving the text embedded and reordering technology to a new peak through advanced structures and training strategies.

Text embedding with the beginning of a statistical language model

In order to use the text data for various machine learning model analyses, it needs to be converted into a standardized format. The first question is how to convert text information into flat vectors, which is also the first step of the text data signature project.

Word bag model

One word bag model

In the Bag of Words model,Each text document (a sentence or sentences) is converted into a numerical vector corresponding to a point in high-dimensional space.

This number vector contains all the words that may appear in the vocabulary (or all the words that we need to study) and determines the structure of the vector according to the number of times the word appears in this document. If not, the vector is zero.

The wordbag model does not represent any word level, so it is called a flat model. It's a natural problem. "not bad" Meaning "good"But in the wordbag model, it'll be understood as "bad" Meaning. Distinction of the phrase structure entails inevitable loss of meaning.

n Phrase Bag

$n$ The wordbag (bag of n-grams) is a natural extension of the wordbag model, and we would like to retain the meaning of the sequence structure in some texts.

After some verbology,$n$ The meta-bag converts the entire document into a number of vectors, each of which represents a group of words that can be repeated.

$n$ The meta-bag model. $n$ indicates that the maximum number of words allowed contains the gram.$n$ The larger the content, the more informative the language is, but also the higher vector dimension.

Filter problem for BOW

BOW is the core instrument of the research embedded, but it also has natural flaws. The question of how to separate meaningful information and noise from the text in an appropriate way is a question that we need to study and, after filtering, models will be more effective.

Disable Word

Discontinuing words such as the,a,on, which have no practical meaning, are largely devoid of real meaning and are generally used only in emotional analysis. For the purpose of understanding, the discontinuation could be completely removed.

The NLTK package for Python contains a list of inactive words constructed by linguists, all of which are lowercase and contain a set number, and uses this package to remove the disabled word.

HF words

A high-efficiency technique for handling high-frequency words: HF words appearing in a language library are likely to be discontinued. After the word has been deleted, it is likely to be a very common term in a particular language library and its relevance needs to be considered separately.

Rare words

Very rare words in a language library are also worth considering; they may be some kind of spelling error or a real rare word.

The presence of very rare words leads to excessive vector dimensions of the word bag model, which often cannot be used as a basis for prediction, but can lead to huge computing costs. Removing the rare words in the wordbag model is a very common NLP feature engineering technique.

Word dry extraction problem

In the NLP problem, the variations of a word are very common data. They are counted separately in word bags as different words, but they are very effective in combining their counts because they do have the same meaning.

The Python NLTK package provides an interface for word dry extraction, but it is not almighty. Sometimes a term that is close in form but has different meanings leads to a bad effect after a dry extraction.

TF-IDF

The word frequency count in the wordbag model does not automatically remove the disabled word. Although it can be removed at a later stage, the model itself is not as appropriate.

The core idea of TF-IDF is:The value of a word to a document is proportional both to the frequency it appears in the document and to the extent to which it is prevalent throughout the language library. Pass."Counter-document Frequency"To punish words that are too common.

---

TF-IDF is the product of two components:

Term Frequence, TF Expression $t$ In Document $d$ The frequency of occurrence is commonly defined as:

Original Frequency $$\text{TF}(t, d) = \text{count}(t, d)$$ Normalization frequency (most commonly used) $$\text{TF}(t, d) = \frac{\text{count}(t, d)}{\text{总词数 in } d}$$ logarithmic scaling frequency (for excessively long documents) $\text{TF}(t, d) = 1 + \\log(\text{count}(t, d))\quad (\text{if}\text{count} > 0)$$

Counter-document frequency (Inverse Document Frequency, IDF) Rareness of measured terms, definitions $$\text{IDF}(t, D) = \log\left( \frac{N}{\text{df}(t)} \right)$$ of which $N$ is the total number of documents,$dt(t)$ It's a word. $t$ , the number of documents is obtained according to custom.

In order to avoid a zero error, smoothing items are often added in practice: $$\text{IDF}(t, D) = \log\left( \frac{N + 1}{\text{df}(t) + 1} \right) + 1$$

The final TF-IDF value is: $$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \text{IDF}(t, D)$$ Every document $d$ It is a vector whose value is the TF-IDF weight of a word in the vocabulary of each dimension.

---

TF-IDF (inverted document frequency) asSimple and efficientThe NLP feature extraction technique is the most recommended option, more reliable and easier to use than BOW, if a simplest and comprehensible model is needed to study machine learning of text data.

Rare search technology peak: BM25

Okapi-TREC

Before discussing embedded models, it is important to understand the powerful Baseline that it needs to transcend. BM25 is not one."Embedded Model", instead a lexicon search (Lexical Retrieval) algorithm. BM25 is the core technology in modern information retrieval, and almost all modern search engines use BM25 algorithms or their improved form.

BM25 is not a cross-cutting development, it's...Classic probability model in the field of information retrieval(Probabilistic Model) Natural evolution by Okapi projectThe BM series is entitled Okapi-TrEC, where BM means Best Match.

BM25 and TF-IDF are classic text weights used to measure the importance of words in documents. Although they're similar in form."Word Frequency"and"Counter-document Frequency"Elements) but the actual design thinking is not the same.

TF-IDF HopeQuantify the documentBM25 estimates that one document is correctRating of the relevance of the given queryFor sorting. The former is used to create usable features for the document, while the latter is used to retrieve existing documents based on queries. Therefore, the application of the scenario was inconsistent.

BM1

Okapi in TREC-1 is original Robertson-Sparck Jones Weightto give a weight to a word under the existing document library:

$$w(1) = \log \frac{(r + 0.5)/(R - r + 0.5)}{(n - r + 0.5)/(N - n - R + r + 0.5)}$$ of which $N$ is the total number of documents,$n$ It's a word. $t$ .$R$ is the number of documents known to be clearly associated with the word,$r$ Is the words contained in these relevant documents $t$ Number. If there is no pre-reference, the two are degraded to 0, and the pattern is similar IDF Format: $$w(1) \approx \log \frac{N - n + 0.5}{n + 0.5} \propto \text{IDF}$$

BM1 is given a score algorithm based on the weight of words and documents that can be calculated earlier. For one. $m$ Word Query $Q={q_1,q_2,...q_m}$ And a document. $D$, BM1 score formula is: $\text{Score}{\text{BM1}}(D, Q) = \sum{i\capD}w(1) q i}$$ Peace onlyWords that appear in both Query and Document DConducted (i.e., intersecting).

---

BM1 is oneBinary model(binary model), only"Query words appearing in document"Peace. If word $q_i$ Not in Document $D$ , it $D$ ScoreNo contribution..and its contribution to $w(1)_{q_i}$BM1 doesn't consider word frequency TF.

General in physical search $R = r = 0$, because the document cannot be confirmed when the search occurs. At this point, the score calculation portion of the BM1 algorithm automatically degrades to the smooth form of the IDF.

BM11 and BM15

In order to address the limitations of BM1, two variants were proposed in TREC-2, addressing three issues: (1) not processing word frequency - one and 10 scores; (2) not considering the impact of document length on the probability of word occurrence - - Longer than possible."By chance."Include query words even if their content is not relevant; (3)Unable to distinguish the importance of query words— No matter how many times a query word appears, no weight is added.

BM15Just right.Word FrequencySaturation,Do not consider document length： $$\text{score} \propto \frac{\text{tf}}{k_1 + \text{tf}} \cdot w(1)$$ of which $tf$ It's a word frequency.$k_1$ As the control of word saturation as an ultra-parameter,$w(1)$ Calculates the weight of a word to the document according to the previous method. BM15 still uses the preceding request and method in calculating the total points - only"Query words appearing in document"Peace, but consider $tf$ The weight.

BM11On the basis of BM15,Include document length harmonization(based on"Redundancy assumptions"）： $$\text{score} \propto \frac{\text{tf}}{k_1 \left( (1 - b) + b \cdot \frac{\text{dl}}{\text{avdl}} \right) + \text{tf}} \cdot w(1)$$ of which $dl$ The length of the current document,$avdl$ The average length of all documents,$b$ is the super-parameter for the control length to the unified strength. BM11 still uses the preceding requirements and methods in calculating the total points - only"Query words appearing in document"Peace, but consider $tf$、$dl$ The weight.

The core idea of the redundancy scenario is that long documents contain more words because they contain a large number of words."Redundancy"or"Repeat"Content, not more.

BM25

In TREC-3, the author found BM11 punishment for document lengthToo heavy.（"Redundancy assumptions"May exaggerate the length effect, original BM11 is taken in part $b=1$) and then one.General form, unite BM11 with BM15:

BM25 Core Formula (for single query words) $q_i$ In Document $D$ : $$\text{score} (D, )=sum (n)\text{IDF}(q i)\cdot\frac{text{tf}{q_i,D} \cdot (k_1 + 1)}{ \text{tf}{i,D} \k 1\left1 - b \cdot\frac{dl}{\text{avgdl}}$$ Of which IDF is similar to the front $w(1)$ No form of feedback,$tf$ This means the frequency at which the word appears in the document,$dl$ The length of the current document,$avdl$ is the average length of all documents.

$k_1$ Control of word saturation as ultra-parameters: degradation of the model to binary values at close to 0, closer to infinite proximity $tf$ The linear growth is usually between 1 and 2.$b$ It's the hyper-parameter for the control length to the unified strength:$b=0$ It's not like you're going to be able to do it with the same length.$b=1$ Full integration (equivalent to BM11).

Supplementary note: qtf in BM25, although standard BM25 achieves constant omission of qtf, the original formula consists of: $$\text{score} (D, )=sum (n)\text{IDF}(q i)\cdot\frac{text{tf}{q_i,D} \cdot (k_1 + 1)}{ \text{tf}{q_i,D} + k_1 \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}} \right) } \cdot \underbrace{ \frac{\text{qtf}{q_i} \cdot (k_3 + 1)}{ \text{qtf}{q_i} + k_3 } }_{\text{query term frequency component}}$$

• If the user repeats a word "AI AI AI") increases its weight.
• Exactly.Distinguishing the importance of query wordsA mechanism.

However, in modern physical search, the query usually comes from the topic CONCEPTS field (without repetition), so qtf is often 1 and the item is ignored.

Summary

Design ideas for BM25:

• Word Saturation(TF Saturation): 10 or 20 times a word appears in a document and the difference of importance should be less than 1 change and 2 times. A logarithmic growth structure was therefore introduced to curb linear expansion of word frequency.
• Harmonization of document lengthLong files naturally contain more words and require punishment. Use $b$ Parameters are flexible in controlling the degree of integration and adaptation to different data sets.
• Keep Probability Model Foundation: IDF items still originate from Robertson-Sparck JonesProbability estimates for relevanceIt has a theoretical basis.

BM25BM11/BM15withQuery Extension(pseudo-relevance feedback) andParagraph Search(passage retrieval) The effect is further enhanced to become the core algorithm of the Okapi system in TREC-3. Still.One of the most widely used search sequence algorithms in industry and academia。

Embedding Technology Beginning: Word2Vec

Thought

Word2Vec and "Efficient Estimation of Word Representations in Vector Space" It's one of the core creations in the modern NLP field. It proved, for the first time, that there was a great deal of semantic relations between words that could be learned through a simple neural network and that it was truly universal."Word Embedding"This concept.

Word2Vec's core idea is:The meaning of a word depends on the words around it.I don't know. Word2Vec automatically learns the mathematical expression -- a dense vector -- of each word in the context of a large volume of text.

Word2Vec contains two main models:

1. CBOW: Conjecture the central word from the context
2. Skip-gram: Conjecture context information based on the central word

Word2Vec wants to learn more complex language meaning information from big data than the low dimensions (50-100-D) that were previously studied. I'm also concerned that word vectors can be captured.Multilevel Similarity(if "big : bigger :: small : smaller"And even...Semantic analogy(if "King – Man + Woman ≈ Queen") and learning efficiency of models on mega-data sets.

Model structure

Structure overview

Overall objectives: Input: a large language library (e.g. Google News, 6 billion words) Output: Each word corresponds to one D Weighted Real Vector(e. g. 300-D) allows a syntax analogy where a semantic/syllable similarity is close in vector space and supports vector operation.

At the pre-processing stage, the removal of particularly low-frequency words (which are still maintained on a million-scale vocabulary) is intended to avoid data contamination by spelling errors. Each remaining word is assigned a single integer ID, which is used in subsequent narratives $V$ is the number of high frequency words.

Word2Vec does not directly model language models, but willContext forecast tasksConvertClassification issues。

For CBOW:

• Objective: Centre Word $w_t$
• Context: Other words in the window, e.g. $w_{t-1},w_{t-2},w_{t+1},w_{t+2}$(window size) $C$=2）
• Training sample：(context → target)

For Skip-gram:

• Objective: a word in the context, if $w_{t+1}$
• Input: Centre Word $w_t$
• Training sample：(center → context_word)
• Skip-gram produces multiple samples for each central word (one for each context)

Shared component of model structure: term vector matrix (Embeding Matrix)

• Define One $V \times D$ Other Organiser $W$of which $i$ Words. $i$ vector means $v_i$。
• This matrix is the ultimate learning.Word embeddings。
• In CBOW and Skip-gram,$W$ Both.Enter layer weight, or as an output layer weight.

CBOW

Structure process:

1. Input Layer: context words $w_{t-1},w_{t-2},w_{t+1},w_{t+2}$ one-hot vector
2. Projection Layer：
   1. Each one-hot vector multiplied by $W$♪ Got a match ♪ $D$ dimension vector.
   2. These vectorsAverage(or peace) $h$, the structure of a wordbag discards the context order.
   3. Note: Matrix used here $W$ Finally, the words learned are embedded in themselves, because the input is actually one-hot encoded, and the product only needs to be used. $W$ One line in.
3. Output Layer：
   1. Will $h$ Enter to One softmax CatalogAnd predict the central word.
   2. Use the matrix $W^ before entering the Softmax classifier'}$ 将 $h$ 映射回 $V.D.
   3. Calculated projected losses for reverse dissemination.

Training objective: to maximize the logarithmic of the correct central words: $$\mathcal{L} = \log P(w_t \mid w_{t-C}, \dots, w_{t-1}, w_{t+1}, \dots, w_{t+C})$$

Skip-gram

Structure process:

1. Input Layer: Centre Word $w_t$ one-hot vector
2. Projection Layer: multiplied by $W$♪ Got a match ♪ $D$ Dimension vector $v_t$。
3. Output Layer: For each context position $w_{t+k}$ Use $v_t$ Go predict the word, use the matrix.'}$ 将 $v_t$ 映射回 $V.D. The losses projected in each context are combined for reverse dissemination as projected losses.

Training objective: Maximum logarithmic $$\mathcal{L} = \sum_{k=-C,, k \neq 0}^{C} \log P(w_{t+k} \mid w_t)$$

On computing efficiency

This is a neural network structure involving large-scale data. To ensure their calculability at the required scale, many adjustments to model structures and computational strategies are required.

In traditional neuronet language models (e.g. NNLM), the complexity of training is mainly derived from:

• Non-linear hidden layer
• Super-high Softmax Header

Even with the use of parallel training, it is difficult to extend to very large language material.

To that end, Word2Vec Moves unless a linear hidden layer becomes a log-linear ModelI don't know. CBOW and Skip-gram Remove the hidden layer completely., the projection layer (the average word vector or a single word vector) directly connects the output layer, i.e. transforms itself into a fully linear sorter, thus reducing the computational burden.

Word2Vec to deal with Softmax Use Hierarchical Softmax (horizontal Softmax)Give up the whole size of each sample. $V$ The list of words (millions of degrees). It organizes the glossary as Huffman, fork tree.Change the projection mission to a series of root-to-leaf pathsClass II (sigmoid), the ultimate probability is the product of all sigmoids on the path, thereby reducing linear complexity to logarithmic complexity.

The very simple design framework runs throughout: whether CBOW or Skip-gram, the model structure is simple and the layers are extremely low, all to ease the counting efficiency bottlenecks. While benefiting from team strengths, Word2Vec uses Google internal DistBelief Distribution Framework, use multi-machine small-volume walk-through training and central server parameters to accelerate computing efficiency.

Summary

This article found:

• Increased amount or dimension of data enhances performance, but existsMarginal benefits diminishingIn order to continuously upgrade model capabilities, both need to be upgraded. This finding also guides the current training of large language models.
• Skip-gram significantly outperformed other models in semantic tasks; CBOW was slightly superior in terms of terms, predicting other words from the central word to make models learn better semantic information.
• The dense vectors obtained by embedding these instruments allow semantic analogy, excavating information through simple calculations between vectors, but there are certain errors.
• Simple model (CBOW/Skip-gram) to efficiently train high quality word vectors, and could continue to expand almost indefinitely to any task requiring text embedding.

Word2VecIt's an era of pre-training words vectors., guided the development of many NLPs, their Open Source Toolkit word2vec It has become an essential tool for NLP missions. The great thing about Word2Vec is:With a very simple structure + big data, it reveals the deep structure of the language.I don't know. It proves."Simple Model + Big Data"It can go beyond that."Complex Model + Small Data"This idea profoundly affects the whole AI field.

Transformer

Attention, Bert and GPT

The Transformer structure presented in the 2017 paper “Attention Is All You Need” led to two different pre-training paradigms. They address the context in different ways and collectively redefine modern NLPs. For a detailed discussion of the self-directional mechanism Self-care mechanisms and Transformer。

---

BERT (Pre-training of Deep Bidirectional Transports for Language University) is the beginning of a deep two-way understanding of language models, where a real two-way context is embedded. BERT is a radical paradigm revolution that solves the dilemma of static embedding that cannot deal with the multidimensional meaning of the word.

Its core self-care mechanism allows the model to handle a word while obtaining context information across the board. The key innovation is the masked language model (MLM): random in the sentence"Query"It also allows the model to project according to the two-way context, forcing the model to integrate the context in depth and generate dynamic, context-sensitive word vectors. BERT's representative."Pre-training -- fine-tuning"The paradigm still dominates the entire NLP field.

---

GPT is another technical route with Bert. It uses "Decoder-only" The Transformer structure with the goal of generating a coherent text. In order to predict the next word accurately, the GPT model forms a deep internal understanding and efficient expression of the above (left context).

The success of GPT has also revealed the Scaling Laws — the larger the models, the larger the data and the larger the resources, the greater the capabilities of the models, including their representational capabilities. Its strong capacity for generation, in turn, has generated a huge demand for high-quality retrieval systems (i.e. RAG).

BERT Embedding

BERT type modelEmbeding through [CLS] Mark or average pool acquisition.[CLS] The tag itself does not contain any semantics, and when the sequence flows through multiple layers Transformer Encoder, the self-care mechanism at each level allows [CLS] Mark it."Attention"all other words in the sequence. Through this two-way exchange of information,[CLS] Marked vectors are trained to be able toAbsorption and summary of semantic information for the entire input sequence。

In the last hidden layer of the model output (lower layer is the restoration to high-dimensional execution Softmax), directly removedWith input [CLS] The final hidden state vector corresponding to the mark, a 768-dimensional vector is considered to be Embeding the entire input text. Or get the last output of the model by average pool.All Charactersto hide the state vector, to average the Embedding vector for all elements.

GPT Embedding

Use with BERT in generating embedded [CLS] The most natural and efficient way to create a GPT series model is to embed it.The hidden status of the last mark in the input seriesI don't know. The logic behind it is closely linked to its goal of self-return training. In embedded tasks of GPT models, except for the beginning <|begin_of_text|>It usually adds at the end. EOS That's... <|end_of_text|> Mark.

The hidden status of each tag is updated when the sequence passes the Transformer layer of GPT. But the last word is the only one that can."Yeah."The whole input sequence (from <s> Present. token_n. The last character is the end point of the entire information stream and is the final carrier of all context information.

The embedded vector of the sentence is from the model.Last Hidden LayerIn the hidden layer before returning to the high-dimensional Softmax, extractLast Input TagWhether it's the original last word, or... EOS Marks the corresponding hidden state vector. For GPT-2, it's also 768D.

Embedded models based on GPT structures allow commands to control embedded generation and adapt it to different downstream tasks, that is,Command fine-tuningI don't know. It's based on the Bert structure that can't do it. Allowing instructions to fine-tune means allowingA model is used for a variety of different types of downstream tasks.It is no longer necessary to train a dedicated model for each mission. Finally.The GPT scale method to improve performance。

Qwen3 Embedding

This section summarizes the technical report that was uploaded by the Qwen team in June 2025 Qwen3 Embeding: Advancing Text Embeding and Reraning Through Foundation Models. The Qwen3 Embeding model (8B, 4B, 0.6B) is ranked 2-4 on the MTEB Integrated List, after the unknown parameter Gemini Embeding 001. The whole series of models has followed the Apace 2.0 license open source.

Introduction

Text embedding and reordering is an important part of the NLP task, and high-quality semantic embedding is the basis for many important tasks, such as RAG and Agens. Despite notable progress, embedded and re-sorting models where training is well performed in terms of scalability, context understanding and alignment with specific downstream tasks remain challenging.

This paper introduces the Qwen3 Embedding series of models based on the Qwen3 Foundation model, making full use of their powerful multilingual text understanding and generation capacity to unleash their potential in training embedded models and reordering models.

In order to train in embedded models, a multi-stage training process was implemented: large-scale unsupervised pre-training, and monitored fine-tuning of high-quality data sets. Model Merge was used to integrate different checkpoints in order to enhance modelity and generalization. The Qwen3 directive model was used to synthesize training data, with a high-quality component for the second phase of supervisory training.

A similar two-stage training programme was used for the reordering model. Three embedded and reordered models (8B, 4B, 0.6B) were eventually released based on the Qwen3 model of different scales. To facilitate applications in downstream missions, the Qwen3 Embeding series model supports elastic dimension expression and customisation instructions.

Model structure

The core idea of embedded and reordered models is to assess relevance in a mission perception manner. For queries $q$ and documents $d$, the model needs to follow instructions $I$ To assess the correlation between the two. To this end, the model uses the following forms of data for training: $${ I_i, q_i, d_i^+, d_{i,1}^-, \dots, d_{i,n}^- }$$ Separately indicates the command, the query, the relevant unrelated document. Training to cover various types of similar data pairs will expand their performance in downstream tasks in different fields.

Structure: Qwen3 Embedded and Reordered Models are based on a dense version of the Qwen3 Foundation Model, using Dense version and containing three parameter sizes, and initializing with pre-trained parameters to use the capabilities they have acquired in large-scale pre-training.

Embedded Model: Add one at the end of the input series using a large language model with causal attention mechanisms EOS Mark. The final embedded vector corresponds to the last layer. EOS Marks the hidden status generation.

To ensure that embedded vectors follow instructions in downstream missions, willCommand & Query Spellas a single input context, andDocument part remains unchanged until sent to LLM processingI don't know. The input sequence for command query is: {Instruction} {Query} <|endoftext|>, where <|endoftext|> Qwen series is always used EOS Mark. The document is {Doc} <|endoftext|>。

Reorder Model: For a more accurate assessment of text similarities, the large-language model (LLM) is used in a single context point-wise is reordered. Similar to embedded models, enter the instructions in the context to enable compliance. Use standard dialogue templates and model similar assessment missions as a question of classification. LLM input follows the following template:

<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: {Instruction}
<Query>: {Query}
<Document>: {Document}<|im_end|>
<|im_start|>assistant
<think>\n\n</think>\n\n

The next word is assessed to be "yes" or "no" Possible. The mathematical expression is as follows: $$\text{score}(q, d) = \frac{e^{P(\text{yes} \mid I, q, d)}}{e^{P(\text{yes} \mid I, q, d)} + e^{P(\text{no} \mid I, q, d)}}$$ Reordering model does not use the embedded vector of the last layer, but allows the model to run the next word projection of the last layer, generating a dimension equal to that of the vocabulary logits vector, then give this vector to the Softmax layer for a probability distribution. The probability of both words is what we need.

The Qwen3 Text Embedding model supports MRL (Matryoshka RepresentationLearning) and allows adjustments to the dimensions of the embedded vector. The Text Relanging model does not address this function because it does not involve an output embedded vector. They all support the implementation of an integrated Aware that is embedded or reordered on the basis of command dynamics. Models of all sizes support 32K context, with slightly different layers.

Training and evaluation

This section describes the multi-stage training process used and describes the key elements of the training programme, including training objectives, training data synthesis and the screening of high-quality training data.

Training objectives

For embedded models, an improved version based on the InfoNCE comparative learning loss framework is used. For a size of $N$ , whose loss function is: $$L_{\text{embedding}} = -\frac{1}{N} \sum_{i}^{N} \log \frac{e^{s(q_i, d_i^+) / \tau}}{Z_i}$$ of which $s()$ is a function used to calculate similarities, using cosine similarities;$\tau$ is the super-parameter for temperature control;$Z_i$ is a normalized factor that combines the similarity scores of positive and negative samples: $$Z_i = e^{s(q_i, d_i^+) / \tau} + \sum_{k}^{K} m_{ik} e^{s(q_i, d_{i,k}^-) / \tau} + \sum_{j \neq i} m_{ij} e^{s(q_i, q_j) / \tau} + \sum_{j \neq i} m_{ij} e^{s(d_i^+, d_j) / \tau} + \sum_{j \neq i} m_{ij} e^{s(q_i, d_j) / \tau}$$

The parameters are as follows, in order from left to right:

• $d_i^{+}$ indicates that the correct document is paired and the positive document corresponding to this query is measured;
• $K$ and $d_{i,k}^-$ represents the number of negative documents and the number of negative documents to which this query corresponds;
• $q_j$ represent other queries in the bat, measure this query and other queries in the bat;
• $d_j$ represent other documents in the bat, measure positive documents and other documents;
• The last item does not involve new parameters, but only measures the relationship between this query and other documents.

Mask Factor $m_{ij}$ With a view to mitigating the effects of false negatives, the definition is as follows: $m= \begin{cases} CC BY-NC-ND 2.0 & \text{if } s_{ij} > s(q_i, d_i^+) + 0.1 \text{ or } d_j == d_i^+, \ 1 & \text{otherwise} I'm sorry. of which $s_{ij}$ is the correlation factor between the query and the document.

For reordering models, optimized oversight fine-tuning (SFT) loss functions are defined as follows: $$L_{\text{reranking}} = - \log p(l \mid \mathcal{P}(q, d)),$$ of which $p(\cdot \mid *)$ This indicates the probability of being allocated by the Large Language Model (LLM). Label $l$ For regular documents as "yes", for the negative document "no"I don't know. The loss function encourages the model to assign a higher probability to the correct label, thereby increasing the ranking performance.

Multi-stage training

Multi-stage training is a common practice for training text embedding models. This strategy usually begins with initial training on large-scale semi-oversight data containing noise, followed by fine-tuning using smaller but high-quality monitoring data sets.

The large-scale weak monitoring training data contribute significantly to the model ' s ability to extend, while the fine-tuning of subsequent phases using high-quality data further enhances model performance. A combination of two-step processes will result in embedded models that have a higher capacity for generalization and performance.Note that the training process of the reordering model does not include weak monitoring training.

Building on the existing multi-stage training framework, the Qwen3 Embeding series introduced the following innovations:

• Weak surveillance training driven by large-scale synthetic data: Directly synthesized into data pairs using the underlying model's strong text understanding and generation capability. This method allows flexibility in defining the multiple dimensions of the data required in a synthetic reminder, such as task type, language, length and difficulty. Data synthesis driven by basic models is more manageable than data collection from open-area sources.
• Application of quality synthetic data in monitoring fine-tuning: The excellent performance of the Qwen3 base model has resulted in significantly higher quality of synthetic data, further enhancing the overall performance and generalization of the model.
• Model integration: Following completion of the supervisory fine-tuning, model integration techniques based on meta-line interpolation (SLERP) are used. The technology combines multiple models kept during fine-tuning with checkpoints, which are designed to enhance the robustness and generalization of models in different data distributions.

Synthetic Data Set

In order to build a robust synthetic data set for training models to perform various similar tasks, we have produced diversified text pairs covering search, bilingual excavation, classification and semantic text similarities (STS). The quality of these synthetic data is ensured by data synthesis using the Qwen3-32B model as the base model.

We have designed a variety of warning strategies to enhance the richness and authenticity of the data generated: assigning specific roles, and simulated scenarios of potential users searching the document. This way of infusing a user perspective increases the diversity and relevance of synthetic queries. The hint template also incorporates dimensions such as type of query (keyword type, fact type, summary type, judgement type), length of query, difficulty and language. This multi-dimensional design ensures the quality and diversity of synthetic data.

Ultimately, approximately 150 million groups of weak monitoring training data were created. Experimental results show that embedded models trained in the use of these synthetic data perform very well in downstream assessments, significantly exceeding many previous monitoring models.

For the second stage of training, a simple cosine similarity calculation method is used to screen data pairs: preserve a cosine similarity greater than 0.7 from random sampling data. Some 12 million high-quality monitoring training data were eventually selected for use in follow-up training.

Evaluation

We made a comprehensive and fair assessment of the Qwen3 Embeding model on multiple benchmark tests. For text embedded models, useLarge-scale Multilingual Text Embedding Benchmark Test (MMTEB), covering over 500 quality control assessment tasks in more than 250 languages.

In addition to the traditional text tasks (e.g., similarity of various types of search, classification and semantic text), MMTEB also contains a series of challenging and novel tasks, such as command compliance, long-document search and code retrieval, which are currently the largest and most widely covered assessment tasks in the model field.

of which Qwen3-Reranker-8B best performed in most missions, the overall performance is only slightly behind the latest version of Gemini-Embeding.

Conclusions

The digestion experiment showed thatLarge-scale and weak pre-training stages of oversight are critical to achieving excellence，Model integration is also a key element in building strong models.。

This technical report officially publishes the Qwen3-Embeding series, a comprehensive text embedded and reordered model based on the Qwen3 base model. These models are designed to perform well in the embedded and reordered tasks of various text types and cover multiple scenarios such as multilingual retrieval, code retrieval and complex command compliance.

The Qwen3-Embeding model is based on a robust multi-stage training process that combines large-scale weak monitoring pre-training on synthetic data with monitored fine-tuning models on high-quality data sets. In doing so, the Qwen3 Large Language Model has played a key role in synthesizing diversified training data across multiple languages and missions, thus effectively enhancing the capabilities of the Model.

A comprehensive assessment showed that the Qwen3-Embeding model achieved SOTA performance in MTEB, CMTEB, MMTEB and several other search baseline tests. The model follows the Apache 2.0 license open source in GitHub.

Appendix

The appendix contains additional relevant information on the Attention Mechanism, Transformer, BERT, GPT, citing notes from other parts as appendices.

Appendix A: Focusing Mechanisms

See below for details of the self-care mechanism. Self-care mechanisms。

Appendix B: Transformer Architecture

For more details about the Transformer structure, see Transformer Structure。

Appendix C: Self-supervised Learning

For details on self-monitoring learning see Self-supervised learning。