Efficiency revolution in the era of small models

In May 2024, Mistral, a little-known startup, released the Mixtral 8x7B. This model has only 46.7B parameters, but its performance is comparable to GPT-3.5, and even exceeds GPT-4 in some tasks. More importantly, its reasoning cost is only 1/10 of GPT-4. This is not an exception-in the past year, small models (usually models with parameters between 1B and 30B) have been rising at an alarming rate, and they have only one trump card: efficiency .

Efficiency, a word that was once concealed by the aura of big models, is becoming a new protagonist in the AI field. When we talk about small models, what we are really talking about is an efficiency revolution-how to do more with less resources, how to get AI from the laboratory to the real world, how to make technology truly serve people, not the other way around.

efficiency trio: computing power, data, architecture

The efficiency improvement of small models is not a single breakthrough, but the result of a trio: optimization of computing power, efficient utilization of data, and architectural innovation.

1. Computing power: from "burning money" to "calculating carefully"

The training cost of large models is prohibitively high. The training cost of GPT-3 is approximately $4.6 million, while GPT-4 is estimated to be as high as $100 million. In contrast, the training cost of small models is usually between tens of thousands and hundreds of thousands of dollars. For example, Meta's Llama 38B model costs about $500,000 to train, but its performance reaches GPT-3.5 levels.

More critical is the cost of reasoning. According to SemiAnalysis, the single inference cost of GPT-4 is about $0.03, while the inference cost of Mixstral 8x7B is only $0.003-only one-tenth of the former. This means that the small model can handle 10 times the number of requests for the same budget.

Behind the optimization of computing power is a series of technological innovations:

• Quantization Technology : Compresses model weights from 32-bit floating point numbers to 8-bit or even 4-bit integers, significantly reducing memory and computing requirements. For example, after 4-bit quantization, Llama 38B triples the inference speed and reduces memory usage by 75%.
• sparsity : Utilize the sparsity of model parameters to activate only some neurons and reduce the amount of calculation. Mixtral 8x7B is implemented through the sparse mixed expert (MoE) architecture.
• Hardware optimization : Optimize for specific hardware (such as NVIDIA's TensorRT, AMD's ROCm) to improve computing efficiency. For example, NVIDIA's H100 GPU can achieve throughput of 1000 tokens/second when running Llama 38B, while GPT-4 can achieve throughput of only 100 tokens/second on the same hardware.

2. Data: From "flood irrigation" to "precise drip irrigation"

Training large models usually requires trillions of tokens of data, while small models often only require tens of billions to hundreds of billions of tokens. The logic behind this is: Data quality is more important than quantity .

Research from Stanford University has shown that on the same task, small models trained using high-quality 100B token data can outperform large models trained using 1T token data. For example, Phi-3-mini (3.8B parameters) outperformed many 10B level models after training using carefully filtered 3.3T token data.

Key technologies for efficient data utilization include:

• Data Screening : Use heuristic rules or small models to filter low-quality data. For example, when Meta trained Llama 3, it used a special data filtering model to filter out 80% of the original data.
• Data Synthesis : Generating high-quality synthesized data through models. For example, Microsoft uses a lot of synthetic textbook-level data when training the Phi series of models.
• Continuous learning : Models continue to learn from high-quality data after deployment to avoid catastrophic forgetting. For example, Google's Gemma model supports online fine-tuning, which can be continuously optimized after deployment.

3. Architecture: From "unification" to "modularity"

Large models usually use a single Transformer architecture, while small models tend to be more modular. This design not only improves efficiency, but also enhances flexibility.

Typical modular architectures include:

• Mixed Experts (MoE): Divides the model into multiple expert subnetworks, with each input activating only some experts. For example, the Mixtral 8x7B consists of 8 7B experts, and each input activates only 2 experts, allowing efficient reasoning.
• Recursive Transformer: Reduce the number of parameters through recursive structure. For example, Recurrent Gemma maintains performance comparable to Gemma through a cyclic structure while reducing the number of parameters by 50%.
• Multi-modal fusion : Separates processing modules of different modes to improve efficiency. For example, Llava-Next enables efficient multimodal reasoning by separating visual and language modules.

Steelman: Questions and responses from the opposition

Of course, the rise of small models is not without controversy. Opponents usually have the following views:

1. "Small models have insufficient performance to handle complex tasks"

Counter-view: Small models perform poorly in tasks such as complex reasoning and long text understanding and cannot replace large models.

Response: This view ignores two key points:

• Task Decomposition : Complex tasks can be decomposed into multiple simple tasks, which are processed separately by small models. For example, Google research shows that by decomposing a long text understanding task into multiple short text understanding tasks, the performance of a small model can be close to that of a large model.
• Context Window : By optimizing the context window and attention mechanism, small models can handle longer text. For example, Llama 3's context window is 8K tokens long, enough to handle most long-text tasks.

2. "The small model is just a transition plan, and in the end, we still need to rely on the large model"

Opposing view: Small models are just temporary measures. As computing power increases, large models will eventually dominate.

Response: This view ignores the long-term value of efficiency:

• Declining marginal effect : As the model size increases, the marginal effect of performance improvement decreases. For example, from GPT-2 to GPT-3, the number of parameters has increased by 100 times, but the performance improvement is much less than 100 times.
• Diversified application scenarios : Different scenarios have different requirements for models. For example, on mobile devices, small models may always be better suited than large models.

3. "Optimization techniques for small models are also applicable to large models"

Counter-view: The optimization techniques used by the small model (such as quantification and sparseness) can also be used for the large model, so the small model has no essential advantage.

Response: This view ignores the scale effect:

• Different optimization space : Small models have larger optimization space. For example, quantization techniques can compress parameters to 4 bits on small models, while typically only 8 bits on large models.
• Hardware adaptation : Small models make it easier to adapt to different hardware. For example, small models can be easily deployed on mobile devices or edge devices, while large models require high-performance server clusters.

QKPFX17 Behind the QK efficiency revolution: from "large-scale steel refining" to "lean production"

The rise of small models reminds me of a key transformation in the Industrial Revolution: from mass production to lean production . In the early 1900s, Ford's assembly-line production model reduced costs through economies of scale, but it also brought rigidity and waste. Toyota's lean production model achieves higher flexibility and lower costs by reducing waste and improving efficiency.

How similar are the development paths in the AI field:

• The big model era is similar to "big steel refining"-it improves performance by piling up resources (data, computing power, parameters), but ignores efficiency and flexibility. The - small model era is more like "lean production"-by optimizing processes, reducing waste, and improving efficiency, achieving higher cost performance and wider application scenarios.

The logic behind this shift is: Efficiency is not just about reducing costs, but also about releasing possibilities .

In the era of large models, AI application scenarios are limited by high costs and complex deployments. For example, autonomous driving company Wayve has said that using large models to make real-time decisions costs as much as $1 per mile, making commercialization almost impossible. The emergence of small models makes this application feasible. For example, Tesla's FSD v12 system uses a small model with 1.5B parameters to enable real-time autonomous driving decisions at a cost down to $0.10 per mile.

Zhang in ## Data Center

I know an engineer who works in a data center. We call him Lao Zhang. Lao Zhang is responsible for maintaining a batch of GPU servers that were once used to train large models. As the craze for big models recedes, these servers gradually become idle.

Last year, the company decided to try deploying small models. Lao Zhang was very skeptical at first: "What can these small models do? The performance is definitely not good. "But when he saw the deployment results, he completely changed his mind.

The first is energy consumption. In the past, when running the large model, the power consumption of a server was as high as 1000W, but now running the small model, the power consumption has dropped to 300W. This means a direct reduction in electricity bills by 70%. What surprised Lao Zhang even more was that the reasoning speed of these small models was much faster than the large models. Customers used to complain about high latency, but now the latency has dropped to less than 100 ms.

What impressed Lao Zhang most was the flexibility of deployment. In the past, deploying large models required an entire cabinet, but now it can be done with a single cabinet. Customers can rent on demand and even run it on edge devices. Lao Zhang said: "We used to sell cannons, but now we sell rifles. Although the power of a single unit is small, it is flexible and cheap, and can be used by everyone. "

Lao Zhang's story reminds me that the efficiency revolution is not just a change at the technical level, but also a change in business models and user experience. When technology moves from "high above" to "within reach," its influence really begins to be released.

QKPFX21 The Cost of QK Efficiencism: Are we overly pursuing efficiency?

The efficiency improvements brought by small models are obvious, but we also need to be vigilant about the traps of efficiency. Efficiency gains are often accompanied by two potential costs:

1. Compression of innovation space

When we focus too much on efficiency, we may overlook innovation directions that will not see a return in the short term. For example, the exploration of emergence capabilities by large models may be ignored by the rise of small models. Emergence capabilities-new capabilities that models suddenly gain as they grow in size-are one of the most exciting discoveries in the AI field. If we concentrate resources on small models too early, we may miss these breakthroughs.

2. Loss of diversity

Optimization of small models often relies on specific hardware or architecture, which can lead to a single technology path. For example, most current small models are based on the Transformer architecture, while other architectures (such as RNN, state space models) may be ignored due to lack of resources. The loss of technological diversity may limit the long-term development of AI.

3. Short-sighted business model

Efficiency improvements often lead to changes in business models, but such changes may be too short-sighted. For example, the low cost of small models may lead to price wars for AI services, which in turn compresses R & D investment. In the long run, this could undermine the innovation capabilities of the entire industry.

Conclusion: Efficiency is not the end, but the starting point

The rise of small models marks the transition from "scale competition" to "efficiency competition" in the AI field. This efficiency revolution is not just a change at the technical level, but also a change in business models, user experience and social impact. It allows AI to move from the laboratory to the real world, from a toy for the few to a tool for the masses.

But efficiency is not the end, but the starting point. We need to improve efficiency while maintaining the pursuit of innovation, respect for diversity, and focus on long-term values. Only in this way can AI truly serve mankind rather than become another tool kidnapped by efficiency.

The ultimate question is not "How efficient can we be?" but "What do we want to do with efficiency?"