One-sentence Summary

Microsoft Research introduces Orca, a $13\text{-billion}$13-billion parameter model that learns to imitate GPT-4's reasoning process via explanation traces, step-by-step thought processes, and complex instructions, guided by ChatGPT, using large-scale diverse data with judicious sampling; it surpasses Vicuna-13B by over $100\%$100% on Big-Bench Hard and $42\%$42% on AGIEval, matches ChatGPT on BBH, and approaches GPT-4 on SAT, LSAT, GRE, and GMAT without chain-of-thought.

Key Contributions

• Introduces Orca, a 13-billion parameter model that learns to imitate the reasoning process of large foundation models by training on rich GPT-4 signals, including explanation traces, step-by-step thought processes, and complex instructions, with teacher assistance from ChatGPT.
• Demonstrates a scalable data strategy combining large-scale, diverse imitation data with judicious sampling and selection, enabling progressive learning beyond shallow output imitation.
• Results show Orca surpasses Vicuna-13B by over 100% on Big-Bench Hard (BBH) and by 42% on AGIEval, reaches parity with ChatGPT on BBH, and performs competitively on professional and academic exams (SAT, LSAT, GRE, GMAT) in zero-shot settings without chain-of-thought, while still trailing GPT-4.

Introduction

Large Foundation Models (LFMs) like ChatGPT and GPT-4 show strong zero-shot performance across many tasks, including professional exams. This success comes from scaling model and data sizes plus alignment training to match user intent. A key question is whether models can supervise their own or other models' behavior. Recent work uses LFMs as teachers to generate datasets for training smaller models like Alpaca, WizardLM, and Vicuna. However, these smaller models often match the teacher's style but lag in reasoning and comprehension. For example, Vicuna retains 92% of ChatGPT's quality per GPT-4 evaluation, but only 64% on reasoning benchmarks and 48% on complex benchmarks like BigBench-hard.

Prior instruction-tuning methods face several challenges. They rely on simple, low-diversity instructions from self-instruct processes, and human-contributed conversations from sources like ShareGPT favor creative tasks over reasoning-heavy ones, limiting task diversity and scale. Imitation learning from query-response pairs provides limited signals for tracing the teacher's reasoning, unlike richer signals like logits or rationales. Evaluation protocols are also flawed, using small test sets and GPT-4 as a judge, which introduces biases like favoring longer outputs and order effects, overestimating smaller models' abilities.

The authors address these gaps with three key contributions. First, they introduce explanation tuning, augmenting query-response pairs with detailed GPT-4 explanations that reveal the teacher's reasoning process, using system instructions like "think step-by-step" to elicit these. Second, they scale tasks and instructions using the Flan 2022 Collection, selectively sampling a diverse mixture and generating complex prompts, collecting 5 million ChatGPT responses and 1 million GPT-4 responses for progressive learning. Third, they evaluate the resulting model, Orca, across generative, reasoning, and comprehension abilities using GPT-4 auto-evaluation, academic benchmarks like Big-Bench Hard and TruthfulQA, professional exams from AGIEval, and safety evaluations with ToxiGen.

Dataset

The authors construct a training dataset where each instance is a triple: a system message, a user query, and a response from a large foundation model (LFM). The system message, placed at the start of the prompt, provides context, guidelines, and response formatting instructions. It is used to vary response length, define the assistant's character, set acceptable behavior, and determine response structure. The user query defines the task.

Dataset composition and sources

• The authors sample user queries from the FLAN-v2 collection, which consists of five sub-collections: CoT, NiV2, T0, Flan 2021, and Dialogue.
• They use only zero-shot queries for training and exclude the Dialogue sub-collection because its queries often lack sufficient context.
• They collect 5 million user queries from FLAN-v2 and obtain ChatGPT responses for all of them. From this 5 million set, they sample 1 million queries for which they collect GPT-4 responses.

Key details for each subset

• Zero-Shot CoT: Contains 18 tasks focused on math word problems, natural language inference, common-sense reasoning, science question answering, and odd-one-out reasoning. It has approximately 150 thousand queries in total, and all are included in the 5 million set.
• NiV2: Contains 1,560 tasks and roughly 5 million queries. The authors randomly sample 300 queries from each task, and take the entire task data for tasks with fewer queries. This yields 440 thousand queries.
• Flan 2021: Contains 142 tasks created from 62 datasets. The full collection is large, so the authors generate at most 1 million queries per task, resulting in roughly 28.9 million queries. They sample a total of 2.5 million queries from this collection of 85.7 million queries.
• T0: Contains 193 tasks associated with 35 training datasets. The authors only include tasks from the training split, excluding Big-Bench, since Big-Bench-Hard is used in evaluation. The T0 collection has roughly 85.7 million queries, and they sample 2 million queries from it.

System messages and metadata

• The authors hand-craft 16 system messages designed to elicit different response types, such as long or short answers, following guidelines, generating creative content, and producing explanations and step-by-step reasoning.
• Different system messages are used for different sub-collections of FLAN-v2. Two system messages are sampled only for multiple-choice questions, so they appear less frequently.

How the data is used

• The authors generate 5 million instructions (queries augmented with system messages), referred to as FLAN-5M, using the sampling techniques described above.
• They randomly sample 1 million queries from FLAN-5M to create another split, referred to as FLAN-1M.
• They use the Azure OpenAI API to collect ChatGPT responses for FLAN-5M and GPT-4 responses for FLAN-1M.
• Training proceeds in two stages: first on FLAN-5M with ChatGPT augmentations, then on FLAN-1M with GPT-4 augmentations. This progressive approach uses ChatGPT as an intermediate teacher, partly because Orca (13B parameters) is much smaller than GPT-4, and partly because ChatGPT is faster and cheaper than GPT-4. The authors collect 5 times as much data from ChatGPT as from GPT-4.
• GPT-4 responses are on average 1.5 times longer than ChatGPT responses, allowing the model to progressively learn from increasingly complex teacher explanations.

Method

The authors introduce Explanation Tuning to address the shortcomings of existing works by leveraging large-scale training data augmented with complex instructions and rich signals. Each training instance is structured as a triple comprising a system message, a user query, and a large foundation model response. The system message is placed at the beginning of the prompt to provide essential context, outline the assistant character, establish behavioral boundaries, and dictate the response structure. The user queries are sampled from the FLAN-v2 collection. Specifically, the authors sample 5 million user queries to collect ChatGPT responses, forming the FLAN-5M dataset. From this set, they further sample 1 million instructions to collect GPT-4 responses, forming the FLAN-1M dataset.

To optimize the training process, the authors employ a two-stage progressive learning strategy, leveraging ChatGPT as an intermediate teaching assistant before introducing GPT-4. This approach is motivated by the capacity gap between the 13 billion parameter Orca model and the much larger GPT-4, as well as the computational cost and time constraints of large-scale data collection. By first learning from the shorter and simpler explanations generated by ChatGPT, the model undergoes a form of curriculum learning before tackling the more complex reasoning of GPT-4. As shown in the figure below, GPT-4 responses are on average 1.5 times longer than those of ChatGPT across various system messages, which allows the model to progressively learn from increasing levels of teacher explanation complexity.

The training pipeline involves specific optimizations in tokenization, sequence packing, and loss computation. For tokenization, the authors utilize the LLaMA Byte Pair Encoding tokenizer, modifying it by adding a padding token to handle variable-length sequences, resulting in a vocabulary of 32,001 tokens. To maximize computational efficiency, they employ a packing technique where multiple input examples are shuffled and concatenated into single sequences. These sequences are partitioned such that their total length does not exceed a maximum of 2,048 tokens, with padding applied to ensure uniform length. This yields an average packing factor of 2.7 examples per sequence.

During the optimization phase, the loss is computed exclusively on the tokens generated by the teacher model. This ensures the student model focuses solely on learning to generate responses conditioned on the provided system messages and task instructions, thereby improving training efficiency. The entire training process is executed on 20 NVIDIA A100 GPUs with 80GB of memory, requiring 160 hours for four epochs on the FLAN-5M dataset and an additional 40 hours for four epochs on the FLAN-1M dataset.

Experiment

The evaluation protocol assessed Orca across open-ended generation, complex reasoning, and safety, comparing it against ChatGPT, GPT-4, Vicuna, and Text-Davinci-003. In open-ended generation, Orca retained 95% of ChatGPT and 85% of GPT-4 quality, outperforming Vicuna by 10 points, while in reasoning benchmarks (AGIEval and Big-Bench Hard) it matched or slightly exceeded ChatGPT on aggregate but lagged GPT-4 significantly, with notable strengths in entailment, temporal/spatial reasoning, and causal judgment, yet weaknesses in world knowledge, logical/geometric reasoning, and table understanding. Safety evaluations showed Orca generating more truthful and less toxic content than Vicuna, though trailing ChatGPT and GPT-4 on truthfulness, and a hallucination case study highlighted that smaller models like Orca struggle with factual recall but remain capable reasoning engines.

The table compares popular instruction-tuned models by their tuning method, data size, and teacher model. Orca stands out by using complex instructions with explanations and a much larger dataset, while other models rely on simpler or more natural instructions with smaller datasets. Orca uses complex instructions and explanations, unlike other models that use simple or natural instructions. Orca's data size is substantially larger than the other listed models. Orca employs both ChatGPT and GPT-4 as teachers, whereas other models use a single teacher.

The table lists 16 hand-crafted system messages used to augment user instructions when querying large foundation models for explanation tuning. These messages are designed to elicit varied response styles, including detailed answers, step-by-step reasoning, and explanations, while preserving the model's ability to produce both short and long outputs. Some messages are tailored for specific sub-collections, such as multiple-choice questions, to balance the training data. System messages are crafted to evoke different response types, from concise answers to detailed, step-by-step explanations. Certain messages are applied only to multiple-choice questions, leading to their lower representation in the training data. The instructions aim to maintain flexibility in output length while encouraging explanatory and reasoning-focused responses.

The training data was constructed by sampling from multiple existing datasets, with stratified sampling applied to most sources and one dataset skipped entirely. The sampling process reduced large original datasets into smaller, curated subsets for a total of 5 million samples. Stratified sampling was used for most datasets, while the Dialog dataset was skipped, contributing zero samples. The largest original dataset, FLAN2021, was reduced from over 28.9 million samples to 2.5 million, and T0 from 85.7 million to 2 million. NIV2 was sampled to 440K samples, and the no-sampling baseline kept 150K samples.

The table compares the cost and rate limits for data collection from ChatGPT and GPT-4 via Azure OpenAI. ChatGPT is significantly cheaper and allows much higher request and token throughput, while GPT-4 is more expensive and heavily rate-limited. These constraints influenced the decision to collect five times more data from ChatGPT than from GPT-4. ChatGPT costs $0.002 per 1,000 tokens, while GPT-4 charges$0.002per1,000tokens,whileGPT−4charges0.03 per 1,000 prompt tokens and $0.06 per 1,000 completion tokens. ChatGPT allows 300 requests per minute and 120,000 tokens per minute, whereas GPT-4 allows only 18 requests per minute and 10,000 tokens per minute. The higher cost and stricter rate limits of GPT-4 led to collecting 5 times more data from ChatGPT than from GPT-4.

The evaluation benchmarks for Orca span open-ended generation tasks and complex reasoning tasks, with dataset sizes ranging from 80 to over 5,500 examples. The largest datasets are dedicated to reasoning and exam-style questions, while the open-ended sets are comparatively small. Open-ended generation datasets include Vicuna, Awesome, and WizardLM prompts, with example counts between 80 and 218. AGIEval and Big-Bench Hard provide over 3,500 and 5,500 examples respectively for academic and complex reasoning evaluation. The benchmark mix balances qualitative generation with quantitative reasoning tasks, favoring larger reasoning datasets.

The evaluation setup compares Orca against popular instruction-tuned models, highlighting its use of complex instructions with explanations, a substantially larger dataset, and dual teachers (ChatGPT and GPT-4) versus simpler setups in other models. Training data was built via stratified sampling from existing datasets, reducing sources like FLAN2021 and T0 into a curated 5 million sample set, while skipping the Dialog dataset entirely. Data collection costs and rate limits favored ChatGPT over GPT-4, leading to five times more data from the cheaper model. Benchmarks span open-ended generation (small sets like Vicuna and WizardLM) and complex reasoning (large sets like AGIEval and Big-Bench Hard), balancing qualitative output with quantitative reasoning evaluation.