ProfBench is a dataset released by NVIDIA in 2025 that covers the evaluation of professional tasks across multiple domains. The associated paper is "ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge", aiming to provide researchers and developers with a standard benchmark for evaluating the performance of large language models in professional fields.

The dataset contains 40 records, covering 40 human-annotated tasks involving professional domains such as PhD in Chemistry, PhD in Physics, Financial MBA, and Management Consulting.

Dataset Composition

• ID: The identifier for the sample.
• Domain: Domain classification, including PhD in Chemistry, PhD in Physics, Financial MBA, and Consulting MBA.
• Prompt: The instruction for the large language model.
• Rubrics: Unique criteria used to evaluate the model's final output, ranging from 15 to 59 items.
• Model Responses: Responses from three models: OpenAI o3, xAI Grok4, and DeepSeek R1-0528.

Citation

@misc{wang2025profbenchmultidomainrubricsrequiring,
      title={ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge}, 
      author={Zhilin Wang and Jaehun Jung and Ximing Lu and Shizhe Diao and Ellie Evans and Jiaqi Zeng and Pavlo Molchanov and Yejin Choi and Jan Kautz and Yi Dong},
      year={2025},
      eprint={2510.18941},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.18941}, 
}