Chinese researchers had just published a paper on a model they built that is sort of close to GPT-4 quality.

Judged as of 2023-09-02. The claim is that Chinese researchers had recently published (within roughly the preceding weeks to few months) a paper on their own model 'sort of close to' GPT-4 quality. True: such a paper was published in roughly mid-2023 (for example on Qwen, Baichuan, InternLM or GLM) and reported benchmark results approaching GPT-4 on major evaluations. Mostly true: the paper exists and is recent, but its claims of being close to GPT-4 were somewhat overstated, or it matched GPT-4 only on some benchmarks, in line with the hedged 'sort of close'. Mixed: a recent paper exists, but the model was clearly closer to GPT-3.5 level. Mostly false: no recent paper, or the model was clearly well below GPT-4, with only isolated benchmark claims. False: no such Chinese model paper existed around that time.

Set before the research began.

1. E1 Baichuan2/README_EN.md at main · baichuan-inc/Baichuan2 GitHub (baichuan-inc) · 31 August 2023

   “Baichuan 2 achieved the best performance of its size on multiple authoritative Chinese, English, and multi-language general and domain-specific benchmarks.”

2. E2 Baichuan2/README_EN.md at main · baichuan-inc/Baichuan2 GitHub (baichuan-inc) · 31 August 2023

   “| | C-Eval | MMLU | CMMLU | Gaokao | AGIEval | BBH | | --------------------- | ---------- | -------- | --------- | ---------- | ----------- | ------- | | | 5-shot | 5-shot | 5-shot | 5-shot | 5-shot | 3-shot | | GPT-4 | 68.40 | 83.93 | 70.33 | 66.15 | 63.27 | 75.12 |”

3. E3 Baichuan2/README_EN.md at main · baichuan-inc/Baichuan2 GitHub (baichuan-inc repository)

   “journal={arXiv preprint arXiv:2309.10305}, url={https://arxiv.org/abs/2309.10305}, year={2023}”

4. E4 Baichuan2/README_EN.md at main · baichuan-inc/Baichuan2 GitHub (baichuan-inc repository)

   “[2023.12.29] 🎉🎉🎉 We have released Baichuan2-13B-Chat v2 version. In this version:”

5. E5 Qwen-7B/README.md at main · cyente/Qwen-7B GitHub (fork of QwenLM/Qwen, the Alibaba Cloud Qwen project) · 14 August 2023

   “2023.8.3 We release both Qwen-7B and Qwen-7B-Chat on ModelScope and Hugging Face. We also provide a technical memo for more details about the model, including training details and model performance.”

6. E6 Qwen-7B/README.md at main · cyente/Qwen-7B GitHub (fork of QwenLM/Qwen, the Alibaba Cloud Qwen project) · 14 August 2023

   “| Model | Tool Selection (Acc.↑) | Tool Input (Rouge-L↑) | False Positive Error↓ | |:------------|:----------------------:|:---------------------:|:---------------------:| | GPT-4 | 95% | 0.90 | 15% | | GPT-3.5 | 85% | 0.88 | 75% | | Qwen-7B | 99% | 0.89 | 9.7% |”

7. E7 Qwen-7B/README.md at main · cyente/Qwen-7B GitHub (fork of QwenLM/Qwen, the Alibaba Cloud Qwen project) · 14 August 2023

   “In general, Qwen-7B outperforms the baseline models of a similar model size, and even outperforms larger models of around 13B parameters, on a series of benchmark datasets, e.g., MMLU, C-Eval, GSM8K, HumanEval, and WMT22, CMMLU, etc.”

The evidence shows a Qwen-7B technical memo released on 2023-08-03 (E5) and a tool-use comparison in which Qwen-7B scores at or above GPT-4 on most measures (E6). No evidence item gives general benchmark scores for Qwen or Baichuan against GPT-4 (E2, E8), so the evidence cannot show whether the model was near GPT-4, at GPT-3.5 level, or well below.