Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 4 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
77 65 88
All types combined: 82 (likely 77 to 86, high confidence), from 94 claims: 77 factual claims, 17 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Satya Nadella's claims
43 claims
“It's unlike anything we've seen in the past when you're taking the most knowledge work task, which is software development and seeing 50 plus percent improvement.”
Satya Nadella, Microsoft CEO 15 November 2023 CNBC Transcript: Nadella on The Exchange (CNBC)
Software development with GitHub Copilot is seeing a 50-plus percent improvement.
Mostly true True in substance, wrong in a detail.
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2023-11-15. True: published studies available by that date (for example, GitHub/Microsoft research showing developers completed tasks about 55% faster with Copilot) show an improvement of 50% or more in a developer productivity measure from GitHub Copilot, and the claim's generality is reasonably supported. Mostly true: a 50%+ improvement is documented, but only in a controlled study or a specific task (such as task completion speed), which the claim presents somewhat more broadly. Mixed: studies show improvements, some of them 50%+ on certain measures but much lower on others, so that the evidence is roughly split. Mostly false: the 50%+ figure comes from a single narrow experiment (for example, one coding task) and is presented as a general improvement in software development, while broader evidence shows much smaller gains. False: no study shows anything close to a 50% improvement, or the evidence shows no improvement.
Evidence
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot arXiv Own record
“The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group.”
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot arXiv Own record
“Productivity benefits may vary across specific tasks and programming languages, so more research is needed to understand how our results generalizes to other tasks.”
- Devs gaining little (if anything) from AI coding assistants CIO · 26 September 2024
“It found no significant improvements for developers using Copilot.”
- The economic impact of the AI-powered developer lifecycle and lessons from GitHub Copilot GitHub Blog · 27 June 2023 Own record
“Engineering teams at Duolingo, for instance, have used GitHub Copilot for Business to achieve a 25% increase in developer velocity.”
Reasoning
The GitHub Copilot study (E1) reports that developers with Copilot completed the task 55.8% faster than the control group. That is a published 50%+ improvement on one productivity measure in a controlled setting. The same paper notes that benefits may vary across tasks and languages and that more research is needed on whether the results generalize (E2). That makes the claim broader than the study supports. Another GitHub source from before the judging date reports a 25% velocity gain at Duolingo (E4), and E3 is dated after 2023-11-15, so it was not used.
Cite this claim
Satya Nadella, “Software development with GitHub Copilot is seeing a 50-plus percent improvement.” (said 15 November 2023). Verdict: mostly true. Reliability API, claim c_cdbfb6f121d3, method 0.1.0+28da144b. https://epistacklabs.com/claims/c_cdbfb6f121d3
Claims without a verdict
Is a verdict here wrong, or do you speak for Satya Nadella? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.