Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
69 49 87 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
69 49 87
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 77 factual claims, 5 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Sam Altman's claims
930 claims · page 10 of 10
“Basic search with a lot of compute power has just worked better than expected.”
Sam Altman, blog author February 2015 Machine intelligence, part 1 (Sam Altman's blog)
Basic search with a lot of compute power has worked better than expected in machine intelligence.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judged as of February 2015, against the state of machine intelligence research then. True: by early 2015, published results and expert commentary showed that simple or general methods scaled with large compute (for example deep neural nets trained on GPUs, as in ImageNet results, or search-based game playing) performed better than many experts had expected. Mostly true: this was broadly the view, though the 'basic search' framing fits only some of the leading results. Mixed: the evidence is split, with some fields showing compute-driven gains beyond expectations and others not. Mostly false: the gains came mainly from new algorithmic ideas rather than simple methods plus compute, and this is overstated. False: the evidence shows compute-heavy simple methods had underperformed expectations by 2015.
Evidence
- Microsoft researchers say their newest deep learning system beats humans -- and Google VentureBeat · 10 February 2015
“The Microsoft creation got a 4.94 percent error rate for the correct classification of images in the 2012 version of the widely recognized ImageNet data set , compared with a 5.1 percent error rate among humans, according to the paper.”
- CMU School of Computer Science Carnegie Mellon University (hosting the NIPS 2012 paper by Krizhevsky, Sutskever and Hinton)
“We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.”
- Microsoft, Google Beat Humans at Image Recognition EE Times · 19 February 2015
“Microsoft did use Nvidia GPUs, but bought and configured their own supercomputer using them to simulate parametric rectified linear neural units to become the “1st to beat a human” at image classification.”
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification - Microsoft Research Microsoft Research · 1 February 2015
“Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities.”
Reasoning
By early 2015, published results showed that large deep networks trained on GPUs were doing very well. AlexNet won ILSVRC-2012 with a 15.3% top-5 error, against 26.2% for the second-best entry (E2). In February 2015, Microsoft reported a 4.94% ImageNet error, below the 5.1% human rate, using GPU hardware it bought and configured itself (E1, E3). However, the Microsoft paper also credits algorithmic work, including a new initialization method (E4), and the system used parametric rectified linear units (E3), so the "basic search plus compute" framing fits only part of the leading results.
Cite this claim
Sam Altman, “Basic search with a lot of compute power has worked better than expected in machine intelligence.” (said February 2015). Verdict: mostly true. Reliability API, claim c_a8b62a045781, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_a8b62a045781
Claims without a verdict
Is a verdict here wrong, or do you speak for Sam Altman? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.