As of 2021, neural networks still struggle with notions that are obvious to babies.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2021-04-13. True: published research or benchmarks from around 2021 show that state-of-the-art neural networks performed poorly on basic concepts that infants understand, such as object permanence, intuitive physics, causality or numerosity (e.g. results on IntPhys, infant-inspired AI benchmarks). Mostly true: the research shows widespread failures, but some systems handled certain such notions reasonably well. Mixed: the evidence was evenly split, with neural networks doing well on about half of the commonly cited infant-level notions. Mostly false: neural networks broadly handled such notions by 2021, with only narrow failures. False: the evidence showed neural networks reliably mastered infant-level concepts.
Set before the research began.
Evidence (1)
-
E1 Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others arXiv
“Nevertheless, recently proposed, deep-learning-based agency reasoning models fail to show infant-like reasoning, leaving BIB an open challenge.”
Reasoning
E1 reports that recently proposed deep-learning models for reasoning about agents fail to show infant-like reasoning, but it is undated, so it cannot be placed relative to 2021-04-13. It also covers only one area (reasoning about agents) and does not show how neural networks did on the other infant-level notions named in the criteria, so it does not settle the claim.
Cite this claim
Gary Marcus, “As of 2021, neural networks still struggle with notions that are obvious to babies.” (said 13 April 2021). Verdict: unresolvable. Reliability API, claim c_87d3c9f16c09, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_87d3c9f16c09
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Gary Marcus
-
Mostly true
“Reviews by Joanna Stern and Marques Brownle of one early prototype were damning;”
Reviews by Joanna Stern and Marques Brownlee of one early humanoid robot prototype were damning.
-
Mostly true
“Many would agree we have reached a point of diminishing returns for scaling; faith in scaling as a route to AGI has dissipated.”
-
Mostly true
“Despite all the hype, agents didn’t turn out to be reliable.”
-
Mixed
“but the systems don’t have the knowledge graph to do that.”
-
Resolved true
“Hasn’t worked so far. That’s for sure.”
Bolting extra components onto an LLM and scaling it up has so far not produced human-level AGI.
Verdicts are made automatically; how verdicts are made.