Under the current optimization paradigm there is no general method known to get particular inner properties into an AI system or verify they are present.

Judge as of 2022-06-10. The question is whether, under the gradient-descent and deep-learning paradigm, a general method was then known both to instil chosen internal properties (goals or cognition) in a trained system and to verify that they are present. True: reviews, surveys or expert statements from around mid-2022 say that no such general method existed, and that interpretability and alignment techniques were narrow, partial or unproven. Mostly true: that is broadly the case, but some methods did give limited verification or instilling of specific properties in restricted settings, which the word 'general' in the claim allows for. Mixed: the experts of the time were substantially divided on whether existing methods amounted to a general approach. Mostly false: credible evidence shows that methods existed by then which did general work in one of the two directions (instilling or verifying), and the claim overstates their absence. False: a demonstrated, broadly applicable method for both instilling and verifying such properties was known by June 2022.

Set before the research began.

1. E1 Unsolved Problems in ML Safety arXiv (Hendrycks et al.)

   “In contrast to typical software, ML control flows are specified by inscrutable weights learned by gradient optimizers rather than programmed with explicit instructions and general rules from humans.”

2. E2 Interpretable machine learning: Fundamental principles and 10 grand challenges Statistics Surveys (hosted by NSF Public Access Repository) · 2022

   “In that sense, explainability methods are often used as an excuse to use a black box model–whether or not one is actually needed.”

3. E3 The Alignment Problem from a Deep Learning Perspective arXiv (ar5iv rendering)

   “First, since present neural networks are effectively black boxes (Buhrmester et al. 2021), we cannot formally verify that they will reliably behave as intended, and need to rely more on informal analysis.”

4. E4 EIS III: Broad Critiques of Interpretability Research — LessWrong LessWrong (cross-posted to AI Alignment Forum) · 14 February 2023 Own record

   “The interpretability field probably should -- but does not yet -- have clear and consistent evaluation methods.”

Independent reviews from around 2021–2022 describe deep-learning systems as governed by inscrutable weights learned by gradient optimizers [E1]. A review also says that because networks are effectively black boxes, their intended behaviour cannot be formally verified and more reliance on informal analysis is needed [E3]. A 2023 critique says interpretability still lacked clear and consistent evaluation methods [E4], which fits the claim that verification techniques were partial. The evidence supports the verification half of the claim, although partly through sources dated after the judging date, and speaks to the instilling half only indirectly. So the claim is broadly supported, and not every part of it is directly confirmed.