In this talk I discuss some of the epistemological and methodological difficulties with identifying the cognitive competencies of large language models. I begin with Chomksy’s performance/competence distinction, and show that inferences from performance to competence can fail in two ways. Inferences from poor performance to the absence of competence can fail when poor performance is caused by auxiliary factors. Inferences from good performance to presence of competence can fail when the test used to assess competence can be passed by means of shortcuts of various kinds. The talk concludes with a discussion of benchmark testing methods, and how they contribute to both inferential failures.