Psychometric properties of the modified International Cognitive Ability Resource (ICAR) test battery

Kornilova, T.V. Lomonosov Moscow State University

Kornilov Sergei A. Lomonosov Moscow State University

Fig 1. Alternative Confirmation Factor Models (CFA) structures of the extended ICAR test battery. The numbering of the models is given in accordance with Table 3.

Fig 2. Psychometric properties of tasks and the distribution of scores in the ICAR visually-spatial scales and the extended test battery. Difficulty is pecentage of correct answers. MD - missing data.

Table 1. Psychometric properties of ICAR visual spatial scales and ROADS verbal scales.

	Temporary restriction -			Temporary restriction +
Subtest	α	wh	ECV	a	wh	ECV
Rotation	0.94	0.80	74%	0.90	0.67	58%
Matrices	0.68	0.44	37%	0.55	0.44	34%
МХ Scale	0.79	0.51	45%	0.79	0.55	49%
Analogies	0.65	0.32	29%	0.56	0.20	16%
Rotation + Matrices (gf)	0.92	0.74	66%	0.86	0.54	47%
Scale МХ + Analogies (gc)	0.84	0.48	37%	0.80	0.53	39%
All subtests (g)	0.91	0.45	26%	0.80	0.41	18%

NB: α - Cronbach alpha coefficient of internal consistency. wh is the reliability coefficient of MacDonald omega. ECV  is the percentage of the total variance due to the general factor.

Table 2. Psychometric properties of individual tasks of visual and ICAR spatial subtests.

		Temporary restriction -		Temporary restriction +		Temporary restriction -			Temporary restriction +
Subtest	Task	M	pBis	M	pBis	Gf	GS1	GS2	Gf	GS1	GS2
Rotation	R3D.01	0.25	0.57	0.16	0.40	0.75	0.25	-	0.57	-0.39	-
	R3D.02	0.11	-0.25	0.15	-0.24	-0.45	-0.24	-	-0.46	0.24	-
	R3D.03	0.06	-0.24	0.07	-0.24	-0.54	-0.32	-	-0.66	0.40	-
	R3D.04	0.46	0.71	0.45	0.45	0.84	0.23	-	0.63	-0.36	-
	R3D.05	0.48	0.65	0.41	0.53	0.68	0.49	-	0.52	-0.55	-
	R3D.06	0.56	0.65	0.54	0.53	0.65	0.56	-	0.48	-0.58	-
	R3D.07	0.44	0.66	0.37	0.53	0.93	-0.08	-	0.78	-0.40	-
	R3D.08	0.50	0.73	0.38	0.51	0.81	0.40	-	0.56	-0.48	-
	R3D.09	0.56	0.65	0.50	0.56	0.72	0.47	-	0.59	-0.56	-
	R3D.10	0.41	0.68	0.31	0.57	0.73	0.43	-	0.50	-0.59	-
	R3D.11	0.35	0.67	0.24	0.54	0.74	0.48	-	0.61	-0.52	-
	R3D.12	0.41	0.69	0.35	0.60	0.81	0.29	-	0.50	-0.63	-
	R3D.13	0.33	0.68	0.23	0.54	0.82	0.25	-	0.58	-0.53	-
	R3D.14	0.48	0.67	0.40	0.52	0.93	-0.06	-	0.52	-0.53	-
	R3D.15	0.49	0.70	0.40	0.61	0.85	0.20	-	0.40	-0.71	-
	R3D.16	0.57	0.60	0.48	0.56	0.65	0.45	-	0.23	-0.76	-
	R3D.17	0.48	0.74	0.39	0.59	0.83	0.31	-	0.33	-0.74	-
	R3D.18	0.06	-0.29	0.07	-0.22	-0.87	0.07	-	-0.44	0.39	-
	R3D.19	0.52	0.64	0.39	0.57	0.89	-0.05	-	0.26	-0.76	-
	R3D.20	0.62	0.63	0.47	0.49	0.68	0.53	-	-0.10	-0.87	-
	R3D.21	0.32	0.66	0.18	0.47	0.81	0.30	-	0.14	-0.72	-
	R3D.22	0.60	0.62	0.37	0.46	0.64	0.57	-	-0.19	-0.85	-
	R3D.23	0.43	0.47	0.29	0.40	0.46	0.50	-	-0.18	-0.75	-
	R3D.24	0.45	0.64	0.31	0.60	0.67	0.42	-	0.01	-0.90	-
Matrices	MR.43	0.86	0.37	0.85	0.16	0.28	-	0.60	0.05	-	0.29
	MR.44	0.73	0.29	0.75	0.18	0.08	-	0.50	0.36	-	0.21
	MR.45	0.74	0.32	0.70	0.21	0.25	-	0.47	0.36	-	0.20
	MR.46	0.70	0.37	0.66	0.22	0.24	-	0.56	0.24	-	0.26
	MR.47	0.76	0.28	0.73	0.22	0.40	-	0.35	0.32	-	0.34
	MR.48	0.69	0.37	0.72	0.18	0.08	-	0.61	0.06	-	0.32
	MR.50	0.43	0.23	0.39	0.27	0.31	-	0.26	0.07	-	0.40
	MR.53	0.75	0.36	0.66	0.35	0.34	-	0.46	0.10	-	0.62
	MR.54	0.43	0.16	0.42	0.21	-0.02	-	0.29	-0.08	-	0.42
	MR.55	0.54	0.42	0.35	0.31	0.31	-	0.59	-0.06	-	0.69
	MR.56	0.55	0.38	0.40	0.24	0.13	-	0.58	-0.23	-	0.62

NB: Task IDs are in accordance with the original ICAR nomenclature. M is task complexity (% of correct answers). pBis is the point-biserial correlation coefficient between the task and the total scale score. Gf is standardized factor load of the task (common factor). GS1, GS2 are standardized factor load indicators (subtest-specific factors).

Тable 3. Indicators of suitability models of abilities

	Visual spatial subtests ICAR (MIRT)					All subtests, expanded battery (CFA)
Model	М2	df	p	CFI	RMSEA	Y-B	df	p	CFI	RMSEA
М1 – 1 factoe	1141.05	560	<0.001	0.960	0.061	368.69	20	<0.001	0.627	0.245
М2 – 2 orthogonal factor	915.16	560	<0.001	0.975	0.047	165.94	20	<0.001	0.838	0.158
М3 – 2 oblique factor	888.33	559	<0.001	0.977	0.046	127.87	19	<0.001	0.879	0.140
М4 – bi-factor*	679.80	525	<0.001	0.989	0.032	22.63	12	0.031	0.988	0.053
М5 – factor of the 2nd order	-	-	-	-	-	127.87	19	<0.001	0.879	0.140
М6 – bi-factor model with Speed**factor	949.25	539	<0.001	0.958	0.044	-	-	-	-

NB: “Temporary Limitation -” M2 - test function of the model; df is the number of degrees of freedom. p is the significance level when testing the hypothesis of the absolute suitability of the model. CFI is a comparative fitness index. Y-B - chi-square corrected for data abnormality. RMSEA is the root of the mean square of the approximation error. * - the model that showed the best suitability indices.

** The indicators for the condition "Temporary Limit +" are given.

References:

Beaujean, A.A. (2015). John Carroll’s view of intelligence: Bi-factor vs higher-order models. Journal of Intelligence, 3(4), 121–136. doi: 10.3390/jintelligence3040121

Becker, N., Koch, M., Schult, J., & Spinath, F.M. (2017). Setting Doesn’t Matter Much. A Meta-Analytic Comparison of the Results of Intelligence Tests Obtained in Group and Individual Settings. European Journal of Psychological Assessment, 33, 1–8.

Canivez, G.L., & Youngstrom, E.A. (2019). Challenges to the Cattell-Horn-Carroll Theory: Empirical, Clinical, and Policy Implications. Applied Measurement in Education, (32)3, 232–248. doi:10.1080/08957347.2019.1619562

Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press. doi: 10.1017/CBO9780511571312

Cattell, R.B. (1987). Intelligence: Its structure, growth and action. New York: Elsevier.

Chalmers, R.P. (2012). Mirt: a Multidimensional Item Response Theory Package for the R. Environment. Journal of Statistical Software, 48(6), 1–19. doi: 10.18637/jss.v048.i06

Condon, D.M., & Revelle, W. (2014). The International Cognitive Ability Resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52–64.  doi: 10.1016/j.intell.2014.01.004

Cucina, J., & Byle, K. (2017). The bifactor model fits better than the higher-order model in more than 90% of comparisons for mental abilities test batteries. Journal of Intelligence, 5(27), 1–21. doi: 10.3390/jintelligence5030027

Davydov D.G., & Chmykhova E.V. (2016). Application of the test Standard Progressive Raven matrices in the time limit mode. [Voprosy psikhologii], 4, 129–139. doi: 10.1098/rstb.2017.0284

Dubois, J., Galdi, P., Paul, L.K., & Adolphs, R. (2018). A distributed brain network predicts general intelligence from resting-state human neuroimaging data. Philosophical Transaction of the Royal Society B: Biological Sciences, 373 (1756).

Gardner G. (2007). The structure of the mind. Theory of multiple intelligence. Moscow, OOO I.D. Williams.

Gottfredson, L.S. (2016). A g theorist on why Kovacs and Conway’s Process Overlap Theory Amplifies, Not Opposes, g theory. Psychological Inquiry, 27(3), 210–217. doi: 10.1080/1047840X.2016.1203232

Hamel, R., & Schmittmann, V.D. (2006). The 20-Minute Version as a Predictor of the Raven Advanced Progressive Matrices Test. Educational and Psychological Measurement, 66(6), 1039–1046. doi: 10.1177/0013164406288169

Horn, J.L., & Blankson, N. (2005). Foundations for better understanding of cognitive abilities. In: D.P. Flanagan & P.L. Harrison (Eds.), Contemporary intellectual assessment. Theories, tests, and issues (2nd ed.). New York: Guilford Press, 41–68.

Horn, J.L., & Cattell, R.B. (1966). Refinement and test of the theory of fluid and crystallized general intelligences. Journal of Educational Psychology, 57(5б) 253–270. doi: 10.1037/h0023816

Horn, J.L., & Noll, J. (1997). Human cognitive capabilities: Gf-Gc theory. In: Flanagan, D.P., Genshaft, J.L., & Harrison, P.L., Eds. Contemporary Intellectual Assessment: Theories, Tests and Issues. New York: Guilford Press, 53–91.

Jensen, A.R. (1998). The g factor: The science of mental ability. Westport, CT: Praeger.

Kirkegaard, E.O.W., & Nordbjerg, O. (2015). Validating a Danish translation of the International Cognitive Ability Resource sample test and Cognitive Reflection Test in a student sample Group, 4(5). 10.26775/ODP.2015.07.31

Koller, M., & Stahel, W.A. (2011). Sharpening wald-type inference in robust regression for small samples. Computational Statistics & Data Analysis, 55(8), 2504-2515. doi: 10.1016/j.csda.2011.02.014

Kornilov S.A., & Grigorenko E.L. (2010). Methodological complex for the diagnosis of academic, creative and practical abilities. [Psikhologicheskiy zhurnal], 31(2), 90–103.

Kornilova T.V. (2016). Intellectual and personal potential of a person in conditions of uncertainty and risk. Moscow, Nestor – Istoriya.

Kovacs, K., & Conway, A.R.A. (2016). Process overlap theory: A unified account of the general factor of intelligence. Psychological Inquiry, 27(3), 151–177. doi: 10.1080/1047840X.2016.1153946

Krasavtseva Yu.V., & Kornilova T.V. (2018). Emotional and academic intelligence as predictors of strategies in the Iowa Game Problem (IGT). [Psikhologicheskiy zhurnal], 39(3), 29-43.

Kyllonen, P., Hartman, R., Sprenger, A., Weeks, J., Bertling, M., McGrew, K., Kriz, S., Bertling, J., Fife, J., & Stankov, L. (2019). General fluid/inductive reasoning battery for a high-ability population. Behavior Research Methods, 51(2), 507–522. doi: 10.3758/s13428-018-1098-4

Mayer, J.D. (2014). Personal Intelligence: The power of personality and how it shapes our lives. N. Y.: Farrar, Straus and Giroux.

Mayer, J.D., Caruso, D.R., & Salovey, P. (2016). The ability model of emotional intelligence: principles and updates. Emotion review, 8(4), 290–300. doi: 10.1177/1754073916639667

McDonald, R.P. (1999). Test theory: A unified treatment. L. Erlbaum Associates, Mahwah.

Nizbett R. (2013). What is intelligence and how to develop it. Moscow, Alpina nonfikshn.

Raven J., Raven J.K., & Cort J.H. (2012). A Guide to Raven's Progressive Matrices and Vocabulary Scales. Moscow, “Kogito-Tsentr”. doi: 10.1037/t10910-000

Raven, J. (1989). The Raven Progressive Matrices: A Review of National Norming Studies and Ethnic and Socioeconomic Variation Within the United States. Journal of Educational Measurement, 26(1), 1–16. doi: 10.1111/j.1745-3984.1989.tb00314.x

Raven, J.C., Court J.H., &Raven J. (1992). Manual for Raven’s Progressive Matrices and Mill Hill Vocabulary Scales. Oxford: Oxford Psychologists Press. doi: 10.1080/00273171.2012.715555

Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696.

Ren, X., Wang, T., Sun, S., Deng, M., & Schweizer, K. (2018). Speeded testing in the assessment of intelligence gives rise to a speed factor. Intelligence, 64–71. doi: 10.1016/j.intell.2017.11.004

Rosseel, Y. (2012). Lavaan: An r package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. doi: 10.18637/jss.v048.i02

Salovey, P., & Mayer, J.D. (1990). Emotional intelligence. Imagin. Cogn. Personal,  9, 185–211. doi: 10.2190/DUGG-P24E-52WK-6CDG

Schneider, W.J., & McGrew, K.S. (2019). Process Overlap Theory is a milestone achievement among intelligence theories. Journal of Applied Research in Memory and Cognition.  32(3). doi: 10.1016/j.jarmac.2019.06.006

Schneider, W.J.,   Mayer, J.D., & Newman, D.A. (2016). Integrating Hot and Cool Intelligences: Thinking Broadly about Broad Abilities. Journal of Intelligence, 4(1), Art.1. doi: 10.3390/jintelligence4010001

Spearman, C.E. (1927). The abilities of man: Their nature and measurement. London: MacMillan.

Sternberg, R.J. (1999). The theory of successful intelligence. Review of General Psychology,  3(4), 292–316. doi: 10.1037/1089-2680.3.4.292

Sternberg R.J., & The Rainbow Project Collaborators (2006). The Rainbow Project: Enhancing the SAT through assessments of analytical, practical and creative skills. Intelligence. 34(4), 321–350. doi: 10.1016/j.intell.2006.01.002

Takagi, Y., Hirayama, J., & Tanaka, S.C. (2018) State-Unspecific Modes of Whole-Brain Functional Connectivity Predict intelligence and life outcomes. bioRxiv, preprint first posted.  doi: 10.1101/283846