ISSN 2079-6617 (Print)
ISSN 2309-9828 (Online)
Articles

Article

MainArticlesVolumes

Kornilova T.V., Kornilov S.A., Zirenko M.S., Chumakova M. A. (2019). Psychometric properties of the modified International Cognitive Ability Resource (ICAR) test battery. National Psychological Journal, [Natsional’nyy psikhologicheskiy zhurnal], (12)3, 32–45.

Abstract

Background. The emergence of the psychometric tradition in Russian psychology necessitates a thorough exploration and adoption of the international experience in the development of appropriate measurement instruments and methodologies. The goal of the present study was thus to perform a comprehensive evaluation of the International Cognitive Ability Resource (ICAR) instrument in a Russian sample. The modified ICAR instrument consisted of visuo-spatial subtests Three-dimensional Rotation, Mental Reasoning (original ICAR subtests), as well as the previously developed verbal subtests Mill-Hill verbal scale and Analogies (from the ROADS test battery).

Design. The ICAR battery was administered to n=681 individuals (377 females) in the age from 17 to 59 years (Med=23, M=25.83, SD=7.58 лет) who were either college students or adults with a college degree from the city of Moscow who volunteered for the study. The test battery was administered without (n=284) as well as with (n=397) a time limit.

Results. The study demonstrated adequate psychometric properties of the modified ICAR battery, and revealed a fundamentally bifactor structure of the battery both at the level of individual items as well as at the level of subtests. Thus, individual’s performance on each item or subtest can be conceptualized as being driven by specific (e.g., fluid or verbal) as well as general (g) intelligence factors. We also show that introducing the time limit distorts the psychometric structure of the battery, lowers internal consistency, and reduces the g-saturation of the resulting scores, a finding that has important implications for the theory and practice of testing.

Received: 09/07/2019
Accepted: 09/15/2019
Pages: 32-45
DOI: 10.11621/npj.2019.0304

Sections: Cognitive Psychology;

PDF: Download

Keywords: cognitive ability; intelligence; Cattell-Horn-Carroll theory; International Cognitive Ability Resource (ICAR);

Available Online 20.10.2019


Fig 1. Alternative Confirmation Factor Models (CFA) structures of the extended ICAR test battery. The numbering of the models is given in accordance with Table 3.


Fig 2. Psychometric properties of tasks and the distribution of scores in the ICAR visually-spatial scales and the extended test battery. Difficulty is pecentage of correct answers. MD - missing data.

Table 1. Psychometric properties of ICAR visual spatial scales and ROADS verbal scales.

Temporary
restriction -

Temporary
restriction +

Subtest

α

wh

ECV

a

wh

ECV

Rotation

0.94

0.80

74%

0.90

0.67

58%

Matrices

0.68

0.44

37%

0.55

0.44

34%

МХ Scale

0.79

0.51

45%

0.79

0.55

49%

Analogies

0.65

0.32

29%

0.56

0.20

16%

Rotation + Matrices (gf)

0.92

0.74

66%

0.86

0.54

47%

Scale МХ + Analogies (gc)

0.84

0.48

37%

0.80

0.53

39%

All subtests (g)

0.91

0.45

26%

0.80

0.41

18%

NB: α - Cronbach alpha coefficient of internal consistency. wh is the reliability coefficient of MacDonald omega. ECV  is the percentage of the total variance due to the general factor.

Table 2. Psychometric properties of individual tasks of visual and ICAR spatial subtests.

Temporary
restriction -

Temporary
restriction +

Temporary
restriction -

Temporary
restriction +

Subtest

Task

M

pBis

M

pBis

Gf

GS1

GS2

Gf

GS1

GS2

Rotation

R3D.01

0.25

0.57

0.16

0.40

0.75

0.25

-

0.57

-0.39

-

R3D.02

0.11

-0.25

0.15

-0.24

-0.45

-0.24

-

-0.46

0.24

-

R3D.03

0.06

-0.24

0.07

-0.24

-0.54

-0.32

-

-0.66

0.40

-

R3D.04

0.46

0.71

0.45

0.45

0.84

0.23

-

0.63

-0.36

-

R3D.05

0.48

0.65

0.41

0.53

0.68

0.49

-

0.52

-0.55

-

R3D.06

0.56

0.65

0.54

0.53

0.65

0.56

-

0.48

-0.58

-

R3D.07

0.44

0.66

0.37

0.53

0.93

-0.08

-

0.78

-0.40

-

R3D.08

0.50

0.73

0.38

0.51

0.81

0.40

-

0.56

-0.48

-

R3D.09

0.56

0.65

0.50

0.56

0.72

0.47

-

0.59

-0.56

-

R3D.10

0.41

0.68

0.31

0.57

0.73

0.43

-

0.50

-0.59

-

R3D.11

0.35

0.67

0.24

0.54

0.74

0.48

-

0.61

-0.52

-

R3D.12

0.41

0.69

0.35

0.60

0.81

0.29

-

0.50

-0.63

-

R3D.13

0.33

0.68

0.23

0.54

0.82

0.25

-

0.58

-0.53

-

R3D.14

0.48

0.67

0.40

0.52

0.93

-0.06

-

0.52

-0.53

-

R3D.15

0.49

0.70

0.40

0.61

0.85

0.20

-

0.40

-0.71

-

R3D.16

0.57

0.60

0.48

0.56

0.65

0.45

-

0.23

-0.76

-

R3D.17

0.48

0.74

0.39

0.59

0.83

0.31

-

0.33

-0.74

-

R3D.18

0.06

-0.29

0.07

-0.22

-0.87

0.07

-

-0.44

0.39

-

R3D.19

0.52

0.64

0.39

0.57

0.89

-0.05

-

0.26

-0.76

-

R3D.20

0.62

0.63

0.47

0.49

0.68

0.53

-

-0.10

-0.87

-

R3D.21

0.32

0.66

0.18

0.47

0.81

0.30

-

0.14

-0.72

-

R3D.22

0.60

0.62

0.37

0.46

0.64

0.57

-

-0.19

-0.85

-

R3D.23

0.43

0.47

0.29

0.40

0.46

0.50

-

-0.18

-0.75

-

R3D.24

0.45

0.64

0.31

0.60

0.67

0.42

-

0.01

-0.90

-

Matrices

MR.43

0.86

0.37

0.85

0.16

0.28

-

0.60

0.05

-

0.29

MR.44

0.73

0.29

0.75

0.18

0.08

-

0.50

0.36

-

0.21

MR.45

0.74

0.32

0.70

0.21

0.25

-

0.47

0.36

-

0.20

MR.46

0.70

0.37

0.66

0.22

0.24

-

0.56

0.24

-

0.26

MR.47

0.76

0.28

0.73

0.22

0.40

-

0.35

0.32

-

0.34

MR.48

0.69

0.37

0.72

0.18

0.08

-

0.61

0.06

-

0.32

MR.50

0.43

0.23

0.39

0.27

0.31

-

0.26

0.07

-

0.40

MR.53

0.75

0.36

0.66

0.35

0.34

-

0.46

0.10

-

0.62

MR.54

0.43

0.16

0.42

0.21

-0.02

-

0.29

-0.08

-

0.42

MR.55

0.54

0.42

0.35

0.31

0.31

-

0.59

-0.06

-

0.69

MR.56

0.55

0.38

0.40

0.24

0.13

-

0.58

-0.23

-

0.62

NB: Task IDs are in accordance with the original ICAR nomenclature. M is task complexity (% of correct answers). pBis is the point-biserial correlation coefficient between the task and the total scale score. Gf is standardized factor load of the task (common factor). GS1, GS2 are standardized factor load indicators (subtest-specific factors).

Тable 3. Indicators of suitability models of abilities

Visual spatial subtests ICAR (MIRT)

All subtests, expanded battery (CFA)

Model

М2

df

p

CFI

RMSEA

Y-B

df

p

CFI

RMSEA

М1 – 1 factoe

1141.05

560

<0.001

0.960

0.061

368.69

20

<0.001

0.627

0.245

М2 – 2 orthogonal factor

915.16

560

<0.001

0.975

0.047

165.94

20

<0.001

0.838

0.158

М3 – 2 oblique factor

888.33

559

<0.001

0.977

0.046

127.87

19

<0.001

0.879

0.140

М4 – bi-factor*

679.80

525

<0.001

0.989

0.032

22.63

12

0.031

0.988

0.053

М5 – factor of the 2nd order

-

-

-

-

-

127.87

19

<0.001

0.879

0.140

М6 – bi-factor model with Speed**factor

949.25

539

<0.001

0.958

0.044

-

-

-

-

NB: “Temporary Limitation -” M2 - test function of the model; df is the number of degrees of freedom. p is the significance level when testing the hypothesis of the absolute suitability of the model. CFI is a comparative fitness index. Y-B - chi-square corrected for data abnormality. RMSEA is the root of the mean square of the approximation error. * - the model that showed the best suitability indices.

** The indicators for the condition "Temporary Limit +" are given.

References:

Beaujean, A.A. (2015). John Carroll’s view of intelligence: Bi-factor vs higher-order models. Journal of Intelligence, 3(4), 121–136. doi: 10.3390/jintelligence3040121

Becker, N., Koch, M., Schult, J., & Spinath, F.M. (2017). Setting Doesn’t Matter Much. A Meta-Analytic Comparison of the Results of Intelligence Tests Obtained in Group and Individual Settings. European Journal of Psychological Assessment, 33, 1–8.

Canivez, G.L., & Youngstrom, E.A. (2019). Challenges to the Cattell-Horn-Carroll Theory: Empirical, Clinical, and Policy Implications. Applied Measurement in Education, (32)3, 232–248. doi:10.1080/08957347.2019.1619562

Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press. doi: 10.1017/CBO9780511571312

Cattell, R.B. (1987). Intelligence: Its structure, growth and action. New York: Elsevier.

Chalmers, R.P. (2012). Mirt: a Multidimensional Item Response Theory Package for the R. Environment. Journal of Statistical Software, 48(6), 1–19. doi: 10.18637/jss.v048.i06

Condon, D.M., & Revelle, W. (2014). The International Cognitive Ability Resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52–64.  doi: 10.1016/j.intell.2014.01.004

Cucina, J., & Byle, K. (2017). The bifactor model fits better than the higher-order model in more than 90% of comparisons for mental abilities test batteries. Journal of Intelligence, 5(27), 1–21. doi: 10.3390/jintelligence5030027

Davydov D.G., & Chmykhova E.V. (2016). Application of the test Standard Progressive Raven matrices in the time limit mode. [Voprosy psikhologii], 4, 129–139. doi: 10.1098/rstb.2017.0284

Dubois, J., Galdi, P., Paul, L.K., & Adolphs, R. (2018). A distributed brain network predicts general intelligence from resting-state human neuroimaging data. Philosophical Transaction of the Royal Society B: Biological Sciences, 373 (1756).

Gardner G. (2007). The structure of the mind. Theory of multiple intelligence. Moscow, OOO I.D. Williams.

Gottfredson, L.S. (2016). A g theorist on why Kovacs and Conway’s Process Overlap Theory Amplifies, Not Opposes, g theory. Psychological Inquiry, 27(3), 210–217. doi: 10.1080/1047840X.2016.1203232

Hamel, R., & Schmittmann, V.D. (2006). The 20-Minute Version as a Predictor of the Raven Advanced Progressive Matrices Test. Educational and Psychological Measurement, 66(6), 1039–1046. doi: 10.1177/0013164406288169

Horn, J.L., & Blankson, N. (2005). Foundations for better understanding of cognitive abilities. In: D.P. Flanagan & P.L. Harrison (Eds.), Contemporary intellectual assessment. Theories, tests, and issues (2nd ed.). New York: Guilford Press, 41–68.

Horn, J.L., & Cattell, R.B. (1966). Refinement and test of the theory of fluid and crystallized general intelligences. Journal of Educational Psychology, 57(5б) 253–270. doi: 10.1037/h0023816

Horn, J.L., & Noll, J. (1997). Human cognitive capabilities: Gf-Gc theory. In: Flanagan, D.P., Genshaft, J.L., & Harrison, P.L., Eds. Contemporary Intellectual Assessment: Theories, Tests and Issues. New York: Guilford Press, 53–91.

Jensen, A.R. (1998). The g factor: The science of mental ability. Westport, CT: Praeger.

Kirkegaard, E.O.W., & Nordbjerg, O. (2015). Validating a Danish translation of the International Cognitive Ability Resource sample test and Cognitive Reflection Test in a student sample Group, 4(5). 10.26775/ODP.2015.07.31

Koller, M., & Stahel, W.A. (2011). Sharpening wald-type inference in robust regression for small samples. Computational Statistics & Data Analysis, 55(8), 2504-2515. doi: 10.1016/j.csda.2011.02.014

Kornilov S.A., & Grigorenko E.L. (2010). Methodological complex for the diagnosis of academic, creative and practical abilities. [Psikhologicheskiy zhurnal], 31(2), 90–103.

Kornilova T.V. (2016). Intellectual and personal potential of a person in conditions of uncertainty and risk. Moscow, Nestor – Istoriya.

Kovacs, K., & Conway, A.R.A. (2016). Process overlap theory: A unified account of the general factor of intelligence. Psychological Inquiry, 27(3), 151–177. doi: 10.1080/1047840X.2016.1153946

Krasavtseva Yu.V., & Kornilova T.V. (2018). Emotional and academic intelligence as predictors of strategies in the Iowa Game Problem (IGT). [Psikhologicheskiy zhurnal], 39(3), 29-43.

Kyllonen, P., Hartman, R., Sprenger, A., Weeks, J., Bertling, M., McGrew, K., Kriz, S., Bertling, J., Fife, J., & Stankov, L. (2019). General fluid/inductive reasoning battery for a high-ability population. Behavior Research Methods, 51(2), 507–522. doi: 10.3758/s13428-018-1098-4

Mayer, J.D. (2014). Personal Intelligence: The power of personality and how it shapes our lives. N. Y.: Farrar, Straus and Giroux.

Mayer, J.D., Caruso, D.R., & Salovey, P. (2016). The ability model of emotional intelligence: principles and updates. Emotion review, 8(4), 290–300. doi: 10.1177/1754073916639667

McDonald, R.P. (1999). Test theory: A unified treatment. L. Erlbaum Associates, Mahwah.

Nizbett R. (2013). What is intelligence and how to develop it. Moscow, Alpina nonfikshn.

Raven J., Raven J.K., & Cort J.H. (2012). A Guide to Raven's Progressive Matrices and Vocabulary Scales. Moscow, “Kogito-Tsentr”. doi: 10.1037/t10910-000

Raven, J. (1989). The Raven Progressive Matrices: A Review of National Norming Studies and Ethnic and Socioeconomic Variation Within the United States. Journal of Educational Measurement, 26(1), 1–16. doi: 10.1111/j.1745-3984.1989.tb00314.x

Raven, J.C., Court J.H., &Raven J. (1992). Manual for Raven’s Progressive Matrices and Mill Hill Vocabulary Scales. Oxford: Oxford Psychologists Press. doi: 10.1080/00273171.2012.715555

Reise, S.P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696.

Ren, X., Wang, T., Sun, S., Deng, M., & Schweizer, K. (2018). Speeded testing in the assessment of intelligence gives rise to a speed factor. Intelligence, 64–71. doi: 10.1016/j.intell.2017.11.004

Rosseel, Y. (2012). Lavaan: An r package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. doi: 10.18637/jss.v048.i02

Salovey, P., & Mayer, J.D. (1990). Emotional intelligence. Imagin. Cogn. Personal,  9, 185–211. doi: 10.2190/DUGG-P24E-52WK-6CDG

Schneider, W.J., & McGrew, K.S. (2019). Process Overlap Theory is a milestone achievement among intelligence theories. Journal of Applied Research in Memory and Cognition.  32(3). doi: 10.1016/j.jarmac.2019.06.006

Schneider, W.J.,   Mayer, J.D., & Newman, D.A. (2016). Integrating Hot and Cool Intelligences: Thinking Broadly about Broad Abilities. Journal of Intelligence, 4(1), Art.1. doi: 10.3390/jintelligence4010001

Spearman, C.E. (1927). The abilities of man: Their nature and measurement. London: MacMillan.

Sternberg, R.J. (1999). The theory of successful intelligence. Review of General Psychology,  3(4), 292–316. doi: 10.1037/1089-2680.3.4.292

Sternberg R.J., & The Rainbow Project Collaborators (2006). The Rainbow Project: Enhancing the SAT through assessments of analytical, practical and creative skills. Intelligence. 34(4), 321–350. doi: 10.1016/j.intell.2006.01.002

Takagi, Y., Hirayama, J., & Tanaka, S.C. (2018) State-Unspecific Modes of Whole-Brain Functional Connectivity Predict intelligence and life outcomes. bioRxiv, preprint first posted.  doi: 10.1101/283846



For citing this article:

Kornilova T.V., Kornilov S.A., Zirenko M.S., Chumakova M. A. (2019). Psychometric properties of the modified International Cognitive Ability Resource (ICAR) test battery. National Psychological Journal, [Natsional’nyy psikhologicheskiy zhurnal], (12)3, 32–45.