Elaboration of Screening Scales for Early Diagnosis of Developmental Delay in Four- to Five-Year-Old Children in Russia

Nasledov, A.D. St. Petersburg State University, St. Petersburg, Russia

Miroshnikov, S.A. St. Petersburg State University, St. Petersburg, Russia

Tkacheva, L.O. St. Petersburg State University, St. Petersburg, Russia

Introduction

Traditionally, the diagnostic tools used in developmental psychology had to meet high psychometric requirements. However, presently, the efficiency and flexibility of their application are becoming more and more relevant, especially when the tools are used in the diagnosis of developmental problems in preschool children. Efficiency is understood to be the possibility of fast screening in large groups to identify children whose development is slightly distorted, but this needs to be addressed even before there are obvious signs of delays.

Traditional tests that are designed for manual execution and processing cannot meet these requirements. The main reason of such imperfection is that traditional methods sacrifice the accuracy of the results when collecting data using forms and manual processing of the results: the important details of the tasks are missed (for example, the number of levels of the elements of a complex task are not taken into consideration). Nevertheless, in Russia, there is even a lack of traditional methods for diagnosing preschool children, which correspond to the standard psychometric requirements for diagnostic tools (reliability and validity, the presence of reasonable norms, legality of distribution). These diagnostic obstacles lead to belated and ineffective attempts to correct a variety of developmental problems, among which the most common are developmental delays (DD) in various forms. Developmental delay can be defined as a significant delay in two or more developmental domains: gross and fine motor skills; speech and language; cognition; personal and social development; or activities of daily living (Majnemer, Shevell, 1995). Recently the number of children with DD in Russia has been increasing (Isaev, 2017), mainly due to medical advances and assistance for premature and somatically weakened children. It is also necessary to note that children who are initially diagnosed as at risk are always in need of further special support, otherwise the positive effect of early medical care may be lost (Collins et al., 2017). It is well known that diagnosis at an early age is significant for intervention: the younger the child, the more flexible his/her brain and the richer the compensatory possibilities (Dennis, 2013). DD has a high potential for rehabilitation, but when only when timely diagnosed and corrected (Barnett, 1995): wasted time significantly increases the probability of an adverse outcome (Kok, 2016).

It is customary to distinguish the many factors and markers that influence mental development at an early age. Among them are motor development (Frick & Mohring, 2013), speech development (Tsao, Liu & Kuhl , 2004), working memory (Cowan & Alloway, 2009), sustained attention (Voelke & Roebers, 2016), spatial thinking (Lauer & Lourenco, 2016), cognitive control mechanisms (Engel de Abreu, Conway & Gathercole, 2010), and social intellect (Wellman et al., 2008. However, the role of these factors as predictors of DD at a particular age is still unclear. While various screenings are used in the West, such as Griffiths (Luiz, 2006) and Vineland (Van Duijn, 2009), there is no such system for monitoring the mental development of children in Russia. Presumably, it could be possible to adapt one of the existing Western screenings to the Russian sample; however, this approach seems to be doubtful, given the complexity of the adaptation itself, which is comparable to the development of new tools, and the legal and financial problems of licensing. According to Carroll A. E., the application of computerized cognitive development screenings to detect DD significantly increased the number of diagnosed children and the timely intervention for children at an early age (Carroll, 2014). Therefore, we aimed to create a system for screening diagnostics of mental development in 4- and 5-year-olds, specific to the Russian sample and allowing for the usage of technologies to obtain reliable results on the basis of a relatively small set of data.

Most studies of DD are based on the classical ideas about the factor structure of intelligence that is specified a priori in tests. However, the usage of structured tests with several types of tasks forming the subtests and factors significantly limits the possibility of studying the real factor structure of abilities, partly due to the questionnaire testing tradition and the manual processing of its results (Macmann & Barnett, 1994). For the purpose of our study, a broader set of initial characteristics was chosen, which allowed for a looser grouping of the factors. We also carefully recorded the results of the child's performance on each task and took into account the performance success of each level of the task and the age of the child (up to one day), etc.

Method

Materials

For data collection, we used the multifactor study of the mental development tool as part of a software complex for longitudinal research (Автор, 2001). It is a complex tool made up of formal observations and tests, which includes 502 tasks covering a wide range of abilities (from motor skills to cognitive and social development). The complexity of the tasks was chosen depending on the calendar age of the child (from 2 months to 7 years). The bank of tasks was formed on the basis of a survey of a large number of expert practitioner psychologists (the content of tasks is presented on the Internet resource of the project http://info11.testpsy.net). Most of the tasks used a categorical (dichotomous) response format: Yes = 2 (the child can perform a control action) or No = 1 (cannot). The baseline data included 847 dichotomous points: 236 points for 4-year-olds and 349 points for 5-year-olds were used in this study. Examples of the tasks are presented in Table 1.

Procedure

The diagnosis of DD was established outside the scope of this study by other experts, representatives of advisory centers and commissions with the participation of neurologists, pathologists, and psychiatrists (1 – Norm, 2 – DD). Data collection was conducted by psychologists involved in psychological and pedagogical support of children in regular and specialized preschool educational institutions (Saint-Petersburg, Murmansk, Belgorod, and others in the period from 2015 to 2017). The diagnostics was performed as a part of routine testing of children after receiving written parental consents.

The sample

In total, 628 4-year-old (575 Norm, 53 DD) and 604 5-year-old children (527 Norm, 77 DD) were surveyed.

Statistical data analysis

The 236 variables (points) for 4-year-olds and 349 variables for 5-year-olds were processed separately for children of each age (4- and 5-year-olds) to: а) identify a set of scales most accurately predicting the diagnosis (belonging to the norm/DD group), which have a simple factor structure and sufficient reliability (according to Cronbach's alpha); b) explain the relationships between predictors and the relative contribution of the scales to the prediction of the diagnosis; and c) develop an algorithm for the rapid assessment of the risk of DD based on the results of the screening application. The analysis was carried out separately, but in the same sequence, for the sample of 4- and then 5-year-olds. All statistical analysis was performed using IBM SPSS software and AMOS version 25.

Results

Selection of variables

A discriminant analysis (DA) of the variables with a stepwise method was conducted to select a set of variables that would best predict group allocation (Norm, DD). More detailed statistical analysis is presented in the article (Автор, 2018). The standard procedure for discriminant analysis was used (IBM SPSS Statistics 25), stepwise method, probability of F: entry 0.05; removal 0.1.; grouping variables: diagnosis (Norm, DD), independent: 236 variables for 4-year-olds, 349 for 5-year-olds. The result: list of variables included in the analysis at the last step of stepwise method application (56 variables for 4-year-olds and 52 for 5-year-olds).

The formation of scales

Using factor analysis (FA) and, thereafter, DA on the calculated factors, variables were selected according to the following requirements: each of the remaining variables was included in only one factor with a load of at least 0.4; each factor included at least five of these variables; calculated factors together provided the maximum accuracy of the diagnosis prediction (DA); the variables included in each factor formed a fairly reliable scale for internal consistency (Cronbach's alpha); the factor validity of the scales was verified by confirmatory factor analysis (AMOS).

For each age (4 and 5 years), a set of three factors, satisfying all the requirements, was obtained: 20 points for 4-year-olds, 19 points for 5-year-olds (6 to 7 points for each factor). The factors coincided by the type of variables united in each of them regardless of age (for 4-year-olds – simpler, for 5-year-olds – more complex): 1) "general awareness" (the breadth of the child's knowledge of the world), seven items for 4-year-olds, six items for 5-year-olds; 2) "motor skills" (sufficient development of gross- and fine-differentiated motor skills), seven items for 4-year-olds, six items for 5-year-olds; 3) "logical reasoning" (the ability to make a logical conclusion based on the comparison operation and consideration of the conditions of the task), six items for 4-year-olds, seven items for 5-year-olds. Examples of the tasks (items) are presented in Table 1.

Table 1

Task examples (items)

Age	Task
Scale (Factor) General awareness
4-year-old	Can give the names of wild animals (3–5).
	Can give the names of some cities (1–2).
5-year-old	Can navigate in a sequence of days of the week.
	Can answer the question: “How old will you be in one year?”
Scale (Factor) Motor skills
4-year-old	Can draw a person.
	Can draw a triangle by looking at the example.
5-year-old	Can cut out paper with scissors on a simple contour.
	Can navigate in a sheet of checkered paper, performing tasks according to the instructions.
Scale (Factor) Logical reasoning
4-year-old	Can distinguish objects by height.
	Can distinguish between true and false depicted in pictures.
5-year-old	Can explain what the car needs brakes for.
	Can answer the question: “What do an axe and a hammer have in common?”

Verification of reliability, relative contribution of scales and prognosis accuracy

To verify the stability of predictions with a given set of predictors, the whole sample was divided into two age groups according to the median of age (in days) of the DD groups: younger and older (median for 4-year-olds was 1,677 calendar days, median for 5-year-olds was 2,041 calendar days). The reliability of the scales (Cronbach's alpha) was separately defined for 4- and 5-year-olds and for parallel subgroups of the younger and older groups for each age. A sufficiently high reliability of each scale (from 0.779 to 0.922) was confirmed in the parallel subgroups. The scales’ values were calculated as the sum of their variables (for each age specifically). Checking the reliability of the scales (Cronbach's alpha) and the subsequent verification of the accuracy of the forecast (discriminant analysis) were carried out six times, three times for each age group (4 - and 5-year-olds): for the entire sample, for younger, and for older groups. In all six cases, the factors "logical reasoning," "motor skills," and "general awareness" made the biggest contribution to the accuracy of the diagnosis prediction (Norm/DD) in descending order: the higher their value, the stronger the probability of belonging to the norm. The sensitivity (accuracy of prediction of DD) and specificity (accuracy of prediction of norm) of the prognosis for 4-year-olds were not lower than 96.2%, and for 5-year-olds, a sensitivity not lower than 89.5%, and a specificity not lower than 97.5%. The sequence of analysis and the results for 4-year-olds are presented in more details in the article (Автор, 2018).

Structural equation modeling (SEM) was used to test the following assumptions: the selected scales are indicators of the general factor (G-factor), which is the main predictor of the diagnosis; age has an indirect impact on the diagnosis: the mediator of this effect is the G-factor. The verification was also carried six times on the samples of each age and their subgroups (younger and older).

All six models confirmed the initial assumptions on the indexes of agreement and statistical significance of parameters (Byrne, 2010); example one model is shown in Fig. 1 (for 4-year-olds). The models differed only in the statistical significance of the influence of age on the separate scales: this effect was stronger for 4-year-olds, but weaker for 5-year-olds, until the loss of statistical significance for the subgroup of older 5-year children. The fit indices for the models are presented in Table 2.

Table 2

Fit indices for models

Sample	CMIN	df	p	GFI	CFI	RMSEA	Pclose
All, 4-years old	5.500	3	.139	.995	.978	.036	.611
Age < 1677 days	1.863	2	.394	.997	1.000	.000	.651
Age >1677 days	1.439	4	.837	.996	1.000	.000	.944
All, 5-years old	3.467	3	.325	.998	1.000	.016	.772
Age <2041 days	4.135	4	.388	.995	1.000	.010	.731
Age >2041 days	6.450	5	.265	.990	.998	.034	.576

All models are almost identical and correspond well to the original data. The model for younger 4-year-olds additionally included a negative relationship between errors e1 and e4 (β = –.21), and in the model for older 4-year–olds, the relationship age - > s1 (p > .1) is not statistically significant. In the model for all 5-year–olds, the relationship age - > S3 ("logical reasoning") is not statistically reliable. For younger 5-year-olds, the influence of age on S2 ("motor skills") loses its statistical significance, and in the model for older 5-year-olds, age does not have a statistically significant effect on the other variables of the model.

Note. Numbers at arrows - standardized regression coefficients; numbers at contours of variables - squares of multiple correlation; S1 – "general awareness", S2 – "motor skills", S3 – "logical reasoning."

Development of test scales was aimed to divide groups into Norm and DD with maximum accuracy. The scales were constructed separately for 4 - and 5-year-olds. For this purpose, the results of DA were used to determine the discriminant function (DF) – an axis passing through the centroids of the separable classes (Norm, DD) and to estimate the probability of belonging to DD for each discriminant score (DS) of this function (Klecka, 1980). As a result, nonstandardized DF coefficients (separately for 4-and 5-year-olds), included in the linear equation for calculating DS by predictor values (age in days, S1, S2, S3), were extracted:

where: A –intercept; B1 – B4 – DF coefficients; i – number of the child; DS_i – its discriminant assessment; S1_i, S2_i, S3_i – values of the corresponding scales for the child.

The calculated DS for the entire sample for each age group were the raw scores to be scaled. Both DS distributions differ significantly from the normal distribution with pronounced right-side asymmetry and a long tail toward negative values. Therefore, different variants of the nonlinear transformation of the scales were compared to develop norms. The 50-point percentile scales appeared to be the most accurate, in the construction of which the entire range of raw scores (DS) was divided into 50 intervals (2% of the sample per interval), and the upper limits of DS for each interval were calculated.

The developed scales demonstrate high accuracy of the prognosis. For 4-year-olds, the upper limit of the 12th percentile (P12) at 100% sensitivity (accuracy of prediction of DD) provided 96% specificity (accuracy of prediction of norm). And for 5-year-olds, the upper limit of P16 corresponded to 94.8% sensitivity and 95.4% specificity. It is believed that good screening should have a specificity of 70% to 80%, and sensitivity in the range of 70% to 80% is considered acceptable (Glascoe, 2005). In most existing screenings, sensitivity is 82% and specificity is 78% (Tonelly, 2016), that is, much lower than the expected accuracy of the scales developed in our study.

Discussion

The constructed scales differ significantly from standard ability tests because their contents are not determined by individual differences (development vectors) of normally developing children. The procedure of elimination of the scales determined contents: it was a vector that polarized to the best extent children at risk for DD from their peers without developmental problems.

The greatest difference between groups norm and DD was obtained on the following factors, specified in descending order of contribution: "logical reasoning," "motor skills," and "general awareness." The factor "logical reasoning" was the most powerful predictor of DD. However, it should be emphasized that the prognostic ability of indicators is valid only in their totality.

Thus, the orientation of the scales is generally consistent with previous studies. For example, it was shown that the level of primary understanding of the concept of numbers in early childhood affects the development of logical reasoning in preschool age children (Clements, Sarama, 2007; Hollister Sandberg, 2010). Also, it has been shown that 8- to 12-year-olds with borderline intellectual functioning have a relatively low capacity for logical reasoning capacity and for autonomy and social responsibility in everyday activity (Baglio et al., 2016). The role of logical reasoning as a predictor of development for 4- to 5-year-old children is extremely important for Russia – in 67% of cases initially diagnosed children with DD in Russia have a speech delay (Isaev, 2017), which naturally inhibits the formation of logical reasoning. The importance of motor development at different ages of early childhood and its impact on further cognitive development has been emphasized in many studies (Farber, Beteleva, 2005; Dornelas, et al., 2016; Hernandez, Cacola, 2015; Baglio et al., 2014; Cheng et al., 2014; Mithyantha et al., 2017). It is known that at the age of 4, a typically developed child still maintains a close interaction between visual perception and motor activity, inherent at an earlier age; at this age, manual manipulations with objects are considered as an essential factor of visual identification (Farber, Beteleva, 2005). Likewise, in a longitudinal study, it was shown that a delay in motor development in 2-year-old children as the most significant marker of DD, increases the likelihood of maintaining this diagnosis during the school years (Dornelas, et al., 2016). While in another longitudinal study, it was shown that a high level of motor development during infancy determines the level of cognitive development in 4-year-olds (Hernandez, Cacola, 2015). According to the results of an MRI study of 7- to 8-year-old children with DD in comparison with the norm, the most significant differences were recorded in parts of the brain associated with motor skills, perception, and the regulation of behavior (Baglio et al., 2014). General awareness is considered as the width of a child's knowledge of the world; therefore, it is often used as a predictive marker of DD (Cheng et al., 2014; Mithyantha et al., 2017). But it should be noted that general awareness can be trained to a much greater extent than the nonverbal abilities of the child, as was demonstrated in the study of the training for the Bright Start program for young children with developmental language delays (Tzuriel et al., 2017).

The peculiarity of the scales is reflected not only in the process of their elaboration, but also in their effective application, which involves the use of computers, as focused on the fullest access to the obtained information (data on individual specifics of the tasks performance, the levels and complexities of tasks, etc., and the calendar age of a child up to the day). An attainment of the final result on the scales requires more complex calculations than is provided for in ordinary manual tests, but the widespread usage of computers by modern specialists will welcome the application of these types of scales in scientific studies and for practical assesments. Moreover, the use of computerized diagnostics optimizes the work of a specialist at all stages of the examination: an automation of the selection of the tasks and the stimulus, depending on the age of a child, and the preservation and consolidation of data processing with output, not just the numeric results, but also prior interpretations and possible recommendations.

Conclusion

The main result of this study is the elaboration of short scales for rapid (30 minutes) diagnosis (Norm/DD), allowing practitioners to quickly identify the risk group among 4- and 5-year-old children with a very high accuracy: at least 95%. It exceeds far more than existing standards for screening tests (Glascoe, 2005; Tonelly, 2016). Currently, this scale can be used on the basis of data collected by the longitud program complex (Автор, 2001). In the future, as the scales expand to other age ranges, independent software will be created that is more concise in terms of data collection and is designed only for use in screening mode to identify children at high risk of DD. However, more detailed individual psychodiagnostics will be needed to clarify the diagnosis and classify a kind of developmental delay to a certain nosological group and to determine the necessary corrective measures. Thus, the expected effect of the application of the screening is an early detection of the risk of DD to change the potential trajectory of a child's development. The desired outcome of the screening is to optimize the work of psychological services, reducing the number of children requiring a detailed individual diagnosis, which takes a lot longer than screening..

Limitations

The scales are designed to identify a high risk of delayed development in a timely manner, when the lag has not yet become apparent and an accurate nosological classification is still hindered. In this respect, the obtained results in the primary examination (screening) by these scales should be considered as a basis for refining the child’s educational route and planning further observations but not as a diagnosis. A specialist examination is always necessary for the final conclusion about the presence of DD and its particular type, regardless of the severity of the signs revealed during the first computerized assesment.

These scales are not intended for the examination of children with behavioral or other specific disorders that may affect the results of observations and the successfulness of the task performance.

Acknowledgements

The reported study was funded by the Russian Foundation for Fundamental Researches, grant № 17-06-00989-OGN “Longitudinal study of the predictive patterns of delayed psychomotor development of preschool children.”

References

Baglio, F., et al. (2014). Abnormal development of sensory-motor, visual temporal and parahippocampal cortex in children with learning disabilities and borderline intellectual functioning. Frontiers in Human Neuroscience, 8(806). https://doi.org/10.3389/fnhum.2014.00806

Baglio, G., et al. (2016). Social competence in children with borderline intellectual functioning: delayed development of theory of mind across all complexity levels. Frontiers in Psychology, 7, 1604. https://doi.org/10.3389/fpsyg.2016.01604

Barnett, S.W. (1995) Long-term effects of early childhood programs on cognitive and school outcomes. Future Child, 5, 25–50. https://doi.org/10.2307/1602366

Byrne, B. M. (2010) Structural equation modeling with AMOS: Basic concepts, applications and programming. 2nd ed. (Multivariate applications series). New York: Taylor & Francis Group.

Carroll, A.E. et al. (2014) Use of a Computerized Decision Aid for Developmental Surveillance and Screening: A Randomized Clinical Trial. JAMA Pediatr. 168(9), 815-821. https://doi.org/10.1001/jamapediatrics.2014.464

Cheng, E.R. (2014) Cognitive Delay and Behavior Problems Prior to School Age. Pediatrics, 134(3), 749-757. https://doi.org/10.1542/peds.2014-0259

Clements, D.H., Sarama, S. (2007) Early childhood mathematics learning. Second handbook of research on mathematics teaching and learning. US: Information Age Publishing.

Collins, P.Y. et al. (2017). Global services and support for children with developmental delays and disabilities: Bridging research and policy gaps. PLoS Med, 14(9), e1002393. https://doi.org/10.1371/journal.pmed.1002393

Cowan, N., & Alloway, T. (2009). Development of working memory in childhood. In M. L. Courage, & N. Cowan (Eds.), The development of memory in infancy and childhood. Hove, East Sussex, UK: Psychology Press, 303-342.

Dennis, M., et al. (2013) Age, plasticity, and homeostasis in childhood brain disorders. Neuroscience & Biobehavioral Reviews, 37, 2760-2773. https://doi.org/10.1016/j.neubiorev.2013.09.010

Dornelas, L.F., Duarte, N.M., Morales, N.M., et al. (2016) Functional Outcome of School Children With History of Global Developmental Delay. Journal of Child Neurology, 31(8), 1041-1051. https://doi.org/10.1177/0883073816636224

Engel de Abreu, P., Conway, A., & Gathercole, S. (2010). Working memory and fluid intelligence in young children. Intelligence, 38, 552-561. https://doi.org/10.1016/j.intell.2010.07.003

Farber, D. A., & Beteleva, T. G. (2005) Formation of the system of visual perception in ontogeny. Human Physiology, 31, 515–524. https://doi.org/10.1007/s10747-005-0091-3

Frick, A., & Mohring, W. (2013). Mental object rotation and motor development in 8- and 10-month-old infants. Journal of Experimental Child Psychology, 115, 708–720. https://doi.org/10.1016/j.jecp.2013.04.001

Glascoe, F.P. (2005) Standards for Screening Test Construction. Retrieved from http://www.aap.org/sections/dbpeds/pdf/Standards%20for%20Screening%20Test%20 Construction.pdf.

Hernandez, A.M., Cacola, P. (2015) Motor proficiency predicts cognitive ability in four-year-olds. European Early Childhood Education Research Journal, 23(4), 573-584. https://doi.org/10.1080/1350293X.2014.991094

Hollister Sandberg, E., McCullough, M. B. (2010) The Development of Reasoning Skills. In E. Hollister Sandberg & B.L. Spritz (Eds.), A Clinician’s Guide to Normal Cognitive Development in Childhood. New York: Routledge. 179–198.

Isaev, D.N. (2017). Psihiatrija detskogo vozrasta. Psihopatologija razvitija. [Psychiatry of childhood: psychopathology of development.] Russia, Saint-Petersburg: Spetslit.

Ivanova, A.E., Miroshnikov, S.A. (2001). Metodicheskie materialy k ekspertnoi sisteme individual'nogo soprovozhdeniya «Longityud» [Methodical Materials for the Expert System of Individual Tracking «Longitude»]. St. Petersburg: S.-Peterb. gos. universitet.

Klecka, W. R. (1980). Discriminant Analysis. Beverly Hills, California: Sage Publications https://doi.org/10.4135/9781412983938

Kok, L., et al. (2016) The effectiveness of psychosocial interventions for children with a psychiatric disorder and mild intellectual disability to borderline intellectual functioning: A systematic literature review and meta-analysis. Clinical Child Psychology and Psychiatry, 21, 156–171. https://doi.org/10.1177/1359104514567579

Lauer, J. E., & Lourenco, S. F. (2016). Spatial processing in infancy predicts both spatial and mathematical aptitude in childhood. Psychological Science, 27, 1291-1298. https://doi.org/10.1177/0956797616655977

Luiz D., et al. (2006) Griffiths Mental Development Scales, Extended Revised: 2 to 8 years. Association for Research in Infant and Child Development (ARICD), Hogrefe.

Macmann, G. M., & Barnett, D. W. (1994) Some additional lessons from the wechsler scales: A rejoinder to Kaufman and Keith. School Psychology Quarterly, 9, 223-236. https://doi.org/10.1037/h0088288

Majnemer, A., & Shevell, M. I. (1995) Diagnostic yield of the neurologic assessment of the developmentally delayed child. J Pediatr, 127, 193–199. https://doi.org/10.1016/S0022-3476(95)70294-6

Mithyantha, R., et al. (2017) Current evidence-based recommendations on investigating children with global developmental delay. Archives of Disease in Childhood,102, 1071–1076. https://doi.org/10.1136/archdischild-2016-311271

Tonelly, M. et al. (2016) Recommendations on screening for developmental delay.CMAJ, 188, 8,579-587. https://doi.org/10.1503/cmaj.151437

Tsao, F. M., Liu, H. M., & Kuhl, P. K. (2004). Speech perception in infancy predicts language development in the second year of life: A longitudinal study. Child Development, 75, 1067–1084. https://doi.org/10.1111/j.1467-8624.2004.00726.x

Tzuriel, D., et al. (2017) Effects of teaching classification on classification, verbal conceptualization, and analogical reasoning in children with developmental language delays. Journal of cognitive education and psychology, 16, 107-124. https://doi.org/10.1891/1945-8959.16.1.107

Van Duijn, G., et al. (2009) Vineland Screener 0-12 years research version (NL). Constructing a screening instrument to assess adaptive behavior. Int J Methods Psychiatr Res., 18(2), 110-7. https://doi.org/10.1002/mpr.282

Voelke, A., & Roebers, C. (2016). Sustained attention and its relationship to fluid intelligence and working memory in children. Journal of Educational and Developmental Psychology, 6, 131-139. https://doi.org/10.5539/jedp.v6n1p131

Wellman, H. M., et al. (2008). Infant attention to intentional action predicts preschool theory of mind. Developmental Psychology, 44, 618–623. https://doi.org/10.1037/0012-1649.44.2.618