Eye movement parameters while reading show cognitive processes of structural analysis of written speech

Chernorizov, Alexsander M. Lomonosov Moscow State University, Moscow, Russia

Latanov, A.V. Lomonosov Moscow State University, Moscow, Russia

Anisimov, V.N. Lomonosov Moscow State University, Moscow, Russia

Language and linguistic activity, taken in the whole variety of their types (speech, writing, listening, and reading) and being among the most complex forms of higher cognitive functions, have a number of complex physiological and cognitive mecha­nisms. Consequently, they are the subject of several disciplines, including physio­logy, psychology and psycholinguistics (Luria, 1975; Leontiev, 1997; Baars, 2007). The founder of generative grammar theory, N. Chomsky, laid down the theoretical foundations for studying cognitive difficulties in communication connected with linguistic form (or verbal expression). A special type of this broader issue is the prob­lem of the origin of and ways of disambiguating syntactic ambiguity in language.

Reading is a highly complicated linguistic activity, for it is not only a perceptual and a motor but also a cognitive process. At each moment while reading, incoming visual information is continuously recognized and eye movements are prepared and executed; each of these is made up of different stages. The process of reading involves a set of sensory, motor, and mental processes (attention, memory, pattern recognition, decision making) related to the processing of visually perceived words. While reading, the eyes make a series of rapid movements (called saccades), sepa­rated by periods of time in which the eyes are relatively still (called fixations). It is only during fixations that the new visual information is encoded from the text, be­cause vision is functionally suppressed during saccades. The process of recognizing signs that comprise morphemes and words occurs during fixations only. As a result, the visual properties of the letters are converted to orthographic and phonological patterns that are further processed into linguistic information that eventually leads to understanding of the text.

The anatomical and functional substrate of these processes constitutes hierarchi­cally organized wide areas of the frontal, temporal, and parietal lobes (Luria, 1975; Vigneau et al., 2006). The contemporary framework for understanding the mecha­nisms and localization of speech functions is based on neuropsychological (Luria, 1975; Ellis, 2012) and neurophysiological data obtained with functional magnetic resonance imaging (fMRI) (Hagoort, 2005; Hasson & Small, 2008) and evoked po­tentials (Kutas & Hillyard, 1984; Hagoort et al., 2004; Steinhauer & Connolly, 2008). But neurophysiological methods impose certain limitations on experimental study of reading, since they require that eye movements be eliminated from investigation. Eye movements produce significant artifacts and thereby mask the neuronal signals of the actual speech process. In real life, reading is carried out with the help of eye movements, and in such conditions the processes of motor control are inevitably su­perimposed on the speech analysis processes, which makes it impossible to separate these effects. However, the parameters of saccades and fixations indirectly represent the brain’s linguistic processes because they are closely related to the cognitive func­tions (attention, memory, letter pattern recognition, decision making) that achieve recognition and comprehension of written language (Clifton et al., 2007). The un­deniable advantage of recording eye movement during reading consists in the pos­sibility of tracking the gaze position in real time; therefore, for several decades, eye movement parameters have been widely used to study linguistic processes (Rayner, 1998; Underwood, 2005; Clifton et al., 2007; Rayner et al., 2012).

The recording of eye movements makes it possible to determine the time it takes to read various parts of a text. When associating the time spent on a par­ticular phrase with its linguistic features, it is possible to draw certain conclusions about the physiological and mental processes involved in recognition and compre­hension this phrase. The gaze position points at the phrase currently being pro­cessed, and the time spent on it manifests the involvement of mental activity. These assumptions have been confirmed by many experiments showing that the linguistic peculiarities of the phrase determine the time spent to read it (Underwood, 2005; Rayner et al., 2006, 2012; Clifton et al., 2007; Rayner, 2009). Thus, eye movement parameters and the mental processes associated with analyzing the currently per­ceived and processed phrase are assumed to be related to each other, as is supported by a large body of experimental data and experience in psycholinguistic studies (Underwood, 2005; Rayner et al., 2006; Clifton et al., 2007; Rayner, 2009).

The effects of lexical ambiguity on eye movements during reading

Studies in the English language demonstrate that fixation duration while reading certain phrases of a sentence and consequently the total reading time for some par­ticular phrase depend on the lexical properties of the words that present special dif­ficulty for interpretation (Rayner et al., 1977, 2006; Rayner & duffy, 1986; Clifton et al., 2007; Reichle et al., 2007). For example, while reading the most frequently used nouns in the English vocabulary (more than 100 words/million in the English corpus dictionary, the CELEX English lexical database), fixations are shorter by 25–40 ms than fixations while reading the rare nouns (less than 10 words/million) (Rayner & duffy, 1986; Rayner et al., 2006; Clifton et al., 2007). Semantic properties of the words (e.g., predictability) also affect the fixation durations. While reading highly plausible words they become shorter by 40 ms and more than fixation dura­tions while reading less plausible words (Rayner et al., 2006; Clifton et al., 2007). Moreover, fixation durations significantly increase when reading semantic anoma­lies, misspellings, and pseudo-words, compared to the fixation duration of the ex­pected words (Rayner et al., 2006; Clifton et al., 2007). Processing a polysemantic word (lexical ambiguity) also increases reading time (Rayner & duffy, 1986; Clifton et al., 2007). These results imply that when reading unusual phrases, their linguis­tic analysis is slowed down in terms of various parameters, because additional mental processes are involved. Higher attention activation and increased load on mnemonic processes inhibit decision making. These processes are represented by electrical activity of the brain such as evoked potentials (Kutas & Hillyard, 1984; Hagoort et al., 2004; Steinhauer & Connolly, 2008) and are correlated with the he­modynamics of certain brain areas as shown by fMRI studies (Hagoort, 2005; Has-son, Small, 2008).

Eye movements in syntactic analysis of locally ambiguous sentences during reading

Structural features and syntactic variables of the sentences also influence eye move­ment parameters (Rayner, 1998; Clifton et al., 2007), as reported in a substantial number of articles (for a review, see Clifton et al., 2007). The number of fixations, their durations, saccadic amplitudes, and the frequency of regressive saccades (re­gressions) that are executed during the second path of reading certain phrases, are associated with certain difficulties in interpreting the sentence structure and, there­fore, comprehension of the text. In general, different types of syntactic ambiguities slow down the reading of ambiguous sentences compared to unambiguous sen­tences in different languages (English, Spanish, German, French, and some others) (Frazier & Rayner, 1982; Cuetos & Mitchell, 1988; Schriefers et al., 1995; Frenck-Mestre & Pynte, 1997; Traxler et al., 2002; Staub, 2010).

Two forms of syntactic ambiguity are differentiated in linguistics: local and global ambiguity. Local ambiguity is temporary until it is resolved after reading the phrase following the critical one; this leads to one distinct interpretation of the processed sentence. The globally ambiguous sentence has two equally possible interpretations. However, most studies of eye movements while reading focus on disambiguation of local syntactic ambiguity (Frazier & Rayner, 1982; Cuetos & Mitchell, 1988; Traxler et al., 2002; Staub, 2010).

Different eye movement parameters clearly represent this disambiguation pro­cess. This means that the time for reading the critical words in locally ambiguous sentences is longer than the time for reading functionally analogous word com­binations of unambiguous sentences (Frazier & Rayner, 1982; Cuetos & Mitchell, 1988; Staub, 2010). Consider the example of local ambiguity that was discussed in the pioneering work by Frazier and Rayner (1982, p. 179). While reading a clause of the sentence Since Jay always jogs a mile (Figure 1) the noun a mile is ambiguous. It could be defined as part of either the first or the second clause, and the reader can’t choose the model of interpretation. Sentences 1a and 1b give the possible interpretations in a more explicit manner. In sentence 1a, the demonstra­tive pronoun this at the beginning of the second part of the sentence forces the reader to define the word a mile as the direct object of the verb jogs. In sentence 1b, a mile is the subject of the verb seems in the second clause and the sentence has another interpretation. The authors gave the subjects 16 pairs of structurally similar sentences to read.

Figure 1. Examples of two sentences with local syntactic ambiguity (Frazier & Rayner, 1982). The numbers above words indicate the average fixation durations (ms) during reading them (ibid., Table 4, p.103). The arrows indicate the regressions for rereading critical words and the numbers nearby indicate frequency of regressions of the total num­ber of sentences (ibid., Table 9, p. 199).

In English, the comprehension of syntactically ambiguous sentences is driven by two very general parsing strategies: late closure (LC) and minimal attachment (Frazier & Rayner, 1982). These strategies are defined as follows:

“Late closure: When possible, attach the incoming lexical items into the clause or phrase currently being processed,” i.e., the lowest possible nonterminal node dominating the last item analyzed.

“Minimal attachment: Attach the incoming material into the phrase-marker be­ing constructed using the fewest nodes consistent with the well-formedness rules of the language,” i.e., the principle of least links (“shortest path”) between the words.

LC strategy implies that in the phrase jogs a mile (Figure 1, sentence 1a) the item currently being processed, jogs, is added to the direct object, a mile. In other words, in this case the lowest possible nonterminal node is established; it links the item — jogs — and the latest item analyzed — a mile. Sentence 1b is an example of an ambiguous sentence with early closure (EC) or high attachment, which is in clear cor­relation with the already formed patterns, typical for the language being studied.

Fixations during reading the critical words (i.e., disambiguating the local am­biguity) this (1a) and seems (1b) were found to be on average longer than fixations on the surrounding words. Thus the fixation duration in reading the item this (268 ms), solving the target ambiguity in 1a with LC, is shorter than the average fixa­tion duration on the item seems (283 ms), solving the ambiguity in 1b with EC. In addition, the readers sometimes execute regressions to reread some phrases of the sentences. These saccades are often (about 2/3 of their total number) executed by the subjects to return to the critical words this (1a) and seems (1b) in order to verify interpretation of the sentence. It should be noted that regressions are less frequent when reading sentences with LC (0.30), the dominant closure pattern in English, than with EC (0.38).

The increased fixation durations while reading the critical words as described by Frazier and Rayner (1982) and supported by other studies (Cuetos & Mitchell, 1988; Staub, 2010; Clifton et al., 2007) are connected with an additional cognitive load, voluntary attention and working memory being involved with interpreting the whole sentence. At the same time, the execution of regressions and additional fixations for rereading an ambiguous phrase signify a reanalysis of the primary sentence interpretation.

The first comparative study of reading similar sentences with local ambiguity was conducted in English and Spanish with native speakers of both languages (Car­reiras & Clifton, 1999). However, the eye movement parameters for these languages turned out to be opposite. Fixation durations in reading the critical words in Eng­lish were shown to rise when subjects were reading sentences with EC, whereas in Spanish this phenomenon was shown for reading sentences with LC. These results suggest that in Spanish, unlike English, the principle of attachment of lexical units in resolving local ambiguity dominates, namely EC or high attachment.

Eye movements in syntactic analysis of globally ambiguous sentences during reading

Eye movement parameters in disambiguating global syntactic ambiguity have only been investigated in a few studies, still mostly in English (Traxler et al., 1998; Van Gompel et al., 2001, 2005). A special case of global ambiguity is complex sentenc­es with relative clauses, as in the well-known example: Someone shot the servant of the actress who was on the balcony (its equivalent in Russian: Некто застрелил служанку актрисы, которая стояла на балконе). In giving the answer to the question Who stood on the balcony? the English speakers often chose the second noun (the actress) (Cuetos & Mitchell, 1988; Carreiras & Clifton, 1999; Staub, 2010), while Russian speakers chose the first noun (the servant) more often (Fedorova & Yanovich, 2004). Otherwise stated, the English readers still prefer the LC strategy when interpreting sentences with global ambiguity, as was the case in sentences with local ambiguity (Frazier & Rayner, 1982). Apart from English, the LC strategy also prevails in Portuguese (Brazil), Norwegian, Swedish, Finnish, Arabic, and Romanian (Fodor, 1998). However, in other languages, including Russian the opposite trend is manifested. The EC strategy is preferred in French, Greek, Japanese, Spanish, dutch, German, Polish, Croatian, and Bulgarian (Cuetos & Mitchell, 1988; Sekerina, 1997; Fodor, 1998; Carreiras & Clifton, 1999; Fedorova & Yanovich, 2004).

Traxler and coauthors (Traxler et al., 1998; Van Gompel et al., 2001, 2005) noted that the reading time for complex sentences with relative clauses containing global syntactic ambiguity is shorter compared with local syntactic ambiguity, but is still somewhat longer than the reading time of sentences without any ambiguity. Based on the their results, Traxler and coauthors put forward the hypothesis that global syntactic ambiguity does not induce significant difficulties in comprehending the sentence and its further interpretation is likely to be predetermined. difficulty in analyzing sentences with global ambiguity as well as those with local ambiguity involves additional mental effort and, consequently, increases the reading time.

Until recently there were no studies of eye movement parameters while reading sentences with global syntactic ambiguity in Slavonic languages. The first study in Russian (Anisimov et al., 2013, 2014) with the original high frequency (250 Hz) eye tracker (Anisimov et al., 2012) was carried out by the authors of the present paper. In our experiments, 28 subjects read 40 complex sentences with feminine relative clauses with global syntactic ambiguity (Figure 2, sentence 2a). To verify our findings, we included 40 similar control sentences, in a random order, without any ambiguity being given to subjects (Figure 2, sentence 2b).

Figure 2. Two sentences with global syntactic ambiguity (Anisimov et al., 2013, 2014). All sentences were presented in three lines, the second line (in 2a) containing the global syntactic ambiguity.

We compared eye movement parameters while reading the second line of the syn­tactically ambiguous sentences (2a) and the control sentences with only one inter­pretation (2b). All parameters were averaged over all subjects and all sentences. We found significant differences between eye movement parameters while reading ambiguous and control sentences (Table 1). We found a similar effect of ambiguity on all eye movement parameters, as was previously reported for other languages.

As already mentioned, the EC principle dominates in Russian, particularly in resolving ambiguity of a sentence with a relative clause (Fodor, 1998; Fedorova & Yanovich, 2004). In our study, the average proportion of EC for all sentences was

0.67. A similar proportion of EC was also reported for some other languages (Cue­tos & Mitchell, 1988; Sekerina, 1997; Fodor, 1998; Carreiras & Clifton, 1999; Fedo­rova & Yanovich, 2004). According to the studies of global syntactic ambiguity dis­ambiguation in English, the proportion of LC is the same 0.67 (Frazier & Rayner, 1982, Carreiras & Clifton, 1999; Clifton et al., 2007). table 1. Eye movement parameters in reading second lines in ambiguous and unambiguous sentences (Anisimov et al., 2014).

The data averages (M ± SEM) were calculated for all subjects and all sentences. The size of the sample is given in parentheses. The influence of the type of the sen­tence with ambiguity levels “ambiguous” and “unambiguous” on all parameters (at the significance level p) was evaluated by the ANOVA repeated measures method.

Our results open the perspective of objective comparative study of linguistic processes and their mental counterparts, particularly ways to resolve syntactic am­biguity in Slavonic and Romano-Germanic languages.

Acknowledgments

This research was supported by RFBR grant No. 15-04-05745.

References

Anisimov, V.N., Fedorova, O.V., & Latanov, A.V. (2013). Parametry dvizheniia glaz kak marker razresheniia globalnoi sintakticheskoi neodnoznachnosti v russkom yazyke [Eye movement parameters as a marker of the disambiguation of global syntactic ambiguity in Russian]. Kognitivnaia nauka v Moskve: novye issledovaniia. Materialy konferentsii [Cognitive scienc­es in Moscow: New studies. Abstracts] (pp. 24–28). Moscow: OOO Buki Vedi.

Anisimov, V.A., Fedorova, O.V., & Latanov A.V. (2014). Eye movement parameters in reading sentences with syntactic ambiguities in Russian. Human Physiology, 40(5), 521–531. doi: 10.1134/S0362119714040033.

Anisimov, V.N., Ermachenko, N.S., Ermachenko, A.A., Tereschenko, L.V., & Latanov, A.V. (2012). Eksperimentalnyi kompleks dla odnovremennoi registratsii dvizhenii glaz i elek­troentsefaligrammy [Experimental setup for synchronous recording of eye movements and EEG]. Izvestiia SFedU [Southern Federal University Bulletin]. Engineering Sciences, 11(136), 116–120.

Baars, B.J. (2010). Language. In B.J. Baars, & N.M. Gage (Eds.), Cognition, brain, and conscious­ness. Introduction to cognitive neuroscience (pp. 371–396). Amsterdam, London, NY: Aca­demic Press. doi: 10.1016/B978-0-12-375070-9.00011-5

Carreiras, M., & Clifton, C. (1999). Another word on parsing relative clauses: Eyetracking evidence from Spanish and English. Memory & Cognition

Chomsky, N. (1962). Sintakticheskie struktury [Syntactic structures]. In V.A. Zvegintsev (Ed.), Novoe v lingvistike [New in linguistics] (Vol. 2, pp. 412–527). Мoscow: Foreign Literature Publishing.

Chomsky, N. (1972). Yazyk i myshlenie [Language and mind]. Moscow: Moscow University Press.

Clifton Jr., C., Staub, A., & Rayner, K. (2007). Eye movements in reading words and sentenc­es. In R.P.G. Van Gompel, M.H. Fischer, W.S. Murray, & R.L. Hill (Eds.), Eye movements: A window on mind and brain (pp. 341–371). Oxford, UK: Elsevier. doi:10.1016/B978­008044980-7/50017-3

Cuetos, F., & Mitchell, D. (1988). Cross-linguistic differences in parsing: Restrictions on the use of the late closure strategy in Spanish. Cognition, 30(1), 73–105. doi:10.1016/0010-0277­(88)90004-2

Ellis, A.W. (2012).The acquisition, retention, and loss of vocabulary in aphasia, dementia, and other neuropsychological conditions. In M. Faust (Ed.), The handbook of the neuropsy­chology of language (Part 2, pp. 635–660). Oxford, UK: Wiley-Blackwell. doi:10.1111/ b.9781444330403.2012.00035.x

Fedorova, O.V., & Yanovich, I.S. (2004). Ob odnom tipe sintakticheskoi mnogoznachnosti, ili Kto stoyal na balkone [One type of syntactic ambiguity, or who stood on the balcony]. In Komputernaia lingvistika i intellektualnye tekhnologii: Trudy mezhdunarodnoi konferentsii ‘Dialog 2004’ [Computer linguistics and intellectual technologies: Abstracts of the interna­tional conference ‘dialog 2004’] (pp. 644–649). Moscow: Nauka.

Fodor, J.D. (1998). Learning to parse? Journal of Psycholinguistic Research, 27(2), 285–319. doi: 10.1023/A:1023258301588

Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178–210. doi: 10.1016/0010-0285(82)90008-1

Frenck-Mestre, C., & Pynte, J. (1997). Syntactic ambiguity resolution while reading in second and native languages. The Quarterly Journal of Experimental Psychology, 50A(1), 119–148. doi: 10.1080/027249897392251

Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cognitive Sci­ence, 9(9), 416–423. doi: 10.1016/j.tics.2005.07.004

Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. (2004). Integration of word meaning and world knowledge in language comprehension. Science, 304(5669), 438–441. doi: 10.1126/ science.1095455

Hasson, U., & Small, S.L. (2008). Functional magnetic resonance imaging (fMRI) research of language. In B. Stemmer & H.A. Whitaker (Eds.). Handbook of the neuroscience of language (pp. 81–89). Amsterdam, Netherlands: Elsevier. doi: 10.1016/B978-0-08-045352-1.00008-2

Kutas, M., & Hillyard, S. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307(5947), 161–163. doi: 10.1038/307161a0

Leontiev, A.A. (1997). Osnovy psikholingvistiki [Fundamentals of psycholinguistics]. Moscow: Smysl.

Luria, A.R. (1975). Osnovnye problemy neirolingvistiki [Basic problems of neurolinguistics]. Moscow: Moscow State University Publishing.

Rayner, K. (1977). Visual attention in reading: Eye movements reflect cognitive processes. Mem­ory & Cognition, 5(4), 443–448. doi: 10.3758/BF03197383

Rayner, K. (2009). Eye movements and attention in reading, scene perception and visual search. The Quarterly Journal of Experimental Psychology, 62(8), 1457–1506. doi: 10.1080/ 17470210902816461

Rayner K., Pollatsek A., Ashby J., & Clifton Jr., C. (2012). The psychology of reading (2nd ed). NY: Psychology Press. doi: 10.1002/9781118133880.hop204019

Rayner, K. (1998) Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372–422. doi: 10.1037//0033-2909.124.3.372

Rayner, K. & Duffy, S. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 24(3), 191-201. doi: 10.3758/BF03197692

Rayner, K., Reichle, E., Stroud, M., Williams, C., & Pollatsek, A. (2006). The effect of word fre­quency, word predictability, and font difficulty on the eye movements of young and older readers. Psychology and Aging, 21(3), 448–465. doi: 10.1037/0882-7974.21.3.448

Reichle, E. d., Pollatsek, A., & Rayner, K. (2007). Modeling the effects of lexical ambiguity on eye movements during reading. In R.P.G. Van Gompel, M.F. Fischer, W.S. Murray, & R.L. Hill (Eds.), Eye movements: A window on mind and brain (pp. 271–292). Oxford, England: Elsevier. doi:10.1016/B978-008044980-7/50014-8

Schriefers, H., Friederici, A., & Kuhn, K. (1995). The processing of locally ambiguous rela­tive clauses in German. Journal of Memory and Language, 34(4), 499–520. doi: 10.1006/ jmla.1995.1023

Sekerina, I. (1997). The late closure principle vs. the balance principle: Evidence from on-line processing of ambiguous Russian sentences. In Proceedings of the second European confer­ence on formal description of Slavic languages. Current approaches to formal Slavic languages. Potsdam Meeting (pp. 205–217). Frankfurt: Peter Lang.

Staub, A. (2010). Eye movements and processing difficulty in object relative clauses. Cognition, 116(1), 71–86. doi: 10.1016/j.cognition.2010.04.002

Steinhauer, K., & Connolly, J.F. (2008). Event-related potentials in the study of language. In B. Stemmer & H.A. Whitaker (Eds.). Handbook of the neuroscience of language (pp. 91–104). Amsterdam: Elsevier. doi: 10.1016/B978-0-08-045352-1.00009-4

Traxler, M., Pickering, M., & Clifton, C. (1998). Adjunct attachment is not a form of lexical ambiguity resolution. Journal of Memory and Language, 39(4), 558–592. doi:10.1006/ jmla.1998.2600

Traxler, M.J., Morris, R.K., & Seely, R.E. (2002). Processing subject and object relative claus­es: Evidence from eye movements. Journal of Memory and Language, 47(1), 69–90. doi: 10.1006/jmla.2001.2836

Underwood, G. (2005). Cognitive processes in eye guidance. Oxford, UK: Oxford University Press. doi: 10.1093/acprof:oso/9780198566816.001.0001

Van Gompel, R., Pickering, M., & Traxler, M. (2001). Reanalysis in sentence processing: Evi­dence against current constraint-based and two-stage models. Journal of Memory and Lan­guage, 45(2), 225–258. doi:10.1006/jmla.2001.2773

Van Gompel, R., Pickering, M., Pearson, J., & Liversedge S. (2005). Evidence against competition during syntactic ambiguity resolution. Journal of Memory and Language, 52(2), 284–307. doi:10.1016/j.jml.2004.11.003

Vigneau, M., Beaucousin, V., Hervé, P., duffau, H., Crivello, F., Houde, O., Mazoyer, B., & Tzourio-Mazoyera, N. (2006). Meta-analyzing left hemisphere language areas: Phono­logy, semantics, and sentence processing. Neuroimage, 30(4), 1414–1432. doi: 10.1016/j. neuroimage.2005.11.002