Structural dialectical approach in psychology: problems and research results.
In this article dialectical thinking is regarded as one of the central cognitive processes. Because of this cognitive function we can analyze the development of processes and objects. It also determines the possibilities for the creative transformation of some content and for solving problems.
The article presents a description and the results of experimental studies. This evidence proves that dialectical thinking is a specific line of cognitive development in children and adults. This line can degrade during school time if the educational program follows formal logical principles, or it can become significantly stronger if the pedagogy is based on dialectical methodology.
Keywords: structural-dialectical approach, dialectical thinking, dialectical tasks, strategy of mental handling of oppositions
The structural-dialectical approach in psychology, which was developed by a team led by Nikolay Veraksa, is based on dialectical logic. Dialectical logic regards any object as developing, as constantly being in a dynamical context. “Dialectness” is an essential characteristic of a developing object; in other words, a developing object constitutes a unity of oppositions. This feature of complex objects had been defined in a most strict way by Hegel’s logic (1970) and has been confirmed many times by scientific research. In the case of a developing object it is correct to discuss not the contradiction but the special type of relations of oppositions that allow them to exclude and to presuppose each other at the same time.
As dialectical thinking is a form of mental process different from formal logical thinking, its researchers face a contradiction. They emphasize the absolute dialectness of nature and society and see inalterability as secondary, as a single moment within the constant changing of objects and phenomena. They also understand the importance of dialectical thinking as a cognitive activity assigned to reflect this dialectness. Yet, although they acknowledge dialectical thinking to be the supreme form of thinking, it appears relatively late in a person’s development. The contradiction lies in the fact that if children come across dialectics of things in the world and dialectical thinking appears much later in their development process, some other form of thinking must provide them with the ability to reflect reality adequately; such thinking is not yet dialectical thinking (because it is supreme), but still has to be dialectical (as it’s oriented on the dialectics of things). This contradiction was explored in the works of J. Piaget, who distinguished two kinds of dialectics — “elementary” and “supreme.”
Therefore the problem can be set as follows: Are there really two kinds of dialectical thinking—“elementary” and “supreme theoretical,” or is there only one kind, whose structures are similar on all levels and whose distinctions are determined by the content?
We proceed from the understanding that dialectical thinking is a special form of cognitive activity and that it is different from the traditional way of thinking and has a certain unity on all levels of representation. On the basis of this understanding only, one can speculate that dialectical thinking, as an independent line in cognitive development, can be ascertained in the mental processes not only of children but also of adults. The unity of dialectical thinking at all ages originates from the dialectical nature of strategies for the transformation of situations whose essential characteristic is the determination of the relations of opposition. Along with the unity of the mechanism of dialectical thinking, certain distinctions are presupposed by the specific features of the cognitive means implemented by a person while in the process of thinking. This difference in means in its turn depends on the activity undertaken by the child or adult (Shiyan, 2011a).
Within the framework of the structural-dialectical approach, dialectical thinking is regarded as creative, productive thinking (Bayanova, 2013; Krasheninnikov, 2008; Shiyan, 1999; Veraksa, 1990, 2007). A person using dialectical thinking can see any object, process, or phenomena as content in which significant oppositions are represented (Shiyan, 2009). Therefore, this content can be transformed dialectically; for example, a resolution of the opposition can be found, or a new situation can be constructed that is opposite to the initial one. Transformations are performed through the following dialectical acts (Krasheninnikov, 2005):
- Dialectical transformation. For any objects, ideas, phenomena, situations, a person finds oppositional ones.
- Dialectical integration. In the structure of any object or situation a person establishes the presence of oppositions that mutually disaffirm each other. There can be several pairs of this kind, and each of them characterizes prominent features of the object.
- Dialectical mediation. For any pair of oppositions a person finds or constructs an object in which these oppositions are present simultaneously.
- Dialectical seriation. A person regards any object or situation as intermediate between the initial condition and the final condition, which is the opposite of the initial one.
- Dialectical transaction. While exploring a process, a person can regard it in reverse order: that which was initially considered as an end of the process is regarded as its beginning, and the initial condition is understood as the final one.
- Dialectical change of alternative. A person regards an object in the context of one pair of oppositions and in the context of another pair.
- Dialectical identification. A person first sees objects and phenomena as op- positional and then establishes their identity and similarity.
- Dialectical dis-identification. The things initially seen as identical now are understood as oppositional.
Dialectical cognitive activities reveal the process of the transformation of objects and situations. In fact, they describe the space of the possible transformations of an object. In this sense, dialectical logic appears to be the logic of the creation of new opportunities through transition from one kind of opposition constituting an object or situation to another kind by means of developing the operations of dialectical thinking. Thus any object becomes a fragment of a structure in the space of opportunities. The object turns up as an entity and as a part. The object is an entity in regard to the substructures on which it is based as a cycle of opportunities. And the object can be seen as a part in relation to higher-order cycles. In other words, the object becomes a “single point,” “a knot” of the realization of various opportunities. The structure of an object is determined by the relations of opposition among different features and links existing around it. The object is, so to say, interlaced into various structures of relations of opposition. While exploring it, a researcher finds himself or herself at one point in the space of opportunities. An analysis based on the logic of opportunities is actually the movement in this space of opportunities, of this very object. Accordingly, an analysis of the object as an entity is connected to the establishment of a cyclic structure within the object, which is the object itself. By reference to the structural-dialectical method, this cyclic structure consists of oppositions. That is why the problem of the analysis of the object as an entity is traced to distinguishing those oppositions above all. Put the other way around, within this object there is a multitude of other oppositional pairs and, therefore, other cycles. So it is crucial to learn how to pass from one pair of oppositions to another. This transition is performed through analyzing the features of the object and is tightly connected to the implementation of dialectical cognitive acts.
Thus, when we discuss the implementation of the structural-dialectical method, we understand it as an intentional “development” of a dialectical cycle in an object that seemed initially static (and therefore, maybe, problematic). It is important to emphasize that when we speak about intentional and not spontaneous implementation of the operations of dialectical thinking, these operations still remain possible vectors of structural development, common principles, but they are not an exhaustive manual of instruction for the “production” of creative things.
A description and the results of experimental studies can be found below. Taken together, this evidence proves that dialectical thinking is a specific line of cognitive development in children and adults. This line can degrade during school time if the educational program follows formal logical principles, or it can strengthen significantly if the pedagogy is based on dialectical methodology.
Development of a Dialectical strategy for the Mental handling of oppositions in children and adults
The development of dialectical thinking is a complex, multilevel process deter- mined by the specifics of the educational situations occurring within the cognitive activity of a person.
We performed a benchmarking study of the age-related specifics of the development of dialectical acts and different strategies for the operational handling of oppositions within the life cycle. We implemented diagnostics of the structures of dialectical thinking for four age groups: preschool children, elementary school students, senior school students, and adults. The experimental part of the research was conducted in two Moscow kindergartens—#1602 (the instructional program in this kindergarten is “dialectics”) and #1511 (the instructional program in this kindergarten is “development”)—and two universities (students in the evening course of the faculty of pedagogics and psychology at Moscow State Pedagogical University and students in the evening course of the faculty of social psychology at Moscow State University of Psychology and Education). In total 232 persons participated in the research.
The technique called “What cannot be simultaneously?” is an authorial modification of the technique “What can be simultaneously?” developed by Veraksa (1987, 1990, 2007, 2009, 2010, 2011). This method consists of two parts. The first part is dedicated to examining the respondent’s ability to distinguish oppositions from the whole variety of an object’s characteristics; and the second part is designed to explore strategies for the handling of oppositions by people of different ages in a situation that has been binary-structured in advance.
The method called “an unusual tree” is widely implemented in most research conducted within the framework of the structural-dialectical approach. It was also developed by Veraksa (1987, 1990, 2009, 2011). This technique allows us to explore dialectical transaction as an operation of dialectical thinking. It helps us trace the sequence of using the means of formal and dialectical logics within the process of solving an unstructured creative problem.
Parameters for Analysis
We chose the following parameters for the analysis of the age-related specifics of the mental handling of oppositions (Belolutskaya, 2011):
- The ability to independently find and specify the oppositions hidden within a variety of features.
- The share of refuses-to-operate-oppositions in the whole mass of responses. This parameter characterizes the extent to which human thinking rigidly obeys the formal logical rule of noncontradiction.
- The average share of dialectical solutions per group. This evidence reflects the extent in which respondents of different ages are willing to implement the operations of dialectical mediation and transaction.
- The interconnection of the ability to distinguish oppositions individually and the implementation of the strategy of dialectical mediation (group evidence and individual diagnostics data). Examination of this parameter allows us to answer the question (frequently asked by those who criticize the structural-dialectical approach) of whether people (including children) really transform the contradiction and do not just make up a chaotic “heap” of all possible characteristics. In other words, do they really understand the handling of oppositions?
- The interconnection of mediation and transaction as dialectical acts. Analysis of this parameter contributes to the study of dialectical thinking as a system mechanism that begins to be established at preschool age.
- Statistically significant differences between the groups of respondents who are the same age but are studying in different instructional programs; the difference is based on the criterion of the frequency of applying the strategy of handling oppositions. Once we analyze this parameter we can ascertain and describe those features of the structure of the concept of contradiction that depend strictly on age and those that can be controlled and affected by varying the educational conditions.
Specifics of the implementation of the Dialectical strategy of handling oppositions by Respondents of Different ages
1. Ability to Find Oppositions Individually Within a Variety of Characteristics
The ability to distinguish oppositions significantly and steadily increases during a lifetime, even though it does not reach the 100% level in adults’ thinking.
We found a significant difference on this parameter between the two groups of preschool children (a 40% success rate for the students from kindergarten #1602 and a 71.6% success rate for the students from kindergarten #1511). This difference can be explained by the characteristics of the educational program and the high sensitivity of dialectical mental structures to the impact of the pedagogy employed.
2. Share of Refuses-to-Operate-Oppositions in the Whole Mass of Responses
The principles of the formal logics dominate and reliably show up in the thinking of a person of any age: more than 50% of the proposed solutions belonged to the formal logics. The other half of the solutions were unequally distributed among four false strategies for handling oppositions and implementing a productive dialectical strategy.
Only one group was significantly distinct from the others, with 75% refuses: the students from kindergarten #1511.
3. Average Share of Dialectical Solutions per Group
Within the framework of our research we considered responses “dialectical” if they were made up through implementation of the operation of dialectical mediation (the “What can be simultaneously?” technique) and the operation of dialectical transaction (the “an unusual tree” method).
In calculating the share of dialectical responses we considered the number of positive responses as 100%; in other words, the cases in which respondents refused to operate oppositions were not counted. If we had considered the total number of responses as 100%, the shares would be approximately twice as low. Therefore, if we regard the distribution of the average number of responses, one half or a bit more are formal logical solutions (refuses), 7% to 10% are dialectical, and 30% to 40% are mistakes of different kinds.
The additional results of diagnostic are presented in tables 2–3.
Statistically significant differences are underlined.
One might come to the following conclusions based on the evidence presented in tables 1, 2, 3:
- Respondents of any age demonstrate the implementation of dialectical logic very rarely: five to six times less than those implementing the formal logics (refuses-to-operate-the-oppositions) and three to four times less than those making mistakes of all kinds.
- Preschool children and adults produce a significantly higher share of dialectical solutions than do the elementary and senior school students.
- The results for preschool children and adults do not seriously differ.
- The lowest share of dialectical solutions was offered by the elementary school students.
- The results for the elementary and senior school students do not significantly differ.
Table 1. Implementation of the Operation of Dialectical Mediation
Implementation of the operation
of dialectical mediation (%)
|Preschool, kindergarten #1602||16.31|
|Preschool, kindergarten #1511||5.52|
|Elementary school #1716||16.63|
|Elementary school #689||13.91|
|Senior school #1716||15.31|
|Senior school #689||12.72|
Table 2. Significance of the Distinctions Between the Groups Under the Parameter of Dialectical Mediation (χ2-Pearson Criterion)
|Preschool, kindergarten #1602||#||4.74||2.466||29.44||0.47||0.13||2.39|
|Preschool, kindergarten #1511||#||#||8.35||3.62||9.95||5.01||6.31|
|Elem. school #1716||#||#||#||0.73||0.53||7.018||0.19|
|Elem. school #689||#||#||#||#||0.66||0.66||0.66|
|Senior school #1716||#||#||#||#||#||0.19||0.67|
|Senior school #689||#||#||#||#||#||#||5.52|
Table 3. Implementation of the Operation of Dialectical Transaction
|Preschool, kindergarten #1602||8.31|
|Preschool, kindergarten #1511||1.34|
|Elementary school #1716||1.72|
|Elementary school #689||1.61|
|Senior school #1716||10|
|Senior school #689||3.41|
The preschool children from kindergarten #1511 stand out from the general trend because of the obvious influence of the educational program.
4. Interconnection of the Ability to Distinguish Oppositions Individually and the Implementation of the Strategy of Dialectical Mediation
We examined this parameter through determination of a correlation between the results of Part 1 and Part 2 of the “What can be simultaneously?” technique.
There was a significant correlation between these parameters in the groups of preschool age only (0.62 in kindergarten #1602 and -0.62 in kindergarten #1511). Thus, the ability of those in the older age groups to distinguish oppositions individually from a variety of characteristics was not any more dominating than it was in relation to the parameter of elaboration of dialectical solutions. Because success in finding oppositions sharply increases along with growing up, one can assume that the problem set in Part 1 of the “What can be simultaneously?” technique was too simple for the respondents. However, if a new technique is constructed and dedicated to determining sensitivity to oppositions but is based on more complicated material, a significant correlation could also be found in the other age groups. This thesis requires a further experimental check.
One can see a significant negative correlation in the group of preschool children in kindergarten #1511, who were studying in the development program. Combined with their very high level of success in identifying oppositions (almost equal to the indexes of adult respondents), the highest percentage of refuses-to-handle oppositions, and the lowest rates of dialectical thinking, this evidence could be interpreted as a reflection of the rigidly established formal logical prohibition against handling oppositions, a prohibition set by the specifics of the educational program. Thus these children can figure out oppositions very well and make very few mistakes from the point of view of the formal logics. Still it is difficult for them to regard these oppositions in the context of dynamical changes and to see them as poles that can be mutually transformed one into the other.
There was a significantly positive correlation in the results of children studying in the dialectics program in kindergarten #1602. In this group the ability to identify oppositions did not block implementation of dialectical thinking. Furthermore, if we turn to the results of the individual diagnostics of students of both groups that produced dialectical solutions for the tasks of the second part of the technique, we notice that all these students succeeded in Part 1 also. On this basis, we can conclude that the ability to distinguish oppositions is a necessary condition for the demonstration of dialectical thinking.
5. Interconnection of Mediation and Transaction as Dialectical Acts
This interconnection was examined through correlation within the data received as a result of the two techniques (“What cannot be simultaneously?” and “an unusual tree”).
In regard to this evidence, we come to the following conclusions:
- There was a significant positive interconnection among the parameters in the groups of preschool children, senior school students, and adults. This finding confirms our assumption that dialectical thinking is a system mechanism consisting of interconnected actions.
- The fact that there was no correlation in the groups of elementary school students causes us to assume the existence of a special sensitivity in children to an educational program based on formal logical principles.
6. Statistically Significant Differences Between the Groups of Respondents Who Are the Same Age but Are Studying in Different Educational Programs
We emphasize that statistically significant differences in the implementation of the dialectical strategy of handling oppositions between groups of the same age but following different educational programs were found in the preschool groups only. Still, the steadiness of this trend attracts attention; one can see significant differences in an index of five parameters (the ability to find oppositions, the implementation of dialectical acts of mediation, the implementation of dialectical acts of transaction, the implementation of the strategy of handling oppositions based on “formal mediation,” and “metaphorical integration”).
We can thus come to a conclusion about the specifically high sensitivity of the dialectical structures of 5- to 7-year old children to the impact of the pedagogy. This sensitivity makes the elaboration of a complete educational program for senior and preschool children as well as for students in the elementary grades extremely relevant and important.
Development of Dialectical Thinking During the Process of solving Dialectical Problems
A longitudinal experimental study was conducted from September 2009 to May 2011.
The solution of dialectical problems is a mechanism of the formation of dialectical mental structures in preschool-age children.
Selection Criteria for Experimental and Control Groups
The experimental group consisted of 22 5-year-old children, and the control group had 23 5-year-olds. All respondents attended kindergarten and studied in similar educational programs.
Before the beginning of the lesson we conducted diagnostics of dialectical thinking by means of the “What can be simultaneously?” and “an unusual tree” techniques. A significant difference between the experimental and control groups was revealed by the “What can be simultaneously?” method: the average index for the experimental group was 0.36, and for the control group it equaled 1.11. The results of the “an unusual tree” technique revealed no significant difference (the average index was 0.06 and 0.05, respectively). The percentage of children who produced at least one dialectical answer was 40.91% for the experimental group and 56.52% for the control group. Thus children from the control group initially had more success solving problems requiring implementation of dialectical mental actions.
Children from the experimental group attended three 30-minute-long classes a week over 2 years. The classes were based on the principles of the structural-dialectical approach.
The essence of these classes was the construction of educational situations of a specific kind—specifically, dialectical tasks. A dialectical problem is an analytical one that requires dialectical mental actions in order to be solved. This kind of problem posed for children of preschool age is based on material from myths and fairy tales. For example, the teacher asks a speculative question and then starts to “undermine” proposed points of view. For instance, was the tail brought by Winnie the Pooh to Donkey (Eeyore) for his birthday really a present? If the children say “Yes,” the teacher immediately argues, “But how it could be a present if it was Donkey’s own tail?” When the children switch to “No,” the teacher switches to “Actually, at that moment the tail had already been lost and belonged to Owl, so it could be a birthday present.” During the discussion the teacher uses visual models and schemes that help the children to “fix” oppositional conditions of the same object in their thinking and then to perform the “integration”—in this case, coming to the conclusion that the tail is simultaneously “a present” and “not a present.”
Classes of this kind are conducted also by using material from well-known fairy tales. For example, is the old woman in “The Goldfish” by Alexander Pushkin the same in the beginning and in the end of the story? In the myth about Orpheus and the Sirens, was Orpheus the listener or the singer? (Shiyan, 2012, 2011).
Results of the experiment are presented in Table 4.
Table 4. Results of the Diagnostics of Dialectical Thinking in the Experimental Group (EG) and the Control Group (CG)
|Measure||September 2009. initial diagnostics (average index)||May 2010. Intermediate diagnostics (average index)||May 2011. Final diagnostics (average index)|
|“What can be simultaneously?” technique||0.36||1.11||1.69||1.82||2.05||1.20|
|“An unusual tree” technique||0.06||0.05||0.27||0.06||0.48||0.36|
|% of children who produced dialectical answers||40.91||56.52||50||46.15||65.22||50|
The experimental results thus allow us to calculate the reliable difference between the level of development in the control and experimental groups. Furthermore, over time, the percentage of children in the experimental group producing dialectical answers significantly increased.
Results of the Pilot study on cross-cultural Differences in the Development of Dialectical Thinking
In order to receive additional evidence for the idea that dialectical thinking is an independent line of cognitive development, we conducted a pilot study on cross- cultural differences in the development of dialectical thinking.
Characteristics of the Respondents
One hundred nineteen students at Moscow State University of Psychology and Education (Russia) and 117 students and graduate students at the Central University of Arkansas (United States) took part in the study.
The techniques “What can be simultaneously?” and “an unusual tree” by Veraksa were used in the study.
The average index for dialectical solutions of all test problems was 1.31 for the Russian students and 1.35 for the American respondents. Thus the frequency of the production of dialectical answers appeared almost equal.
Based on our research, we can formulate some common trends in the establishment of dialectical thinking within the human life cycle. The share of respondents demonstrating dialectical thinking in the various age groups was approximately equal, even with the 5% to 10% decrease in elementary school students.
Dialectical thinking is a system mechanism for handling oppositions; such thinking occurs during the whole life of a person. The first signs of dialectical thinking can be noticed already in preschool-age children. At that time dialectical structures are at the stage of active development and therefore are extremely sensitive to the influence of the pedagogy employed. In preschool the connection between the concept of oppositions and the method of handling them is being formed. The ability to distinguish oppositions among a variety of different characteristics is an essential condition and appears basic for the further establishment of dialectical thinking. However, the specific features of the educational program determine whether the concept of oppositions becomes a new resource for a person (if the person becomes able to transform them) or whether thinking will develop along the lines of formal logic, which bans operations with oppositions because of the rule of noncontradiction.
The start of school is the crucial moment for the establishment of a child’s dialectical thinking. Teaching in elementary school is based on formal logical principles that can lead to partial corruption of dialectical structures.
One should remember that the frequency of the implementation of a dialectical strategy for handling oppositions does not differ significantly in senior school students and preschool children.
Once the school period is over, the implementation of dialectical mental operations increases. This finding leads us to suggest the prospective development of an educational program for dialectical thinking in adults. The research results show us that within the framework of the structural-dialectical approach age should be considered not as a factor that strictly determines the development of the mental process and that sets certain limits for it but as presenting an opportunity for the purposeful formation of dialectical thinking within a specially designed educational program.
- The development of dialectical thinking is a complex, multilevel process that occurs within the cognitive activity of a person and is determined by the specific features of educational situations.
- Recognizing and distinguishing oppositions is a basic mechanism of dialectical thinking. Development of this ability is extremely crucial at preschool age.
- The educational system is an important agent in the development of dialectical thinking.
The research was supported by Russian Humanitarian Scientific Foundation grant No13-06-00669a
Bayanova L.F. (2013) Vygotsky’s Hamlet: the dialectic method and personality psychology. Psychology in Russia: State of the Art, 6, 35–42.
Belolutskaya, А. К. (2011). Vozrastnye osobennosti formirivaniya struktur dialekticheskogo myshleniya. [Age-related specific characteristics of the formation of structures of dialectical thinking]. Socialo Zinatnu Vestnesis [Social sciences review], 1, 124–141.
Hegel, G. W. (1970). Raboty raznykh let. [Compilations of works of different years]. 2 vols. Moscow: Nauka.
Krasheninnikov, Е. Е. (2005). Dialekticheskaya psikhologiya. [Dialectical psychology]. Ufa, Russia: BGPU.
Krasheninnikov, Е. Е. (2008). Tvorchestvo i dialekticheskoye myshleniye. [Creativity and dialectical thinking]. Sovremennoe doshkol’noe obrazovaniye. Teoriyai praktika. [Preschool Education Today: Theory and Practice], 4, 42–49.
Shiyan, I. B. (1999). Predvoskhischaiuschiy obraz v strukyure dialekticheskogo myshleniya doshkol’nikov. [Anticipatory image in the structure of dialectical thinking of preschool children]. Voprosy psikhologii [Issues in psychology], 3, 57–65.
Shiyan, I. B. (2011). A propos d’une méthode diagnostic de la réflexion dialectique chez les enfants d’âge préscolaire. Diotome [About a diagnostic method of dialectical thinking in preschool age]. Revue internacionale de didactique de la philosophie, 49, 121–134.
Shiyan, I. B. (2011a). Strukturno-dialekticheskaya psikhologiya razvitiya i kul’turnoistoricheskaya teoriya L.S.Vygotskogo. [Structural-dialectical developmental psychology and L.S.Vygotsky’s Cultural-historical theory]. Vestnik Moskovskogo gorodskogo pedagogicheskogo universiteta. Seriya: Pedagogika i psikhologiya. [Journal of Moscow City Teachers’ TrainingUniversity. Pedagogic and Psychology]. 1, 34–43.
Shiyan, О. А. (2009). Dialekticheskaya zadacha dlya doshkol’nikov. [Dialectical task for preschool age children]. Sovremennoe doshkol’noe obrazovaniye. Teoriyai praktika. [PreschoolEducation Today: Theory and Practice], 4, 42–50.
Shiyan, О. А. (2012). Razvitie tvorcheskogo myshleniya. Rabotaem po skazke. [Development ofcreative thinking. Working with fairy tales]. Moscow: Mozaika-Sintez.
Veraksa, N. Е. (1987). Razvitie predposylok dialekticheskogo myshleniya v doshkol’nom vozraste. [Development of the suppositions of dialectical thinking at preschool age]. Voprosy psikhologii [Issues in psychology], 4, 135–139.
Veraksa, N. Е. (1990). Dialekticheskoe mysheniye i tvorchestvo [Dialectical thinking and creativity]. Voprosy psikhologii [Issues in psychology], 4, 125–132.
Veraksa, N. Е. (2007). Pensamiento dialectico y creatividad. Eclecta. [Dialectical thinking andcreativity]. Revista de Psicologia General, 5(11), 7–14.
Veraksa, N. Е. (2009). Struktura i soderzhanie dialekticheskogo myshleniya. [Structure and content of dialectical thinking]. Moscow: Moscow State University of Psychology and Education.
Veraksa, N. Е. (2010). Structural approach to dialectic cognition. Psychology in Russia: State of the Art, 3, pp. 227–239. doi: 10.11621/pir.2010.0011