Psychological prerequisites and effects of using multimedia content in the mass media

Kulchitskaya, D.Yu. Lomonosov Moscow State University, Moscow, Russia

Introduction

Multimedia technologies have increased during the last decade. They have expanded their reach tremendously and have entered our everyday lives. Currently, such spheres as tourism, learning, computing and business are widely using multimedia in various projects. However, the entertainment sphere is now the main user of multimedia. Gaming and cinema industries are relying greatly on multimedia effects and exploring the new possibilities of virtual reality. Some scholars see this trend as a consequence of multimedia having a strong emotional charge and affecting the potential user with various expressive means (Shlykova, 2004).

However, multimedia technologies can be used for more than just entertainment purposes. The use of multimedia content in the mass media proves this point. Today, various media outlets turn to multimedia genres to convey various meanings and present complex matters in an easy, compelling way. The various psychological effects multimedia might have on the audience have not been thoroughly studied and remain a grey area in research. Therefore, in this theoretical article, we will try to outline the general psychological effects that multimedia products might have on the users and give a theoretical explanation of these effects. We will try to prove that multimedia works have a greater potential than being only entertainment tools, as their nature is closely connected with some specific features of human perception. The article will also provide an overview of the existing psychological research in the sphere of multimedia. The main focus of the paper, however, is the use of multimedia in various media outlets.

Current psychological research trends in the sphere of multimedia

The general lack of research in some aspects of multimedia psychology does not indicate that psychologists have completely ignored the phenomenon. During the last few decades, when multimedia became a wide-spread technology, psychological studies have focused mainly on how multimedia systems may be used to acquire better educational outcomes and help learning. A pioneer in this sphere is the California University professor Richard Mayer, who developed the so-called cognitive theory of multimedia learning, which is based on three key points:

• Human beings have two main channels for processing information (auditory and visual)
• Each of these channels has a limited carrying capacity
• The learning process is an active process of filtering, selecting, organizing and integrating information. (Mayer, 2001)

Based on experiments, Mayer proves that humans memorize information better if it is coded not only in words but also in visual images. Additionally, he distinguished another regularity, which he called the principle of modality, which states that the efficiency of information assimilation is greater if data are delivered as a combination of graphic images and verbal narrative, compared to being coded as graphic images and print text.

Mayer’s studies explain why multimedia is considered to be an effective tool in the education sphere. He argues that multimedia products stimulate an integral processing of verbal and iconic information in the working memory of humans. The integral processing requires that graphic and verbal information is delivered to the working memory simultaneously (Mayer, 2001).

Mayer’s legacy is used and developed by many contemporary scholars. Various studies have been performed in the sphere of educational psychology. Some of them try to build on the cognitive theory of multimedia learning (Bartlett & Strough, 2003; Bower, 2004; Koeber, 2005; Windell, Wiebe & others, 2006; Mitchem, Koury, Fitzgerald & others, 2009; Eysink, Jong, Berthold & others, 2009; Chen & Catrambone, 2014), and others are exploring the way people with various learning problems and diseases can benefit from multimedia (Wissick, 1996; Hagiwara & Smith Myles, 1999; Tjus, Heimann & Nelson, 2001; Elder-Hinshaw, Manset-Williamson, Nelson & others, 2006; Morrow, D’andrea, Stine-Morrow&others, 2012; Kennedy, Deshler& Wills Lloyd, 2015)

During the last decade, multimedia systems have become more complex and sophisticated, which forced many scholars to develop methods to construct efficient multimedia products. In this pursuit of better technological and design solutions, researchers again underlined the importance of understanding the psychological mechanisms of perception and the particularities of various types of media. For example, in his work Multimedia and Virtual Reality, the specialist on human-computer interaction A. Sutcliffe (2003) highlights that to be able to design comprehensible and effective multimedia, one should study the physiology and psychology of perception. He formulates some rules on how to create better multimedia products and how to avoid cognitive overload, fatigue, or stress of the user. The general multimedia principles include the following:

• Thematic congruence: the parts of the multimedia product presented in different media should fit together well and should form a coherent whole.
• Manageable information loading: in designing multimedia products, we should consider the possible overload of the user’s information-processing capacity; the parts of the multimedia product should be presented to the user at a manageable pace.
• Ensure compatibility with the user’s understanding: the images, diagrams or other information in the multimedia products should be compatible with the existing knowledge of the user; the signs and symbols used should be understandable and clear for the person assimilating information.
• Complementary viewpoints: different aspects of one matter should be delivered in various media formats in a complimentary way to ensure better memorization and understanding of the subject.
• Consistency: the interface of the multimedia system should use the same media every time to deliver specific information, which ensures better orientation in the system (for example, use a similar sound signal every time you need to show that the image is changing).
• Reinforce messages: trying to expose the user to the same concept using different modalities is also good for forming lasting memory cues (Sutcliffe, 2003).

Studying multimedia systems has also furthered the understanding of various media outlets, the combination of which creates multimedia. Researchers have distinguished static and dynamic (or time-based) media, which have a different effect on the human brain and can be used to disseminate various types of information. Static media include verbal text and non-moving images, whereas dynamic media are represented by video, audio and interactive graphic materials. Static media form a better base for multimedia products and convey the core concepts and meanings, as a way to make the user memorize them better. Dynamic media are good for attracting attention quickly and for creating an emotional effect (Chapman & Chapman, 2009).

Researchers have investigated the specific features of the way online news is perceived if presented with the help of multiple media. A pioneer in this sphere is S. Shyam Sundar, who is a director at the Media Effects research laboratory at Penn State University. In his study, he tries to discover whether there is a difference in the way text-only materials and multimedia materials are perceived by readers (Shyam Sundar, 2000). He found that multimedia sometimes hinders memory for the content of the story but leads to a positive attitude towards advertisements. However, these results may be because the study was performed long ago when multimedia content was not properly designed.

Since the beginning of the 21st century, there have been more studies on how multimedia elements contribute to the way people perceive news. For example, in a study by a U.S. researcher from the University of Florida, the modality of the news presentation did not directly impact the way people perceive news credibility. Similarly, multimedia content often contributes to a positive view of credibility by the audience (Kiousis, 2006).

Multimedia as an integral stimulus to human perception

However, despite the various approaches to multimedia, few studies have mentioned a quite obvious, but rather unnoticed, feature of various multimedia systems: the fact that multimedia products appeal to almost all of the senses of the human being and therefore create an integral immersive effect. A recent study from Russia examined the psychological effects of multimedia on human beings. A.V. Krapivenko (2009), in his work Multimedia technologies and perception of sensations, specifically focuses on this feature of multimedia. The researcher presents an original definition of the phenomenon and emphasiszes the effect it has on human senses:

Multimedia — a contemporary computer technology, which enables us to unite different types of multimedia data in a programme system (images, audio, video, tactile sensations, etc.) for the creation of a unified information environment to affect human perception through the sensory organs (Krapivenko, 2009).

The Russian scholar also argues that multimedia designers should know not only the psychological patterns of human perception but also understand the physical nature of various phenomena affecting the human senses. According to him, this knowledge is essential, as we will discover which physical phenomena are imitated in multimedia properly and which are imitated improperly. Krapivenko also mentions the possibility to train human sensory organs through multi- media programmes. He cites a study that showed that a group of students playing shooter games managed to improve their peripheral vision after several sets of the games (Krapivenko, 2009). This observation is particularly important and relevant as psychology finds various manners of usage for multimedia and virtual reality. For example, contemporary research has shown that curing various phobias and other psychological disorders with the help of virtual reality immersive systems is very effective (Zinchenko, Menshikova, Bayakovsky, Chernorizov, & Voiskounsky, 2010, p. 16).

Based on the aforementioned findings, multimedia is a new tool that has a substantial influence on the human senses and imitates reality by sending signals to all (or several) sensory organs and re-creating the multi-layered world in which we live. In other words, by describing multimedia products, we can say that they are designed with regards to the two following intertwined phenomena: multimodality of perception and polyphony of reality. We will define these two terms, which still are not widely used in the theory of communication and media psychology.

Polyphony of reality is a characteristic of the objective world to disseminate information using various channels (visual, auditory, tactile, gustatory and olfactory).

This specific characteristic of the objective world has never previously been described terminologically, but it has been noticed by many scholars from various fields of study. The famous Soviet psychologist A.N. Leontiev spoke of a phenomenon similar to this theory. He introduced the term “image of the world”, which he defined as the following:

“The world doesn’t consist of light, colours, vibrations...warmth, cold... It has its characteristics and is represented in these properties... only in the process of world perception through these modalities ... i.e., not as a complex of sensations, but as a reality, which “speaks” of itself in the languages of these sensory modalities” (A.N. Leontiev, 1975). Additionally, another term is similar to the phenomenon described above. It was introduced to psychology by the U.S. scholar James Gibson and named distal stimulus: a notion that is designed to describe the object of perception in its various forms and qualities.

The term multimodality of perception is more widely used in research and is a common notion for contemporary psychologists. It is defined by the following:

Multimodality of perception — the capacity of human beings to receive information about the outer world using all end-organs (visual, auditory, tactile, gustatory and olfactory).

Both of the mentioned phenomena may be considered prerequisites to the emergence of multimedia. The orientation towards the reproduction of the sensible world (i.e., the polyphony of reality) and the fact that multimedia products are designed inwith regards to the specific feature of human perception called multimodality facilitates the consumption of multimedia materials.

The two intertwined phenomena described above lead to various psychological effects that multimedia products may have on the end user. Knowing and understanding these effects of multimedia content may achieve better results and present interesting topics in a compelling way. This knowledge is crucial for journalists as well, as the amount of multimedia content is increasing rapidly.

Multimedia products in the mass media

In the mass media, an orientation towards general visualization of the news occurred a long time ago. The second part of the twentieth century was dominated by TV, which gradually became the main medium. However, the rapid development of Internet replaced TV sets with computers, and currently, the majority of the audience receives news through the screen of a PC or a tablet. As the electronic devices develop and there are more carriers of multimedia content, journalists turn to products, often combining several media. Similarly, in the mass media, multimedia technologies are often used only as an entertainment tool. This method is a general trend of the 21st century during which the entire media system has integrated into the leisure market and the industry of free time (Vartanova, 2013).

However, by using multimedia only as an entertainment tool, journalists neglect the potential of multimedia, which can be used in information assimilation.

We will try to enumerate several psychological effects, which multimedia may have on the audience and all of which can be connected to the multimodality of perception and the polyphony of reality

First, multimedia products that are well-designed create a sense of presence and make the reader/user believe everything is really unfolding in front of his/her eyes. This immersive effect is achieved because of the various stimuli used by multimedia products that are familiar to the user and make him experience something, which is very close and similar to the real world. This specific sense of presence may also be explained by the feeling of reality that has a sensual basis (Leontiev, 1975). If we describe an event using sensory images, we make the recipient feel that everything described is authentic and actually exists. This perception may open some room for manipulation, as the newest technologies help adjust and change images, video, and sound files. However, a professional journalist will not use this tool for this form of distortion and will rely on the potential multimedia to describe events accurately and in a realistic way.

Multimedia materials with a presence effect are gaining popularity in today’s mass media. For example, 3D photo galleries and separate multimedia stories perfectly recreate the scenes of natural disasters or wars. After the earthquake in Haiti, many news outlets used immersive multimedia materials to present realistic and persuasive images of the tragedy on the island.

Multimedia content also has a substantial emotional charge and may be used as a tool for achieving the so-called empathy effect in the audience. This notion may be explained in several ways. First, multimedia strongly relies on visualized information and uses various forms of media. In previous research, it has been stated that visual images (both in static and dynamic forms, such as still photography or moving video) have an immense effect on our brain and arouse various emotions: from fear and disgust to delight and happiness. This orientation towards visual culture has been thoroughly analysed by the U.S. media theorist W. J.T. Mitchell (2005), who found roots of this emotional value of pictures in the unconscious sphere. He argues that people often treat the iconic images as real objects, which goes back to animalism and totemism.

Another interesting feature is the empathy effect, which can be achieved through multimedia products. Empathy, the ability to react to and share the feelings of other people, has been achieved by movies and TV shows due to their emotional effect and realistic display of heroes and events. The same applies to multimedia content, which is well-designed and appeals to the reader/user. This feature is also connected to the fact that multimedia systems produce multimodal stimuli and give the illusion that everything described is unfolding currently.

This emotional “hook” effect is already used in some media products, such as the so called long-read materials designed by The New York Times and other prominent newspapers using multimedia. The Snowfall project, describing an expedition to a snow peak and issued on the site of the publication in 2012, arouses various feelings in the users, as they see and hear the witness accounts of the participants in the expedition and experience 3D models of the snow peak and the route of the travellers.

The third psychological effect that multimedia has on the audience, is the so- called simplification effect. It can have both negative and positive outcomes for the

reader/user. In the learning process, complex and complicated problems and information are often presented in a playful, simple way so that the audience understands the core issues without being bored or annoyed by the explanation. In contemporary literature, this phenomenon is called edutainment. Multimedia may use its entertaining potential and be a tool for conveying information in a way that is easy for assimilation. In the mass media, if great loads of complicated information should be delivered, journalists often turn to multimedia techniques. An example of this approach may be the story by the famous British newspaper The Guardian, which did a multimedia story based on the U.S. National Security Agency (NSA) materials.

Another feature of multimedia products is the interactivity. As many scholars note, this phenomenon is not new and has been in use before. With the advent of the Internet and multimedia technologies, it has increased its importance and scale. Interactivity makes the process of consuming multimedia products compelling and interesting. Therefore, users have the feeling that they participate actively in the story that is unfolding and feel emotionally attached to the heroes and events described.

All of the effects enumerated above are sometimes used by the journalists, but they are often used unconsciously and inappropriately. Knowing these psychological particularities will make the authors be more selective and create well-conceived and designed materials, which will meet their end goals and make the assimilation of information an easy and interesting process.

Although psychological effects of multimedia content usage are not widely described and analysed in research, journalists are approaching this knowledge via practical experiments. In online journalism, there is even an emerging format, which helps achieve emotional and presence effects. It is generally called a multi- media story, but sometimes other terms are used, such as interactive feature, web-documentary, etc. Although it is a new phenomenon, there have been attempts to define it. Russian researchers suggest the following definition:

A multimedia story is a journalistic story in which the main topic is explained using various media — such as text and audio-visual means — which altogether create an integral and multidimensional picture of an event (Lukina, 2010).

It is also crucial that a multimedia story is designed in a way that all of the elements in it are complimentary, and their combination makes the story an integral whole. Additionally, a multimedia story may be delivered in various genres (for example, multimedia interview, multimedia feature, etc.).

Practitioners and scholars note that not all stories may be presented in the format of a multimedia story. Here are some of the main requirements for a story, which can be transferred to a multimedia story:

• The event should evolve in time
• The event should include visual episodes that can be described with the epithet ‘the most’
• The story includes details that can be easily visualized rather than described in words
• Many opportunities to create video footage
• Substantial background information
• The event may potentially be developed by user generated content (Lukina, 2010).

Multimedia stories are well designed for creating a presence effect and the effect of emotional attachment, as they appeal to various senses and create a “digital imprint” of an event. However, their potential has not been used fully, and there are many ways to explore it.

Conclusion

Multimedia technologies are an invention of the 21st century, but they have roots in various phenomena in the history of communication and the arts. Their evolution and development are still in progress, and it is important to understand that the very core idea of multimedia is natural for the human being, who lives in a multimodal world and perceives it with all five senses.

In communication and journalism theory, there have not been many attempts to comprehend the psychological basis underlying in the multimedia phenomenon. Additionally, media psychology has neglected this sphere and focused mainly on the analysis of gaming and cinema applications of multimedia. However, as we can see from this work, a thorough analysis of the psychological regularities that exist in multimedia may help journalists and media workers attract a wider audience and make their works more compelling and comprehensible. By knowing the specific features of each medium and the general effects, which result from their combination, media outlets would better understand various multimedia formats and would achieve better comprehension and assimilation of their information products. Neglecting some particularities of human perception would result in a bad outcome for the media outlet and cause negative feedback from the audience and a decline in readership. Therefore, further psychological research is needed to understand the nature of multimedia as a complex phenomenon, which can be effectively used in the mass media. Focus groups, analysis of users’ perception experience and the study of various psychological effects of multimedia texts in journalism may be useful both for the audience and for the mass media sphere.

Scholars and practitioners should regard multimedia not as a technology, which is alien to the human world, but as a tool that can greatly benefit the process of information dissemination if it is well-designed.

References

Bartlett, R., & Strough, J. (2003). Multimedia versus traditional course instruction in introductory social psychology. Teaching Psychology, 30(4), 335-338. doi; 10.1207/S15328023TOP3004_07

Bower, J. I. (2004). The impact of asynchronous multimedia communications on understanding and recall. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 48(3), 431-435. dot 10.1177/154193120404800333

Chapman, N., & Chapman, J. (2009). Digital multimedia. West Sussex: John Willey&Sons.

Chen, D., & Catrambone, R. (2014). Effects of multimedia interactivity on spatial task learning outcomes. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 1356-1360. doi: 10.1177/1541931214581283

Elder-Hinshaw, R., Manset-Williamson, G., Nelson, J., & Dunn, M. (2006). Engaging older students with reading disabilities: Multimedia inquiry projects supported by reading assistive technology. Teaching Exceptional Children, 39(1), 6-11.

D. Yu. Kulchitskaya Eysink, T., Jong, T., Berthold, K., Kolloffel, B., Opfermann, M., & Wouters, P. (2009). Learner performance in multimedia learning arrangements: An analysis across instructional approaches. American Educational Research Journal, 46(4), 1107-1149. doi: 10.3102/0002831209340235

Hagiwara, T., & Smith Myles, B. (1999). A multimedia social story intervention: Teaching skills to children with autism. Focus on Autism and Other Developmental Disabilities, 14(2), 82-95. doi: 10.1177/108835769901400203

Kennedy, M., Deshler, D., & Wills Lloyd, J. (2015). Effects of multimedia vocabulary instruction on adolescents with learning disabilities. Journal of Learning Disabilities, 48(1), 22-38. doi: 10.1177/0022219413487406

Kennedy, M. J., Newman T. C., Meyer, J. P., Alves, K., & Wills Lloyd, J. (2014). Using evidence- based multimedia to improve vocabulary performance of adolescents with LD. Learning Disability Quarterly, 37(2), 71-86. doi: 10.1177/0731948713507262

Kiousis, S. (2006). Exploring the impact of modality on perceptions of credibility for online news stories. Journalism studies, 7(2), 348-359. doi: 10.1080/14616700500533668

Koeber, C. (2005). Introducing multimedia presentations and a course website to an introductory sociology course: How technology affects student perceptions of teaching effectiveness. Teaching Sociology, 33(3), 285-300. doi: 10.1177/0092055X0503300309

Krapivenko, A. V. (2009). Tekhnologii multimedia i vospriatie oschusheniy [Multimedia technologies and perception of sensations]. Moscow: Binom.

Leontiev, A. N. (1975). Deyatelnost. Soznanie. Lichnost [Activity. Consciousness. Personality]. Moscow: Moscow University Press.

Lukina, M. M. (Ed.). (2010). Internet-SMI: teoria ipraktika [Online media: theory and practice]. Moscow: Aspekt Press.

Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press, doi: 10.1017/ CB09781139164603

Mitchell, W. J. T. (2005). What do pictures want? Chicago: The University of Chicago Press.

Mitchem, K., Koury, K., Fitzgerald, G., Hollingsead, C., Miller, K., Tsai, H., & Zha, S. (2009). The effects of instructional implementation on learning with interactive multimedia case-based instruction. Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, 32(4), 297-318. doi: 10.1177/0888406409343520

Morrow, D., d’Andrea, L., Stine-Morrow, Shake, M., Bertel, S., Chin, J., Kopren, K., Gao, X., Conner-Garcia, T., Graumlich, J., & Murray, M. (2012). Comprehension of multimedia health information among older adults with chronic illness. Visual Communication, 11(3), 347-362. doi: 10.1177/1470357212446413

Shlykova, О. V. (2004). Kultura multimedia [Multimedia culture]. Moscow: Fair-press.

Shyam Sundar, S. (2000). Multimedia effects on processing and perception of online news: A study of picture, audio, and video downloads. Journalism and Mass Communication Quarterly, 77(3), 480-499. doi: 10.1177/107769900007700302

Sutcliffe, A. (2003). Multimedia and virtual reality. New Jersey: Lawrence Erlbaum Associates.

Tjus, T, Heimann, M., & Nelson, K. (2001) .Interaction patterns between children and their teachers when using a specific multimedia and communication strategy: Observations from children with autism and mixed intellectual disabilities. Autism, 5(2), 175-187. doi: 10.1177/1362361301005002007

Vartanova, E. L. (2013). The media and the individual: economic and psychological interrelations. Psychology in Russia: State of the Art, 6(1), 110-118. doi: 10.11621/pir.2013.0110

Zinchenko, Yu. P., Menshikova, G. Ya., Bayakovsky, Yu. M., Chernorizov, A. M., & Voiskoun- sky, A. E. (2010). Technologies of virtual reality in the context of worldwide and Russian psychology: Methodology, comparison with traditional methods, achievements and perspectives. Psychology in Russia: State of the Art, 3,12-45. doi: 10.11621/pir.2010.0001

Windell, D., Wiebe, E., Converse-Lane, S., & Beith, B. (2006). A comparison of two mental workload instruments in multimedia instruction.Proceedings of the Human Factors and Ergonomics Society Annual Meeting,50(17), 1764-1766. doi: 10.1177/154193120605001709

Wissick, C. (1996). Multimedia: Enhancing instruction for students with learning disabilities. Journal of Learning Disabilities, 29(5), 494-503. doi: 10.1177/002221949602900504