Film Cognition

What is going on in the mind of a viewer when they watch an audiovisual narrative? Film and TV evolved formal techniques to minimise the effort involved in processing their highly unnatural flow of sensory information. Here we study cinematic techniques such as cinematography, editing, and sound design as well as their moment-by-moment influences on cognition and emotion to both understand why film evolved the way it did and what this tells us about how we process real-world dynamic scenes.

Our work into film cognition tackles many aspects including film literacy, the role of sound, narrative comprehension and edit blindness. But a large part of our work has focussed on the perception of continuity across edited film sequences and how we can use eye tracking as a window into the real-time cognition occurring during film viewing. All of these areas are covered in our publications below but for a quick overview of these two main areas first check out these publications:

​

​

For on-going ramblings on this topic check out Dr. Tim J. Smith's research blog, Continuity Boy.

Introductory Videos on Film Cognition

You can find a video lecture introducing our work on Film Cognition presented as part of the Zurcher Dokumentarfilmtagung (ZDOK; Documentary Film Workshop) in Zurich on 22nd March, 2018 here: Summary of Dr. Tim J. Smith's 2015 Academy of Motion Picture Arts and Sciences (AMPAS) event, Movies in Your Brain.



Demo Videos & Resources

Other demo videos and film/eye movement corpora:

DIEM Project



Key Publications

Ildirar Kirbas, Sermin and Smith, Tim J. (2018) How infants perceive animated films. In: Uhrig, M. (ed.) Emotion in Animated Films. Routledge Advances in Film Studies. Abingdon, UK: Routledge. ISBN 9781138303287. Loschky, L. C. and Hutson, J. P. and Smith, M. E. and Smith, Tim J. and Magliano, J. P. (2018) Viewing static visual narratives through the lens of the Scene Perception and Event Comprehension Theory (SPECT). In: Dunst, A. and Laubrock, J. and Wildfeuer, J. (eds.) Empirical Comics Research: Digital, Multimodal, and Cognitive Methods. Routledge Advances In Comics Studies. Abingdon, UK: Routledge, pp. 217-238. ISBN 9781138737440. Batten, J. and Smith, Tim J. (2018) Looking at sound: sound design and the audiovisual influences on gaze. In: Dwyer, T. and Perkins, C. and Redmond, S. and Sita, J. (eds.) Seeing into Screens: Eye Tracking and the Moving Image. London, UK: Bloomsbury Publishing. ISBN 9781501329029. Hinde, S. J. and Smith, Tim J. and Gilchrist, I. D. (2018) Does narrative drive dynamic attention to a prolonged stimulus? Cognitive Research: Principles and Implications 3 (1), p. 45. ISSN 2365-7464. Ildirar, S. and Levin, D. T. and Schwan, S. and Smith, Tim J. (2017) Audio facilitates the perception of cinematic continuity by first-time viewers. Perception 47 (3), pp. 276-295. ISSN 0301-0066. Hutson, J. P. and Smith, Tim J. and Magliano, J. P. and Loschky, L. C. (2017) What is the role of the film viewer? The effects of narrative comprehension and viewing task on gaze control in film. Cognitive Research: Principles and Implications 2 (1), ISSN 2365-7464. Hinde, S. J. and Smith, Tim J. and Gilchrist, I. D. (2017) In search of oculomotor capture during film viewing: implications for the balance of top-down and bottom-up control in the saccadic system. Vision Research 134 , pp. 7-17. ISSN 0042-6989. Smith, Tim J. and Martin-Portugues Santacreu, J. Y. (2017) Match-action: the role of motion and audio in creating global change blindness in film. Media Psychology 20 (2), pp. 317-348. ISSN 1521-3269. Loschky, L. C. and Larson, A. M. and Magliano, J. P. and Smith, Tim J. (2015) What would Jaws do? The tyranny of film and the relationship between gaze and higher-level narrative film comprehension. PLoS One 10 (11), e0142474. ISSN 1932-6203. Smith, Tim J. (2015) Read, watch, listen: a commentary on eye tracking and moving images. Refractory: A Journal of Entertainment Media 25 (9), ISSN 1447-4905. Wass, S. V. and Smith, Tim J. (2015) Visual motherese? Signal-to-noise ratios in toddler-directed television. Developmental Science 18 (1), pp. 24-37. ISSN 1363-755x. Smith, Tim J. (2014) Audiovisual correspondences in Sergei Eisenstein’s Alexander Nevsky: a case study in viewer attention. In: Taberham, P. and Nannicelli, T. (eds.) Cognitive Media Theory (AFI Film Reader),. AFI Film Readers. Abingdon, UK: Taylor & Francis. ISBN 9780415629874. Smith, Tim J. (2013) Watching you watch movies: using eye tracking to inform film theory. In: Shimamura, A (ed.) Psychocinematics: Exploring Cognition at the Movies. New York, U.S.: Oxford University Press, pp. 165-191. ISBN 9780199862139. Smith, Tim J. and Mital, P. K. (2013) Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes. Journal of Vision 13 (8), ISSN 1534-7362. Smith, Tim J. and Levin, D. and Cutting, J. E. (2012) A window on reality: perceiving edited moving images. Current Directions in Psychological Science 21 (2), pp. 107-113. ISSN 0963-7214. Smith, Tim J. (2012) Extending AToCC: a reply. Projections 6 (1), pp. 71-78. ISSN 1934-9688. Smith, Tim J. (2012) The attentional theory of cinematic continuity. Projections 6 (1), pp. 1-27. ISSN 1934-9688. Smith, Tim J. (2010) Film (cinema) perception. In: Goldstein, B. E. (ed.) Encyclopedia of Perception. Thousand Oaks, U.S.: Sage. ISBN 9781412940818. Smith, Tim J. and Henderson, J. M. (2008) Edit blindness: the relationship between attention and global change blindness in dynamic scenes. Journal of Eye Movement Research 2 (2), pp. 1-17. ISSN 1995-8692. Smith, Tim J. (2006) An attentional theory of continuity editing. Doctoral thesis, University of Edinburgh.



External Links

The Society for Cognitiive Studies of The Moving Image (SCSMI) and associated Facebook group. James Cutting’s (Cornell) comprehensive statistical analysis of the evolution of cinematic features and their cognitive consequences. David Bordwell’s constantly eye-opening blog on film in all of its many forms: Observations on film art.