tA/v\Am
Narrative:
Historically, writing has been the primary mode for analyzing art and music. Analysts generally turn to the written word (books, journal articles, etc.) in their efforts to find and explicate meaning within works of art. While much writing has transferred into the digital medium—online journals and e-books, for example—it clings to the format of its predecessor: words (and images) on a white canvas. Unfortunately, words and images may be either insufficient or inefficient in analyzing the sensational richness of multimedia and interactive art/music.
When attempting to write about video pieces, video games, or other forms of A/V art, one must use extensive language to describe the visual aesthetics, sonic material, the experience of empathizing for characters on the screen, etc., before one may begin writing an in-depth and informative analysis of the work. To be sure, analysts may incorporate static screen shots of the video and/or musical examples of the audio to demonstrate the media being described; however, these snapshots fall short of truly representing the experience of the media, and in many cases, certain audiences may not be able to read music notation or understand how to interpret spectrograms.
In an attempt to address these concerns, analysts may impose their written analyses directly onto the medium in the form of subtitles. In this way, all of the important data (sound, visuals, and analytical text) exist in one place and are presented efficiently. However, due to the fixed limits of the basic video medium and its playback, the subtitle format often fails to adhere to the viewer's preferred pace. Differing levels of familiarity, interest, and theoretical backgrounds of the media being analyzed make it desirable to provide the possibility of adjusting the speed of the video such that users may repeat, skip, or pause sections of the analysis.
Analysts may also want to spend a large portion of an analysis highlighting a specific moment in a piece of A/V art. If the analysis is embedded within the A/V media, it may be necessary to pause playback of the image and sound in order for the viewer to read in-depth analyses of a small section. For any discussion of the audio to occur, the aural experience must be suspended until viewers are finished reading the text.
In creating tA/v\Am (The Audio/Visual Analysis Machine), I believe I have solved many of these problems by building an audio/video player that gives the user complete control over the playback rate, direction, and cursor position without affecting the pitch content of the audio. This allows the user to control the pacing of the text and sensory data, as well as repeat playback at a comfortable rate. Thus, users may slow down, reverse, and pause/hold audio and video information in order to experience a single moment that may be critical to the analysis. Utilizing a granular synthesis module, the pitch and timbre of the audio are maintained as the user manipulates time. This provides analysts with a tool with which they can freeze audio to demonstrate the techniques being used at a particular point in time.
While tA/v\Am may be used to analyze any medium containing audio and/or video, it is especially elegant in analyzing video games, since both are interactive audio/video formats. Those interested in analyzing video game music will be able to effortlessly show, for example, how a character's interaction with its environment affects the sounds or music within a game. tA/v\Am, however, also proves valuable to music theorists looking to provide interactive audio/video musical examples. For example, theorists may use tA/v\Am to display audiovisual diagrams, encouraging readers/users to manipulate and hear for themselves the musical relationships that the theorist is illustrating. tA/v\Am may also be applied to the music theory classroom. Instructors wishing to demonstrate long-form tonal trajectories to students can simply generate videos, synchronizing audio with, for example, Schenkerian diagrams. Then, the instructor can easily 'scrub' through the video, allowing students to hear the ursatz directly, and how it is articulated by the foreground/middleground structures. Finally, theorists may also use the software to enhance conference paper presentations by scrubbing through their own interactive analyses in front of an audience, zooming into moments by pausing the video for in-depth discussions on the sounding sonority. This brings together the theoretical discussion with the audio example, solving the otherwise detached method of presenting the theory first, followed by the audio example.
When attempting to write about video pieces, video games, or other forms of A/V art, one must use extensive language to describe the visual aesthetics, sonic material, the experience of empathizing for characters on the screen, etc., before one may begin writing an in-depth and informative analysis of the work. To be sure, analysts may incorporate static screen shots of the video and/or musical examples of the audio to demonstrate the media being described; however, these snapshots fall short of truly representing the experience of the media, and in many cases, certain audiences may not be able to read music notation or understand how to interpret spectrograms.
In an attempt to address these concerns, analysts may impose their written analyses directly onto the medium in the form of subtitles. In this way, all of the important data (sound, visuals, and analytical text) exist in one place and are presented efficiently. However, due to the fixed limits of the basic video medium and its playback, the subtitle format often fails to adhere to the viewer's preferred pace. Differing levels of familiarity, interest, and theoretical backgrounds of the media being analyzed make it desirable to provide the possibility of adjusting the speed of the video such that users may repeat, skip, or pause sections of the analysis.
Analysts may also want to spend a large portion of an analysis highlighting a specific moment in a piece of A/V art. If the analysis is embedded within the A/V media, it may be necessary to pause playback of the image and sound in order for the viewer to read in-depth analyses of a small section. For any discussion of the audio to occur, the aural experience must be suspended until viewers are finished reading the text.
In creating tA/v\Am (The Audio/Visual Analysis Machine), I believe I have solved many of these problems by building an audio/video player that gives the user complete control over the playback rate, direction, and cursor position without affecting the pitch content of the audio. This allows the user to control the pacing of the text and sensory data, as well as repeat playback at a comfortable rate. Thus, users may slow down, reverse, and pause/hold audio and video information in order to experience a single moment that may be critical to the analysis. Utilizing a granular synthesis module, the pitch and timbre of the audio are maintained as the user manipulates time. This provides analysts with a tool with which they can freeze audio to demonstrate the techniques being used at a particular point in time.
While tA/v\Am may be used to analyze any medium containing audio and/or video, it is especially elegant in analyzing video games, since both are interactive audio/video formats. Those interested in analyzing video game music will be able to effortlessly show, for example, how a character's interaction with its environment affects the sounds or music within a game. tA/v\Am, however, also proves valuable to music theorists looking to provide interactive audio/video musical examples. For example, theorists may use tA/v\Am to display audiovisual diagrams, encouraging readers/users to manipulate and hear for themselves the musical relationships that the theorist is illustrating. tA/v\Am may also be applied to the music theory classroom. Instructors wishing to demonstrate long-form tonal trajectories to students can simply generate videos, synchronizing audio with, for example, Schenkerian diagrams. Then, the instructor can easily 'scrub' through the video, allowing students to hear the ursatz directly, and how it is articulated by the foreground/middleground structures. Finally, theorists may also use the software to enhance conference paper presentations by scrubbing through their own interactive analyses in front of an audience, zooming into moments by pausing the video for in-depth discussions on the sounding sonority. This brings together the theoretical discussion with the audio example, solving the otherwise detached method of presenting the theory first, followed by the audio example.