The Art of Melding Image and Text

As a graduate student, Susan Hagan didn’t think the required English courses would help with her studies in design. This view changed when she noticed that writing improved her drawing, especially scenic writing. She found this exercise forced her to notice how sounds and actions connected to line and texture. This realization cracked open the window to a new world to explore, one where the effects of language were implicit.

Communication design combines the elements of image, text and typeface. By studying examples, Hagan could show how meaning changed when any of the elements are altered. Communication was not implicit in the text or image alone but something that needed its own framework, a framework that Hagen calls ‘inter-play.’

“Conventionally, semantics and syntax contribute to meaning,” said Hagan, associate teaching professor in the Information Systems program, jointly offered by Dietrich College of Humanities and Social Sciences and Heinz College of Information Systems and Public Policy. “There are limits to text that make visual information necessary [that is lost in text alone].”

This realization has profoundly affected the course of Hagan’s research and career. Today, she explores the interaction of words, images and typeface, both static and in motion and brings these concepts to the courses that she teaches at Carnegie Mellon University. In April 2023, her work coalesced in a thought-provoking book, “The Space between Look and Read: Designing Complementary Meaning,” from MIT Press.

In her book, Hagan bridges the fields of design and rhetoric, arguing that the collaboration of image, text and typeface could be useful in some situations. Seeing and reading delivers messages that are both not easily forgotten and can challenge bias and inexperience by providing visual with textual reasons or proof for claims. This work builds perspective that bias would not and inexperience could not allow.

As an example, Hagen references the picture of the tiger. The images could provoke a variety of thoughts and feeling, but when paired with the simple phrase, ‘I am not a rug,’ the meaning is crystalized. The complementary union of these different pairings produce new associations that are absent with just an image, text or typeface alone. The image suggests life, while language adds a new class to the group that should not be made into rugs.

The mechanisms that bring these elements together — grouping ties and bridging ties — lead to loose to tight configurations and potentially add value to the overall meaning of the concept being portrayed. While inter-play can be merely decorative or manipulative, she notes that one outcome, complementary meaning, can be used to improve communication and transparency and challenge existing views.

“The book provides a framework of outcomes,” said Hagan. “The take-away message is that it is important how we communicate with people who are inexperienced or have a bias.”

Hagan believes that CMU students in human-computer interaction, information systems and design could benefit from these concepts. She also sees benefits in organizations that are working to address hate, conspiracy and bias.

In the future, Hagan wants to extend these concepts, bringing sound and motion into the equation to address aspects that may be missed by text and images.

Hagan maintains a split appointment between the main CMU campus and the campus in Qatar, spending a semester at each campus annually.