Knowledge Design Meets Media Design: Multimodal Communication of Knowledge Structures

Forming knowledge by designing media

“Design is making sense of things.”¹ Klaus Krippendorff makes this statement to emphasize that designing communicative objects has far-reaching consequences. Design impacts on the messages, meanings, and knowledge to be communicated. It is Krippendorff’s interpretation of a basic position well-known in media and communication studies: ‘Design’ is primarily considered ‘expressive’ rather than ‘aesthetically appealing’. Historians, too, are familiar with this concept. When examining material aspects of history, for instance, the architecture of a historic building becomes meaningful in terms of cultural, social and political ideas. The same applies to the design of memorials, ancient ritual objects, elaborate diplomatic documents, or any other symbolic form of sense-making. These communicative designs are meaningful to historical research.

But how about our own products of communication in historiography?² Our publications are traditionally texts, and this is by no means coincidental, for text as a medium is well-suited to represent logical argumentation and narrative structure. It employs an overall demonstration along defined sections with detailed references, linking academic debates from specific areas. We know about these text language capabilities from the fundamental works of Arthur C. Danto,³ Hayden White,⁴ Anthony Grafton⁵ and others. All in all, text language seems predestined for historiographical demonstration.

Beyond text: multimodal historiography

So why should we even ask for other media of expression for historiography? In recent decades, attempts to answer this question have come notably from postmodern or cultural history perspectives: Texts have a somewhat linear structure, they say, making it hard to use them to demonstrate multi-perspectivity. Consequently, something complex like transnational entanglements could not be adequately expressed. The same applies when historians intend to show layers/sediments of time and their interrelations or refer to historical actors and groups of fluid identities. These complex and flexible conceptions are not easily demonstrated by text and its linear progression – at least if we want to avoid countless repetitions and cross-references, or straying into detail.

Pluralistic historiography represents interpretations as “structures of meaning”. These call for media of expression that should avoid the linearity of text and causality, as Stefan Haas argues.⁶ Many scientists and scholars explain the limitations of text to convey complexity, contingency and ambivalence with the qualities inherent in text as a medium. As an alternative they bring into play non-linear media such as hypertext⁷ or visual media such as concept maps.⁸ In that sense, David J. Staley argues that visualizations should not be considered illustrations subordinate to written demonstrations. Instead, he defines them as “meaningful conceptual tools of both inquiry and insight in their own right.”⁹

Displaying coherence structures

Take this network graph as an example. It is part of Pathfinders, an online publication by Stuart Moulthrop and Dene Grigar that explores the history of early digital literature:

The graph represents the pages that tell the story of early digital literature as orange nodes (two nodes highlighted by clicking on them). At the same time, the blue nodes indicate how pages have been weaved together in narrative collections called “paths.” These paths may share pages, making them a device to structure content flexibly. They are a traversal to any overarching structure, such as sections and sub-sections, and used by Moulthrop and Grigar to represent the relations they deem essential for their demonstration. As for the visual display, it not only provides a coherent overview. Beyond that, it is an interactive means of exploring the entire contents. Users may switch to any page or path by clicking on the respective node.

From a theoretical point of view, we might well ask what exactly is the benefit of such a multimodal publication? Is it more suitable for comprehension than text, does it offer a clearer view of complex relations? So, is it a matter of ‘mere’ media-pedagogic advantages? Or what else is the point of talking about “meaningful conceptual tools of inquiry and insight in their own right” as Staley does? In my view, this question is of particular theoretical interest.

A first attempt to answer it could begin with the various semiotic qualities and perceptions that distinguish visualizations from texts. Here, Staley emphasizes a two- or three-dimensional visualization syntax, in contrast to the one-dimensionality of (prose) text.¹⁰ Studies of multimodality research go beyond this observation. They demonstrate that visual media are ideally suited to provide an overview of narrative and argumentative structures because, semiotically speaking, the visual is iconic and leads to a direct, simultaneous and holistic perception.¹¹ Against this background, the network graph from Pathfinders represents an “analytical structure.” According to the pioneers of multimodality research, Gunther Kress and Theo van Leeuwen, these analytical structures present elements in their relation to an overall configuration (“meronymical relations”).¹² The visualization gives an overview of the coherence structure, and this overview converges with the propositional text that discursively elaborates on academic arguments and narratives. This is the multimodal character of the entire demonstration.

Such an analytic perspective can help to identify the critical semiotic and perceptual differences between text and visual media – how differently they ‘function’ in this respect. We might then intuitively think that visual or multimodal media are more ‘adequate’ when it comes to representing complex coherence structures: Are they not themselves complex and non-linear? Do they not therefore immediately demonstrate what they are supposed to express? I would argue, however, that such an intuitive conclusion stops halfway and says almost nothing about what ‘adequacy’ means here. What exactly does a visual or multimodal publication format accomplish?

A proposal for a theoretical backdrop: revealing the (de facto) construction of knowledge structures

In attempting to answer this question, it is worth taking a closer look at the contexts and relations that every historian ultimately seeks to convey: Their historiographical works explain the applied approach and selection of sources, and argue for the plausibility of the given interpretations. Authors must reveal all of the relevant diachronic, causal, contingent, and other relations that characterize the historiographical conception. Selection and coherence shape the narrative and argumentative character of historiography.

This weaving of contexts and relations is, beyond doubt, a constructive process. Having said this, I use ‘constructive’ in a moderate sense, highlighting that – in practice – we de facto build up narratives and arguments when writing coherent works. I do not, however, engage in the old epistemic debate on realism/positivism versus constructivism, a meta-theoretical question referring to whether historians convey truths in a positivist sense. It may well lead the way when it comes to the historian’s self-image while producing a coherent historiography. But the question is not crucial from a praxeological point of view, which focuses on historiography as a practical enterprise to build up coherence (structural knowledge design) with media. In other words, I apply a non-dualistic concept of constructivism (no comparison between academic conclusions and reality). Josef Mitterer,¹³ in particular, has promoted non-dualistic philosophy. Siegfried J. Schmidt follows a similar route by his postulate of not mentioning objects of knowledge, as these entities are hard to prove or characterize. Instead, he emphasizes processes of argumentative reasoning and interpretation justification.¹⁴ Both authors avoid the term ‘constructivism,’ to be sure. I consider it helpful, however, to accentuate the constructive process of forging coherent arguments and narratives. Consequently, historiography is taken seriously as a fabrication result, but without taking sides in fundamental epistemic debates. On this basis, we can assess the potential of multimodal publications more precisely: Do they demonstrate the very constructedness of historiography? For digital history, Stephen Robertson and Lincoln A. Mullen have similarly argued that digital publication formats have the potential to represent “patterns of historical interpretation” but are rarely utilized in that way.¹⁵

Against this background, visualizations like the Pathfinders network graph become a means to communicate something important: It shows the relations that Moulthrop und Grigar consider relevant and intend to express. More precisely, the coherent structure of these relations is immediately (that is to say: simultaneously) visible – something that text cannot achieve. What Haas calls “structures of meaning,” I consider the syntax or ‘architecture’ of presented knowledge. It appears before the eye in an overview. A visualization depicts which elements are closely linked and which ones are more or less isolated, or whatever other patterns form up. And precisely these pieces of structural information support the spectator’s assessment of the logical role of an individual element in the overall configuration. As mentioned above, text tends to blur such relations and configurations if these are highly complex. Many intertextual connections between sources would remain unclear, for instance, if the text were to express the diverse contexts involved in creating cultural works, as Pathfinders does. The complexity increases if an author respects different layers/sediments of time. Processes of historical knowledge transfer may also become pertinent or multiple lines of discourse. Linearly designed texts oversimplify and often cut off these complex interrelations of interest. This is because the text and its flow force us to limit any articulation in order to create a coherent and understandable piece. Ted Nelson, founder of the term ‘hypertext,’ has postulated that the issue lies with the fundamental medial qualities of text: “The problem with creating sequential documents is this: any sequence cuts connections, just as any grouping omits items. The cutting of connections is the loss of information. […] Before hypertext, these problems were intrinsic to writing and publication.”¹⁶ One might object that we would sacrifice the cogency of academic writing, gaining instead over-complex alternatives: a confusing demonstration of any connections. I do not think this objection would be justified. Pathfinders paths, for instance, are a means of creating coherent structures – at the same time, the online publication depicts complexity. Digital history, too, often combines (interactive) visualizations with text to complement detailed data discussion with ‘the big picture’ of connections between data points. 

Visual media certainly portray structural qualities well but barely convey content. They cannot, for example, argue in the sense of unfolding logical reasoning. Neither can they communicate the content and course of narratives. For this reason, I do not entirely agree with Staley when he defines a visualization of historical phenomena as a “meta-narrative.”¹⁷ It is the textual part, as in Pathfinders, that adopts that role. At the same time, it complements the visual display, which makes the whole demonstration so powerful. Consequently, I regard a multimodal design as more fruitful than a purely visual or purely textual design. Multimodality, as understood here, combines the semiotic strengths of both visuality and textuality. Ultimately, this renders possible an essential communication performance: the transparent expression of knowledge structures.

You can comment in German, English, French or Italian under the german version of the blog contribution.

---

Christian Wachter is a postdoctoral research fellow at the chair of Theory and Methods of History, University of Göttingen. He received his Ph.D. for his thesis on hypertext as a medium for digital historiography in 2021. In a current research project, he conducts digital discourse analysis, inquiring about political discourse in the early twentieth century.

https://www.uni-goettingen.de/de/christian+wachter/621240.html

christian.wachter@uni-goettingen.de

---

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

1. Klaus Krippendorff, The Semantic Turn. A New Foundation for Design. Boca Raton (FL): CRC Press, 2005, xiii. [↩]
2. Cf. Christian Wachter, Geschichte digital schreiben. Hypertext als non-lineare Wissensrepräsentation in der Digital History. Bielefeld: transcript, 2021. [↩]
3. Arthur C. Danto, Narration and Knowledge. New York (NY): Columbia University Press, 2007. [↩]
4. Esp.: Hayden V. White, Metahistory. The Historical Imagination in Nineteenth-Century Europe. Baltimore (MD): Johns Hopkins University Press, 1973. [↩]
5. Anthony Grafton, The Footnote. A Curious History. Cambridge (MA): Harvard University Press, 1999. [↩]
6. Stefan Haas, “Designing Knowledge. Theoretische und pragmatische Perspektiven der medialen Bedingungen der Erkenntnisformulierung und -vermittlung in den Kultur- und Sozialwissenschaften,” in Fabio Crivellari u.a. (eds): Die Medien der Geschichte. Historizität und Medialität in interdisziplinärer Perspektive, ed. Fabio Crivellari, Kay Kirchmann, Marcus Sandl et al. Konstanz: UVK, 2004, 211-236, here 233-234. [↩]
7. Jakob Krameritsch, Geschichte(n) im Netzwerk. Hypertext und dessen Potenziale für die Produktion, Repräsentation und Rezeption der historischen Erzählung. Münster: Waxmann, 2007; Wachter, Geschichte digital schreiben. [↩]
8. David J. Staley, Computers, Visualization, and History. How New Technology Will Transform Our Understanding of the Past. Armonk (NY): Sharpe, 2014,142-167. [↩]
9. Ibid., 39-40. [↩]
10. Ibid., 40-52. [↩]
11. Cf. Hartmut Stöckl, “Sprache-Bild-Texte lesen. Bausteine zur Methodik einer Grundkompetenz,“ in Bildlinguistik. Theorien – Methoden – Fallbeispiele, ed. Hajo Diekmannshenke, Michael Klemm, Hartmut Stöckl. Berlin: Erich Schmidt, 2011, 45-70, here 48. [↩]
12. Gunther and Theo van Leeuwen Kress, Reading Images. The Grammar of Visual Design. London: Routledge, 2021, 76. [↩]
13. Josef Mitterer, Die Flucht aus der Beliebigkeit. Frankfurt am Main: S. Fischer, 2001. Josef Mitterer, Das Jenseits der Philosophie. Wider das dualistische Erkenntnisprinzip. Vienna: Passagen, 2000. [↩]
14. Siegfried J. Schmidt, “From Objects to Processes. A Proposal to Rewrite Radical Constructivism,” Constructivist Foundations 7, no. 1 (2011), 1-9; Siegfried J. Schmidt, Die Endgültigkeit der Vorläufigkeit. Prozessualität als Argumentationsstrategie. Weilerswist: Velbrück Wissenschaft, 2010; Siegfried J. Schmidt, Geschichten & Diskurse. Abschied vom Konstruktivismus. Reinbek bei Hamburg: Rowohlt Taschenbuch, 2003. [↩]
15. Stephen Robertson and Lincoln A. Mullen, “Arguing with Digital History. Patterns of Historical Interpretation,” in Journal of Social History 54, no. 4 (2021), 1005-1022. DOI: 10.1093/jsh/shab015. CC-BY-NC-ND. [↩]
16. Theodor Holm Nelson, Philosophy of Hypertext, PhD diss, (Keio University, Tokyo, 2002), 24. [↩]
17. Staley, Computers, Visualization, and History, 156. [↩]

---

---