GraphManuscribble

Within HisDoc 2.0 we have developed a new user-centred and intuitive tool GraphManuscribble for directly interacting with a digital facsimile of a manuscript; particularly to segment, extract, or mark its contents. It exploits the new human-computer interaction patterns evolving around touch-screen devices that are operated with a stylus, such as Microsoft Surface, and builds upon document graphs that capture the structure of a manuscript similar to human perception. Specifically, users of the tool scribble directly onto a digital facsimile of a manuscript in order to select or annotate manuscript parts. They are assisted by a semi-automatic system that facilitates imprecise interactions since accurate operations such as the region segmentation are done automatically. A user study including participants who did not have any experience with pen-based interaction has shown a promising potential of the proposed tool.

Graphs in the context of computer science are a powerful means to model structural relationships. They are capable of representing data in a fashion similar to our perception given an appropriate definition of their topology and criterions for defining relations: graphs are a set of points connected by edges. To fix ideas let us explore the analogy of maps, more particularly road networks, and graphs. Graphs capture entities and their relationships rather similar to a road network connecting neuralgic points in a city. In the network, there are roads that are more crucial than others, are wider, have more lanes, or a higher speed limit, and thus, higher traffic throughput. A graph can similarly encode different relationship qualities in its edges.

We propose a user-centred system that is inspired by the humans’ perceptual capabilities and aims at representing the document image in a way that resembles our understanding using graphs. The goal of our first prototype tool is to assist a user in selecting and segmenting entities in a manuscript in an intuitive manner guided by the humans’ connotation of structure and belonging in a document. The user edits this representation with a stylus on a touch-sensitive screen. The range of application of such a tool extends from extracting parts of a manuscript e.g. for selecting samples for word spotting or illustration retrieval, in order to find other appearances of the same or word or illuminations, decorations, or drop caps; to a tool for scholars that empowers them to work on the digital facsimile of a manuscript in an intuitive and natural fashion. A scholar is empowered to directly mark and annotate a facsimile with simple and intuitive tools.

Binary image with the graph in the background (green)        Polygons resulting from the user annotations
and user annotations (scribbles) in purple, magenta
and cyan (for different classes)

We conducted a user study with the aim of assessing the efficiency of the tool for annotating complex historical manuscript pages, the user fatigue, and the quality of the proposed scribbling interaction pattern. It demonstrated that our tool and the interaction with a pen on a touch screen that displays the facsimile does neither require prior training, algorithmic knowledge, nor extensive computer skills.

GraphManuscribble Videos on Youtube

GraphManuscribble Use Case: Segmenting an illustration in a historical manuscript with a few scribbles.

GraphManuscribble - the power of graphs explained visually: Why do we need a graph in the background? How does it make our life easer?

GraphManuscribble - the power of graphs: using the graph for segmentation vs drawing polygons by hand

Publications

A. Garz, M. Seuret, A. Fischer, and R. Ingold (2016). A User-Centered Segmentation Method for Complex Historical Manuscripts Based on Document Graphs, IEEE Transactions on Human-Machine Systems, (to appear).

A. Garz, M. Seuret, F. Simistira, A. Fischer, and R. Ingold (2016). Creating Ground Truth for Historical Manuscripts with Document Graphs and Scribbling Interaction, International Workshop on Document on Document Analysis Systems, p. 126-131.
PDF

A. Garz, M. Seuret, A. Fischer, and R. Ingold (2016). GraphManuscribble: Interact Intuitively with Digital Facsimiles, Second International Conference on Natural Sciences and Technology in Manuscript Analysis, p. 61-63.
PDF

Software

We plan to release GraphManuscribble to the community after summer 2016.

A bit of fun with GraphManuscribble - Annotate wherever you are ...

@ DAS 2016, Santorini

GraphManuscribbling @ Turmbar, Hamburg 2016