Jean-Christophe Burie: Position Paper

Problematics

The visually rich publications and especially the comics that will be discussed during the workshop are often poor semantically. This comes mainly from the legacy of the print chain industry where images are never described. The only data and metadata available being often those provided by the publishers which aim to feed the product page for the e-retailers.

This lack of semantics has several consequences

• The addition of a geometric or textual semantics is not or hardly foreseen in publishing standards such as EPUB.
• Indexing of these contents in search and recommendation technologies is very basic
• The accessibility of such content seems utopian today

Extracting the semantic content from BD/Comics/Manga

Understanding aspect: Use of Machine learning and Artificial Intelligence

The SAIL has developed different robust methods to extract the complex layout (+ the reading order) of the comics and also to detect and to recognize: the text, the faces of the characters and the characters themselves in ALL comics artistic styles considering the cultural heritage of many countries. The main difficulty come from the diversity of drawings. Each author has his own drawing style, his own writing style.

Some of these methods are based on machine learning and artificial intelligence approaches. The developed research is able to extract elements such as panels, speech balloons, text, named entities, faces, mains characters from digitized comics and in a near future born-digital comics based on the EDRLab BDCoMa Working Group.

Representation aspect: CBML

An important part of the work consists in the understanding of the content in order to obtain a semantic description of the digitized or born-digital “document”. In the comics/manga filed the SAIL has chosen Comic Book Markup Language which is an extension of the TEI. The language mainly focus on digitized contents and a work has started to enrich CBML format by including some characteristics of born digital document.

On one hand, the publication (EPUB or Web publication), its future developments and one the other hands its representation in CBML (or other?) are topics that must move forward together. Discussions with people from the web and from the publishing on these topics should be interesting.

For which use?

The precise extraction of the layout allows to propose automatically a panel by panel reading for non manga content and more generally of any documents. The detection and recognition of text could improve the accessibility by providing text speech for visually impaired people. Some advanced uses like colorization of some words for dyslexic people could be considered.

More generally, the extraction of any elements (faces, characters, objects, part of the decor) will allow to develop interactivity between the reader and the device and is the basis for an audio description of the visual rich publication.

Goals

The objective of this proposal is to present what it is possible to extract and to understand from automatic analysis and then to stimulate discussions with experts of the domains to propose either new kind of interactive publications or how the extracted information could be embedded into a future technical standard in order to create new forms of digital books.

About the S.A.I.L

The Sequential Art Image Laboratory is a common laboratory between the Computer Science department of University La Rochelle and Actialuna/Sequencity, a digital comics platform and supported by the French National Research Agency.

Background

I am full professor and assistant director at L3i laboratory in La Rochelle University France. My research topics are image processing, image analysis, pattern recognition, machine learning. In 2011, I and my team began the e-BDtheque project. The aim of this project is to extract relevant information from comics, mangas or “bandes dessinées”. Then a second R&D project (iiBD Intelligence in Bande dessinée) supported by the French Government and leaded by Actialuna/Sequencity has been carried out to develop innovative services. We also work in collaboration with researchers from Osaka Prefecture University on a project dealing with automatic manga analysis.