Meeting minutes
<Wendy Reid> /w3c.zoom.us/j/88131559816?pwd=KGwOotiwS2iBagzxHeCxK0zUHEEjf4.1///docs.google.com/document/d/1FfUjiK8PrKfqeVCAnjpgZ_rpq7ZSAziloyLT5AkZKjQ/edit?tab=t.0
EPUB 3.4 Testing and Documentation
We are in a good place, except we haven't received many test results
Wendy Reid: We are in a good place, except we haven't received many test results
… we will probably have to do our own testing
… and we can now flag that as third party vs implementer
… If we get to the point where we need to do it, we will inform them that we have done the tests and give the a chance to add their own
Ivan Herman: Good news - the big challenge was the roll feature, since that was the only new thing. And we have already received 2 tests, so that is done
… The other thing, for non-roll we have old tests (some have changed names), maybe we could take them over
… Fresh tests would be nicer, but it is an option
… Charter ends Feb, so we really want to get the tests in and voting done by end of January
… I would prefer tests in by beginning of Dec
Wendy Reid: So please get any results in by end of Nov
Gautier Chomel_: I am planning the tests in Dec
Ivan Herman: We may have to wait then
Wendy Reid: Mid December may be ok, we only need 28 days for the AC vote
Brady Duga: Trying to understand who I need to bug about testing
… it takes a while and is a lot of work and people are busy
… is it better for us to let people use the old tests and move on?
… do we need fresh tests
Ivan Herman: For things like Apple/Google there haven't been many changes, so we can probably reuse
… the reason I am putting Gautier Chomel_ aside is the Thorium has changed a lot
… so if we have Apple/Google then I am fine using the old tests
… but we still have to figure out what tests changed names
… so I agree we could take at least these two
… and we have some Voyager that are older, and we have some new ones, we could merge them.
Susan Neuhaus: I talked to production people and there was enthusiasm for participation. But this is very reading system focused.
… maybe for the future we could open this up to production side people
Gautier Chomel_: I think it is concerning that there hasn't been much evolution in readers
… And I agree that having producers involved would be good
Wendy Reid: Maybe the better way to put it is the big players are quite stable, and not much has changed
… I don't want to give the impression they aren't evolving, it is just that they favor stability
Hadrien Gardeur: On our side with readium toolkits on iOS. We have added support for images in spine, with and without fallback, mixed layouts (iOS only), still not to roll, but it does impact some tests
… There were some Readium systems that were tested, but Swift, iOS, and Kotlin are different right now
… we plan to move it to Kotlin next year, but that is too late. We aren't really unique, adding features takes time
… in general this can be a problem (timing), since we won't have everything done when the spec releases (e.g. roll)
Wendy Reid: That segues to tools being part of the test process. And we will need to do something about that in Annotations (are the proper annotations produced)
… For annotations there are two sides, testing the result of export and testing that import works
Ivan Herman: I don't know how we test that - do we describe scenarios?
… so how do we provide that description?
Brady Duga: We do need to look, we only need to test the RFC 2119 statements, they are much more specific and clear
… should we describe what you must do and allow
… we don't have conformance statements for these behaviours, we don't even say a human has to do anything
… a LLM could do it all
… we have to be careful about what we're testing
… currently we don't have many conformance statements
Ivan Herman: Which was very intentional, as we only have an interchange format
Wendy Reid: Though we are still figuring it out. I expect we will add a lot more. Perhaps understanding what needs testing will help direct the spec
… what do we want to test?
<Ivan Herman> /join #tpac-chat
Laurent Le Meur: First we must test the spec, not the RS
… most of the tests will be driven by that
… I don't think we want heavy conformance statements on reading systems
… We have 3 points:
<Shinya Takami> /w3c.zoom.us/j/8603454797?pwd=PXWIgtQr0SCEtRI53FtSDEbeg4nV5X.1///w3c.zoom.us/j/88131559816?pwd=KGwOotiwS2iBagzxHeCxK0zUHEEjf4.1
Laurent Le Meur: [preparing to present slides]
… First test is can our samples be imported
… using different selectors and refinements
… we will need about 5 sets to do that
… we also need to have changed epubs to make sure they still work (for some selectors)
… we need to make sure they produce valid json, which is easy once we have the schema
Hadrien Gardeur: This doesn't handle selectors
Laurent Le Meur: Yes, that is still an issue
… and we need to make sure orphans degrade well
Ivan Herman: we need to make sure the interchange works
Laurent Le Meur: For that we will need 2 reading systems
… currently at the table we have thorium and zotero. Others are interested but can't do it soon
Ivan Herman: Annotation will be in the next charter, so we have some time
Laurent Le Meur: We do have 2 thorium implementations that are completely different
Ivan Herman: AC review might be an issue, so they may want proof
Laurent Le Meur: Not a problem. Also we need someone else to implement it
Hadrien Gardeur: we need to reach out to some more people. We often talk to the same set of people, which is a problem
Susan Neuhaus: Will we provide the slightly changed document as well?
Laurent Le Meur: Yes, we will provide that
Hadrien Gardeur: Some implement KO reader sync. It is not the same thing, but they kind of support annotations
… So there are a lot of reading systems that support this, but they do it with sync instead of export
… and lots of people are annoyed at KO sync. So there are a lot of people interested in this, but doing it as a protocol
… There is an active community that is interested in self hosting, which hasn't been on our radar
Wendy Reid: There is KO, there is also Calibre annotations, so off the top we have 2 communities to reach out to
… They don't participate here, but they are enthusiastic implementers
… colibrio is another reader we should connect with
… We do want to do more outreach with the next charter
Ivan Herman: We are in an interesting position, the features we are discussing for the next charter are more for these communities than the traditional ones we have connected with in the past
Wendy Reid: Any other testing topics that aren't part of the charter?
… When we designed the epub test site, we did it with 3.3 in mind. We might need to consider how we change things for new versions (let's never change names again)
… should we have a central test repo for all the tests, including annotations?
Ivan Herman: No, that will be a mess. It will make it too confusing, so we should have a new repo for annotations
Wendy Reid: But we do want a central home for all the tests from a documentation standpoint
Ivan Herman: Repos are cheap, and I don't like the old IDPF way of a giant repo
… some groups have lots of repos
Wendy Reid: Readers of the spec don't care about github repos
… we can have multiple repos, but do we have one home?
… So the tests tells us what works in epub, having one place to see them makes sense
<Ivan Herman> w3c/
<Ivan Herman> w3c/
<Ivan Herman> https://
Ivan Herman: We had a repo set up as a response to this request
… we could make it an entry point for epub in general
… people try to get to one place called "epub" so we set this up as a place to go
… but it needs work.
Documentation
Gautier Chomel_: Documentation has come up in the CG as well
… People are looking at MDN, etc for epub info
… we also have the DAISY knowledge base, which focuses on a11y.
… do we want something similar for the full spec? It is all a big question
… we had a TF for documention, but they left no documentation
… Maybe this is a good thing for the CG
<Charles LaPierre> https://
Gautier Chomel_: I love the idea of the epub repo to do that, even if it just gets exported elsewhere. People need an entry point for EPUB
… I propose we take this to the CG
Susan Neuhaus: I was thinking about this a few months ago and created a sheet of all the documents we already have
<Susan Neuhaus> https://
Susan Neuhaus: if you think of anything I can add you as an editor to the sheet
… but we might need multiple entry points (e.g. reading system dev vs indie author)
Ivan Herman: At some point someone (Tzviya? Dan?) was trying to get us set up with MDN
… I just don't think an MDN branch for epub is realistic
… Moving docs to CG is a great idea if we have the manpower
… which group is secondary to who will actually do the work
Wendy Reid: There are 2 sides to the project. We have always had the question of MDN for epub.
… MDN is always very open to it if we provide the info
… we have terrible organization for the existing docs
… Searches for some topics come up for IDPF sites
… A fantastic first effort would be to use the repo as a landing page with links to all the normative and informative documents
… once we have a home for that, then maybe we can add new documentation
… but a great start would be to organize what we have now
… this would be a good CG topic
<Charles LaPierre> EPUB Official Documentation Search Engine (Created by Brady)
<Charles LaPierre> https://
Charles LaPierre: Right now the best way to find things is the search engine
Gautier Chomel_: There is manpower that will be happy to contribute
… we have to clearly define a strategy. Once we have that we can find the people to help with the actual work
… a lot of people are building their own docs, let's get them building it for official docs instead
… But github can be scary for some people so we may want to accept contributions without needing github knowledge
… so we should have a group that can discuss strategy
Ivan Herman: CG is the best thing to do, but we should have it on our agenda in the WG to report/cross-pollinated
… so the first point is we need more people. 3 or 4 who want to work on docs.
Wendy Reid: It is essentially joint. So we want people with knowledge, but we also want people from the other side who want to help us improve
… In the WG we become numb because we know the repos to hunt through
… I like a joint thing
… sometimes we have gotten too ambitious, everything is written somewhere, we just need to collect it
… there is the KB, there are blog posts, etc
Brady Duga: One issue I found looking for things, was links to pages of links of presentations, that took you somewhere unrelated
… the original document was lost, we have lost a lot, it will be a task to find the content and eliminate old stale, links
Shinya Takami: We have notes from CG and WG, should we distinguish these?
Ivan Herman: It isn't clear to me why some are WG and some are CG
… when we started, several notes were created by pulling from IDPF specs or documents. A lot of those ended up in WG.
… Then the CG has done some work which they published as notes
… There is no systemic content reason for it, it is just historical oddity
… CG notes have been recently reorganized by Matt for a11y documents
Gregorio Pellegrino: There are a lot of things on stack overflow that are just wrong
… Just a warning. I think the effort will be important
… in terms of notes, the WG requires being formal members, so some notes went to the CG for better community contributions
Wendy Reid: It matters a lot to us, but not the community. WG Note or CG final report is the same to everyone else in the world
… we have to very important docs as CG reports (eg a11y ux report)
… some of these might want to be WG notes
… we might want to consider a pipeline for this
… There are probably a lot of documents meet this criteria. Fine to do the work in CG, then elevate to WG note
… There has been some consolidation
Susan Neuhaus: We have discussed taking down the IDPF site, since it is out of date
… do we want to forward? 301 rediirect? Just 404?
Charles LaPierre: I was going to suggest the same thing - a path to move from CG to WG
… Some of this has to happen in the CG, but it gives it some weight to be in the WG
… we do have to think about how we still allow CG contrbutions
… As to IDPF, some information may get lost
… we don't control that
various: Not true, we own the IDPF website
Ivan Herman: It is off to the side and kept so it isn't lost. There is limited access to it. We should see if we can do something with it
… So we could just add a redirect to homepages, but we can leave deep links
… originally I was going to disagree with CG and WG
https://
Ivan Herman: We just add to the confusion of having a report duplicated into the WG. The community doesn't really care, and having two copies is terrible. So it adds no value to the community
Hadrien Gardeur: I don't think the community has any idea of what we are talking about. CG, WG, note, spec, they don't know the difference. Keep that out of the way for.
… we also have a bunch of old useless notes in the WG from old IDPF days, but then we have notes from the CG that are more important than specs in some cases
… We have to think about this in a more practical sense - which are important, not the track they are on
… I use the old IDPF site to tell what the changes are
… removing them would be painful for me
Ivan Herman: We can't remove them
Hadrien Gardeur: I would be careful with redirect. A big red banner pointing to the new version would be fine
<Ivan Herman> qq
Hadrien Gardeur: But the old specs are very important, especially for us
Wendy Reid: I think the latest proposal was a banner
Gautier Chomel_: First we need a clear epub web page
… A11y display guide is one that has been very quickly adopted
… partly because of legal reasons, but we had a lot of people working on it, so the process was very good
… Next step migh to discusss with Brady Duga and Susan Neuhaus about launching a TF
Wendy Reid: I agree with Hadrien Gardeur. A lot of this is exposing our quirks to the outside world, who do not care about our quirks
… The display guide is a great example. It is really important and heavily used, but it lives in a weird place. Having a stable URL is really important for it
Charles LaPierre: When we do a final report in the CG, is there a WG vote?
various: no
Charles LaPierre: Can we? Some of these reports never make it to the WG, they never see it
Ivan Herman: Process wise they are entirely unrelated. To bring it to the WG, we would have to make it a WG note, go through a transition request, get a URL, etc. It's too much
… informally, we could agree with the CG that the WG would give it a nod to say "yes we saw it"
Ivan Herman: W3C is very picky about not removing things from the web
… so We will not remove 3.0.1
Wendy Reid: We wanted to take a moment to recognize Avneesh
… For the Digital Publishing award at the Frankfurt book fair
Avneesh Singh: I would thank Bill K and Cristina for. And DAISY, and of course my wife who made it all possible
Wendy Reid: [Break]
Annotations
Laurent Le Meur: (shares screen) here is the status
… we started 3 years ago, implemented it on Thorium desktop
… focus on EPUB, multimodal applications, removed CFI selector, text selector instead, less publication metadata
… we have a stable draft now, with a few issues about wording
… we can discuss the issues that are more structural
… first, which publication metadata is most useful?
… how to order annotations by reading system?
… which detailed processing model at export?
… should annotations be human readable?
… how to test?
… most important is the human readable question
… use case, if I push annotations to a website, and I want access to my annotations, but the website doesn't have the original publication. If I want to see these, we would need some context in the annotations
… the next usecase— if I export annotations, and then lose access to the book. but I would like to turn the annotations into human readable citations.
… should annotations be transformable into human readable annotations?
… to give context to the human, we need to export the annotated text, and their may be copyright or other issues
… this would require more information about the context like chapter, page, position
… why would we want to convert json to human readable when the RS can already do that?
gpelligrino: another use case is users with dyslexia, who may need to export annotations and be able to rearrange or reformat them including the original text
Ivan Herman: an annotation is the combination of the note and the annotated text itself. Do you translate the annotation selector into something a human understands, as opposed to CSS
… we agreed previously that this would be text only, with no formatting. What exactly does it mean to transform this to human readable?
Laurent Le Meur: (shows slide with kindle and annotations output)
… you get the annotation book name, the page and location text, date, and your note
Ivan Herman: we have another spec for identification, which is json. Another discussion is render methods. It is very complicated. There are so many approaches. Even starting with the transformation engine and the html display.
… I am worried about introducing this here
Hadrien Gardeur: we have not looked enough about how this is done elsewhere
… a lot of the big RS are fine with having some of the original content in the export
… you can have a selector that is useful for the machine but not the human
… KO Reading and Calibre both include the whole text
… we could have a requirement for that. This discussion is important and would have big impact on users
… secondly about Kindle, they don't have original content or a relation to the table of contents
… in colibre they have a reference to the TOC, like "chapter 1, section 1"
… a lot of formats do this. We could probably provide a bit more guidance to align with what is already out there.
… I'm not a fan of location numbers, since everyone does that a bit differently.
Brady Duga: Ivan Herman mentioned the complexity in rendering, where is the complexity you are worried about
Ivan Herman: in both the formatting and the export
… is this really something we have to standardized? Or can the individual reading systems ready to provide this? what do we have to standardize?
Brady Duga: just for a basic display, Playbooks includes the chapter title, and that's something we have to standardize. We would need access to the book to know that.
… we don't want to make something broken from the start, we could just say this isn't a usecase that works in 1.0
… or we could say, here's a thing you could use, we don't know how, but it is up to the RS to figure out. This could also cause problems
Wendy Reid: For a lot of RS and publishers, there is an agreed upon number of characters that can be exported. If the publisher doesn't allow that, we can make annotations, but we cannot export them. You just get a message that the publisher doesn't allow this
… this poses an opportunity for a fix for this. Perhaps if you share your annotation, even without prohibited text copied, we could still share it. The next user could find the annotation in the book based on the locator.
Benjamin Young (observer): with the amazon example and the location number mismatch, its kind of on them to fix that. I think you've got pieces already, and you'll be affected by limitations, maybe you are OK
Laurent Le Meur: It depends on what we want to do and who wants to go there
Benjamin Young: maybe you could take chunks of metadata and put it into something like YAML LD
Hadrien Gardeur: About metadata, in KO readers they have an option to include other metadata, saying we don't use it, but it might be useful to someone else.
… Colibre keeps this in the database, but hasn't really solved the problem.
… KO reader has basically the title and the author
… with selectors it is usually the highlighted text plus another selector. The second selector isn't consistent across systems.
… there is quite a bit to learn by looking at what has been done
Brady Duga: when we originally talked about the annotations, we talked about the text people type in, which is pretty basic. But as we talked about exporting this in a human readable way, I don't think we can ignore markup.
Laurent Le Meur: It is good to have some proposals—
… here is something in between doing nothing and being too prescriptive
… (shows slide) this includes the fragment selector, like the beginning and end of selected text.
… I've added two things, one is context, where you can include Chapter, section or page#
… and "progression" which is a float, and can represent the progress in the book, like 20%
… we may change the name from context to avoid confusion
… these new things will give the reading systems some options
Charles LaPierre: did we decide if we support annotating images that don't have any text
Laurent Le Meur: the context cannot be as accurate but we can annotate images
… perhaps you could provide the alt text
Hadrien Gardeur: since the plain text discussion was based on what the user can put into a note, some system does have support for HTML markdown.
… sometimes the metadata changes by the type of publication, for fixed layout and PDF they use page numbers, for audiobooks they use timestamp. It gets tricky with reflowable
… progression could be useful and pretty transferable. Probably the scroll position on the y access. Sometimes it is difficult to calculate but could be still be useful. Media overlays will have an id or timestamp
… to include the original text if possible it is most useful. Especially if someone loses access to the initial publication.
George Kerscher: As a student reading stem content, will I be able to annotate tables and math ML
Hadrien Gardeur: yes, but it will be trickier if you lose access to the book
Wendy Reid: I agree we should change "context" to avoid confusion with "@context"
… about progression, I think different systems calculate that differently
Laurent Le Meur: with other information, the progression will be useful
Ivan Herman: will we specify what progression is?
Laurent Le Meur: we will leave that to the reading system
Ivan Herman: before we adopt this, we will have to go back to the contacts we have who are already doing this, and see if this makes sense to them.
Benjamin Young: I'm curious about your use of the fragment selector
Laurent Le Meur: this is a new idea, we explicitly state that we hope this will become a standard.
… this is a new way to do what the text selector was supposed to do 10 years ago
Benjamin Young: If you want to display the start and end of your quote you'll have to separate them.
Ivan Herman: since this isn't already standard, we are kind of off the hook, but browsers already do this
Hadrien Gardeur: we decided to use this because there are complexities, and this isn't ours to solve, since these are inherent in HTML. It's heavily used even if it is under specified. We didn't want to be struggling with unicode and white space. We though it is safer if others do that
Ivan Herman: we can only signal that this is needed, and there are open issues in the HTML working group. We can hope. Practically all the browsers provide a way to use this
Benjamin Young: that is encoding the selector, but not the display, and it may break when you hit emojiis
Wendy Reid: we should also consider, are we demanding that they must use this selector, but we are offering it an option. There are other selectors you can use. Like CFI
Ivan Herman: we will talk about that tommorrow
Wendy Reid: people will ask if we don't include it
… is what Laurent Le Meur has proposed enough that we can start getting some feedback from people outside the group?
Charles LaPierre: In addition to the selector itself, why couldn't we add another text block with whatever markdown we want. The machine can read what it can read, and humans can read this part.
Hadrien Gardeur: that was my point, we've been arguing about this for ever, with text fragments, etc. It might not be a bad thing to have both human and machine readable. Maybe text only
Benjamin Young: you could include the annotation creator, an @id, a valid version of the identifier, and you'd still get the version you want
Laurent Le Meur: the text will be transformed before display, you make it base 64, you can show it as html.
Ivan Herman: will that make it unnecessarily complicated?
Laurent Le Meur: the detailed process model (shows slide) Normative? or are we OK with going with the current Expectations of reading systems
… do we have to do this normative work and what would it look like
Brady Duga: those are extremes, perhaps we can do some simple things. What if I get an annotations file that isn't json LD, should I try to process it? Do we clarify if someone is right or wrong? Should we offer some guidance about things that don't match or throw errors. If we don't handle these edge cases, we will get inconsistent results. We should
… have some clarity about what to expect.
… there is a consistent processing model in css for handling a "//" even if it is broken, everyone knows what to do with it
Laurent Le Meur: I'm not sure there is a practical difference here. In both cases there will be good and bad citizens
Ivan Herman: the difference is if we have to test it or not
… what this also shows, is that this goes beyond what we describe as an exchange format but addresses what the reading system does with it
… we should specify what to do if things go wrong. We need at least some levels of guidance for failures
<Wendy Reid> https://
Wendy Reid: I've shared the link from epub reading systems section about error handling
… it might be as simple as this, we can say
"you are not required to display invalid content" and leave it up to the RS to decide how far to go with broken content and display.
… I could see annotation being similar, there will be situations where the annotations are garbled, or simply incorrect color
Ivan Herman: this is consistent with our standard
Charles LaPierre: how concerned are we about security?
Laurent Le Meur: there is a section in the draft about privacy and security
Brady Duga: we're in a different situation with EPUB. A lot of EPUBs get rejected by publishers. With annotations its more like CSS and HTML, there will be more variation, and you can't go back to the original annotator for a new file in the same way. HTML has a well-defined error model, so we could give them some guide, a complete model, or just let
… them work it out
Ivan Herman: the person creating the annotations is not the one creating the exchange file. The reading system where the annotation is created is required to produce something valid
Marisa DeMeglio: is there a place to say which reading system made the file
Brady Duga: we took that out
Wendy Reid: we removed it for security and privacy reasons
Wendy Reid: this is tricky, I don't want to give people nothing about error handling, but in this case we probably don't want a bad user experience, but we have a relatively set structure.
… we could brainstorm some of the error situations, like the locators are not present in this book…
Brady Duga: some of the errors are expected, and we should say something about what to do. Other things may come up, like if someone uses a selector that is not used in our spec?
… what happens if there is a missing break in the json?
Wendy Reid: if the json isn't working, can we go look at the json LD processing model
Benjamin Young: with malformed json LD the system will output what it can
… there is incentive for doing json processing especially if you're using multiple languages
Laurent Le Meur: a reading system will use the "open json" command and if it isn't right it won't display
… like Brady Duga said, perhaps we could have a small set of rules
niklasl: on the LD side of things you can read the partial stuff
Charles LaPierre: are we considering creating something similar to EPUB check for validation?
Laurent Le Meur: yes, it would be based on the json schema. And include more than semantics, we could have something that would check the book, but the CSS selector may or may not provide the right text
Brady Duga: we may not even know what is wrong, the URI may not even exist in another edition of the same book. I think in practice people can always use text selectors because it's easy to search
Charles LaPierre: it might be useful to have the checker, to make sure it validates on LD, and we might be able to say that a text selector is correct, but that might be out of scope.
Wendy Reid: we agree would like some kind of model, maybe something light weight.
Ivan Herman: we still have to agree on the name of the annotation file
<Benjamin Young> https://
Benjamin Young: I added a link about the use
LUNCH
Wendy Reid: with that we could break for lunch
Sync Media and Media Overlays
Marisa DeMeglio: We have a new community group is called the...
<Marisa DeMeglio> Accessible Synchronized Multimedia for Publishing
Marisa DeMeglio: the idea is to have an open place to discussion the revision on media overlay for publishing and the web
… we're meeting every two weeks, we're using GitHub and issue trackers
<Gautier Chomel> w3c-cg/
Marisa DeMeglio: we have 14 use cases till now
… from use cased we create user requirements, out of user requirements we make technical requirements
… a new topic is synchronized video in EPUB
… another topic is multiple granularities
… it's the first major revision of media overlays, maybe this work may go in EPUB 3.5
<Charles LaPierre> use cases: https://
Marisa DeMeglio: another topic is metadata for multi tracks publications
… another topic is to maintain assistive technologies access to the content
… like someone is listing a narrated audiobook, and they want to dive deeper, and they want to switch to screenreader
… usually the screenreader is not synced with the narrated audio
<Marisa DeMeglio> w3c-cg/
<Marisa DeMeglio> w3c-cg/
Marisa DeMeglio: I think that having it in EPUB 3.5 is important, we will not change the whole spec, but we'll make some refinements
Use case: Life-critical public information for residents with disabilities in a nuclear emergency planning zone (w3c-cg/
(reading the description of the use case available at the posted link)
… this is a very rich use case
Use case: A publication that ages with its reader — deafblind reading across a sensory trajectory (w3c-cg/
(reading the description of the use case available at the posted link)
… maybe we'll not cover all these use cases in EPUB 3.5
Hadrien Gardeur: my own perspective in working with media overlays
… I'm really interested in accessibility for manga
… I've spoken with several specialized libraries who have produced accessible comics
… they made them using pre-recorded audio
… what they did worked in some special reading solutions, not standardized
… since the object model defined by media overlays in EPUB, they had to reference to text elements (even when they were referencing images)
… so I opened an issue to suggest to enable the possibility to sync audio with images (and potentially portions of images)
… I was discussing it because I was working on releasing the iOS Thorium version
… digging into the spec I've found out that every reading system was implementing the spec differently
… and here is where we are: we don't have interoperability in media overlays
… most of the work was based on Apple implementation for fixed layout content
… but most of specialized libraries are interested in reflow content with media overlay
… that's why I think we should make a big effort to defining the roots of media overlays
… with interoperability in mind
Hadrien Gardeur: not sure how much we can test in the EPUB 3.4 test run, but we'll find differences
Brady Duga: the way media overlays are implemented are different from how were designed
… a lot of this work was made retro-engineering a specific publication
… I think we need to have both sides of the people at the table: the content creator and the reading solutions
… may you explain the sign language use case?
Marisa DeMeglio: we have an ebook with audio, and we want to synchronize a sign language video
Brady Duga: is this the sign language version of the audio, or is it the sign language version of the text?
Marisa DeMeglio: both
Brady Duga: for the community group: how do you see the collaboration with the Publishing Maintenance Working Group?
Marisa DeMeglio: it's something to discuss today
Marisa DeMeglio: originally media overlays came from DAISY books that were always reflowable
… it was implemented on fixed layout because of difficulties in implementing it in reading systems
George Kerscher: when Steve Jobs announced the iPad, when EPUB 3 was coming out, they see a big market for audio narrated children books
Ivan Herman: I'm trying to understand the direction: take what we have and add some SMIL spec and add it to EPUB
… another possibility is to use another technology instead of SMIL (without deprecating SMIL)
… another possibility is to use SMIL syntaxt but to define our processing specification
… we have all the possibilities with the new charter
Marisa DeMeglio: I don't know what we'll do...
… but I agree it's time to look at all the requirements, and then decide which path to take
Ivan Herman: the bad news is that we have to define the charter before Dicember
Hadrien Gardeur: for all the issues I've opened, SMIL is never the problem, it's the way we adopted SMIL in EPUB
… speaking with creators (like MTM) who produced a lot of EPUBs with Media Overlays
… it's clear that the ecosystem is lacking
… I think we can just take what we have and implement it, instead of being too ambitious
Marisa DeMeglio: I think we should make the implementation not difficult (complex)
… I think we made an issue following too strict the rules of SMIL
… that makes it difficult to be implemented
Ivan Herman: please elaborate more
Marisa DeMeglio: we've put a lot of restrictions on our "profile" of SMIL
… like you always have to have a text element
… or having a single audio track
… there are other specs that are simpler like web VTT
… it's not a perfect solution, but may be worth to explore
Ivan Herman: we adopted around 10% of SMIL
Marisa DeMeglio: it was defined before HTML 5 and media support in browsers
Laurent Le Meur: Using the XML structure in SMIL is an issue, but the strength is to being able to map fragments in one content, to fragments in another content
… this is powerful for the use cases we have
… I think VTT is less powerful in this mutual sync
… I think that get rid of SMIL is wrong
… also to have another technology along with SMIL would create a mess
… I agree to improve the supporto of SMIL in EPUB
… like support other types of references to the content (apart from IDs)
Marisa DeMeglio: I don't disagree
Wendy Reid: EPUB is around 15 years old, SMIL is 20
… in a perfect group we may ask to the WG maintaining SMIL to update it
… but I think there is no one
… the question is: how can we profile SMIL to work better?
… because the spec itself will not be updated
Susan Neuhaus: why do we have to express a direction in the charter?
Ivan Herman: there is a tendency to be specific in charters, so to have some direction
Marisa DeMeglio: something that we'll never resolve is if we're trying to refine SMIL
… instead of get rid of SMIL
Laurent Le Meur: in the use cases I didn't see any requirements that may not be supported by SMIL at the current status
Marisa DeMeglio: the SMIL in EPUB is divergent from SMIL 3
… like the way we defined the text elements
… we intend references, in SMIL they put the text directly
Ivan Herman: looking at the RS specification and EPUB specification
… the dependences to SMIL specification is really weak
… we can even say in the spec that we were "inspired" by SMIL
Hadrien Gardeur: I don't think SMIL is the issue, I think what we wrote in the spec is the issue...
… because the use case was narrated children books
… what I see is people that want to use SMIL in EPUB
… but they're now able
… I want that people that want to make accessible comics can make it
… my idea is to get back to SMIL spec and understand what we need to adopt to make the situation better
Samuel Petit: I'm specialized in manga and comics, speaking about UI/UX use case
… I share some slides to display different needs for different types of disabilities
… you need different ways of accessing and navigating the content, for different types of disabilities
… displaying a demo of a reading application with panel by panel navigation
… to make this user experiences we're using a lot of AI to understand the context, not only the text (emotions, background, main text, characters)
… I'm not sure if SMIL is enough
… in comics we need sound, visual effects, panel navigation, etc.
… and we need them all to have an accessible experience with comics
… while respecting the creativity of the author/publisher, having the possibility to offer multiple way to access the content
Hadrien Gardeur: the example I've build supports most of the things you showed
… the only thing missing is variants of images
… same thing for variants of text
Samuel Petit: in Japan there is a publisher making alternative versions of comics for users with disabilities
… another possibility is to click on a bubble and to listen the audio only of that bubble
Hadrien Gardeur: the problem we had with variants is related to what we had with multiple renditions
… I think this is bigger than media overlays
… I propose to split che requirements/expectations
Wendy Reid: the complication of different accommodations and to have a single EPUB to fit them all is challenging
… I think this research goes in the "accessibility experiments" in the charter
… where we try to test new things
… I think we should split into different requirements for SMIL, for EPUB, etc.
Ivan Herman: do we have to reopen the multiple renditions?
all: noooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
Hadrien Gardeur: synchronizing two media together works (audio - text, video - text, etc.)
… this is possible with the specs we have
… what we cannot do at the moment is to have multiple variations of the same content (example simplified version, gray scale version, etc.)
… I think we can dream in steps, one step at a time
Samuel Petit: we're building a software for creating accessible comics
… but we need the spec to support everything
Marisa DeMeglio: the use case we have is not about synchronizing media with text, like having the EPUB and the eBraille version
… to being able to switch from braille display (eBraille) to screenreader (EPUB)
Ivan Herman: I think that having some kind of multiple alternatives for content it's something that would be useful
Hadrien Gardeur: I took part of the specification of multiple renditions
… the key concept is to have multiple reading orders for the same content (fxl vs reflow, french vs english)
… the other concept is to map equivalent resources
… I agree that we don't need multiple OPFs to manage it
… but it's a long journey
Ivan Herman: if we have to discuss it, it's now
Charles LaPierre: I don't see problems: we're synchronizing video, with audio, with text, with eBraille (potentially)
Ivan Herman: but it doesn't cover the same language
<Gautier Chomel> just a quick reminder that PCG proposed a simplified MR guide https://
Avneesh Singh: the other usecase is the one from Janina (APA)
… speaking about SMIL: if we want to do something about SMIL, we should have a limited scope
… at the same time the Sync Media CG continues as a research project
… and maybe we'll adopt something in the future
… so I propose to use limited SMIL for the moment, and at the same time look for another solution
[BREAK TIME!]
Accessibility for Manga
Shinya Takami: DPFJ is an organization of publishers who has prepared a presenation for us
Suzuki: Thank you for having us, accessibility for Manga
… [shares slides]
… introduction, from the chairman, and from the agency for cultural affairs
… technical issues by Shinya Takami
… content issues with demostration presented by Kodansha
… DPFJ.manga industry shared framework for manga accessibility developed with government, publishers, and RS developers
… DPFJ mission is to share Japan's culture globally
… 67 member companies
… in July we launched a dedicated term for ebook accessibility
… ensure readers around the world can fully enjoy japanese content without barriers
… short term focus is manga
… digital manga, developing a proof of concept to explore transforming complex visual storytelling
… into accessible format
… goal here is to share our approach, and make this a reality
… head of the working team is Nobuhiro Oga, president of Shogakukan, one of the leading publishers
… committed to producing accessible content that preserves content integrity
… [video from Oga-san]
Suzuki: Next video is from Ito Gakuji, 24th commissioner for cultural affairs
… [video]
… Shinya Takami will present on technical issues
Shinya Takami: Technical issues for manga accessibility
… issues in current EPUB spec, only one hour today, so we plan to have a breakout session later this week
… alternative text for roll and spread
… comic XHTML, we use SVG wrapped image in EPUB files
… almost no reading system can handle these SVG alternative text
… no alt attribute for SVG
Shinya Takami: Alternative text for roll comics, and in the same manner of EPUB fxl comics, images split, and sometimes speech bubbles are split over images
… another challenge
… and challenges with explaining the content on a page-by-page basis
… left speech balloon is spoken by the character on the right page
… difficult to understand
… another spread layout issue, inconsistencies in speech bubble order
… content passes over both sides of the spread, the first is on the right, but the second is on the left page, and third is back on the right
… difficult to embed the correct reading order
… supplementary text to provide background information
… is text based content explanation sufficient, is the combination of text and image enough, but EPUB can include alt for text or images
… maybe supplementary text, like the introduction or of characters, when added in the EPUB file, we don't have specs for this
Ivan Herman: One of the first slides, SVG, I realize the syntax is weird, but you can add metadata to SVG
Ivan Herman: You could add metadata that describes the image the way alt would in HTML, it doesn't impede the visual content, not sure if people use
Gregorio Pellegrino: Not sure it's supported by assistive technologies
Ivan Herman: Which is strange
Shinya Takami: AG's best practice is "title" or "desc"
Hadrien Gardeur: A few things you mention in the slides, align with our discussions on best practices
… true spreads or cosmetic spreads, cosmetic where each page is independent, but true spread where both sides are necessary
… if we don't take into account accessibility, it's a challenge because if you read in a scrolled way, you have the same problem, it's partly an authoring issue
… because content is always separated into 2 images, it can create issues for people who don't need accessibility features
… we haven't spoken much about roll, but similar problem where images are the same height, but if you had variable images you could avoid cutting off bubbles or parts of the page
… and the two discussions could be merged across groups, globally
Max Gendler: Overly basic question maybe, in the true spreads scenario, is there any indication that there is more on the second page that explains the content, is there a signal to the reader?
Hadrien Gardeur: Spread placement exists, attempts before were misued
Max Gendler: It's been explored in the past but not used in the standard
<Brady Duga> Wendy Reid: We are talking about signals to the reading system
<Brady Duga> ... We don't want the bounce between two pages readig experience. Spec way we have is left and right, with you must to do this with a spread
<Brady Duga> ... But that may be overriden. And we don't have a good way of forcing two together
<Brady Duga> ... That was the original intent (or might have been)
Max Gendler: More from the reader perspective, to clarify, when I asked, I don't necessarily think it lives in the standard, different publishers want to do different things, I wanted to know in practice if thst happens
Brady Duga: I implemented something like this, we implemented something that forced what we thought was a true spread, even if the publisher didn't tell us to put them together
… if we could detect, we would
… in the sense of implemetation, it's possible and works well
… even bubble by bubble navigation
… you have to do it in the reading system, no affordance in the spec, but you can figure it out
… my real reason, the title or ARIA, is there an issue of we have text and the description, those might be separate, might want one or the other, what are you thoughts on affordances for combining those
… making it more of a narrative experience
Shinya Takami: Maybe possible, but for publishers technically challenging
Gregorio Pellegrino: Just to be clear, with the current stack of technology it's not feasible to have accessible comics
… it may pass checkers, it may be technically compliant, but then its not usable by users
… what is the point of super-long alt text that describes the entire page
<Avneesh Singh> +1 Gregorio
Gregorio Pellegrino: in the EAA, this kind of content must be as accessible as technology allows, and we know we don't have ways currently
… even with SMIL, we can't jump from two pages
Hadrien Gardeur: You can
Gregorio Pellegrino: That may be one solution, some of these issues were addressed in FXL guidelines, one suggestion was to not split spreads
… but everything in alt text would not work
… technically you can put aria-label and long text, but the result would not be good
Samuel Petit: To spreads, in terms of implementation, even we have challenges, we do it manually when publishers don't
… we make panel detection, something interesting about that
… a proposal from Hadrien Gardeur, the proposal to implement reading ordewr
… you can understand there is a two page spread if you have a reading order that spans
… there is a clean way to understand the double page, by injecting panel ordering
… would help all reading platforms
… would solve a lot of issues
… there are some extreme cases when bubbles are connected or overused
… one page can be one image, or many slices of many images
… we need to ignore that to present a semantic presentation of the page
… we've seen all sorts of layouts and designs that machines can't understand, we need to separate the graphical and semantic presentation
Shinya Takami: Next part of the slides
… how can we make manga accessible?
Tomikura-san: AI generated alt text for manga, human-generated is time consuming
… we have been experimenting, but many only describe panels line by line
… we are trying to ensure the content is communicated properly,
… understanding unique japanese manga conventions, and user needs
… first requirement, manga has its own visual grammar, we must provide context, reading order, objectivity, stay true to the original, respect copyright, and instruct AI to check visual cues, and context limits
… second requirement, we conducted over 20 experts and users with disabilities, and learned
… multi-speaker TTS is essential, and we must focus on the story, not just the details, keep only what moves the plot forward
… based on the readers, we set more prompting rules
… transcribed all dialog, conveyed natual context, kept things concise
… how did we put this into practice
… generated the transcipts and then audio using multispeaker TTS
… and different genres have different alt needs
… we'll explore this on the breakout session, but here are the samples
… first example is The Drops of God
… [video]
… there are notes in the sample, this is where AI gets confused
… next example is One Piece
… translating fast paced action and adventure into words
… [video]
… next sample is The Level 999 Villager, from the isekai genre
… [video]
… next example is First Love Honey
… romance genre
… [video]
… to wrap up, we've faced 5 major challenges, we have no universal TTS tags
… multispeaker audio is necessary but we can't tag it
… AI makes mistakes, and humans need to review
… authorial integrity vs practicality
… it's impossible for authors to review every line
… implementation into reading systems, and who pays for AI costs?
… who should bear the costs?
Gregorio Pellegrino: What you showed us is media overlays in the end, TTS engines can handle multiple voices on the device, it might be easier to handle as multi track audio
… as Hadrien Gardeur mentioned
Suzuki: [something to Daihei Shiohama in Japanese]
Daihei Shiohama: Repeat the question
Gregorio Pellegrino: In EPUB 3.5, synchronizing the audio with media overlays instead of TTS
… might be easier than relying on the device
Daihei Shiohama: [something in Japanese]
Tomikura-san: making the audio files is too much for the publishers
Shinya Takami: It's different from publishers, we're using TTS to make manga accessible
Samuel Petit: I'm sharing authoring software to make accessible content
… showing manga with extracted audio and different voices, emotions
Shinya Takami: Shorter explanation might be better, from our reseach
Hadrien Gardeur: As a general comment, I find it convincing not just for accessibility but in general use
… I take the subway a lot, it would be nice to have this for everyone, many people could enjoy consuming manga like this
… I think the balance of not too long but pleasant
… as you said, there are exemptiosn that apply, but what is important, there are specialized libraries that can do this work, and they need to do it
… and we need to make the tools for them to do it
… the first gap we should be solving
… solving this for adapted content
… what you have shown is interesting, we don't have a good way to point to panels
… it's possible, the example I have, it's consistent with one exception, I also included text, text is very important
… if people rely on braille or other means, text is essential
… very convincing
Daihei Shiohama: [japanese again]
… [translation of what Hadrien Gardeur said]
Dahei Intelligence :)
Brady Duga: You dismissed using media overlays because of publishers, I would hate to see many users hit Gemini to get this content, I think media overlays is a way to do it, then if we can figure out the tech
… if we can cache it once somehow, there is more to do
Suzuki: Mention during the breakout session
Wendy Reid: The thing I wanted to mention is what has been demoed is the ideal, and the ideal is challenging to achieve at scale. What might be more realistic is content that consistently has the text and semantics to enable other systems to build these high quality TTS experiences or even other
Laurent Le Meur: Wendy said what I wanted to say. There is no problem making an alliance between media overlays and content so we can make voices change
… simple evolutions
… to solve minimal problems of publishers and maximal problems of specialist libraries
Hadrien Gardeur: Question of what is possible today is quickly evolving
… more and more that are open weight, open source, run on device and very good
… and some you can run on your phone are capable of doing voice cloning or generating based on a prompt
… the issue is having information in the file to do this
… TTS is mostly solved now or will be soon
… question becomes what do we do
… don't go directly to the very best, background music and so on, but if we had the script, it could just be a single voice
… just having that, is already valuable
… and if we want to go beyond, being able to uniquely identify characters, who is speakign when
… those bring us to multi-voice TTS
… its more of an us problem if we want to tackle it
… there is a cost question of course, it's quite a bit of work, it's out of the equation for libraries
… if we do things step by step, instead of trying to go to the end, a marathon not a sprint
Marisa DeMeglio: One thing that comes to mind with synchonized audio, text, images, is having a situation where text is only in MO
… if they want to read in a screenreader, in a reading system that doesn't have MO, they'd have no access
… something to keep in mind
Charles LaPierre: Quick note, this can go beyond manga and comics, even in books, who is speaking, TTS could change voices, could go far
Shinya Takami: Manga is the culture of Japan and we want to lead this area, along with accessibility people. thank you for the session.
https://
Laurent Le Meur: Wendy said what I wanted to say. There is no problem making an alliance between
Wendy Reid: The thing I wanted to mention is what has been demoed is the ideal, and the ideal is challenging to achieve at scale. What might be more realistic is content that consistently has the text and semantics to enable other systems to build these high quality TTS experiences or even other accessibility features based on the text and metadata.
accessibility features based on the text and metadata./
media overlays and content so we can make voices change/
<Wendy Reid> i/... accessibility features based on the text and metadata./Laurent Le Meur: Wendy said what I wanted to say. There is no problem making an alliance between media overlays and content so we can make voices change/