Meeting minutes
<wolfgang> Agenda: Nick Brown (VP Product Vitalsource): From Principles to Practice - Responsible AI for Enhanced Student Engagement in Reading Systems
wolfgang: welcome to all
Nick: From Principles to Practice overview
… VitalSource overview... talking about 'the engagement challenge'
… who are we? 28 m units delivered last year, 19+m users served, users around the world world, localized in 37 languages
… we act as a 'learning delivery network' sitting between learning providers and institutions/students/bookstores
… How do we do this? 1. Driving day 1 affordable access for millions. 2. Helping students stay engaged on Day 2 and beyond
… https://
… more than 30 published papers about learning science
… engagement is step 1
https://
… shows students not reading
14% average number of assigned textbook pages read by students
… one solution: VitalSource CoachMe.
feature inside Bookshelf reader, adds AI generated low-stakes formative practice questions inside the textbook reading experience
generated from the textbook, not a general LLM model
instructors can assign engagement with these questions as a part of their grade
uses VitalSource proprietary AI for automatic question generation. Based on the Doer Effect from Carnegie Mellon OLI work
'practice while reading causes 6x more learning gain than reading alone'
bring active learning to a 'passive medium'
(deeper dive into how it works)
timeline of deployment and achievement
21 million AI generated questions answered in a learning context
published 35 peer reviewed papers
3 best paper awards
examination at Iowa State University course with A/B test with the same book, looking at how many times students opened their book showing dramatic improvement in engagement when CoachMe questions are assigned as a part of the grade
multiple school study showing increase in engagement by students
… 'but what about Gen AI?'
https://
explosion of GenAI usage
Key opportunities: things we can do for students and faculty that could not be done before
Pro's and Con's of using AI
The importance of responsible AI use: https://
referenced 1EdTech rubric https://
Why do these principles matter?
avoid "AI for AI's sake', focus on real learning gains not hype, maintain strong publisher and institutional partnerships
where we are heading next...
High quality AI answers aligned with textbook content, no model training, no IP leakage, SOC2 compliant use with LLMs, DRM protected
Q&A
gautierchomel: when content/questions are generated, we need an evaluation methodology and a way to advise the user this is AI generated. Have you been thinking about this? Is it possible?
uptownnickbrown: that concern drove a lot of our development and decisions. 'generation' is a misnomer, as nothing is 'generated'. There is zero risk, as the sentence for the 'fill in the blank' is from the book.
… there are feedback mechanisms inside the book to give positive/negative feedback on the questions
… recommend: have clear disclaimers that this is AI generated content. It's the right ethical thing to do.
… measuring quality is also critical, and hard.
… automated judges that use LLMs to judge LLM responses, trained to do different tasks in different ways. Judge a few different things at the same time (factually accurate, ...)
… reduce hallucinations by having the concrete source material
… comfortable with a 'the book does not address that question' response
… also trying to evaluate the underlying pedagogy behind the model
… 'can answer that... best I can do is recommend you read page 62 of the book' type of answers
liisamk: have you thought about using AI for 'other things' with your reading system?
uptownnickbrown: a few ways to tie this into the reading system. Things like making good flashcards (hard to do now).
… also looking at ways to evolve search beyond a 'find' function
… search with mixed languages...
… alt-text for screen readers
… lots of places to pervade the reading system
Michalis0: are questions created in real time, or pre-loaded?
uptownnickbrown: yes, the CoachMe questions are pre-generated
… also aligned with where to ask the question in the book flow
… may evolve over time as AI improvements come
… over 1,000,000 questions in production at scale
wolfgang: am I right that you mainly use NLP and not LLMs?
uptownnickbrown: Yes. That's exactly how we built CoachMe
wolfgang: did you take into account the AI legislation from the EU?
uptownnickbrown: we are starting to think more about that now, as it was not in place at the time.
rssagent, generate minutes