Science of Reading: The Podcast

S10 E5: Reimagining comprehension assessment, with Gina Biancarosa, Ed.D.

• Amplify Education • Season 10 • Episode 5

Share episode

In this episode of Science of Reading: The Podcast, Susan Lambert is joined by University of Oregon College of Education Professor and Ann Swindells Chair in Education Gina Biancarosa, Ed.D., to explore how best to assess for comprehension. Gina elaborates on her extensive work developing more precise and informative measurements of reading comprehension and discusses think-aloud research, demonstrating how to infer for coherence, and examining how students who are struggling with comprehension tend to rely too heavily on making inferences or paraphrasing.

Show notes:

Quotes:

"A lot of what we know about reading comprehension comes from think-alouds where you ask someone to tell you what they're thinking as they read." —Gina Biancarosa, Ed.D

"To model reading comprehension, [try] thinking aloud in front of a classroom of students in a way that is instructive for them, and also authentic to the reading process." —Gina Biancarosa, Ed.D.

"Students are making causal inferences in their daily lives, when they watch movies, and when they're hearing stories. And so what we're really trying to do is get them to generalize these behaviors that they engage in outside of the task of reading, during reading." —Gina Biancarosa, Ed.D.

Episode Timestamps:
02:00 Introduction: Gina Biancarosa, Ed.D. and comprehension assessment
08:00 How do we assess comprehension?
14:00 Think-aloud research
21:00 MOCCA (Multiple-Choice Online Causal Comprehension Assessment)
24:00 Causal coherence
30:00 Paraphrasers and elaborators
33:00 Comprehension assessment research
39:00 Professional development and comprehension assessment
42:00 Closing thoughts
*Timestamps are approximate, rounded to nearest minute

[00:00:00] Gina Biancarosa: Other reading comprehension assessments I was talking about can tell you if someone's not a very good comprehender, but they don't tell you what to do about it.

[00:00:12] Susan Lambert: This is Susan Lambert and welcome to Science of Reading: The Podcast from Amplify. So far on this comprehension-focused season, we've explored why comprehension is so key and we've unpacked some of comprehension's foundational components. Today we're turning to a new angle: comprehension assessment. Our guest has done groundbreaking research in this field, and she's joining us to explain why it's critical to build on traditional methods of comprehension assessment.

Dr. Gina Biancarosa is the Ann Swindells Chair and a full professor in the department of special education and clinical sciences in the College of Education at the University of Oregon. She's also the director of the Center on Teaching and Learning. In this episode, she'll detail her work to develop more precise and informative measurement of reading comprehension.

This new assessment, called MOCCA, is something you will definitely be hearing more about in the coming months. Quick reminder, we want to tackle your comprehension questions too. Please share what you'd like to learn about at amplify.com/SORmailbag. And now here's Dr. Gina Biancarosa.

Well, Dr. Gina Biancarosa, it is so amazing to have you here on the podcast. Thanks for joining us today.

[00:01:34] Gina Biancarosa: Thank you. Thank you for having me.

[00:01:36] Susan Lambert: You know, listeners can't hear all the things that we do before we hit the record button, but I was saying how amazing it is that we are finally getting you on, and how surprising it is that we haven't had you on the podcast before, because you know a lot about both early reading development and how that relates to assessment. So. I'd love it though if you could introduce yourself to our listeners and tell us a little bit about your background and how you ended up in this world of literacy.

[00:02:02] Gina Biancarosa: Yeah, yeah. Well, um, I grew up an avid reader myself and, uh, really, um, was passionate about sharing that with others. And so, you know, even as a child I in, I was involved in tutoring other students who were younger than me, um, you know, volunteered as a, a tutor for young children and, um, took a little, um side trip into hospital administration and suddenly...

[00:02:35] Susan Lambert: Wow.

[00:02:35] Gina Biancarosa: Yeah. Yeah. And realized, you know, after a few years of that, that this was not my passion and that I wanted to get back to reading.

And so I pursued, uh, higher degrees in language and literacy at Harvard Graduate School of Education and, um, was intending to be a literacy specialist actually. Um, but was seduced by, uh, research and found that I had a knack for it. So, um, and became really particularly, um, obsessed with the notion of improving our assessments of reading and, and particularly of reading comprehension.

[00:03:16] Susan Lambert: Hmm. That's an interesting story. I think that's the first former hospital administrator turned researcher, so that's, that's really cool. It's encouraging though, because I, you know, I often talk about the fact that when we have a passion for something, sometimes we don't understand the range of opportunities that's available to us. And I'm sure nobody thinks, "Well, when I grow up, I'm going to be a reading researcher around assessment." So, that's really fun. You are at the University of Oregon.

[00:03:46] Gina Biancarosa: Yes. Yeah. I'm a professor in the Ann Swindells Chair in Education and also the director of the Center on Teaching and Learning.

[00:03:54] Susan Lambert: Congratulations.

[00:03:55] Gina Biancarosa: Thanks.

[00:03:55] Susan Lambert: That's an amazing, an amazing role. So, um, you actually do have a lot of influence in terms of preparing our teachers to go into the world, along with this work that you're doing in reading comprehension research.

So, you know, this season of our podcast is, Season 10 is all about reading comprehension. And I really would love, since our focus is on reading comprehension, if you could provide your sort of definition of that and maybe some of the challenges teachers encounter, um, when they're faced with, you know, trying to teach comprehension.

[00:04:32] Gina Biancarosa: Yeah. Yeah. So reading comprehension, a lot of times people talk about it as though it's, it's only the end product, sort of what you have after you're done reading. But I really, um, think of reading comprehension as a process, as what happens as you're reading, as you're sort of constructing a coherent mental model of what it is that you're reading, the, the message, um, or the story. And so when you focus on reading comprehension as a process, it really, um, attunes you more to the cognitive, uh, strategies and processes that we engage in in order to create that coherent mental model as we read.

That really for me is reading comprehension and it emphasizes the, the interactive nature of reading comprehension, that the, the reader is actually an active agent in creating meaning, um, which, uh, that, that it's not a static message, um, coming from the text, but rather that it's the reader creating that coherent, uh, mental model.

[00:05:44] Susan Lambert: Mm-hmm. That was a big aha to me too.

I don't think I understood reading comprehension as a process when I was actually a teacher in the classroom, but always, just like you said, saw, saw it as a product and as an outcome. Um. And didn't always understand that, well, I say it now that I think this is right, you're, you're the expert, so correct me if I'm wrong, but you don't just, even when you think about comprehension as an end product, it's not that you comprehended it or you didn't comprehend it. Comprehension is a range and a little more complex than that. Do I have that right?

[00:06:21] Gina Biancarosa: Yeah, absolutely. Yeah. There's, there's not a singular message to be obtained and once you have it, you're done. Um, it, there's really, um, a spectrum and a lot of, a lot of how much we comprehend and the depth with which we comprehend what we read depends on our goals for reading, right?

We read a news article differently than how we read a novel for pleasure differently than how we read something that's been assigned for a class, right? And, um, and even when something's been assigned for a class, oftentimes, you know, there are different purposes, um, within that, um, in terms of, you know, paying, uh, more attention to the details versus the gist. A lot depends on what we're being asked to do with what we've read.

[00:07:09] Susan Lambert: Hmm, that's a really good reminder. You know, we spent a little bit of time talking about word recognition as the gateway into reading comprehension that, right, if we can't get word-level reading accurate and automatic, it really impedes reading comprehension.

[00:07:27] Gina Biancarosa: Yeah.

[00:07:27] Susan Lambert: And we've talked a lot about, you know, how do, how do we assess that? And so through that area of reading, that word level, is a little more straightforward in terms of assessment, but, can you tell us a little bit about how difficult and how complex assessing reading comprehension is? Especially if we're talking about it as a process and not a product. Wow, that's even more complicated.

[00:07:52] Gina Biancarosa: Exactly. Exactly. And historically, you know, for over a hundred years, the way that we've assessed reading comprehension is by asking questions after someone's done reading, right? And the problem with that, it's twofold. One, you're not getting at that process. You're only kind of getting at the end state. But even more importantly, once you ask a question, you can change what it is that someone has understood from what they read. So, so ...

[00:08:23] Susan Lambert: Tell me more.

[00:08:25] Gina Biancarosa: Well, well, uh, basically we could ask questions about, uh, inferences that one might have made while they were reading, but you could have a reader who didn't make an inference while they were reading and is, it's only instigated by the question itself.

You know, when we ask, sort of, you know, why did Harry hide under the, the invisibility cloak? Right? They, they might not have been thinking about what was motivating Harry. Um, sorry. A little bit of a Harry Potter fan. Um, but once we ask that question, suddenly you can think, "Hmm, I wonder why." So, uh, you may not have been wondering why while you were reading, but the question itself instigates that thought process.

And so that, that's really the problem with asking questions, period. You know, even if you were to ask them during reading, which folks like Kristen McMaster have done, has done a lot of, uh, research on, do we improve comprehension by asking questions during reading versus after reading? And, and the issue with asking those questions is again, you're, oh, you're changing that process, potentially changing that process. For some readers, they may have made those inferences naturally, um, but you have no way of knowing whether they, they were made naturally or not when you ask questions after the fact. There are a lot of other ways that people have tried to get at reading comprehension. One of, uh, the stranger tasks that, that folks have engaged in is, um, having, uh, students break apart words.

Um, I'm trying to remember what that test is called. It's like the TORC, the test of reading comprehension. I'm not sure, but um, what they do is you have a text where all the words are pushed together, all the spaces are gone.

[00:10:14] Susan Lambert: Okay.

[00:10:14] Gina Biancarosa: The task of the reader, yeah, is to break up the, the stream of letters into words that create sentences that create meaning—which is a really strange task, and um, but it also taps into one's ability to recognize a meaningful message when they see one.

[00:10:36] Susan Lambert: That's interesting. Never heard of that one before.

[00:10:39] Gina Biancarosa: There's another test called the Test of Silent Reading Efficiency and Comprehension, where um, I think of it as an assessment of basic reading comprehension, where you read a sentence and you rate it as true or false.

The idea being, you have to understand the message of the sentence in order to understand whether it's a true statement or a false statement. And the statements are very, sort of, common knowledge that everyone would know. The sky is, is green. False, right? Um, the grass is blue. False. Um, fish live in the water. True. Things like that. Another approach that, uh, has been developed is something called a sentence verification technique where individuals read a passage and then the passage is taken away and they have to rate sentences that are related to that passage. Some of them are verbatim from the passage, some of them are paraphrases, some of them, um, are paraphrases, but with meaning changes.

And again, you're rating things as yes or no, this was part of the passage, was part of the meaning of the passage. So there are a lot of different ways that we, we get at this. Um, in DIBELS we use a maze approach where we're reading and every seventh or, or nth word is deleted, and we have to choose among three options, what best fits into the passage and the place where that word has been deleted.

They really only get it sentence-level comprehension, they don't get it, that deeper comprehension that we're generally trying to build as we read. So it's, it's a really challenging task and there's a reason that we've been assessing it in the same way for over a hundred years by asking questions after reading, because people have had a really hard time improving upon that.

As, as you know, I'm, uh, one of the lead authors behind a more innovative measure called MOCCA that we developed to try and get at that process of reading. And, um, this one is really based on think aloud research. So a lot of what we know about reading comprehension comes from think alouds, where you ask someone to tell you what they're thinking about as they read. So they kind of pause after each sentence.

[00:13:00] Susan Lambert: Yeah. Yeah.

[00:13:00] Gina Biancarosa: And give you insight into, "Oh, okay. Uh, I think Harry is hiding under the invisibility cloak because of this, that, and the other thing."

[00:13:10] Susan Lambert: So, so that's generated by the reader, himself, and not like, sort of, prompted by a teacher, so...

[00:13:17] Gina Biancarosa: Exactly. Exactly.

[00:13:18] Susan Lambert: Exactly. Yeah. Yeah.

[00:13:18] Gina Biancarosa: So they're just, they're, they're trained to, after each sentence, tell me what you're thinking. And so it's freely generated by the reader, and then you code what they're saying, um, in order to understand what processes are they engaging in. Are they paraphrasing? Are they making inferences? Are they making evaluations?

Um. Are, are they, um, connecting to their personal experience? There's tons of codes for sort of, what are the things that they're doing. And so that's how we've learned about what good readers do when they're reading. Um, and that's where so many of the reading comprehension strategies that we teach children come from is, is from the think aloud research.

[00:14:03] Susan Lambert: How, how far back does that? Think aloud, research, go?

[00:14:06] Gina Biancarosa: Think aloud research goes back, gosh, at least to the '50s. It's probably older than that actually. Yeah.

[00:14:13] Susan Lambert: Wow. Okay.

[00:14:13] Gina Biancarosa: Yeah. Yeah. And um, you know, there, there was a lot of, um, work done specifically around reading comprehension with think alouds in the '80s.

That's when sort of the reading comprehension strategy instruction sort of was born more in sort of the '90s, um, as, as based on that think aloud research.

[00:14:34] Susan Lambert: That is really interesting because I can't imagine as a former classroom teacher that I would be able to action that in my classroom to be able to get inside a, that's like a, that's like a heavy one-to-one sort of process. Was it ever actioned like that in the classroom or was this more of a literally a research project?

[00:14:53] Gina Biancarosa: Uh, so it's, it's primarily a research approach. There are in individuals like Isabel Beck, who have, um, promoted think aloud as a way of modeling reading comprehension for children.

[00:15:06] Susan Lambert: Okay. Yeah.

[00:15:07] Gina Biancarosa: But that's also really difficult. It's really difficult to model reading comprehension, thinking aloud in front of a classroom of students in a way that is instructive for them, but also authentic to the reading process. So it's, it takes quite a bit of training, but, um, it is one way that you can teach reading comprehension strategies, is by sort of modeling that through think alouds.

[00:15:34] Susan Lambert: So there definitely this, this think aloud strategy has more potential benefits than what we, what we are doing with more traditional assessment. Is that, is that what I'm getting?

[00:15:46] Gina Biancarosa: Yeah. Yeah. I mean, it, it's been one of the primary ways that we get at the process itself. Um, there's also been sort of retrospective interviews where you ask people, "What were you thinking when you read that sentence?"

But think alouds are preferred because they're kind of natural in the moment, um, spontaneous thought.

[00:16:05] Susan Lambert: So really then the think aloud is, it's the only mechanism so far to get at this process, that comprehension process, or what the reader's doing in the, in the process of reading.

[00:16:15] Gina Biancarosa: Well this, this assessment that we developed called MOCCA is an attempt to kind of get around having to do think alouds. So one of the other products that has come out of think aloud research is we've learned that students who are struggling with comprehension tend to do one of two things when they're thinking aloud. They either, um, paraphrase a whole lot, like more than 80% of the time. And I, I like to think of those readers as very text bound.

You know, they're, they're reading the text and they're receiving the message, but they're not doing anything with it. They're not getting interactive with the text in, in the way that good comprehenders do. And then you have a second group of these students who are struggling with comprehension who tend to make inferences.

Um. Some of them are valid, some of them are invalid, but they're just, they're making inferences all the time. Not all the time, that's a little bit overstating the case, but they're, they're making inferences a lot. What we tend to think of about these readers is that they're not being selective with their inferences. So they're, they're making all kinds of these inferences that, um, may or may not be helpful for building, um, what we call causal coherence, um, in terms of the story that they're reading. And that really, when we're trying to build a coherent understanding of what we're reading, we're really sticking to the central message of the text.

So we're not getting waylaid by details and sort of, you know, saying, "You know, I'm reading this, this, uh, story about the character," and doesn't say anything about how the character looks, but I'm making lots of inferences about, "I bet this character has blonde hair and blue eyes, and, um, is, you know, average height and um, but a little overweight."

Um. What have you. Like these inferences that are, are nice to have, right? It kind of gives you a visual image but aren't necessarily central to the meaning of the text. And so, um, the second group of readers we tend to refer to in MOCCA as elaborators, because they're elaborating the text, which is great, this is something that good comprehenders do, but they're doing it to, they're, they're not being selective with the inferences that they're making.

So what MOCCA does is using, um, a sort of modified maze approach, we, we, instead of deleting a word, we delete one sentence from, uh, a passage, and it's always the next to last sentence. And what that does is it creates a causal gap in the story, because you've got the story going along, then there's this missing sentence and then there's a conclusion. And so there's like something missing that gets you from how the story evolved up until the missing sentence to the conclusion.

[00:19:21] Susan Lambert: Hmm.

[00:19:21] Gina Biancarosa: And students are asked to select the sentence that best completes the story. Um, so we don't ask them any questions. We just say, "What best completes the story?" And that's up to them. And so what we find is that our poor comprehenders tend to pick paraphrases, um, for those sentences or elaborations that don't fill that causal gap.

Whereas our good comprehenders recognize, "No wait. Something has to happen to, to motivate this ending, to make it make sense." And so they're picking the causally coherent inference, the one that really ties the whole story together, whereas your elaborator are picking something that sort of, you know, goes off in left field.

[00:20:07] Susan Lambert: Interesting.

[00:20:08] Gina Biancarosa: Yeah. Yeah.

[00:20:10] Susan Lambert: Can we, can we just take a quick step back?

[00:20:12] Gina Biancarosa: Mm-hmm.

[00:20:12] Susan Lambert: You mentioned MOCCA, it must be an acronym for something. So can you let us know what it stands for?

[00:20:20] Gina Biancarosa: Yes. I'm sorry. We speak about it in shorthand because the name is a mouthful. It's the Multiple-Choice Online Causal Comprehension Assessment.

[00:20:31] Susan Lambert: It makes me think of really good coffee though. Right? So mocha and coffee, so that makes sense. I mean, now I have all, all kinds of questions of how, how did you get to this place where you are like, "Oh, we're just going to remove a sentence from the, these two", I'm assuming that they're, they're shorter. They must be shorter passages...

[00:20:53] Gina Biancarosa: Yes.

[00:20:53] Susan Lambert: That you put in front of kids. Okay.

How did you get to the place where you're like, "We just, if we remove that sentence, it's going to do this thing and we're going to be able to, to see kind of the process that that readers are using to get there?"

[00:21:07] Gina Biancarosa: That's, that's a great question. My colleagues, uh, Dr. Sarah Carlson and Dr. Ben Seipel, uh, really came up with the brainchild of MOCCA when they were still doc students at University of Minnesota.

[00:21:20] Susan Lambert: Interesting.

[00:21:21] Gina Biancarosa: And the idea is that, you know, when you're deleting a word, it's, it's very constrained. It's constrained both by the passage, but more specifically by the sentence that it's in. Whereas when you delete an entire sentence, you're deleting a thought. Right?

[00:21:39] Susan Lambert: Right. Yeah.

[00:21:39] Gina Biancarosa: Something that is, is an occurrence in the story. And, and so their, their idea was to create a gap in a story such that it forced the reader to make an inference, in order for the story to make sense. They really drew on a lot of the work of Paul van den Broek, um, uh, and um, his colleagues who have done a lot of work around this idea of causal coherence, um, in terms of the, the role that that plays in reading comprehension; that good comprehenders are really prioritizing that, uh, the causal coherence of what they're reading in order to make meaning, and that that's what really distinguishes them from poorer comprehenders or comprehenders who are really, um, struggling more with establishing that coherence.

[00:22:32] Susan Lambert: Hmm.

[00:22:33] Gina Biancarosa: It draws on, um, the landscape theory of, um, reading comprehension, which sort of, Uh—I, I won't get into here because it's, it's, it's actually pretty heady stuff. But, um, it's this idea that the way that we draw coherence from what we read is by paying attention to reinstantiated concepts and, um, understanding that our role as a reader is to keep those ideas active as we're trying to make meaning.

[00:23:06] Susan Lambert: That makes sense. I'm going to actually pull out the article—so we have a couple of articles that I read in preparation for this. Super fascinating, this idea of causal coherence.

[00:23:19] Gina Biancarosa: Mm-hmm.

[00:23:19] Susan Lambert: In this one article, you have, uh, three short sentences. It says, "Consider the following very short story." So I'm going to read these to you, and I want you to talk a little bit about this idea of inferencing for coherence.

[00:23:31] Gina Biancarosa: Sure.

[00:23:31] Susan Lambert: Tyrese decided to bake a pumpkin pie for dessert. He looked in the pantry. Tyrese was disappointed.

[00:23:39] Gina Biancarosa: Yeah.

[00:23:39] Susan Lambert: So what does a good reader have to do to get to, to comprehend those three short sentences?

[00:23:46] Gina Biancarosa: Yeah, so, so most of your listeners, uh, probably all of them, immediately inferred that Tyrese was missing some ingredients, and so he's not able to make the pie and that's why he's disappointed. None of that is explicit in the passage.

[00:24:02] Susan Lambert: Right.

[00:24:02] Gina Biancarosa: Um. And, and that is very typical of the things that we read where, um, there's this reading between the lines that we have to do in order for the story to make sense. Because without that inference, it's, it's a string of non sequiturs. It's, it's, it's like Tyrese wants this thing. Tyrese does this thing. Now Tyrese is feeling something. Like, what's the connection? The connection is made by the reader. Right? So even going to look in, in the pantry is, is a motivated action by the fact that, you know, we're, we're inferring, "Oh, okay. So he is looking to check for the ingredients."

And so that's what good readers are doing when they're reading, oftentimes completely unbeknownst to themselves because it, it's such an automatic process for a proficient reader. For our struggling readers, you've got, um, your paraphraser for whom this really is a string of non sequiturs. They can tell you what the story said, but, as to why Tyrese looked in the pantry, why he was disappointed, they're not making those connections. And your elaborator the one who's making inferences but not like selective inferences, they might have imagined what Tyrese looked like or where the pantry was in his kitchen. They might have even imagined what's in the pantry. Right? Or, or, or basically details that, again, do enrich the story, but don't necessarily make it make sense, that don't make it cohere.

[00:25:46] Susan Lambert: Yeah, that makes sense. I love that example because it's three short sentences. And a couple of things, what you said about inferencing... I think it took me a while to recognize the fact that you can infer make inferences automatically, without ever knowing it, right? So there, there's some level that it can be an automatic response. I think as a teacher, I always thought, "Oh, you've got to teach that process." Right? But at some level it gets to be automatic. The other thing is that the elaborators, right, they bring all, probably this rich background knowledge. They understand what a pantry is. They understand what's in a pantry. They understand, you know... But yet they're not making the connection to, to what the passage actually means or the purpose of that passage.

[00:26:33] Gina Biancarosa: Right.

[00:26:34] Susan Lambert: And to me, that is actionable then. And I think this is where you're going with some of why MOCCA is really helpful. Because if you say, "Well, hey, Susan is a poor comprehender, what do we do next?" "I don't know." Right?

[00:26:52] Gina Biancarosa: Right, right. Yeah. Yeah. No, I mean that's, that's exactly the issue with all of the reading comprehension, uh, uh, other reading comprehension assessments I was talking about, is like, they can tell you if someone's not a very good comprehender, but they don't tell you what to do about it.

[00:27:09] Susan Lambert: Right.

[00:27:10] Gina Biancarosa: And if the problem is word reading, we've got lots of assessments that will identify that and we know what to do. Right? We've, we've got the interventions that are kind of all set up there. But what MOCCA is trying to do is to reveal what is going on in the mind of the reader? What is their process?

And then what's the next step? Right? And the next step is really informed again by our colleague Kristen McMaster over at University of Minnesota. Both Sarah Carlson and Ben Seipel worked with her as doc students. And you know, they've developed these questioning interventions that really prompt our struggling comprehenders to make the connections that they need to make in order to improve their comprehension.

And so for your paraphrasers, you're really just trying to get them to interact with the text at all. So a lot of the questions that you ask are really asking them, how does that connect to what we already know? How does that connect to the sentence just before it? So that they're trying to just get them to realize you've got a role in connecting ideas across the text, versus the students who are elaborating. They're making inferences and that's great, but we want to get them more directive or, or more selective with those inferences.

And so we use a more specific questioning technique called causal questioning, where we ask them, why did Tyrese go to the pantry? Why was Tyrese disappointed? And by constantly asking those "why" questions, we're really prompting them to attend to really the causal mechanisms in what they're reading.

And I should emphasize, we always kind of illustrate this with, um, narrative texts and with stories, and MOCCA relies very heavily on stories. But this causal, um, structure is also, um, also exists in expository texts, particularly, um, science and historical texts.

[00:29:08] Susan Lambert: Hmm, that's very interesting. Just thinking about this idea of the dimensions of comprehension, I'm wondering, I don't know if that's the right way to say it or not, but, I'm wondering how understanding what paraphrasers and elaborators might need to have to support their comprehension, is that also beneficial for, you know, someone like me who is actually a good comprehender? Um. Can I learn from some of the, that, you know, the strategies that you engage paraphrases and elaborator with?

[00:29:46] Gina Biancarosa: Yeah. You're not going to be harmed by being asked these questions, right? I mean, in fact, you probably already know the answer when I ask you these "why" questions? You've, you've already inferred that, so you know, it, it reinforces your attention to the, the causal mechanisms in the, in the story. You know, when you were talking about the dimensions of comprehension there, it brought to mind, um, situation model theory, Kintsch and van Dijk talk about, or they're the authors of that theory and, and the idea is that we have at least sort of three dimensions to the meaning that we're making as we read.

And the first is sort of the text base. And so that's the, the first level, it's kind of, "Do you get the, the literal message of the text?" And that's where I sort of see those paraphrasers stuck at, is that text-based level.

The next level up is, is what we call the propositional level, where you're sort of extract extracting the most central ideas, maybe letting some details go. So really, um, getting at the, at the gist. But the level above that is that situation model, that coherent situation model. And that's where we start to make connections to our background knowledge, to um, you know, potentially even learning, right? And changing our, our background knowledge, um, as this situation model is built. So, um, that's another way to think about the dimensionality of, of reading comprehension.

[00:31:17] Susan Lambert: I love that you brought that up. Um, I often use a sentence, a couple of sentences when I do trainings or, or, um, anything like that about reading comprehension, or presentations. And I think this is the way it goes—I should have it memorized by now; Carla forgot her umbrella and got wet on the way to school. And the question is, why did Carla get wet? And if you say, well, because she forgot her umbrella, that would be one that's just text-based. It's right there in the text.

[00:31:48] Gina Biancarosa: Mm-hmm.

[00:31:49] Susan Lambert: And other people can say, oh, because it was raining.

[00:31:52] Gina Biancarosa: Mm-hmm.

[00:31:52] Susan Lambert: But the passage doesn't say it's raining. So that would be that deeper level of connection and comprehension.

[00:31:58] Gina Biancarosa: Exactly. And that's your causal, causally coherent comprehender right there. The one who's like, you know, it goes beyond what the text says. There's more, there's more to this story that makes it make sense.

[00:32:09] Susan Lambert: Hmm. And that to me is depth of comprehension.

Just like you outlined with Kintsch's model is that well, it is not wrong to say she got wet because she forgot her umbrella. That's absolutely right. Right?

[00:32:22] Gina Biancarosa: Right.

[00:32:22] Susan Lambert: But there's another, there's another layer.

[00:32:24] Gina Biancarosa: Wow. Yeah.

[00:32:25] Susan Lambert: I just had a good aha. So thank you for that. So you've been working on this research for, for some time now. Is there anything that really surprised you in terms of what you learned in the process of doing this research?

[00:32:41] Gina Biancarosa: Yeah, I, I think what surprises me is, you know, we've been working on this for more than a decade now, and we have, we started working in, in grades 3 to 5 because that's seen as like this transition to, you know, reading, uh, reading to learn, right?

And so, uh, the importance of, of readers becoming active agents in their own comprehension. And what has surprised me is that there's been a real hunger for this at the middle grade level. And we have expanded up to grade 6, 7, 8, and even nine. And, uh, Ben and Sarah, um, have even developed a sort of college version of, of MOCCA.

And what surprises me is that you find these paraphrasers and elaborators throughout that developmental arc. So one of the things we had sort of hypothesized was that maybe paraphrasing is sort of an earlier developmental stage, right? Then you're an elaborator and then finally you're a causal comprehender and that, that may or may not be true. Um, I don't think we have enough evidence to say whether that's how an individual develops in their reading comprehension. But what was surprising is that you can get to eighth grade and still find that you've got like 10 or 20% of your students who are really stuck at that text-based level who are paraphrasers.

That, for me was a real aha. So it's, it's been a real learning experience working with middle grade teachers and, and understanding that these same issues are, are being experienced by their readers.

[00:34:22] Susan Lambert: Is there any, have you done any work with this same kind of idea in the earlier grades where it's related to listening comprehension? Like can we sort of look at K-2 in some read aloud environment or something and sort of say, "Oh, this group of students is exhibiting behaviors that might be paraphrasing?"

[00:34:44] Gina Biancarosa: Yeah. Yeah. We, we have actually expanded it down to grade 2. I think one of the issues is that decoding starts to be this bottleneck, right? So if they're not decoding well enough, MOCCA is overly challenging for them, and, you know, you end up with what we call an inconclusive categorization where it's just like they're, they're guessing, um, because it's too hard a task. We have thought about putting it into a listening comprehension, sort of, mode, but we haven't gone there yet.

I think that's one of the, the sort of next steps for the work that we do. I, I will say, I, I've, I've been a little bit less, um, motivated to do that because, um—I, I keep mentioning Kristen McMaster. Um, she, she's a good colleague and does a lot of, um, related work. She and Panayiota Kendeou, um, also at the University of Minnesota, have developed the Minnesota Inference Assessment, or MIA, which is for grades K-2.

[00:35:49] Susan Lambert: Okay.

[00:35:49] Gina Biancarosa: It actually uses video, which I think is, is, is potentially a little advantageous to just relying on listening comprehension. Um. So students watch a little narrative and then are asked to make an inference about what they've seen. Now their assessment isn't focused solely on causal inferences, so they're also assessing more, um, sort of lower-level just connective inferences.

Um. So, you know, to go back to this Tyrese story, you know, when he, you say he goes to the pantry, you know, you, you need to infer he is in his kitchen, right? Pantries are in the kitchen. Things like that, um, that do lead you towards coherence, but aren't solely focused on coherence. But yeah, I, I think the challenge with this is when you think about doing it for, for listening alone, is the MOCCA task requires a lot of, um, sort of, working memory. And so by having the text in front of you, you can revisit propositions that, or sentences that were in the assessment. And, uh, one of the details that the um, uh, the experience of the assessment, um, is if you click on a sentence, it appears in the story, so they don't have to do the mental effort of like, "Okay, I am reading," and then, "Oh, this sentence..." and then back up here.

[00:37:14] Susan Lambert: Oh, I gotcha. Yeah.

[00:37:14] Gina Biancarosa: It helps them kind of see where it fits in the story and read it within the context of the story to kind of settle on which, which of the sentences best fits. And I worry that when we only do it in a listening context, that it's an overly demanding, uh, working memory task. So, so that's the challenge that I think we'll have to tackle if we want to go to the younger grades, is how do we keep it within the working memory capacity of our, you know, kindergarten to second graders.

[00:37:47] Susan Lambert: Yeah, that makes sense, because otherwise then it's just an assessment of working memory as opposed to an assessment of comprehension.

[00:37:54] Gina Biancarosa: Right. Exactly.

[00:37:55] Susan Lambert: This is, um, I'm going to underscore this for our listeners, but this is, uh, an exciting new breakthrough, if you will, in assessment in an assessment that actually delivers instructional support for teachers. How do you see this assessment actually impacting classrooms?

[00:38:22] Gina Biancarosa: Yeah, I mean we're, we're seeing it impact classrooms already, you know, where, where the, the hunger for, um, professional development around these questioning strategies is real. And, um, that's sort of one of the next, um, developmental tasks that, um, Sarah and Kristen are actually really focused on, is how do we give teachers more support around how they can ask these questions within their instruction?

And I talk about it as professional development rather than an intervention because a scripted intervention will only get you so far. What you really want to do is, sort of, arm teachers with the knowledge to understand where these causal gaps are in texts that they have their students reading, being able to identify where those causal gaps exist, and then knowing the kinds of questions to ask at that moment is really the, the skill that we want to ingrain in teachers.

[00:39:27] Susan Lambert: Hmm. That's awesome. And can you do one more review of that? So if we have, and we notice a student is a paraphraser, in other words, they're just sort of parroting back the things that they see in the text, what kind of questions do we want to ask that student?

[00:39:45] Gina Biancarosa: So we want to ask them, um, connecting questions, um, or, uh, it's, it's also just called general questioning, but, but, um, it's, it's really sort of like, as they, um, come across new information, asking them, how does that connect to what we already know?

And that's really, if like, if I had one question I could teach teachers, that would be it for the paraphrasers, is how does that connect to what we already know?

[00:40:12] Susan Lambert: That's awesome.

[00:40:12] Gina Biancarosa: So, so getting them to kind of go back to the text and, and really try to glue together the concepts as they're coming in.

For the, um, elaborators, right, the questioning technique that we really, um, recommend... There are these "why" questions. Why did the character do this? Why is this important? And so, you know, you've got your how and your why. Um, so that, that's kind of my shorthand for the two questioning approaches.

[00:40:43] Susan Lambert: I love that. That's a good sort of, you know, something to remember the being mindful of asking the "how" questions and the "why" questions, and not just because they feel like good questions to ask, but they're actually productive in terms of helping students think about those connections within passages.

That's super helpful. Thank you for going back and reviewing that.

[00:41:04] Gina Biancarosa: Yeah. Yeah.

[00:41:05] Susan Lambert: Any closing thoughts or advice for our listeners as we wrap up?

[00:41:11] Gina Biancarosa: Um, yeah, I, you know, one thing that I really also try to emphasize when I'm talking to teachers is that in some ways comprehension, what, what's so baffling about it is, you know, it's so hard to move the dial on it, to actually, um, enact change in, in what a student is comprehending as they read.

But on the other hand, students are comprehending all the time, right? They're, they're actually making causal inferences in their daily lives when they watch movies and when they're hearing stories, right? And so what we're really trying to do is get them to generalize these behaviors that they engage in outside of the task of reading, during reading. So we're really trying to get them to, sort of, raise their standards for how much they should understand of what they read when they read.

[00:42:09] Susan Lambert: Hmm. That's such great, such great words of wisdom to think about. There are things that students are already doing outside of the process of reading that we can actually bring in and help support them while they're reading, so that's awesome.

Well, Gina, it's been a pleasure. Thank you for the work that you're doing. We can't wait to watch it all unfold and move forward even more. We'll link our listeners in the, in the show notes to an article or two. Um, but thank you so much for joining us and, and again, it's been a pleasure. Thanks for the work that you do.

[00:42:42] Gina Biancarosa: Yeah, it's been an absolute pleasure. I always love talking to you, Susan.

[00:42:46] Susan Lambert: Thank you.

That was Dr. Gina Biancarosa. She's the Ann Swindells Chair in Education and a full professor in the department of special education and clinical sciences in the College of Education at the University of Oregon. A special thanks to Gina for helping make an article we discussed open access.

Check out the show notes for a link to, "Diagnostic and Instructionally Relevant Measurement of Reading Comprehension." Keep your eyes peeled for more news about the MOCCA assessment in the coming months.

Next time we're continuing to explore this fascinating area of comprehension assessment. We'll be joined by Dr. Melissa Farrell, author of Reading Assessment: Linking Language Literacy and Cognition.

[00:43:34] Melissa Farrell: I think we ultimately really have to ask ourselves what it is that we want to learn from this assessment.

[00:43:40] Susan Lambert: That's next time. You can also access a new collection of free comprehension resources, including ebooks and on-demand professional learning, by visiting at.amplify.com/comprehension101.

Also don't forget to submit your questions about comprehension at amplify.com/SORmailbag. On the latest episode of The Beyond My Years podcast, Dr. Tina Boogren discusses research-backed strategies for improving teacher wellbeing.

[00:44:13] Tina Boogren: When I was an instructional coach and mentor working with new teachers, I would have teachers that, they were showing me they were in dissolution, but I didn't know what to do, so I just kept shoving instructional strategies down their throat.

And then I learned, "Oh, let me take a step back," and, and typically what I, my number one question that I'll ask is, "Tell me how you're sleeping." Oftentimes the tears will start flowing.

[00:44:37] Susan Lambert: That's available now in the Beyond My Years feed. There's a link in the show notes, Science of Reading: the Podcast is brought to you by Amplify.

I'm Susan Lambert. Thank you so much for listening.