When AI Helps Students Learn & When It Doesn’t: An Emory Prof’s Groundbreaking Study

Q&A WITH GARETH JAMES, DEAN OF EMORY GOIZUETA BUSINESS SCHOOL

Poets&Quants: I have to ask, am I speaking with the real Gareth James right now and not the Digi-Dean?

Gareth James: I feel like the real Gareth James, but maybe you can decide as we go along if it feels right.

All right. If there are any glitches, I’ll know that you’ve got better things to do.

Well, it’s the interesting thing with these avatars, you can usually tell them apart because they don’t make glitches. Oftentimes if you put two videos side by side, people tend to pick the real person as the avatar because it doesn’t seem as high-quality or something.

That’s a little scary, isn’t it? There are some scary implications there.

Oh yeah, there are very concerning implications. I was both excited and shocked at how easy it really was to just take two minutes of video of someone — we did it in our studio, but you don’t even have to have that high quality of video — and feed it into various software packages, and then you can just make the person say whatever you want. So suddenly all my faculty are going to get a 10% pay raise and everything. You can imagine that anything could happen. So certainly I know from a personal perspective, I certainly will no longer take video evidence of anything at face value. Right? It’s clear that I don’t know what to take at face value anymore, but certainly that can’t be just assumed to be accurate.

Does that make it incumbent upon the school to talk about all the ethics involved, but also label things and make sure people know this in a course? 

Yeah. One of the interesting questions to me is, what are the appropriate applications of the avatar technology in an educational setting? And I can imagine multiple different examples. So maybe I’ll just quickly go through a couple of them to think through. So the first one that occurred to me was that my previous institution, we created an entire online MBA program, and it was about 50% synchronous material and 50% asynchronous. And there the faculty spent literally hundreds of hours in this business school, our studio, creating all of this material. It was a real team effort with the staff and everyone.

And then of course, that material quickly becomes stale as new things come along or the faculty member moves on to another teaching assignment or another institution and you have a new person coming in — and they either have to use the old person’s videos and it makes them look like a TA in their own class because the videos are from someone else, from the professor, and you also have stale material in there with the avatar technology. If you want to update just two minutes of the video, you could just update the text and in 30 seconds you’ve got the updated material as well. Or if you want to move to a new professor, again, you just feed that text that’s already been created in with the new person and almost instantly you’ve updated all the videos. All that.

That’s a huge time saver.

Yeah. And that affects the price point, right? Education is incredibly expensive — the high-quality version, anyway. So this allows us to open it up to a wider audience. It also means you can instantly take those videos and transform them into any language you like, so you can switch it to a much wider international audience.

I saw you speaking Chinese in your video.

Yeah, exactly.

Presumably you don’t speak Chinese.

You feed in the English text and it does the adaptation for you. I barely speak English, and that’s the only language I know — but the people who speak Chinese and things and watch these videos tell me that it is really high-quality. It’s an accurate translation. There’s that aspect to it. There’s also the opportunity to present a lower price-point option to a wider audience. But I think the gold standard is always going to be the professor in person in the classroom speaking to a relatively smaller group of students. But that’s also a very expensive model, both in terms of the price point for those students, but also for the fact that that knowledge can only be conveyed to a relatively small group of individuals. So the avatar allows you to potentially present that incredible knowledge that our faculty have to a much wider audience at a lower price point, at a quality level that’s not that far below what you might create with these high-quality studio recordings and all of this sort of thing.

Professor Garg’s team found that student performance was actually highest when there was human-generated content, but an AI avatar. So does that mean AI can actually improve student performance?

That was a great example. I think there’s way more research do on this topic. But yeah, it illustrates what I think is an important point: that you can have faculty who are great researchers and also great at coming up with material to explain their topic, but their presentation skills may not be the best. There’s a whole range of presentation skills that people have. So you could easily imagine a situation where we have a great faculty member who’s a great researcher and great at coming out with a way to present the material, but their presentation skills are weak; you take the avatar version, and the students will understand that better.

I could also imagine a setting where I take my regular class that I’m teaching in person to my students and I want to create a Q&A document: What are the common questions that the students have on various different things? Where do they often get confused? As faculty, we often do that for our students, but then they have to read through all of this material and such. I could now feed this into my avatar and in 30 seconds create a five-minute video explaining all of this. It’s no extra work for me. I already created this text. But for the student learning experience, it improves their learning. And I think it raises interesting questions about how we learn this material. A lot of these ideas, they exist in books already. You could in theory go and just read a book and learn all about this — but I don’t know about for you, but for me and for lots of other people, it’s almost impossible to just read a book for hours on end and self-learn all of this material.

So the classroom experience and the way that the professor presents that material, the explanations they give — that’s what adds on and makes it a truly educational learning experience. So these avatars allow us to actually take what’s already existing in a book and create a mechanism for students to learn that material without necessarily having to have the highest-price-point, gold standard faculty member themselves standing in front of the students presenting it.

Are you already employing this stuff or is that right around the corner or what?

Some of our faculty are starting to work with students in their class for using generative AI. The avatar technology itself, I don’t think we’ve actually created any for our classes directly, but this is actually the next topic. This video that we created was really just the first example, it was to present to my faculty so that they could see the technology has evolved now to the point and literally months ago, I don’t think it was there. Now I think it is.

It’s moving so quickly.

I think it’s still got a little way to go. I think it’s going to continue to improve, but literally in a matter of months, not years. So I could imagine in six months it’s looking even dramatically better than the stage that it’s at now — and so we as a school need to think about how we are going to use this. And I mentioned several different ways that it could be used. And so that’s going to be an internal discussion with our faculty and staff about what are the best applications of it. But I think we’re all sort of scrambling to try to figure that out at the moment.

And the pace of this means you’re going to be constantly scrambling. Things are going to change so quickly and you’re going to have to stay on top of them.

So this is the new normal in business schools for sure. The avatar technology is just one example of this, but I think it does present a challenge in general for higher education. So our faculty are used to spending four or five years on a Ph.D. going in-depth into a topic. You can imagine how much this technology evolves in a four- or five-year time period, so it’s actually forced our faculty to adopt more collaborative learning models in their classes instead of the hierarchical faculty delivering the material down to the students and they just absorb it. There’s more of a group learning thing because many of these students, of course, are self-learning this material as well. And so the faculty and students to a larger extent now are working in a collaborative fashion to figure out how this works. And the faculty member can often bring in expertise and experience they have from other areas that existed five, 10 years ago and layer it on top of this new technology and provide insights that the students maybe wouldn’t have picked up themselves — but the faculty member can’t possibly be expected to suddenly be an expert on all aspects of this technology that’s been around for literally months.

I think overall that’s a good thing. I think the cultural norms within our society have been moving in that more collaborative teaching style anyway, over the last decade or so.

Things like the online MBA program that we created were actually quite challenging for the faculty involved because they were used to working by themselves and just presenting the material, doing everything. When that material was being created for an online program, they had to work as part of a team, and that was hard for them, but they adapted, and as a result we created a much better product than just the faculty member standing and randomly talking in front of a camera. So I think this is the way learning’s going anyway.

Are you in the classroom this semester, or do you hope to get back into the classroom?

This semester, no — I am doing so much traveling. I’m actually in New Zealand at the moment, and I am doing a lot of traveling, so it’s almost impossible to actually teach a full-semester class. But last semester I did get to teach a great one-unit class for our undergraduate students on leadership. I team-taught it with General (Ken) Keen, who’s a retired three-star lieutenant general on our faculty. So we team-taught military leadership, academic leadership, and then we brought in guest speakers from industry and government and things like that. So that was a lot of fun.

Will your Digi-Dean make another appearance next time you make it into the classroom, whatever the subject?

Yeah, I would love to. I can definitely say that if I was able to teach full classes again — my expertise is on machine learning, data science, those topics — I would absolutely bring that technology in. Maybe not into the primary classroom itself, but at the very least into the sort of supplemental materials and things for the students, for sure.

As the comfort level rises, then there will be more applications and more opportunities.

Oftentimes our best faculty spend a lot of time outside the classroom recording for themselves, help videos and things like that. The opportunity cost of creating these videos becomes almost zero. If you can just feed in text that you already have to explain it to the students, maybe you cut and paste different things together and you create a one- or two-minute video explaining some concept, it might take five minutes to create that video from scratch. With this technology, it is quite impressive. Now, I think there are limitations. I think that the technology doesn’t quite capture the dynamism of a good instructor and things like that, at the moment. But again, I think we might be only months away from seeing that level of impact as well.

That’s next. And speaking Klingon, too. You need to get that.

Exactly. Yeah, probably makeup languages and yeah, you can see why we are having these actors’ strikes, for example, in Hollywood.

That was the basis of it, right?

You can really see how we could rapidly get to the point where humans are not even required to create education, all sorts of material.

Well, we’d  better step carefully then.

We should! What I would say on that topic is that technology has been evolving for hundreds of years, and every time there’s a major technological advance, humans get concerned about it. There are always winners and losers as a result, but humanity or society overall has benefited from those technologies, and the winners are the ones that learn how to evolve with that technology — and take generative AI as an example here.

So I think for you as a writer, I think it’s not that generative AI is going to take your job away, at least at the moment. I don’t see it as high enough quality to entirely replace a human in writing up articles or stories or things like that. But it will get to the point — I think it already is at the point — where people that refuse to use generative AI to get a first draft and a starting point for a document or something like that are going to fall behind other people who can present and produce 10 articles in the same time. It might take a person not using generative AI to create one or two. And so that’s where the adaptation comes, that sort of synchronization between the human and the artificial intelligence aspect of it.

DON’T MISS LEARNING AI ‘FROM ITS CORE’: INSIDE PROFESSOR TINGLONG DAI’S COURSE AT JOHNS HOPKINS CAREY BUSINESS SCHOOL and THE AI BOOM IN EXECUTIVE EDUCATION: WHAT YOU CAN STUDY RIGHT NOW AT THE WORLD’S TOP BUSINESS SCHOOLS