A commentary on the 2016 Bloomberg Businessweek ranking of undergraduate business programs
Thank goodness Bloomberg Businessweek is getting out of the rankings business—at least the business of cranking out undergraduate business school lists.
Today (April 19), the magazine published the last of its undergraduate business program rankings and it’s a doozy. The University of Pennsylvania’s Wharton School, regarded by most in higher education as the No. 1 program, is ranked 16th, down nine spots from a rank of seventh two years ago. That puts Villanova, Bentley, William & Mary, and Wake Forest, among others, comfortably ahead of Wharton. Villanova, by the way, is first in this ranking, up 23 places from its rank of 24 just two years ago.
It’s certainly not your traditional ranking. By and large, this is a list that is upside down on some world class business programs and favors large state schools except for the big three—Villanova, Notre Dame, and Boston College, all Jesuit Catholic schools—at the top. The new ranking is also the result of a highly controversial, updated methodology that attempts—but ultimately fails—to put, in the words of Businessweek, a “bigger focus on the outcome most students want from B-school: the brightest possible career path.”
NO EFFORT TO MEASURE THE QUALITY OF INCOMING STUDENTS OR THE ACADEMIC EXPERIENCE
The magazine uses four metrics to rank the schools: A survey of employers, worth 40%; a survey of students, worth 35%, starting salaries, 15%, and the percentage of students who had at least one internship during their time in college, 10%. If the results from Businessweek’s surveys made some sense, this approach could yield useful information.
Not that there aren’t other core problems with the methodology. As Andrea S. Hershatter, senior associate dean at Emory University’s Goizueta School of Business, explains, the new approach fails to measure many attributes that distinguish undergraduate business programs. “While many outstanding programs did very well, to truly assess the value of a school’s offering, it seems to me that one would want to take into account the quality of entering students and the differentiating factors of the academic experience,” believes Hershatter.
“Even if one accepts that placement is the focus, I believe the methodology misrepresents student opportunities. This ranking intentionally discounts the weight of alumni who recruit at the school, undervalues the impressions of elite firms that recruit a highly-selective number of graduates, and flattens salaries proportionately by function such that the percent of students placed in the most sought after industries is not taken into account. That overlooks the relative ability of a program to place its graduates in the positions to which they most aspire.”
WHERE IN THE WORLD IS USC, BABSON & TEMPLE?
There are plenty of other problems with this ranking as well. Though the magazine places numerical ranks on more than 100 undergraduate programs, you will notice that some of the best business schools are conspicuously missing from the list. Consider the University of Southern California’s Marshall School of Business, which boasts one of the best undergraduate business programs in the world. Where in the world is USC? The response rate to Businessweek‘s student survey at USC was a healthy 21%, but the magazine had a minimum 30% cutoff.
So USC Marshall is nowhere to be found in the ranking. That’s a nonsensical decision, considering the importance of the school, the fact that a 21% response rate is solid, and that Businessweek insists a school can’t tell its students why they should open and respond to the survey because the editors worry about potential coaching.
Also MIA from the list is are other major players in the undergraduate space, including Babson College, the highly regarded business school outside Boston known for its entrepreneurship curriculum. The change in methodology led Babson officials to simply walk from the survey and decline to participate, further eroding the list’s validity. That also was the reason why the Fox School of Business at Temple University, which boasts one of the largest undergraduate business programs in the world,decided a few years ago to cease its participation in the ranking. Fox leaders believe that Businessweek’s all-too-frequent methodology changes made year-over-year progress by any program impossible to measure.
WHO’S ON FIRST AMONG EMPLOYERS? JOHN CARROLL UNIVERSITY?
Still, the most damning evidence that casts great doubt on the results is this incredibly revealing factoid: John Carroll University’s Boler School of Business takes first in Businessweek’s employer survey. Huh? Yes, Boler ranks ahead of all the obvious players you would expect employers to rank highly (Villanova is second on this measure, and Bentley College is third): Wharton, Berkeley, Michigan, NYU, UVA, UNC, Texas, Cornell, Indiana, Emory, Washington University, Georgetown, and USC, not to mention many other world class undergraduate business programs.
The reason for the head-scratching, less-than-credible results? It’s how the magazine choose to survey the employers who come on campus to recruit students. The editors say they surveyed 1,079 recruiters at 582 firms, with many of the recruiters being alumni of the schools they were rating. Though Businessweek says it excluded alumni votes for their own schools in part of the methodology, the magazine conceded that “alumni opinions of their own schools….remained a factor in the average (quality) rating.”
Even worse, Businessweek commits the ultimate journalistic faux pas in reporting on a survey. It fails to tell its readers how many recruiters actually responded to its questionnaire. Without a reported response rate, the reader has no idea how much credibility to attach to the results. Yet, oddly, the magazine chose to report its impressive 42.5% response rate for the student survey it conducted.
THE FACTS: EMPLOYERS PAY WHARTON GRADS $73,690 VS. $38,000 FOR JOHN CARROLL GRADS
How reliable is the employer portion of the ranking? There are two ways to get at the answer. One is to simply look at what employers are willing to pay the graduates of these programs. Clearly that is the ultimate test of the market. The fact that employers pay Wharton undergrads more than any other—$73,690 for the Class of 2015—is enough proof that they hardly belive Wharton could possibly rank 66th among the employers of business undergrads. (By the way, Businessweek’s pay metric in the ranking only measures salaries—and not bonuses. Yet, 75% of Wharton’s graduating class reported receiving a sign-on bonus, the average of which was $10,020, while 87% reported expecting an annual bonus, the average of which was $28,275.)
Businessweek’s No. 1 school in the employer survey, John Carroll, hasn’t even updated its employment outcomes for 2015, nearly a year after graduation. Carroll’s 2014 report, however, shows that its graduates had “overall median income” of $38,000. Compare that to Wharton’s outsized $105,000 average salary and bonus, with bonuses adjusted for the percentage of graduates getting them. Sure, Wharton sends a large share of its graduates into high paying finance jobs. Still, if employers are willing to pay Wharton undergrads some $67,000 more than Carroll graduates, employers are casting the ultimate market vote on the perceived quality of those graduates.
The other way to shed light on this question is to look at the conventional wisdom in the higher education industry. The only available proxy for this is U.S. News & World Report’s undergraduate business ranking based on its survey of deans and directors. Though a flawed measure as well, it at least provides a point of comparison. The deans list of the top five programs are No. 1 Wharton, No. 2 MIT and Berkeley, No 4 Michigan, and No. 5 NYU. For comparison sake, Businessweek’s employer survey puts No. 1 Wharton at 66th, No. 2 MIT not ranked at all, No 2 Berkeley at 41st, No. 4 Michigan at 27th, and No. 5 NYU at 6th (the latter more likely a consequence of coincidence than a reflection of the methodology’s crediblity.)
GARBAGE IN, GARBAGE OUT
The fatal flaw in the employer rank stems from a well-intentioned, though ill-informed, decision by a statistician who generally believes that having more data leads to a more authoritative result. So instead of surveying the recruiters who oversee undergraduate employment at a company, a far smaller number than those actually polled by Businessweek, the magazine surveyed just about every recruiter who steps foot on a campus—even if they only visit and know one school. Obviously, that recruiter has no basis to make comparative judgments on a number of undergraduate programs. It’s not possible for an alum who revisits a campus for the purpose of recruiting to have thoughtful opinions on rival schools. Yet that is exactly what Businessweek’s survey approach does. That decision alone inevitably leads to wide distortions and highly misleading and unfair results.
There’s an old saying in business that appears apt in this regard: Garbage in, garbage out. If a journalist gets a garbage result, he or she should exercise the judgment to toss it where it belongs, in the garbage. That’s what happens in quality reporting when a person provides distorted or inaccurate information. That’s what should happen when a survey is obviously ill-conceived and flawed. You learn from your mistakes and you redo it.
Thank goodness, Businessweek is getting out of the undergraduate ranking business.
John A. Byrne is editor-in-chief of Poets&Quants and was executive editor of Businessweek when he assigned the magazine’s team to rank undergraduate business programs. He created the first regularly published ranking of full-time MBA programs at Businessweek in 1988.
DON’T MISS: VILLANOVA TOPS FLAWED BUSINESSWEEK RANKING OF UNDERGRADUATE BUSINESS PROGRAMS
Questions about this article? Email us or leave a comment below.