GMAT validity studies

Does the GMAT matter?

If you either know me personally or are a regular reader of my GMAT blog, you know that I’m not always fond of the GMAT.  I honestly enjoy some of the intellectual and psychological challenges presented by the exam, but I often cringe at the actual content of the test.  I’m simply not convinced that your knowledge of number properties or modifier placement will ever help you to become the CEO of a successful company. And that brings us to a (somewhat unanswerable) question:  what is the GMAT good for, anyway?  And can we prove that the GMAT is a useful measure of… something?

As I mentioned in an earlier blog post about the new Integrated Reasoning section, the writers of the GMAT spend a lot of time proving that the GMAT is “valid,” and their official GMAT blog often goes to great lengths to remind us of the GMAT's value.  GMAC’s definition of the word “valid” is interesting, though:  most GMAT validity studies simply examine the correlation between GMAT scores and students' grades during the first year of business school.

By this measure, the GMAT is an outstanding exam, and it does an excellent job of predicting test-takers’ academic performance upon entering business school. If you feel like geeking out on some of the numbers, you can check out a classic GMAT validity study here.

But the GMAT isn’t really designed to predict anything else.  Yes, the GMAT correlates strongly to your grades during your first year of business school.  But your grades in an MBA program don’t correlate strongly to success in the business world—and neither do your GMAT scores.  According to a recent article in Forbes, the GMAT doesn’t seem to predict to post-MBA salaries at all, despite the fact that a handful of high-paying finance and consulting firms still use business school grades during the hiring process.

Many of us have an unfortunate habit of blowing the GMAT way out of proportion, and we act as though it’s a measure of general intelligence or business acumen.  It isn’t either of those things.  The GMAT is a fantastic measure of a specific set of quantitative and verbal reasoning skills, and validity studies have proven that those reasoning skills will help you get good grades in business school.  Beyond that, the GMAT isn’t a great indicator of success.

So next time you get your butt kicked by some GMAT questions, don’t take it personally.  The GMAT isn’t telling you that you’re stupid, or that you won’t succeed in business.  It’s just telling you that you might not have the specific skills required to get a high score on a very specific standardized test.  And even if you’re getting your butt kicked by number properties or modifier placement rules, you might still be an absolutely spectacular business mind, destined for a lifetime of financial success.

IR might be really important... in 2017

If you’re applying to MBA programs during the current (2012-13) admissions season, you’ve probably already read a few dozen articles about the new Integrated Reasoning (IR) section. As many other GMAT tutors and bloggers and forum participants have suggested, there’s little reason to think that the IR section will have any meaningful impact on your odds of admission at top business schools this year. The section is simply too new, and MBA admissions committees have absolutely no basis for evaluating the new section. But what if you’re looking ahead, and you’re preparing to submit your MBA applications in late 2013 or beyond? Should you worry about Integrated Reasoning at all? And if so, how much of your GMAT preparation time should you devote to the IR section?

Although GMAC is doing its best to convince everybody that Integrated Reasoning is extremely important (click here or here or here or here to see their official GMAT blog posts about the awesomeness of Integrated Reasoning), I would argue that there’s still no good reason to spend much time studying for the Integrated Reasoning section... for now.  And here are three reasons why:

Reason #1: five years of GMAT fairness

GMAT scores are valid for a full five years, and a substantial percentage of applicants will submit “old” GMAT scores (taken before the IR section existed) during the next few admissions cycles. And it simply isn’t reasonable for schools to use Integrated Reasoning scores to compare applicants, when a certain percentage of applicants haven’t taken the new section at all.

In theory, MBA programs could require all applicants to submit an Integrated Reasoning score beginning with the 2013-14 admissions cycle, but this seems incredibly unlikely. We still know very little about the value of the Integrated Reasoning section (see reason #2 below), and there’s absolutely no incentive for MBA programs to aggressively require an IR score before the five-year window is over.

Reason #2: GMAC needs time to determine IR test validity

Somewhere in the bowels of GMAC headquarters, researchers are busy calculating the “validity” of various portions of the GMAT exam. Basically, those researchers are interested in determining what, exactly, the GMAT tells us about test-takers. Do GMAT scores correlate to performance in business school? Do GMAT scores correlate to success in the business world?

For what it’s worth, most of the studies I’ve read (and yes, I’m apparently dorky enough to read academic studies about the GMAT) suggest that the quant and verbal sections do an excellent job of predicting MBA students’ grades in business school, but the GMAT does a pretty lousy job of predicting post-MBA success… and the AWA isn’t really a great predictor of anything, which is why one of the AWA tasks has been replaced by Integrated Reasoning. If you’re curious and want to geek out on some old GMAT validity studies, you could start by clicking here.

Anyway, the bottom line is that researchers need time to “prove” that the GMAT’s new Integrated Reasoning section actually means something. Until that happens, why would MBA programs worry about your IR score?

Reason #3: MBA applicant information overload

By the time you submit your MBA application, business schools know a ton about you. They have your work history, academic transcripts, lists of extracurricular activities, two or three references, at least a few essays, maybe a cover letter, possibly a slide presentation, an interview, and probably some extra interactions with you from campus visits or other events. Oh yeah, and they have your GMAT quant score, your GMAT verbal score, your GMAT composite score, and your GMAT AWA score.

Really, do you think the adcom needs yet another data point? And do you think that the adcom is likely to care about a data point that doesn’t show up in any MBA rankings?

Don’t get me wrong: someday, all of this will probably change. If U.S. News and World Reports decides to include IR in its MBA program rankings, adcoms will suddenly care—a lot—about the IR section. I also believe that the GMAT validity studies will someday make the IR section look really, really good; frankly, the GMAT should be testing your ability to analyze basic data tables, and the exam should include some non-multiple choice questions. In my (probably very irrelevant) opinion, the Integrated Reasoning section has plenty of potential to be a valuable tool for evaluating MBA applicants.

And someday, the IR section really will matter. But not yet. Call me in 2016 or 2017, and maybe I’ll tell you to start worrying about it then.

In the meantime, your approach to Integrated Reasoning should be the same as your approach to the AWA section: both tasks are warmups for your quant and verbal sections, and it’s not worth burning much of your precious GMAT energy on IR or AWA. The IR section is not adaptive at all, so just answer the easy ones to avoid complete embarrassment, and let the harder ones go. It just isn’t worth spraining any brain cells for a section that has another four years of irrelevance ahead of it.

But again, call me in 2016 or 2017—the story might change by then.