Monthly Archives: October 2016

That official GMAT question might cost $3000

 

I spend a lot of time telling GMAT students that no GMAT practice problems can ever be as good as real, retired questions from official GMAT tests and publications. And if you read a bunch of GMAT blog posts written by other GMAT test-prep “experts”, you’ll find that many of them seem to disagree with me – since they’re trying to sell you a nice bundle of non-official GMAT questions and exams.

Sometimes, even my own GMAT students object when I tell them to avoid using knockoff GMAT tests from, say, Manhattan GMAT or Veritas: “How terrible could those materials be? There are smart people working at those GMAT test-prep companies, right?”

Yes, of course there are smart people working at GMAT test-prep companies – though some test-prep firms clearly employ smarter GMAT question-writers than others. But even at their best, no GMAT test-prep company can possibly compete with the precision of real, official GMAT questions. Part of the reason is that copying the style of an intricate standardized test is inherently challenging, but the bigger reason is basic economics.

If you’re reading a GMAT blog, you’re probably a business-savvy future MBA student, and you probably have some intuition for the business models of companies like Kaplan or Manhattan GMAT. So take a guess: how much do you think those test-prep companies spend developing each individual question on their GMAT practice tests or in their GMAT books? Go ahead and think of a number.

OK, got something in mind? Great. Personally, I would imagine that GMAT test-prep companies spend something in the range of $5-30 per “knockoff” GMAT question if they actually want to make a profit, but I could be wrong.

Now let’s ask Dr. Lawrence Rudner, former GMAT Chief Psychometrician, how much the GMAT’s developers spend on each practice question:

Test items are costly to develop, often in the range of US$1,500 to US$2,500 per item.

This comes straight from a paper that Rudner presented at the GMAC Conference on Computer Adaptive Testing way back in 2007. If question development costs have kept pace with inflation, then each GMAT question would cost somewhere between $1,700 to $3,000 per question in 2016, though that figure could be even higher if shortages of trained psychometricians continue to inflate GMAT development costs.

(Incidentally, I got halfway through a PhD in education policy and psychometrics – the statistical science behind standardized testing – before I came to my senses and realized that I’m much happier as a GMAT tutor than as an academic or psychometrician. If you’re interested in a career in psychometrics, send me an email, and I’ll do my best to help out.)

So yeah: that GMAT question you saw on the latest version of the exam? It might have cost $3000 or more to develop – perhaps 100 times more than a GMAT test-prep company could reasonably spend on each question.

And if that sounds batsh*t crazy – or at least wildly inefficient – it probably isn’t. Actual GMAT test items go through a painstaking cycle of editing, testing (often as “experimental questions” on actual GMAT tests), re-editing, and re-testing before they actually become part of a real GMAT exam. For every question that actually makes its way onto the GMAT, countless others are discarded for any number of reasons. For example, there might be evidence that the discarded GMAT questions were phrased badly, that they don’t seem to capture the skills the GMAT promises to measure, or that they (accidentally) discriminate against GMAT test-takers from certain demographic groups. Much of this testing requires substantial skill and training – and employees with these skills certainly aren’t inexpensive.

You probably know that I’m not particularly impressed by the GMAT’s ability to identify the most talented business leaders, but the GMAT’s creators deserve credit for producing subtle and meticulously phrased questions. How could any GMAT test-prep company possibly produce anything of comparable quality or precision, given test-prep firms’ economic limitations?

So whenever you work through practice tests from major GMAT test-prep companies – or GMAT practice worksheets from an obscure little GMAT tutor dude in Colorado – remember to take your results with a huge grain of salt. Sure, our “knockoff” GMAT questions can probably help you build skills, but none of us can ever give you a truly accurate GMAT practice test – simply because none of us will ever be able to afford to do so if we actually want to keep our doors open.

 

GRE vs. GMAT, part V: look Ma, no MBA rankings!

 

In four earlier GMAT blog posts highlighting the differences between the GMAT and the GRE exams (you might want to start at the beginning of the GRE vs. GMAT series), I held back on one big factor: if you’re not a great standardized test-taker, then you should probably consider taking the GRE instead of the GMAT. Why? Because the GRE still isn’t included in MBA rankings schemes.

So let’s take a step back: why do MBA programs care about the GMAT and GRE tests in the first place? I’d argue that there are two major reasons. First, the original intent of the GRE and GMAT was to help graduate schools figure out whether applicants can handle graduate-level coursework – after all, undergraduate GPAs can be hard to trust. But over the past couple of decades, the tail has begun to wag the dog: MBA rankings have forced MBA admissions committees to “play the GMAT game”, and GMAT score averages have inflated to ridiculous levels.

This little GMAT blog already contains plenty of whinging about rising GMAT scores, but to be fair, let’s think about it from an MBA program’s point of view. Imagine that you’re on an MBA admissions committee, and you’re choosing between two candidates. Candidate A has a 720 GMAT, while Candidate B has a 750 GMAT. The two are otherwise similar, except that Candidate A is qualitatively a bit more appealing than Candidate B – more interesting MBA application essays, a sharper interview, and more convincingly effusive recommendations, but nothing quantifiable.

Would you blame the admissions committee for worrying about their MBA rankings and selecting Candidate B? I certainly wouldn’t, even though the two candidates’ 30-point GMAT score difference may be nothing more than measurement error. MBA rankings matter – a lot – whether we like it or not. “Candidate A is cooler” also matters, but in practice, that argument is almost always trumped by hard numbers.

But the GRE isn’t factored into MBA rankings – at least not yet. So if you submit a GRE score instead of a GMAT score, you’re effectively removing yourself from the GMAT/MBA rankings game.

Is that necessarily a good thing? Unfortunately, it depends. That Candidate B fellow, with a 750 GMAT? I’d call that guy “GMAT-positive” – he’ll almost certainly help an MBA program increase its GMAT score average, so swapping the GMAT for the GRE would be a mistake. The same would be true if your GMAT score is simply high for your demographic: if, for example, most applicants from your country or industry have lower GMAT scores than you do, then stick with the GMAT.

But if your GMAT score might be a liability in your MBA applications, then maybe the GRE is a good idea. I’m convinced that there’s always a little voice inside the adcom’s head, nervously babbling about average GMAT scores and MBA rankings. By taking the GRE instead of the GMAT, you can hush those voices, at least a little bit – and then maybe the adcom might be more attentive to how qualitatively cool you are in your sassy MBA admissions essays.

 

Although this little corner of the internet is called “GMAT Ninja,” the author of this blog post also offers GRE tutoring services in Denver, Colorado and online via Skype. If you’re not sure which test to take, call or email for more information… or try starting from the beginning of this series on the GRE vs. the GMAT. And if you want some insights into what, exactly, a “good” GRE score might look like, check out GRE vs. GMAT, part VI.

 

Inflated GMAT scores… and deflated quant percentiles

 

In the most-commented GMAT blog post I’ve ever written, I argued that the so-called “80th percentile rule” – the idea that top-tier MBA applicants need to score above the 80th percentile on both the GMAT quant and the GMAT verbal section – is mostly a myth.

Why? Well, the unfortunate truth is that GMAT quant scores have increased dramatically in recent years, and you’d now need to earn a 50 (!) on the GMAT quant section to score above the 80th percentile, according to the most recent GMAT data. Here, check this out:

From mba.com. http://www.mba.com/us/the-gmat-exam/gmat-exam-scoring/your-score-report/what-percentile-rankings-mean.aspx#tab4. Retrieved October 3, 2016.

From mba.com, based on 2013-15 GMAT data. Retrieved on October 3, 2016.

Plenty of things are alarming here. When the GMAT score scale was originally designed, the average quant score was supposed to be 30. Now a quant score of 30 is just the 20th percentile (!!), and the mean GMAT quant score is a 39 – a whopping nine points higher than it “should be” if the GMAT score distribution were actually a bell curve, as originally intended.

Even worse: if you want to score above the 80th percentile, you’ll need a 50 on the quant section. Since a 50 is one heck of a high GMAT score (a 51, of course, is a perfect GMAT quant score), I would still argue that no MBA admissions person in their right minds should actually expect it of you. So if anything, the “80th percentile rule” is more of a myth than ever.

But here’s the part of my original GMAT blog post that may not be true anymore, just a few years later:

A 710/47Q/40V is still an outstanding score that eliminates all rational doubt about your academic abilities. For the vast majority of MBA applicants, a GMAT quant score in the 73rd percentile is enough to placate MBA admissions officers, and your odds of admission will depend almost entirely on other elements of your profile.

Let’s pull this apart a little bit. Would a 710/47Q/40V eliminate all rational doubt about your academic abilities? Yes, definitely. If you can get a GMAT score that high, there’s no good reason to question your ability to handle MBA-level coursework. But is that GMAT score enough to “placate MBA admissions officers”? Probably not anymore, depending on which MBA programs you’re applying to — and, of course, on the rest of your MBA profile.

Here’s the problem: GMAT score averages for top-tier MBA programs have continued to shoot through the roof in the past few years. At Kellogg, for example, average GMAT scores have risen from 712 to 728 in just six years. Stanford’s incoming class of 2018 had an average GMAT score of 737.

I’m sorry, but those GMAT scores are ridiculous. Clearly, the GMAT arms race has gotten absolutely out of control – and I say that as a test-prep tutor who profits from that very same GMAT arms race. It’s as if MBA admissions committees have zero understanding of how standardized tests such as the GMAT actually work: the GMAT’s creators would tell you that the difference between a 710 and a 740 may be nothing more than measurement error.

And that’s before we get into the fact that there’s little evidence that the GMAT is predictive of post-MBA success. But that’s another topic entirely.

To be fair, I can’t blame MBA admissions committees for allowing GMAT score averages to spiral upward. Frankly, it’s much quicker and easier for admissions committees to whittle down a stack of applications based on GMAT scores than it is to ponder the non-quantitative aspects of each MBA applicant’s profile. Plus, MBA programs are forced to play the rankings game – and average GMAT scores are part of those MBA rankings, whether we like it or not.

Unsurprisingly, MBA admissions consultants are advising their clients to chase GMAT scores in the mid-700s – and applicants to elite MBA programs are working harder than ever to raise their GMAT scores. It’s a vicious cycle, and I’d bet that GMAT scores will continue to creep upward.

In any case, if you’re reading this little GMAT blog, you probably can’t do a damned thing about the way things work: you probably just want to get into a good MBA program. So what should you do if an MBA admissions consultant – or worse, an MBA admissions committee member – tells you that you need a GMAT score in the mid-700s?

Well, take solace in the fact that you’re not alone. And if you think it’s unreasonable that you might need a 750 on the GMAT exam, take solace in the fact that you’re probably right.

And then join the rest of the herd. If you have a profile that will allow you to get away with a lower GMAT score, then consider yourself lucky. But if you’re like most people, all you can do is work as hard – and as smart – as you can for as long as you can.

And if you’re already scoring in the low-to-mid 700s but don’t have time to continue studying, maybe it’s worth taking an extra couple of shots at the GMAT, just to see if dumb luck (or measurement error, if you want to be technical about it) might hand you an extra few points next time around. And if you’re a really long way from the GMAT score you want, then maybe it’s time to think about the GRE — but that’s another topic entirely.

Then, please graduate from your top MBA program, become a bigshot in the MBA world, and change the way that the GMAT is handled by MBA admissions committees – because I think that the GMAT arms race encourages even more great business talents to slip through the MBA cracks.