GMAT test prep companies

That official GMAT question might cost $3000

  I spend a lot of time telling GMAT students that no GMAT practice problems can ever be as good as real, retired questions from official GMAT tests and publications. And if you read a bunch of GMAT blog posts written by other GMAT test-prep “experts”, you’ll find that many of them seem to disagree with me – since they're trying to sell you a nice bundle of non-official GMAT questions and exams.

Sometimes, even my own GMAT students object when I tell them to avoid using knockoff GMAT tests from, say, Manhattan GMAT or Veritas: “How terrible could those materials be? There are smart people working at those GMAT test-prep companies, right?”

Yes, of course there are smart people working at GMAT test-prep companies – though some test-prep firms clearly employ smarter GMAT question-writers than others. But even at their best, no GMAT test-prep company can possibly compete with the precision of real, official GMAT questions. Part of the reason is that copying the style of an intricate standardized test is inherently challenging, but the bigger reason is basic economics.

If you’re reading a GMAT blog, you’re probably a business-savvy future MBA student, and you probably have some intuition for the business models of companies like Kaplan or Manhattan GMAT. So take a guess: how much do you think those test-prep companies spend developing each individual question on their GMAT practice tests or in their GMAT books? Go ahead and think of a number.

OK, got something in mind? Great. Personally, I would imagine that GMAT test-prep companies spend something in the range of $5-30 per “knockoff” GMAT question if they actually want to make a profit, but I could be wrong.

Now let’s ask Dr. Lawrence Rudner, former GMAT Chief Psychometrician, how much the GMAT’s developers spend on each practice question:

Test items are costly to develop, often in the range of US$1,500 to US$2,500 per item.

This comes straight from a paper that Rudner presented at the GMAC Conference on Computer Adaptive Testing way back in 2007. If question development costs have kept pace with inflation, then each GMAT question would cost somewhere between $1,700 to $3,000 per question in 2016, though that figure could be even higher if shortages of trained psychometricians continue to inflate GMAT development costs.

(Incidentally, I got halfway through a PhD in education policy and psychometrics – the statistical science behind standardized testing – before I came to my senses and realized that I’m much happier as a GMAT tutor than as an academic or psychometrician. If you’re interested in a career in psychometrics, send me an email, and I’ll do my best to help out.)

So yeah: that GMAT question you saw on the latest version of the exam? It might have cost $3000 or more to develop – perhaps 100 times more than a GMAT test-prep company could reasonably spend on each question.

And if that sounds batsh*t crazy – or at least wildly inefficient – it probably isn’t. Actual GMAT test items go through a painstaking cycle of editing, testing (often as “experimental questions” on actual GMAT tests), re-editing, and re-testing before they actually become part of a real GMAT exam. For every question that actually makes its way onto the GMAT, countless others are discarded for any number of reasons. For example, there might be evidence that the discarded GMAT questions were phrased badly, that they don’t seem to capture the skills the GMAT promises to measure, or that they (accidentally) discriminate against GMAT test-takers from certain demographic groups. Much of this testing requires substantial skill and training – and employees with these skills certainly aren’t inexpensive.

You probably know that I'm not particularly impressed by the GMAT’s ability to identify the most talented business leaders, but the GMAT’s creators deserve credit for producing subtle and meticulously phrased questions. How could any GMAT test-prep company possibly produce anything of comparable quality or precision, given test-prep firms’ economic limitations?

So whenever you work through practice tests from major GMAT test-prep companies – or GMAT practice worksheets from an obscure little GMAT tutor dude in Colorado – remember to take your results with a huge grain of salt. Sure, our “knockoff” GMAT questions can probably help you build skills, but none of us can ever give you a truly accurate GMAT practice test – simply because none of us will ever be able to afford to do so if we actually want to keep our doors open.


beware the knockoffs, part I

I was reading a thread on a GMAT forum a few days ago, and saw that the owner of a test-prep firm was telling students that they should avoid independent GMAT tutors who rely on commercially available GMAT materials.  The owner's logic sounded good (hey, look, we know enough about the GMAT to write our own book!  hire us!), but I wholeheartedly disagree with the idea that you should use GMAT materials written by random test-prep companies.  The only GMAT materials you should completely rely on are the GMAT official guides--which are, of course, commercially available.  Anybody who claims that they've written something better is either lying, or completely deluded about the complexity of the GMAT. It is extraordinarily difficult to copy the writing style of the GMAT, even on math questions that contain few, if any, words.  The GMAT is a ridiculously complex test, which is exactly why I love it.  (And, perhaps, exactly why you hate it.)  I don't know exactly how many brilliant test-writers are employed by GMAC, but I suspect that it's a fairly large fleet of people, and I'm certain that questions are very thoroughly vetted before they are ever inserted into the scoring algorithm for the real test.

The challenge for "knockoff" GMAT writers is even greater on the verbal section.  I've written and edited TOEFL practice tests, and I can say that took an enormous amount of effort to make our material sound even vaguely like the real thing.  GMAT verbal questions are even more detailed than TOEFL questions, and it's extraordinarily hard to write GMAT questions that will be "hard in the same ways" as the real thing.  I've found countless mistakes in widely-published GMAT verbal material, and the companies seem to struggle with all three verbal question types.  Frankly, most of the mass-market GMAT test-prep publications will do more harm than good if you're using them as your primary verbal prep resources.

A majority of my students contact me after they've already taken either a GMAT prep course, or a reasonably thorough self-study course.  I've seen refugees from Veritas, Manhattan, and Kaplan in roughly equal numbers, as well as an occasional graduate of Princeton Review or other (less well-known) competitors.  Some of these GMAT companies are better than others (and one, in my opinion, stands head and shoulders above the rest), but the bottom line is that all of them are inevitably limited in their ability to replicate the GMAT experience.

For what it's worth, the company that makes the very best "knockoff" GMAT materials manages to capture about 80-90% of the real GMAT experience, and that's an astounding achievement.  (I feel that it would be inappropriate to make an endorsement here, so I won't name the company.)  This same company probably makes its verbal questions look like the actual GMAT only 70% of the time--it's amazing how often I see (or hear of) real GMAT questions that fall outside the scope of even the best "knockoff" curriculum.  Again, I think that this company is absolutely amazing for doing as well as they do, but that shows how hard the task of "reproducing" the GMAT really is.

The other GMAT test-prep companies?  Not even close, and I've looked at material from some of the most expensive, "boutique" test-prep firms, as well as the household names.  As a general rule, the "knockoff" verbal questions will do more harm than good, and even the second-best large company produces GMAT tests that are riddled with typos and errors.

I mean none of this disrespectfully--quite the opposite, actually.  These firms throw a ton of resources into an extremely difficult task, and their results are necessarily mixed.  But before you buy a GMAT practice resource sold by a small company, be very aware that the material will probably be filled with inaccuracies.  There are presumably a few gems out there, but all of them should be handled with extreme care.

I'm regularly asked if I include some sort of textbook or workbook or self-published GMAT guide as the backbone of my tutoring program.  I write supplementary GMAT questions all the time, but trust me on this:  you probably wouldn't want to hire me if I relied primarily on a self-published resource.  A good GMAT tutor or test-prep firm will supplement lessons with their own materials, but any independent tutor who says that they've written a magical, best-in-the-industry GMAT resource is probably exaggerating, if not egregiously lying.