That Official GMAT Question Might Cost $3,000

Official GMAT practice questions, the real, retired questions from past exams and official publications, are worth far more than any third-party “knockoff” questions, and the reason comes down to cost. Each official question can cost $1,500 to $3,500 or more to develop, while a prep company spends just a few dollars per question. Here’s why that gap should shape how you study.

I spend a lot of time telling GMAT students that no GMAT practice problems can ever be as good as real, retired questions from official GMAT tests and publications. And if you read a bunch of GMAT blog posts written by other GMAT test-prep “experts,” you’ll find that many of them seem to disagree with me, since they’re trying to sell you a nice bundle of non-official GMAT questions and exams.

Sometimes even my own GMAT students object when I tell them to avoid using knockoff GMAT tests from other companies: “How terrible could those materials be? There are smart people working at those GMAT test-prep companies, right?”

Yes, of course there are smart people working at GMAT test-prep companies, though some test-prep firms clearly employ better GMAT question writers than others. But even at their best, no GMAT test-prep company can possibly compete with the precision of real, official GMAT questions. Part of the reason is that copying the style of an intricate standardized test is inherently challenging, but the bigger reason is basic economics.

Developing an Official GMAT Question Costs Money… a Lot of Money

If you’re reading a GMAT blog, you’re probably a business-savvy future MBA student, and you probably have some intuition for the business models of companies like Kaplan or Princeton Review. So take a guess: how much do you think those test-prep companies spend developing each individual question on their GMAT practice tests or in their GMAT books? Go ahead and think of a number.

OK, got something in mind? Great. Personally, I would imagine that GMAT test-prep companies spend somewhere in the range of $10 to $50 per “knockoff” GMAT question if they actually want to make a profit, though I could certainly be wrong.

Now let’s ask Dr. Lawrence Rudner, a former GMAT Chief Psychometrician, what the GMAT’s developers spend on each question:

Test items are costly to develop, often in the range of US$1,500 to US$2,500 per item.

That figure comes from a paper Rudner presented at a GMAC conference back in 2007. Pay attention to the date, because it’s the floor, not the ceiling. Adjust Rudner’s figure for nearly two decades of inflation and you’re comfortably past $3,000 per question today, before even accounting for how much more elaborate the development process has become. 

(Incidentally, I got halfway through a PhD in education policy and psychometrics, the statistical science behind standardized testing, before I came to my senses and realized that I’m much happier as a GMAT tutor than as an academic or psychometrician. If you’re interested in a career in psychometrics, contact me, and I’ll do my best to either help out or scare you away – whichever you’d prefer.)

So yeah, that GMAT question you saw on the latest version of the exam? It may have cost $3,000 or more to develop—perhaps a hundred times more than a profitable GMAT test-prep company could reasonably spend on each question. 

How a Real GMAT Question Gets Made

And if that sounds crazy, or at least wildly inefficient, it probably isn’t. According to GMAC, every potential GMAT question runs through a seven-step development and review process before it’s ever allowed to count on a real exam.

It starts with qualified subject-matter experts writing each item to detailed specifications. From there, the question is checked against a quality rubric to confirm it measures what it’s supposed to, then it goes through editorial and “soundness” reviews to scrub out flaws in its language or logic. After that, an international fairness panel, made up of people who reflect the global population of test-takers, reviews every item to screen out cultural bias or anything that might unfairly disadvantage one group. 

Only then does the question move to live pre-testing, where it appears as an unscored “experimental” question on real exams so GMAC can gather response data. Finally, the team statistically calibrates the item using item response theory to estimate its difficulty, how well it separates stronger test-takers from weaker ones, and how long people take to answer it. Start to finish, a single question can spend a long time working through that pipeline before it ever appears as a scored question. 

For every question that survives the process and makes it onto the GMAT, plenty of others are discarded along the way for being poorly phrased, for failing to measure the intended skill, or for quietly favoring one group of test-takers over another. 

You probably know that I’m not necessarily impressed by the GMAT’s ability to identify the most talented business leaders, but the GMAT’s creators deserve real credit for producing subtle, meticulously vetted questions. So ask yourself: how could any test-prep company possibly produce something of comparable quality or precision, given the economics we just walked through? 

So whenever you work through practice tests from major GMAT test-prep companies, or practice worksheets from a small GMAT tutoring company based in Colorado, remember to take your results with a huge grain of salt. Sure, our “knockoff” GMAT questions can absolutely help you build skills, but none of us can produce a truly accurate GMAT practice test, simply because none of us can afford the development process required to do so while keeping our business open.