Gartner MQ, Forrester Wave: A Disservice to Innovation?

Note: Read this blog more as a wakeup call to vendors than as a criticism of Industry Analysts. By inadvertently playing into analysts’ narrow definition of products and categories, vendors can become distracted from their real goal – pursuing meaningful, valuable, and unique innovation. 

Prologue
Industry analysts have an important job in the technology industry: To help customers better understand trends, technologies, and vendors, so they can make better, more confident purchasing decisions. And for vendors, analysts are helpful by providing market research, trend analysis and evaluation of products and technologies.

But in my experience, analysts also have a fine line to walk when they make vendor evaluations. Since analysts take money from both vendors and customers, they put themselves in potential conflict-of-interest positions, if not also risking bias.  This is made worse by lack of transparency in their evaluation methodologies.

The most popular – and influential – evaluation tools analysts offer are periodic product analyses and ratings. The well-known Gartner Magic Quadrant (MQ) and Forrester Wave are at the top of the list, with others such as KuppingerCole’s Leadership Compass. These reports are pervasive, with Wikipedia listing 66 different Gartner MQs – with many more unlisted.

Unintended consequence: Category myopia

A closer look at how the MQs, Waves, and other evaluation are constructed reveals massive spreadsheets that vendors are asked to complete. They largely focus on product feature comparisons, and to a lesser degree, company operations. I’ve personally helped complete countless numbers of these – and it’s often a multi-person, multi-week vendor effort.

But the spreadsheet questionnaires have an unintended consequence: They inadvertently treat each vendor product/category as a fungible, semi-generic solution to meet a limited set of problems.  Further, they often track technology categories that can quickly become outdated, even as analysts struggle to update them.

The result is that vendors are compelled to play the analyst’s game... not their own.   The unintended myopia – the forced “thinking-inside-the-box” – manifests itself this way:

• Vendors end up spending time, R&D resources, and marketing expense to chase feature boxes that will yield them high evaluation scores.  This inadvertent “keeping up with the Jones’ ” for feature parity does an injustice to innovation they would otherwise pursue.  It is the rare visionary vendor that’s able to say “screw the features the analysts want, we want to innovate in a different direction”.  And, unfortunately, those same vendors might be penalized in the next vendor evaluation for not “checking the boxes” even though they may have a breakthrough approach to the market.
• Customers may also be misled by these evaluations, often narrowly viewing the product sector through the narrow lens presented by the analysts.  While some customers will benefit by an apples-to-apples comparison of features, many may miss appreciating the variation in vendor options, approaches, and overall direction/strategy – things not generally reflected in simple checkbox evaluations. 

True innovators are penalized
A great set of podcasts by Christopher Lochhead focus on “legendary marketers” and innovators who re-think their products, market strategies, and ultimately create new concepts and categories.  But in the world of  standard analyst categories, innovators of new categories are penalized, because (a) they are not being considered for a MQ or wave, or (b) receive poorer ratings on the standard category ratings... even if they offer a truly revolutionary approach to solving a technology problem.


Now, to play my own devil’s advocate, I recognize the need for analysts to create some level of standardized evaluation criteria – the industry needs this. However, analyst criteria can be mistakenly held as the end-all and be-all, rather than as general guidance.  Further, most analysts fail to go the extra mile to fully explain the differentiations between vendors and products.

Note to Vendors: Analysts assess the Finite Game, not the Infinite Game
In his excellent new book The Infinite Game, Simon Sinek outlines the notion of great companies focusing on the “infinite game” – one where there is vision, constant reinvention and constant shifting of the playing field... and even shifting the definition of what it means to succeed.  This is in contrast to playing the “finite game” where the rules are defined, there is a limited set of metrics, and a singular clear goal to win.

The current mindset which is rewarded by MQs and Waves is that of the Finite Game... where vendors are encouraged play to their competition, innovate with the “check-the-box” mentality, and where customers might errantly treat vendor solutions as generic and fungible.

My candid vendor advice is this: Yes, we need to play the analyst’s game. There is absolutely a service to our customers using this approach. But proceed with caution and intelligence – There may not be not a need to “check all the boxes”, nor necessarily should you. Balance that effort against your own vision, direction and approach to differentiation. And ensure that if you opt for a powerful direction, albeit one that could weaken your evaluation, emphasize your believe/vision with the analysts.  Remember: There are lots of billion-dollar firms that aren’t leadership quadrant companies.

My advice if you’re a customer: Don’t blindly choose to put the top 3 vendors of a leadership quadrant on your short-list.  Look more deeply, and have your own set of criteria developed when selecting a vendor.   All-too-often, a vendor might have a solid offering in a given category, but ultimately fail to demonstrate the strategy and direction that will carry your company forward a few years in the future.

Coda: A note to analysts: 
I have the greatest respect for you – and for the incredible knowledge you have and advice you give.  (Even some of my own best friends are Analysts).  But please emphasize that your evaluations are standardized.  Go the extra mile to really understand and communicate

• how vendors differentiate their products
• where vendor visions lead and/or diverge
• adjacent categories to the vendors’ own (or categories that overlap)
• how the vendor’s products are either “pure-play” in the category, or how it expands the category definition. 

I look forward to your comments/feedback