Analyse this…
An article in yesterday’s BusinessWeek challenged the methodology employed by technology analyst firm Brown & Wilson Group. This outfit apparently produces a ‘black book’ of rankings of IT outsourcing firms. However, the firm has been criticised for allegedly employing dodgy methods to arrive at its rankings. The result is that a firm’s position can fluctuate wildly from one publication to the next.
The article implies that Brown & Wilson’s methodology is suspect in the context of other tech analysts. According to the piece, “Brown & Wilson’s approach is different from most established tech researchers. Firms such as Gartner and Forrester Research employ dozens of analysts who use detailed phone surveys of high-ranking tech purchasing executives as one of their research methods. Gartner’s so-called Magic Quadrant reports typically show only gradual movement in companies’ positions from year to year.”
Hmm. I’m not so sure. As someone who has worked in tech and B2B marketing for years it’s still not clear to me what methodology Gartner employs to arrive at its magic quadrants. These things map companies on two continua: ability to execute and ‘vision’. The holy grail for a tech firm is to make it to the upper right quartile i.e. to be perceived as a visionary leader.
However the means by which companies make it onto a magic quadrant is not clear – and appears to be very subjective. Early stage companies find it notoriously difficult – especially as access to Gartner analysts is very restricted if vendors are not actually clients of the firm’s. I have faced situations in the past where analysts are made available for a briefing but business development reps have sat-in.
Of course, everyone has to make a crust – and Gartner has made quite a few billion of them. But the magic quadrant is Gartner’s cash cow. It provides an easy to understand spatial relationship between vendors in terms of their technology and business acumen. Therefore it makes the job of a CIO easier in terms of making vendor choices. However, the number of magic quadrants has proliferated over the years making it appear that Gartner sees the product as a key money-spinner.
Magic quadrants and black books must teach the wider market research and business intelligence community a more fundamental lesson, however. If research can be condensed into a simple metric it’s more likely to become useful and pervasive. Too many research companies fixate on data rather than the simple message it conveys. But in the case of most survey research firms they do at least have sound methodologies behind their findings.
