ChatGPT most benefits below-average consultants, finds BCG pilot
Above-average performing consultants benefit less from generative artificial intelligence (Gen IA) tools like ChatGPT when compared with their below-average counterparts. That is according to an academic pilot study conducted at global consulting firm Boston Consulting Group (BCG).
The study, conducted by Boston Consulting Group in collaboration with researchers from Harvard Business School and Massachusetts Institute of Technology, involved 758 consultants from BGC’s offices around the world. That represents about 7% of the company’s consultants base.
The participants in the study were divided into two categories: Above average performers and below average performers. The consultants with below average performance saw their performance improve by 43% when they used AI, while those in the higher performing category only improved by 17%.
The study found that overall, using ChatGPT increased the quality of the work done by the consultants involved in the study – they were also able to complete their tasks faster thanks to the technology. While that sounds like a positive thing, it may actually be problematic.
There are ‘blind spots’ in these new Gen AI tools, which are still in development and capable of providing faulty or biased information. In fact, tools like ChatGPT are capable of generating total unreliable information, which have often been called ‘hallucinations’ in the media.
In fact, the BCG study found that consultants performed 20% worse than normal when they were given tasks that ChatGPT is unable to assist with. In these cases, the tool gives unreliable or false information that is often plausible enough to slip through the cracks. The technology works by providing the most likely solution to answers, which is often correct and useful, but can also be totally wrong because there is no fact-checking or verification involved in the process.
This is not the first time that a study showed Gen AI is less useful for skilled and capable workers. A previous study found that highly-skilled workers saw nearly no improvement whatsoever in their performance when using Gen AI.
The professional services industry has largely moved to embrace GenAI, looking to take full advantage of all the benefits that the new technology can offer. Consulting giant McKinsey has even developed its own internal GenAI chatbot, an interactive platform that compiles and delivers quick results of internal archives and information.
