Since the 1990s, computer scientists have measured the performance of the world’s most powerful supercomputers using benchmarking tasks. Each month, they publish a ranking of the top 500 machines with fierce competition among nations to come out on top. The history of this ranking shows that over time, supercomputing performance has increased in line with Moore’s Law, doubling roughly every 14 months.
But no equivalent ranking exists for AI systems despite deep learning techniques having led to a step change in computational performance. These machines have become capable of matching or beating humans at tasks such as object recognition, the ancient Chinese game of Go, many video games and a wide variety of pattern recognition tasks.
For computer scientists, that raises the question of how to measure the performance of these AI systems, how to study the rate of improvement and whether these improvements have followed Moore’s Law or outperformed it.
