In 1965, American engineer Gordon Moore made the prediction that the number of transistors integrated on a silicon chip doubles every two years or so. This has proven to be true to this day, allowing software developers to double the performance of their applications. However, the performance of artificial intelligence (AI) algorithms seems to have outpaced Moore’s Law.
According to a new report produced by Stanford University, AI computational power is accelerating at a much higher rate than the development of processor chips.
“Prior to 2012, AI results closely tracked Moore’s Law, with compute doubling every two years,” the authors of the report wrote. “Post-2012, compute has been doubling every 3.4 months.”
Stanford’s AI Index 2019 annual report examined how AI algorithms have improved over time. In one chapter, the authors tracked the performance of image classification programs based on ImageNet, one of the most widely used training datasets for machine learning.
According to the authors, over a time span of 18 months, the time required to train a network for supervised image recognition fell from about three hours in late 2017 to about 88 seconds in July 2019.
This phenomenal jump in training time didn’t compromise accuracy. When the Stanford researchers analyzed the ResNet image classification model, they found the algorithm needed 13 days of training time to achieve 93% accuracy in 2017. The cost of training was estimated at $2,323. Only one year later, the same performance cost only $12.
The report also highlighted dramatic improvements in computer vision that can automatically recognize human actions and activities from videos.
These findings highlight the dramatic pace at which AI is advancing. They mean that, more often than not, a new algorithm running on an older computer will be better than an older algorithm on a newer computer.
Other key insights from the report include:
- AI is the buzzword all over the news, but also in classrooms and labs across academia. Many Ph.D. candidates in computer science choose an AI field for their specialization in 2018 (21%).
- From 1998 to 2018, peer-reviewed AI research grew by 300%.
- In 2019, global private AI investment was over $70 billion, with startup investment $37 billion, mergers and acquisitions $34 billion, IPOs $5 billion, and minority stake $2 billion.
- In terms of volume, China now publishes the most journal and conference papers in AI, having surpassed Europe last year. It’s been in front of the US since 2006.
- But that’s just volume, qualitatively-speaking researchers in North America lead the field — more than 40% of AI conference paper citations are attributed to authors from North America, and about 1 in 3 come from East Asia.
- Singapore, Brazil, Australia, Canada, and India experienced the fastest growth in AI hiring from 2015 to 2019.
- The vast majority of AI patents filed between 2014-2018 were filed in nations like the U.S. and Canada, and 94% of patents are filed in wealthy nations.
- Between 2010 and 2019, the total number of AI papers on arXiv increased 20 times.