Jack Clark (whose Import AI newsletter
is consistently excellent) reports
on an interesting AI study out of China
. Researchers trained a “brain-scale” model with over a trillion
parameters (and demonstrated that the model should scale to close to 200 trillion parameters). These are enormous numbers. OpenAI’s GPT-3, which we’ve discussed many times before
, was trained with 175 billion
parameters. It’s not as simple as “more parameters = more powerful model”, but as we discussed in TiB 128
, there’s good evidence that we’ve not yet reached the point where the benefits of scaling existing machine learning techniques are tapped out.
This has important implications for the future of AI. I recommend Gwern’s write up of what’s known as the “scaling hypothesis
” for more on this. Two other points worth noting. First, we’ve discussed before (see TiB 159
) the growing strategic value for a country of owning the full “AI stack”. This paper is a good example: there’s a hardware as well as a software innovation here. The model was trained on a new Chinese supercomputer; much of the novelty comes from the way they set up the machine to deal with such enormous scale.
Second, as Jack notes, this paper is evidence of the sort of public/private collaboration which is rare in AI research in the West. Some of the authors come from Chinese tech giant Alibaba, but others from Tsinghua University and the Beijing Academy of Artificial Intelligence. Why does this matter? In Clark’s words:
[I]nitiatives like this are a rarity in the West, which is dangerous, because it means Western countries are handing over the talent base for large-scale AI development to a small set of private actors who aren’t incentivized to care much about national security, relative to profits
If governments in the West are serious about the strategic power of AI (and they should be!), it’s crucial that they think through and plan for this dynamic. As some commentators have pointed out recently (excellent thread
), it’s easier and cheaper for governments to worry about AI ethics than investing in actual AI.