TiB 223: "Democratic AI"; designing research funders; the geopolitics of rare earths; and more...

Welcome new readers! Thoughts in Between is a newsletter (mainly) about how technology is changing politics, culture and society and how we might do it better.

It goes out every Tuesday to thousands of entrepreneurs, investors, policy makers, politicians and others. It’s free.

Forwarded this email? Subscribe here. Enjoy this email? Forward it to a friend.

Can AI design popular policies?

DeepMind published an interesting piece in Nature which introduces an idea the authors call "Democratic AI" (New Scientist has a good short write up). The paper demonstrates an AI model that designs "policies" for a simple political economy-type game; these policies prove to be more popular with human players than those created by other humans. The authors argue that this is a potentially important result for AI alignment, in that the AI learns how to deliver a policy mechanism to which humans consent through a majoritarian democratic process.

The game itself is simple. Each player starts with an initial endowment of resources and each turn has to decide how much of this to contribute to a public investment fund. These investments, by assumption, have a positive return and the proceeds are distributed among the players according to a policy. For example, the pot could be split equally ("strict egalitarianism"), proportional to contribution ("libertarianism") or proportional to the share of initial endowment contributed ("liberal egalitarianism"). The AI's task is to design a policy that human players will select in a majoritarian vote. It did this by broadly "rediscovering" liberal egalitarianism, but where players receive almost no payout unless they contribute at least half of their endowment.

We've discussed AI-designed policy before, back in TiB 113, in the context of Salesforce AI's "AI economist". This is quite different though: DeepMind's goal is to not to optimise for a particular policy outcome, but for democratic consent. As David Dalrymple points out in this excellent thread, this may not be a good thing! (To be clear, the DeepMind authors acknowledge this possibility) Aligning human and AI values is important, but there’s a potential tension between designing for what we want and what is good for us… and teaching AIs to learn how to give us the former without the latter has some dangerous failure modes.

How to design a research funder

My friend Ben Reinhardt (see TiB 125 and 159) has a great post on the design philosophies of research funding organisations, a favourite TiB topic. He starts by noting that two pieces of conventional wisdom about research seem to be in tension: on the one hand, great research requires a high degree of intrinsic motivation (and hence freedom for researchers); on the other, research management (i.e. somewhat reducing freedom) matters a lot. Ben spends the rest of the post reconciling those positions and laying out some of the ways that the great research organisations of the past have dealt with the tension (Bonus: if, like me, you're a sucker for a 2x2 diagram, it includes a very good one).

I've thought a lot about this problem because a version of it comes up a lot in building any sort of programmatic support for entrepreneurs (as I do in my day job at Entrepreneur First). Great founders, we generally assume, don't need or want to be told what to do, so what use is a programme? Ben provides one* very good answer: "context changes the types of ideas people have". In other words, one useful function that a programme like YC or EF is to make a particular kind of ambition legible (See also my essay on the history of ambition). The existence of startup investment programmes therefore increases the supply of entrepreneurs.

In the same way, the "bat signals" (see last week's TiB) that the design of research organisations send up will profoundly shape the supply of researchers:

in a world where research is funded through broad grant calls to PIs for them to pursue specific projects where the the majority of work is done by grad students and which are judged on their scientific contribution and publication record, you’re going to see a lot of ideas that appeal to broad grant calls that can be done by grad students that are seeking publishable scientific contributions

... and of course we should hope that new kinds of research organisation can increase the supply of the visionary, risk-taking scientists we need (and that we've talked about a lot in TiB - see TiB 103, 165 186 211, 217, among many others). How to do this is not a fully solved problem, but as Ben shows, the history of the space suggests some interesting starting points.

*There are certainly others - not least creating "scenius", as we discussed in TiB 116 and 129 - but alas I'm out of space to discuss them this week!

Stubborn persistence of the physical: rare earth edition

We've talked a lot here about what I've called "the stubborn persistence of the physical". The idea is that while software really is eating the world and the pandemic really has accelerated the virtualisation of everyday life (e.g. remote work is here to stay - fascinating new data), these trends have not erased the strategic importance of physical territory and possession. In fact, where the world of bits relies on the world of atoms, control of those atoms is arguably more valuable than ever. We've talked about this in the context of COVID PPE, internet security, corporate control and, above all, the semiconductor manufacturing.

This week saw intriguing news that fits squarely into this theme: Turkey has discovered nearly 700m tonnes of rare earth elements (REEs) in its territory and plans to begin mining them this year. That might sound very dull, but REEs have become key components in the manufacture of many important products, including electronics and electric motors. More here. Until now, the vast majority of the world's REE reserves were in China, which has attempted to use this position to its strategic advantage (just as the US continues to do with its control of the semiconductor supply chain - see the new story about US pressure on ASML - see TiB 174 - this week).

Unsurprisingly, you can already find Chinese commentary playing down the find - and it's absolutely true that REE deposits aren't very useful without the associated processing and refinement supply chain (more here on this, including some skeptical voices). But it's a useful reminder that while some of the strategic consequences of the "stubborn persistence of the physical" will take decades to change (as we've discussed before, China can't, for example, build domestic 5nm chip manufacturing capability overnight; nor can Europe, it seems, easily wean itself off Russian gas), others like REE are subject to unpredictable exogenous shocks.

Quick links

Of course, my work replicates. What is the best predictor of a study replicating? Troubling.
Barbells in everything, movie edition. Cinema continues to hollow out.
"How science actually works". Excellent book list from TiB friend Sam Arbesman (see also: TiB podcast episode with Sam).
If you're a daytime talkshow watcher, how come you're so rich? Interesting data on media consumption by income level.
Bad news for guinea worms... great news for the world!

Here we go again...

As always, thanks for reading. If you like TiB, I'd love you to forward it to a friend or share it on Twitter.

Do feel free to hit reply you have comments, feedback or suggestions.

Until next week,

Matt Clifford

PS: Lots of newsletters get stuck in Gmail’s Promotions tab. If you find it in there, please help train the algorithm by dragging it to Primary. It makes a big difference.