The radicalism of Positive Alignment

Could it go further?

Last week researchers from DeepMind, OpenAI, Anthropic, Oxford and other organisations co-wrote a paper on Positive Alignment - building AI where ‘human flourishing is a technical target’.

Raphael, School of Athens

What is positive alignment?

But what does ‘Human Flourising’ mean? This is where ‘positive alignment’ is a radical departure compared to standard software. Most consumer technology optimises for revealed short-term behaviour (based on clicks, scrolls, session length). But is my ‘scrolling in bed at 8am’ self really the one I want? In fact, many products are deliberatlely built to hijack these short-term impulses via the dopamine sysyem (see Hooked for more details on this this.) Relatedly, mainstream economics (especially from 1950s-70s in the West) was based on the assumption of ‘Rational Choice Theory’. Human preferences were modelled based on observed choices. Behavioural economics has significantly complicated this model (e.g. Amartya Sen’s Rational Fools or behavioural economics literature) but demand estimation, welfare analysis and policy evaluation all still tend to use revealed preference logic as their basis.

Current AI models inherit a similar notion for discerning human preferences. RLHF (Reinforcement Learning with Human Feedback) is also based on revealed preferences. The original RLHF in InstructGPT (Current methods are rarely disclosed by the major AI labs) involved the model generating several candidate responses to a prompt and human labellers ranking these responses. The rankings were converted to a single scalar reward using the Bradley-Terry model. This model comes from utility theory in psychology and it places the preferences onto a single linear metric. Crucially, in this form of RLHF post-training the labelling is done in the moment. There is no longer time-horizon consideration. Just as when I’m lying in bed scrolling I’m not prioritising my long-term self, RLHF structurally prioritises in-the-moment preferences.

The most radical move by proponents of positive alignment is that human flourishing should be modelled in a much richer and more dynamic way than either current AI models (or our economic models) currently achieve. This isn’t simply a question of how to use AI, but the authors argue we can build a vision of the good life into the fabric of AI models.

Source

How does Positive Alignment work?

The paper considers (approximately) four layers to implementing their programme of positive aligment.

1. Whole AI lifecyele: Technically, positive alignment should operate throughout the whole lifecycle of model creation, not just as a bolt-on at the end (like RLHF in post-training is currently).

2. Long time horizons: Perhaps their most interesting thought is that we should consider the impact of the technology on an individual over longer time horizons. Unlike social media, AI is too important a technology to optimise to our 8am scrolling in bed selves. As the authors note, ‘flourishing lives are not necessarily smooth’.

3. Multi-dimensional good lives: They want to see they see the four theoretical approaches discussed (hedonic, conative, objective list, perfectionist) as working together in a ‘virtuous circle’). They acknowledge that currently LLMs are more homogenous than the populations they serve and simply picking one framework (utilitarianism) and optimising for it won’t represent the variety of users.

4. Role-based approach to flourishing: One of the most important points raised in the paper (following Zhi et al) is to suggest that positive alignment shouldn’t just be based on preferences, but on role-based normative standards. This means we should think about an AI tool for learners, for example, based on the role of a learning (what do they need pedagogically).

They suggest a range of technical proposals to support this, including positive alignment data curation, where specifically chosen data that pushes the model towards valuing human flourishing are prioritised. ‘Reinforcement Learning from AI Feedback (RLAIF) to optimize for active virtues like honesty and helpfulness’, ‘modular pluralism’ (achieved by specialised community language models), a benchmark measuring morality (MoReBench) and middleware markets for alignment packages. This would see multiple competing alignment wrappers competing (but based on the same underlying models).

Will Positive Alignment work?

The authors themselves point out that the emergent behaviours of these ‘strange new minds’ suggests that it isn’t easy to specify and control this technology. In fact, this is one of their criticisms of existing ‘negative’ alignment work. The idea that ‘providing positive attractors that naturally lead models away from shallower attractors such as sycophancy’ is worth seriously testing and working on, but is ambitious. We should be clear-eyed about the inherent level of uncertainty grown into these systems.

The governance layer of their proposal is most questionable aspect. Despite utilising the theory of Elinor Ostrom’s polycentricity, a number of the institutional solutions proposed reach for markets and competition. This includes ‘middleware markets’ where they provide the example of selling a homeschool alignment layer product. This ‘alignment-as-a-service’ approach, as well as regulatory markets with competitive providers is a contradictory solution to reach for. Given that the paper argues against revealed preferences as a basis for discerning human flourishing, isn’t this exactly what an alignment layer market would be based on (except the revealed preference is buying)? The idea of a competitive market for alignment packages to create diversity might fall into the trap of other market-based solutions. There have been many markets which were supposed to decentralise (the browser market, search market, mobile OS market, social media market, app store market, cloud market) but didn’t. Would this be different?

It is also questionable as to whether this analysis can work if the unit of analysis is the technology itself. Many of the authors themselves work in major labs, which means this paper is more likely to be consequential, and explains why the unit of analysis is the technology itself. However, would a comprehensive vision of Positive Alignment not need to include professional associations, civic bodies, international coordination?

One omission for a paper this ambitious is the theory of change. How does positive alignment get implemented? Is it just that AI labs and governments think it’s a good idea and decide to invest time and effort into it? Given many of the co-authors are in positions at major AI labs, perhaps this is the assumed path. However, it’s hard to see how positive alignment won’t be political, and without a political analysis the programme risks sliding into solutionism. This is a radical proposal and needs a commensurately radical theory of change to bring it about.

There are many unanswered questions. If RLAIF is used to mediate between AI and our understanding of the good life what happens? What happens when a compressed version of human values mediates with AI, also a compression of human culture? Is there space to decompress (or generate, in the true sense of the word)?

This paper is an initial attempt to set a direction for a new sub-field, and in that sense it is both important and exciting. Most critically, the paper acknowledges that AI is not just a technical challenge with technical solutions without explicitly named values. It is a cultural technology that requires explicit value-based goals.