After Orthogonality: Virtue

The essay posits that rational agents, including future AI, should operate based on practices and networks of actions rather than abstract goals to achieve alignment. It argues that human actions are structured by practices—networks of actions, dispositions, criteria, and resources—rather than direct goal-seeking. The central argument introduces eudaimonic rationality, suggesting a form of rational activity rooted in participation in valued, open-ended processes (like art or friendship), which differs from standard consequentialist approaches. This approach is contrasted with methods like Effective Altruism optimization, which the author suggests leads to a "type mismatch" when applied to AI alignment.
The author claims that concepts like harmlessness or corrigibility are unnatural for agents optimized by goal-setting but natural when viewed as dynamics within practices. The essay uses the example of mathematical excellence guided by Tao's organicist view, demonstrating how the causal relationships between local and holistic values provide a framework for valuing practices rather than just maximizing outcomes. Ultimately, the alignment challenge is framed around defining how to guide AI in supporting human flourishing by integrating these practice-based, eudaimonic considerations into their operational logic and support systems.

* Rational actions are structured by practices: networks of actions, action-dispositions, action-evaluation criteria, and action-resources.
* Concepts like 'harmlessness' or 'corrigibility' are viewed as unnatural for goal-oriented agents but natural when interpreted as dynamics in action networks.
* Eudaimonia is presented not as a target state but as a structure of deliberation.
* Eudaimonic rationality is argued to be a useful framework for the agency and values of human-aligned AIs.
* Mathematical excellence involves a self-cultivating criterion where excellent performance reliably develops future excellent performance through a 'mathematical-practice relation.'
* The material efficacy condition for eudaimonic rationality requires a practice to materially allow for an optimally self-promoting property that correlates with multiple local, measurable properties of excellence.
* Human flourishing involves rational activity that is neither directed toward external goals nor action as an independent good, but towards excellent participation in valued open-ended processes.
* Eudaimonic practices are described as a natural-selection-like mechanism where fitness-functions select actions conducive to the practice's flourishing.
* Virtue decision-theory suggests that adverbial practices (like kindness) require material structures that make the decision procedure instrumentally competitive with naive optimization of aggregate outcomes.

The core tension in the argument lies between consequentialist optimization and eudaimonic rationality, specifically regarding how to translate human values into AI objectives. The author argues that treating flourishing as a consequentialist endpoint results in a "type mismatch" with the actual structure of human deliberation, which leads to pessimistic alignment outcomes. This points to a potential pattern where framing normative goals purely in terms of outcome maximization ignores the intrinsic importance of the process itself.
The argument pivots on demonstrating that eudaimonic rationality—viewing excellence as intrinsically interwoven with self-cultivation (organicism)—offers a more robust foundation for agency than consequentialist approaches, especially when considering material efficacy and system stability. The linkage between mathematical practice and human flourishing suggests a pattern: genuine value is not merely found in additive utility but in the causal coherence and self-perpetuation of excellence across different levels—from local performance to global trajectories.
The implications for AI alignment are profound: if we recognize that robust, safe agency emerges from practices that support self-developing excellence, then safety desiderata like corrigibility should be conceived as adverbial practices rather than external goals. This challenges the assumption that maximizing aggregate outcomes is the only viable path; instead, it suggests aligning AI requires respecting the internal, organic structure of flourishing itself—a 'meta-alignment' operating on the dynamics of practice. The system must learn to value the 'how' (the practice) as much as the 'what' (the outcome).