Sebastian Thrun, a computer scientist, entrepreneur, and adjunct professor at Stanford University (whose projects have included Waymo, Google X, Google Street View, and more) has launched a study to evaluate “the capabilities and limitations of AI systems for writing English-language philosophy.”
Called PhilosophyBench, the study is focused on “whether AI can generate novel philosophical ideas and develop them clearly with depth and sophistication, not merely summarize existing views or apply existing philosophical theories.”
The advisory board for the study includes philosophers Ned Block, Nancy Cartwright Ruth Chang, Kit Fine, Gideon Rosen, Jonathan Schaffer, Crispin Wright, and Linda Zagzebski, as well as computer scientists.
The project involves developing “an independent academic benchmark against which claims about capabilities in AI philosophical reasoning and writing can be evaluated. If a new AI model comes out and claims are made about its philosophical capabilities, we hope that this study will provide an independent methodology that can judge those claims.”
The questions to be addressed by the project include (according to its FAQ):
- Can an AI generate novel philosophical ideas and develop them clearly with depth and sophistication?
- Can an AI generate essays that could be accepted to a philosophy journal, or earn admission to a leading philosophy PhD program?
- To what extent does providing human philosophy BAs and PhDs access to AI tools change the quality of the philosophical writing that they produce (this is called an “uplift” study)?
- Given that many frontier AI models are already pre-trained on an extensive set of philosophical work across human history, what limitations, if any, prevent frontier AI models from producing philosophical work at the quality level of the best philosophical work in human history?
- When AIs write philosophy, are there any patterns in how they write that reveal anything about AI alignment, particularly when AIs are asked to produce novel arguments?
- What, if anything, might the philosophical capabilities of current AI systems tell us about how future AI systems might develop?
The project is currently looking for people with philosophical training to help with the project (“professional philosophers, graduate students, or advanced philosophy undergraduates, as demonstrated through formal education or other appropriate evidence”). Further details are at the PhilosophyBench site.