Computer Science > Computation and Language
[Submitted on 26 May 2026]
Title:A Virtual Member of a Community of Practice for the Society of Petroleum Engineers: From Prototype to Deployment
View PDFAbstract:We describe the evolution of a virtual assistant, called ATHENA, designed to support the capture, retrieval, and dissemination of knowledge for members of a Community of Practice (CoP) related to the Oil and Gas sector. An evaluation of a first prototype involving 75 professionals from the Society of Petroleum Engineering (SPE) showed that ATHENA dramatically improved both their productivity and performance equality on a set of realistic well-planning tasks compare to their use of a state-of-the-art RAG baseline system. However, the evaluation also identified areas for improvement. This paper describes technical advances to our first prototype in the areas of multi-document retrieval, support for answer validation, and more focused proactive dissemination. Evaluation results show that this enhanced version of ATHENA provides better support for completing knowledge-intensive tasks related to well planning than does a state-of-the-art baseline. ATHENA has been integrated into the SPE Research Portal and is being deployed for use by the society's membership.
Current browse context:
cs.CL
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Facts Only
* ATHENA is a virtual assistant for a Community of Practice in the Oil and Gas sector.
* The system supports knowledge capture, retrieval, and dissemination.
* A first prototype was evaluated by 75 professionals from the Society of Petroleum Engineering (SPE).
* The evaluation compared ATHENA to a state-of-the-art RAG baseline system using well-planning tasks.
* The first prototype showed improvements in productivity and performance equality.
* Technical updates to the system include multi-document retrieval, answer validation support, and proactive dissemination.
* The enhanced version of ATHENA was evaluated against a state-of-the-art baseline for knowledge-intensive well-planning tasks.
* ATHENA is integrated into the SPE Research Portal.
* The system is being deployed for use by the SPE membership.
* The work was submitted on 26 May 2026.
Executive Summary
ATHENA is a specialized virtual assistant developed to assist the Society of Petroleum Engineering (SPE) in managing the lifecycle of technical knowledge within the Oil and Gas sector. By focusing on the specific needs of a Community of Practice, the tool aims to streamline the capture and retrieval of information necessary for complex well-planning tasks. Initial testing with 75 professionals indicated that the system outperformed standard Retrieval-Augmented Generation (RAG) baselines, specifically in increasing productivity and reducing performance variance among users.
Following the initial prototype, the system was upgraded to include enhanced multi-document retrieval, mechanisms for validating answers, and proactive information dissemination. These technical advancements further improved the assistant's ability to support knowledge-intensive workflows. ATHENA has now moved from the prototype stage to active deployment within the SPE Research Portal, where it is available to the broader membership.
Full Take
The transition of ATHENA from a prototype to a deployed tool represents a significant application of domain-specific AI. From a methodological standpoint, the use of a RAG baseline provides a necessary control, and the sample size of 75 professionals offers a reasonable initial signal. However, a rigorous peer reviewer would seek more transparency regarding the "performance equality" metric—specifically, whether this implies the tool elevated low-performers to a median level or simply standardized the output regardless of user expertise. The claim of "dramatic" improvement in productivity is a high bar; the underlying data on time-to-completion versus accuracy would be essential to verify if this is a genuine gain or a perceived efficiency.
This work extends the current understanding of AI by moving beyond general-purpose LLMs toward "Virtual Members" of a community. It suggests that for high-stakes engineering, the value is not in the generative capacity of the AI, but in the precision of the retrieval and the validity of the source documents. If the findings hold, the implication is a shift in professional training: the "expert" may no longer be the one who possesses the most knowledge, but the one most capable of validating AI-retrieved technical data.
To further strengthen these claims, a longitudinal study is required to determine if reliance on ATHENA leads to "knowledge atrophy" among junior engineers over time.
Bridge Questions:
1. Does the standardization of performance equality come at the cost of the "expert's edge" in complex, non-standard scenarios?
2. How does the system handle conflicting technical data from different documents within the same community of practice?