The biggest unresolved question in AI right now, and more takeaways from Madrona’s IA40 Summit

Executives from Microsoft, Amazon, Anthropic, and Stripe discussed the future of AI agent relationships with customers and data ownership at Madrona’s IA40 Summit. A key unresolved question was who retains control over customer data generated by AI agents. Executives noted that AI agents are expected to mediate between companies and customers. For example, an executive from Microsoft predicted most business software would be used by AI assistants, potentially reducing software makers' pricing power. Furthermore, AI agents can operate applications like a person, leading to the concept of "thin apps" where the application itself matters less than the AI's ability to interact with the computer. In retail, commerce mediated by AI agents faced stagnation until recent sharp growth, as agents bypass traditional checkout and ad revenue streams. Regarding data ownership, Anthropic’s CTO stated that software providers will use available data to improve agents. Amazon Web Services introduced AWS Context to map company data for agent use across platforms. Concerns were raised about whether agents will adhere to company rules, leading some executives to suggest pairing agents with separate systems to enforce compliance. The concentration of capital in a few AI entities remains significant, with OpenAI, Anthropic, and Databricks accounting for a large share of funding.

The narrative surrounding AI agents is framed by a tension between technological capability and institutional control. The central conflict revolves around emergent agency: if AI can operate systems autonomously, the traditional ownership structures of software, commerce, and data become porous. The discussion highlights that the technical mechanism for operationalizing AI (e.g., AWS Context) seems to be an infrastructural layer being built beneath the philosophical debate on governance. The pattern observed is a defensive reaction from incumbents—like Anthropic arguing for ecosystem benefits—against the potential fragmentation of control promoted by agentic systems, which inherently favors those with superior integration or control over the underlying data infrastructure. The argument that "the bottleneck is human" reveals an underlying systemic friction: the gap between current AI capability and organizational capacity to manage it securely and ethically is where real constraints lie, not in the code itself. This suggests a pattern of centralized power attempting to maintain relevance by demanding that external actors improve systems in ways that benefit the existing structure. The call for distributed ownership, articulated by figures like Carlos Guestrin, counters the path of consolidation evident in the massive capital concentration, suggesting that true resilience will depend on decentralized control mechanisms rather than monolithic model ownership.