A Massively Multi-user Datastore, Synced with Mobile Clients

The system for managing merchant catalog data was re-architected to address challenges associated with large, distributed datasets that require synchronization with mobile clients, support for complex querying, and flexibility for custom extensions. The design adopted an entity-attribute-value data model combined with an append-only structure to ensure transactionality and auditability without relying on underlying database transactions for core operations.
The entity-attribute-value model structures data where objects have unique tokens and types, and attributes have predefined definitions that are namespaced for ownership. A key feature is the handling of location awareness, allowing some clients to see only specific locations while others see all, and allowing object availability to be toggled per location. Synchronizable constraints are modeled as catalog objects to enforce data structure and validation across all clients dynamically.
The append-only model facilitates consistent paging by encoding catalog versions in paging tokens, enabling reliable synchronization across divergent client states. Transactionality is implemented through locking mechanisms that allow long-running operations to execute atomically without blocking other users. This structure enables history tracking, allowing for versioning and rollbacks of the catalog state while maintaining audit trails of all modifications linked to specific callers.

* The system manages large amounts of information for merchants, including products, prices, taxes, and configurations, referred to as a merchant’s catalog.
* Data must be synced with mobile devices, which may operate offline, causing catalog versions to diverge.
* The design required writing data without impacting reads until the operation is complete across multiple API calls.
* The data needed consistent paging even when other clients were writing to the catalog.
* A per-user model was necessary so that one user's activity did not impact others.
* The data structure needed flexibility for rapid feature development and custom merchant data creation without schema changes.
* An entity-attribute-value data model was used, where objects have tokens and types, and attributes require predefined definitions.
* Location awareness is supported by assigning location values to attributes, allowing clients to operate on subsets of data based on their location knowledge.
* Synchronizable constraints are modeled as catalog objects created via API, enforcing structure and validation across clients.
* The core uses an append-only data model, handling modifications by creating deletions at the attribute level.
* A paging token encodes the current catalog version for consistent pagination during updates.
* Transactions can be opened using a lock token to ensure atomic operations; rollbacks are supported by deleting changes made after the lock version.
* Every write is tagged with caller information, providing auditability of specific changes.

The architectural shift towards an entity-attribute-value store combined with an append-only model represents a deliberate tradeoff: sacrificing traditional database transaction guarantees for highly flexible, distributed, and client-aware data management capabilities essential for mobile synchronization. The pattern of modeling constraints as syncable catalog objects, rather than relying on rigid schema migrations, suggests a tension between centralized consistency and decentralized evolution—enabling flexibility by pushing validation logic into the client environment.
The mechanism for history and transaction rollback is particularly interesting. By encoding versioning directly into an append-only stream via paging tokens, the system achieves strong temporal consistency across disconnected states without heavy locking overhead on the underlying storage. This pattern suggests a preference for eventual consistency in distributed systems over strict ACID isolation during data ingestion, allowing operations to be fast and highly available, at the cost of immediate, monolithic transactional guarantees that traditional relational systems enforce strictly.
The concept of location awareness introduces complexity in state management; allowing clients to operate on subsets of data while maintaining the full global context for merchant-aware clients requires a sophisticated layered abstraction over the base store. The design acknowledges that different client needs—some requiring scope limitation and others requiring comprehensive visibility—demand different views of the same underlying append-only stream. This points to a necessary cognitive segmentation where the system facilitates multiple, internally consistent realities based on the consumer's context.
What assumptions are baked into prioritizing this model? Does the flexibility gained through custom attribute definitions and dynamic constraints inherently increase the surface area for inconsistency if client implementations drift? How does the necessity of providing "easy" rollbacks interact with the append-only principle to ensure that an erroneous state is fully and cleanly erased without leaving ghost data or ambiguous states regarding prior versions? What are the real-world costs associated with ensuring synchronization across these potentially divergent, yet constrained, views?