Transaction integrity when the reply is lost.
A reference decision about retries, operation identity, and the boundary that commits a transfer.
- Decision
- Retain, repair, or replace the transaction path
- System
- Reference transfer service
- Audience
- Engineering sponsor, system owner, operations
This is an illustrative engineering assessment of a reference scenario, not client work. The executable demonstration uses request-isolated, in-memory state. Production checks remain explicitly proposed.
Retain the service shape. Repair the transaction boundary.
For this scenario, a caller may lose the reply after a transfer has been applied. Before expanding retry behavior, make the original operation and its outcome recoverable under one durable identity.
The recommended next phase preserves the service interface and establishes atomic ownership of the operation record, ledger mutation, and receipt. A replacement engine would add a larger migration before there is evidence that storage or workload limits require one.
What would change the recommendation
Retain the path unchanged if the existing system passes the proposed integrity and recovery tests. Reopen replacement if its persistence model cannot make the required writes atomic, or a measured workload demonstrates a limit that a bounded repair cannot address.
One transfer from request to recoverable result.
The modeled production path accepts a transfer command, authorizes its accounts, applies the debit and credit, and returns a receipt. The caller can retry when it cannot determine the outcome. A shared ledger and overlapping application versions are assumptions for the proposed design.
Included
- Command identity and content binding
- Balance conservation and receipt replay
- Ambiguous acknowledgement and retry behavior
- Persistence, compatibility, and recovery acceptance
Excluded
- External payment settlement and provider reconciliation
- Foreign exchange, fees, and financial reporting
- Fraud policy and regulatory assessment
- A production capacity claim or release authorization
The executable Rust reference narrows that scope to two fixed accounts, one request, and sequential calls. Each run starts with 1,000 units in the operating account and zero in reserve. It has no persistence layer or tenant model.
The contract that each option must preserve.
- I-01
One operation has one meaning.
Bind the operation identity to the account pair, amount, currency, and authorized scope. Reusing it with different content must return a conflict without changing balances.
- I-02
A transfer preserves the total.
The debit and credit either both commit or neither commits. This scenario excludes fees, exchange, and issuance; its total balance must remain unchanged.
- I-03
A lost reply does not authorize another transfer.
A retry with the same identity and content must recover the original result. The caller's uncertainty cannot create a second ledger mutation.
- I-04
The receipt outlives the process.
A production design must commit the operation record, balance mutation, and recoverable result in one durable transaction. A process restart cannot erase deduplication history.
- I-05
Scope and version remain part of the contract.
Authorize access to both accounts at the ledger boundary. Old and new writers must agree on identity and content comparison throughout a deployment.
Separate the reference from the system still to be tested.
The observed evidence below comes from local tests of the Rust reference. Its scenarios cover command replay, content conflict, and a simulated lost acknowledgement. The proposed tests address the persistence and operating conditions that demonstration does not represent.
Identical command repeated
Local Rust tests apply a 250-unit transfer and repeat the same command. They verify the original receipt is replayed, with balances of 750 and 250 and one transfer recorded.
One request, sequential calls, fixed accounts, in-memory state.
Identity reused with changed content
Local Rust tests apply 250 units, then attempt 400 units under the same identity. They verify a conflict, unchanged balances, and one transfer recorded.
No claim about database isolation or concurrent writers.
Reply lost after application
Local Rust tests withhold the first acknowledgement, then repeat the command. They verify recovery of the original receipt with one transfer recorded.
Acknowledgement loss is simulated; this is not a real network failure.
Crash between balance and receipt writes
Interrupt the database transaction at each write boundary, restart the service, and retry the command. Check the ledger and retained operation record together.
Needs the actual database, isolation level, and recovery configuration.
Concurrent duplicates and unauthorized access
Race identical and conflicting commands. Attempt the same identity from another tenant and with revoked account access. Verify mutation and result visibility separately.
The reference has no tenant model and does not test distributed concurrency.
Capacity, latency, and recovery time
Establish a workload and service objective with the owner, then measure contention, tail latency, and restoration from backup.
No production workload, service target, or benchmark result is assumed.
Three paths, with different change surfaces.
The recommendation is conditional on the assumptions above. The next phase must resolve them before the sponsor commits to an implementation program.
Fund the work in reviewable stages.
Establish the contract
System owner + Atom technical lead
Document command identity, authorization scope, content comparison, result retention, and the caller's retry behavior. Preserve representative incident evidence.
Acceptance
Owners agree on the invariants and exclusions. A deterministic test reproduces each material failure the investigation identifies.
Expand and verify
Implementation team + independent verifier
Introduce the compatible operation record and atomic write path. Test concurrency, rollback, process termination, and recovery against the real database.
Acceptance
Failure tests preserve balances and return one stable result. Old and new versions coexist without disagreeing on operation identity.
Activate with a recovery plan
Release owner + operations
Activate for a bounded workload, compare receipts with ledger entries, and exercise the agreed recovery procedure. Keep the previous compatible code deployable.
Acceptance
Reconciliation has no unexplained differences; agreed latency and error limits hold. Operators can detect and resolve an ambiguous outcome.
Conclude the transition
System owner
Retire the old write path only after all writers have migrated and reconciliation is complete. Retain operation records for the agreed replay horizon.
Acceptance
No active caller depends on the old path. The owner accepts the evidence record, runbook, retention policy, and remaining risks.
Recovery is part of the change.
A code rollback does not undo committed transfers. Preserve durable operation records and reconcile uncertain outcomes before another attempt. If incompatible data has been written, use the agreed forward recovery procedure; redeploying old code alone is insufficient.
Estimate from access and boundaries.
A scoped proposal would name the system owner, technical lead, and independent verifier. It would assume access to the write path, relevant schema and migration history, representative traces, and a test environment that can safely exercise failure.
- What determines effort
- The number of writers, persistence model, version overlap, history requiring reconciliation, and the time needed to obtain representative evidence.
- What can expand the scope
- Multiple ledgers, external side effects, undocumented callers, or missing operation history. Resolve these before extending the engagement.
- What remains unestimated
- Implementation duration, staffing, and production performance. This reference has no customer access or workload evidence from which to derive them.
- Handover
- A decision brief, architecture boundary, evidence record with reproduction steps, options, acceptance tests, and an execution plan owned by the customer.
The customer can use these artifacts with its own team or another provider. Production activation requires a separate decision supported by the agreed acceptance evidence.