Mini project / POC · end to end
Distributed Payment & Wallet Service: a complete Java 21 ledger mini project.
A double-entry ledger with pessimistic-locked atomic transfers, two-layer idempotency, the transactional outbox pattern, and scheduled balance reconciliation — the exact mechanics interviewers actually probe when they ask "how do you prevent a duplicate charge."
Project overview and requirements
Every payment system interview eventually asks the same three questions: how do you stop a retried request from charging someone twice, how do you guarantee money never silently disappears between two accounts, and how do you know your ledger is actually correct in production. This project builds real, working answers to all three instead of just describing them.
Functional requirements
Non-functional requirements
Technology stack and why each piece is there
| Technology | Role in this project | Why this one |
|---|---|---|
| Java 21 + Spring Boot 3.x | Runtime and framework for all three services. | Same reasoning as the other two mini projects — virtual threads, records, auto-configuration. |
| PostgreSQL | Wallet Service's single source of truth for balances and the ledger. | Row-level locking (SELECT ... FOR UPDATE) and real ACID transactions are non-negotiable for a ledger — this is not a place for eventual consistency. |
| Spring Data JPA | Persistence, including pessimistic lock annotations on wallet reads. | First-class support for @Lock(PESSIMISTIC_WRITE) without hand-written native SQL. |
| Apache Kafka | Downstream events only — notification and reconciliation alerts, never the transfer itself. | The transfer must never depend on Kafka being available; Kafka only carries what happens after a transfer is already durably committed. |
| Spring Security + JWT | Stateless auth with CUSTOMER / MERCHANT / ADMIN roles. | Same pattern as the other two projects — every service verifies its own JWT. |
| OpenFeign | Payment Service → Wallet Service synchronous transfer call. | A payment's success genuinely depends on the transfer's outcome, so this has to be synchronous, not fire-and-forget. |
Jump to a section
High-level architecture
Payment Service is the public-facing orchestrator; Wallet Service is the only service that ever mutates a balance, and it owns every wallet in one database — not one database per customer, not sharded by user. That single decision is what makes a transfer a simple local transaction instead of a distributed one.
Service breakdown
Payment Service — public API, idempotency, and reliable eventing
The only service a client talks to directly. It enforces client-facing idempotency, calls Wallet Service synchronously to execute the actual transfer, and uses the transactional outbox pattern so a downstream event is never lost even if the process crashes right after committing.
POST /payments— CUSTOMER, requires anIdempotency-KeyheaderGET /payments/{id}— CUSTOMER (own), MERCHANT (received), ADMIN (any)- scheduled outbox relay — polls unpublished events and sends them to Kafka
Wallet Service — the double-entry ledger, and the only source of truth for balances
Owns every wallet and every ledger entry in one strongly-consistent database. Its transfer endpoint is idempotent at the database level and performs the debit and credit as one local transaction under row-level locks acquired in a fixed order.
POST /internal/transfers— internal only, called by Payment Service, idempotentGET /wallets/{id},GET /wallets/{id}/ledger— owner or ADMIN
Reconciliation & Notification Service — the independent safety net
Consumes payment events to notify customers and merchants, and separately runs a scheduled job that recomputes every wallet's balance directly from ledger entries and compares it against Wallet Service's cached balance — catching bugs that transactions alone wouldn't.
- consumes
payment.completed/payment.failed— sends notifications - scheduled job — recomputes and compares every wallet's balance
- publishes
reconciliation.discrepancyif a mismatch is found
Database and ledger design
The ledger is append-only and double-entry: every transfer produces exactly two rows — a debit and a credit — that share a transfer_id and always sum to zero. The wallets.balance column is a cached, fast-to-read projection of "sum of this wallet's ledger entries," not the source of truth itself.
The atomic double-entry transfer
This is the piece interviewers care about most: how do you move money between two wallets so that it's impossible for the debit to happen without the credit, impossible for two concurrent transfers on the same wallet to corrupt the balance, and impossible for a retried request to apply twice.
// Wallet.java
@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("select w from Wallet w where w.id = :id")
Wallet findByIdForUpdate(@Param("id") Long id); // WalletRepository
// WalletTransferService.java
@Transactional
public TransferResult transfer(String idempotencyKey, Long payerWalletId, Long payeeWalletId, BigDecimal amount) {
Optional existing = processedTransferRepository.findById(idempotencyKey);
if (existing.isPresent()) {
return existing.get().toResult(); // safe to retry -- same result, no reprocessing
}
Long firstId = Math.min(payerWalletId, payeeWalletId); // fixed lock order prevents deadlock
Long secondId = Math.max(payerWalletId, payeeWalletId);
Wallet first = walletRepository.findByIdForUpdate(firstId);
Wallet second = walletRepository.findByIdForUpdate(secondId);
Wallet payer = payerWalletId.equals(first.getId()) ? first : second;
Wallet payee = payerWalletId.equals(first.getId()) ? second : first;
if (payer.getBalance().compareTo(amount) < 0) {
throw new InsufficientFundsException(payerWalletId);
}
payer.setBalance(payer.getBalance().subtract(amount));
payee.setBalance(payee.getBalance().add(amount));
String transferId = UUID.randomUUID().toString();
ledgerRepository.save(LedgerEntry.debit(payer.getId(), transferId, amount, payer.getBalance()));
ledgerRepository.save(LedgerEntry.credit(payee.getId(), transferId, amount, payee.getBalance()));
processedTransferRepository.save(new ProcessedTransfer(idempotencyKey, transferId, "COMPLETED"));
return new TransferResult(transferId, "COMPLETED");
} amount and balance are BigDecimal, never double or float — binary floating point cannot represent most decimal fractions exactly, and a rounding error in money code is a bug report, not a footnote.The transactional outbox pattern
After Payment Service commits a completed payment, it must publish a payment.completed event — but a crash between "commit the payment" and "publish to Kafka" would otherwise lose the event forever, even though the money already moved. The outbox pattern closes that gap by writing the event to the same database, in the same transaction, as the payment itself.
// PaymentService.java
@Transactional
public Payment createPayment(String idempotencyKey, PaymentRequest request) {
Optional existing = paymentRepository.findByIdempotencyKey(idempotencyKey);
if (existing.isPresent()) {
return existing.get(); // client retried -- return the original result
}
Payment payment = paymentRepository.save(Payment.pending(idempotencyKey, request));
TransferResult result = walletClient.transfer(
payment.getInternalIdempotencyKey(), request.payerWalletId(), request.payeeWalletId(), request.amount());
payment.markCompleted(result.transferId());
paymentRepository.save(payment);
// same local transaction, same commit as the payment row above
outboxRepository.save(OutboxEvent.of("PaymentCompleted", payment.getId(), toJson(payment)));
return payment;
}
// OutboxRelay.java -- separate process/thread, decoupled from the request path
@Scheduled(fixedDelay = 2000)
public void relayPendingEvents() {
List pending = outboxRepository.findTop50ByPublishedFalseOrderByCreatedAtAsc();
for (OutboxEvent event : pending) {
kafkaTemplate.send("payment." + event.getEventType(), event.getAggregateId().toString(), event.getPayload());
event.markPublished();
outboxRepository.save(event);
}
} payment.id, which is a far easier property to guarantee than exactly-once delivery across a network.Security and role-based access control
Same JWT/RBAC pattern as the other two mini projects: every service independently verifies the signed token. The role set here is deliberately narrow — nobody, including ADMIN, gets an endpoint that directly edits a balance.
| Role | Can do |
|---|---|
ROLE_CUSTOMER | Initiate payments from own wallet, view own wallet and ledger history. |
ROLE_MERCHANT | View received payments and own wallet, cannot initiate transfers from other wallets. |
ROLE_ADMIN | View any wallet and ledger, view reconciliation reports — never a direct balance-edit endpoint. |
PATCH /wallets/{id}/balance endpoint at any privilege level. Every balance change must go through the transfer path so it always produces a matching ledger entry — an admin "quick fix" endpoint would be the one way to make a balance and its ledger silently disagree.Service communication: the payment sequence, including a client retry
Idempotency-Key) stops the client's retry from calling Wallet Service a second time at all. Layer two (Wallet Service, keyed on an internal idempotency key Payment Service generates per attempt) stops double-processing even if Payment Service itself retries the internal call after a timeout without knowing whether the first attempt succeeded.Balance reconciliation: the independent safety net
Transactions and locking prevent corruption from concurrent access, but they don't protect against a logic bug, a bad migration, or a manual data fix that silently breaks the invariant that a wallet's balance always equals the sum of its ledger entries. Reconciliation is a second, independent check of that invariant — the same principle as double-entry bookkeeping in traditional accounting.
@Scheduled(cron = "0 0 * * * *") // hourly
public void reconcileAllWallets() {
for (WalletSummary w : walletClient.getAllWalletBalances()) {
BigDecimal derived = ledgerRepository.sumEntriesForWallet(w.walletId());
if (derived.compareTo(w.cachedBalance()) != 0) {
reconciliationRepository.save(ReconciliationRun.discrepancy(w.walletId(), w.cachedBalance(), derived));
kafkaTemplate.send("reconciliation.discrepancy",
new DiscrepancyEvent(w.walletId(), w.cachedBalance(), derived));
}
}
}Project folder structure
payment-wallet-poc/ ├── api-gateway/ │ ├── src/main/java/com/jiquest/gateway/GatewayApplication.java │ ├── src/main/resources/application.yml │ ├── Dockerfile │ └── pom.xml │ ├── payment-service/ │ ├── src/main/java/com/jiquest/payment/ │ │ ├── PaymentServiceApplication.java │ │ ├── config/SecurityConfig.java, KafkaProducerConfig.java │ │ ├── client/WalletClient.java // OpenFeign │ │ ├── controller/PaymentController.java │ │ ├── service/PaymentService.java, OutboxRelay.java │ │ ├── repository/PaymentRepository.java, OutboxRepository.java │ │ └── entity/Payment.java, OutboxEvent.java │ ├── src/main/resources/application.yml │ ├── Dockerfile │ └── pom.xml │ ├── wallet-service/ │ ├── src/main/java/com/jiquest/wallet/ │ │ ├── WalletServiceApplication.java │ │ ├── config/SecurityConfig.java │ │ ├── controller/WalletController.java, TransferController.java │ │ ├── service/WalletTransferService.java │ │ ├── repository/WalletRepository.java, LedgerEntryRepository.java, ProcessedTransferRepository.java │ │ ├── entity/Wallet.java, LedgerEntry.java, ProcessedTransfer.java │ │ └── exception/InsufficientFundsException.java │ ├── src/main/resources/application.yml │ ├── src/test/java/com/jiquest/wallet/ConcurrentTransferIT.java │ ├── Dockerfile │ └── pom.xml │ ├── reconciliation-service/ │ ├── src/main/java/com/jiquest/reconciliation/ │ │ ├── ReconciliationServiceApplication.java │ │ ├── config/KafkaConsumerConfig.java │ │ ├── client/WalletClient.java │ │ ├── listener/PaymentEventListener.java │ │ ├── service/ReconciliationJob.java, NotificationService.java │ │ ├── repository/ReconciliationRunRepository.java, NotificationRepository.java │ │ └── entity/ReconciliationRun.java, Notification.java │ ├── src/main/resources/application.yml │ ├── Dockerfile │ └── pom.xml │ ├── docker-compose.yml ├── .env.example └── README.md
Local development setup
# docker-compose.yml (excerpt)
services:
postgres-payment:
image: postgres:16
environment: { POSTGRES_DB: payment_db, POSTGRES_PASSWORD: postgres }
ports: ["5433:5432"]
postgres-wallet:
image: postgres:16
environment: { POSTGRES_DB: wallet_db, POSTGRES_PASSWORD: postgres }
ports: ["5434:5432"]
postgres-reconciliation:
image: postgres:16
environment: { POSTGRES_DB: reconciliation_db, POSTGRES_PASSWORD: postgres }
ports: ["5435:5432"]
kafka:
image: apache/kafka:3.7.0
ports: ["9092:9092"]
wallet-service:
build: ./wallet-service
depends_on: [postgres-wallet]
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres-wallet:5432/wallet_db
ports: ["8082:8080"]
# payment-service and reconciliation-service follow the same shape as the other mini projectsTesting strategy: proving the concurrency guarantees
Unit tests aren't enough here — the entire point of the locking design is to behave correctly under concurrent access, so the most important test in this project fires many transfers at the same wallet pair simultaneously and asserts the final balance is exactly right.
@Testcontainers
@SpringBootTest
class ConcurrentTransferIT {
@Container
static PostgreSQLContainer postgres = new PostgreSQLContainer<>("postgres:16");
@Test
void hundredConcurrentTransfersNeverCorruptTheBalance() throws InterruptedException {
Wallet payer = createWallet(BigDecimal.valueOf(1000));
Wallet payee = createWallet(BigDecimal.ZERO);
ExecutorService pool = Executors.newFixedThreadPool(20);
CountDownLatch latch = new CountDownLatch(100);
for (int i = 0; i < 100; i++) {
String key = "test-key-" + i; // each a DISTINCT idempotency key -- 100 real transfers
pool.submit(() -> {
transferService.transfer(key, payer.getId(), payee.getId(), BigDecimal.TEN);
latch.countDown();
});
}
latch.await(30, TimeUnit.SECONDS);
assertThat(walletRepository.findById(payer.getId()).getBalance()).isEqualByComparingTo("0");
assertThat(walletRepository.findById(payee.getId()).getBalance()).isEqualByComparingTo("1000");
}
@Test
void retryingTheSameIdempotencyKeyNeverDoubleCharges() {
Wallet payer = createWallet(BigDecimal.valueOf(100));
Wallet payee = createWallet(BigDecimal.ZERO);
transferService.transfer("same-key", payer.getId(), payee.getId(), BigDecimal.TEN);
transferService.transfer("same-key", payer.getId(), payee.getId(), BigDecimal.TEN); // retry
assertThat(walletRepository.findById(payer.getId()).getBalance()).isEqualByComparingTo("90"); // not 80
}
}Observability
processed_transfers or payments lookups find an existing key — a proxy for client retry behavior and network reliability.Path to production on AWS
| Local (this POC) | AWS equivalent |
|---|---|
| 3 services in Docker Compose | ECS Fargate or EKS, one service per task definition |
| 3 Postgres containers | Amazon RDS PostgreSQL, Multi-AZ for wallet_db specifically given its criticality |
| Single-broker Kafka container | Amazon MSK |
Scheduled @Scheduled jobs | Consider moving the outbox relay and reconciliation job to independent, separately-scalable workers rather than in-process schedulers once volume grows |
Key design decisions and interview talking points
Why does a single transfer only ever touch one database instead of two?
Wallet Service owns every wallet in one database rather than sharding by customer, so both sides of any transfer are always local rows in the same database. That service-boundary decision is what avoids needing a distributed transaction (two-phase commit or a SAGA) for what is fundamentally a single, simple operation.
Why row-level SELECT ... FOR UPDATE with ordered locking instead of just using SERIALIZABLE isolation for the whole transaction?
SERIALIZABLE would also work, but it detects conflicts by aborting and retrying transactions, which degrades badly under high contention on a popular wallet. Explicit ordered row locks pay a small, predictable wait cost up front instead of unpredictable abort-and-retry storms, and the lock ordering rule is simple enough to reason about directly.
How exactly does the idempotency key prevent a double charge on client retry?
Before doing any work, Payment Service checks whether a payment with that Idempotency-Key already exists; if it does, it returns the stored result immediately without calling Wallet Service again. The check-and-insert is protected by a unique database constraint on the key, so even two near-simultaneous retries can't both slip through a race in the application code.
Why two separate idempotency keys instead of just passing the client's key all the way through to Wallet Service?
The client's key protects against the client retrying; Payment Service's own internally generated key protects against Payment Service itself retrying the call to Wallet Service after an ambiguous timeout, without needing to know or trust anything about the client's key format or uniqueness guarantees. Two layers, two independent failure modes covered.
What actually breaks if the transactional outbox pattern is removed and Payment Service just calls Kafka directly after committing?
A crash in the narrow window after the database commit but before the Kafka send would lose the event permanently — the payment is real and money moved, but nothing downstream (notification, reconciliation) ever finds out. The outbox makes that window disappear by writing the event in the same transaction as the payment itself, so "payment committed" and "event will eventually be published" become the same guarantee.
The outbox relay can publish an event and then crash before marking it published — doesn't that cause duplicates?
Yes, and that's an accepted trade-off: the system gets at-least-once delivery, not exactly-once, which is a much easier guarantee to build correctly. The fix is making every consumer of these events idempotent on the payment id, the same discipline applied everywhere else in this project.
Why keep a cached balance column on wallets at all if the ledger is the real source of truth?
Recomputing a balance by summing every historical ledger entry on every read would get slower as a wallet accumulates transaction history, so the cached column exists purely for fast reads. The ledger stays authoritative precisely because reconciliation periodically checks the cache against it — the cache is a performance optimization with a verification safety net, not a second source of truth.
What does the reconciliation job protect against that the transaction and locking design doesn't?
Locking and transactions protect against concurrent-access bugs during a transfer; they do nothing to catch an unrelated bug, like a migration script that directly updates a balance, or a future code change that writes a ledger entry without updating the cached balance to match. Reconciliation is an independent, continuously-running proof that the invariant still holds regardless of how it might have been violated.
Why BigDecimal instead of double or float for money amounts?
Binary floating point cannot exactly represent most decimal fractions (0.1 has no exact binary representation), so repeated arithmetic on money stored as double silently accumulates rounding error. BigDecimal represents decimal values exactly and lets you control rounding explicitly, which is why virtually every production financial system uses a decimal type, never floating point, for currency.
How would you handle a currency mismatch, like trying to transfer USD into a EUR wallet?
Reject it at the validation layer before any locking happens — a transfer request's currency must match both wallets' currency exactly in this design. Supporting cross-currency transfers would require an explicit conversion step with its own rate source and rounding rules, which is a meaningfully different (and riskier) feature, not a small extension.
Isn't a version column (optimistic locking) redundant with the pessimistic SELECT ... FOR UPDATE locks already in place?
Within the transfer path itself, yes — the pessimistic lock alone is sufficient there. The version column matters for any other code path that might read and later write a wallet outside the transfer service (an admin tool, a batch job), giving those paths a cheap way to detect a stale read without needing to take a pessimistic lock themselves.
How would you extend this design to support multi-currency wallets per customer?
Model it as one customer owning multiple single-currency wallet rows (a USD wallet and a EUR wallet, each with its own id) rather than one wallet with a currency field that changes meaning — transfers stay single-currency and simple, and any cross-currency conversion becomes an explicit, separately-designed operation between two of that customer's own wallets.
What happens to a payment request if Wallet Service is briefly unavailable?
The OpenFeign call fails or times out, Payment Service's local transaction rolls back (no payment row is committed), and the client sees an error and can safely retry with the same Idempotency-Key — because nothing was ever committed, that retry is a fresh attempt, not a double-processing risk.
Could a malicious client replay an old, legitimate Idempotency-Key days later to trigger a duplicate payment?
No — replaying the same key against the same payment just returns the original cached result again, by design; it can never trigger a second transfer. The actual risk to guard against is a leaked API credential being used to submit new payments with fresh keys, which is an authentication and rate-limiting concern, not an idempotency one.
Why can't even ADMIN directly edit a wallet's balance through the API?
Any balance change that doesn't go through the transfer path produces a balance with no corresponding ledger entry, which is exactly the kind of drift the reconciliation job exists to catch and exactly the kind of unauditable change a financial system should never allow, regardless of who's making it. A correction, if ever needed, should itself be modeled as a transfer (from a system "adjustments" wallet) so it leaves the same audit trail as everything else.
How would you scale Wallet Service if it becomes a write bottleneck, given it's deliberately one strongly-consistent database?
Vertical scaling and read replicas for the read-heavy endpoints buy real headroom first. Beyond that, the next real lever is partitioning wallets across multiple database shards by wallet id range — which reintroduces the cross-shard-transfer problem this design currently avoids, so it should only be taken once single-database scaling is genuinely exhausted, not as a default.
If an interviewer asks for the single most important design decision in this project, what do you say?
Keeping every wallet in one strongly-consistent database so a transfer is always a single local ACID transaction — it's the decision that makes the rest of the system simple, because it means idempotency, locking, and the outbox pattern only ever have to solve well-understood, single-database problems instead of the much harder distributed-consensus problem a naive multi-database design would have created.
Related guides
Update these hrefs to your published Blogger post URLs once each page is live.
Post a Comment
Add