How Retries, Race Conditions, and Duplicate Jobs Can Cost You Real Money
Your payment job worked perfectly.
The customer was charged. The database was updated. The API returned a successful response.
Then the worker crashed.
Laravel did exactly what you configured it to do: it retried the job.
And your customer got charged again.
Congratulations. Your queue is working as designed. Your payment system isn’t.
This is one of the most dangerous misunderstandings in backend development: successful job execution and correct business execution are two completely different things.
Laravel provides an excellent queue system. Redis, Horizon, retries, backoff strategies, job middleware, and unique jobs make background processing relatively straightforward.
But none of these features automatically guarantees that a financial operation happens exactly once.
And when money is involved, that’s a rather important distinction.
Let’s examine what actually goes wrong, why common Laravel solutions aren’t sufficient, and how I would approach designing a payment-processing backend that can survive repeated execution.
1. The $10,000 Duplicate Payment Problem
Imagine an e-commerce platform processing payments asynchronously.
A customer places an order for $500. Your application dispatches a background job:
ProcessPayment::dispatch($order->id);The worker picks it up:
class ProcessPayment implements ShouldQueue
{
public function handle(PaymentGateway $gateway): void
{
$order = Order::findOrFail($this->orderId);
$gateway->charge(
customerId: $order->customer_id,
amount: $order->amount,
);
$order->update([
'status' => 'paid',
]);
}
}Everything looks reasonable.
Until you introduce a production failure.
Consider this sequence:
- The worker sends a payment request.
- The provider successfully charges $500.
- The provider returns HTTP 200.
- The worker crashes before updating the database.
- Laravel eventually makes the job available for another attempt.
- Another worker processes the job.
- The payment request is sent again.
The customer is charged $1,000.
Your application database still contains one order.
Your accounting department has a problem. Your customer has a bigger one.
The interesting part is that neither Laravel nor the payment provider necessarily did anything wrong.
Two independent systems performed their responsibilities correctly, but your application failed to coordinate their state.
This is a distributed systems problem.
Your database and an external payment provider cannot participate in one ordinary database transaction.
A successful external operation can remain unrecorded locally.
And that’s where things become interesting.
2. Retries Are Not Business-Level Recovery
Retries are essential.
Networks fail. APIs become unavailable. Database connections disappear. Containers restart. Workers get terminated.
Without retries, temporary infrastructure problems would become permanent application failures.
Laravel makes retry configuration straightforward:
class ProcessPayment implements ShouldQueue
{
public int $tries = 5;
public int $timeout = 60;
public function backoff(): array
{
return [10, 30, 60, 120];
}
}But there’s an important distinction:
Retrying a job means executing its code again. It doesn’t mean continuing from the exact instruction where the previous execution stopped.
Laravel doesn’t automatically know whether an external payment provider completed an operation before the connection was interrupted.
Consider a different failure:
Application
|
|---- Charge $500 -------------------->
| |
| Payment Provider
| |
|<---- Payment successful -------------|
|
X Worker crashes
|
|---- Job retried --------------------->
|
|---- Charge $500 again ------------->From the application’s perspective, the outcome of the first request might be unknown.
A timeout doesn’t prove that the payment failed.
A connection reset doesn’t prove that the provider didn’t process the request.
This is why blindly retrying financial operations is dangerous.
The correct response to an ambiguous payment outcome is not necessarily another charge request. Sometimes the application must first determine what happened to the original operation.
3. The Illusion of ShouldBeUnique
A common response to duplicate jobs is:
“Just implement ShouldBeUnique."
Laravel provides this mechanism for a good reason.
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
class ProcessPayment implements ShouldQueue, ShouldBeUnique
{
public function __construct(
public readonly int $orderId,
) {}
public function uniqueId(): string
{
return (string) $this->orderId;
}
}Laravel attempts to prevent multiple instances of the same unique job from being dispatched while the uniqueness lock is held.
That’s useful.
But here’s where developers sometimes make a dangerous assumption.
Job uniqueness is not payment idempotency.
These mechanisms protect different things.
ShouldBeUnique helps control duplicate job dispatching. It does not guarantee that an external payment operation will never be repeated.
Consider these scenarios:
- The payment provider successfully processes a charge, but the worker crashes before completing the job.
- A uniqueness lock expires.
- A separate application endpoint initiates the same payment.
- An administrator manually retries an operation.
- A customer submits the same request twice.
You need protection at the business-operation level.
Laravel also provides WithoutOverlapping middleware, which helps prevent concurrent execution for a given lock key.
But preventing two workers from processing the same resource simultaneously isn’t equivalent to guaranteeing that a payment happens only once.
Use these mechanisms for what they actually solve.
Don’t turn a queue lock into a financial integrity strategy.
4. Idempotency: The Business Operation Must Survive Repetition
Let’s introduce the most important concept in this discussion: idempotency.
An operation is idempotent when repeating the same logical request doesn’t create additional unintended effects.
For payment processing, the same logical payment request must not create another charge simply because the application sent it again.
The first step is to assign every payment attempt a stable identifier.
final readonly class PaymentRequest
{
public function __construct(
public int $orderId,
public int $paymentAttemptId,
public int $amount,
public string $currency,
) {}
public function idempotencyKey(): string
{
return sprintf(
'order:%d:payment:%d',
$this->orderId,
$this->paymentAttemptId,
);
}
}The payment gateway receives this identifier:
$result = $gateway->charge(
amount: $payment->amount,
currency: $payment->currency,
customerId: $payment->customerId,
idempotencyKey: $request->idempotencyKey(),
);Every retry of the same logical operation must reuse the same key.
First attempt:
order:123:payment:1
Second attempt:
order:123:payment:1
Third attempt:
order:123:payment:1The provider can recognize these requests as repetitions of the same operation, provided its idempotency implementation supports that behavior.
A different intentional payment attempt should have a different key.
For example, a customer changing their payment method after a genuinely failed attempt may initiate a new logical operation.
There’s another important detail: idempotency keys need to be persisted, not generated randomly every time a job executes.
Otherwise, your retry mechanism effectively creates a new payment request on every attempt.
Also, external provider behavior matters. Key retention periods, parameter validation, concurrent requests, and error handling differ between providers. You must understand those semantics before relying on them.
Idempotency is not a magical HTTP header.
It’s a business invariant that must be implemented across the entire payment lifecycle.
5. Your Database Must Enforce Business Invariants
External idempotency is important, but your own database needs protection too.
Imagine a payments table:
CREATE TABLE payments (
id BIGSERIAL PRIMARY KEY,
order_id BIGINT NOT NULL,
idempotency_key VARCHAR(255) NOT NULL,
status VARCHAR(30) NOT NULL,
amount DECIMAL(12, 2) NOT NULL,
provider_transaction_id VARCHAR(255),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
CONSTRAINT payments_idempotency_key_unique
UNIQUE (idempotency_key)
);The unique constraint is important.
Don’t rely exclusively on application-level checks:
if (! Payment::where('idempotency_key', $key)->exists()) {
Payment::create([...]);
}Two concurrent workers can both pass the exists() check before either inserts a record.
That’s a classic race condition.
The database must enforce uniqueness atomically.
A safer approach is to use a database constraint and handle the resulting unique-constraint violation appropriately.
try {
$payment = Payment::create([
'order_id' => $order->id,
'idempotency_key' => $key,
'status' => 'pending',
'amount' => $order->amount,
]);
} catch (UniqueConstraintViolationException) {
$payment = Payment::where(
'idempotency_key',
$key
)->firstOrFail();
}The exception class and handling details should match your database driver and Laravel version.
One more important rule:
Never hold a database transaction open while waiting for an external payment API.
You don’t control the provider’s response time.
Long-running transactions can create unnecessary locks, increase contention, and hurt database performance.
Use short transactions to establish state, perform external operations outside the transaction, and persist the result in another short transaction.
Your database should protect business invariants. Your payment provider should protect external payment operations. Your application must coordinate both.
6. The Database-to-Queue Gap: Where Transactional Outbox Comes In
Now we’ve solved one problem and uncovered another.
What happens if the payment record is successfully created in PostgreSQL, but publishing the corresponding message to Redis fails?
Consider this implementation:
DB::transaction(function () use ($order) {
$payment = Payment::create([
'order_id' => $order->id,
'status' => 'pending',
]);
ProcessPayment::dispatch($payment->id);
});Using Laravel’s after_commit behavior is a good practice. It prevents jobs from being processed before their surrounding database transaction commits.
But it doesn’t make the database commit and Redis publication one atomic operation.
There is still a failure window:
1. Database transaction commits.
2. Application attempts to publish a job.
3. Redis is unavailable.
4. No job reaches the queue.
5. Payment remains pending.Your business data exists.
Your background operation doesn’t.
This is a different kind of reliability problem.
And it requires a different architectural solution.
Transactional Outbox
The idea is straightforward:
Instead of directly publishing an event to an external message broker, persist the event in an outbox table inside the same database transaction as the business operation.
For example:
DB::transaction(function () use ($order, $outboxService) {
$payment = Payment::create([
'order_id' => $order->id,
'status' => 'pending',
]);
$outboxService->store(
aggregateType: 'Payment',
aggregateId: (string) $payment->id,
eventType: 'payment.requested',
payload: [
'payment_id' => $payment->id,
'order_id' => $order->id,
],
);
});Now both records are persisted atomically.
Either the payment and its corresponding outbox event are committed, or neither is.
A separate worker processes unpublished messages and sends them to the configured destination.
If Redis, RabbitMQ, or another broker becomes unavailable, the message remains in the database and can be retried later.
This changes the failure model significantly.
You no longer depend on successfully publishing a message at the exact moment your business transaction commits.
However, there is an important limitation.
The publisher itself can successfully send a message and crash before marking it as published.
The same message may therefore be delivered again.
That’s expected.
The Outbox Pattern provides reliable publication through durable persistence and retryable delivery. It doesn’t magically provide exactly-once message delivery.
Consumers must still be idempotent.
Using a Laravel implementation
I built an open-source Laravel package specifically to make this infrastructure easier to integrate into Laravel applications:
The package targets PHP 8.4+ and Laravel 12+.
Instead of manually implementing message persistence, worker coordination, retries, and publisher integration in every application, you can use the package’s existing infrastructure.
Installation:
composer require bugfix666/laravel-outboxPublish its configuration and migrations:
php artisan vendor:publish --tag=outbox-config
php artisan vendor:publish --tag=outbox-migrations
php artisan migrateStore an event within your business transaction:
use Bugfix666\LaravelOutbox\Services\OutboxService;
use Illuminate\Support\Facades\DB;
DB::transaction(function () use ($outboxService) {
$payment = Payment::create([
'amount' => 1000,
'currency' => 'USD',
'status' => 'pending',
]);
$outboxService->store(
aggregateType: 'Payment',
aggregateId: (string) $payment->id,
eventType: 'payment.requested',
payload: [
'payment_id' => $payment->id,
'amount' => $payment->amount,
],
);
});Start the outbox processor:
php artisan outbox:processThe package includes:
- Atomic persistence of business data and outbox messages.
- Background processing with batch support.
- Worker locking to coordinate processing.
- Configurable retry behavior with exponential backoff.
- Custom publisher implementations through a contract.
- Processing and failure events for monitoring.
- Retention and cleanup of processed messages.
- Failed-message retention for inspection and manual intervention.
It also allows you to implement your own publisher for infrastructure such as RabbitMQ, Kafka, or other messaging systems.
The important architectural distinction is that the package handles reliable event publication. It does not replace payment idempotency, provider-side safeguards, or consumer-level deduplication.
Those responsibilities remain part of the application architecture.
I prefer explicit infrastructure over invisible magic. When something goes wrong in a financial system, developers should be able to understand exactly which operation failed, what state was persisted, and what needs to happen next.
That’s the engineering principle behind this implementation.
Repository: https://github.com/bugfix666/laravel-outbox
7. Designing the Complete Payment Pipeline
Let’s combine these concepts into one architecture.
HTTP Request
|
v
Create Payment
|
v
Database Transaction
/ \
v v
Payment Outbox Event
\ /
\ /
v v
PostgreSQL
|
v
Outbox Worker
|
v
Message Publisher
|
v
Redis / Broker
|
v
Laravel Worker
|
v
Payment Service
|
v
Idempotent Gateway
|
v
Payment Provider
|
v
Persist Final State
|
v
ReconciliationEach component has a specific responsibility.
Database
- Enforces business constraints.
- Stores payment state.
- Persists outbox messages atomically.
Outbox Worker
- Publishes persisted events.
- Retries failed publications.
- Coordinates concurrent processing.
Message Broker
- Distributes work to consumers.
- Provides asynchronous delivery.
Laravel Worker
- Executes background operations.
- Handles retries and transient failures.
Payment Service
- Uses stable idempotency keys.
- Handles ambiguous payment outcomes.
- Separates payment attempts from job attempts.
Reconciliation
- Detects discrepancies between local and provider state.
- Investigates payments stuck in pending status.
- Recovers from situations where the outcome of an external operation is unknown.
Notice something?
The queue is just one component in the architecture.
It’s not responsible for guaranteeing financial correctness.
8. What I Would Actually Deploy
If I were building a payment-processing backend today with PHP 8.4, Laravel 12, PostgreSQL, and Redis, this is the checklist I’d start with.
Payment integrity
- Every logical payment operation has a stable idempotency key.
- Database constraints enforce uniqueness.
- Payment attempts are separate from queue attempts.
- External payment requests use provider-supported idempotency.
- Ambiguous outcomes are reconciled instead of blindly retried.
- Provider webhooks are authenticated and processed idempotently.
Queue reliability
- Retry and backoff policies are explicitly configured.
- Worker timeout and queue
retry_afterare configured consistently. - External HTTP requests have connection and response timeouts.
- Failed jobs are monitored.
- Queue workers restart safely during deployment.
- Duplicate execution is treated as a normal failure scenario.
Database and messaging
- Business state transitions are validated.
- Database transactions are short.
- Outbox messages are persisted atomically with business changes.
- Outbox publication is retryable.
- Consumers are idempotent.
- Failed messages can be inspected and recovered.
- Reconciliation detects inconsistencies between local and provider state.
Observability
- Every payment has a correlation ID.
- Provider requests and responses are traceable.
- Duplicate operations generate useful logs and metrics.
- Payment states can be reconciled automatically.
- Alerts exist for stuck pending payments and unusual failure rates.
One more thing: don’t confuse a successful queue job with a successful payment.
Your monitoring should track business outcomes, not just worker health.
A green Horizon dashboard doesn’t mean your payment system is healthy.
It means your workers are probably having a good day.
Your customers’ bank statements are a much more interesting metric.
9. Stop Designing for the Happy Path
Distributed systems fail in unpredictable ways.
Workers crash. Networks disappear. Responses arrive late. Messages are delivered more than once. External services behave unexpectedly.
You cannot eliminate every failure.
But you can design your system so that repeating an operation doesn’t destroy its correctness.
That’s the difference between a backend that works in a development environment and one that can safely process real money in production.
Laravel gives you excellent tools:
- Queues and workers.
- Retry and backoff mechanisms.
- Unique jobs and atomic locks.
- Database transactions.
- Events and listeners.
- Horizon for queue monitoring.
But the architecture around those tools is still your responsibility.
Don’t build a payment system that assumes jobs execute once. Build one that remains correct when they execute twice, three times, or ten times.
Because sooner or later, they will.
And when that happens, you want your customers to notice nothing.
Not a second charge.
Not a missing payment.
Not an apology email from your support team.
Just a successful transaction.
The queue is allowed to retry. Your payment must not happen twice.
A note from the author
I built laravel-outbox to bring explicit, reliable event publishing to modern Laravel applications without requiring every project to reinvent the same infrastructure.
If you’re working with Laravel 12, PHP 8.4+, and distributed application workflows, take a look at the repository. Contributions and technical feedback are welcome.
Комментарии (0)
Пока нет комментариев — будьте первым.