Payment & Order Processing

This case describes investigation and a targeted implementation change within the order-module work I contribute to. The wider platform, integrations, and delivery are team work; the pipeline described here was an existing system.

The platform took a customer order through a chain of stages - stock validation, payment authorisation, order creation, order-group splitting, and downstream fulfilment through Dynamics 365. Each stage handed off to the next through Azure Service Bus, so the work proceeded asynchronously once the HTTP request that started the order had returned.

On a normal request the chain ran cleanly: stock was reserved, the payment gateway was called, an order was created in commerce, and the order was split into the line-level groups that fulfilment consumed. The downstream D365 integration picked those groups up and pushed the order forward into fulfilment.

In production, a class of orders was reaching a point where the payment had succeeded and the order had been written - but no further progress was being made. From the customer's point of view, the order looked paid; from the system's point of view, the chain had stopped mid-way and nothing downstream knew about it.

• Payment was authorised and the order was created, but the chain did not move past the order-group step.
• The downstream fulfilment and D365 integration never received anything for those orders.
• Stock had been consumed against an order that the rest of the system had effectively forgotten.
• Engineers had to reconcile the state across services before they could explain what had happened.

Each request moved through the same set of stages. Service Bus carried the message that told the next stage to do its work; the commerce database recorded the order itself; D365 was the system of record for fulfilment.

The investigation worked backwards from the symptom - orders that were paid but not moving - towards the last stage that had clearly completed. Payment gateway logs confirmed authorisation. The commerce database confirmed the order existed. The next stage, the order-group split, was where the trail stopped.

The investigation followed the Service Bus handoff around the order-group step. The expected message was not being enqueued, leaving downstream workers without the input they needed. This narrowed the issue to message publication rather than the downstream fulfilment logic.

The decision was to treat the missing enqueue as a first-class failure mode rather than a flaky message. The fix targeted the order-group step's responsibility to publish its outgoing message, so that a failure to publish was visible, retryable, and observable - rather than being absorbed silently into the next stage's queue.

• Make the order-group step's enqueue of its outgoing message an explicit, instrumented action - not a side effect of completing the work.
• Surface enqueue failures as visible failures of the step, so they are retried with the rest of the work and reported through the same operational channels as any other error.
• Keep the rest of the chain downstream of the order-group step unchanged. The downstream stages were correct; they were simply starved of input.

At the architectural level, the change was scoped to the order-group step and the message contract that surrounded it. The step's work - splitting the paid order into its fulfilment groups - remained the same. What changed was the way the step's outgoing message was produced and how a failure to produce it was treated.

• The order-group step was made responsible for the full lifecycle of its outgoing Service Bus message, including publishing and acknowledgement, rather than relying on a side-effect path that could complete without the message being enqueued.
• Enqueue failures were promoted from silent misses to first-class errors, so a step that could not publish its outgoing message failed in the same way as a step that could not do its work - making the failure visible to retry, alerting, and the on-call rotation.
• The downstream contract - that inventory and D365 integration would only act on a message - was preserved. Because the downstream stages were correct, they were left alone; they were not the source of the failure.

• Coupling the order-group step to its outgoing message publication, so a publish failure is now a step failure - accepted in exchange for visibility and a single, well-defined retry boundary.
• Treating the downstream stages as correct, even though they were the user-visible symptom - accepted because reworking them would have expanded the scope without addressing the root cause.
• Carrying an explicit acknowledgement for the outgoing message in the order-group step - accepted as a small amount of added structure in exchange for the chain being auditable end-to-end.

After the fix, the class of incident where a paid order stopped at the order-group step stopped recurring as a silent failure. When the order-group step could not publish its outgoing message, the failure was visible to the same retry and alerting path as any other step error, and the order was either retried through the existing flow or surfaced for human attention - rather than sitting indefinitely in an inconsistent state.

The downstream chain - inventory commit and the D365 fulfilment push - continued to operate exactly as it had before. The change is best described as a reliability fix to a single step's message contract, with the visible effect that paid orders moved through the chain the way the rest of the system expected them to.

• An asynchronous chain is only as reliable as the message that hands off from one stage to the next. A step that completes its work but does not publish is, from the rest of the system's point of view, indistinguishable from a step that never ran.
• The right place to look for a cascading failure is often the last stage that observably completed, not the first stage that observably failed.
• Treating a publish failure as a first-class error - with the same retry, alerting, and on-call visibility as any other error - is usually a more durable fix than adding compensating logic downstream.
• Production fixes are often narrower than they look. The temptation to redesign the chain around the symptom is strong; the higher-leverage move is usually to find the single missing signal and make it visible.