However, per this helpful drupalcommerce.org post, when using Paypal WPS, the "order first paid in full" event (on IPN) may happen before the user redirects back to Commerce (and , which means that the "Completing the checkout process" is skipped and never fires.

This causes all sorts of pain, because we have generic things we want to happen on completing the checkout process (like creating a user for anon users, etc.)

Thanks to spoonbow for the detailed debugging over on drupalcommerce.org.

Comments

In Commerce_payment.module, we have this code. The code inside the "if" doesn't fire if the order status is 'completed'. So the correct rule never fires. Would it be possible to just change it to say if ($order_status['state'] == 'checkout' || $order_status['state'] == 'completed') ?

This seems rather fragile for the case that a user doesn't return from paypal even though they completed the purchase, too...

function commerce_payment_redirect_pane_next_page($order) { // Load the order status object for the current order. $order_status = commerce_order_status_load($order->status);

Wouldn't it be better to move the call to "commerce_checkout_complete" into the "commerce_order_status_update" so that it will be fired even if the status is changed from the "when the order is first payed in full" rule?

DC cannot expect people to return to the main website after the payment, because this often doesn't happen, so.. once we have the payment we should move on to the next step without waiting for user interaction. (UPDATED: I just tested this again after removing the rule that changed the order status on IPN and it seems other checkout completion rules are now fired)

This way the same events would fire also when an administrator administratively change an order status. Currently I can mark an order as "completed", but all of my rules won't trigger...

It seems to me this is a problem that can be solved through an alternate configuration: namely, you know the IPN is coming in, and we now see that changing the order status on "When an order is first paid in full" will prevent the IPN activated "When completing the checkout process" event from firing - thus opening the door for that event to never fire in the likely event that the customer doesn't return to the site. In this case, the order status update can just be moved from the "When an order is first paid in full" action to the "When completing the checkout process" event, and a condition can be used to check the balance of the order and ensure that payment was actually received.

As to the actual solution, I'm not sure. It could be we just need to change the checkout page before we save the transaction in the PayPal module and advise other modules to do the same. However, that would be contrary to the way on-site payment works with respect to those events, and I think it's better to keep them uniform. The other option is to trigger the "When completing the checkout process" event based on some alternate criteria. The conditions in that function that Randy pointed out are there to ensure this event doesn't get invoked twice on any given order; if we see the order is still in checkout, then we know it needs to be "completed." It could be we just depend on some alternate bit of data tucked away in the order object like we do for ensuring "When an order is first paid in full" only executes once.

However it works out, this isn't really a critical issue, so I've demoted it to normal. Until we solve the core issue, please use the alternate configuration I recommended above.

Unfortunately, the alternate config that you suggest never fires for *anything* if the customer doesn't return to the site to complete the checkout process. So then they've *paid* and done everything we've asked of them, but because they didn't follow funky paypal links back to our site, they're left with a not-completed order. However, your proposed workflow is better than what I'm doing now. The condition would be "Order balance Comparison".

Edit: Ryan's alternate workflow then *only* works when the IPN comes in before "when completing the checkout process". If the race condition has the user completing before the IPN, again, we lose the ability to respond to the fact that the wonderful event has happened.

If the user beats the IPN back to your site, presumably "When completing the checkout process" has already fired before the IPN came. That event gets invoked when the user returns to the site and is forwarded on to the checkout completion page.

To summarize: whether the user returns or not, you will always have the "When completing the checkout process" event. It will either be triggered by the IPN - assuming you haven't changed the order status on "When an order is first paid in full" - or by the user advancing to the completion page. If this is your only payment method and the only way for people to access this event, then there's no harm in updating the order status on that event, assuming the order was paid in full, instead of "When an order is first paid in full."

My actual experience here now that I've changed it as suggested is that *some* orders (presumably the ones where the user does not return to the site) are being left in the Checkout:Payment state after successfully paying and after the IPN coming through. So I suspect that #5 may not be the whole picture.

Hmm, ok. It'd be good to figure out why these orders are not progressing. Perhaps you can setup another administrative e-mail or something that also sends an e-mail to you on the checkout completion event. This will let us know if the event itself isn't being triggered or if something is wrong with the Rule. For a counter-example, on RealMilkCheese.com we use PayPal WPS for payment and have never had an order get stuck in Checkout: Payment whether the customer came back to our site or not.

Ok, great. I got to thinking to that we can probably confirm in the watchdog that the IPN was even received / verified and processed. It could be something actually prevented the processing or introduced an error in that process somewhere such that the loop was never completed. I'm not aware of any problems with the IPN handling code, but it's never something I feel I've "nailed."

It would be great to have that in the watchdog... But in this case I'm getting the payment completely updated by the IPN, so I'd say that probably the IPN does not necessarily cause "When completing the checkout process" to fire.

Hmm, I'm not sure what you're saying - you're not seeing IPN debug logs in the watchdog even with logging turned on but somehow they're still being processed? If you look at the top of the function commerce_paypal_wps_paypal_ipn_process() where PayPal WPS IPNs are actually processed, you can see that it's bailing out for unknown payment statuses. Perhaps your successful IPNs are submitting a status other than "Completed"? Is that something you can investigate in the debug logs?

The failing order (left in Checkout:Payment) had two IPN watchdogs, "IPN Validated" and then "IPN Processed". But the "When completing the checkout process" event did not fire. There were no errors or time-related entries in the watchdog around the time of the processed payment. I have to assume that these people just didn't come back from paypal.

I'm 99.99% sure that's a red herring, because most of my customers don't come back and the code simply doesn't leave that option open (a processed IPN for an order on the Checkout:Payment status without a checkout completion event being fired). If you have a message in the watchdog that the IPN processed, the line right before that message in commerce_paypal_wps_paypal_ipn_process() calls commerce_payment_redirect_pane_next_page(). This is the function that moves an order forward if the order is still in the payment page status, the exact same function that will be called if the customer happens to return to the site before the IPN.

Can you confirm this code in your version of commerce_paypal_wps.module? The lines in question are 236-237. If they're different, feel free to post the function and we can take it from there. Otherwise you might want to put an additional watchdog() or two in those functions to track it down further. If you need a temporary fix, too, you can implement hook_commerce_paypal_ipn_process() in a custom module to update orders that got skipped for whatever reason.

Just a thought as I'm sheepishly trying to figure out how I had code that old. On this site I tried to use all stable releases, but apparently failed. I note that Commerce Paypal doesn't have one though. Couldn't it have an alpha release at least? That way update module would have informed me of the error of my ways.

Might help some people.
I tried to recreate "Completing the checkout process" rule from scratch to add extra functionality and it didn't work.
When I cloned the original rule "Send an order notification e-mail" and modified it, it worked no problems.
Sounds like a bug as two rules were exactly the same.

Nope, you'd just need to change the way you're using checkout completion rules so the order status isn't being updated on "When an order is first paid in full" if the order status is still in checkout. Conversely, make it so the order status is being updated on "When the customer completes checkout" if the order balance is <= 0.

[...] the line right before that message in commerce_paypal_wps_paypal_ipn_process() calls commerce_payment_redirect_pane_next_page(). This is the function that moves an order forward if the order is still in the payment page status, the exact same function that will be called if the customer happens to return to the site before the IPN.

...then making sure the order progresses to order complete (in the case the user doesn't trigger it) is the responsibility of the Payment module? I think this is an important issue, as other Payment modules clearly have a bug in this regard.

Same issue. I have multiple orders that have come on, and IPN did not report the data correctly. Now there are numerous orders which do not have payment posted, without a method to post payment after the fact.

A method to post payment should exist manually for such situations.

Currently am having to look up other methods of posting payments (COD)?

I'm also having the same issues and the emails are not being sent to either the customer or the store owner when adding them into the mix / emails which are sent. Ryan: can you post an example rule which you descried earlier for us please?

Example 1. I have a rule that acts When an order is first paid in full. I change the order status of it.
IPN reaches first than user returns to site, so all rules from Completing the checkout process.
Hence, I don't send no e-mail to user, etc etc.

Example 2. I remove rule that acts When an order is first paid in full. So IPN reaches the website, but nothing happens.
User doesn't return to website with the link it should. My order will stay in checkout: payment status forever.

What's not the worst situation then?
Maybe i'm missing something, but it seems I'm always dependent of a user returning from the paypal with the link that's provided there, right? That's the only way Completing the checkout process will react.

I'll just ask, even if it sounds "dumb". What if I use the hook hook_commerce_payment_order_paid_in_full and inside the hook i'll just invoke the rules with event Completing the checkout process ?

I went inspecting this, and one thing I wasn't truly aware. My dev environment can't receive IPN's, so I didn't know that Completing the checkout process event can be triggered by 2 things. The user returning to the checkout process, or the IPN is received. So the example 2 I referenced previously never happens.

Ok, reaching a better scenario. But, on that event the status order is only changed to "Pending", I didn't have yet a way of changing the status of the order, reflecting that the payment is accepted. I can't use When an order is first paid in full because it is fired first that event before.

A bit more of searching and this thread shed some more light on how everything works, which led into and consequently to

Bottom line, now I think I am more aware of the explanations given on #3 and #21. The trick, for now is really to combine two more rules for Completing the checkout process and When an order is first paid in full.
To refer that I added another condition on the When an order is first paid in full event, to see if order is not on cart, and if not is on checkout also.

So am I right in thinking that the default "out of the box" checkout rules which come with commerce kickstart never work unless the customer clicks on the link after paying with paypal to get back to the site? Also is the solution to this to edit all checkout rules to have the event "when first paid in full" rather than "on completing the checkout process"?

I'm only confused as the defaults don't appear to work and I just need to know what to edit in order for the notification emails to be sent out and anonymous account creations to happen etc...

im still having this issue....in my case the ipn status in paypal is stuck at "retrying" event thought the callback url is correct...pasting the address in the browser hits the ipn succesfully...any ideas??

no its not...although telnetting to port 443 is taking like 10 sec....paypal says their threshold is 3....ALSO
Im seeing in the logs that the ipn address is getting hit, but there is no post data...any ideas?

I'm having the same issues.
Everything works perfectly when using the Example Payment method but when I enable PayPal WPS the orders get stuck in the shopping cart page with status Checkout: Confirm order. Emails are not sent out to the order email and accounts for anonymous users are not created.
I have a fairly vanilla setup with commerce discounts and coupons.

...in fact none of my checkout rules were firing as the event 'completing the checkout process' was never reached.
My only workaround was by adding 'When an order is first paid in full' as the event trigger instead of 'Completing the checkout process' to every rule in the checkout rules set.