Dev4Press No Script
Back to GD Mail Queue (Lite) Forum

Duplicated Emails and Slow Queueing Time

Published on: February 7, 2024 at 6:39 pm · By: pharocattle
Author
Topic
#77183

I’m brand new to this plugin. It was recommended in the WP Foro forums because we were having issues with errors when forum posts were made due to a high number of email notifications being sent out. It’s at least partially resolved the issue we were facing; however, it still takes around 30 seconds for a forum post/reply to be made with ~600 subscribers being notified. With no subscriber notifications, the same post/reply takes less than 1 second.

First question…is there a way to further speed up queueing the email to reduce load time when making a post/reply.

Second, and most important question…we’re having some users get duplicated emails. It’s not consistent across our user base, so it seems somehow related to mail queueing or possibly re-queueing of emails. I tried lowering the number of emails in a batch, but I’m not sure what else could be done to troubleshoot and/or resolve this. Any advice?

Note…we’re on v6.4.3 of WordPress.

Viewing 13 replies - 1 through 13 (of 13 total)
Author
Replies
  • #77184

    Hello,

    If your post has 600 subscribers, plugin needs to create 600 email records and store it in the database. That should be very quick under normal conditions. But, if your database (or hosting) server are slow, that can be long process. You say that it takes 30 seconds for 600 subscribers, that is only 20 subscribers a second, and that is very slow. Duplicated emails can be also caused by the slow database server, where the update queries are slow, and not properly synced, so some emails don’t get fully processed.

    What kind of server do you use (detailed specs if you know)? The problems you described are easily explained by the underpowered server.

    I plan to have major updates to the plugin processing of large email batches, but this is still not close to be ready, hopefully later this year. But, in general, I have not seen many complaints about plugin speed that is not caused by the server.

    Regards,
    Milan

    Dev4Press - Premium plugins for WordPress.

    1 user thanked author for this post.
  • #77185

    Milan, thanks for your reply. We’re now hosted on WP Engine, which is hosted on GCP. We were formerly on GoDaddy’s managed WP hosting, and we were having a lot of issues. So far, everything seems very fast on WP Engine except for this one issue.

    1 user thanked author for this post.
  • #77186

    Yeah, WPEngine is usually set correctly, and hosted on GCP. My websites are on Siteground, also on GCP architecture. I recently did some tests on my server, it was able to insert 6000 rows into database table in about 5 seconds. WPEngine shouldn’t have trouble with 600.

    If you can get the slow queries log from WPEngine, we can maybe see if the SQL is really the issue or something else. Issues related to server are hard to debug without special setup to do it. And, testing on staging is not really an option with WPEngine. No matter how good WPEngine is, their staging setups are very slow (at least 5 times slower than live).

    Regards,
    Milan

    Dev4Press - Premium plugins for WordPress.

  • #77187

    That’s interesting! If you could give me an example query, I could pause the mail queue and run a query in PHPMyAdmin to see how long it takes in production. Do you think that would help?

  • #77188

    I can’t give you example of the query, query is not generated by my plugin, this is insert method generated by the WordPress DB object. That’s why slow queries log is helpful, because it lists the queries that MySQL server flags as slow.

    Dev4Press - Premium plugins for WordPress.

  • #77287

    Hi Milan,

    I finally had a chance to chat with WP Engine today. Here was there response about the slow queries log:

    I’m actually not seeing any slow or killed queries on the server for pharocattle

    server’s mysql and PHP usage are also pretty stellar

    They did note that we had our PHP memory set to ~50MB, so in case that could affect this, we upped to 256MB.

    Their other observation was that your plugin’s not listed as compatible with our version of WordPress. Do you know if that could cause any issues, and do you know when it will be updated again?

  • #77288

    The latest plugin version is compatible with the latest WordPress. I may have not updated the compatibility tag only. But, 50 MB of memory is very low for WordPress, so that may be responsible for issues. Usually, WPEngine has 128MB or 256MB set for the PHP (for websites I worked on WPEngine). The fact that they have not found any slow queries points to no SQL issues at least, but the memory limit can cause problems – if the process runs out of memory, it will fail without completing all it needs, so some emails will not be marked as sent, even if they are sent. Test it with higher memory limit and le me know if that helped.

    Dev4Press - Premium plugins for WordPress.

  • #77313

    Hi Milan,

    I just tested again with the higher memory limit, and unfortunately there’s no noticeable difference.

  • #77314

    I am really sorry, but there is not much I can do. I can fix the problems that I can reproduce, but if you don’t have errors in the log, or the slow queries, there is nothing I can test for.

    Just one more question: what are you use for sending, is it a custom SMTP set? Do you have some other plugin that deals with delivery, which is not advised?

    Dev4Press - Premium plugins for WordPress.

  • #77315

    Hi Milan,

    We’re only using GD Mail Queue for sending email via SMTP.com via the PHPMailer right now. I just purchased the Pro version of the plugin, and I’ve been testing in our staging environment with using it to send via the AWS SES API. As soon as I get approved with SES for production sending, I plan to move that over to production.

    I feel like we’re a little stuck right now because in the WP Foro forums, they say the solution is to use GD Mail Queue, which has definitely helped, but it hasn’t fully resolved the issue. I hope that I can take this conversation back to WP Foro to see if there’s anything more that they can do from their end.

  • #77415

    Hi Milan,

    I wanted to come back around to update you since the last that I posted. I ended up paying for WP Foro’s premium support, and they basically told me they couldn’t help me. After a lot more testing and more research, I came across the Offload SES plugin. Since everything else has struck out so far, I figured I’d give it a shot since I have been approved for production SES access. This plugin helped tremendously!! Based upon this, I think the delay does have something to do with the GD Mail Queue plugin not working as expected. I’m adding the results of my testing with GD Mail Queue and Offload SES below so you can see the difference. Keep in mind that these are all on WP Engine in our staging environment.

    GD Mail Queue

      1000 Emails – 1.2 minutes – Resulted in 502 Error
      900 Emails – 1.2 minutes – Okay
      800 Emails – 1.0 minute – Okay
      700 Emails – 52.21 seconds – Okay
      600 Emails – 44.86 seconds – Okay
      500 Emails – 37.56 seconds – Okay
      400 Emails – 48.72 seconds – Okay
      300 Emails – 32.40 seconds – Okay
      200 Emails – 15.49 seconds – Okay
      100 Emails – 8.67 seconds – Okay

    Offload SES

      1000 Emails – 22.12 seconds – Okay

    If I can help any with getting to the bottom of what’s going on with how long it’s taking to get the emails queued, just let me know.

  • #77416

    My plugin works quite differently from SES plugin. My plugin takes emails and puts it into internal queue, with SES plugin they are sending emails as is through the SES infrastructure, and all the processing is done on SES side.

    But, the most important thing here is this: wpForo sends notifications with individual emails => for 1000 subscribers it does send 1000 emails. On the other side, bbPress forums, for 1000 subscribers will send 1 email. Now, here is the difference with my plugin and SES offload. My plugin takes each plugin and adds to queue (in case of bbPress noticiations – it takes one email and splits it into 1000 individual emails and adds them to queue), with SES it just routes 1000 emails to the Amazon infrastructure. My plugin can control sending speed and other things, add HTML template to the email, it can work with a lot of different sending methods.

    Finally, I got to test the wpForo sending on its own, and I made demo website with 1000 users subscribed to a single topic, and when 1000 notifications needed to be generated, this process started, and sent around 400 emails, and then the server timed out. wpForo sending is fundamentally broken, because they don’t impose any limit on sending, and if your sending reaches server timeout, that’s it, it will break and you have no idea what is send at all. When I switch sending to SMTP with authentication (still not using my plugin), I managed to get to 120 emails sent, until server timeout (I used 30 seconds timeout, which is standard for most PHP installations). I used my plugin, and it managed to capture around 700 emails in 30 seconds, because it is adding them to DB, not sending. With faster server, I think it can get to 1000 emails in 10-15 seconds, and why SES offload manages to pass them all to the Amazon SES. But, I am pretty sure that if you use your server to send emails directly it will not manage to send 1000 emails with or without my plugin or SES offload. Any email sending method takes time to send and verify the email has been sent, and I think that’s why you tried my plugin in the first place to see if it can handle the large number of emails all sent individually. In most cases, your examples will not work in any variant it you don’t increase server timeout.

    wpForo has to have two major changes to make email handing better: send emails in the background (if the new topic is posted, don’t send email in the same process that is publishing the topic – but instead schedule background job to send emails without locking main process), and use server timeout to send emails in batches. They never tested this process with large number of subscribers, and they never experienced the issue on their own. You found the issue, and they can’t solve it with the current system. What if you want to send these emails without paying expansive third party email processor? wpForo email sending is bad, and they need to implementing properly to work by using background processes to send emails and avoid main process to freeze.

    I plan to make changes to GD Mail Queue and improve the queue speed, but, if the source of emails is badly optimized, I can’t do much to offset that. Again, SES offload works only because of the huge Amazon infastructure, and the fact that passing emails through API is much faster than using wp_mail and SMTP. Even with Amazon, at some point you will run into same issue. And, for my money, having 20+ seconds process freeze when new topic is published is bad for the end user, which again, goes back to bad wpForo implementation.

    Regards,
    Milan

    Dev4Press - Premium plugins for WordPress.

  • #77421

    Hi Milan,

    Thanks for looking at this and your response! Agreed that WPForo needs to do some work on sending emails. Sending them in the same process (like you mention) as posting on the front-end just doesn’t make sense for any sizable forum. I hope they fix this in a future version. It sounds like this gave you some information for improving your plugin as well, so that’s great!

    Let me know if I can help further.

Viewing 13 replies - 1 through 13 (of 13 total)
  • You must be logged in to reply to this topic.
0
0
0
0
0
0
0
0