problems with large volume of emails in coldfusion

Hey there,
I currently run coldfusion enterprise 6.1.... I have written an auto contact app for a clients website which parses our database (over 3 million users) and sends them personalized email newsletters. The problem Im running into is that the scheduled task which generates and sends the emial does not seem to send any mail whatsoever... the memory usage on the server gets gigantic and then I get a coldfusion out of memory error.... how do I correct this? I chose coldfusion 6.1 for this solution because of the 1000000+ emails an hour it can supposedly output..... does anyone have a solution to this problem... I can post the code if need be.

SELECT TOP 500 *
FROM yourtable
WHERE userID NOT IN (SELECT userID FROM tblSentMail)
send out the emails with cfquery and use the query attribute on the tag, do not put a loop around the cfmail tag

when done, flag those 500 as having send them an email, you could create a bew table
[mailOutID] [userID]
mailOutID is an id created at the beginning of the process and userID is each user that has been send an email

around all this you can do a loop with a condition of while queryName.recordCount

I do not have a solution, but am interested in this as we are about to write an app that will do the same, althought no where near as big. Probably around 2,000-3,000. Ours is more of an email notice that some new document has been added to the site, (users will register to get "alerted" for specific types of documents). The email will contain links to the documents that the users registered to get notices on. No attachments in the mail, and much smaller email size overall.

I can only think of a few suggestions which you may have already done...

1) Does the schedule task dump any mail into the "undelivr" folder (in the "Mail" folder under your cf root).

2) Does the app work if you run it "manually" as opposed to via a scheduled task.

3) Do the logs show any other errors occuring before the out of memory error

4) Are you running the latest updater for cfmx (a new one was released Aug 25)

dldeeds,
1) no it doesnt go anywhere at all..... it doesnt write anything to the disk....
2) no the app does not run properly run manually. It does the same thing.
3) no there are no other errors
4) I am running version 6.1 as of some time in aug, with updated drivers.

Here is the code exactly as it appears in my cfm page. As I stated before It just sucks up a ton of memory and then ends with an out of memory error.

<!---- Send an email campaign: THIS IS A SCHEDULED TASK! ---->
<!---- first see if the process is already running ---->
<!----<cfif isdefined("application.delivery")>---->
<!---- <cfif application.delivery neq "true">---->
<!---- <cfset application.delivery="true">---->
<!---- first get the next campaign record from the database ---->

Has the mail server been configured properly in CFAdmin ?
If yes - then is the mail server that is configured responding - so as to effectively send emails.
Can u check the mail logs & see whats happening there ?

anandkp,
yes the mail server verifies under the administrator, and works when sending single emails...... the server is configured to run 150 threads for mail delivery, but I have tried scaling this number up and down..... nothing fixed it... the MTA it is relaying to is PMTA by port 25 solutions.
The machine it is running on is a dual p4 xeon at 2.0ghz with 3 gigs of ram, raid 0 for speed when spooling.

How abt the from & to address ?
Check on the domain usuage restirictions with ur ISP [some ISP may block the use of a different domain names to avoid spamming]
how abt calling that file - without using the Scheduler ...

anandkp,
different domains is not a problem in this scenario either..... they are their own ISP #1, and #2 it is doing this in my test environment which is definitely not restricted. I have also run it without the scheduler and it does the same thing.

"When returning large character data queries from a database, the memory print for Macromedia ColdFusion MX is twice as large as that of ColdFusion 5 and can eventually cause the server to hang with java.lang.OutOfMemoryError errors."

Tacobell777,
Ok I have made the changes you asked for.... and good point on the REReplaceNoCase thing... totally an oversight on my part.
I have eliminiated the cfloop, and made it internal to cfmail which poses some problems as far as generating the correct messageid's and subject lines, however, that aside, coldfusion still crashed with an out of memory error. I have since tried it with a smaller segment of the list. It seems to work up to 300,000 names at a time.
a quick question however... during the test of 300,000 names that worked in the end I got an error for an invalid email address being found... How do I make coldfusion ignore the invalid addresses if found.

Tacobell777,
Ok the <cftry><cfcatch> thing worked the way you said.. I completely overlooked the placement of the tags. Very strange results now however. The task runs and says it completed successfully. However the mail server reports only having received 30000 of the 100000 emails that were supposed to be sent. I have verified the queries, and had the cfm page even output what it is working with.... It says 100000 were processed but only 30,000 were sent out. I checked the mail.log. I have this error occurring several times (once for each execution of the page).

mrichmon,
I have not in anyway forgotten about this question. I have been waiting to hear a response to the error I am experiencing. I really hope this is not abandoned as this is a big issue for me right now. I have been waiting with my fingers crossed for the answer.

Hello,
Well I have scrapped rendering large volumes of email with coldfusion. No matter what approach I take to doing it (i have done 8 major rewrites with no change in performance, as well as tried all sorts of high end hardware options) I have written a solution in java, using NIO functions. It performs about 2.4 million render and feeds to the transport agent an hour. Bottom line: I dont recommend doing large volumes of email in coldfusion. It by no means lives up to its claim of 1mm an hour.

I will leave points to those I feel were most helpful.

Cheers,
Rick Simnett

0

Featured Post

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

The purpose of this video is to demonstrate how to create a Printer Friendly PDF on a WordPress
Page.
This will be demonstrated using a Windows 8 PC. Tools Used are Photoshop, Awesome
Screenshot” Google Chrome Extension, and SmallPDF.com
Log…

The purpose of this video is to demonstrate how to set up basic WordPress SEO.
This will be demonstrated using a Windows 8 PC. The plugin used will be WordPress SEO by Yoast.
Go to your WordPress login page. This will look like the following: myw…