Hard-coded 1 minute timeout for syncing to IMAP sent folder

1

Hi

i have inordinate problems any time I send a larger file to someone. The file is sent by SMTP no problem, but when trying to sync the mail to the sent folder, it times out, and it’s causing real headaches.

        ---> System.TimeoutException: Server did not respond in 00:01:00

The log seems to indicate there’s a hard-coded 1 minute timeout. I shouldn’t have to tell people that 1 minute is short, especially if the file is big. Why even have a timeout for this? f you’re going to have a time out it should either be editable, or smarter (e.g. scale based on message size).

At the moment each time I make a mistake of sending an audio file attachment, I have to go through a lot of steps to prevent emCLient trying to sync to the sent items, and manually clean up files in the server and restart it.

2

i have inordinate problems any time I send a larger file to someone. The file is sent by SMTP no problem, but when trying to sync the mail to the sent folder, it times out, and it’s causing real headaches.

That could be caused “the size of the file you are sending”, so takes time for your SMTP server to acknowledge it’s sent & then update the client.

Or if your internet connection “is not constant enough”, so is getting delays updating the sent folder in eM Client from your SMTP server.

Or could be due to “some local program running” on the computer delaying the SMTP server updating the sent folder.

How large is the file you are sending ?

Also how fast is your internet connection sending u/l speed ? You can test the upload speed via eg: https://www.speedtest.net

Also is your internet connection allways constant, or do you get any internet connection speed delay / lag at anytime when uploading ?

Lastly do you run any optionally installed software like eg: Firewall / Security programs or optionally installed Antivirus programs or VPNs ?

3

Hi

It’s nothing to do with SMTP. The mail is accepted for delivery, and forwarded to the end user no problem.

The problem is in IMAP syncing the mail to the sent folder. The SMTP Server does not support BURL, and so the IMAP client has to upload another copy of the message.

The link speed is 100MB, but I see in the emClient logs, that it is timing out the upload, and every time it’s just 1 minute.

I think this is a new bug, I don’t recall this happening in the past.

I just sent a file again, and it’s only 20MB and same problem. The drive that the mail store is on is on a 1Gb-connected QNAP NAS, so maybe that’s a slight delay, but the client should not be timing out with a hard-coded value regardless of message size, that is so 1994.

4

It’s possibly taking more than 1 min to update the index file on the server.

I’ll need to check the IMAP server code that writes the index file.

This may need to happen before the server will acknowledge the upload, and so the 1 minute may be between the client thinking it’s uploaded the message, and getting the acknowledgement.

5

It’s nothing to do with SMTP. The mail is accepted for delivery, and forwarded to the end user no problem

The problem is in IMAP syncing the mail to the sent folder. The link speed is 100MB, but I see in the emClient logs, that it is timing out the upload, and every time it’s just 1 minute.

I think this is a new bug, I don’t recall this happening in the past.

I carn’t personally replicate timeouts updating the sent box when sending any size file attachments using a few different IMAP global and local ISP mail servers with the latest eM Client V9 or V10 on PC Win 10/11 or Mac Sonoma / Sequoia.

So if you already are using V9 or V10 & one of those OS’s, then that sounds like something outside of the mail client whether mail server configuration or physical connection issues.

I’d suggest to contact the technical support of the company “who makes that specific mail server” for assistance. If they then claim it’s an eM Client issue, then if you have a current active Pro or Personal version go to the following VIP support page and login and lodge a support request to have that investigated further. If you do lodge a support ticket let them know what eM Client version you have and your OS version.

For a test, I’d suggest to try sending emails wirh large file attachments such as 20+ mb via a eg: cloud drive in eM Client such as Google drive, OneDrive, Dropbox etc, and see if that will even update the sent mail folder without timeouts.

6

the IMAP server is actually written by us (WinGate).

I can see the complete message file in the mailbox, however the index isn’t updated, and I think it’s because the client disconnects prior to this.

I can add some more instrumentation code to WinGate to see exactly where the client disconnects, but I don’t think we can issue a response indicating a successful APPEND prior to all the steps being completed.

An append to a mailbox should be quick to write the index (seek to end and write just that index entry). However moving the message file from temp folder to the mailbox volume could take various amounts of time. The QNAP NAS is not the quickest in this situation. This will come down to load and size.

I think 1min hard coded timeout after upload for a server response, with no workaround, and no contemplation of message size is showing its fragility here.

7

When I ran into timeout issues due to large attachments with different email clients (it is not limited to eM Client), it had to do with the time it took the server to process the incoming message after the “DATA” send process completed, and when the server sent the “250” code that the message sent ok. Different clients have different time-out periods, where if it doesn’t get the 250 code after the DATA process finishes in a specified amount of time, the client will give a “Timed Out” error.

To improve the issue the server code had to be modified to process things a little faster between the end of the data sent and reporting the 250 message sent ok. Even then, there is a limit on how much you can do because beyond a certain point it’s likely to take too long to process. So when it comes to very large attachment sizes beyond a specific size, it is best to do what @cyberzork said and utilize cloud based storage attachments. The additional advantage to this is that it avoids the MIME/Base64 encoding overhead since it is just inserting the link to a file that was uploaded using other protocol methods then SMTP (HTTP for example).

1 Like
8

DATA and 250 is SMTP not IMAP

There’s no point me creating extra work for cloud-based storage of attachments for my own mail when I host my own mail server.

I think clients need to consider that some servers in some cases will have processing delays.

For example our SMTP server has a scanning step with AV in between acceptance and acknowledgement.

For a large file with many contained files, the unpacking and scanning process can be quite time consuming.

1 minute hard-coded timeouts are something that really should be avoided. I say this because I have done them myself before, or had team-members do it, and it caused only pain.

9

Just for interest sake, from memory (if it doesn’t fail me) the SMTP spec recommends 10 minutes as a minimum timeout for DATA after sending.

10

Yes I know, where did I say it was IMAP?

I was just telling you in our instance when we ran into a similar issue as you described, it was due to the delay between issuing the DATA and 250 commands. I am only providing this information as a possible solution to the issue, but then again maybe your issue is completely unrelated to that if you say it is IMAP related. I assume SMTP will become an issue if the time between the DATA and 250 commands is well over 60 seconds. In instances where it is a SMTP issue, you usually see the email stuck in the “Outbox”, which will cause eM Client to keep trying to send again and timing out again (basically a loop).

I agree SMTP servers do various processing between the acceptance and acknowledgement, which is why in our case the server side code was adjusted to speed up the process. Of course this doesn’t fix everything since the amount of time the data is being sent is variable.

I agree hard-coded limits such as this are bad, and there should be an option to change. Of course it was noticed email clients had different default time-out values, for example I think Thunderbird has a default value of 100 seconds, which can be changed in the advanced settings by adjusting the “mailnews.tcptimeout” setting. Of course most normal users will not edit this and will most likely use the default client time-out, so this would be a niche case for use.

Using the power of AI (assuming it doesn’t hallucinate) here is the result I got back about the SMTP spec recommendation you state:
According to the SMTP specification (RFC 5321), while there isn’t a strict, explicitly defined “DATA timeout” value, best practice suggests setting a timeout for the DATA command of at least 5 minutes when receiving email data from a sender, allowing ample time for large messages to be transferred without interruption; this is especially important for reliable email delivery.

So I agree 1 minute is not long enough if that is indeed what eM Client uses for any IMAP or SMTP time-out, and this should be more like 5 minutes. Additionally I agree this should be a setting where it can be changed from any default. I am not arguing your point at all when it comes to increasing this limit, and allowing it to be adjusted. The problem is your most likely still going to hit a limit at some point where using something other then SMTP is recommended to send very large attachments.

So lets hope someone from eM Client reads this and could comment further about any SMTP or IMAP time-out limits, what exactly they are, and if they can be adjusted to allow larger files to be sent using SMTP, or copied using IMAP.

11

Yes the problem is happening in IMAP, there’s no problem with SMTP. The mail is accepted for delivery, and forwarded no problem, and this goes for much much larger attachments (over 100MB).

The problem occurs in STORE command in IMAP when the client is trying to make a copy (additional upload) to the IMAP server to store the mail in the Sent Items folder.

So yes, the client would need to alter IMAP client code, not SMTP client code.

I agree in the end any time you have a set timeout, you will find edge cases to defeat it. Another common strategy if you want to minimize the timeout with fewer (maybe) problems, is to allow the timeout to be calculated based on the size of the file, or other factors. But even this is an assumption that the time is related to the size. It’s impossible for a client to predict how long a server will take to complete processing after upload prior to response, and the spec doesn’t contemplate giving the client an interim response to tell it everything is still ok. I know a client can’t rely on a TCP disconnect either.

Probably the simplest fix would be an advanced setting to allow to either disable or greatly extend the timeout.

Regards

Adrien

1 Like