Checking IMAP-Folders still very slow

It would be really great if the speed at which EM-Client checks IMAP-Folders could be improved, because it’s extremely slow.

For a customer popup-email-notifier, I wrote a script some time ago, using Limilab’s - mail.dll, that checks my 60 IMAP-Folders for new emails and reports them.
With PowerShell, which is not known for its speed, that takes under 4 seconds.

eM-Client takes 30 seconds for the same task.

I had high hopes for v10, but there is seemingly no improvement.

With PowerShell, which is not known for its speed, that takes under 4 seconds.

eM-Client takes 30 seconds for the same task

Could be the eg: verification of the IMAP folder changes using Limilabs.dll “isn’t as stringent as eM Clients” so is quicker when you run it.

Or Limilabs might use eg: alot less overhead when running it so is then quicker to verify changes.

I’m seeing slow IMAP syncs even on emClient on IPadOS.

To be fair my five accounts, customized on iPadOS, don’t take 30 seconds to finalise the requests, however the same five accounts on my Android Samsung, using FairMail, take only up to 5 secs to finish.

This was bugging me as well, as I was testing Emclient.
I’m thinking of replacing Thunderbird because of the database, but I have a lot of folders, and it takes forever…

Hi! The listing of IMAP folders should not be this slow, can you perhaps send us more details so we can look into it?

Ideally please enable IMAP logs in Menu>Settings>Advanced for the affected account.
Restart eM Client.
Wait for the slow loading to happen again - note down the time it started, and once the folders finish loading, close eM Client again.
Then send us the created log files and the time where you noticed the slow loading (so we can check the same time in the log). You can find the logs via Menu>Help>Open Logs Directory.
Send these log files to [email protected] with a link to this forum post in the body.

Could you also perhapd send us the Limilab’s - mail.dll you mention so we can look what that tool does exactly?

Thank you for your cooperation.

Hi Olivia, i’ve sent the requested logs. Btw this is not a random occurrence; the script and EM-Client running on the same machine and querying the same mail-account. And em-client, unfortunately, is always about 8-10 times slower.
Thanks!

Thank you! I confirm I see the email with the logs and will forward them to our IMAP specialist to look into :slight_smile:

I consulted with our IMAP expert and unfortunately, I don’t have the best news.

The reason for the slower folder check you’re seeing is because your IMAP server (GMX) does not support the CONDSTORE extension (RFC 4551). Without it, eM Client cannot efficiently ask what has changed since the last sync, so it must recheck message flags for all messages in each folder to keep a correct and consistent local copy.

Tools or scripts that only check whether new messages arrived and don’t verify what happened to existing messages can be dramatically faster.
With a large message history, this difference can easily be orders of magnitude.

What can improve the behavior at this time is using time limited sync in IMAP settings in Accounts section (so there are fewer messages to check), or ignoring/not syncing some folders (Ignored paths in IMAP tab). But that would not be a solution if you want to keep your full message history available in eM Client.

We’re aware this is not an ideal state for servers like GMX, and our developers have considered alternate ideas on how to address this (for example, limiting how many messages are rechecked even when full synchronization is enabled).
However, this could lead to inconsistencies compared to webmail if changes are made to older messages, so we have not found a good solution to this specific behavior.

We will continue looking into whether further optimizations are possible without losing functionality, or maybe find a way to have an optional setting to run in a mode similar to Limilab’s mail.dll.

In general, however, without server-side support for useful IMAP extensions like CONDSTORE, there are inherent limits.

1 Like

Thank you for your detailed answer; I appreciate that.

But :blush: if this was a mailservice-provider/server-feature issue, why does the limilabs-dll has seemingly no problem with it?

Because my script does in fact check if any mails are marked as “unseen”, so it does check (like em-client) every email (I have about 109.000). Only much much faster.

I can mark any email in any of my folders, no matter how old, manually as “unseen” and a few seconds later my script notifies me of this mail as being “new”.

You mentioned your script checks for new messages in the original post, so we assumed that is the difference, but if it indeed checks each message, we’d need to see the actual communication with the server to figure out what the difference might be.
Would it be possible to get a server log from when the script checks the messages? Or is your script able to generate some kind of IMAP log?

Or another idea of course is that your script checks just that one detail (only the unread flag) or small set or parameters, and nothing else about the message, while we as a full desktop client check for all parameters of the message that can be changed, which can be other flags, including hidden Server tags (\Answered \Flagged \Draft $Forwarded $MDNSent…), or if the messages was moved or deleted etc. - checking for a removed message can be a more complex task than just checking unread status.

But also to accomodate other functionality (such as getting messages in Inbox or currently opened folder first) we cannot do this in one single action - yes with your script you know how your specific server will respond, but we need to have the system optimized for many different servers, including those we cannot directly test ourselves.
So we need to be prepared for cases what if the server stops responding in the middle? What if the response is huge and we cannot parse it in time?
Without having an out in the middle this could block all other sync for the account or even other accounts.
We need to consider these and many other questions when changing the IMAP synchronization.

we need to have the system optimised for many different servers, including those we cannot directly test ourselves.

Agree with @Olivia_Rust that currently eM Client works, so adjusting it might stuff up alot of other IMAP mail server connection. In my opinion id personally leave it alone.

GMX should just support the CONDSTORE extension ([RFC 4551] like most other good mailbox providors.

1 Like

Since I’m facing slow IMAP requests, too, running emClient on iPadOS 18, I’m not sure if it’s caused by missing protocols on server side solely.

I’ve configured Yahoo, GMX and Goneo accounts, 6 in total and while emClient seems slow when checking for new emails, another email client (Canary EMail), configured with the same accounts as above, is performing well.

Additionally, same accounts on Android, email client is FairEmail, checks for new emails are finished within a second or two.

Just my 2 ct’s - I’m not the IMAP expert, though :wink: and checking for new mails in inbox only is certainly a different process than checking a whole bunch of IMAP folders.

I get all that, of course, but I don’t think that is it. :slight_smile:
For the heck of it, i’ve rewritten the script to use MailKit (a popular open source mail library), which I wanted to do anyway and… its even faster (not by much, but still).

I have a lot of processing and checking in the script, e.g. I am checking each mail for spam flags, querying a text-db for duplicates and building a preview in the notification window.
All of which is done on the slow PowerShell side.Sure, Emclient might have even more overhead, but I mean, I am doing it with a script. :slight_smile:
I can send you the script if that helps; no magic in there.

I’ve also created verbose logs with the mailkit-logger; I send both to [email protected].
I wouldn’t mind if emClient was just a bit slower, but nearly ten times? That seems a lot.

1 Like

@Olivia_Rust Any news?

Sorry for the delay, I was waiting for our IMAP expert to have time to analyze the script and the server log.

From what we can tell, the script seems to work a little like POP (but on multiple folders) since it just opens each folder, and checks for unread messages (in the part where it actually contacts the server).
It does not seem to check for any other flags or check for changed message count (ie deleted messages) or any other additional information that I mentioned before that we need in our app.
This seems to correspond with the Server log as well: it just returns search for unread messages - a limited action without checking for other information.

In short, as a full desktop email client, our app requires significantly more information and must work reliably with both small and very large mailboxes across many different server providers. With a large number of messages, your script would likely become much slower, but since it’s tailored for personal use rather than a general solution, it can afford to be more lightweight.

For these reasons, we unfortunately can’t promise a specific performance improvement for GMX at this time. As mentioned earlier, our system has to support many different setups and be robust against unexpected server behavior and in-app actions.
But maybe our team can consider some optional simpler mode in the future, though it would need to be considered which other features would be affected by this and if it’s feasible.

If you are able to log a feature request with GMX as well, adding CONDSTORE to their servers would improve the speed a lot (and it is supported by most IMAP servers nowadays).

Mhm, what do you consider a “large number” of messages?
I have about 110,000 emails in my account. :upside_down_face:

But anyway, I hate to be a pain in the ***, but I still think this explanation is, sorry, wrong.

To prove it, I just reinstalled Thunderbird and logged the check of the same account.
To my surprise, it is even faster than my script: about 2 sec. And about 12-15 times faster than eM Client.
(I have sent you the log file.)

And I would think Thunderbird qualifies as “full desktop email client, [that] works reliably with both small and very large mailboxes across many different server providers,” does it not?

So I think there has to be another explanation for eM Client’s slowness.

Yes, we have to be ready for mailboxes and databases much, much bigger than that.

Thunderbird has its own IMAP implementation and simplifies some other steps as well - but on the other hand they can struggle with different accounts that we can handle easily. We currently prioritize all the things I already mentioned - being able to handle many different server types of all sizes, on all types of devices and both fast and slow connections.

We might consider adding a separate IMAP mode for older servers and those without CONDSTORE in the future, but I cannot confirm that at the moment.

That’s a pity. Thanks anyway.