xImapPusher - based on ImapPusher-0.60

Fixup · Dec 2, 2009

This program is based on the source code of Imap Pusher Service (17762) V0.60:

http://www.codeplex.com/ImapPusherService

Improvements:

1) No longer quit;
2) Auto recovery from socket errors;
3) Auto re-connect;
4) No mail sync after a re-connection, good for battery life.

Many service providers timeout the connection very shortly. For example, Verizon closes inactive sockets after 5 minutes. While xImapPusher can recovery from these 5-min timeouts, it is better not to raise a mail sync each and every 5 min.

5) Added a connection management service, XinConnManager.

If data connection is disconnected, this service will know immediately and reconnect and restart the push service. This service is truly event driven, so there is no periodical polling, idle all the time.

The multiple version can push multiple IMAP accounts, based on the 061 source code made by camro. Tested to work with Gmail and Fastmail.

Enjoy!

[For best battery life, set refresh interval to 28 and the multiple version is recommended even if you have only one account to sync.]

camro · Dec 2, 2009

SOHKis · Dec 2, 2009

DLing & testing now! thanks fixup

any chance of being able to add more than 1 account to the .xml..?

rsenden · Dec 2, 2009

Battery drain & stability

Until recently I was using Windows Live to get push email. This was working quite well, but nonetheless I've switched to GMail & IMAP for various reasons. I'm currently using EmailScheduler for IMAP IDLE support, but it looses connection quite frequently (as in 1-2 times per day). I also noticed that it drains the battery much more than the Windows Live solution. I did some tests with ImapPusher quite some time ago, but it wasn't stable enough.

Would xImapPusher solve these problems?

I noticed xImapPusher comes with a XinConnManager DLL; do you have the sources available for that component as well (I would like to know what I install on my device ;-).

Kind regards,
Ruud.

rsenden · Dec 2, 2009

Possible improvement?

I've been having an idea that could possible improve performance and battery usage of IMAP IDLE solutions like these.

Basically, one would implement some kind of IMAP proxy. This proxy would consist of a socket listener (localhost only) and an IMAP IDLE client. Pocket Outlook is then configured to point to this proxy instead of the actual IMAP server.

The IMAP IDLE client would basically remain the same. However, whenever Pocket Outlook connects to the proxy and issues a LOGON command, IMAP IDLE service is suspended. All remaining Pocket Outlook commands are simply passed through over the same connection that was previously used for the IDLE command. Once a LOGOFF command comes in and Outlook closes the connection to the proxy, the IMAP IDLE client resumes IDLE operation (again, over the same connection). Of course, LOGON, LOGOFF and similar commands issues by Pocket Outlook will not be passed on to the actual IMAP server; the connection will be managed by the IMAP IDLE client.

This way, you prevent Pocket Outlook from setting up its own connection to, and authorizing with, the remote IMAP server for each incoming email. Instead it will re-use an existing authorized connection.

I'm not sure whether it makes that much of a difference, but it might be worth the effort.

Kind regards,
Ruud.

camro · Dec 2, 2009

Fixup · Dec 2, 2009

I've done embedded programming for 15 years, so you can trust my code.

Many programs, such as PortSip and EmailScheduler, are not event-driven, they keep polling which sucks battery, runs the phone hot and not reliable. XinConnManager is a true event-driven service, has no such problems.

When I have time, I'll re-write the entire ImapPusher from VB to native C++. Anything simply sucks when touchs .Net and VB.

Actually my current push solution is direct http push, very reliable and no any impact to battery life. Here is how to do it:

1) Run Depeche, a http server for PPC. Modify the SimplTex ISAPI to tell Outlook to sync mail when it gets a http request.

2) Run Pocket DynDNS to update "mypocketIP.homeip.net". This program works, but it uses Sleep(60 seconds) to update DynDNS. You know, Sleep() does not work as soon as the phone goes into suspend mode. So I made "XinServerManager", a service similar to "XinConnManager", to keep the connection and meanwhile call PDD to update dyndns immediately after a new (re)connection. Then kill the PDD.

3) Run PPC emulator on a XP machine and run a slightly modified xImapPusher on top of it. When email arrives, it sends a http request to my phone and the phone then will do a sync to get the new mail.

This is by far my push solution. It does not need any open connection between the phone and the server. True and pure push. This solves the porblem of 5-min timeout with many 3G providers such as Verizon. Such 5-min timeout kills battery. Before, 4 hours of push running would drain 10% battery, now with my direct http push, 10% will run the phone for 24 hours, like it is no push running at all!

bsinclair · Dec 2, 2009

Battery will always be worst using IDLE, this is one thing Microsoft did right. If you wanna know a little bit more, see here:

http://gudensoft.com/?p=140

Instead of using this, I moved to active sync as now Google supports it.

Fixup · Dec 2, 2009

bsinclair said:
Battery will always be worst using IDLE, this is one thing Microsoft did right. If you wanna know a little bit more, see here:

http://gudensoft.com/?p=140

Instead of using this, I moved to active sync as now Google supports it.

That article is totally wrong, see my comments at the bottom. Both require an open connection, just inactive, no difference here between these two.

IMAP idle is actually better on this regard: it'll detect the 5-min timeout and make a reconnection, no push lost.

With ActiveSync, it won't detect the timeout and any mail arrived after 5 min until the next beatheart will not be pushed in time.

That's why AS makes you feel a little better on battery life. The default heartbeat is 8 min. So you do get slightly better battery life, but you pay for that with push delay. You may set "MinHeartBeat" to 300 (5 min) so you won't suffer from delay, but then the battery life is no any better than IMAP idle.

The only reason MS does not add IMAP support to WM is because they want to sell Exchange servers and the ActiveSync Direct Push service. Both Apple and Google went to this stupid route.

To address this 5-min timeout issue, I set xImapPusher to re-send the heartbeat without raising a mail sync.

BTW, MS Live push is based on IMAP idle, not ActiveSync. You know, years ago MS bought Hotmail which was Linux based, seems still Linux. Live is far more reliable than AS. Like rsenden, I too was using Live because it does not have all kinds of troubles with Gmail's AS push. However, Verizon applied the stupid 5-min timeout sometime this year or so, so I had to figure out my direct http push.

SOHKis · Dec 2, 2009

Fixup said:
I've done embedded programming for 15 years, so you can trust my code.

Many programs, such as PortSip and EmailScheduler, are not event-driven, they keep polling which sucks battery, runs the phone hot and not reliable. XinConnManager is a true event-driven service, has no such problems.

When I have time, I'll re-write the entire ImapPusher from VB to native C++. Anything simply sucks when touchs .Net and VB.

Actually my current push solution is direct http push, very reliable and no any impact to battery life. Here is how to do it:

1) Run Depeche, a http server for PPC. Modify the SimplTex ISAPI to tell Outlook to sync mail when it gets a http request.

2) Run Pocket DynDNS to update "mypocketIP.homeip.net". This program works, but it uses Sleep(60 seconds) to update DynDNS. You know, Sleep() does not work as soon as the phone goes into suspend mode. So I made "XinServerManager", a service similar to "XinConnManager", to keep the connection and meanwhile call PDD to update dyndns immediately after a new (re)connection. Then kill the PDD.

3) Run PPC emulator on a XP machine and run a slightly modified xImapPusher on top of it. When email arrives, it sends a http request to my phone and the phone then will do a sync to get the new mail.

This is by far my push solution. It does not need any open connection between the phone and the server. True and pure push. This solves the porblem of 5-min timeout with many 3G providers such as Verizon. Such 5-min timeout kills battery. Before, 4 hours of push running would drain 10% battery, now with my direct http push, 10% will run the phone for 24 hours, like it is no push running at all!

this sounds like a nearly perfect solution, now if it could only be done w/2-3 accounts.

i'll have to give this method a try someday when i get some freetime on my hands. thanks man!

camro · Dec 2, 2009

Fixup · Dec 3, 2009

The phone keeps connections and sockets in suspend mode, it is the service provider (mine is Verizon) that closes inactive sockets (not the connection) after a timeout (5 min in case of Verizon). You have no control about that.

However, the connection does get disconnected sometime, a random thing, that's why your program needs to wait for the "NOTIFICATION_EVENT_NET_DISCONNECT" event. Note that when you unplug your phone from our computer, it also generates this event, so your program needs to deal with that too.

How long you can use Sleep() is defined by the power settings. For example, if you set your phone to suspend in 60 seconds, you cannot use Sleep(61000). Anyway, you should not use Sleep() at all, because it'll either prevent your phone from suspending or hang your program till the phone wake up.

This is how I do wait when needed:

Code:

SYSTEMTIME GetTimeAfterSeconds(DWORD nTimeInSeconds)
{
	SYSTEMTIME sTime;
	FILETIME fTime;
	GetLocalTime (&sTime);
	SystemTimeToFileTime(&sTime, &fTime);
	ULARGE_INTEGER uTime;
	uTime.HighPart = fTime.dwHighDateTime;
	uTime.LowPart = fTime.dwLowDateTime;
	uTime.LowPart += nTimeInSeconds*1000*1000*10;
	fTime.dwHighDateTime = uTime.HighPart;
	fTime.dwLowDateTime = uTime.LowPart;
	FileTimeToSystemTime(&fTime, &sTime);

	return sTime;
}

void Wait(HANDLE hEventTimer, DWORD dwTime)
{
	CeRunAppAtTime (_T("\\\\.\\Notifications\\NamedEvents\\EventTimer"),&GetTimeAfterSeconds(dwTime));
	WaitForSingleObject(hEventTimer, INFINITE);
}

main()
{
	HANDLE hEventTimer = CreateEvent (NULL, false, false, _T("EventTimer"));
	......
	Wait(hEventTimer,60);
	......
	CloseHandle(hEventTimer);
}

Note that WaitForSingleObject(hObject, 60000) won't work, like Sleep(60000).

Fixup · Dec 3, 2009

SOHKis said:
this sounds like a nearly perfect solution, now if it could only be done w/2-3 accounts.

i'll have to give this method a try someday when i get some freetime on my hands. thanks man!

Multiple accounts sync should be easy, just tell which account to sync in the http request.

rsenden · Dec 3, 2009

Fixup said:
I've done embedded programming for 15 years, so you can trust my code.

It's not that I don't trust your code; as a programmer (non-embedded Java and some C#) I'm just interested in what components like these exactly do. Anyway, I had some time left yesterday so I installed xImapPusher. So far it's working good, too early to compare battery usage with EmailScheduler yet though.

Fixup said:
Actually my current push solution is direct http push, very reliable and no any impact to battery life.

This indeed looks like a very good solution. Unfortunately it currently requires a Windows machine running 24/7 (which I don't have) and is too difficult to set up for the average user. For me, a good alternative would be to run the 'IMAP IDLE to HTTP request' component on my CH3SNAS.

Of course, the ultimate solution would be to have a hosted solution on which users can register their IMAP accounts, together with a program on their PPC that both listens for push notifications send by the hosted solution, and sends IP updates to the hosted solution.

rsenden · Dec 3, 2009

camro said:
i'm not sure, if this save battery ..... maybe you are interested on this more in detail

Basically, I got the idea by reading somewhere that Pocket Outlook IMAP solutions are always less efficient than, for example, ActiveSync. This is because every time Outlook needs to retrieve mail, it needs to set up a TCP/IP connection, set up SSL if applicable and do a LOGON (and afterwards do a LOGOFF, close the connection).

With an IMAP IDLE solution like xImapPusher you already have a secured and authenticated connection set up, so why not have Pocket Outlook re-use that same connection? However, I don't know how much data and processing power is involved in setting up and closing the connection, and thus how much impact this could have on battery life and time taken to retrieve new mail.

So basically, xImapPusher would not only function as an IMAP client, but also as an IMAP server. The xImapPusher server component handles incoming connections, together with LOGON and LOGOFF requests. All commands between LOGON and LOGOFF are simply passed on to the real IMAP server via the connection that was previously set up by the IMAP IDLE client component (which has temporarily suspended IDLE operation).

SOHKis · Dec 3, 2009

Fixup said:
Multiple accounts sync should be easy, just tell which account to sync in the http request.

sorry, the multple accounts was in reference to the re-writing of imappusher to c++

still gonna give the http method a shot when i get some time. thanks again man

camro · Dec 3, 2009

camro · Dec 3, 2009

Fixup · Dec 3, 2009

rsenden said:
Basically, I got the idea by reading somewhere that Pocket Outlook IMAP solutions are always less efficient than, for example, ActiveSync. This is because every time Outlook needs to retrieve mail, it needs to set up a TCP/IP connection, set up SSL if applicable and do a LOGON (and afterwards do a LOGOFF, close the connection).

With an IMAP IDLE solution like xImapPusher you already have a secured and authenticated connection set up, so why not have Pocket Outlook re-use that same connection? However, I don't know how much data and processing power is involved in setting up and closing the connection, and thus how much impact this could have on battery life and time taken to retrieve new mail.

So basically, xImapPusher would not only function as an IMAP client, but also as an IMAP server. The xImapPusher server component handles incoming connections, together with LOGON and LOGOFF requests. All commands between LOGON and LOGOFF are simply passed on to the real IMAP server via the connection that was previously set up by the IMAP IDLE client component (which has temporarily suspended IDLE operation).

Compared to the excessive battery drain from the 5-min timeouts, nothing else matters.

Don't believe too easily what you see on the net. Highschool bloggers dont know what they talk about. If IMAP idle is not good, why MS Live uses it?

I've done extensive tests between imap idle and ActiveSync. I also tested z-push (AS clone) and many other things in searching for the best push solution.

The real problem with imap idle is there is only few mobile clients support it and none of them work. That's why I spent time on this project.

rsenden · Dec 6, 2009

Fixup said:
Compared to the excessive battery drain from the 5-min timeouts, nothing else matters.

Don't believe too easily what you see on the net. Highschool bloggers dont know what they talk about. If IMAP idle is not good, why MS Live uses it?

I don't believe everything I see ;-). I was just trying to find an explanation why for example EmailScheduler drained the battery so much faster than Windows Live. After trying out xImapSync for the last few days, I now know that it is just caused by the way EmailScheduler is implemented.

With xImapPusher battery drain is about the same as Windows Live I think, and it's also more stable than EmailScheduler (e.g. I don't have to force a reconnect every now and then). So thank you for your contribution to this project!

xImapPusher - based on ImapPusher-0.60

Senior Member

Attachments

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Similar threads