Check your free memory. This normally happens when your launcher gets thrown out to make space. Maybe get rid of some bloat.
Edit: Another option could be raising the memory limits to throw out apps earlier (hopefully not the launcher, lol, but apparently that gets kinda elevated treatment).
Sent from my XT910 using Tapatalk 2