Default nexus 5 immersive and nova launcher prime

If I enable immersive mode it disables Nova's "swipe down to open notifications" though swipe up for app drawer still works.
Disabling immersive mode makes it right again...

Does anyone have suggestions as to how I can keep immersive mode but have the swipe down notifications?!