Disable notification when fullscreen app is detected not working

I have the option for “Disable notifications when fullscreen application is detected” enabled. However, I will still get calendar reminders while i have a game full screen, this causes my game to crash.

Is there another setting to help with this?