Issue Reminders stopped Reminding (Jira DC 7.13.8)
On Sept 30 a user reported that he was not receiving notifications for reminders he created using the Issue Reminders plug-in. Upon testing 10, 15, and 30 minute reminders, it became obvious that no reminders were being triggered regardless of project or user or permission.
The reminders continued to show the clock icon instead of the checkmark icon implying that they were not being triggered. There were also no comments which are added when the reminder is triggered. All of this pointed to an issue with the core plugin and not email connectivity or permissions. Testing Reminders from the global admin panel worked fine, so we know emails worked.
Logs were collected and the plugin uninstalled then reinstalled from the Atlassian Marketplace. Immediately after reinstallation, all past reminders were sent and test reminders set for the future also started working.
We increased Issue Reminders logging to DEBUG couldn't determine from the logs what might have went wrong. There were no ERROR messages logged or stack traces.
Attached is a screenshot showing what happened to use for a specific test. The reminder was scheduled for 30-Sept at 3:30pm (about 15 minutes in the future at the time, we are aware of the issue with reminders 5 minutes or less in the future). You can see on the panel on the left that the reminder fired at 30-Sept 8:14pm which was when we reinstalled the plugin.
Do you have some advice about where we could look that might have caused this? Was an event queue backed up? Is there a database table we can examine? Is this maybe related to our use of Data Center?
Jira Data Center 7.13.8
You may disable sending reminders to user groups from add-on settings. If a user selects "jira-users" for a reminder it will generate lots of logs and may overload email system.
No luck finding anything useful in the logs.
We have left debug logging on for the Jira Issues Reminders plugin and also added an alert if the “Time to send reminders:” message suddenly disappears from the logs. So if this occurs again we will know what happened and what else was going on that might be related. We also will know this has failed before our users start missing reminders.
Thanks for the sharing that code snippet. It helps greatly to see the logic.
Based on the code, I would expect to see in the log either the “The add-on received a pending…” WARN or the “Acquiring cluster lock” DEBUG. In the logs I captured I don’t see either of these which implies that sendPendingReminders is not being called. (FWIW there are no “Previous instance is still running” messages).
So this makes me think it was something to do with the Scheduler. I’ll go through the logs for the entire day and see if anything was captured there.
We acquire "cluster lock" to ensure that only one server in a cluster will send reminders. If you are suspecting a cluster lock is not released for any reason you can follow the procedures explained here, https://confluence.atlassian.com/jirakb/healthcheck-cluster-locks-738722353.html
Logging "acquiring cluster lock" message is one of the first things we do when send reminders is triggered as shown in the following code. Do you see any "The add-on received a send pending reminders request while already processing reminders You may be sending too many reminders, for example you may sending a reminder to large jira user group" message. If it is not also written to logs, that means scheduler for the reminder is not triggered. If that happens again you should see an explanation mark on "scheduler Administration" page with an explanation. Can you also check whether do you see any "Previous instance is still running" message in the logs?
The tests were all setup at least 10 minutes in the future, we are aware of the 5 minute limitation.
Thanks for sharing the expected debug message sequence that is helpful. In our current Jira I can see regular log messages that match that sequence. However in the logs from when the plugin was not reminding I don’t see any cluster messages across any of the three nodes:
I didn’t learn anything by examining "Scheduler Administration” as it simply shows the current status. It is working now so I just see success messages.
What would cause the “Acquiring cluster lock for sending reminders" message from generating?