Retry a recording status update if it fails

[smarquard]

Galicaster will try once and only once to update the recording status of an event when it starts recording. (example below). If this fails, it doesn't retry.

Galicaster should retry the recording status update periodically (for example on the long counter) until it succeeds, or the recording ends.

galicaster      2019-03-05 08:59:59,946 INFO    scheduler/scheduler     Start time for recording e9c05e74-05e4-4773-934f-dcff729fef85, duration 3300000 ms
galicaster      2019-03-05 08:59:59,969 INFO    recorder/service        Recording (current status: preview)
galicaster      2019-03-05 08:59:59,970 INFO    recorder/service        Recording to MP e9c05e74-05e4-4773-934f-dcff729fef85
galicaster      2019-03-05 08:59:59,984 INFO    opencast/service        Sending state capturing for the scheduled MP (u'e9c05e74-05e4-4773-934f-dcff729fef85', u'EEE3091F LG01 09:00 14', datetime.datetime(
2019, 3, 5, 7, 0))
galicaster      2019-03-05 09:00:00,026 INFO    mediapackage/repository Temporary data saved to /usr/share/galicaster-repository/rectemp/info.json
galicaster      2019-03-05 09:00:02,947 DEBUG   scheduler/heartbeat     galicaster-notify-short in 15
galicaster      2019-03-05 09:00:16,950 ERROR   opencast/client call error in http://media.uct.ac.za/capture-admin/recordings/e9c05e74-05e4-4773-934f-dcff729fef85, status code {502}: 502 Proxy Error
galicaster      2019-03-05 09:00:16,950 WARNING opencast/service        Problems to connect to opencast server: Error in Opencast client
galicaster      2019-03-05 09:00:17,946 DEBUG   scheduler/heartbeat     galicaster-notify-short in 15
galicaster      2019-03-05 09:00:17,947 INFO    opencast/service        Init opencast client
galicaster      2019-03-05 09:00:17,962 INFO    opencast/service        Be careful using profiles and opencast scheduler
galicaster      2019-03-05 09:00:17,962 INFO    opencast/service        Updating series from server
galicaster      2019-03-05 09:00:32,947 DEBUG   scheduler/heartbeat     galicaster-notify-short in 15
galicaster      2019-03-05 09:00:32,947 INFO    opencast/service        Set status capturing to server
galicaster      2019-03-05 09:00:32,947 INFO    opencast/client Sending state capturing

[smarquard]

The #614 improvement is good. We do see cases though where Opencast is up but a status update fails for some transient problem (e.g. network glitch, timeout on the request because Opencast is very busy or something).

So Galicaster needs some code that checks periodically if opencast is up, and if a recording is in progress, and if a status update for the recording has not been successfully sent, then send a recording status update.

[Alfro]

Hi @smarquard
This improvement should work as you specify.

Even if only that particular endpoint is failing, Opencast is set to "down" and, after the current changes, we will retry to set the state, which will set Opencast as "down" if it fails. As there is already a periodic check for whether Opencast is up, Galicaster will retry to set the state.

The only issue I can see here with the current change is that if Opencast is working, but that endpoint is failing (maybe because of a high load), it may not help if all Galicasters are retrying every long beat.