Recorders that synchronize time with NTP servers may show problems with audio quality in recordings on analogue channels and in some cases recording may stop on some or all analogue channels after a period of time. In the latter case, recording resumes as expected on all analogue channels after restarting the recorder.
The issue is caused by what we consider to be large time drifts between the time on the NTP server and the recorder. This can be the result of an inaccurate/faulty NTP server or (most likely) a large network latency between the recorder and NTP server networks.
Typical average drifts between a recorder and a reliable NTP time source should be <10-20ms. Total Recall VR recorders are real-time systems and need a reliable time source to operate correctly.
Audio quality issue happen when the time on the recorder moves back by more than 100ms at the time when the recorder clock is synchronized to the clock of the NTP server. For example, messages such as the following in the Linux log and the recorder log are typical when audio quality issues are present:
Linux log:
Feb 16 15:48:50 totalrecallvr ntpd[2426]: time reset -0.405405 s
Recorder log:
2018-02-16 15:48:51,509 WARN DSP Provider Worker - DSP code: 1, DSP board: 0, DSP Board 0 missed xfer oldCnt=4679381 newCnt=4679384
The log basically shows that the time shifted back by >405ms. As a result, timers that are responsible to fetch audio data from the analogue channel cards missed to read audio data from the cards 4 times. This audio data is lost and as a result there is audio quality issues in the recording (basically up to 400ms of audio is missing).
If such time changes occur on regular basis and result in “missed” data from an analogue channel card, then the recorder will “think” that the analogue channel card is faulty and eventually exclude it from recording. In this case this is what will appear in the system and recorder log:
Linux log:
Feb 17 04:21:13 totalrecallvr ntpd[2426]: time reset +0.242920 s
Recorder log:
2018-02-17 04:21:14,025 WARN DSP Provider Worker - DSP code: 1, DSP board: 0, DSP Board 0 missed xfer oldCnt=4829857 newCnt=73
2018-02-17 04:21:14,120 WARN DSP Provider Worker - DSP code: 1, DSP board: 0, DSP Board 0 missed xfer oldCnt=73 newCnt=73
2018-02-17 04:21:14,025 WARN DSP Provider Worker - DSP code: 1, DSP board: 0, DSP Board 0 missed xfer oldCnt=4829857 newCnt=73
2018-02-17 04:21:14,120 WARN DSP Provider Worker - DSP code: 1, DSP board: 0, DSP Board 0 missed xfer oldCnt=73 newCnt=73
2018-02-17 04:21:14,629 WARN DSP Provider Worker - DSP code: 2, DSP board: 0, DSP Board 0 may have trouble locking buffers (4, 0)
2018-02-17 04:21:15,139 WARN DSP Provider Worker - DSP code: 2, DSP board: 0, DSP Board 0 may have trouble locking buffers (4, 1)
2018-02-17 04:21:15,649 WARN DSP Provider Worker - DSP code: 2, DSP board: 0, DSP Board 0 may have trouble locking buffers (4, 2)
2018-02-17 04:21:15,651 ERROR DSP Provider Worker - DSP code: 5, DSP board: 0, DSP Board 0 may have to bootload to recover from problems (4, 3)
2018-02-17 04:21:15,652 ERROR DSP Provider Worker - DSP board 0 may need to be replaced if the problem persists after system restart
2018-02-17 04:21:15,738 ERROR DSP Provider Worker - Call not active on channel: 3
2018-02-17 04:21:15,740 ERROR DSP Provider Worker - Call not active on channel: 4
2018-02-17 04:21:15,745 ERROR DSP Provider Worker - Call not active on channel: 5
2018-02-17 04:21:15,747 ERROR DSP Provider Worker - Call not active on channel: 6
2018-02-17 04:21:15,749 ERROR DSP Provider Worker - Call not active on channel: 7
The result is loss of recordings on some or all analogue channels until the recorder is restarted.
To avoid these issues when using NTP as a time source you must synchronize the recorder to a reliable NTP server and maintain an average time drift of <10-20ms.