Post Upgrade Verification Checklist

complete these checks within 10 minutes of the broadcaster restarting use this checklist immediately after every broadcaster or zen master upgrade to confirm that streams, outputs, and reporting have all recovered correctly streams and connectivity • all sources reconnected in zen master, verify all expected sources show as active with normal bitrates sources that remain in connecting state more than 3 minutes after restart need investigation • all targets reconnected verify all outputs are connected in zen master check each failover group is on the correct primary leg • no unexpected stream errors check the zen master event log for any new error events generated after the restart a small number of reconnection events is normal sustained errors or packet loss events are not • bitrates at expected levels spot check bitrates on key sources and targets a bitrate significantly lower than normal after reconnection can indicate a source issue unrelated to the upgrade transcoding and adaptive outputs • transcoded channels are online in zen master or the broadcaster ui, check all transcoded outputs any output showing as offline after the restart with a codec or encoding error may indicate a regression in the new build • adaptive bitrate outputs reconnected verify each adaptive output is online and shows the correct profile ladder an adaptive output showing "offline unsupported encoding" after an upgrade is a known regression pattern see the companion upgrade checklist guide for steps • audio codec outputs are functioning if any transcoded outputs use he aacv2, verify that the audio bitrate is present and correct multi channel audio sources can surface transcoding issues after an upgrade see the multi channel audio downmix troubleshooting guide if affected zen master reporting • stream statistics are populating verify that bitrate graphs, packet loss stats, and content analysis data are appearing for active streams if zen master shows charts with no data, wait 5 minutes chart data can take time to populate after a restart • alert configuration is intact go to settings, then alerts, and confirm all alert rules are present • historical charts are accessible check that historical charts from before the upgrade are still visible if history is missing, try a hard browser refresh first (ctrl+shift+r) if still missing after 30 minutes reach out to support bonding skip this section if bonding is not configured on this broadcaster • bonding statistics are populating verify that bonding stats including link utilization and packet distribution are showing correctly in zen master or the broadcaster ui if bonding stats to display as zero after upgrade, restart the bonding group and refresh the browser before escalating • all bonding group members are active confirm all links in each bonding group are active and contributing traffic recording and storage skip this section if recording is not configured on this broadcaster • s3 recording settings are intact go to settings, then recording, and verify the s3 url and credentials are saved correctly s3 urls can appear to save but silently revert to disk only after an upgrade after saving, navigate away and return to confirm the url is still present • ts recording is capturing if scheduled ts recording is active, verify a capture is running or completed successfully after the restart post upgrade decision tree broadcaster restarted and streams not recovering as expected? streams still connecting after 5 minutes? > check upstream source/encoder is online > force reconnect from zen master > still offline after 10 min > initiate rollback adaptive output shows "offline – unsupported encoding"? > note target version > contact zixi support with version and output config bonding stats showing zero? > wait 5 min > refresh browser > restart bonding group > still zero after 10 min > escalate with version zen master charts missing history? > hard refresh browser > wait 15 min > still missing open support ticket with zen master links and details s3 recording not saving to s3? > re enter and save s3 url > verify credentials > navigate away and back to confirm persistence transcoding output offline with audio or codec error? > see multi channel audio downmix guide or confirm codec settings match source pid broadcaster not starting at all? > check systemctl status for error > review /var/zixi/broadcaster/log/ > reinstall multiple failures across multiple streams 10+ minutes post restart? > collect broadcaster logs > initiate rollback reinstall previous build from backup, restore config, restart service no clear cause and streams still offline > collect broadcaster logs, version details, and stream list and escalate to zixi support rollback procedure initiate rollback if streams are not recovering within the agreed threshold (recommend 10 minutes) rollback restores the previous broadcaster build and configuration stop the broadcaster service sudo systemctl stop zixi broadcaster reinstall the previous build use the previous installer run the installer to overwrite the new build restore the configuration backup copy the backed up config directory back to /var/zixi/broadcaster/config/ (linux) or \[install dir]\config\ (windows) restore the license file copy the backed up license to /var/zixi/broadcaster/conf/broadcaster lic start the broadcaster sudo systemctl start zixi broadcaster verify version is the previous build check help > about in the broadcaster ui notify zixi support open a support ticket describing what failed during the upgrade and that rollback was performed include the target version and the failure symptoms zixi support will advise on a safe path forward escalation checklist when escalating to zixi support after an upgrade failure, include as much of the following as possible broadcaster version before upgrade (e g 17 9 45910) broadcaster version after upgrade (e g 18 10 46622) host os and version (e g almalinux 9, ubuntu 22 04, windows server 2022) deployment type on premises / aws / zaas / other cloud list of affected streams/outputs with their current status and any error messages shown in zen master or the broadcaster ui provide zen master links where possible broadcaster logs post upgrade from /var/zixi/broadcaster/log/ (linux) or \[install dir]\log\ (windows) include at minimum the log file covering the restart and the 10 minutes following zen master event log exported for the period around the upgrade description of which section 3 checks failed and what was observed whether rollback was performed and what the outcome was time of the upgrade restart and time issues were first observed