There was a planned renovation in the building but unexpected power outages for a few hours on select days. This caused the systems to be shutoff.
We have ups battery to bypass short electricity outages, but did not last long enough unfourtunately.
Our second problem incurred when a good number of our nodes were not guarded by ups even though they were connected to "controlled by master" connectors. This was unfourtunate and misleading. We will need to understand this behavour in the future. For now, we moved the systems to the connector specified "battery backup" and tested the reliability.
We still need to purchase a ups for the computational cluster.