Old battery's probably swelled up and that's why it's getting stuck. Don't be shy about removing the entire top cover and front of the frame, if needed.
For mission critical business systems, UPSes are a critical necessity as a single brown out, power surge or power outage could easily cost thousands of dollars worth of lost time, lost data and ...
I have an APC XS 1500VA unit that just finally failed on me (batteries crapping out). It's been quite useful, but power here has been less-than-good, mostly due to me tripping out a breaker roughly ...