The Monday Morning the Database Went Dark

We got the call at about 8:30 AM today.

One of our clients — a construction company here in South Florida that handles large scale office building and residential construction projects — had close to 60 people trying to start their week: office staff, crews already on job sites, a few people working from home. And their core estimating / change order / billing software software wouldn't open. Not slow, not glitchy. Just unresponsive, for everyone, all at once. For privacy reasons, we are not disclosing the name of our client or their software vendor.

The First Call Went Nowhere

Before calling us, they'd already called their software vendor's support line. That call didn't go anywhere fast, for a few reasons that are worth knowing about:

Their software vendor is based out west, exactly two time zones behind. They have no early support coverage for anyone on the East Coast. And because it was a Monday, an automated message for the support line advised them that if they were calling on a Monday or after a holiday, callbacks could take as long as 24 hours due to everything that may have piled up over the weekend/holiday.

The recording did offer one piece of advice: restart everything, including all workstations and servers. A properly performed restart is normally safe, but recommending it without first understanding the environment is still risky. It can disconnect users from active sessions, interrupt running jobs, disrupt services, and complicate troubleshooting by erasing useful evidence of the original problem. It is the technical equivalent of restarting a car simply because a warning light appeared: the light may go away, but the underlying problem remains unknown and may return.

What Was Actually Wrong

We logged in remotely and found the real issue in a few minutes: two background services that the database software depends on had hung. Not the entire server and not the network but just two specific services that had quietly stopped responding.

We manually restarted those two services directly, they came up fine, and the software become online immediately. All the staff were back in their software before most of them finished their coffees.

That part was the easy fix. The part that actually matters is what happens next — someone still needs to find out *why* those services hung in the first place, and why nothing alerted anyone before people started logging in Monday morning. That's on the software vendor to dig into, and we've reached out to them to actually do it.

The Real Lesson Here Isn't About This One Vendor

This wasn't really a story about bad software. It's a story about what happens when you don't know the answers to a few questions *before* something breaks — not after.

If you're evaluating business software, here's what's actually worth asking the vendor about their support options, before you sign anything:

• Does their support actually cover your hours? If you're in the Eastern time zone and their team starts at 9 AM Pacific, you're out of luck for half a business day - at a minimum.
• Do they cover weekends, or does everything wait until Monday? If your business runs early mornings or weekends, find out now whether their support does too.
• What does after-hours support actually cost? Sometimes it's included. Sometimes it's a surprise ($$$) invoice.
• Are they okay with outside IT being involved? Some vendors are fine with us logging in to help diagnose things. Others push back hard on it, which just means everything has to route through them, every time.
• What about software updates? Are updates or upgrades to the program as easy as downloading a patch and a few clicks or does it require full involvement of vendor's staff, extended downtimes?
• Will you get billed for updates? Are software updates or upgrades covered by the software's yearly maintenance fees or will you get a bill every time the software needs to be updated, upgraded or simply patched?

Where We Actually Fit In

We don’t manage this particular client’s day-to-day application as that responsibility belongs to their software vendor. Our role is to ensure the servers, network, and supporting infrastructure are configured precisely to the vendor’s requirements. That removes the opportunity for the anyone to blame somebody else when a problem arises and prevents the familiar ping pong game of "its your fault, no it isn't, yes it is". And when something does go wrong at zero dark seven on a Monday, we’re glad to take the call, work with your team, the vendor -anybody- to get everything running again.

If you’ve ever been stuck on hold with a software vendor while your whole team sat around waiting, that’s exactly the kind of gap we’re here to close. We’re also available before you choose a software package or vendor, so we can ask the critical technical questions that are easy to overlook and help ensure the solution will actually work with your existing systems, security requirements, and day-to-day operations.. Start a conversation, or give us a call at (954) 274-9020.