Technology

Incident Response for Small SaaS Teams: A Practical Guide to Post Mortems

Outages happen to everyone. What matters is how calmly you handle the next one. Here is the lightweight process I use with small teams.

Incident Response for Small SaaS Teams: A Practical Guide to Post Mortems

Incident response for a small SaaS team does not need a 40 page policy or a dedicated on call rota. It needs a few agreed habits that stop panic turning a bad hour into a bad week. I have run production systems for 17 years, and the outages I remember most are the ones where nobody knew who was in charge.

If you are a handful of developers running a product with paying customers, this is how I would set things up. It is simple on purpose.

Decide what counts as an incident

Before anything breaks, agree what you are treating as an incident. Otherwise every wobble becomes a drama, or worse, a real problem gets shrugged off as "probably nothing".

I use three levels. Severity one means customers cannot use the product or data is at risk. Severity two means a major feature is broken or badly degraded. Severity three is everything annoying that can wait until morning.

Write the levels down

Put the definitions somewhere everyone can find them, with an example of each. When it is 11pm and you are not sure, a one page document beats a debate. Keep it short enough that people will actually read it.

Name one person in charge

The biggest failure I see is five developers all debugging the same thing in five different directions. Nobody talks to the customer, nobody checks what changed and two people accidentally undo each other's fix.

Pick one incident lead for each incident. Their job is not to fix the problem. Their job is to coordinate, make decisions and keep everyone else informed. In a team of three, that might be the founder, and that is fine.

Split the roles even in a tiny team

You can still separate the jobs. One person investigates, one person handles communication, and the lead keeps notes. If you only have two people, the lead does the notes and talks to customers while the other digs into the code.

Fix first, understand later

When something is on fire, the aim is to stop customers suffering, not to find the elegant root cause. Roll back the last release. Switch off the new feature flag. Restart the service. You can work out why once things are stable.

The first question I always ask is "what changed?". A deployment, a config tweak, a certificate expiring or a third party having a bad day accounts for most incidents. Good monitoring and alerting makes that question quick to answer, and a tidy deployment pipeline makes rolling back painless.

Talk to your customers early

Customers forgive outages. They do not forgive silence. A short, honest message saying "we know about it, we are working on it, next update in 30 minutes" calms people down more than you would believe.

Set up a simple status page before you need it. Post when you start investigating, when you find the cause and when it is fixed. Do not blame a supplier in public and do not promise a time you cannot keep.

Be straight about data

If personal data may have been affected, that is a different conversation, and in the UK you may have a legal duty to tell the ICO within 72 hours. Know that before it happens. My post on securing a SaaS without a security team covers the basics worth having in place.

Run a blameless post mortem

This is where the real value sits. Within a few days of a serious incident, get the team together for 30 minutes and write down what happened. Keep it blameless. If people fear being punished, they will hide things, and you will have the same outage again.

Focus on the system, not the person. If one developer pushed a bad change, the question is why your process let a bad change reach production, not why that developer made a mistake.

What to write down

  • A short timeline of what happened and when.
  • How you found out, and how long it took.
  • What the impact was on customers.
  • The root cause, and the contributing factors.
  • What went well, because there is always something.
  • Two or three actions with a named owner and a date.

Keep the actions few and real. A post mortem with fifteen action items is a wish list nobody will ever finish. Pick the ones that would genuinely stop a repeat and put them into the backlog like any other work.

Practise before it matters

Once or twice a year, rehearse. Pretend the database is down and walk through what you would do. Try restoring a backup to see if it actually works. My piece on disaster recovery for small SaaS goes deeper on that side, and it is the sort of job that always gets pushed back until it is too late.

If you would like help putting a sensible incident process in place, or a second opinion on how resilient your platform really is, that is something I do through my consulting services. It is far cheaper to sort out before the first big outage than after it.

More from the blog

Technology•6 min read

Monitoring and Alerting for Small SaaS Teams

Read more
Business•6 min read

How to Reduce SaaS Churn When You Are Bootstrapped

Read more
Travel•6 min read

NC500 in a Motorhome: Is It Worth It and How to Do It Properly

Read more