Jacob Kaplan-Moss

Hangar’s Dumb Security Questionnaire

Update, August 2024: In January 2021, I published this Dumb Security Questionnaire that I developed at Hangar on their tech blog. That blog has since gone offline, so I’m republishing it here. The original was released into the public domain, so I’ve done likewise.

Last year, one of our startups needed to buy a SaaS product (case management and workflow software). There were several promising vendors, all with products that looked impressive. Technically, all had the features and APIs we were looking for. However, we had security concerns. We planned on storing extremely sensitive data in this tool, and wanted to understand their security posture before we selected a vendor.

This a common problem; you’ve probably ran into it yourself. With SaaS software, how do you verify its security? As an industry, our answers are … poor. We have various certifications (PCI, HIPAA/HITECH, FedRAMP, etc), but all too often these are box-ticking exercises with no real security value – just ask SolarWinds.

So, security teams often end up sending potential vendors a bunch of questions – what Latacora calls a Dumb Security Questionnaire. As the title implies, the practice … isn’t great. Most DSQs are full of questions with no security value (which explains why most security teams don’t actually read the responses!). But unfortunately, they’re also kinda the best we can do. Huge companies might be able to afford real human-powered security audits and convince vendors to allow them, but small startups like ours don’t have a better option than relying on DSQs.

As Latacora points out, it turns out there isn’t really a ton that you need to ask. As a startup, we’re not looking for our vendor to be Fort Knox; we just need to know that storing data with them is at least as safe as storing it internally. There are a limited number of likely ways that our vendor might get owned; we just need to ask about those few basic things. Latacora ends their post with this:

Someone — and I am not volunteering — should write the DSQ that just nails these basic things. 10 questions, no diagrams.

Well, we tried; here it is.

Who this is for

This questionnaire is designed for smaller, probably early-stage companies who need to evaluate a vendor. It makes some assumptions:

  • You have a security person (or team) who can evaluate the answers and is part of the decision-making process. If nobody’s going to read the answers, don’t waste your vendor’s time.
  • Your answers to these questions are already pretty ok. If not, stop wasting time evaluating vendors. Just buy the thing and use the time you save to get your own house in order.
  • This is for some SaaS product that’ll have particularly high security impact - i.e. a breach of the vendor would be a major, potentially company-ending event. If you’re just trying to decide which ticket tracking system to use, again: just buy one and move on.
  • You have no particular compliance requirements – so you’re looking purely evaluate security posture, not compliance.

Hangar’s Dumb Security Questionnaire

(1) Please answer these questions:

  1. If you use a cloud provider (GCP/AWS/Azure/etc):

    • describe how credentials are provisioned, managed, and stored.
    • If an attacker gained access to an individual developer’s cloud credentials, what actions could that attacker perform, and how would you detect and respond to the breach?

  2. If you don’t use a cloud provider: why not?

  3. Describe how staff authenticate to company services (e.g. servers, email, SaaS products), particularly highlighting your use of password managers, 2FA, and SSO.

  4. What development practices do you use to protect against the OWASP Top 10?

  5. Describe the steps a developer or operations person takes to push new code to production.

  6. Have you had any security breaches in the last two years?

    • If yes: please explain the breach, and provide copies of any postmortem/root cause analysis/after-action reports.

(2) And, please provide as many of the following as possible:

  1. A recent (last 12 months) penetration test report, with information on what follow-ups/remediation steps were taken. Reports by an external firm are preferable; internal tests are OK. A redacted summary is fine as long as it includes basic vulnerability descriptions and severity levels.

  2. Documentation on your organization’s Secure Development Lifecycle, or similar, if you have them. If not, please summarize the steps you take to help engineers write secure code.

  3. A copy of any security training material provided to staff, particularly software developers. A summary/description of classes and content is fine.

  4. A link to your Vulnerability Disclosure Policy and/or bug bounty program. If you don’t have either: please explain.

(10 questions, no diagrams :)

License

This questionnaire is released to the public domain. I’d appreciate it if you credit Hangar if you release a derivative of this, but do what you like.