The importance of doing reference checks (2/2)

This is part two of Jenny's article about the importance of reference checks. If you haven't read the first part yet, start here.

Last time we spoke about WHY you should do reference checks and what impact a bad hire can have on your organization. In this second part I’d like to share my personal experience as well as some outcomes that have recently been discussed within the Point Nine family around the HOW.

General thoughts on the HOW:

1. If you do reference checks, make sure they are one of the last steps of your recruitment process. Because if you do them right (I’ll explain further what that means) it will cost you time. You want to make sure that time is invested in the right candidate.
2. The number of reference checks people do varies greatly between 1 to 15 checks per person. You want to find inconsistencies within the feedback you receive about the person. Depending on how senior the candidate is, ask for more references. A good start is 2 references for junior/entry roles that you want to increase to 3 to 5 references for middle management positions and 6 to 10 references for a senior leadership position.
3. Make sure you have a mix of suggested references by the candidate as well some that you happen to know or you proactively approach to give you feedback. A good way to start is with the suggested ones. You can ask those guys who else they find you should talk to. Make sure you still top that up with your own research e.g. via LinkedIn or your personal network.

Now how to do them well?!

• Have questions prepared. Make sure you know what to ask rather just go for the “So, tell me something about John” kind of question. Ask specific questions about the candidate. Remember, you want to find inconsistencies! Don’t be okay with foggy answers, worst case rephrase the question.
• Try to establish a relationship with the reference you are talking to. Do a video call if possible and try to avoid written references. Take your 2-5 minutes small talk time at the beginning. Why? It’s much harder to lie into your face when I like you :)
• Don’t only ask direct peers. It’s much more interesting to talk to people e.g. that got fired by that person (they are usually much more honest and open) or in general someone that has worked below him and got directly impacted by that person. A good mix of people above, around and below works quite well!
• Obviously try to avoid asking someone at the current company the person works unless he got recommended by the candidate.
• A good way to challenge the integrity upfront is to ask the following question once you’ve received the suggested references: “Would you mind If I talk to Peter, Fran and George as well”? Watch the reaction closely!
• Always remember: You hire someone for his strengths not lack of weaknesses so tailor your questions about the strengths you are looking for and make sure that you weigh the feedback you receive according to what you really need. Example: Reference says: “Steve, in his role as Head of Sales, was too pushy in his general approach and sometimes went too fast for the Executive Team and the organization, so it was really hard to follow”. This is generally not the best feedback, but if your company is currently trying to attack a market, someone like Steve might be just the right fit for this period of time ;)

Here are some random questions that might be helpful for your day to day reference check. I usually use a mix of those and tailored ones to the specific candidate and role. Remember we’re trying to find inconsistencies, so use a good mix of questions for every reference check.

Some general questions that I find helpful:

• What is your relationship with Peter like?
• What was it like to work with Peter?
• What was Peter’s management style like?
• Can you describe a tough/very challenging moment Peter was in and how he has managed to get ouf it?
• If you think about the time you and Peter have worked together, what’s the first memory that pops up your mind?
• Would you say Peter is more a team player or does he excel more when he’s on his own? Can you give a specific example for it?
• Would you describe Peter as a hands on person?
• Do you regret that Peter has left your company? Why?
• Should I hire Peter? Why?

There is plenty of reading out there with good stuff on that topic. One post that I can highly recommend is Mark Suster’s post “How to make better reference calls”.

What are your thoughts? What are your favorite questions that you use in reference calls to cut through the bullshit politeness?

The importance of doing reference checks (1(2)

This is a guest post by Jenny Buch, who recently joined us as a Talent Manager. It's the first in a series of two posts. The second one will appear here soon. 

To follow up on the recently posted interview with Netflix CEO Reed Hastings, I’d love to share my experience about reference checks with you.

So, many of you probably made the experience of hiring someone that you would have stated as “a really promising candidate” upfront. But after four months into the job it turns out that the hire was actually a total fail, that your staff is thinking you’re an idiot for bringing him on board (even if they don’t tell you) and that you now have to pay the debts by firing that person and start a whole new time consuming hiring process again to reduce the mess you’ve just done to your organization.

Well, even though things like these sometimes just happen and can have many reasons, there are ways to dramatically reduce the likelihood. One of them is to have a strong hiring process in place with its most important asset, you can guess it – reference checks!

Or, to say it with Christoph’s words “In God we trust, all others bring references”

Reference checks have been quite common in the US and most of the English speaking countries ever since but are still fairly new to many of the European countries. This is due to cultural differences and different common practice that was established in each country years ago. So why changing that good old practice and do references checks? Here is why:

There is only so much you can get out of a certificate, a CV or an interview. Your goal is to get to know your new “promising candidate” as best as you can within a short period of time to make sure you don’t mess things up! Today, and especially in the startup scene, there’s a different need for different skills than there was when “written references” were the way you did it. Here are some classics of the “must have skills” for any kind of candidate that can only really be proved by a good reference check:

• Interpersonal skills: Interpersonal skills for any kind of role matter much more these days than they did in the past (or to state it correct: people are more aware of them, they’ve always mattered). The interview situation is not the best way to find out whether or not your candidate is actually a great fit since some candidates are extremely good at selling themselves during an interview.
• Integrity: Today, an intern can become the CFO. Maybe the company sucks and he only became CFO because he went to the same university as one of the founders. As Ben Horowitz likes to say in his amazing book “The Hard Thing about Hard Things”. “There are two kinds of companies in this world. One where matters what you do and one where matters who you are. You can either be the first one or suck.” You want to make sure that his previous company was out of the first category and that besides his great university connections, he actually was the best candidate for the role and that’s why he got it - because he knows shit, works hard and is an awesome guy. I’m not saying that hiring from your university environment is bad, but it shouldn’t be the only reason for a hire - which it is sadly in many startups these days.
• Right kind of ambition: You are looking for people with the right kind of ambition. So people that love your idea, bring a “get shit done mentality to work” and that thrive to make your company successful. As a side effect it will help them grow their career - not the other way around.
• Right kind of person: You are not looking for “the Facebook Head of Sales” or the “CMO from Google”. Even though those guys do an amazing job at their current companies, every company is different and every time in every company is different. You need to find the right candidate for your company at this time. So one of your challenges is to make sure that your candidate has the right skillset for YOUR COMPANY.

I guess I don’t have to tell you how big the impact of a bad hire can be for your organisation. Simply do the math. Really do it! Sit down and calculate how much time and money it takes you to get rid of the wrong hire and find a new person and tell the people that you are sorry rather than using this time to talk to 5 people for 10 minutes upfront. If you do that math correctly you’ll figure out that doing reference checks is going to be the easy, cheap and most efficient way for busy startups to get the right people on their rocket ship!

Cohort Analysis: A (practical) Q&A [Guest Post]

My colleague Nicolas wrote a great guide with tips and tricks on how to do cohort analyses which I'd like to share with the readers of this blog. Thanks, Nicolas, for allowing me to guest publish it here. Without further ado, here it is!

- - - - - - - - - -

At Point Nine we believe that the only way to get a real sense of user retention and customer lifetime is doing a proper cohort analysis. Much has been said and written about them and Christoph has a published a great template and guide on the topic if the concept is new to you.

With this Q&A I want to focus on some of the more practical questions that might arise when you are actually implementing a cohort analysis for your startup. After close to two years of working with SaaS companies and doing numerous of these analysis I have learned that in most cases there is no perfect step-by-step procedure. But although you will always have to do some customisation for a cohort analysis to perfectly fit your business, there are a handful of questions and pitfalls that I have seen over again and again and want to share so that you can avoid them.

Now let's get into it!

Q: Which users should I include in the base number of the cohort?

There are two parts to the answer as it depends on what you want to measure. If you want to find out your overall user retention and have a free plan, then you should include all signups of a specific month.

However if you are trying to calculate your customer lifetime value, you should only look at the number of paid conversions. I only count an account as a paid one when the user has or will be charged for a period. So if you offer a 30-day free trial for example, wait to see if the user converts into a paying plan before you include him in the cohort. This way the numbers won't be biased with users that actually never paid for your service.

If possible without too much effort, you should also try to eliminate all 'buddy plans' that you have given to friends, your team or investors. If they are not paying, they are not representative for the real cohorts.

Q: How do I treat churn within the first / base month?

There are different approaches here, but in my view taking churn within the first month into account is the most accurate representation of reality. That means that in your first month the retention could be less than 100%, if people cancel their paid subscription within that month. It would look something like this:

I do this because I don't want the analysis to exaggerate churn in the second month and understate it in the first / base month. After all the reasons for churning in the first 1-4 weeks could be very different than after 5-8 weeks.

Q: Should I treat team and individual accounts differently?

If you are at a very early stage or sell mostly (90%+) individual plans it is probably sufficient to mix them all in the same analysis. But when team plans make up a significant part of your paid accounts, or your product has a very different user experience when a whole team uses it, you should probably look at both type of accounts separately.

Findings could include that team accounts are a lot more active, churn less and see a lower drop-off in the first month than individual plans. Or not. :)

Q: What about annual vs. monthly plans?

Again, if you are focusing on how active your users are over their lifetime it is OK to mix both plans. If you just want to see how many of the people that signed up still come back after X months, no need to split hairs.

If you are however focused on churn, you should only look at paid accounts that could have churned in that month. This is one of the 9 Worst Practices in SaaS Metrics and means that you should exclude all annual plans that are not expiring in the respective month. Including these in the denominator would otherwise skew churn numbers.

Q: Now that I have it, what can I take away from it?

The two most obvious take-aways are depicted in this (KISSmetrics) retention grid. Note that this is a most likely an analysis for a mobile app and the numbers for your SaaS solution should be significantly higher:

(click for larger version)

Moving horizontally you can see how the retention of a cohort decreases over the users lifetime. Interesting here is where the highest drop-offs occur and whether the numbers stabilise after a few months.

Vertically, you can (ideally) see how the retention of your cohorts change over the product lifetime. Assuming you are not twiddling your thumbs while catching up with House of Cards or sipping Mai Tai’s at the beach once your product launches, you should see an improvement in user retention with younger cohorts as the product improves. If this is not the case, you should consider whether the hypotheses or features you are working on are the right focus.

Most importantly though, this data will be the basis to give you a sense for your customer lifetime value (CLTV). If you take the weighed retention data for the 6th or ideally 12th month and extrapolate it, you will get an approximation for the average lifetime of your customers. Multiplying this with the average revenue per account (ARPA) or respective plan that you are looking at (e.g individual / team) it will give you your CLTV. This number is really the quint essence of the cohort analysis, as it gives you an idea about how profitable your business model is (=how much more money are you making with than what you are paying to acquire him). Subsequently it will also tell you the highest price you can spend on customer acquisition to grow profitably. It is important to note here that although super valuable, especially in the early stages of a startup this number will always be an estimation and most likely not 100% accurate. So keep in mind to continually track and fine-tune your CLTV calculations.

And one last thing: If you have accounted only for paid subscriptions as defined at the first question above, then the base rates of each month will also give you the most accurate number for paid customer growth and subsequently MRR growth. Two charts you will want to have at hand when talking to investors.

Q: Is that it?

For this post, yup! If you want to learn more about cohort analysis or SaaS Metrics, I would strongly suggest to check out Christoph’s and David Skok’s blog. And in case you have any questions on the above or something is unclear, feel free to ask away in the comments or send me a mail and I will do my best to answer you (or forward the hard questions to Christoph). ;)

- - - - - - - - - -

Like this post? Make sure you add Nicolas' blog to your reading list.