A Research-driven Recruitment Story

As you may know, if you’ve already read our increasing diversity in recruitment post, the way we hire at Bytemark is different. This is owing to our meticulously-designed, anonymous, decentralised recruitment process.

We’ve previously outlined some of the reasons why we wanted to make this change to lead the way to a fairer, people-focused local tech community. But we wanted to take the time to share the research behind it all. How we got from the idea of inclusive recruitment to the system we use today, which has allowed us to process 200 applications and successfully hire 14 members of staff. We hope it can inspire others to think about the way they hire and make and share their own improvements.

Our problem with traditional recruiting

For the first decade or so of Bytemark’s history,  our co-founder (Matthew Bloch) took on the role of recruiting staff. He’d put a few adverts out, collect some CVs, schedule interviews, everyone would dress up, and then he’d make a decision about who he felt would be best for the role.

Over time, this method became increasingly problematic. The entire recruitment workload was in the hands of one person. In Matthew’s own words:

I became increasingly aware of how inappropriate biases had influenced our previous process, and how slow we were to spot them. For instance until a few years ago, we used to grill prospective system administrators on the particulars of their home computer network – how many did they have, how complex was the routing and so on. We figured that because we had staff who already built overly complex home networks, this was a good signifier for the kind of sysadmin we wanted. Instead of course, we were testing for all kinds of privilege, a certain size of house, a set of interests we’d shared in college, everything but the work we wanted them to do.

— Matthew Bloch, Co-founder

In sum, we knew that we were missing many great candidates because of our “network-based” process. We treated existing candidates inconsistently and gave them a poor (and inaccurate) impression of our company.

These local concerns about bias were reinforced by bigger, more global problems in the tech community. Stories of harassment were gaining publicity, highlighting how toxic the environment had become in some circles.

At the same time, new awarenesses surfaced about problems with diversity in tech, the challenges for minority groups to stay in the industry, and the issues surrounding environments where there was minimal to no processes for hiring, e.g. Heddleston’s null process.

These problems necessitated a rethinking of how we do things at Bytemark. We needed to build a better team through better hiring policies.

A New Recruitment Process

Doing some research into the subject of diversity, Matthew stumbled upon an innovative solution from an entirely different field. Anonymity.

A 2000 paper [1] by Claudia Goldin and Cecilia Rouse, two economists, had highlighted the positive effect of an anonymous audition process on gender diversity in US orchestras during the 70s and 80s. The authors showed that when auditions were held behind a stage screen, women were 50% more likely to advance to the second stage of the process compared to when they were visible to the panel.

Taking the idea of an anonymous audition as his cue, Matthew pieced together a technological version of the stage ‘screen’, and in early 2015, careers.bytemark.co.uk was born.

The First Anonymous Applications

With a new process comes new issues. We loved trialling the process but found some teething problems. Specifically, we found that while reducing biases in some areas, we introduced new ones in others. Because candidates had no frame of reference for our process, they started to make (often incorrect) inferences about what we were up to, and how they were being evaluated. A common source of anxiety was about staying anonymous:

I’m always thinking about whether or not I’m accidentally giving myself away.
Answering reveals too much about my background, but didn’t want to be vague either. These questions didn’t let me shine or share relevant things about myself.

– Successful Careers applicants at Bytemark

[It] was initially frustrating to not be able to tell you things about who I am and what I’ve done. But it’s great that it’s a level playing field. By the final interview I was liking the process so much that I was reluctant to share my CV and de-anonymize myself.

– Successful Careers applicant at Bytemark

We also encountered many obstacles which were situational or involved miscommunication of some form. For example, when our (new and not-yet-added-to-our-website) HR Manager, Pam Hinds, responded to candidates by text messages including her name, several of these candidates started to wonder who the mystery ‘Pam’ was — an anonymous person? an acronym (perhaps ‘Personal Application Manager’)? or possibly even a friendly chatbot. These are the sorts of findings you can never anticipate, but you must discover and address!

We were filling in the gaps, too. Without research, it’s difficult to know if we’re attracting more diverse candidates. Most importantly, our guesses about the psychological barriers our candidates faced were just that, guesses. We needed to find out what these realities were, how they were affecting people, and then take action to address the unmet needs.

A Research-Driven Recruitment Update

By slowing down and vetting product & design decisions more carefully, you can uncover gaps and weaknesses that would breach your core users’ trust, and avoid costly problems down the line.

— Eric Meyer, Design for Real Life

It can sometimes be challenging to know where to start with a project that may potentially have several important issues that need addressing. However, with a carefully-reasoned approach, some practical empathy, and a supportive team, it becomes clearer not only to see what’s most pressing but how to implement an effective solution for each aspect of the problem.

With this in mind, we set out to talk to a wide range of people to find out how we could further improve our process.

Successful Applicants & Company Stakeholders

Our Research Manager, Kim Witten, led a 6-week development cycle. The research included:

• 12+ hours of interviews
  • six with anonymously-hired candidates
  • four stakeholder interviews
• 3-hour data workshop
• More than half the company was involved in the process in one way or another

What did we learn?

Our research team uncovered many psychological barriers that we’d unintentionally introduced through anonymous recruiting. However, our biggest challenge by far was setting expectations and communicating well with candidates.

This led to us revisiting the entire process and resulted in an extensive re-write of our current recruitment guidance for both candidates and staff.

Additionally, the research alerted us to several bugs, feature requests and other issues needing to be addressed.

In our 3-hour workshop, we prioritised and sorted these findings. This resulted in over 80 separate issues. All of which we tracked, assigned and completed during the final two weeks of the development cycle. This process was immensely valuable to our development team:

Design and development are so much easier when you understand people’s needs. Decisions make themselves, instead of an endless back and forth of second guessing. What would have been features distill themselves into tiny text changes. You are not implementing notifications, you are making sure people don’t get ignored.

– Steve Urmston, Product Manager

We also learned of many encouraging insights about our process, and our successful candidates too! Our anonymously-hired staff were unanimously positive about the anonymous hiring concept, they appreciated the focus on skills first, and they were enthusiastic about research that would improve the process based on their feedback.

However, relying only on the word of successful candidates is problematic. They got the job, so it follows that they would view the hiring process positively. Our results were almost certainly be influenced by survivorship bias.

Potential Applicants

We wanted to go back to first principles and understand the obstacles faced by all applicants, regardless of their success. So, we continued our research mission by speaking to a wide pool of people at a number of conferences, including AlterConf and PHP Yorkshire, to get a better understanding of the general issues people face when looking for a job.

Our discussions focused on:

• Which aspects of job hunting do potential applicants find the most difficult (and also the easiest)?
• Which aspects of job hunting are the most important to them (and also the least important)?

Kim, and our Placement Year Junior UX developer, Hannah, then conducted usability testing. They met with people to go through the existing careers site and answer the following questions:

• What are people’s initial impressions of the Careers site?
• What do they want to find immediately?
• Can they find it easily?

What did we learn?

Our interviews allowed us to validate some of the steps in our process and also gain a better understanding of the exact information we need to provide for people before they started their application.

The usability testing then helped us to focus on improving the aspects that potential applicants found the hardest and were most important to them, while also addressing current usability issues.

Our New Careers Website

We took all of the information we had gathered and used it to overhaul our careers website. We’ve gone from this:

To this:

Many of the changes appear cosmetic, but they do make a real difference to the applicant experience.

6 Key Changes to the Website

1. We’ve redesigned the home page so the job openings are the first thing people see.
2. Pictures of current staff members were added to show applicants the friendly faces of their interview panel.
3. We introduced a sidebar to check the application’s progress.
4. Information was re-jigged on the application and review pages. This made submit buttons clearer and the important information more visible.
5. Similar changes were made to the handle (anonymised username) page to make the button to generate a random handle more obvious.
6. A less visible update, we also added and rewrote a lot of the copy. We organised the guidance into a table of contents, making expectations clear at each step along the way.

Together, we think these updates make the website easier to use and reduce applicant stress using an unfamiliar process, Less wondering about whether or not your username is ok or if what you wear to the final interview matters (it doesn’t); we want you to be able to focus on making a great impression with as little stress as possible!

Our work doesn’t stop here

---

[1] Goldin, Claudia and Cecilia Rouse. 2000. “Orchestrating Impartiality: The Impact of “Blind” Auditions on Female Musicians.” American Economic Review, 90(4): 715-741.

---