Technology work is very progressive. Rapid progress drives the industry to constantly learn new ways to meet objectives and innovate. A ubiquitous theme is that we keep up to avoid catching up. Why, then, does it seem that interviewing to measure technical aptitude has neither kept up nor caught up?
A while back I was connected with Amanda Richardson, CEO at CoderPad, through the great team at Mainsail Partners. Amanda has shared objective advice on several occasions since then. She recently dropped a LinkedIn article about the interview process that got me thinking about that topic myself. I think Amanda is right, interviewing does suck. I spoke to a few people about it and came away with 3 angles on technical interviews.
1. The Candidate
The first perspective is from me acting in the role of a candidate. The best technical interview I’ve had was here at SentryOne way back in 2006. There were questions, but they were designed to solicit conversation surrounding a technology or pattern. Further, I left the building with a problem to solve. I was given real time (2 days) to solve it, and left to solve it autonomously. I presented my solution(s) in a code review. This experience replicated what the job would actually be like. The technical portion of that interview was not an interview at all. It was more of a trial, and I loved it.
For highly technical roles, in what setting do people create the most value? That is a difficult question to answer, but I know where it generally does not happen. It does not happen sitting across from one or more people throwing trivia questions out until they “stump-the-chump.” It does not happen in awkward live auditions where candidates whiteboard source code for manufactured problems in front of an eagerly critical audience. These scenarios are not reflective of realistic environments or conditions. They also do little to determine technical aptitude or identify the behaviors that activate technical aptitude toward individual and team performance.
In today’s world, technical people work to solve many types of problems. There are multitude tools available to help find the best solution from a pool of nearly infinite possible solutions. The old trivia and public audition technical interview methods do not reflect reality in the modern environment. If I could have a better experience 14 years ago, shouldn’t we be able to expect a better experience now?
It took some time to prepare this article. I didn’t want my perspective to be stale. A lot can change in 14 years, after all. To prepare, I asked friends from three different companies to help me understand, and potentially engage with, the technical portion of their interview processes. I did this to gain understanding, not to pick on any of these friends or their companies. For that reason, I’m only revealing industry verticals and a basic description of how technical screening is approached for each.
- Big Finance
The technical screen that would be used for candidates working on a data warehousing team was provided. In this case, it comes very early in the staffing process. Recruiters provide an online assessment consisting of 20 multiple choice questions. It was timed at 30 minutes. The questions were all related to business intelligence, data warehousing, ETL, and reporting concepts. - Big Tech
I wasn’t able to engage directly in the technical portion for this one. This is because there isn’t a separate technical screen. They engage in behavioral interviewing, and technical discussions branch from the stories candidates share when answering behavioral questions.Of the three, this is the most similar to what I would appreciate most as a candidate. At the same time, I believe that it could be subject to moods that change with the ebb and flow of conversation. There could also be entire important subjects that never get discussed at all.
- Big Retail
A contact working in tech for a large retailer shared this process with me. Similar to the big tech firm, this process has no separate technical screen. They focus, instead, on identifying alignment with vision and purpose. They also use a “hire fast, fire fast” mentality. If they’ve made a poor hiring decision, they want it identified and corrected as quickly as possible. Any sort of technical ineptitude is discovered and resolved, one way or another, within 90 days of the start date.
In reviewing the technical vetting processes commonly used by these three friends, it became clear to me that this is still an area where a candidate will not know what to expect. One example is using the simplest form of the “trivia” style. A second is taking what they can get as it is revealed in behavioral conversations. A third foregoes the screen completely, opting for an on-the-job extended trial instead.
2. The Interviewer
This perspective comes from Richard Taylor, Sr. VP of Engineering at SentryOne. Richard heads our Platform Engineering department consisting of software developers, cloud engineers, DevOps and automation engineers, data scientists, and data engineers. The diversity of the platform engineering group gives Richard the chance to conduct interviews for many different technical disciplines.
Richard has evolved the technical interview beyond what I experienced at SentryOne years ago, and I believe he has taken it in a great direction.
Richard’s process involves four steps:
The pre-screen with HR is used to determine if the candidate meets basic requirements. It also allows our recruiters and the candidate to get a feel for culture, purpose, and mission. If both sides agree that we may have a fit, then Richard plans the next step.
The screen with Richard or the hiring manager is similar to the HR screen, but this one provides a closer look at the role being filled, the candidate’s relevant work history, and—most importantly—their personality. Richard asserts that this screening helps short circuit the process for any candidates who, for whatever reason, may not mesh well with the rest of the team. If a personality might introduce discord or negatively impact the team dynamic, there is little point to continuing the process regardless of how technically brilliant the candidate may or may not be. Richard puts the team level mission, culture, and performance first.
The third step is the technical assessment. Just as I had experienced, the candidate is provided with several problems to solve “open book” style. They are left to their own devices to solve them. The solutions a candidate delivers are not reviewed as right or wrong. They are reviewed with the goal of gaining an understanding of how the candidate thinks and solves problems. Richard and team are looking for critical thinking, sound logic, and the ability to troubleshoot issues. The only true failure for this assessment would be blatant plagiarism.
The fourth step is a more lengthy on-site interview with Richard and the team. Here, any questions from the technical assessment can be addressed and the team and candidate can dive deeper into determining the cultural and personality fit.
Of these four steps, Richard cites the technical assessment as the lower priority. While all four steps are important, he relies on the technical assessment as more of a measurement for where to place the candidate in terms of seniority and project work. It is also used to determine a path for on-boarding and training. Richard lends more weight to the other three steps in deciding whether to extend an offer.
This type of process represents a style that is becoming more prevalent as technology functions in business mature. We’re seeing more authenticity and servitude from technology leaders and less reliance on heroic or maverick individuals.
3. The Recruiter
This perspective comes from Heather Del Bene, Talent Acquisition Manager at SentryOne. Heather is experienced in placing highly technical roles, and her perspective on technical interviews in uniquely valuable among these three. Rarely does a candidate or interviewer gain a true understanding of how technical screens impact goals from the viewpoint of recruiting.
Heather told me about a time when earlier forms of technical screens made serving her client very difficult. This screen was a variant of the audition style of technical screen combined with elements of the trivia style. Candidates were set up with a timed assessment, given no access to reference material, and placed in a controlled environment alone. Further, the assessment was designed to be extremely difficult from beginning to end.
Heather was sending candidates to interview who were top performers in every current and prior role. Somehow, all of these candidates were failing the assessment. There was no consideration of how the problems were approached, and no discussion of why certain answers were selected or coded. The assessment was strictly pass/fail. Pass and move in; fail and move on.
Heather explained that this client gained a reputation through this process. Eventually the anxiety inducing assessment that seemed engineered to induce failure was discontinued. By the time it was retired, though, the hiring firm was already having a difficult time appealing to local candidates. Very few engineers were interested in interviewing for the company with the impossible technical assessment.
From the perspective of recruiting, poorly run technical screens directly impact your ability to meet the goals of your client. If the technical interview process leaves a memorable scar on candidates, it may even damage the firm’s reputation. This type of reputation damage increases the time it takes to fill technical roles. If the firm’s business is technical in nature, it could also impact their ability to reach these candidates as potential customers.
Summary
Our first perspective—that of a candidate—had me recanting an experience of my own from long ago. I chose that as a relevant story, because it does prove that some places were getting it right, even back then.
We’ve also heard from an engineering executive in Richard. He provides insight into how interviews that involve technical screens are changing for the better, and much of it has to do with how leadership roles in technology have matured along with technology work itself.
Finally, Heather shared her perspective as a talent manager. She helped illustrate how damaging negative experiences with technical interviews can be. They can be problematic for the recruiter, the candidates, and the hiring company – causing real and lasting impact to reputation.
Technical interviews should evolve to be more consistent and reflective of what the work entails. This could mean they take longer, or involve help from collaborative tools like CoderPad. It could also mean taking a critical look at your interview processes for technical roles in general. Some companies have been doing it well for years, but many are still missing out on potentially great candidates due to this aspect of interviewing for technical roles.
