A story about hiring the wrong person
As a young engineer before joining AgileEngine, I had a co-worker. Let’s call him Dave.
I met Dave when the company interviewed him for a Java development job. His technical interview with us focused on simple, direct questions to assess his knowledge of specific backend technologies, like Java and SQL. He did well, showing strong knowledge of the programming side of backend web development. The company hired him.
But after he came on board, Dave struggled. His managers delegated new web application features for him to implement, expecting him to own the outcomes and deliver. But he often didn’t know where to start his work or how to break it down into manageable tasks for himself. For the work to get done, he needed his specific tasks defined for him. Though he knew the tech, he just couldn’t solve problems on his own.
In his defense, the requirements for these features were not always crystal clear. These vague inputs made Dave’s job harder. Nevertheless, the job description mentioned the need for a person to own such efforts — including elaborating requirements — and such was the company’s expectation.
After a year of low productivity, Dave left the company.
The problem: The gap between technical knowledge and solving unstructured problems
Technology organizations frequently face this challenge. Many engineers intimately know the details of their chosen technologies but fail to solve unstructured problems that cover unfamiliar territory. Such difficulties might involve a bleeding-edge technology, a company-wide business flow, or a novel challenge without an existing pattern to apply.
To overcome these or similar difficulties, engineering teams need broader creative skills to innovate and tackle new problems.
The solution: Building general problem-solving skills
In my experience, solving complex real-world software engineering problems requires an ability to:
- Define or clarify stakeholders’ desired outcomes
- Identify root causes of difficulties
- Decompose a challenge (or solution) into smaller pieces that are easier to solve (or manage)
- Ideate solutions, both alone and while collaborating with other team members
- Plan and conduct quality assurance
These skills are really about system design in general and do not belong to any single technical domain. Engineers with these broad skills can tackle just about any analytical or engineering difficulty as long as they also have the requisite (and complementary) technical knowledge.
One more observation: while some hard problem-solving skills (e.g., root cause analysis) can be taught via courses or books, the ability described here also depends on things like temperament and aptitude. As with computer programming, a person typically either has a knack for it or doesn’t. Thus, one of the best ways to develop this capacity in your organization is to screen for it when hiring.
Our approach: Evaluate candidates for broad problem-solving during interviews
In remote software engineering engagement models with higher degrees of autonomy and initiative, like dedicated teams or managed projects, problem-solving is critical for success. For this reason, our hiring process involves an evaluation of general problem-solving skills in addition to specific technical knowledge.
At AgileEngine, extensive screening for problem-solving skills is mandatory for senior, lead, or higher engineering positions. We also include it for any seniority level when selecting experts for a dedicated team with a demanding problem-solving culture and a high sense of product ownership.
To evaluate these skills during the candidate’s interview, we present them with an open-ended design challenge that roughly corresponds to their technical specialty. We give them a broad, slightly ambiguous technical problem and ask them to explain how they would go about solving it.
Naturally, we have different expectations for the different seniority levels. We give each candidate a score of 0 to 4 based on the following criteria.
Skills expectations for a junior engineer
- Understands basic requirements and has the ability to ask questions to clarify when needed.
Skills expectations for a middle engineer
- May suggest small or mid-size feature estimates.
- Knows how to split tasks, especially for their tech stack.
- Has analytical thinking skills, breaking down issues to identify causes.
- Can come up with ideas that can be used as a starting point when tackling complex tasks.
Skill expectations for a senior engineer
- Performs discovery for a feature, asking questions and collecting data.
- Organizes the feature breakdown and provides estimations for both backend and frontend.
- Understands what to do if there are no QA engineers on the team.
- Can come up with a strategy to locate the source of the difficulty and can also mention different strategies for solving it.
- Can analyze and discuss trade-offs for proposed solutions.
Skill expectations for a lead engineer
- Has solid experience driving the development of new products from scratch.
- Drives communication with the client to collect functional and non-functional requirements and knows how to initiate and conduct the necessary discovery activities.
- Organizes the feature breakdown and provides estimations.
- Has awareness of the team’s strengths and capabilities.
- Considers time for automated testing and cares about future maintainability and scalability.
- Can diagnose and solve complex problems and help others to solve them.
- Can propose innovative solutions to challenging problems often requiring optimization or efficiency improvements.
- Uses systematic problem-solving methodologies (e.g., root cause analysis, hypothesis testing).
Our results: Correlation between general problem-solving excellence and passing our clients’ interview process
Since implementing the evaluation in our interviews, we have seen a dramatic increase in client satisfaction with the team members we have brought forward into our engagements. Our candidates who receive top scores for general problem-solving are up to three times more likely to pass the full interview process than those with low scores. The precise difference depends on the technology stack.
Here are the metrics broken down by tech stack or specialization.
In the hiring process for Java engineering roles, AgileEngine’s candidates with higher general problem-solving scores (2 or 3) were twice as likely to be accepted by clients as those with low scores (0).
React.js candidates with higher general problem-solving scores (2, 3, or 4) were roughly three times more likely to be accepted by clients than those with low scores (0 or 1).
Node.js candidates with higher general problem-solving scores (2) were roughly twice as likely to be accepted by clients as those with low scores (0 or 1).
QA engineering candidates with higher general problem-solving scores (3 or 4) were roughly 33% more likely to be accepted by clients than those with low scores (0, 1, or 2).
The evidence above speaks to the significance that companies attach to problem-solving skills when engaging engineering talent. Engineering teams often have to deal with challenges characterized by unclear or changing scope and complexity that goes beyond technology. By prioritizing candidates with strong problem-solving skills, companies can tackle such challenges faster and gain an edge over the competition.
Conclusion
In today’s increasingly competitive and quickly evolving world, you need people who can find innovative solutions to non-trivial challenges. Specific technical skills are often not enough for this — you must find engineers who can analyze, define, and decompose problems that haven’t been solved before.
AgileEngine has vast expertise in ensuring that new members of our clients’ remote teams have this capability. Our approach to candidate selection empowers our clients to create innovative solutions that win awards, get top-rated by the Play Store and App Store, and land top spots on G2 and Gartner. Talk to our experts and explore what positive impact this approach can have on your digital product.
