Wed 12 Dec 2007
Periodically I read articles claiming that there is a programmer shortage in the US. Bill Gates made that claim earlier this year. A quick search on Google turned up similar articles going back to 1998. Do we have a problem with too few programmers in this country? I contend that we do not.
Certainly shortages arise in specific specialties as some technologies become more popular. Likewise, some geographic areas may see more competition for programmers than others. As lovely as it is, not all of us want to live in California. These articles are talking about large-scale, industry wide shortages and not the kind that can be explained by local factors. So what’s going on?
I believe that there are a number of problems at fault.
Programmer Shortage as Means of Lowering Programmer Pay
When Mr. Gates decries the lack of available programmers, he offers the solution that we should increase the number of H-1b visas. Indeed, if there truly is a shortage then this seems like a reasonable solution. However, this is more likely a ploy to address the true problem that these companies aren’t willing to admit: there aren’t enough programmers available at the salary they are willing to pay.
I worked at a company that would post positions that had ridiculous job requirements that almost no one could meet. They would then use the unfilled positions as justification to bring in programmers who were on H-1b visas that were, frankly, no more qualified than the American programmers that were rejected. The ones I knew were paid far less than their American counterparts.
This is a peripheral issue and not what I deem to be the problem at most companies.
What is a Qualified Programmer?
The trouble is that many companies don’t know how to identify a qualified programmer. I say that because their job postings frequently contain combinations of specific technical abilities stated in years of experience. In a previous blog, The Mythical Year of Experience, I discussed why that approach is a failure.
Even without relying too much on years of experience, most companies put too much emphasis on experience with a specific combination of technologies. This includes things like Oracle, Struts, .NET, XML, etc. I consider these to be extrinsic abilities. Knowledge of these things must be specifically learned and aren’t generally applicable to other areas. In contrast, intrinsic abilities are more portable and can be applied in many areas. These include problem solving, communication, and the ability to learn quickly.
I’m not saying that you should hire people who are totally unfamiliar with the technologies you are using, just that this need is too often overstated and the value of the intrinsic abilities is rarely weighed.
Let’s take XML as an example. Several years ago, I worked on a project that processed mortgage applications. We decided to use XML to store the documents while they were being processed. Those of us who designed the document structure and laid the fundamental access patterns needed a thorough knowledge of XML. Once that was finished, I was able to show my team members how to access fields very quickly.
If we had an opening on this team, many companies would make the mistake of requiring a couple years of experience in XML since it is so important to the project. A good programmer, who learns quickly, would be able to get up to speed on our use of XML in no more than a day.
To illustrate the importance of these intrinsic abilities, let’s look at another project I worked on. This was a C++ project where we had a requirement to run on several platforms. We ran into portability issues with the C++ code and decided we would be better off if we rewrote the system in Java. Though none of us had experience with Java, we quickly learned the language and proceeded with only a few minor hiccups. Our intrinsic abilities as great programmers allowed us to quickly adapt to this change.
The one constant in technology is that it is going to change. Hiring people with a greater focus on their intrinsic abilities will create a stronger work force that is better able to adapt to change. This will help your company avoid the short-term programmer shortages created by these shifting technologies.
How Do You Recognize a Qualified Programmer?
You can also tell how much trouble companies have with this by looking at their interview processes and the questions they ask. They are as likely to hire a truly effective programmer using their approach as if they had candidates randomly draw “Hire” or “Don’t Hire” cards from a deck. After some of the interviews I’ve been through, that approach would have been more satisfying and more respectful of my time.
Most companies ask the same old questions about your previous experience. You spend your time describing what the project did, not what you did. Any member of your team is likely to be able to answer these questions just as effectively, regardless of how little they actually contributed.
Some companies ask ridiculous questions with no specific answer, like “How many manhole covers are there?” They then look for candidates who try to reason out an answer. Or they might give you some kind of problem to solve about weighing balls or crossing a bridge. Tell me, what analog does this have to your daily programming tasks? Sure, problem solving is important but something a little more closely related to the programming domain might be nice. These problems too often rely on a flash of insight, which may be difficult to come by when standing in front of strangers with a potential job on the line.
Some companies ask questions that are more pertinent to programming, but get off in the weeds on very detailed algorithmic knowledge, like “How do your reverse an array without a temporary variable?” Or “How do you detect a loop in a graph?” These are good tests of your memory if you’ve been exposed to them before, but you will probably not puzzle out a great solution during an interview. Certainly, you could look up the answer more quickly, which is what you would do on the job.
There are jobs for which these kinds of questions may be suitable. But for most, they will just help you disqualify good candidates. Keep in mind that the answers to these kinds of questions seem more obvious to the interviewers because they already know the answer and have watched several candidates wrestle with them. Because of this, interviewers tend to be unduly critical of those who do not give the optimal answer.
No Unqualified Programmers?
So I’m saying that there are no unqualified programmers? NO! As a manager who has interviewed my fair share of candidates I know first-hand that many of the applicants are unqualified. The sheer volume of candidates whose knowledge and abilities fall short of your expectations can easily lead you to believe that there are no qualified programmers.
Like any population, the bell curve is at work with programmers. The highly talented, top contributors that you seek are outliers in that population. You will need to screen many, many candidates to find the ones you need. Just be sure you’re screening for the right abilities.
So What Do I Do?
When hiring, consider what you truly need in your candidates. Don’t fall into the trap of just listing all the technologies you use. Determine which can be learned on the job and which truly require experience.
Examine your interview process. How does each step specifically help you identify the kind of programmers you need? Are you properly balancing intrinsic and extrinsic abilities?
Get your most talented engineers involved with screening candidates. “It takes one to know one.” Great engineers may not be able to describe how to recognize another great engineer, but they know one when they interact with one. The time they spend interviewing candidates will pale in comparison to the time they spend fixing problems created by less capable programmers.