Periodically I read articles claiming that there is a programmer shortage in the US. Bill Gates made that claim earlier this year. A quick search on Google turned up similar articles going back to 1998. Do we have a problem with too few programmers in this country? I contend that we do not.
Certainly shortages arise in specific specialties as some technologies become more popular. Likewise, some geographic areas may see more competition for programmers than others. As lovely as it is, not all of us want to live in California. These articles are talking about large-scale, industry wide shortages and not the kind that can be explained by local factors. So what’s going on?
I believe that there are a number of problems at fault.
Programmer Shortage as Means of Lowering Programmer Pay
When Mr. Gates decries the lack of available programmers, he offers the solution that we should increase the number of H-1b visas. Indeed, if there truly is a shortage then this seems like a reasonable solution. However, this is more likely a ploy to address the true problem that these companies aren’t willing to admit: there aren’t enough programmers available at the salary they are willing to pay.
I worked at a company that would post positions that had ridiculous job requirements that almost no one could meet. They would then use the unfilled positions as justification to bring in programmers who were on H-1b visas that were, frankly, no more qualified than the American programmers that were rejected. The ones I knew were paid far less than their American counterparts.
This is a peripheral issue and not what I deem to be the problem at most companies.
What is a Qualified Programmer?
The trouble is that many companies don’t know how to identify a qualified programmer. I say that because their job postings frequently contain combinations of specific technical abilities stated in years of experience. In a previous blog, The Mythical Year of Experience, I discussed why that approach is a failure.
Even without relying too much on years of experience, most companies put too much emphasis on experience with a specific combination of technologies. This includes things like Oracle, Struts, .NET, XML, etc. I consider these to be extrinsic abilities. Knowledge of these things must be specifically learned and aren’t generally applicable to other areas. In contrast, intrinsic abilities are more portable and can be applied in many areas. These include problem solving, communication, and the ability to learn quickly.
I’m not saying that you should hire people who are totally unfamiliar with the technologies you are using, just that this need is too often overstated and the value of the intrinsic abilities is rarely weighed.
Let’s take XML as an example. Several years ago, I worked on a project that processed mortgage applications. We decided to use XML to store the documents while they were being processed. Those of us who designed the document structure and laid the fundamental access patterns needed a thorough knowledge of XML. Once that was finished, I was able to show my team members how to access fields very quickly.
If we had an opening on this team, many companies would make the mistake of requiring a couple years of experience in XML since it is so important to the project. A good programmer, who learns quickly, would be able to get up to speed on our use of XML in no more than a day.
To illustrate the importance of these intrinsic abilities, let’s look at another project I worked on. This was a C++ project where we had a requirement to run on several platforms. We ran into portability issues with the C++ code and decided we would be better off if we rewrote the system in Java. Though none of us had experience with Java, we quickly learned the language and proceeded with only a few minor hiccups. Our intrinsic abilities as great programmers allowed us to quickly adapt to this change.
The one constant in technology is that it is going to change. Hiring people with a greater focus on their intrinsic abilities will create a stronger work force that is better able to adapt to change. This will help your company avoid the short-term programmer shortages created by these shifting technologies.
How Do You Recognize a Qualified Programmer?
You can also tell how much trouble companies have with this by looking at their interview processes and the questions they ask. They are as likely to hire a truly effective programmer using their approach as if they had candidates randomly draw “Hire” or “Don’t Hire” cards from a deck. After some of the interviews I’ve been through, that approach would have been more satisfying and more respectful of my time.
Most companies ask the same old questions about your previous experience. You spend your time describing what the project did, not what you did. Any member of your team is likely to be able to answer these questions just as effectively, regardless of how little they actually contributed.
Some companies ask ridiculous questions with no specific answer, like “How many manhole covers are there?” They then look for candidates who try to reason out an answer. Or they might give you some kind of problem to solve about weighing balls or crossing a bridge. Tell me, what analog does this have to your daily programming tasks? Sure, problem solving is important but something a little more closely related to the programming domain might be nice. These problems too often rely on a flash of insight, which may be difficult to come by when standing in front of strangers with a potential job on the line.
Some companies ask questions that are more pertinent to programming, but get off in the weeds on very detailed algorithmic knowledge, like “How do your reverse an array without a temporary variable?” Or “How do you detect a loop in a graph?” These are good tests of your memory if you’ve been exposed to them before, but you will probably not puzzle out a great solution during an interview. Certainly, you could look up the answer more quickly, which is what you would do on the job.
There are jobs for which these kinds of questions may be suitable. But for most, they will just help you disqualify good candidates. Keep in mind that the answers to these kinds of questions seem more obvious to the interviewers because they already know the answer and have watched several candidates wrestle with them. Because of this, interviewers tend to be unduly critical of those who do not give the optimal answer.
No Unqualified Programmers?
So I’m saying that there are no unqualified programmers? NO! As a manager who has interviewed my fair share of candidates I know first-hand that many of the applicants are unqualified. The sheer volume of candidates whose knowledge and abilities fall short of your expectations can easily lead you to believe that there are no qualified programmers.
Like any population, the bell curve is at work with programmers. The highly talented, top contributors that you seek are outliers in that population. You will need to screen many, many candidates to find the ones you need. Just be sure you’re screening for the right abilities.
So What Do I Do?
When hiring, consider what you truly need in your candidates. Don’t fall into the trap of just listing all the technologies you use. Determine which can be learned on the job and which truly require experience.
Examine your interview process. How does each step specifically help you identify the kind of programmers you need? Are you properly balancing intrinsic and extrinsic abilities?
Get your most talented engineers involved with screening candidates. “It takes one to know one.” Great engineers may not be able to describe how to recognize another great engineer, but they know one when they interact with one. The time they spend interviewing candidates will pale in comparison to the time they spend fixing problems created by less capable programmers.
Posted in Programming
I’m seriously tempted to send out a few custom resumes in response to various job offering; those resumes would not contain any buzzword and very little technical names, but they will be filled with what you call “intrinsic abilities”, even in the parts where I describe my past experience.
I’m really curious to see how many and what kind of interview they would land me
Really good post, I totally agree with what you write
The best ridiculous question I got when applying for a programming job was “What is the acoustical resonance of a half full beer can?”. And I replied with 12 or 16 ounce can? And I ended up getting the job.
Maybe we are better off having more H-1b visas and taking lower wages here in the US, then having high wages and no jobs.
Here’s a way of identifying a weak programmer:
“Maybe we are better off having more H-1b visas and taking lower wages here in the US, then having high wages and no jobs.”
Worst Question I was asked.
If you could be any animal, which one would you be?
this was my first interview for a programmer job right after college. Being stumped I replied Tiger; because he likes to take control?
before leaving I askt the interviewer the same question and he said chimpanzee; because they effectively use the tools they have. Tools being sticks.
H1B is such a massive racket.
Consider the average H1B Indian programmer. His BS in Computer Science was paid for by the Indian Government in India. His Masters, typically from a middle tier U.S. university, was paid for by the Indian government. Then, he gets out of school with no student loans and competes with U.S. graduates at a significant advantage.
Corporations are lobbying so hard for H1B slots because they are significantly cheaper to hire. The corporations actively cheat the inadequate safeguards that are in place to prevent the H1B workers from displacing U.S. workers.
H1B should not be expanded, it should be abolished and all the scab labor imported from the rest of the world should go home.
Fred,
this (“H1B should not be expanded, it should be abolished and all the scab labor imported from the rest of the world should go home.”) is a sign of how weak as a programmer you are.
The competition is just as healthy as possible, and if you’re afraid of losing your job to an indian (or whoever), you should stop worrying about it and begin to improve yourself as a programmer.
FYI..
Little facts about Indian Education Systems.
In India, to get B.S in engineering or Medicine, you have to write an entrance exam. 100s of thousands of people write them. 10% of them get the adimission. You will have to pay for the education. Parents usually take the pain.
Then same things happens for M.S. Here also, the parents pay expenses. Once in a while, there are scholarships available for them at this level. There are cases, where people do this on educational loans too..
Now, in america, what I notice is most of the teens spend their time on entertainment like video games and sports. I am not saying they should not..They should, but should give preference to education first. Then comes entertainment. That’s what is missing here. Citizens of U.S.A have good resources and opportunities. But, are under utlizing them.
I did my M.S. here on education loan, and paid off responsibly after living a stingy life in one year. I got a decent job and not to mention, got some patents for company which was again founded by Indians. This company is now employing people all over the world and created thousands of jobs in U.S.
One other question that comes to my mind is why companies should pay more salary for programmers. Any one with programming skills can do that. Or they can be learned on duty. Did you ever notice how much a mechanical, electrical or a civil engineer gets paid.
I have gone to the University of Waterloo in Canada, and around 60% of people from my school join Microsoft/Google/Amazon/Apple after graduation.
The average total payout is around 95-120K for a new graduate. That is definitely above the country-wide average and around the average salary for West Coast. Defies your theory about H1B visas being used to bring in cheap labor.
I also can’t see a single problem with companies scouting the talent globally.
Usually an article like this one would not come under my sphere of reading material, but surprisingly this was ranked 1 on reddit. My comments here are certainly going to be flamed, but still I cant keep my hands still.
So, here goes… If you have not known so already from my screen name “I am one of them – the enemy” you so rightly hate.
Hey first of all – lets say hypothetically speaking, if our situation were reversed and I thought that someone from lets say Bangladesh started coming to India and taking away IT jobs there, there would be riots on the streets. So, I understand …
You have every right to your opinions, and frankly I don’t care. I am not trying to change them. But, there are a lot of misconceptions on which you base your opinions on. Let me try and clear some of them. The rest is up to you.
1. H1b candidates are not paid lower salaries, legally they cannot. If you are sure they did you can anonymously complain to the Labor department and fix it.
2. The ridiculous postings you saw are not for bringing in H1b programmers, they are for making the ones already here Permanent Residents (GC). So, wrong there too.
3. I have interviewed and hired people, likewise been interviewed and been hired here in the US, and I have seen no difference in the process between Visa holders and citizens. In fact it is probably harder for Visa holders because for most companies maintaining Visa workers brings added hassles and associated costs. I personally have been rejected by startups after clearing interviews because they did not want to bear the additional costs. You can see posts on craigslist that say Visa holder need not apply.
4. On one level you maybe right – if you believe in the free market, then you know that price is determined by supply and demand. H1B workers increase the supply of IT workers, so from a macro economic perspective it maybe affecting wages. But that is free market for you.
5. The world has changed a lot from the last time you checked. Though most American Companies started out that way, and by that way meaning “American”, today a large percentage of their revenues comes from outside the US. By what logic do you argue that they should show preference in hiring workers in the US.
I have used SlickEdit at work in the past, it was surprising to read such a superficial post from its author. You sound like Hyde in “That 70’s show” – It’s all a conspiracy man.
And to the Fred – dude who commented, where do you get your information from the “The Mars Times”.
1. Indian Government does not pay for BS education, the student’s parents do. And yes when most Indian students come out of college they are financially debt free. You wanna fix that talk to your dad, give him a hug once in a while.
2. Indian Government does not pay for MS education in the US. Well, this’s got to be the most moronic statement I have heard. Firstly, ’cause they don’t have that money, remember India is a poor country. Secondly, even if they had the money they would do it ’cause they get no benefit out of it. Unlike the US India does not tax its citizens worldwide income. I pay taxes and social security here, and India get nada, zippo.
Now if that made you think – phew at least that one good thing. But, sorry to spoil your party. Well still MS education for an Indian student is expensive, so who pays – they get financial aid at least most of them do. But before you rise up in anger again, let me tell you, this aid is not for students from poor countries. It is open to all – write your GRE/GMAT apply to universities, and let the best man win.
… And dont worry the scab labor will one day go home. In the meanwhile try and read “The art of Computer Programming” By Donal E Knuth. That will keep your job safe from any H1B worker.
PS========================================PS
This can get someone fired. Appeared why I tried to post.
Warning: chk_crypt(): Failed opening ” for inclusion (include_path=’.:/usr/share/pear’) in /var/www/wordpress/wp-content/plugins/cryptographp/crypt/cryptographp.fct.php on line 29
Error: Wrong anti-spam word.
I’m surprised to see the H1-b portion of my article consuming so much attention. Particularly when I dismiss this as a perpheral issue very early in the article.
I have worked with many folks who are here on visas and I do not find them to be any less talented than those with similar experience from the US. No where in my article do I suggest that this is the case.
scabman, I appreciate your thoughtful response, but I don’t understand your objections. Yes, some H1b holders are paid less than there American counterparts–just because it’s illegal doesn’t mean it doesn’t happen. As for the job postings, I am simply relating my experience at one company. No where do I suggest that companies should show preference in hiring US workers. Please be more careful before you mischaractize me or my writing.
I do think that many companies struggle with identifying qualified programmers, whether they are local or not. Before I’m willing to beleive there is a nation-wide shortage of programmers, I’d like to see companies do a better job identifying qualified programmers. When you read about the ridiculous interview practices at some companies, it’s easy to see why they think there is such a shortage.
So, how about more discussion about how you can better recognize qualified programmers? Anyone? Anyone? Bueller? Bueller?
Two parts:
a) Maybe I should be more specific. Restricting the number of programmers coming into the US results in US companies hiring them overseas. Or maybe “izaak” is suggesting that we seek mediocre talent where ever we can find it. The people at the top don’t have to worry about a pay cut due to the number of warm bodies. Look at CEOs, movie stars, sports for examples.
b) How to recognize better programmers? There is a paradox: if your candidate is better than the interviewer, they won’t want to hire on. If they aren’t better, they won’t be hired. So, perhaps the best you can get is people of the same ability as the interviewer.
I’ve given hundreds of interviews — people usually consider me “hard”, “educational”, “fair.”
1. Dive into their best language that you understand well.
I strongly agree with you that specific knowledge is kind of stupid to test as it can be quickly acquired (though I think it takes you at least two years to get solid results from C++ and at least a year for Java, no matter how smart you are).
However, I also think that in order to be a solid programmer, there has to be one language that you’re an expert in — that you can discuss subtleties.
So find out the candidate’s strongest language, get your strongest programmer in that language, and get them to ask questions — big questions that you can chat about, perhaps leading to a more specific question.
In Java, for example, “how do threads work?” is a great question. Let the candidate take the lead. “Imagine I don’t know anything about threads and describe them to me.”
You might be leading to a toy design question. “Imagine this simple program with x, y, z [things like what your business uses.] How would you design the threading?”
2. Find specific expert knowledge and test it.
Again, while I think testing specific knowledge is bad, finding out areas of knowledge where the candidate is supposed to be expert and asking about those is fine. For example, in Java you might say, “What do you know about collection classes?” or even “What are your best-known Java packages?”
3. algorithms and fundamentals.
I agree with you that puzzle questions are stupid.
But I don’t think a question like “find loops in a graph” is at all unreasonable to expect a candidate to solve in an interview — particularly if you specify the problem to be “find ONE loop in a graph” or even “find out IF there are any loops in an undirected graph”.
This is the sort of question you get in school — but it’s also “fundamentals”. You should always be able to solve such problems pretty handily. It’s like asking a pianist to play scales at an audition — they should be able to do them flawlessly.
It’s not like I remembered the solution to this problem — there are only a few ways to do things — in this case you traverse the graph until you run into the same node a second time, which means that a loop exists.
As for reversing an array without a temporary variable — that’s impossible in general as stated though of course there are language tricks that hide the temporary variable from you: eg, in Python
la = len(array)
for i in xrange(0, la / 2):
array[i], array[la - i - 1] = array[la - i - 1], array[i]
pretends not to have a temporary variable but behind the scenes Python is creating two of them for you.
I beleive your last paragraph say it all. Only a good programmer can recognize another one. Anyways, really interesting series of articles. Keep it up.
And for those cheap laborers up here, party while it last, soon enough we’ll protect our jobs and salary ;>
Mr.foobarer,
When you protect your job and salary, you will automatically be a cheap laborer…Guaranteed…
Mr.foobarer is another weak engineer.
Sri
About reversing an array without a temporary variable.
Just use e.g. XOR swap repeatedly.
for i=0..n/2:
swap(i,n-i)
Then again, there’s no point in doing it this way so the question is still stupid.
I second Dimitry.
MS and most big companies pay H1-B at the same level as you, Americans.
Here’s what happened in the actual work-field:
Americans are not loyal to their companies while Asians (including Indians) have a different point of view in terms of loyalty.
Americans are going where the money go whereas Asians prefer a good relationship with co-workers even if the job might be boring or whatsoever.
I’m not saying that Americans are bad, it’s just a different perspective. While you guys are looking for instant-gold via Startup (a.k.a leaving your old company) during any rush, Asians are happy to have a “stable” job.
Whenever there is a “worker shortage”, it is a prelude to an invasion of cheap foreign labor. These “shortages” are always in the interest of a large corporation, or group thereof, and not in the interest of the general public, i.e. Americans.
These drummed-up “shortages” are nothing new to America, to industrial England, to the modern world that is ruled by money.
But we must realize that there is nothing here at work except for the IRON LAW:
wages tend towards a minimum level corresponding to the subsistence needs of the workers
If a worker wishes to earn a higher wage, then that worker needs to find a higher-value ecosystem to contribute to.
“Maybe we are better off having more H-1b visas and taking lower wages here in the US, then having high wages and no jobs.”
True. Programming jobs are going to India, China, etc, to people living in and continuing to live in those countries. Large US outsourcing firms are even transferring their workforce bases from Shanghai,etc to smaller inland Chinese cities because they can halve the salaries they pay.
H1b-visa-holders and US citizens (and Europeans, etc) are in the same boat, and must take pay cuts, gradually and eventually, to keep their programming jobs home-country-based.
I was interviewing for a data analyst. I setup all the interviews for a week, 2 a day. At each I gave out a short list of requirements, 6 short sentences, a pen and blank paper and asked for a logical data model. Then I gave out a modified list of requirements and a new sheet of paper, and asked for a modified model and for them to describe a transition strategy without effecting operations.
I wanted to avoid discussions like tell me what BCNF stands for and means. I’ve been burned by employees would could answer every technical question like that, and had no ability to apply it.
In summary, it was a short test of the skill I was hiring them for. One candidate rose to the top. Half were not even barely qualified.
Hiring a programmer, I look for experience with multiple languages, the ability to understand the business, and maybe some pseudo-code of a solution to a common problem. Then I usually hit them with a really hard question about anything on their resume listed as “expert”. You should see the jaws drop when you ask questions like “So how do you solve the occupation time of a node in a Markov chain? You listed yourself as expert in them on your resume.”
If you look at this idea in reverse, you must as a programmer figure out how to increase your value so that you can get pay increases and keep your programming job.
You can for example look at it as the difference between a laborer at a job site who is merely assembling things to some plan vs that of an architect who understands how everything fits together, the business implications, and is able to tie together disparate ideas into the overall design.
As an aside, I have worked with a number of outsourced programmers over the last 5 years in India, and generally the quality of their work has been pretty terrible. You’re basically throwing man power at a problem because it’s cheap rather than relying on a smaller more skilled workforce.
Even if the quality of work was equal, there’s a great number of benefits of having the people working on your code be well paid, happy people who are sticking around for the long term. Otherwise it’s just like the prisoner’s dilemma, both sides are trying to maximize their gain without concern for the other side.
First off, I’m not a weak programmer. I was a programmer for 15 years. I gave it up, and transitioned over to DBA work 17 years ago. I’ve never regretted that decision. In general I was fed up with having to work with weak programmers that would kill projects.
During that time I’ve worked with many H1B workers. It’s unavoidable. It is the cheapest way to get bodies to throw at a project. Despite being illegal, employers do pay the H1B programmers much less. They use many methods to accomplish this, but the simplest is to contract with one of the many job shops that specialize in importing cheap labor.
So the Indian government doesn’t pay for degrees? News to me. I’ve known many Indian programmers who told me directly that the Indian government paid for both their B.S. in Inda, and for their masters in the U.S. I’ve never had any of them tell me that they or their parents paid for their degree. Since I have 6 H1B programmers on staff here I’ll check around again.
I’m not going to get into the quality of Indian H1B programmers. It’s just like any other pool of people. Some good, and a lot awful. It’s not important really and I don’t have a problem with the workers, just the system that allows a pool of low cost workers into the country.
H1B is not necessary. The complaints of a shortage of qualified people is entirely fabricated by the businesses that want to lower costs.
I suspect that many of the companies suffering shortages probably dumped a great many of their programmers during 2001-2005. (Yes, I know there were some new HTML monkeys who took a Java class, proclaimed themselves programmers, got jobs in unrealistic start-ups, and then got fired in the dot.com crash. But a _great_ many highly skilled people working for established employers also suffered personal financial disasters during periods of unemployement due to mergers and cutbacks.)
Employers want it to be easy to get new programmers now, but guess what? When their business turns down, they’ll dump us on the street again. You want more programmers: go ask other companies for lists of people that were let go in 2001-2005; find those who eventually found jobs in other fields and offer them money and training to entice them back into programming.
Or do like companies did in the 1970s — post jobs in the newspaper asking for recent college graduates with majors or minors in math and other quantitative subjects, offering to hire them into your IT training program.
High-tech is supposed to be a field in which unless you’re incompetent you _never_ have to worry about having a job.
A very good post! I have recommended it at my own blog.
Keep up the good work!
Global labor arbitrage is a critical labor issue for technical/science professionals. I hope to see more technical sites cover this. It’s only software/hardware engineers/scientists who can show how this insane skill list often has skills that take 3 days to learn or the qualification demands are more than a particular technology has been in existence. We have more articles on the NoSlaves.comblog.
If it is only for saving money then companies could get lotz of ppl from Indonesia, china and any other place. Why India?
1. They believe in something more than cost savings.
2. There is no question there are many talented ppl among americans.
But how many of them teach their managers.
Most of them try not to explain to their colleagues – may be for job security.
The net result – projects stumble – duplicate work costing more money.
Indians do teach their peers.
Work odd hours without question.
And they r not a threat to the managers.
They work more to cover up other 10 inefficient american programmers.
Truth will always be bitter. But let us be fair… with the facts.
How many community colleges run computer courses by who never did real projects. I am not sure about the colleges. How many NIITs you have in America. Why not? To be frank NIIT and other private institutions did better job than real IITs and IISc s and Universitites in India.
Why do u call them as computer education bachelor or masters degree when they cannot compete with a 3 months training from India.
Look at the root cause of the problem.
To me, it doesn’t matter you have a degree in History, Politics, Engg, or CS, if u don’t have a passion for programming you cannot do it.
Yes the Industry asks for .net , j2ee, oracle, sap, xml, javascript, ruby on rails etc…
Obviously a person cannot learn all of these in his life time. But take it or leave it that is the requirement for the Industry. It is going to get worse with the ever expanding open source.
How do you expect a manager to know all these and ask questions on them. He should be relying on others for selecting. A big coal company said all the J2EE guys quit. And how r they supposed to select a tech lead. Easy ..the boss decided to conduct a written test. As the dollar is falling it is better to look for job in somewhere else if any one is a real programmer…
And you can be sure your “we are better than the world attitude” is remembered by those “scabs”. We are industrious, and hard working while you are pompous and arrogant. Our businesses are growing, yours are shrinking.
Will I hire Americans? No. Not until real progress is made on a cultural level in America.
Here’s a link to a recent article about whether there is truly a shortage of qualified IT workers in the US: Is There Really an IT Labor Shortage?
I agree with the guy who said that there is a cultural difference. Asian workers in general are prepared to work hard for less money because the life they come from, even lower US or European wages are a significant improvement. Also Aisans are more loyal to their employees than their US and European counterparts. I believe businesses recognize this and this is also a factor, it’s not just the lower wages. And about that lower wages, why not? The world operates on a supply and demand basis. It’s not cheating the system. We all know it’s an unwritten rule that a business does what’s best from a financial perspective first, everything else second. IT really doesn’t matter if the worker is qualified or not, keeping wages low means you have the option to replace people just as quickly if they aren’t up to the task.
Securenext Software
Headquartered in Fresno, California, USA, and its offshore in Chennai, India, we rock on every single projects we develop and venture into upcoming technology with a vision of agile web development & customer satisfaction. Thus, we want to make people feel informed and involved, committing quality and timeliness and ready to flourish using latest technology.