The prevailing wisdom at the time was that it would be easy for us to do a Mac version, “because it’s just UNIX now, and you guys are UNIX already – it’ll just be another port”.  I remember hearing this over and over at trade shows in between requests for Ruby support, and “Do you guys have any free t-shirts?” (Right now, there’s a guy in his basement inventing a new scripting language, convinced his will be the one that will take over the world, obsoleting all other scripting languages.  I really wish somebody would stop him, because it won’t take over and obsolete all other scripting languages, but it will become just popular enough we will have to support it.  When I say “stop him”, I don’t mean hurt him or anything – just take him out to dinner or introduce him to a woman).

It turns out the “It’s just UNIX now” logic is a bit, what’s the word I’m looking for…. wrong.  There is UNIX under there, and obviously that was a good place to start when they revamped MacOS.  Plus, I can get a console window which was always my complaint about Macs “back in the day” (yes, I know you could install something, I maintain it should just be there).  But what the fact that “It’s just UNIX now” got us was the ability to create an X11 application that would run on the Mac.  So we did that – and customers loved it.  Well, that isn’t exactly true.  They acted more like we gave them a new BMW but first used it for one of those things you see at the county fair where you pay a dollar and get to hit a car with a sledgehammer (granted they usually do this with demolition derby cars, not new BMWs).  It was SlickEdit – it worked, but it just didn’t look or feel like people expected it to. Hardcore SlickEdit fans were happy… ish.  They were happy to have SlickEdit, but they missed all the Mac-isms, and were a little perturbed by SlickEdit’s menu being attached right to the top of the application window.

So, starting sometime in 2010 our CTO began the process of exploring how to get us a native version on the Mac – and the GUI overhaul that ensued stayed largely in his office until sometime this past spring when different team members began to have certain controls assigned to them to port to our new GUI framework (I’m not going into which one here, the evidence is “out there”).  I was able to be part of this process, and it was very interesting work.  It is an exciting time to work here at SlickEdit.

We had a nice long beta test – a little longer than we planned. I would like to take a moment to say that our beta testers were some of the best we’ve ever dealt with.  Even when there were problems, the comments almost always went something like “It crashed, but thanks for doing a native Mac version, great work, looking forward to the next version!”.  The beta testers this time were aces – some of them old SlickEdit fans we’ve known for a while, and I believe we made some new friends too.  Thanks guys.

So, our native Mac version is “out there” now.  We hope everybody likes it.  Typically I haven’t been a big Mac fan, but I’m learning to appreciate the things that it does well.  I only wish I could get a few people to join me in an Indiana Jones like quest to track down the eleven remaining single button mice on the planet and then Ctrl+Click could work properly.

If you bought any of our X11 versions on the Mac you can get the new version for free. Here’s the link,or you can give our sales team a call at (919) 473-0070.

Let’s assume you have some file extension which is not already set up accurately in the default SlickEdit Core configuration. It might be something your company only uses internally. Let’s use .data for this example.

I have a workspace created which includes some .data files, and what I first see in the Navigator view is this:

When I go to open one of these files, Eclipse will use the standard Text Editor because it does not know better:

Not terribly useful…clearly the SlickEdit editor has not been opened, and we have no language support whatsoever. These appear to be COBOL files…so we should set up Eclipse to recognize them as such, and also tell SlickEdit that these files should be treated as COBOL code.

The first step is to go to Window > Preferences > Editors > File Associations and add *.data as a “New File Type”. Then select *.data in the list and click the Add button in the “Associated Editors” pane. Now select “SlickEdit Editor”. Make sure you mark “SlickEdit Editor” as the “Default” editor for this file type. Now we have told Eclipse to open all .data files with the SlickEdit Editor.

We are not quite finished yet. Now we need to tell SlickEdit to treat these files as COBOL. So, go to Window > SlickEdit Preferences > Languages > File Extension Manager and create a new extension for data (just data…not *.data), and assign the language to be Cobol 85. Click “OK”, and then click “OK” on the SlickEdit Preferences dialog. Now when you open up one of these .data files, you will see the correct editor opened with the correct language support.

Since the Eclipse IDE has become so popular for mainframe developers, we have been dedicating more and more resources to our mainframe language support. The SlickEdit editor is even now integrated into the Compuware Workbench:

Supports a vast array of programming languages for source code editing, powered by SlickEdit.

PL/I is one of the mainframe languages that got a major support upgrade in the newly released SlickEdit Core v3.7.1. The SlickEdit editor now has full support for parsing structures, includes, and even statement level tagging for PL/I code.

SlickEdit Core also has great support for COBOL, JCL, REXX, and other languages. If you are a mainframe developer coding in an Eclipse-based environment, try out the latest version of the SlickEdit Core plugin for free.

I didn’t buy your book. There are several reasons I walked out the door without shelling out for your title.

First off, fifty bucks on the sticker price subjects a book to a lot of extra scrutiny. When I think of all I can do with that sum, the bar gets set pretty high. This used to be somewhat ameliorated when Borders was still a going concern. Their frequent 30% and 40% discount coupons made me a little more cavalier in my purchasing decisions. I am now limited to shopping Barnes and Noble. They have a good selection of titles, but they are a bit more stingy with the coupons, and a 10% discount barely covers our state sales tax. I’m still willing to part with two Jacksons or more, but you’ve gotta wow me.

Why does your book have 10 authors, including 9 I’ve never heard of? Unless I’m reading a short story anthology, I’d prefer to see just one or two subject matter experts. I get a little suspicious when the author count goes above 3.

Annotated code samples and snippets are wonderful. And bonus points can be earned for legibility-enhancing color coding. But I really don’t care to pay for the trees you chopped down to provide over 200 pages of mostly boilerplate code. I work with computers for a living, so I’m not afraid to download a zip file with source code. Look into it.

Your book won’t stay open. I know a nice binding, especially a hardback binding is going to cost more. But I don’t have two sets of arms. Most of the time I’m reading a computer book while trying to work on the computer. I’d prefer not to have to pin down a page with empty coffee mugs and my left elbow while trying to type in a code sample.

Your index isn’t helping me find what I need. If I’m in the market for a programming title, it’s often because I am unfamiliar with the language or technology. And since I’m not plugged into the lingo yet, I need all the help I can get finding the relevant passages. An index that is 90% class and function names does me little good.

Before you decry how internet retailers are pushing down your margins, let me assure you that I still buy most of my software titles in a brick and mortar store. I’ve been disappointed too many times buying books online, sight unseen. I need to hold a book in my hands and flip few a couple chapters. I’ll buy that book in the store, but you’re going to have to work a little harder for my money.

Sincerely,

TechBookShopper

The primary oddity I see is the scheduling of the release. Hot fixes are easy. A problem comes up, you investigate and fix it, then send out the small update. And for us at SlickEdit, major releases are dictated by our annual release cycle. We *will* have a beta in the early spring and we *will* have the major release out late spring/early summer. Lawns go un-mowed, playoff hockey goes unwatched, and “I’ll be home for dinner” promises are broken in order to meet the schedule. For hot fixes and major releases all of the time pressure is one-way: the sooner the better.

A point release is just harder to time as there are forces pulling at both ends of the schedule. An earlier release means customers don’t have to wait as long for the features we had to let slip initially. There is a shorter wait for resolutions to the small bugs that didn’t warrant a hot fix. An earlier release means that the quality of the trial version is better sooner. And the quicker it goes out the door, the sooner we can focus all attention on feature development for the next major release.

But there are also forces that want to delay the service pack. More time means more polish, fewer bugs, and more special requests that can be accommodated.

This release is a little more poignant for me personally. I primarily work on the Mac version of SlickEdit. And we are committed to providing a native version for 2012, meaning that we’re finally getting rid of our X11-based UI. It’s a welcome (admittedly overdue) change, and I’m enjoying the early progress we’re making. But this release of 16.0.1 represents the last chance we have to make the X11 version work as well as it can.

But now the release notes have been finalized, customers have received the update e-mails, and it’s time to call 16.0.1 done.

Few companies and few developers like to talk about the warts in their software, but let’s face it — for decades, SlickEdit has been a single threaded application, nothing but a bicycle-built-for-one. That single thread rode high and strong and did what it wanted, with all the powers and dangers of the world granted to it to do as it pleased. Let’s call this thread George, King George, as it was.

Then we decided to let another thread into our world. Well, needless to say, King George did not like this newcomer in his kingdom. Suddenly, he had to synchronize his peddling and braking and balancing with another thread, and let me tell you, he was not used to doing so much paperwork. Sorry, George, you have to acquire a Mutex before you can use the memory allocator. Sorry, George, you have to release the Mutex when you are done. George was angry, frustrated, and not accustomed to waiting in line behind these mere peasants. They were supposed to make his life easier. Instead, they pervasively filled his kingdom with landmines. George had to step carefully from now on. His smallest comfort was that those little threads also had to step carefully, very carefully. The King was also comforted to know that there still were places that only he could step.

Converting a single-threaded piece of software into multi-threaded software is a task that requires vision. It also requires strictness and thorough investigation of what-calls-what and what-uses-what. The vision comes from finding ways to utilize asynchronous processing and deciding what parts of the application to make thread safe first. The vision also comes from learning new ways to design less stateful APIs and data structures so that you can minimize access to shared data, or at least make the access thread-safe.

It can’t be done all at once.

The techniques for writing thread-safe software and multi-threaded software in general are well documented. It is certainly not a new science, but each thing you convert poses new challenges.

My challenge in SlickEdit 2010 and SlickEdit 2011 was to make it possible for a thread to parse code and insert indexing information into a tag database. Well, that certainly can’t be real hard. That is, until you meet King George.

Step 1: A thread that parses code

SlickEdit has great parsers for many, many languages; the problem was, none of them were written in a thread-safe manner. The lexical analyzer framework was completely stateful and also lacking in flexibility and power. Every parser would need a new lexer, and it had to perform well. The invocation mechanism depended on the Slick-C interpreter, which is still far away from being a thread-safe component. The parsers used global data when they wanted to and communicated directly with the database. To make a long story short, a lot of code had to change.

Step 2: A tagging job queuing framework

Once we could parse code, we couldn’t just start creating threads all willy-nilly. We needed a block of threads to do the work and pass the results forward to the editor. We needed to define what a tagging job was and how the indexing information was collected. We needed to reconsider every place where the editor would launch tagging jobs and see if those jobs could be done on a thread. As it turned out, the answer was “nearly everywhere”. Oh, by the way, King George liked that.

Step 3: A thread-safe tag database

This is one of the big steps forward in SlickEdit 2011. A thread can write to the tag database, making it possible to do everything required to build or update a tag database completely in the background. This was no easy task, because the tag database is a sophisticated component that was built specifically for King George. We had to rethink how we traversed through items in the database. We had to get rid of shared global variables. We had to refine the database block cache so it could be shared by threads.

Step 4: Getting a list of files to tag on a thread

The final step was to scale the threading up from allowing the main thread to schedule jobs one at a time to having a thread schedule a a list of files to be tagged. We needed this thread to find all the files in your workspace, check dates, figure out what language the files were, and finally schedule anything that was new or out-of-date to be tagged. This meant rewriting a lot of single-threaded Slick-C code as thread-safe C++. The speed gains from this change were significant.

The final result

SlickEdit 2011 is a huge improvement over SlickEdit 2010 with respect to its handling of background tagging. King George is beginning to appreciate what these little threads are doing for him and how they are making his job easier now.

The worst problem: Mutex acquisition order

This is what I regard as the worst thread-safe code synchronization problem. Now, if you are fortunate enough to be writing thread-safe code from scratch, where you have an existing, clearly modularized code whose critical sections are small, short-lived, and well encapsulated, it is not a problem likely to hurt you unless you do something silly. But, when you have a large base of single-threaded code with large amounts of shared data that you have to try to make access to thread safe, things get ugly really fast. Here is the classic deadlock condition.

   VSMUTEX mutex1;
   VSMUTEX mutex2;

In the main thread, King George does this.

    mutex1.lock();
    ...                        // I own mutex1 and rule supremely
    ...                        // except that I don't own mutex2 yet
    mutex2.lock();

In another thread, some peasant does this.

   mutex2.lock();
   ...                          // I own mutex2, nobody else can have it,
   ...                          // not even my reverent King
   mutex1.lock();

When King George has mutex2 and is trying to get mutex1, and simultaneously, another thread has mutex1 and is trying to get mutex2, they will deadlock forever. There are a few ways around this, but the best way is the classic solution of don’t do that. The problem is that it is hard to see how pervasive this problem can be when some special case, such as error handing, causes everything you think you know about what order you acquire Mutex’s in to be reversed. This is a problem that can only be solved with thorough analysis and testing and fundamentally sound designs. But, given the circumstances surrounding migrating older single-threaded code to support threading, such designs are harder to come by than one would hope.

Summary

If you read this whole thing looking for deep insight on how to write solid thread-safe code, well, you probably finished sadly disappointed. If you are planning on renting a bicycle built-for-two with your spouse and were looking for riding tips, then you really, really read the wrong article. If you came here because you always thought SlickEdit should do tagging in the background and wondered why it took us so long to implement it that way, then maybe now you understand the obstacles and hurdles a little better. We are still iteratively improving the system and working to improve the tagging throughput. You can look forward to seeing even more performance gains and scaling in the first update (16.0.1).

SlickEdit is moving forward, and finally, George isn’t the only one peddling the bike.

CodeProject

As we began the 64-bit port we faced the same challenges as any company dealing with a large established code base.  A lot of typing issues.  In the 64-bit world, long means different things on different platforms.  In some cases it doesn’t make any difference, in others it does.  These cases have to be carefully examined.  In some cases the solution was not to use long at all.  There were cases where we had used long because it seemed like the right thing to do, but on careful examination, int is completely sufficient.

However, the most interesting case we came up against was a call to the X-Windows library.  The specific call had looked like this for years:

   XtVaSetValues(xftXtTopLevelShell,
               XtNx, -1000,
               XtNy, -1000,
               XtNwidth, 100,
               XtNheight, 100,
               XtNmappedWhenManaged, False,
               0);

And it was fine in our initial builds and testing.  But as we approached our first beta and started to compile our release configuration, the editor would crash on this line.  I spent an evening trying different compiler options, and different versions of the compiler, because I simply did not see a problem with this line.  But there is one.

The functions that start with XtVa take a variable number of arguments.  In this case, that last argument – the ’0′ – is a pointer.  On a 32-bit system, a pointer is 32-bits wide, and an int is 32-bits wide.  So passing 0 works just fine.  On a 64-bit system, pointers are 64-bits wide, and NULL is a 64-bit unsigned number. 0, however, is a 32-bit signed number.  Which actually works fine… until you turn on compiler optimizations.

Here is the corrected code:

   XtVaSetValues(xftXtTopLevelShell,
               XtNx, -1000,
               XtNy, -1000,
               XtNwidth, 100,
               XtNheight, 100,
               XtNmappedWhenManaged, False,
               NULL);

This wasn’t the most confusing problem I encountered, in fact I’m sure to a seasoned 64-bit Linux guru it should not have a been a problem at all.  But it was one of the things I felt was interesting enough to share here.

Anybody have a good 64-bit porting horror story?  Let’s hear them.

A SlickEdit user thought it would be nice to have SlickEdit displayed on his bookshelf, so we sent him an image and he created his own homemade SlickEdit box. It looks great!

Say you are using Eclipse as shown here (Eclipse 3.6.1), and you want to open another view of the same file.

You can do this easily within Eclipse by using the “New Editor” command. Right-click on the buffer tab of the file and select “New Editor”. For faster access, you should bind this command to a key, or use the mnemonic Alt + W, E for the Window menu. This allows you to create a new editor window without ever having to reach for the mouse. Good so far…but what did we get?

Not really what we wanted, because this is pretty useless. We want to be able to see a side-by-side view of these windows. How can I accomplish this in Eclipse? Well, unfortunately, you have to reach for the mouse. You have to click and drag the buffer tab for one of these windows over to where you want it to be. Then you get what you really want.

With SlickEdit Core for Eclipse (v3.6.1), you can accomplish the same thing very quickly, without ever reaching for the mouse. This might seem like a very small detail, and maybe it is to some folks, but if you try to limit (or eliminate ) your mouse usage, you will find that you are a much more comfortable and productive programmer.

If I have the same file open in Eclipse 3.6.1, but this time using the SlickEdit Core 3.6.1 plugin, I can run the command vsplit-window from the SlickEdit command line, and a new editor window is opened to the right of the current editor window. No intermediate step involving the mouse. Of course, you should bind vsplit-window to a key to be more efficient.

vsplit-window demo

SlickEdit Core also has a command for hsplit-window (Ctrl + H by default) for splitting an editor window horizontally, and you can use next-window (Ctrl + Tab or Ctrl + F6 by default) to rapidly switch between all open editor windows.

There are other plugins available which provide similar functionality. I believe there is an Emacs plugin out there which supports the Emacs-style Ctrl X, 3 window splitting, which also does not require the mouse. But no plugin also provides the level of language support of SlickEdit Core. And SlickEdit Core does have a fully functional Emacs emulation, for any Emacs users out there who want to be able to Ctrl + X, 3 within the Eclipse environment.

Boy: Mr. Turtle, how many licks does it take to get to the Tootsie Roll center of a Tootsie Pop?
Mr. Turtle: I’ve never even made it without biting. Ask Mr. Owl.
Boy: Mr. Owl, how many licks does it take to get to the Tootsie Roll center of a Tootsie Pop?
Mr. Owl: Let’s find out. A One… A two-HOO…A three.
(crunch sound effect)
Mr. Owl: Three!

This illustrates two common problems in oral communication. The first is the iterative process of finding someone who might know the answer to your question. In software development, the questions are typically of a more concrete nature, relating to a specific design or implementation choice.

Most software projects depend on a great deal of local knowledge. This is knowledge that is specific to the project, something you couldn’t know prior to working on the project. This include things like architectural and design details. The local knowledge is rarely written down in an organized manner but is contained in the collective experience of the development team. Because it isn’t written down and cataloged it cannot be searched, except through the iterative query process.

The other problem that this commercial illustrates is that of false experts. Like Mr. Owl, you will find people who will give you a definitive answer but one that is, unfortunately, wrong. In software development, it can be very difficult to spot these wrong answers–you don’t get to see someone just bite into the Tootsie Pop to come up with it.

In some cases, the person may truly believe the answer they give, and we’ll assume that until we have proof to the contrary. At other times, people just don’t like to admit that they don’t know something. So, they make up an answer. In either case, the damage to your work is just as bad.

Folklore is typified by this kind of oral communication. It is the main method for communicating important information in primitive cultures. Knowledge is passed from one person to another. Often, because of miscommunication, the knowledge is corrupted by each successive telling.

It’s like that party game where you start off with a sentence and have each person whisper it to another person. This is repeated several times and then the last person writes down the sentence. In the end, the meaning of the sentence is often radically changed with comical results. The same thing happens in software development, but the results are far less comical.

The solution to both of these problems is to document local knowledge. One form of that is code comments. So much has been written on that topic, that I don’t feel the need to add more. However, many things are not appropriate to store in code comments. It’s that kind of knowledge that is most often passed along as folklore.

Fortunately, technology has given us a solution. A wiki provides a centralized knowledge repository that is edited by the group. This allows for the information to be easily reviewed for accuracy. It also eliminates the need to wander around and find someone who might know the answer to your question.

It also addresses another problem. Many people are hesitant to go ask questions. Either they are afraid of looking stupid, or they don’t want to trouble their coworkers. So, they will often flounder around trying to figure it out on their own, wasting time, or they will try to make due without the answer. A wiki allows people to find an answer without the potential embarrassment of exposing their ignorance.

To be successful, a wiki needs to be searchable and browseable. The search capability is needed to find the answer to specific questions. Browseability allows someone to view a set of topics and delve deeper to get more information. This is important to allow someone to gain the initial knowledge on a topic, particularly for getting new team members up to speed. Most packages do a good job providing the search capability. But very few provide a good mechanism to build a browseable knowledge structure.

At SlickEdit, we use MediaWiki for our shared knowledge. It provides both an excellent search and a convenient means to organize topics. The side menu allows you to structure your top-level categories, and each page automatically includes a table of contents if it contains 4 or more subsections. This makes it simple to build a logical knowledge decomposition. It’s this capability to create a browseable structure that sets MediaWiki apart from other wiki software we tried.

Finding the right wiki software and setting up an initial structure is only the first step, though. The main challenge is getting the team to populate it. You face a couple of common problems here.

The first is the normal reluctance of programmers to document things. Many programmers despise any activity that is not directly related to producing code. This is a tough nut to crack, since this attitude is often deeply ingrained in the developers psyche. How you overcome this depends on the person and the arguments that they will listen to. You can appeal to their sense of professionalism and explain that this is part of professional development. You can appeal to their sense of teamwork and explain how this helps make the team more effective. You will need your full set of management skills to get everyone contributing.

Even if everyone is willing to contribute, one problem that often prevents a wiki from reaching its true potential is that of obviousness. Once you know something, it seems obvious to you. If it’s obvious, then why document it? This same problem plagues code comments. The best solution is to make the wiki an essential part of the problem solving process. When someone is trying to answer a question, they should look in the wiki. If they don’t find the answer, they should make a note to come back and add this information to the wiki after folklore has delivered an answer.

So, how does having a wiki alter the scenario depicted in the commercial?
Boy: Hmm, I wonder how many licks does it take to get to the Tootsie Roll center of a Tootsie Pop?
(Boy looks in wiki for the answer and finds an entry by Mr. Owl: “How many licks does it take to get to the Tootsie Roll center of a Tootsie Pop? Three.”)

Well, at least he didn’t have to go bother curmudgeonly, old Mr. Turtle.

The basic reason for this is that each feature was written in isolation without finding and reusing the capabilities of the other features. This is a very common problem on most software projects. To address this issue, I believe that software development should be broken down into two different workflows: mechanism application and mechanism creation.

We can define a mechanism as a component with a cohesive set of functionality, like the wheel. In most cases, mechanisms are meant to be reusable, though it is possible that there will be only one use for that mechanism in a particular program. Common mechanisms in software include login processors, calendar widgets, and sorted lists. These elements perform a specific function that is independent of what they are used for.

Mechanisms can also be specific solutions to common design problems, like error logging, object persistence, and thread synchronization. Ideally, these would be encapsulated in a library that is used by other developers, but in some instances it is more a matter of following a specified convention.

For example, one project I worked on specified that each function would contain #ifdefs for tracing and logging with specific information that each function should output. This still represents a mechanism that defines a cohesive functionality, in this case, tracing. But instead of writing a reusable library, the mechanism was primarily a set of conventions we all followed when writing our functions.

During mechanism application, the specialized functionality of a feature is built using existing mechanisms. You can work on your own unless you have a question about the specs or need assistance from another developer. The code being written is unique to the feature, so little attention is given to reuse unless a new mechanism is needed. Then you enter mechanism creation.

During mechanism creation, you are developing a generalized capability that is specifically intended to be reusable. This requires a totally different workflow. You have to stop and do a degree of analysis before you create the new mechanism. Several questions need to be answered. Are you sure there isn’t already a mechanism for this? If not, what are the essential behaviors of this mechanism? Where should it be stored?

These questions are best addressed with other members of the team. The goal is to produce a mechanism that can be used by everyone, not just by you. So, you need to get their input to make sure it does more than meet your immediate needs, or at least doesn’t prevent someone from extending it in the future.

When a developer tackles the new mechanism as part of their normal development, two common problems occur. Often the programmer includes the mechanism as part of the feature-specific code, rather than placing the mechanism into the library where others can find it. Even when they do put the code in the proper place, it often is written in such a way that it is only useful for this feature. You can’t make it reusable without breaking the feature that already depends on it.

In most applications, mechanism application is very straightforward, simple programming and requires little creativity. Mechanism creation, on the other hand, requires a great deal of creativity and design effort. You have to think more about algorithms and efficiency.

So why don’t programmers naturally break their work into these separate workflows? I think there are a number of explanations.

First, I think programmers get very focused on completing a feature. If they have to interrupt that work to enter the slower and more deliberate workflow required to create a new mechanism, then it delays the attainment of their goal.

Secondly, programmers like to program. Designing reusable components requires meetings, documentation, and reviews–all of which get in the way of programming.

Thirdly, most programmers aren’t too keen on collaboration. When you talk things over in a group, you have a chance of not getting your way. If you just build it yourself, you can do it the way you think it should be done.

Finally, by separating the two workflows, the creation of a needed mechanism might be assigned to a different programmer. Since the creation of these mechanisms is often more challenging and, therefore, more fun; developers are hesitant to initiate a workflow that would potentially deprive them of this fun.

Basically, it’s more fun to reinvent the wheel than to determine whether a suitable wheel already exists. If the wheel does need to be invented, it’s less fun to do it with a committee of your peers who may disagree with your vision for the wheel.

I worked as a contractor on one such project a few years ago. We were writing a call center application for a large enterprise client. The architect of this system convinced the client that the business analysts would be able to write in a high-level business language (invented by the architect) so that future changes wouldn’t require hiring programmers to modify the system. Each line of business script, it was claimed, had the power of 20 or 30 lines of lower level code. So, this promised even greater efficiency than just eliminating the programmer.

The business language had classes, constructors, destructors, and nearly all of the features of modern object-oriented languages. Maybe it started out as a simplified language that would be accessible to the business team, but it grew into the same level of complexity as the Java code we were writing to implement the rest of the system. There was no way that a business analyst without training as a programmer was going to be able to write effective code with this language.

Even worse, the programming team was struggling to write code with the language. There was no debugger or other tools, not even an editor with color coding. When you had an error, you had to determine if it was a bug in the script or in the interpreter, which was being modified almost daily.

Even if we accept that it is a worthy goal to eliminate the need for programmers, I don’t think it’s possible. That’s because programmers have skills and abilities other than just their knowledge of programming languages.

Whether this is by training or nature, I cannot say. Likely, it is a little of both. Programmers have aptitude in the skills necessary to succeed in our field and these get nurtured through practice. Regardless, I’ve noticed that programmers are often more capable in many ways than their non-programming coworkers.

Programmers think more logically. Working through if-then-else conditions is a core capability for any programmer. While working with business teams on requirements, I have often run across cases the where same ability was lacking.

During one project to develop an expert system for mortgage analysis, I was often given requirements for rules on how to handle loans with particular Loan-To-Value ratios or for a particular loan amounts. Not uncommonly, these requirements would have gaps. I’d have requirements for loans from $0 to $100,000; from $150,000 to $250,000; and for loans over $250,000. But there was a hole between $100,000 and $150,000. At other times I’d have conflicting requirements for overlapping ranges.

Programmers have a superior ability to analyze problems and come up with solutions. They excel at analyzing preconditions, sequences of events, and outcomes. Certainly, this is a key skill in programming, but it is also useful in troubleshooting and business case analysis.

Another key ability where programmers typically have an edge is the ability to make order out of chaos. I think that’s because the programmer is responsible for creating order within the program. We break systems into subsystems, subsystems into modules, and modules into units. There are no physical constraints that dictate the structure of the solution. Whatever order exists is created by the people who write the code. Reflecting on some of the codebases I’ve worked on, it’s clear that this is not a universal trait in all programmers.

While people typically think of programmers as coders, whose main talent lies in writing the arcane syntax of programming languages. I think that their main talent lies in their ability to analyze, troubleshoot, and solve problems. Code is just the physical manifestation that culminates the thought process of the programmer.

Let’s say that someone does manage to write a tool that allows people to define software and control its behavior without having to write code. The person using this tool will still need all of the other mental abilities of a programmer. If this were to happen, we haven’t eliminated the programmer; we just changed the job description a little.

Yes, I can envision a distant future where Artificial Intelligence is used to develop code—but that won’t eliminate programmers, either; it just ported them to a different platform.

Regardless, I’ve grown weary of some wasteful pleasantries, and I’ve been trying to make things more efficient. In particular, I’m tired of saying “please” and “thank you” all the time, particularly with my wife. After nearly 25 years of marriage, haven’t I proven my love and respect? Do I need to continually show it through endless “pleases” and “thank yous”?

So, I asked her if she could just assume that every request I ask is automatically preceeded by a “please” and followed by a “thank you”. I mean really, if she expects one every time, why not just assume it’s there in the first place?

In programmer parlance, this would be equivalent to writing a macro. Macros are substitutions that are inserted inline so that the code is altered, as though the substitution were there all along. I won’t bore you with the syntax, but if you’ve used macros before then you know what a time-saver they can be.

Unfortunately, my wife didn’t go along with this notion. It appears that her ability to execute macros is substantially less effective than a C compiler. So, she still expects a tedious “please” and “thank you” for each request. Sheesh!

Some programmers have taken this notion of macros a little too far. From the many programmers I have worked with, I have noticed the following macros in use:

The Greeting Macro — this is a common one, where the programmer has fully eschewed any responsibility for saying “Good morning”, “Hi”, “How’s it going?” or any of the regular acknowledgements of the other person’s existence.

The “Yes sir, I’ll get right on it” Macro – with this macro, a programmer will no longer let you know that they have heard and understood your requests. You must infer that they will comply because they have ceased their efforts to make you feel stupid for asking.

The “No I don’t think you’re a complete moron” Macro – here, the programmer has given up with trying to smile politely and respect your opinion. The macro is intended to soften his critical and belittling demeanor. It is important to note the stress on the word “complete” in this macro.

The Personal Hygiene Macro – I know it’s boring to shower every day, but the rest of us really appreciate it. Trust me on this one!

please_and_thankyou(“If you enjoyed this article, leave a comment.”)

Cast your mind back to the days of the tall ships, great wooden vessels that plied the oceans under wind power. Imagine one of those majestic ships sailing along when suddenly it hits a reef, tearing a large hole below the waterline. All hands scramble to save the ship. They bail and bail, but the ship continues to sink. As the ship sinks lower, they bail more furiously until finally the ship is lost. The survivors are washed ashore on a nearby island.

As they stare out at the masts of the ship, still visible above the waves, the Captain asks, “Why weren’t we able to plug the leak?”

One of the crew pipes up, “Captain, sir, we were too busy bailing to plug the leak.”

OK, this is a contrived example, and it’s very hard to believe that it could occur in real life. However, very similar situations frequently arise in software development.

Bad Codebase

The most typical way I’ve seen this occur is on projects where the early work has set us up for failure. The infrastructural code is so full of bugs or badly designed that it has become a stumbling block to get the rest of the system built. The team furiously pounds away at the code, driving toward a critical milestone fixing bugs as they can. But with every iteration, things take longer to do and the codebase gets harder to work with.

In this case, the leak is the poor codebase. The only way to plug the leak is to rewrite that part of the system. Often, the programming team is unable to recognize that this code is a lost cause, particularly if they participated in its development. More commonly, the programmers know that this code is the problem, but they can’t talk management into the schedule slip that is necessary to fix it. In the end, the project takes longer to complete than if we had stopped and rewritten the offending code.

Unfortunately, a leak in software is rarely as visible or as easy to appreciate as a gaping hole in a hull. However, the programmers’ work to patch the system instead of fixing it is just as futile as the sailors who just keep bailing. And the result is nearly as predictable.

Poor Process

Another way this phenomenon is manifested is in adherence to bad processes. I worked on a project that used very poor configuration management techniques. The approach to version control was sloppy, and there were no controls for how integration servers were configured or how code was deployed for testing. Consequently, even though we would get successful tests on the integration server, we continually faced problems in production. These problems stemmed from several causes. Sometimes it was because of files that were edited on the integration server and not checked into source control, so they never got deployed to production. Or someone would set up the directory structure differently so that it would work on one machine but not the other.

When these problems appeared, the proffered solution was that people should be more careful. This case might be slightly different. Instead of thinking that you’re too busy bailing to plug the leak; in this case you believe that bailing may be all that’s needed.

I’m a firm believer that your test machines should mirror your production environment in every way possible and that you should deploy your code for testing exactly as it will be deployed to the production environment. And if I ever catch someone editing a file on a test or integration server, I’ll keel haul them!

No Time for Personal Development

I also see the “too busy bailing” mentality at work in peoples’ careers. As a hiring manager, I have interviewed hundreds of candidates for jobs over the years. I’m frequently amazed to learn how little people invest in their careers and the knowledge necessary to advance.

For example, most of my work has been on Object-Oriented projects. So, I typically ask OO questions of the candidates. One question is about how they learned OO programming. What books did they read? How did they hone this knowledge? Typically, I find that most candidates learned OO in college and have done nothing to build on that knowledge, other than their normal work experience.

I’ve also interviewed candidates for web development. So, I’ll get a resume from a bright young college student who is about to graduate. They tell me they want to be a web developer.

“Great!” I say. “So, have you ever built a website?”

“No.”

“Do you know HTML and JavaScript?”

“We had that in a class once.”

“Well, then what have you done to become a web developer?”

“I’m real busy with class right now.”

So, basically, this guy is looking for someone to pay him to become a web developer. I’ll always consider a new grad without formal experience, but I want someone who has done more than just sit through classes on their way to a degree. Show me some passion or interest in your area by working on something outside of class. Or at least have knowledge in the basic tools.

Tool Selection

The final way I see people being too busy bailing is through adherence to less effective tools. I often interact with people who have been using vi their whole career, or some other minimal editor, and just can’t be bothered to change. No amount of telling them how you can save them time will convince them to switch. They may even admit that they know they could be more productive if they switched, but they don’t want to take the time to learn a new tool.

Truthfully, this could be a case of the tool becoming the job rather than being too busy bailing. In that situation, a person spends so much time, say, hammering that they lose sight of why they are hammering: to build houses. So when a new way of driving nails comes along, they stick to their current way of working. “I’ve been hammering for 20 years. I’m not about to switch.”

Switching tools will always involve an investment in time. But if you pick the right tool, that investment will pay off rapidly.

So watch for signs of bailing in your own work and career. Consider whether there is a leak that needs to be plugged first. If not, then bail for all your worth. If you have spotted a leak and no one will listen, maybe you need to do a different kind of bailing.

Ideally, this is done logically and methodically, making sure to choose the technologies that best meet the stated needs and provide for a future growth path. Like most human activities though, this process is often subverted by the foibles of those involved.

Here are a number of Design Fail Patterns, ways that bad designs get chosen.

“To A Man with a Hammer, Everything Looks Like a Nail.” – Mark Twain

Sometimes a team makes an ill choice because of their background. There are more technologies available than ever before (see Software’s Cambrian Explosion). It is impossible to stay on top of every language, framework, and system. You do have to make a choice from the set of technologies you are familiar with or ones you know to explore.

Sometimes the right choice is the tool you have expertise with, even if another is more capable. Predictability is a key factor in the success of a project. Many projects don’t afford the time necessary to learn and master a new technology. So, success depends on doing the best with the tools at hand.

However, I have seen familiarity promote technologies way past their utility. I worked on a real-time communications system a few years ago. The architect on this project was a former Database Administrator. While he was a first-rate DBA, he knew little of object-oriented design or the design of large, complex systems. His design reflected his knowledge. Consequently, the system was designed as a transactional database with a thin veneer of objects to interact with the user and other systems.

I’ve also seen this work the other way. On another project we were building a system to manage patient records. While this was obviously a database project, the team treated this primarily as an object-oriented system, using the database merely to serialize their objects. All permissions, synchronization, and look-up took place in the object layer. To do that, they loaded all of the objects into memory when the system started up, with predictably poor results.

Every system we build has an essence that defines the nature and purpose of the system. For each, there are always technologies that are better or worse suited to that essence. It is very important to look past your familiarity and find the ones that best meet that need.

Design by Resume

Sometimes a team makes a bad choice because of a technology they want to work with. For a technical field, software development is full of fads and fashions. At any moment in time there is some hot, new technology that is supposed to render all others obsolete, and developers clamor to gain experience in these systems.

One such technology was Enterprise Java Beans (EJBs). A few years ago, the media was abuzz with articles extolling the virtues of this almost magical technology. I can hear the voice of Billy Mays now, “Try EJBs, the distributed data solution that removes stains and whitens your teeth.”

For several years, every Java project I worked on desperately tried to find a way to use EJBs. Developers who had never actually used EJBs fell all over each other trying to wedge them into the design. Ultimately, none of the systems I worked on found a good use for them.

Sometimes that hot, new technology will be the right fit. Just be sure you are picking it because it is the right tool and not because you want that experience on your resume.

Design by Magazine Article

There is an entire ecosystem in software engineering that exists just to tell you how glorious new technologies are. That used to be the domain of trade magazines—you know the ones, the large newspaper looking things that are always lying around the break room. Now, websites have taken over as the number one source of telling you what you should be using. Seminars and conferences are another source for this.

Regardless of the source, Design by Magazine Article occurs when the technology direction is chosen because of one of these experts telling you how to do things. This differs from Design by Resume principally in motivation. Design by Resume comes from the desire of the developers to expand their experience. Design by Magazine Article occurs when engineers put too much faith in the so-called experts. Often this occurs in management circles, like when a CTO has time to kill on a long flight in first class.

And this leads us to our final Design Failure Pattern…

Design by Management Fiat

This may be the worst of them all, and certainly is one of the most common. In this situation, upper management decides what tools and technologies will be used. Often this comes as an effort to standardize on a tool set, like when a company decides it’s a Microsoft shop or Java shop. Sometimes it is driven by cost, like when companies standardize on tools like Eclipse.

This pattern is wrong on many levels. First it robs the team of their sense of buy-in. This is an essential element in a team that is functioning at its highest level. Second, no tool or technology is right for every situation. I’ve used both SQL Server and Oracle. In many cases either will do just fine. In some cases, SQL Server is a clear win for its simplicity and lower cost. But sometimes you need the capabilities that only Oracle can give you.

Finally, this pattern is bad because the policy is written in stone. No matter how effective some other solution might be, once management has made up their minds they will not be swayed, at least not until the next hot-shot CTO comes in and talks them into the latest set of must-have technologies.

Friends, let me share with you the one sure thing I’ve learned about technology: it is hard. You are faced with innumerable decisions about which way to go. While there are no roads to guaranteed success, there are many roads to guaranteed failure. You need to keep your options open and make the best choices you can.

You might say that each was well adapted to a particular niche in the programming ecosphere. COBOL was a business language used on mainframes to write banking applications. FORTRAN was the language used by scientists and engineers. C was the choice for writing system code and operating systems. And so on…

Likewise, there were relatively few choices for libraries and frameworks. If you wanted a graphical front-end, you could use X Windows. There were a couple of Database Management Systems available, but usually one that was dominant on your platform.

Since then, there has been an explosion of new languages, technologies, and frameworks. In my mind, this is similar to the Cambrian explosion, a rapid proliferation of life forms that occurred around 530 million years ago. During this period, Earth went from having a relatively few and fairly similar life forms to a vast number of highly diverse life forms.

One explanation for this proliferation is the Snowball Earth hypothesis. According to this theory, there was a period where the entire Earth was covered by a sheet of ice. This created a uniform set of conditions for life. There was little need or opportunity for life forms to adapt to different conditions. So, there were relatively few life forms.

Similarly, the early years of computing saw a fairly uniform ecosystem for programs. Code was written to run on mainframes and later on mini-computers and personal computers. Still, the programs were very similar in that they were largely compiled programs and ran locally. Programs used text to interact with users.

The ice sheet was finally melted, it is believed, by the build-up of greenhouse gases over millions of years. Suddenly there are new conditions for life. New life forms come into being to exploit these conditions, producing the Cambrian explosion.

Similarly, the World Wide Web likely launched software’s version of the Cambrian explosion. The web has added many new challenges and radically altered the way programs are conceived and delivered. There was a great proliferation of languages and technologies to exploit these new conditions.

Though earlier developments like the personal computer and Graphical User Interfaces spawned a degree of new technologies, they cannot compare to the web for sheer number of new mechanisms. Within the Java ecosphere, alone, you can choose from Swing, Spring, JSP, Struts, JFaces, and more to build your interface, most of which were created to help deliver web content.

The Open Source movement is another evolutionary accelerator. There was a time when languages and frameworks were produced by a few individuals working at places like Bell Labs. With open source, communities are collaborating together to produce new technologies. This has greatly increased the rate of change.

I wonder what a Survey of Programming class is like now. It’s hard to imagine how you could cover the major technologies in a semester. Heck, by the end of the semester whole new technologies will likely have come into being. It’s exciting!

Read The Friendly Manual… or not

Many software products have changed over the years to cater to end users who don’t want to spend a lot of time learning how to use new software.  I first saw this in the game industry, where people can’t seem to be bothered to read a manual at all.  These days, games don’t even come with instructions.  They provide a tutorial level or integrated help that watches what you do and introduces you to the things you need to know as you play.  It’s brilliant, really.  The users don’t have to read a thing before playing, and who’d want to?  When I get a new game, I don’t want to waste time reading when I could be already up to level 10.  The game companies win too, because they don’t have to pay for the printed materials.  Everyone’s happy.

But how does this translate outside the game industry?  It’s been tried before, with much less success.  Clippy was the spokesman for the first wave of learn-as-you-go user experiences.  We all know how that turned out.  “One of the worst software design blunders in the annals of computing,” wrote Smithsonian Magazine.

So users didn’t like being told how to use the software while they used it.  Another form of learning that is not always well received is the tip of the day.  I know my first order of business after installing new software is checking the “Never, never, ever show me this again” checkbox on the tip of the day dialog.

Wizards also tried to ease the user into getting up and going without having to wade through options pages and understanding all of the menus and toolbars.  Although wizards have had some success, it’s still a paradigm that’s seems old.  When I think of wizards, I feel like I’m flashing back on an episode of “I Love the 90s“.

At the very least , there hasn’t been a lack of effort to try to help users figure out how to use their software without the manual.  Yet, most of these attempts seem to fail.  It’s as if users don’t want to read how to use the software and can’t be bothered being shown how to use it by the software itself.  Part of me wants to rant about how idiotic that is and the other half of me recognizes that I fall into that group.

Done programming?  Great, now get back to work.

I’ve spent most of my programming career writing programs for in-house use, or writing programs that were bought by large companies to be rolled out in-house.  The thing that all of my work had in common was that the end users were not end users by choice.  The software I helped write was all rolled out in some fashion to a large group of people whose jobs depended on them learning the software.

Here at SlickEdit, I’m no longer writing software that users must adapt to.  Instead, I’m writing software that users  make a conscious choice to use or not use.  The choice to use it hinges heavily on the trial, and if the user can’t get the trial installed and doing spectacular feats in less than five minutes, then the opportunity is completely lost. The bottom line is that the most important software performance metric is not how fast my code works, but how fast the user can use it meaningfully.  If that doesn’t happen, then the user loses interest and all of the great work we’ve done will never be seen.

That’s quite a psychological transition for me, because I’ve never focused before on instant user gratification.  One of our strategies involves the software helping users through an optional tool window, called the Tools Assistant,  which is part of the “Tools for Visual Studio” product.  It stays off to the side or can be closed if the user doesn’t want the help.  But there’s also a lot more to this effort than just adding up-front convenience to the software.  We also have to think up new ways to get users to see the product in action with very little effort on their part, often without even downloading it and installing it.  This involves a lot of non-programming work, like video script writing, screen shots, articles, etc…  It’s certainly an interesting change of pace, but it really pushes the limits outside of my programming turf.  I took a break the other day to fix bugs and it was refreshing just to see code for a while.

I love a change of pace… it keeps things interesting.   However, trying to figure out how to pitch a software product may be one of the biggest challenges I’ve had as part of a software development team.  I certainly never took any marketing classes in college, and at this point they’d all be irrelevant anyway.  So I’m no longer focused on reducing the time it takes to perform complex operations… my focus now is on reducing the amount of time it takes a new user to say, “wow, that is so cool!”  So far I can’t find any way to calculate the big O for that, but I’m trying.

Using Code Annotations to Write Review Comments
The Code Annotations feature in the Editing Toolbox is extremely useful for writing code review comments because it lets you attach comments to specific locations in the code without altering the code itself. You can bring up the Code Annotations tool window by clicking SlickEdit > Show Code Annotations.

[Click for full size image]

The first step when writing code review comments is to create an annotation file to store all of your code review comments. You can then send this file to the code review organizer when you’re done. You can use the Annotation File Manager to create a file called “Code Review – Bob” (or whatever your name may be).

Writing Comments About the Code
After applying the “Scott’s code” visualization scheme that we made in the first article, we can navigate through the sections of code that Scott has written. On line 497 of PackageBaseWhidbey.cs, we see that he’s doing a GUID comparison using a literal string. This goes against coding standards, so we’ll log a comment that the string literal needs to be defined as a constant. Put the cursor on that line and click the New Annotation button on the Code Annotations toolbar. This will bring up the New Annotation dialog.

We’ll create a review comment, which is one of the standard annotation types. We’ll also select the “Global : Code Review – Bob” scope, or category, which will put the annotation into the code review file we created earlier. After filling out the fields and clicking OK, a purple highlighted marker is placed on the line of the new comment.

[Click for full size image]

Once you create an annotation, it gets an entry in the annotation list. You can select that entry to see a summary of the annotation, or you can double click on it to jump directly to the code it is assigned to. This makes it easy to get to the code that you’ve commented on.

We also found a bug on line 431 of FileUtility.cs, so a bug annotation can be logged for that. This new annotation has a red highlight to indicate that it’s a bug type.

[Click for full size image]

It’s helpful to only show annotations that you have created for the code review, without the clutter of others in the list. You can filter out all non-code review annotations by using the Annotation Filter Configuration dialog, available on the code annotations toolbar. Select Filter by Scope and select the “Global : Code Review – Bob” item. Now, only your code review annotations will be visible in the list.

Wrapping Up Your Review
Once all of your code review comments have been completed, they will all be contained in the file “Code Review – Bob.sca“, which we created at the beginning of the article (code annotation files have an extension of .sca by default). That file can then be sent to the code review organizer.

All of the received code annotation files can then be imported by the organizer during the code review meeting, where they can be discussed. Reviewing code this way maintains a tight link between the code review comments and the actual source code, and is far better than trying to manage the process with pen and paper, or a set of text documents.

You can also create a report of the annotations to print before the meeting, or to send to other developers. To do this, click the View annotation Detail toolbar button. The Code Annotation Details dialog will be shown and each annotation in the list will be shown in the report. A small red document icon will appear by each annotation that may be clicked to navigate to that annotation in the source code. The report can be exported to HTML or printed.

[Click for full size image]

The next step will be the actual code review meeting, which will be the focus of part 3 of this series. Once the code annotation files are pulled together from all of the reviewers, those comments can be discussed and updated. After the code review meeting, they can be exported to a single code annotation file and emailed back to the group. The developer being reviewed then has a clear set of action items as a result of the meeting.

Just like a novelist must have a solid foundation of grammar and vocabulary, a good programmer must understand the fundamental APIs for their language. But a novelist cannot write a novel with grammar and vocabulary alone. It takes creativity to turn the words into a meaningful connection with the reader. The equivalent of this in programming is usability for the end user.

Usability is such a bad word. Dictionary.com defines it as “capable of being used”. Thanks for that. However, scrolling down further, it also references the following as it relates to programming:

The effectiveness, efficiency, and satisfaction with which users can achieve tasks in a particular environment of a product. High usability means a system is: easy to learn and remember; efficient, visually pleasing and fun to use; and quick to recover from errors. [1]

Now we’re getting somewhere! This is a perfect analogy to the novelist trying to connect to the reader. Investigating further, I found the HCI (Human Computer Interface) page at Wikipedia. I was expecting to find the softer side of programming, sort of a feel good, warm milk and teddy bears approach to software. However, I pretty quickly realized that it’s a whole other science in itself.

After creating what I think are some really compelling features in our Tools product, I’m having to look back on it now and figure out how to optimize it for the user. That’s so hard to do when you’ve worked with a thing for so long, or even worse if you’ve created the interface to it. Just like I wrote in my “Zen Mind, Programmer’s Mind” post, I really need to go back and approach it as it I knew nothing about it. I’ve always approached HCI by imagining the way that I would want to use it. It’s so easy to forget that I’m just one person and there will be so many more people out there using my software that won’t agree with the way I use it.

HCI is fairly new to me, but I’m always interested in learning something new. The problem to me is how amorphous it is. There are principals, guiding rules, methodologies, and such, but there are no concrete steps. I guess that’s how it is though when you get into the areas of programming that make the leap from straightforward answers to creative problem solving.

I wonder if someday in a hundred years or so we’ll talk about software like we talk about classic literature now. Maybe someday there will be a Shakespeare or a Mark Twain of software. Maybe someday you’ll be able to see a piece of software and say that it’s a distinctive program for a particular developer group. The closest thing we have to that right now is Apple. Someday I hope I can look at may own software and think, “Yeah, that was my style.”

1. The Free On-line Dictionary of Computing, © 1993-2007 Denis Howe

• First, Last
• Next, Previous
• Begin, End
• Push, Pop
• Draw, Erase
• Add, Remove
• High, Low
• Yes, No
• Stop, Go
• OK, Cancel
• And, Or
• Up, Down
• Left, Right
• True, False
• In, Out
• Plus, Minus
• Fixed, Floating
• Open, Close

Each pair is opposite in meaning and different in length. When I study these pairs of words, I can’t help but be absolutely convinced that the inventors of the English language gave absolutely no consideration to the fact that (thousands of years later) there will be millions of programmers writing code in mono-spaced fonts and really, really wanting it to line up nicely, but thwarted by this lack of similarity in length.

If you are smart, you’ll stop reading now, the rest is a bit predictable and boring. The point of this article is that it is annoying to write code that looks like this.

So much is it annoying, that the monosyllabic, four letter abbreviation “Prev” has nearly became a universally accepted substitute for “Previous”, within the confines of software, at least.

You are still reading? Ok, fine. Really, if you have been programming at all, you have already encountered this, and you know everything I’m going to say. You could have stopped at the title, in fact.

There are a few exceptions to this rule of opposites — Get, Set, North, South, East, West, Insert, Delete, Internal, External, Throw, Catch. Get and Set are so loved, I think for their matching size and rhyme, that we have invented new words for those types of functions (getters and setters). Of course, “setter” isn’t a new word, but prior to the explosive discovery that both get and set had three letters, a “setter” usually referred to a breed of canine. What is really impressive are the applications you might find map directions applied to. Not in SlickEdit, but elsewhere, I have seen North and South used for cursor movement where you would have logically used Up and Down. Likewise, East and West have been applied to where you might logically think Left and Right and even, Next and Previous. My theory is that to some, alignment is a worthy cause, so much so that you may chose a naming scheme which involves a level of indirection.

Certain shell languages have employed mirror spellings of keywords as a level of indirection between opposites and like-sized keywords. Thus you find do, od, if, fi, case, esac. The first time you see it, you will think, “What is an esac? If it’s anything like an email, I do not want to touch it.” But, it may grow on you, as the Bourne shell has grown on Unix.

Acronyms were all the rage in the old days when it really did matter that your code be small and fit on a punch card. Some continue to survive and are frequently used, such as Gt for greater, Lt for less than, Eq for Equal, Ne for inequality, Hi for high, Lo for low, T for true, F for false. Sometimes you see hybrid abbreviation pairings, such as Up, Dn, or Add, Del.

Some programming language designers have addressed this issue with more symbols. APL is well known for it’s elaborate character set, “⌊” for floor and “⌈” for ceiling, for example. C++ introduced operator overloading and programmers followed that path of slow, destructive insanity towards standardizing — and ++ as next and previous iterator methods, among others.

But, in the end, this problem of opposites being different is not a programming language issue, it is a natural language issue. It’s a problem with the English vocabulary. And there is little we can do about that. Until we find a natural language invented by linguists who really thought about writing code in monospace fonts, our code will never be aligned as well as it could be.

I can’t say that I’ve met too many developers that eagerly looked forward to reviewing code. Most developers understand the importance of code reviews, but many have experienced the pain of poorly run code reviews. Although SlickEdit Tools does not have a “code review” feature, it provides several features that can be used together to make a code review run smoothly. A well done code review process can be the difference between developer loathing and developer buy-in.

Because code reviews are such a big topic, this article will focus on some of the steps that lead up to a code review. Code reviews typically focus on the work of a single developer, so we’ll start there. We are going to be code reviewing the developer with ID SLICKEDIT\shackett (or Scott) in source control, and we’ll be reviewing his work since March 3, 2008, which is the date of our last (fictional) release.

Finding the Best Files to Review
Our first challenge involves finding the set of files that Scott has worked on since March 3, 2008. We’ll use the Find Version feature of the SlickEdit Tools Versioning Toolbox as our starting point.

Find Version allows you to query versions of the source code files in your solution. The code review organizer can use this feature to find an ideal set of files to review. Simply click SlickEdit > Find Version to bring up the Find Version tool window. For this example, we’ll find versions with the following criteria:

• Since date: Finds all versions checked in since a particular date, such as the last release.
• Developer name: Finds all versions checked in by one or more specific developers.

These two criteria are used to find all versions checked in by Scott during the latest development cycle (March 3, 2008 – present).

No development team has the resources to code review every file in that list, so it’s important to find the best candidate files to review. Therefore, the next task is to look through the results and find a sampling of files that represent the most important or complex code. You might also include some files that have a high number of check-ins by that developer, which indicates high rate of change, or churn. These are the ideal candidate files to be reviewed. Email the names of those files to all developers who will be participating in the code review.

[Click here for the full size image]

In this example, PackageBaseWhidbey.cs has had several changes and is a very important file, so we’ll include that in our code review process.

Loading the Files to Review
When a developer is ready to prepare their review comments, they should close all other files and open the files in the list sent by the code review organizer. This can be done quickly using the Load Files feature in the Editing Toolbox, which allows you to quickly filter the list of files in your solution and open the ones you’re looking for. Click SlickEdit > Load Files to use this feature.

By typing “basew” in the filter, we narrow down the files in the list to only those that include that text in their file name. We’ll select PackageBaseWhidbey.cs and open it.

Isolating a Developer’s Work
One of the most difficult things to do when reviewing the code of another developer is actually identifying the work that that developer did, especially if it was done as part of a team. Once you have PackageBaseWhidbey.cs open, we’ll use the Version Visualizer feature of the Versioning Toolbox to see which parts Scott has worked on. Click SlickEdit > Version Visualizer to show this tool window. Next, select “Default User Scheme” in the drop down at the top of that tool window.

The versions of the file will be processed and the tool window will list the names of all developers who have contributed to the current state of the file. The editor also becomes colorized into segments of lines that identify who wrote each line of the file.

[Click here for the full size image]

You can find Scott’s name (SLICKEDIT\shackett) in the tool window list and double click on it to go to the first line of code he contributed. You can then use the next and previous button on the tool window to navigate through the blocks of his work.

To make Scott’s work stand out more distinctly, select his name (SLICKEDIT\shackett) in the list and click the “Toggle Selection Visibility” toolbar button. When toggled on, Scott’s work will stand out, and all other developer’s work will be dimmed out. This allows you to quickly see what code is Scott’s and what code isn’t. You can click this button again to toggle it back off.

[Click here for the full size image]

Another option is to assign a unique color to Scott. To do this, click the Configure Visualization Schemes button on the Version Visualizer toolbar. This will bring up the Scheme Builder dialog. We’ll create a new scheme to highlight all areas of code worked on by Scott. Click the Add new scheme button and give the scheme a name of “Scott’s code”. We can now add a scheme item for Scott and give it a color of bright red.

[Click here for the full size image]

We can save this visualization scheme and then apply it by selecting it in the Version Visualizer drop down. This same scheme can be applied to any file to highlight areas of the code written by Scott.

[Click here for the full size image]

If you find a block of code written by the developer being reviewed that looks questionable, you may want more context about the developer’s intent, such as the check in comment made when that section of code was checked in. This information is very important for creating meaningful code review comments. The easiest way to do this is to see which version that code was checked in as, and look at the comment associated with that version. To do this, select “Default Relative Age Scheme” in the drop down at the top of the Version Visualizer tool window.

[Click here for the full size image]

Again, the versions of the file will be processed, but now the tool window will list the IDs of all versions that have contributed to the current state of the file. The editor also becomes colorized into segments of lines that identify which version last updated each line of the file. The colorizing of each segment corresponds to the age of the version it was checked in with (dark is oldest, light is most recent).

You can find the version associated with the questionable code and take a look at the comments associated with it. You can also use the next and previous button as before to see all of the code in the file that was updated as a result of that check in. For instance, we may have disagreed with the AddMenuCommandHandler function being public. We can see that change was part of version 762. By hovering over that version in the Version Visualizer tool window, we get a better understanding of why the function was made public. All of these features assist the team in making judgments about which code to review. They also provide some insight into the motivation the developer being reviewed might have had when checking a set of code in.

Next is the task of actually commenting on the code. This task is always best done in electronic format, in a way that can be pulled together as part of the code review meeting. The next article in the series will go through some of the features of SlickEdit Tools that facilitate writing comments about code and tying them directly to the source, without modifying the files.

A programmer’s bookshelf was more than a reference library though; it was your peer resume. Other programmers would look through your bookshelf the same way that animals size each other up in the wild. When you started a new job, your bookshelf contents were like the cologne and nice clothes you put on for a first date. It was a huge part of the first impression you made and helped define what you knew. Did anyone care if you’d really read those books? Of course not! Just seeing Design Patterns, Introduction to Algorithms and The C++ Programming Language was enough to grant you elite status your first day on the job (at least to those who mattered).

At my desk right now, I have approximately thirty books on the shelf. I’ve read them all from cover to cover, some of them several times. However, I haven’t read any of them in the past five years. At times, I’ll crack one open for a quick reference to something I can’t find quickly online, but seriously, they’re relics. I know that I should just do the smart thing and admit to them that the internet has caused their jobs to become redundant. I should tell them that the bookshelf is being downsized, and that they’re jobs have been outsourced to the web servers at Google. I just can’t bring myself to do that, though.

Every one of them has, in some strange way, come to mean more to me than just the information they contain. I can look at the cover of almost any book on my shelf and remember the reason I bought it, the project I was working on when I read it, and the difficult problems that it helped me work my way thorough. Many of the books have pictures of the authors on the cover, personifying them. All of the books make me reminisce to the period of time when they were read, the same way that so many songs from my first year of college do. I feel like I could take my programming books out for a beer and have a blast catching up on old times.

My programming bookshelf isn’t about learning anymore, and it’s certainly not about impressing anyone. I may at some point retire them, but not anytime soon. They sit on my shelf like old friends who I want to share my new programming experience with. They helped me out through tough times and I still want them to be there for all of my latest projects. I think my COM ATL books are grudgingly impressed with all the new-fangled .Net stuff, though I don’t think they trust it. My sockets books like to shake their walkers at web services. And of course, my first HTML book doesn’t think this fancy web 2.0 nonsense looks anywhere near as good as the first Geocities home page it helped me make.

I believe that all teams need management. That’s not to say that they all need a manager; some teams do manage themselves. Effective management gets the most out of a team by setting clear goals and minimizing time lost to indecision. A good manager bridges the gap between the business team and the technical team, making sure that both viewpoints are properly considered. A strong development manager can bring a process-centric view to the development team that seeks to improve daily workflows and operations.

So, what’s wrong with “management”?

I think the core reason that software management is so reviled is that it is so rarely done well. In my programming career, I can divide my previous managers into 3 categories:

1. Benevolent, useless, harmless—this is the category into which fall most of my previous managers. They did a reasonable job of passing along information and managing schedules, but I never really felt they were helping me get my job done. I spent a great deal more time giving them status reports than receiving guidance.
2. Malevolent, clueless, harmful—fortunately this category contains only a few of my previous managers. These are the ones who made the job much harder than it needed to be. Sometimes it was done to further their individual goals, like the manager who stuck us with an unreachable milestone so he could get a 30% bonus. Others, it was because they had no idea what we were doing and didn’t have the courage to admit it.
3. Effective, purposeful, supportive—by far the smallest population, this is the category we wish all managers fell into but so few actually do. These are the ones who truly help you get your work done by providing the kind of guidance and environment you need to succeed. Tom R., you were the best!

How did we get here?

There are so many contributory causes for this problem. The first group, above, is composed of good people who just don’t know what to do. The second group shouldn’t be in charge of themselves, let alone others. I can offer little help to the second group, so let’s focus on the first.

There are two kinds of development managers: those with a strong programming background and those without one (henceforth called “programmers” and “non-programmers”). Even if you have some programming experience, it may not be enough to qualify you as a programmer. You have to have done enough to be considered as a peer on your programming team. I divide managers into these groups because they face different challenges. I’ve worked for managers from both backgrounds and neither is a guarantee of success or failure. My best manager learned his craft managing supermarkets before he went into software development.

To be a successful development manager, you need to know how to manage and enough about software engineering to make appropriate choices. Oddly enough, both groups seem to struggle with the first item. Though the non-programmers tend to have better communication skills than the programmers (if they didn’t, what the heck were they hired for since they’re not programmers?), few of them in my career were skilled managers.

Unique Challenges

The unique challenge faced by the non-programmers is their lack of knowledge about software development. This often hinders their decision-making. A common sign of this is an insistence on keeping to the schedule when a minor slippage could make a huge difference later. When you’ve been around software development long enough, you learn that schedules are fluid. What you do today can make your work tomorrow easier, or harder, depending on the quality of the design and implementation. It takes experience and insight to know when to make these tradeoffs.

Another common problem for the non-programmer manager is not being able to call shenanigans when a developer is snowing you. Even as a programmer, you will never know someone’s code like they will, but you will know from your experience when a developer is likely doing something wrong. My background has helped me to sniff out cases of “design by resume”, where developers chose a particular approach in order to get a trendy new technology on their resume. A non-programmer manager would have a hard time asking the appropriate questions to recognize this.

The programmer managers also face unique challenges. The first is the temptation to code and manage at the same time. I read where others actually recommend this as the desired situation. In my experience, the two activities are incompatible. Effective coding is about immersion, getting into a state of flow, and being very productive. Effective management involves being available to your team to tackle their issues when they arise. Even when I wanted to try to keep coding, I’ve never had a position where the management responsibilities didn’t crowd out any time I allocated for coding. As a manager, my favorite means of being involved with the code is to participate in design discussions and code reviews. This keeps things at a high level and still allows you to contribute your knowledge and experience.

Another problem for the programmer manager is trying to be “one of the guys”. This is manifested as a manager who is hesitant or unable to provide negative feedback. Often, these kinds of managers are promoted from within the team, making this even more of an issue. These managers still view themselves or want to be considered to be a member of the development team rather than its leader.

As a manager, your primary allegiance must be with the business team. Your job is to take their goals and translate them into effective plans to accomplish them. Where you think these goals will lead to problems, you need to point that out. However, if you can’t get them to change the goals, then you need to sign up to them and do your best to accomplish them. Programmer managers sometimes have a difficult time making this change.

Shared Challenges

The most common shared challenge is a lack of knowledge about how to manage. Most of the managers I have worked for had no formal training in management. At best they had a seminar or two. Few ever read any books on management or software development management. Since they, too, have had few truly effective managers, they have no guidance on how to manage. They just don’t know what to do.

An example where this shows up is the weekly status meeting. Most managers have one. They have one because their previous managers had one. In its most typical form, the status meeting lasts for an hour or so while the manager has a conversation with each member of the team, discussing what they are doing. The theory is that having this as a team fosters teamwork and helps everyone to know what everyone else is doing. In reality, the rest of the team already knows or doesn’t care. So this ends up being a large waste of time as the manager has a one-on-one conversation with each team member in front of the rest of the team.

Don’t get me wrong, I’m not saying that status meetings are wrong. But you need to choose to have one because of the benefits the team will get from it, because it is right for this team. Instead, most choose to have one because that’s how they’ve been managed in the past, and it is a visible sign that they are in charge. One of the primary goals of management should be to achieve stated goals with as little interruption in the daily work as possible.

The Solution

What we need to solve this is the same thing we need for all leaders: wise, beneficent individuals with a commitment to goals and compassion for their team. In software development, you need to add in enough knowledge to be mindful of the unique challenges in our field. To erase the pointy-haired boss from our minds or at least remove him as the dominant figure in engineering management, we need enough developers crossing over to become committed managers or enough business managers to gain an appreciation for software development.

As programmers of an application or feature, we quickly develop our own preconceptions as to how the software should work. Every day that we spend working on something, the use patterns of that feature become more and more hard wired into our brain. The end result is often a piece of software that works the way that we would expect it to. However, once that software gets released into the wild, many people are going to be using it, all of whom have never seen or touched it before. As a rule of thumb, each of those people will approach it with different perspectives. Each person will use it in a different way, and have different expectations of what it should and shouldn’t do.

Requirements are always necessary, but it would be impossible to capture everything in the requirements that any user would want or need, especially when it comes to the more subtle areas like UI workflow. Important use cases can also go completely undetected until a real end user actually sits down to work with it. Because of that, it’s important to remember that even the best requirements are not guaranteed to be intuitive or even correct. To help with this, we’ll need to bring in a Zen Buddhist master.

Zen Buddhism in Application Development

There are a handful of books I’ve been fortunate enough to stumble across that have changed me in some way. Those books are the ones I keep going back to, to remind me of that moment when I’d learned something new, yet felt like I’d known it all my life. Zen Mind, Beginner’s Mind by Shunryu Suzuki is one of those books. In it, the author and Zen master teaches the importance of “beginners mind”, which is the concept of constantly seeing with new eyes. It’s a teaching that he recommends for all aspects of daily life, though I doubt he ever imagined software development would be on that list.

As developers, we need to see our work from two perspectives:

• Expert’s perspective: From this view, we make sure that the software is meeting the known requirements, which is our primary goal.
• Beginner’s perspective: From this view, we try to see the software as though we’ve never used it before and constantly ask the question, “Does that make sense?”

These may seem very similar, but they’re not. For instance, let’s create a signature in Microsoft Outlook.

Requirement: Allow the user to specify text to be automatically appended to every outgoing email. (Sounds simple enough…)

Implementation:

1. Click Tools > Options.
2. Click the “Mail Format” tab on the options dialog.
3. Click the “Signatures…” button on that tab pane. This brings up the “Create Signature” dialog.
4. Click the “New” button. This brings up the “Create New Signature” wizard dialog.
5. Enter a name for your signature and select that you want to start with a blank signature. Now click the “Next” button.
6. Type your signature. When you’re done, click “Finish”.
7. Click “OK” on the “Create Signature” dialog.
8. Your new signature is now added to a list of signatures on the “Mail Format” tab pane. Select it and click OK.

Do those steps fulfill the requirement to create a signature? Check. Would a new user to Outlook have any clue how to do that without digging through help? Fail. This is a clear case where the development team did not look at final product and ask themselves, “Did that make sense?” The obvious answer is no.

There was a fantastic blog post recently called “The Great Ubuntu-Girlfriend Experiment“. In this post, the author has his girlfriend use his latest installation of Ubuntu as a way of getting a beginner’s perspective on it. He winds up with some very interesting conclusions, specifically “that the geeky programmers and designers assume too much from the average user”.

End users are demanding more and more ease of use out of the box without having to go to help (much to the dismay of our tech writer). Think about the Geico caveman… buying insurance with Geico.com is so easy even he could do it. Ease of use and intuitiveness are so in demand that Geico launched a massive marketing campaign with that as its centerpiece. The commercials don’t say anything about price, customer service or any of the details of their policies. It’s just easy to use… and that pulls in huge numbers of people.

Putting It Into Practice

Having beginner’s perspective on your own software sounds much easier than it really is. Stepping back and using the software without preconceptions involves being honest with yourself. Sometimes it’s easier to give in to the fact that the software might meet the requirements, and that task can be considered done. However, it’s those little touches like usability that can bring an application to the next level, and move it up from good to best. Finding those usability issues can only be done by being objective about your own work. It’s far better to find those issues during development than when an end user first sits down to try the software.

Whether you use Zen mind, caveman mind or girlfriend mind [ducking behind my chair], try and take the time to step back and ask yourself if you would love your software just as much if you were trying to figure it out for the first time. Just like meditation, it’s a practice that gets easier every time you do it. Soon, it should become a normal part of your daily approach to software development. Of course I’d be lying if I said that I had actually attained this goal, but I try.

I have used the freely available book “The Great Logo Adventure!” as a reference in this article. Please note that I am not an experienced Logo programmer, so if you bear with me, you can learn how to add language support and I can learn something about Logo.

Step 1: Defining the language

Go to Tools > Options…, and navigate to Languages > “Language Manager”, then click on “Add Language…” Enter the following:

• Mode name: Logo
• Associated extensions: logo lgo
• Color coding: leave blank

Step 2: Color Coding

Now navigate to your Logo language settings and find the “Color coding” page.

Click on the top-most “New…” button to create a new Lexer and name it Logo. Now, we need to define the color coding. Let’s work left to right, starting with the Tokens tab.

Logo is case-insensitive, so uncheck “Case sensitive”. The identifier characters can be left pretty much as they are. Logo, in fact, will allow nearly anything to be an identifier, but in practical terms, the typical C language style identifiers are adequate.

Next, we add Keywords. The list below is fairly complete, though there is a fair amount of room for interpretation with respect to what is a keyword and what isn’t in Logo. Note that when you add new Keywords, you can add several at once if you separate them with spaces.

Now switch to Preprocessing and add:

And Punctuation:

And Operators:

Now switch to Library Symbols.

This is where you would typically add coloring for functions that are built into the language or included with the standard language libraries. Since Logo has nearly 500 built-ins, it would be senseless to try to type them all in manually. The attached hotfix includes a file, “logo_builtins.txt” which contains all the built-ins. This file was constructed by opening the help file for MSWLogo (Microsoft Windows Logo, by Softtronics, Inc.) using SlickEdit’s Help Indexer (“Help” > “Configure F1 Index Help”). I added the help file, then clicked on “Export…” to extract the index entries, then edited out the few things which were not library functions. In the Color Coding dialog, click on “Get…”, point it at this file, and it will pull in all the words in the list file.

Note that the color coding engine is not perfect is the keyword coloring. Logo is very context dependent whether a symbol is being used as keyword or just a word or list. Lists are sometimes code, sometimes strings. The color coding engine has no idea about these rules, so it sees a word that matches a keyword and it colors it. This is simultaneously quite useful, quite annoying, and quite unavoidable.

We can now turn our attention to the Numbers tab. The default settings will be adequate here.

In the Strings tab, we have to do something ugly. Check the two items:

• Double quoted strings are always 1 char long
• Single quoted strings are always 1 char long

This is necessary because Logo doesn’t have the typical kind of string constants we have grown to expect in other languages, so we need to trick it, otherwise nearly everything will look like strings. This solution is not ideal, but it is readable. Sometimes, unusual language features will not map to SlickEdit’s default color coding engine, and you have to settle for the best you can do and move on. This is one such case.

The default settings are fine for the Language tab, so let’s move on to Comments. Logo has one type of comment, the semicolon. So, click on “New Line Comment…” and specify that it uses “;” as the delimiter.

We are now done with configuring color coding. Color coding definition are saved in “user.vlx” in your configuration directory. I recommend opening “user.vlx” and copying the entire definition for [Logo], then saving it to a new file “logo.vlx”, also in your configuration directory. This will be used later when we package up the language support.

Step 3: Tuning language settings

We can now configure other Language specific options for Logo. Start with the Indent options. You will probably want to set your tabs to “+3″, or something more in line with your preferences.

Now switch to the Comments options. Define the “Comment line” to use “; ” on the left so that you will be able to use “Document” > “Comment Lines” and “Document” > “Uncomment Lines” in your Logo source files.

On the General tab, make sure “Logo” is selected as the Color coding Lexer name. I also recommend turning on the “Current Line” check box to highlight the current line, since SlickEdit does current line and selection highlighting better than any editor out there.

While we are on the General tab, click on “Language-Specific Project…” to set up a command for loading and running Logo programs in MSWLogo. Go to the “Tools” command and select “Execute”. Enter “logo32.exe -l %f” for the Command Line, and specify the Run from dir to be “%p” (file path). You may have to give a [quoted] absolute path to logo32.exe if it is not on your regular path.

Step 4: Configure aliases

Switch to the Aliases options for Logo. This will allow you to define some typing shortcuts for Logo syntax. I will provide one example and let you go from there.

To define an alias for defining a function, click on “New…” and name the alias “to”. Fill in the alias text with the following:

“%\c” specifies a cursor landing and “%\i” specifies indentation.

Step 5: Create a language support module

All of the above work displays how you can add support for viewing color coded files in SlickEdit, entirely through the configuration dialogs without writing any Slick-C code. However, if you want to hand what you have done over to another developer, the best way is to write a language support module that he can simply load and use. For what we have done so far with Logo, the essentials can be accomplished with the following small amount of code. A complete copy of the language support module (logo.e) is attached to this article.

Step 6: Write tagging support for Logo

Tagging is not rocket science, you can write your own tagging support with a minimal amount of effort, depending on the language you are trying to parse.

In the case of Logo, the critical item is the “to” statement, which defines a procedure.

User-defined tagging support is accomplished by writing a language-specific “proc-search” function, the signature for this function is as follows:

The function returns <0 on error, and 0 on success, also setting “proc_name” to the encoded information about the symbol which was found.

The basic outline for a “proc-search” function is to start by searching for a regular expression which will find lines that symbol declarations happen on. If the search fails, return the failure status and it’s all over. Otherwise, you parse apart the line and get the name of the symbol that was declared. Finally, you set proc_name and return 0. A very simple example for Logo is included below. The “logo_proc_search” included in logo.e is a little more thorough than this, but this is a good starting point for nearly any language.

If you want to support tagging local variables as well as globals, you will need to implement the “list-locals” callback. It’s signature is shown below. The implementation is quite similar to the “logo-proc-search” above, except that rather than working in a start and stop pattern, the list-locals function is just expected to find everything and insert the symbols directly using the tagging API function tag_insert_local2(). I will not detail it’s implementation in this article — please take a look at logo.e to learn more.

Step 7: Adding support for block matching

For many languages, the default block matching which supports parentheses, braces, brackets, and configurable begin-end pairs is adaquate. For other languages, you may want to do more. The prototype hook function for implementing advanced block matching is below. For an excellent example of how to implement this function, look at ada.e.

Step 8: Adding support for syntax expansion

The typical way to add support for syntax expansion is to write commands to handle [space] (syntax expansion) and [enter] (syntax indent). A simple example of this can be found in modula.e. By using the “ext_keys” event table, your language-specific space and enter handlers will be automatically hooked in. For this sample, I am not going to write syntax expansion for Logo — I leave that as an exercise for the reader.

Step 9: Adding support for Context Tagging®

There are six major hook functions necessary to implement support for Context Tagging for a language. Again, I am not going to implement all of these for Logo, but leave that as an exercise for the reader. There are numerous examples in the Slick-C® code of how to implement these functions for a variety of languages. Look at csymbols.e for examples of how they are implemented for a conventional language such as C++. Look at cobol.e for an example of how they can be implemented for less structured languages where you can have function calls without parens, for example.

This function is used to automatically build a tag file for language-specific libraries and built-in functions. You can create a [lang].tagdoc file to document built-in functions. Look at [slickedit]/builtins/basic.tagdoc or html.tagdoc for some examples of how this is done. The attached Logo sample attempts to build a tag file for Microsoft Windows Logo. Note that there is a small problem, because most of the files in their run-time library are extensionless, thus, not recognized immediately as Logo.

This function is used to get information about the code at the current buffer location, including the current identifier under the cursor, the expression before the current identifier, and other supplementary information useful to Context Tagging. If the _[lang]_get_expression_info hook function is not implemented, the editor will use a default implementation which simply returns the current identifier under the cursor and no prefix expression.

Find tags matching the identifier at the current cursor position using the information extracted by {@link _[lang]_get_expression_info()}. This callback is used for symbol navigation, symbol completion, references, and symbol analysis.

Insert tags into the list members tree control matching the given identifier and prefix expression.

If the _[lang]_insert_context_tags() or _[lang]_find_context_tags() hook functions are not implemented, the editor will use a default implementation which simply tries to find a symbol matching the current identifier under the cursor, ignoring any prefix expression.

The _[lang]_fcthelp_get_start() callback is used to find the start of a function call. This determines quickly whether or not we are in the context of a function call. The _[lang]_fcthelp_get() callback then does all the heavy lifting to find the matching functions and parse out their parameter lists and set up all the information in order for the function parameter info feature to display a prototype for the current function.

If the _[lang]_fcthelp_get_start() and _[lang]_fcthelp_get() hook functions are not implemented, the editor will use a default implementation which simply assumes that function parameters are comma-separated lists surrounded by parenthesis.

Step 10: Creating a redistributable hot fix

An easy way to redistribute a set of macros and support files is to package them together using SlickEdit’s hot fix mechanism. To package a hot fix for the Logo support all you would have to do is use the create-hotfix command from the SlickEdit command line. Note that this command requires you to have the Cygwin “zip” utility on your path. You can also construct a hot fix manually by creating your own hotfix.xml manifest and dragging files into a zip file.

The Logo support can be loaded by going to Help > Product Updates > Load Hot Fix… and pointing it to the addons_se13000_logo.zip file.

Summary

This article gives you an initial glance at how to add support for a new language to SlickEdit. It does not detail every single aspect of adding language support. There is still a great deal for you to learn by studying some of the language support code shipped with SlickEdit, such as basic.e, awk.e, ansic.e, pascal.e, modula.e, and fortran.e.

Zip file can be found here

I worried for him while he was telling me about this, and at the same time my mind was half zoning out thinking about my own career. I’ve been in a similar situation… it’s not uncommon for companies to just run out of funding or to have massive sweeping layoffs. The last company that I was at out ran out of money and we all showed up at our office one morning to find movers disassembling our cubes. No prior warning, nothing. Let me tell you, there’s no cup of coffee that can wake you up quicker than the sight of that.

“I just don’t know any other company that’s doing this kind of work,” he continued.

Apparently chem development is not the kind of generic skill that can be easily ported from company to company. I forget sometimes just how lucky I am to have a skill set that is so adaptable. Programming work is so readily available it’s ridiculous. A quick search on monster.com or dice.com will show that there’s a constant need for decent programmers. I say decent because you don’t even have to be a top shelf developer to enjoy this kind of programmer’s market. Most companies are looking for a programmer that can just get a job done. Now, ideally, I try to look for a lot more than that in a job position, but if it came down to being a worker-bee programmer, the money is still good for those jobs and you get to stay on point with your skills. It’s not a bad deal, and there’s always a need for this kind of work.

“The only other pharmaceutical company that has the kind of equipment I work with is two states away, ” he told me.

Again, my brain is thinking about this and wondering what things would be like if my work depended on some niche, specialty apparatus. As a programmer, I’m spoiled by VPNs, remote desktop, portable laptops, 4 gig flash drives and a whole arsenal of gear to help me work from wherever I need to. My friend would have to buy some multi-million dollar research equipment to do his work from home. However, I can connect to my desktop and work from home like I’m right at my desk. The kind of flexibility that gives me is just incredible. How many people are fortunate enough to be as productive from home as they are at work?

“Without being around the lab every day, my skills are going to fall apart… this just isn’t the kind of stuff I can keep up with outside of a workplace,” he told me.

Software developers don’t need professional work to keep their skills up to date. There are many online communities, sites dedicated to providing samples for all aspects of programming, open source projects, and endless opportunities to stay on the cutting edge of software development. Staying current in this way is something many developers do on top of their normal job anyway. It’s not very often that I realize that without those resources, it would be much harder to keep up with all of the change and fluctuation that the software world goes through.

Of course, after all of the things my friend had told me I couldn’t say any of these thoughts I was having. But in the back of my head, I had a real face-to-face moment with the fact that I may be one of the luckiest people in the world simply because I write software. Do I ever worry about losing a job? Sure, everyone does to some degree. Nothing is certain. But I do know that if I ever lost a programming job, I’d have another within a week. No worries.

What does make me lose sleep some nights is what would happen if software development simply went away. It sounds ridiculous, but it’s something that I think about from time to time. Industrial workers in the early 20th century probably never imagined being replaced by robots, but it’s happened on a large scale.  Without software development, I would be in a lot of trouble. I wouldn’t be able to work from home if I needed to. I might have to do something 8 hours a day that I didn’t enjoy doing. I might have to wear a suit.

Even worse, I might have to take a job that didn’t allow me any creative thought. That would be worse than the suit. To have to work at a job where I performed some task over and over, or had some responsibilities that didn’t allow my mind to wonder “What if?” would kill a small part of me. This may be the biggest perk of all in this career choice of software development… the freedom of creativity.

I’m going to spend the better part of today reminding myself of just how lucky I am that there is such a thing as computer programming and that I just happen to absolutely love doing it. When I think about what my life would be without programming, I realize that the stars have aligned in some miraculous way for me.

Here’s my story.

It all started a “few” years ago to when I was in high school. The Rubik’s Cube popularity had just peaked and our school was still just getting computers. I forget the exact specifications of the machine, but I think it was a TRS-80 Model III with about 16K of RAM and no color. I spent a lot of time on that machine because it was old even then and usually free. Most others wanting to use the newer Apples that had color. Wow.

So how do you represent a colored cube on a monochrome display with at best crude graphics? I used letters for the different colors (B = blue, R = red, G = green, O = orange, Y = yellow, W = white) and unwrapped it on the screen. (Fortunately none of the colors started with the same letter.) You would have something like this for a completed cube:

After typing in your sequence of letters representing the colors (You only had one chance. One mistake, and you had to start over.), the program dumped instructions to the printer (“Red Right” “Green Left” “White Around” “Blue Left” etc.) while the arrangement of letters on the screen was updated at each step. Needless to say, it was virtually impossible to take a mixed-up cube, type it’s arrangement into the TRS-80, then follow the instructions from the printout and actually solve the puzzle, but I know a few classmates certainly did try. I really should have made it pause after each turn, so you could check for mistakes as you went.

Here are a few things I remember about the program.

1. I knew the limits of the machine’s memory so made use of the space compression codes. The TRS-80′s BASIC interpreter used the character values from 0xc0 and up to represent sequences of blanks. One byte could be used to display up to 64 spaces. My data structure was one string holding just the letters and space compression codes. I updated the display by printing just this one string.
2. Each of the 18 different moves would change the positions of 20 letters on the display or 20 characters in my string data structure. These character positions were stored in strings of 20 characters. This was much more compact than reading DATA statements. But you couldn’t type these strings with the keyboard because some of their values needed to be higher than the ASCII value of ‘Z’. So I first wrote some code that found the memory location of the array of strings and then used POKE to write the proper values directly into the memory of the program’s instructions. This trick actually made the program manipulated itself.
   I don’t really remember why I did it this way. It did make the program shorter and the code simpler in some places, but I doubt that was the real reason. I think I did it just because I could. Plus is looked cool when you listed the program on the screen. It would beep, clear screen, and show graphical characters inside your program listing. This was a great way to show off to the older junior and seniors when they came in and kicked you off the computer so they could write their boring programs to convert Celsius to Fahrenheit for their computer class. I’m sure they all hated me.
3. Finally, it was my own algorithm. There were a slew books out then to teach you to solve the Cube, but none of that cheating for me. Of course, that also meant that the list of “Yellow Left” “Orange Around” “Green Right” “White Left” etc. probably went on much longer than needed. But that didn’t matter, because you were never going to solve an actual Cube with the program anyway. It was all just a young kid’s way to use up free period.

Best program I ever wrote!

Between contract work and full time work, I’ve worked at a lot of places. There have only been a few that I can say I really loved. Interestingly, the characteristic that all of those places had in common wasn’t interesting work or free food or a ping-pong table. It was my managers.

There are good managers and bad managers, and most programmers can say that they’ve worked with a representative from both groups at least once. However, there’s also a class of top-shelf managers, and they’re the ones I’m talking about here. They have the ability to lead a team, and at the same time make you feel like you’re one of the most important people on it.

Quarterback vs. software manager

Whether you like him or not, most of us (at least in the States) know who Tom Brady is. I wouldn’t normally look at him, or any sports star, and see a clear comparison to the world of software development. However, I recently read an article in Sports Illustrated by Ross Tucker about his time spent playing on the Patriots with Brady:

During my time in New England I worked as a backup lineman and often had to snap to Brady while playing center. In spite of all of the other chaos that he had to sort through, he always found the time to look me squarely in the eye and say, “C’mon Ross, me and you, let’s get a great snap first.”

I never wanted to snap a ball so well in my life.

I was a veteran in my fifth and sixth years in the league while in New England and I had started over 20 games, but Brady’s ability to single me out and make me feel important for the success of the play was unlike anything I had experienced.

That’s powerful stuff. I instantly recognized it as the common characteristic of the jobs I’ve loved working at… managers that practiced the same philosophy as Tom Brady. After reading this, I realized that the need to be recognized as an important part of the team is something universal, regardless of your job or your level of experience. It’s about taking people who work hard and finding ways to make them feel important to the team effort. That includes contractors, junior developers, everyone on the team.

So managers, listen up! Yes, programmers like free food. We like the latest and fastest computers. We like ping-pong and pool tables and a fun atmosphere to work in. But most of all, we like to hear that our ideas are important… that our work matters. So often, we are just looked at as the staff that makes the requirements happen. Sometimes it can seem like we are simply a means to an end, but we are so much more than that. Software developers are creative writers; the designers that make your applications work, the same applications that run your business.

If you give a development team a set of requirements, you hope to get an application that meets those requirements. However, when you let the developers participate in the creative process, and make them feel like their contribution is needed, you’ll probably find a development team that will think up and implement features you couldn’t even dream of. All it takes is trust, and an occasional reminder to development that they are an essential part of the process.

The free pizza doesn’t hurt either…

My first job as “the VB guy”

I graduated with a masters in CS, and my research involved working with the Air Force on a neural network algorithm they were implementing. This was all done in C++ and by the time I graduated, I was proficient in the language. My first job was at a startup with 3 other people, where we needed to get something done very quickly and we used VB4 (the latest VB at the time) to write the entire thing. I had never worked in VB before, but I picked it up quickly and learned how to push it to the edge of what it could do, which was a lot.

I remember being blown away by how fast we could test ideas and write new parts of the system, compared to the work I had done in C++ before. VB didn’t have inheritance and of course it had its own quirky syntax, but the power of how fast you could get things done was amazing.

That project went extremely well and we got an enormous bid to continue our work. At that point, I was firmly rooted in my place as a developer of the initial product and would continue developing it. However, we needed to start writing our own device drivers for new peripherals that we needed to interface with. In came the C++ programmers, and shortly after, I became known not as the lead programmer of the main product, but rather as “the VB guy”. As the VB guy, I was always pretty far outside of the C++ clique, even though a year before I had considered myself a proficient C++ programmer.

It’s a scar I think I still carry to this day.

Any idiot can program [insert high level language]!

Anyone who has worked for a significant amount of time in VB, Ruby, Perl, or even C# and Java probably knows where I’m coming from. There is a large number of people who hate these languages purely because they allow newer, less experienced programmers to develop working code quickly. You don’t need to know what a pointer is to program any of these languages. You don’t need to know how memory is allocated for new objects, and for the most part you don’t have to be conscious of freeing it either. You don’t need to understand a lick of the Win32 API to make a working client application in VB. The list goes on and on and on…

There are also programmers who have battle wounds from debugging memory leaks from unfreed pointers. There are those that learned to program Windows applications from Petzold’s book where you wrote the WinMain function and the message loop by hand. There are programmers who learned COM by hand writing their own implementation of QueryInterface and doing their own reference counting. Many of these programmers watch in horror as new, inexperienced programmers get the same end result in a small fraction of the time because their higher level language wraps up what they’ve spent years mastering, and makes it “just work”.

Check your ego

I believe that’s where so much of today’s language bitterness comes from. All programmers strive for higher levels of abstraction and functionality, whether it’s through libraries or frameworks. However, there’s a level at which the programmers who understand what happens internally start to despise the programmers that make use of that abstraction without understanding those internals. It’s a smug, arrogant attitude that makes a developer believe that they are a better programmer, or that their language is better, because they work at a lower level. Unless you are writing your code in 1s and 0s, there is always a huge amount of “scaffolding” underneath you that you just rely on.

If you’re one of these people, get over yourself. There are both great and terrible programmers in every language. There are brilliant VB programmers and there are frighteningly bad C++ programmers out there. If you resent beginner programmers being able to write apps quickly in VB, or put together a web site quickly in Rails, then you need to seriously rethink what the point of programming is in the first place. Their success should be your happiness.

This is a pretty common programming scenario which can become quite unnerving when you discover that the original function was named perfectly. Of course, you can’t reuse that name because you are programming to a C ABI, or the signature is not different enough to allow you to overload the function.

So, you spend the next half hour of your day figuring out a name for this new function which makes sense. Here is one example, the original function:

   int findMatches(string regex, int flags);

And the new function changes the meaning of one of the flags. So you name it:

   int findMatchesWithNewFlags(string regex, int flags);

But wait, that’s a pretty stupid name, and rather verbose. So you try to capture what is really different about the new function.

   int findMatchesOldReentrantFlagIsReverseFlag(string regex, int flags);

Still pretty verbose, and if we change the function again in the future, this will just get silly. OK, how about this:

   int findMatchesWithReverseOption(string regex, int flags);

This is better, but still verbose. There has to be a better way.

Generational Naming Conventions

Well, on this day of April 1, 2008, let me share with you the dearest naming convention to my heart, generational naming. In this paradigm, we recognize that the original name was perfect, so all we want to capture in the name of the new function is that it is newer. You do this by adding a “2″ to the end of the name.

   int findMatches2(string regex, int flags);

When you need to again change findMatches(), you create the next generation:

   int findMatches3(string regex, int flags);

Extended Naming

There are several less powerful variants of generational naming. For example extended naming, where you add “Ex” to the new function. This is in fact, a common practice at another large software company located in

   int findMatchesEx(string regex, int flags);

And then there is “new naming”:

   int newFindMatches(string regex, int flags);

Which can actually be extended twice over:

   int newerFindMatches(string regex, int flags);
   int newestFindMatches(string regex, int flags);  // buck stops here

Conclusion

The conclusion is simple. You do not need to be smart or clever to write code, you just need to know how to count.

Learn to write well. [*]

“But that’s not a programming skill!” you’re probably saying. Or maybe you’re disappointed because you thought I was going to be talking about the very latest cutting-edge API you can’t live without. Good writing skill is probably one of the most useful talents you can develop as a software engineer, and ignoring it is one of the biggest mistakes most programmers make.

Working in a cave

I used to work with a programmer who we jokingly referred to as a “walking MSDN”. He knew all the Win32 APIs, he could tell you the int values of const definitions and could enumerate the parameter lists to functions overloaded 30 times over. As a programmer, when given an assignment, he would go into his cave, hammer out a solution and would re-emerge from his cave with something that worked. I mean that in the most serious way… he would hammer his code into submission until it worked. There was no such thing as elegant code to him; it worked or it didn’t.

His manager both loved and feared him because he would do everything required on time, but he had no idea what happened in that cave. There was no visibility at all into what he was doing, and his manager had to blindly rely on the fact that he had a history of getting stuff done on time.

Programmers loved and feared him because he was incredibly knowledgeable and could answer any question about how this or that API worked. But whenever someone had to get inside his code, it was a pure mystery. There was no documentation, no models, no code comments… nothing but the raw code itself. For mister MSDN, this was no problem, because it was all in his head. For everyone else, it was like being lost in a big city with no map.

Was he a good programmer? Most people that worked with him would say yes… he was technically proficient and met his deadlines. But most of those same people would also say that they had a very difficult time working with him. That element of fear that he generated was a direct result of not being able, or willing, to communicate. He couldn’t communicate his plans or designs to other programmers. He couldn’t communicate his status with his managers. He couldn’t communicate his interfaces with other teams. Eventually when he left, his code left a wake of terror felt by those that had to pick up where he left off.

The lifespan of a tech skill

Technologies come and go. Seven or so years ago, I felt like I was completely on top of my game when it came to programming technology, at least in the Microsoft world. I had a very deep understanding of COM and ATL, thanks to Chris Sells, Brent Rector and Richard Grimes, and I coupled that with VB6 on the front end. This was a really potent combination and I thought I’d be working with that for years and years. But, as with most technical skills, it didn’t last long… .Net came along a year later, ATL became outdated and VB6 just disappeared into the sunset. Now that I’ve mastered C#, I find that I haven’t even touched the new stuff in .Net 3.5. The point is, no matter what technology you learn, with very few exceptions, it will disappear or be overshadowed by newer technologies (and probably sooner than later).

Writing skills will last you a lifetime, though. You will always benefit from it, regardless of what language or technology your use, or whether you’re even a programmer or not. Good writing skills are universal and will never be replaced.

The golden rule of documenting software design

Try to document whatever you are working on. It doesn’t have to be the worlds most perfect UML and you don’t need an expensive tool to do it. In fact, Wordpad and Paint are sufficient, and don’t tell me you don’t have those. Write in a way that best expresses your intent and your thought process in coming to the design decisions you did. I have a golden rule of documenting software design:

Describe your design to others as you would have others describe their design to you.

When you spend the time to do this, you’ll find that many benefits will follow. You’ll get feedback from others that may have tried to solve a similar problem and have insight you may not have thought of. You’ll leave a clear trail to follow for those that work on the code after you. Most importantly, you’ll shed light on your work, which everyone who depends on your work will appreciate. Even if no one else reads what you write, you’ll still have worked through problems in your head that can only lead to better design in the long run. There is no downside to documenting your designs.

If that wasn’t enough, consider the fact that someday you may be looking for another programming job. You’ll sit down with other developers and managers in an interview and try to describe how great your technical abilities are. You are now one of the vast sea of programmers trying to get that job. With your resume alone, a company has only your answers during the interview to base their decision on. With your documentation in hand, though, you can show them in full detail the type of work you’ve done and give them deeper insight into your thought process than they could ever get from interview questions alone. Your documentation serves as a body of work that travels with you, long after you’ve stopped working on this or that code, and speaks for your experience.

Of course, these skills won’t write code for you. But think about how much time you’ve devoted to learning programming based skills. Think about how many books you’ve purchased and read about programming. It’s a huge investment in time and money to keep up with. The next time you have some free cycles to sit down and pick up something new, consider skipping the software development section of the bookstore and pick up a book about technical writing. Here are some good books on this subject:

• Writing White Papers: How to Capture Readers and Keep Them Engaged
• Developing Quality Technical Information: A Handbook for Writers and Editors
• Documenting Software Architectures: Views and Beyond

I feel like I’ve gotten to know you, that you’re not just a program or a set of files. You have a personality that I spend most of my day with, sometimes more than the people I work with and even my own family. We have our ups and downs. We’ve had days where we hated each other, and you were so stubborn that I wished I had never written you in the first place. But we also had times that made me feel like one of the proudest developers in the world. We learned a lot together and sometimes your bad behavior taught me something new about the way I write software. At other times you’ve amazed me with the things you’re capable of doing, producing results that were useful in ways I’d never originally intended, as if you were secretly developing yourself while I wasn’t looking.

There are going to be users out there and most of them are great people who will be excited to buy you and work with you. But there will also be some that won’t be so nice, and they may say or blog really bad things about you. You may be installed into environments where you have no clue. People aren’t going to be so gentle with you in the real world. They won’t be forgiving when you make mistakes and won’t give you the soft restart that you’re used to when you get debugged. I’ve tried my best to give you everything you need to avoid those situations and handle them gracefully, but sometimes unexpected things happen. Just remember all the good things you can do and let the bad stuff go.

We’ll be seeing each other from time to time, during bug fixes and maintenance, but it won’t be like this last development cycle. I hope I’ll be able to keep improving you and making you the best you can be, but in reality I’ll have new features to work on and a schedule that only allows us so much time together. I know my new features… they’ve started just like you did, some of them as ideas that came to me while I was writing you. But no matter what features I may work on in the future, the time we’ve spent in development together will always be unique and I’ll always remember it, just as I remember the features and programs that came before you.

This will be a milestone build and release and I can’t wait for your introduction to the world. Remember that I believe in you more than you know. Now get out there and show everybody what you can do!

When it comes to extensibility, SlickEdit benefits from the fact that it is largely written in macro code, which can easily be loaded dynamically. Unfortunately, in our Tools product, extensibility is a challenge because all of our product is compiled code. Two of the features of .Net that I rely on to “discover” objects that plug into our architecture are reflection and dynamic loading. Reflection, a term borrowed from Java, refers to the ability to dynamically load an assembly (a .Net or Java compiled binary) and query that assembly’s metadata about the types it contains. In other words, if I’m looking for a class called Foo, I can load any .Net assembly and traverse its types looking for the Foo class.

[i]

Looking for a specific class is pointless for our purposes, though, because we want to dynamically discover unknown objects that fit the description of a plug-in type. To do this, we’ll define an interface called IDoSomething. IDoSomething defines a “get” property called Name and an method called DoSomething().

Any class can implement IDoSomething. One of the beauties of interfaces is that they do not have to obey the law of single inheritance, which states that a class may only derive from at most one other class. Classes in .Net and Java may implement as many interfaces as they choose. That makes interfaces very good candidates for defining the contract between a plug-in class and the framework that creates an object of it. Instead of using reflection to find a specific class, we can use reflection to find classes that implement a specific interface. Therefore, if we scan [n] assemblies and find two classes that implement IDoSomething, we have found two types that are candidate plug-ins.

Our main application does not need to have ever seen these types before because we aren’t binding to these objects through their class interface, we’re binding to them through their IDoSomething interface. Our main application will never know the real identity of these two discovered objects, it only knows them by their IDoSomething interface.

This workflow of dynamic object discovery allows us to set up a pattern for plug-in types. We can create a factory class, called PluginFactory, that scans a set of assemblies for objects implementing a specific interface (IDoSomething in this case).

The main application will create the plug-in factory and will determine which assemblies should be scanned for plug-ins (this is usually all assemblies in some pre-configured directory). The scanning for plug-ins will then be initiated. During the scanning process, when it it finds a class that implements IDoSomething, the plug-in factory object will do the following:

1. Create an object of it
2. Get the object’s logical name via the Name property
3. Get the object’s type
4. Hash that type using the Name property value

Once the scan is done, our application can query the factory object for the list of logical names reported by the objects discovered during reflection. The UI can be populated with this list, either as a combo box or a dynamic menu, to show all available plug-ins.

When the user selects a plug-in item from this list, the application uses the name to ask the factory for an instance of the object it represents. The factory provides a function called CreatePlugin(string) that takes the logical name of a plug-in type and finds that type in the hash table created during the initial reflection scan. Once the type is found, the factory uses Activator.CreateInstance() method in .Net, or the Class.newInstance() method in Java, to create an instance of that object. The main application has no idea what type of object it has… it only knows that that object implements IDoSomething. The main application can now call DoSomething() on that object and the object will do its specific task.

This implementation is very simplistic. However, the concept, like all patterns, is universal. We can even make the factory class generic by implementing it as a generic class in .Net, or a template in Java. We can now create a factory to scan assemblies for any interface. The one dependency is that the interface supports a “get” Name property, since we rely on that property in the reflection scan.

The plug-in factory approach can also lead to trouble, because it opens a doorway to the system which can be exploited by plug-in objects that behave maliciously. For instance, imagine an object whose DoSomething() implementation was to format the C:\ drive. It’s an extreme example, but a serious case to consider.

Reflection and dynamic discovery can be a powerful way to make an application or architecture more flexible. Flexibility and extensibility are key components to any software architecture. In my next blog post, I’ll expand on this technique to describe how to implement UI configuration of dynamically discovered objects.

I could understand if dexplore were processing SETI data on startup, crunching genome data, or possibly computing sophisticated weather patterns. But let’s be honest, it’s a glorified web browser. All of the help it provides can be found online, and chances are, it’s more up to date online than it is dexplore.

Apparently, I’m not the only one that completely hates the help system. AprilR reports in her blog that Microsoft is overhauling MSDN help based on user feedback. That’s great, but why has it taken so long? The only conclusion I can come to is that Microsoft must simply not use it. If anyone on the Visual Studio development team actually used it, it would have been overhauled a long time ago. It’s a shame too, because the actual MSDN content is, in general, very good.

Maybe someday we can go back to chm. It just worked and did the same job that MSDN does, just a million times faster. Well, everything except online content, but… that’s why I have Firefox installed on my computer (yes, Firefox). I set up my own help launcher macro to spawn a web browser, take the selected text and do a Google search on MSDN and the selected text.

Here’s the macro:

Now, if you go into Tools > Options and select Environment / Keyboard, you can bind Macros.MyMacros.HelpModule.LaunchWebBrowser to the F1 key (or whatever you like). Now, when you press F1, it will go right to Google and search for “MSDN [selection]” where [selection] is any selected text in your active document. It’s faster than launching dexplore and gets better results, at least for my purposes. The one thing it can’t do is determine the type context of your selection, but that can be fixed by by adjusting the search query.

Help is one of the most basic needs of every programmer. It’s an “I use it every day” kind of tool. Give Microsoft your feedback, because if there’s one thing that needs help in Visual Studio, it’s the help.

The worst thing, though, is having to hand edit HTML that has been generated by other software, be it an Office document or HTML created by web page authoring software. This was the situation I found myself in recently when I was asked to do some simple web page maintenance for a local volunteer organization.

The site was hosted by Yahoo! Japan and was originally authored by Japanese web site software from a major unnamed corporation, which I will just refer to as HAL. I was not given the original project files from which I could make changes and regenerate the html, so I was going to have edit the HTML directly. The Yahoo! Japan portal provides web-based maintenance tools for the sites they host. Through a browser you can see a list of the directories and files in your site, backup and rename files, and edit and save HTML files in an editor control. You could also upload new files. These are crude tools, but for the changes I had, it should have worked. I didn’t want to manually download and upload each page.

I pulled up the first page in the editor control, and started editing. The file was many times longer than needed, filled with obtuse metadata and other unintelligible nonsense. The page switched between English and Japanese frequently, often many times in one sentence, and each change in language required that section to be enclosed in yet another <span> tag with the proper lang attribute. In many cases, a single word was nested in 2 to 3 identical <span> tags. Most lines were about 80 characters wide, but there were also 500+ character lines just to make things interesting. (I hate HTML.) Anyway, without too much trouble, I was able to find the content and make the changes. It wasn’t hard, just tedious.

I was now ready to view the result. I hit save to close the editor control and return to the file list. I switched to another browser tab that had the page loaded and hit F5 to refresh the display. Every bit of Japanese text was total garbage. Argh!!. I inadvertently changed something that ruined the display of the entire page. I knew I’d have to try again and be more careful. So I open the page in the editor control again and much to my chagrin, now all the Japanese text in the original file was garbage as well. The whole page was ruined. It hadn’t just a bug in my edits causing the display to go bad. (Another lesson in why you should back up your content before you start editing. Fortunately I had. Whew!)

I was stuck. A little experimentation quickly revealed that just loading the file in the Yahoo! Japan editor control and immediately saving it still ruined the Japanese. Not very impressive considering this was a site specifically for hosting and maintaining Japanese websites! (To be fair, this did not happen to every page on the site. Most I could edit just fine.) The problem was that the Yahoo! Japan on-line editor control did not resave the file with the correct encoding. It could read the file, display the file, and edit the file, but not save the file correctly. It was actually rather pitiful. Would it have worked had I been running on a native Japanese version of Windows XP and IE? Who knows? I would hope so, but that wouldn’t help me. I had the same problem in IE and Firefox. Could I have changed my browser settings to fix the problem? I don’t know, but I’ve never had a problem with any other Japanese page.

I was about to resign myself to having to recreate the whole page from scratch and upload it. Then I remembered SlickEdit handles different encodings. So I pulled up the backup copy in the Yahoo! Japan editor control and opened a new blank HTML page in SlickEdit and selected Japanese (Shift-Jis) encoding. After a simple copy and paste into SlickEdit, I had the original contents in a real editor (syntax highlighting, completions, HTML beautifier, parenthesis matching etc..) that could preserve the encoding. In a lot less time, thanks to the features in SlickEdit, I had the changes in again, saved properly to Japanese (Shift-Jis), and uploaded back to Yahoo! Japan.

Thank you SlickEdit.

(But I still hate software generated HTML.)

If you’re familiar with Quake, or pretty much any other FPS, then you know about the deathmatch. It’s my single favorite type of game and, in it’s own way, is really therapeutic when the stress of development, or testing, gets you down. A lot of the other developers I worked with at one point also loved to deathmatch and we used to play during lunch. When testing time came around, we found a way to make it fun… QA Deathmatch.

Here are the rules for QA Deathmatch:

1. Scoring works on a point system: the points correspond to your company’s severity levels for bugs. If you find a bug, you score that bug’s severity level in damage. The person who owns that bug sustains that many points of damage. If your severity level goes in reverse order, then you score ([max severity] – [bug severity] + 1) per bug. For instance, assume your company has a severity level of 1 for critical bugs, and 5 for cosmetic bugs. If you find a bug with a severity of 2, then you score (5 – 2 + 1) or 4 points of damage and the bug’s owner sustains 4 points of damage.
2. Damage can be recovered: by fixing a bug that caused the damage. When a bug is fixed, you reduce your damage sustained by the point total for that bug. However, the person who found the bug does not reduce their damage total.
3. All features are fair game: there are points to be won everywhere in the code base, don’t limit yourself. The more you try all of the features, the more chance you have of scoring big.
4. First to find a bug gets the points: If you find a bug that’s already been logged, you score nothing.

All players have three running stats:

• Damage delivered: The sum of the bug scores found by you.
• Damage sustained: The sum of the scores of the bugs found by other players against your features.
• Damage recovered: The sum of the damage reduced by you by fixing bugs.

At the end of each day, the point totals are taken and five medals of honor are awarded:

• Sniper medal: The person who delivered the most damage
• Stealth medal: The person who sustained the least damage
• Tactics medal: The person who had the best delivered:sustained ratio
• Medic medal: The person who recovered the most damage
• Commander medal: The person with the greatest overall score as calculated by [delivered - sustained + recovered].

The point of all of this is that it makes testing fun. It gives you motivation to find bugs, fix bugs and make your features as solid as you can. It’s amazing what winning one or two medals at the end of the day can do to make you feel great about going into testing for another day. And if you think winning a medal is motivating, wait until you lose a medal to someone else.

Another great thing that QA deathmatch encourages is getting everyone to test all aspects of the system. Too often, developers only test their features and don’t go outside that box. However, there are bugs to be found everywhere, and when you are in scoring mode, you’ll take the time to check out all the new features to see what you can break to score big. It’s also a great facade for getting competitive about finding bugs. Nobody likes the guy that finds a lot of bugs in their features, and even fewer like to be that guy. It’s easy to feel bad about finding a lot of bugs. You’re essentially adding work to other people’s plates. But in QA deathmatch, you are playing a role and it’s a great way to detach personal feeling from the process.

There are other variations of the game too:

1. Team play: Development is split into two or more teams, divided by feature sets. Point totals are not by individual, but by team.
2. Capture the flag: Just like team play, but instead of end-of-day totals, points are calculated when bugs are found or fixed. Instead of medals, flags are used that can be stolen by any team at any time. It feels great to find or fix a bug, then be able to steal a flag from another team.
3. Mercenaries: Testers that aren’t involved in fixing bugs are mercenaries. They may not participate on a team or get any of the normal medals. Instead, a special Mercenary award is given to the one who scores the most damage in a day.

Again, the whole point is to make testing fun. All you need is a whiteboard, a calculator and some gadgets to be used as medals. Of course, you can always add to the game by printing a bunch of these then taping one to a developer’s cube with the bug number and severity penciled in whenever a bug is found. It can only be removed when the bug is fixed. The possibilities for expansion are endless.

QA is a painful process and some days, when 8+ hours of QA work lay ahead of you, it’s just hard to get out of bed in the morning. Don’t let boring work be boring. Not only will you have more fun during the day, but I can guarantee that when you turn QA into a competitive deathmatch, the quality of the final product going out the door will be better than if you are finding and fixing bugs for the sake of quality alone. Happy sniping!

My resolution for 2008 is to get more involved with kids and computers. I’ve been programming since I was 11. I took a course in 5th grade on how to use BASIC. I have no idea how that course was offered or what sparked me to take it, but I went one night a week for a few months and learned to write some simple stuff. I was lucky to have had that course and I wish I could thank that teacher for teaching what was probably considered a novelty subject on her free time. I was also lucky for the fact that my parents were willing to invest what was then an enormous amount of money into my first few computers. I’m sure that at the time they wondered why they were investing so heavily in “toys”, which back then, didn’t manage their money, do their taxes, or calculate my Dad’s golf handicap. Yet, that investment was the start of a love of programming that’s gone on now for over 25 years.

I’ve read a lot of stories lately about how computer science students are on the decline, despite hiring and salary increases in programming. Redmond Developer reports, “[...] survey data showed that since 2000, interest in CS as a potential major had dropped almost 70 percent.” Many other sources report the same trend [1], [2], [3]. As a software developer, that trend bothers me.

My kids are in K and 6th grade, so I see what they’re learning on computers these days. Many kids have access to computers and know how to use them to access the internet, to write and print a report with a word processor, and so on and so forth. These are great life skills which, you could argue, are as important these days as reading and writing. However, there’s very little transition to computer science there, and that’s where most of the computer teaching seems to end.

It’s a shame too, because computers are doing such cool things these days. The games, the web sites, the music applications… there’s so much there that a grade school kid can connect with and see as being essential to their day to day activities. With all of that coolness, you’d think that kids would be knocking down the doors to learn how to make that kind of stuff.

The problem is that you can’t learn to make cool stuff like that the first time you sit down to program. Not the tenth and probably not the hundredth time either. In order for kids to get from the point of being users to being developers, there has to be some motivation to keep them interested. Rob Walling wrote a great article about how to make programming seem cool again to young kids. He’s got the right concept for the hook and I tried to memorize the “Show the Steak” section like it was a script.

But a hook is not sustainable. What kids really need to keep their interest are small, recurring successes. For me, Compute! magazine provided those little successes in the form of one page programs I could type verbatim to get a working game. After a few straight copies, I started tweaking their games, then I added new abilities to those games and then I was writing games from scratch. Once my momentum had reached that point, I really wanted to learn more.

So what are those little successes these days? There’s no more Compute! magazine or anything like it. Even writing a simple program these days can seem impossible because of IDEs, compilers and all sorts of things that precede writing a single line of code. There are some great projects out there like Phrogram, Scratch, Basic for kids and LOGO. Amazingly, despite the unlimited access to this stuff, most of it seems to go relatively unnoticed.

So now that I’ve made my resolution, how do I actually make it happen? Well, the other day I was talking to my son’s Kindergarten teacher before class. She was telling me about a new web site she had set up for the class… a blog where she could post information about what they were up to in class. She’d taken a class about it and was learning HTML. We talked some more and she mentioned her interest in the Give One, Get One program. I’d never heard of it. It turns out that it’s part of the One Laptop Per Child (OLPC) program, an initiative to get affordable laptops to underprivileged children of third world countries. I had definitely heard of that. Originally, this laptop was not planned to be sold commercially, but you can get one by donating one through the Give One, Get One program. This teacher was interested in getting one to use in the class, but didn’t know much about it.

So there I was with my kid’s teacher willing to learn how to use this new device to teach. We put together a small plan, and I bought one a few days ago. She’s buying one also, so there will be two in my son’s class. I’m going to help her learn what it can do and show her things the kids can do with it beyond web surfing and game playing. In particular, I’m interested in the LOGO program it comes with and how that can be introduced to Kindergarteners in the form of repeating patterns (a heavily stressed concept in the lower grades). My highest expectation is that they think it’s fun to tell a pen-dragging turtle where to go.

I don’t exactly know where this is all going, but it feels good to be doing something. I’m excited about getting involved in helping kids learn computers, a subject I’ve spent lots of time thinking about and very little time doing anything about.

The more a task resembles a previously completed task, the easier it is to estimate. Ideally, metrics kept about the previous task will include time spent on various activities and the total time overall. When estimating the new task, you could analyze which activities are likely to change based on completing the previous one and derive an estimate for the new task.

In order to best illustrate the challenges of software estimates, I will use the time-proven technique of the parable.

The Parable of the Cave

Two people stand before a cave. One is the sagely manager of a cave exploring company who’s wisdom is only exceeded by his wit, charm, and humility. Let’s call him, oh, “Scott”. The other is a cave explorer of indeterminate gender who bears no resemblance to any programmers past or present that this author may have worked with and whose size may be big or little. Let’s call him/her “Endian”.

“Endian,” said Scott in a sagely voice that was both commanding and compassionate, “I need you to explore this cave. But before you do, I need to know how long you think it will take, so that I may build a schedule and tell the sales team when the cave will be ready.”

“Great Scott,” replied Endian using the title bestowed upon Scott by his admiring employees, “how can I give you an answer when surely you know I have never been in this cave before? The cave may be vast, with deep chasms. It may contain a labyrinth of underwater passages. It may contain fearsome creatures that must first be vanquished. How can I say how long it will take to explore?”
Scott pondered Endian’s words and after a thorough analysis that might have taken days for others but was completed in but a moment for him, he replied, “Surely this is not the first cave you explored. Are there no other caves in this district? Use your knowledge of those caves to form an estimate.”

Endian heard these words and still his doubt prevailed. “Your words are truly wise,’’ said Endian, “but even within a district the caves may vary, one from another. Surely, an estimate based on the size of another cave cannot be deemed accurate.”

“You have spoken truly, good Endian,” replied Scott in a fatherly, supporting tone that lacked any trace of being patronizing as certain cynical readers may think. “Here, take from me this torch and this assortment of cheeses that you may explore the cave briefly. Return ere the morrow and report what you have learned.”

The parable continues like this for pages, as parables are known to do. Let’s see, Endian enters the cave…something about a wretched beast of surpassing foulness…he continues on…hmm, that’s what the assortment of cheeses were for. Ah! Here we go.

Endian returns to Scott, his t-shirt ripped and his jeans covered in mud. Being always concerned with the well-being of his employees, Scott offers Endian a cool drink, then asks, “Endian, what news of the cave? Have you an estimate that I can use for my schedule? What shall I tell the sales team?”

Endian considers all that he has seen and builds a decomposition containing the many tasks necessary to explore the cave based on his earlier reconnoitering. He factors in variables for risk and unknowns, and then he responds, “Two weeks.”

Applying this Wisdom

To truly appreciate the punch line, you do need to check out The Perfect Estimate.

The point I’m trying to illustrate is that estimating software development is like estimating the time needed to explore a cave you’ve never been in. You don’t really know until you get in there. Still, there are techniques that can help.

First, draw on your experience. Even though technologies may be very different, few automation problems are empirically new. They bear resemblance to other things we’ve done in the past.

Expect the unexpected! Make sure you factor time in for the unknowns. If your schedule only contains the things you can think of, then it’s certainly going to fall short when the unexpected arises. There will always be things you don’t know to put into the schedule.

Provide estimates that are ranges rather than just a single duration. If a task can take 2 to 4 weeks, provide it in that manner. If your project planning software cannot accomodate date ranges, enter the task as 2 weeks and then put in a secondary task for another 2 weeks to account for this. I like to name these tasks, “risk factors”. So, if I have a task called, “Integration Testing”, I would add another called, “Integration Testing, Risk Factor”. This way, I can specifically see how much time has been set aside for this variability, and I can see what happens to the schedule if I increase or decrease that amount of time.

Make your estimates as fine-grained as possible. An estimate for a task that only needs a few hours is far more likely to be accurate than an estimate for one that will need a couple weeks. By making your estimates as detailed as possible you not only improve accuracy but the work necessary to create a detailed estimate will often turn up tasks you might otherwise overlook.

Even if your manager only wants an overall estimate, you should do a detailed decomposition to make sure you build the best estimate possible. This decomposition will also serve as a guide during your work, both as a checklist to make sure you haven’t forgotten something and as a status check to see if you are on time. This can help you timebox activities, applying the available time to a task. Projects can run over due to time spent waffling on design decisions. Timeboxing tells you when it’s time to make a decision and move on.

Build your decompositions with tasks that are either complete or incomplete. When tracking progress, don’t fall into the trap of recording activities as some percentage complete. A task is either finished or not finished. Showing partial completion leads to a false sense of security.

Use incremental and iterative processes. As in the parable, sometimes it’s useful to make a quick, preliminary exploration in order to build a more comprehensive estimate. Continue to refine your estimates and the overall plan. The closer you get to the end, the more accurate your schedule should be.

Work on the riskiest part of the system first. Too often projects run over when someone discovers a large problem in the final stages of the work. Assess the risk of each portion of the work and tackle first those parts that are most likely to exceed their estimates or induce changes in the rest of the system.

Lastly, it is absolutely critical that an environment of trust exists between the development team and the rest of the company. Your job should be to deliver the best estimate you can with appropriate risk factors identified. Too often estimates are inflated to make sure that we can hit them. Equally often, due dates are set artificially early to allow for overages. Both of these strategies prevent the company from building an effective plan for this project and developing appropriate contingencies. Worse, these constraints sometimes lead to poor choices and compromises during development that add further risk.

Certainly shortages arise in specific specialties as some technologies become more popular. Likewise, some geographic areas may see more competition for programmers than others. As lovely as it is, not all of us want to live in California. These articles are talking about large-scale, industry wide shortages and not the kind that can be explained by local factors. So what’s going on?

I believe that there are a number of problems at fault.

Programmer Shortage as Means of Lowering Programmer Pay

When Mr. Gates decries the lack of available programmers, he offers the solution that we should increase the number of H-1b visas. Indeed, if there truly is a shortage then this seems like a reasonable solution. However, this is more likely a ploy to address the true problem that these companies aren’t willing to admit: there aren’t enough programmers available at the salary they are willing to pay.

I worked at a company that would post positions that had ridiculous job requirements that almost no one could meet. They would then use the unfilled positions as justification to bring in programmers who were on H-1b visas that were, frankly, no more qualified than the American programmers that were rejected. The ones I knew were paid far less than their American counterparts.

This is a peripheral issue and not what I deem to be the problem at most companies.

What is a Qualified Programmer?

The trouble is that many companies don’t know how to identify a qualified programmer. I say that because their job postings frequently contain combinations of specific technical abilities stated in years of experience. In a previous blog, The Mythical Year of Experience, I discussed why that approach is a failure.

Even without relying too much on years of experience, most companies put too much emphasis on experience with a specific combination of technologies. This includes things like Oracle, Struts, .NET, XML, etc. I consider these to be extrinsic abilities. Knowledge of these things must be specifically learned and aren’t generally applicable to other areas. In contrast, intrinsic abilities are more portable and can be applied in many areas. These include problem solving, communication, and the ability to learn quickly.

I’m not saying that you should hire people who are totally unfamiliar with the technologies you are using, just that this need is too often overstated and the value of the intrinsic abilities is rarely weighed.

Let’s take XML as an example. Several years ago, I worked on a project that processed mortgage applications. We decided to use XML to store the documents while they were being processed. Those of us who designed the document structure and laid the fundamental access patterns needed a thorough knowledge of XML. Once that was finished, I was able to show my team members how to access fields very quickly.

If we had an opening on this team, many companies would make the mistake of requiring a couple years of experience in XML since it is so important to the project. A good programmer, who learns quickly, would be able to get up to speed on our use of XML in no more than a day.

To illustrate the importance of these intrinsic abilities, let’s look at another project I worked on. This was a C++ project where we had a requirement to run on several platforms. We ran into portability issues with the C++ code and decided we would be better off if we rewrote the system in Java. Though none of us had experience with Java, we quickly learned the language and proceeded with only a few minor hiccups. Our intrinsic abilities as great programmers allowed us to quickly adapt to this change.

The one constant in technology is that it is going to change. Hiring people with a greater focus on their intrinsic abilities will create a stronger work force that is better able to adapt to change. This will help your company avoid the short-term programmer shortages created by these shifting technologies.

How Do You Recognize a Qualified Programmer?

You can also tell how much trouble companies have with this by looking at their interview processes and the questions they ask. They are as likely to hire a truly effective programmer using their approach as if they had candidates randomly draw “Hire” or “Don’t Hire” cards from a deck. After some of the interviews I’ve been through, that approach would have been more satisfying and more respectful of my time.

Most companies ask the same old questions about your previous experience. You spend your time describing what the project did, not what you did. Any member of your team is likely to be able to answer these questions just as effectively, regardless of how little they actually contributed.

Some companies ask ridiculous questions with no specific answer, like “How many manhole covers are there?” They then look for candidates who try to reason out an answer. Or they might give you some kind of problem to solve about weighing balls or crossing a bridge. Tell me, what analog does this have to your daily programming tasks? Sure, problem solving is important but something a little more closely related to the programming domain might be nice. These problems too often rely on a flash of insight, which may be difficult to come by when standing in front of strangers with a potential job on the line.

Some companies ask questions that are more pertinent to programming, but get off in the weeds on very detailed algorithmic knowledge, like “How do your reverse an array without a temporary variable?” Or “How do you detect a loop in a graph?” These are good tests of your memory if you’ve been exposed to them before, but you will probably not puzzle out a great solution during an interview. Certainly, you could look up the answer more quickly, which is what you would do on the job.

There are jobs for which these kinds of questions may be suitable. But for most, they will just help you disqualify good candidates. Keep in mind that the answers to these kinds of questions seem more obvious to the interviewers because they already know the answer and have watched several candidates wrestle with them. Because of this, interviewers tend to be unduly critical of those who do not give the optimal answer.

No Unqualified Programmers?

So I’m saying that there are no unqualified programmers? NO! As a manager who has interviewed my fair share of candidates I know first-hand that many of the applicants are unqualified. The sheer volume of candidates whose knowledge and abilities fall short of your expectations can easily lead you to believe that there are no qualified programmers.

Like any population, the bell curve is at work with programmers. The highly talented, top contributors that you seek are outliers in that population. You will need to screen many, many candidates to find the ones you need. Just be sure you’re screening for the right abilities.

So What Do I Do?

When hiring, consider what you truly need in your candidates. Don’t fall into the trap of just listing all the technologies you use. Determine which can be learned on the job and which truly require experience.

Examine your interview process. How does each step specifically help you identify the kind of programmers you need? Are you properly balancing intrinsic and extrinsic abilities?

Get your most talented engineers involved with screening candidates. “It takes one to know one.” Great engineers may not be able to describe how to recognize another great engineer, but they know one when they interact with one. The time they spend interviewing candidates will pale in comparison to the time they spend fixing problems created by less capable programmers.

I have spent many a late night over the past few years obsessing about my digital music collection. Whether it’s ripped from an old CD or purchased download, I have to have it labeled correctly. But this is a losing battle. Virtually all players are designed for standard pop albums with the artist, album, and song model. That just doesn’t cut it for real music. (OK. I’ll be honest. It bugs the heck out me when Beethoven’s Opus 132 gets referred to as a song.)

Anyway, I got hooked on digital music files a few years ago because the computer had a bad cd-rom drive in it. I couldn’t play a CD on it, but it would read the data and could rip a CD. So I started slowly moving my CD collection over one disk at a time. Well, I didn’t start out with iTunes, I tried a variety of players but mostly used the ziff player. The big decision is what format to use. I chose .ogg. There was no real reason other that I used CDex to rip the same track in a couple of formats at 160 kbps and .ogg sounded the best. The files were smaller as well. The difference wasn’t much, but I felt the .ogg files sounded warmer, especially the string sound. There are no end of websites and blogs promoting one standard over the others, but I’m not going to go there. I just made the decision early on that ogg was the best for me. My collection is not that large, a few hundred CDs, and it took a few years to convert the collection. I wasn’t actively trying to rip everything, but whenever I played a CD if it was on the computer with the bad drive, I would rip it. I eventually had most of my disks ripped and for the most part tagged.

I’ll spare you my discussion on what I think is the best way to tag to your classical music files. Instead, check out the following pages (especially the second) to see how the debate rages on. And these guys just cover what to do with iTunes. I’m sure there are many others.

Tagging Classical Music
Taming iTunes for Classical Music
On Classical Music Tagging (ID3 tags) for iTunes and iPod

Each of these is quite individual and so naturally, I neither totally agree or disagree with any of them. I applaud their careful work, but each method suffers from the same agonizing flaw. All they do is try to make a broken system appear less broken. I don’t want to start down that road. (However, for another time, I have some advise for using iTunes with classical music. Anyone else out there used the iTunes COM for Windows SDK?)

Instead, I ask why are we so stuck on the primitive data model of a single track stored as a single row in a table? Relational database anyone? Authority records? We could even do something pretty good with a little XML.

I just randomly pulled a CD off my shelf. It is a Telarc disc of orchestral music by Barber performed by the ASO and Yoel Levi. When I stick this into iTunes, rather than have the Gracenote’s database give me a few choices for the CD and then fill in the rows with poorly formatted strings, it should just give me a link to an authority record (or some XML) for the CD. This would contain accurate information for composer, compositions, and performers etc. Then within the player, I could choose how to display this information for the CD. Does the Name column contain both the full piece name and the movement name or just the movement name? I could define this through some kind of options dialog in iTunes or just give it my own xsl translation.

I know, you’re wondering how are all these authority records created anyway? The same way they are now, by having users submit them. If I were to put in a CD that was not yet in the Gracenotes database, I would fill in the crucial data much like I do now in iTunes. For composer, I’d highlight the tracks on the CD by Barber and then iTunes and its database would give me a list of known composers to choose from. Pick another track, and it could provide a list of known compositions by Barber etc. Starting with a core database of a few hundred composer records, a few thousand composition records, and a similar number of ensemble and performer records, one could quickly create a new complete CD record for the database based on existing records. Of course, new records for composers and compositions could be submitted and vetted by the database. Most of the CDs would not need to submit any new data beyond what compositions and performers appear in the release.

If you think this data still would be too hard to create or put on-line, then check out ArkivMusic. Granted, ArkivMusic is a retailer and their database serves a different purpose, but they’ve done a whole lot of things correctly. At this site, I don’t never have to differentiate between Samuel Barber and Barber, Samuel, or between Atlanta Symphony Orchestra and Atl Sym Orch. Why can’t putting music in my digital music player work half this easily?

To be fair, I’m certainly not the only one out there grappling with these questions. Kirk McElhearn has started a project called The Well-Tempered Database, though it still seems to be getting off the ground. Of course, if you have the means (~$5,000) you could always get the Fortuna Maestro Classical Music Player and have someone else do your cataloging for you. Let me know about any other projects there may be.

I’ll also take this opportunity to put in a plug for eMusic. Here is the other side of the coin. They have incorporated classical music into their business model just as they did for noise downloads (pop music), one price per track. Their standard fee for 30 downloads is $9.99/month, or ~33 cents per track. Good price for your average 3 minutes noise track, but what a bargain if you go for a big piece of real music like Beethoven’s Ninth. The four tracks cost you $1.33 and you get 65 minutes of music at a higher quality than the standard iTunes download. That’s a steal! The drawback is that it is a subscription service. Ten dollars a month with a use it or lose it policy.

So I admit that I am a classical music snob, but it’s mostly in good humor. I don’t care what you listen to as long as I can listen to what I want to. As proof I offer this recent photo of me playing jazz/rock violin with the band Fujiyama Roll. (Sorry for the low image quality.) I really am a hip guy who can get down to some funky music. Far out, man!

Rock star David O

• Letting the user manage a collection of objects with a dialog
• Creating an MRU menu
• Running a task on a background thread
• Paging a large set of data in a grid
• Writing drag-and-drop code

There are many bite-sized tasks that programmers face every day. It’s the unglamorous and tedious side of programming. Most developers spend the far majority of their time tackling these tasks that are often too specific to be effectively handled by a general purpose framework. Yet, there is a commonality to most of them that all programmers will recognize and respond to with something like, “Oh yeah, I’ve done something like that before.” Reusable frameworks that support the core of a software project are crucial to its long term stability and scalability, but most programmers don’t spend their day working on them. The list of bite-sized tasks is just too big and specific to be encapsulated in a such a framework.

Helper classes

Some frameworks have tried to address these tasks by creating single use objects designed to provide a general starting point. For instance, .Net provides the BackgroundWorker class to help run tasks on a background thread. It’s a very good class and I use it whenever I need to run tasks asynchronously, but it’s a standalone class. It has no relationship to any of the other objects in the .Net framework. There are many classes like this that I would not categorize as “framework” classes… they’re just helpers.

Generalized helper classes certainly serve a need in software development, but even those can only carry the ball so far. For one thing, you still have to bridge the gap between the generalized functionality a helper class provides and the specific functionality you are trying to accomplish. Sometimes this can be just as challenging as not using the helper class at all. Other times, the helper does 90% of what you need and the remaining 10% makes using the helper class impossible. If you can’t get inside the helper class and tweak it to do that extra 10%, then the helper class becomes useless for that task.

Where reuse falls short

This doesn’t mean that those bite-sized tasks haven’t been tackled before, though. On the contrary, they’re tackled over and over and over again by different developers in different companies that never know of each others’ existence. This is a big hole in the state of reuse in today’s software development.

However, outside of the commercial frameworks and components are endless code samples cataloged on the internet. Sites like Source Forge, Code Project, Koders, Google Code Search, CodePlex, O’Reilly Code Search, etc, provide endless amounts of code that do those little tasks. Disassemblers are also available to get the source code from compiled .Net or Java assemblies. Need to read a comma delimited file? Somewhere, someone has done that before and it’s probably on the internet. Find a project that does that, look at the code and use it… it’s that simple.

Ok, maybe not. In order to do that you have to do “code scavenging”. Doing this involves the following process:

1. Finding public code that does what you need to do.
2. Scanning the code and pulling out just what is needed to accomplish the task.
3. Tweaking the found code to make it work with your own code.

Many people will stop reading here and argue that this is software reuse at it’s worst… cut, paste and retrofit to solve the problem. My answer to that is… yes, it is, and that’s ok. If you think about the hierarchy of a software project, the farther we get from the core, the more task-specific the code becomes. At some point in that hierarchy, you reach a point of diminishing returns on object oriented reusability. Striving for reuse simply for reuse’s sake at this level will hurt the code, rather than improve it. These task-specific extremities in the software hierarchy most likely represent the bulk of the day to day coding work. That’s not being pessimistic about reuse, it’s being realistic.

A new methodology

I picture the “code scavenging” style of programming as a new methodology of reusable software development. Practicing this type of reuse is very different than practicing component or framework reuse. It requires resourcefulness to find sample code, technical knowledge to quickly understand the found code, and insight to know how to retool that found code into your own code. Applied properly, though, it can turbo-boost productivity and quickly solve problems that might otherwise take days to work out.

Unfortunately, there doesn’t seem to be very many tools or documented techniques for this type of programming. Some authors have mentioned it ([1], [2], [3]) but overall, little work has been done on how to do it effectively. There are several barriers to code scavenging as a methodology as well. Web sites hosting code samples are disconnected and sometimes poorly cataloged. Samples are not guaranteed to be good or even work properly. They can also be copyrighted by the original author under any number of legal bindings.

Despite the difficulties, I believe there is a need to develop a methodology around code scavenging. This methodology would address best practices in the following areas:

1. Finding sample code: Finding samples that fits your needs is not always as easy as a quick Google search. What are the techniques for finding quality sample code?
2. Determining the root code sections that perform the required task: The root code is the minimum set of code that performs the task you are trying to accomplish. This may be in one or several different locations in the sample. What are the best ways to identify these areas?
3. Finding the “incision points”: Incision points are the areas of code in the sample where you can “cut” that still enables the root code sections to work. This includes initialization functions, non-local variables, etc. How are these best identified?
4. Merging the extracted code into your own work: This is the area where a set of documented techniques would shine. What is the best way to add the found functionality without having to rework a lot of your existing code? Even better, how do you keep the sample decoupled from your code in case it fails to work as you expected. Are these techniques the same for UI vs. back end work?

I certainly don’t have the answers to all of these questions, but I have a lot of ideas and I’m really interested in exploring this further. Code scavenging is a technique that I use often… it’s practically replaced paper books as my method of learning new concepts. Sometimes I just use it as a means to understand a concept before working on a problem. I even used a sample project to learn how to use the BackgroundWorker helper class mentioned earlier. As I said in another blog post, “Just like a picture is worth a thousand words, a sample project is worth a thousand reference books”.

This collection of tips come from the top of my head — think of it as the sort of advice I would pass on to my daughter if she ever found herself writing a memory intensive C++ application. For the record, she is six years old, so please, no complaining if this advice seems obvious or academic to you.

Oh, and disclaimer #2. This article is for C++ programmers only.

1. Pointers need well defined owners

Whenever you do an allocation it is important to identify an owner for that pointer — someone, something that is responsible to delete it when it is done with it. The owner also defines the pointer’s life span. Sometimes the ownership of an allocation can be by committee, for example, when you are using reference counting. Managing the lifetimes of allocations is the most important rule of memory management. If you consistently make ownership and lifetime absolutely clear and obvious in your code, you will avoid lots of headaches and bugs. This is the core of our second admendment rights, the right to own and bear pointers. The grantation of rights comes with a responsibility to bear. Treat pointers with the respect you would treat a firearm — and you may not find yourself shot in your own foot.

2. Use const

If you pass an object to a function by pointer and do not intend to modify the object, mark the object (not the pointer) as const.

Some programmer’s eschew const because they believe that they know better than to modify an object at an inappropriate time. That’s good for them, but if you have any real world experience, you know that you will not be the one maintaining your code for the rest of it’s life. Putting these guardrails in place will prevent a rooky from putting in a bad side-effect then asking to you to figure out why everything is broken.The disadvantage of using const is that it frequently will double the number of getters you need. A very typical C++ idiom is the const and non-const getter:

This class clearly owns a pointer to an object of type T, and it will only surrender that pointer on equal terms. You have to have a non-const reference to someClass to get a non-const pointer to T.A final warning about const. Avoid casting away const-ness. Sometimes, it will be unavoidable. Nevertheless, your mindset should not be “oh, I just need to cast, ok, done.” You should always say to yourself “self, what have I done wrong? Maybe I am not using this code the way the author intended.” If someone took to the trouble to make a variable const, they probably had a reason.

3. Use the new, safer C++ style casts

C++ is quite willing to let you cast any pointer type to any pointer type using conventional C casts. Using the new C++ style casts will help you insure that your casts are safe and sensible.

4. Avoid pointer arithmetic

There are two ways to walk over an array represented by a pointer. Some coder’s prefer this:

This is fast, and effecient, but not easy to trace, especially when multiple pointers are involved. And, by the end of the for loop, you have lost the original value for p. Yes, you can recalculate it, or save and restore it, but that just adds to the complexity. I prefer this approach instead:

This avoids the pointer arithmetic and is easier to read. Let the compiler identify the loop invariant and do the optimization for you. Some poor compiler engineer spent a lot of time working out that optimization. Think of that poor guy, or the next person who’s going to have to read your code. It’s also important to remember that the performance gains you get from using C++ and doing your own memory management will eclipse any peephole optimizations you can achieve using pointer arithmetic. Paraphrased: your code is already fast — now make it easy to read.

5. Use new and delete

Using new and delete ensures that constructors are executed, which ensures that your member variables are initialized. For C code, use malloc and free. There are situations where you will want to write your own allocation routines when you have to squeeze out an extra ounce of performance, but do so with discretion. Why is this important? Because if you do your own memory management, you are bound to make mistakes. There are tools that will help you, such as Rational’s Purify or Numega’s BoundsChecker, but often using your own memory management masks the precise origin and granularity of allocations from these tools.

6. Beware of new[] and delete[]

Many memory leaks and other crashes originate from inconsistent usage of the array allocation (scalar allocation) primitives. The most common mistake is using new[] to allocate and delete (non-scalar) to deallocate. The effect is that you construct an array of objects, then only destruct the first one. Depending on the nature of scalar new and delete, this may also segment memory, cause a crash, or throw a hard exception. The opposite error, using new to allocate and delete[] to deallocate will sometimes go undetected. But, again, it depends on the implement of scalar new and delete. I advise you to find or write an array or vector template class that you are comfortable with, and use that instead any time you are tempted to use new[] and delete[]. This is an area where C++ should have been more strongly typed. Currently the type returned by new T[n] is simply a pointer, T*; it should be an array T[]. Currently C++ makes little distinction between these two types. Distinguishing the types would have allowed the compiler to detect an improper use of delete. It would have also forced programmers to use the array type when they are treating memory as an array and a pointer when it is a pointer to a single item. Taking things one step further, operator[] would not even be allowed for pointer types.

7. Bound arrays

If you do not make use of an array or vector template class, you need to bound your arrays. I prefer to keep track of the number of items allocated. Avoid null terminated arrays — history should have taught us that lesson by now.

8. Initialize pointers to NULL

If you are not immediately initializing a pointer to a valid value, always initialize it to NULL. There is just no excuse other than laziness to write this:

instead of:

With optimizing compilers, it is very, very seldom that this will introduce even an extra cycle to the performance of your code, and will protect you from leaving a pointer un-initialized and later doing this:

You should also assign pointers back to NULL after deleting them, like this:

Again, this traces back to pointer lifetimes. Pointer variables should be NULL whenever they are not pointing to valid memory. To take this to an extreme, I even like to do this in destructors. Think of it as analogous to wiping your fingerprints at the scene of a bank robbery. Leave the place as clean as it was before you showed up. Once memory is deleted, leave no trace that it ever existed.

9. Always test pointers before using them

Do:

Avoid:

This is another peephole optimization that some lazy programmers defend. I admit, there is some merit to the optimization, it avoid a branch test, and we all know that branches clog CPU pipelines. However, unless the function’s contract (documentation) specifies that it can not handle a NULL pointer, you need to be prepared to handle the case in an appropriate manner. A good rule of thumb is to only use pointers when the object is optional to the function.

10. Use pass by reference for non-NULL objects

Do:

Avoid:

Using a reference type makes it clear that someFunction expects a non-NULL object by contract. This also avoids the pointer check mentioned in the previous section.

11. Avoid acquiring pointers to non-heap memory

Avoid:

Otherwise, you may forget that p is pointing to an object whose life is over when the function returns, which will lead to all kinds of dangerous things.

Summary

There is a great deal of over-inflated talk about the dangers of using pointers and the difficulty of doing your own memory management. Experienced Java programmers will even tell you that if you don’t know how to manage memory correctly and establish documented lifetimes for objects, you will have all kinds of memory problems including leaks and painful garbage collection delays. That advice is coming from people who have garbage collection. I submit that learning how to manage memory well is a skill that is actually easier to learn by programming in C++ and using pointers in a sensible manner. These skills are not arcane remnants of a programming language that is past it’s prime. They will make you a better programmer in whatever language you need to use in the future.

SlickEdit C# Project Tutorial

In this tutorial, we’ll build a simple C# console application with SlickEdit. This tutorial assumes you have the .NET Framework 2.0 and the C# compiler (Csc.exe) installed under %WINDIR%\Microsoft.NET\Framework\v2.0.50727. The Windows SDK (v6.0 or later) or the full .NET Framework SDK is required for the mdbg.exe managed code debugger.

Conventions

Dark blue italics represent menu commands: File > New
Bold items represent names of dialog buttons, labels, and controls: Location:
Monospaced fonts are used for environment variables, file paths and names, and text that you type into dialog controls:
Several screenshot links are provided that show how the dialogs are used.

Creating the starter project

• In SlickEdit, go to Project > New. Create a (generic) project. You probably want to use the Create project directory from project name option. Change the Location: to a directory where you want the project to go. Click OK to create the starter project. Dismiss the project properties dialog if it appears afterwards.
• Go to File > New Item from template…  Under Installed Templates \ CSharp, highlight the C# Main Entry Point template. Change the Name in the lower portion of the dialog, and check the “Add to current project” checkbox. Click Add, and the new file will be created. It should appear under the Source Files project folder. Edit the Main() method. Add the following code inside the Main()

  1 2 3 4 5

  #if DEBUG Console.WriteLine("Hello World - Debug"); #else Console.WriteLine("Hello World - Release"); #endif

• Go to Project > Workspace Properties to display the workspace properties dialog. A SlickEdit Workspace (.vpw) is the equivalent of a Solution (.sln) in Visual Studio. Click the Environment button to display the Workspace Environment Options dialog.
• Click Set Environment Variable . Create a new variable named DOTNETDIR , and point it to your .NET framework that you want to use for the C# compiler, like C:\Windows\Microsoft.NET\Framework\v2.0.50727. To use an existing environment variable in the definition of the workspace environment use the %(VARIABLE) syntax, and not the %VARIABLE% syntax. For example: %(WINDIR)\Microsoft.NET\Framework\v2.0.50727
  Be sure to select environment variable names that do not already exist.
• Create another variable named WINSDKDIR , and point it to the Windows SDK or .NET Framework SDK, like C:\Program Files\Microsoft SDKs\Windows\v6.0 . You want this to be a directory where the mdbg.exe managed code debugger can be found.
• Click OK. You will be prompted that you need to close and reopen the workspace for the variables to be set.

Setting up the build command

• Once you’ve closed and reopened the workspace, go to Project > Properties to show the dialog. In the Settings for: combo, select Release, which should be the only other choice right now.
• Go to the Tools tab and remove the Compile and Rebuild tools. (Use the red X button) This should leave Build, Debug, and Execute.
• Highlight the Build tool. Change the Command line: entry field to the following:

  1	%(DOTNETDIR)\csc.exe /warn:3 /target:exe /define:TRACE /debug- /out:%bdSeHello.exe %{*.cs}

  The %bd in the above command line is a variable that represents the build output directory, in this case Release\.
  The %{*.cs} construct means all project files that end with the .cs extension

• Replace SeHello.exe with whatever you want the exe to be named. You shouldn’t need to change the Run from dir: entry field as it should already be %rw
• Make sure the Capture Output and Output to build window options are selected.
• Highlight the Execute tool. Change the command line to “SeHello.exe”, or whatever your exe output name is from the Build tool command line. Change the Run from dir: option to %bd
• Make sure the Capture Output and Output to build window options are selected.
• Click OK to close the dialog and save the changes. You should now be able to execute a build from Build > Build , and see the output of the C# compiler. If the build succeeds, Project > Execute should run the console program and display the “Hello World-Release” results.

Setting up the debug build

• Open the project properties dialog again, and click Configurations . Click New to create the Debug configuration, and copy the settings for the Release configuration.
• Click OK to create the Debug config, and dismiss the dialogs.
• Back on the project properties dialog, highlight the Tools tab, and select Debug from the Settings for: combo box.
• Highlight the Build tool name , and change the Command line: entry field to:

  1	%(DOTNETDIR)\csc.exe /warn:3 /target:exe /define:DEBUG;TRACE /debug+ /out:%bdSeHello.exe %{*.cs}

  All we did was add the DEBUG define and change /debug- to /debug+. Again, use your exe name in place of SeHello.exe if you want.

• Highlight the Execute tool name, and make sure it is the same as you set up for the Release configuration.
• Click OK to save and close the project properties.
• Go to Build > Set Active Configuration … and select Debug.
• Execute the Build > Build command. If the build suceeds, run the Build > Execute command. You should see the “Hello World – Debug” output in the build window.

Setting up the console debugger

• Back on the project properties dialog, highlight the Tools tab, and select Debug from the Settings for: combo box.
• Highlight the Debug tool name, and change the Command line: entry field to:

  1	%(WINSDKDIR)\Bin\mdbg.exe SeHello.exe

  Again, change SeHello.exe to your exe name as needed. Run from dir: should be set to %bd

• Make sure the Capture Output and Output to build window options are selected.
• Click OK to save and close the project properties.
• Run the Build > Debug command. The build window should have focus with a blinking cursor just after the first mdbg> prompt.
• Enter the following commands in order, pressing <enter> after each one.
  • print args to show the value of the args variable.
  • next to do a step over
  • go to continue execution
  • quit to quit the debugger (IMPORTANT!)
• The entire debug session output should look like the following.

C:\Dev\Lab\TestProjects\SeHello\Debug
> C:\WinSDK\Bin\mdbg.exe SeHello.exe
MDbg (Managed debugger) v2.0.50727.312 (rtmLHS.050727-3100) started.
Copyright (C) Microsoft Corporation. All rights reserved.
For information about commands type “help”;
to exit program type “quit”.
run SeHello.exe
STOP: Breakpoint Hit
16: static void Main(string[] args) {
[p#:0, t#:0] mdbg> print args
args=array [0]
[p#:0, t#:0] mdbg> next
19: Console.WriteLine(“Hello World – Debug”);
[p#:0, t#:0] mdbg> go
Hello World – Debug
STOP: Process Exited
mdbg> quit
C:\Dev\Lab\TestProjects\SeHello\Debug
>

Handling complex build commands

A small console application like this one doesn’t have any extensive dependencies, and most of the default options for the C# compiler are fine. However, more complex projects will require many more options to be passed on the command line. For these cases, it can be useful to create an options file for all of the command line switches.

• Go to File > New to create a new text file and add it to the project. For this example, we’ll create the command line options for the debug build in a file called Debug.opts
• Edit the new options file and put in the following contents, which should match your first few options on the Debug build command line:

  1	/warn:3 /target:exe /define:DEBUG;TRACE /debug+

• Open the Project Properties dialog to the Tools tab, select the Debug configuration, and highlight the Build tool. Change the build command to:

  1	%(DOTNETDIR)\csc.exe @Debug.opts /out:%bdSeHello.exe %{*.cs}

• You can now run the Build > Build command to make sure the options file was correctly read.

The Coors Brewing Company of of Golden, Colorado, USA, had a fairly profound impact on the thirty-somethings in the world like myself.  You see, we have been led to believe for thirty years or so that it was once illegal to transport Coors beer East of Texas.  As far as I can tell, this was never true (although it was not nationally available until sometime in the 1990s).  Nonetheless it was the plot of the hit 1977 movie Smokey and the Bandit.  And when I say “hit 1977 movie”, it was the second highest grossing movie of that year.  Star Wars was number one.  It is not Scorcese or [fill in name of artsy director you like here], but it was a big deal.

I don’t want to get off on a big tangent here, but a small one is necessary, stick with me.  For anybody who did not get basic cable over the last thirty years, the basic plot is as follows:

• It is illegal to ship Coors beer East of Texas
• Burt Reynolds and Jerry Reed are paid a large some of money to get 400 cases Coors beer from Texarkana, Texas to Atlanta, Georgia in twenty-eight hours
• Burt Reynolds distracts cops from the speeding eighteen wheeler carrying the 400 cases of beer by performing really cool stunts in a 1977 Trans Am
  • In order to keep a low profile, the eighteen wheeler is painted with a mural of a masked bandit robbing a stagecoach.
• Jerry Reed sings his hit song, “East Bound and Down”
• Hillarity insues

Sounds ridiculous right?  Well, it does lack the realism of “Die Hard”, but it is actually not a bad movie provided that:

1. You are a guy
2. When you watch a movie you want to be entertained and don’t question things too much.

But the point here is that history was changed:

• The mystique of Coors beer was further boosted to those who could not get it East of Texas.  Here I need to emphasize that people thought it was really illegal to take Coors beer East of Texas.
• The Trans-Am became as cool as the (then recently jailed) Evel Knievel with suburban pre-teens across America.
• Jerry Reed had a hit single.

As much as I love old Trans-Ams, and that song, what I want to focus on is the first item in that list: People believed that it was illegal to take Coors beer East of Texas.  This is one of the standout facts people in the United States know about Coors beer.  What they do not know is this: in 1959 Coors was the first beverage in America to be packaged in a two-piece aluminum can.  These cans are a part of common life today.  There are two on my desk right this minute (diet soda, not beer).  They are so common place people my age really cannot picture a world without them.  A world where I am told you had to use a can opener (one of the pointy triangluar ones, not the rotating ones with a blade) simply to drink a beverage.  I imagine the people at The Coors Brewing Company of of Golden, Colorado are proud of having introduce this convenient, and recyclable, can.  But far more people remember that it was illegal to ship Coors beer East of Texas (and when I say “far more people”, I  mean “me”).

What could this possibly have to do with software development?  It can be difficult to figure out what people will latch onto.  In my tenure at SlickEdit, I have seen a lot of sleep lost over features that did not make quite the splash we hoped they would.  Thankfully none of these turned out to be as big a blunder as the Edsel.  I will share the one that personally hit me the hardest though:

1. Launch SlickEdit’s DIFFzilla® setup dialog (Tools>File Difference)
2. Fill in “Path 1″ and “Path 2″ with any two files that SlickEdit tags
3. Click the “Symbols” radio button at the top left



4. Clicking OK will yield output that actually show changes on a symbol level, including modified functions and added/deleted functions.



5. Selecting an item from this list and clicking “Diff” will actually compare just those two functions in the DIFFzilla output dialog.

This can be convenient for a number of reasons.  Sometimes a function moved, and the file itself has changed so much diffing two whole files just confuses matters.  This feature will shine in those cases.

I thought this would be a hit.  Everybody here thought it was a neat idea, and I worked feverishly to sneak this into the 8.0 release.  It was not a hit.  In fact, I am not certain that a user has ever actually used this feature.  It is generally good for a “wow” or two at a trade show though.

The moral of the story?  Be careful when you choose to add features.  The two-piece aluminum can may not look nearly as flashy as a 1977 Trans-Am, but people will still be using it 50 years later.

I vary on my reaction to the dialog. It’s rarely unexpected, as usually the application in question was not responding anyway. But do I send the report? If I know that Microsoft is just going to send me a link that says “Not our problem! Maybe you should talk to the losers who made the software!” then no. If the problem is a mystery to me, then I might send it just for kicks.

If it’s a Microsoft program that crashed, then I send the crap out of it. I was having an issue with Word a few weeks back, where it crashed anytime I opened a specific file. I was so frustrated that I would purposefully cause the crash, just so I could send another bug report. I no longer cared whether I could edit the file, as long as Microsoft knew that I couldn’t do it. Do I have your attention now, Mr. Gates? Is the problem fixed yet? Logically I know that sending the same report from the same computer ten times in one hour probably does not mean instant results. But like bringing down a shoe on one cockroach when you know he’s got a million friends somewhere, it just makes you feel better.

But sometimes, I get another experience with the dialog box, one altogether unique to programmers. The application in question was mine. It crashed because of something that I did. So now I have Microsoft apologizing for my behavior and asking if I would like to turn myself in. Uh, no, thanks, that’s quite all right. I’m just glad there’s no way for Windows to detect that I was running my own program, or I’d get a different message entirely.

In a similar vein, I’d like to expose the myth of the so-called “year of experience”. In hiring, this is the basic unit used to measure ability. Look on almost any job posting and you will see years of experience required in something. Some postings use this metric sparingly, requiring a number of years of experience overall or in a single key technology, like 3 years of C++. Some postings carry this to such an extreme that you wonder if anyone could have that magical combination of experience. Thankfully, you don’t see this as much as you used to. Some ask for ridiculous amounts of experience in some areas. A few years back I even saw a posting for a Java position requiring more years of experience than Java had been around!

What companies need is someone with a certain level of expertise. But how do you quantify expertise, particularly in technical fields with broad areas of knowledge? That’s where the mythical year of experience comes in. It may not be accurate, but it is quantifiable and easily verified from a candidate’s work history.

The reliance on Human Resource departments may also explain the popularity of using years of experience, particularly in technical domains. In general, HR representatives aren’t engineers. They don’t truly understand what we do for a living and are unable to tell a good engineer from a bad one. But it’s their job to screen candidates and provide suitable ones to hiring managers. So some criterion must be used to filter out unsuitable candidates.

Experience != Expertise

There is an assumption that a certain amount of experience will produce the desired expertise. However, experience does not produce expertise! Experience is important in developing expertise, but expertise requires that the person has the will and ability to extend their knowledge and refine their abilities. Without this, experience just reinforces your current methods.

On one project, I worked with a person who was responsible for doing the builds, who had been working on this project for 5 years. In that period, the approach used hadn’t changed significantly since the first year. I’m sure he claims 5 years of experience for that work, but it really should be considered 1 year of experience with 4 years of repetition. I’m not saying that the build process needed any changes or improvements. My point is that repeating the same activity over and over should not count as experience. It certainly does nothing to further your expertise.

By focusing on years of experience rather than needed expertise, you may even rule out better candidates. Say that a position has a stated requirement of 3 years of experience in C++. The best programmers learn new languages and technologies quickly. Indeed, a great programmer will have a better understanding of a new language after six months than the average programmer would have after three years!

Most technologies don’t take several years to master. When learning a new technology, the early part of the learning curve is the most intense. By the end of the first year, most people are in the incremental learning cycle where they are picking up knowledge very slowly. From my observations, there is little difference between the knowledge a person has after 1 year of using a technology and after using it for 3 years–unless something motivates them to learn more. Even great programmers are in the incremental learning phase by the end of the first year, but they have attained a higher level of understanding in that period than many others will gain in a lifetime.

In my career, I’ve interviewed many C++ programmer candidates. Most have done little to extend their knowledge past their initial training, often a training class that lasted a week or two. Most have never read any books on the language, other than the book they used to initially learn the language. It’s rare for me to interview programmers who have read even a single book on Object-Oriented Programming. For a person like that, 3 years of experience will not lead to much expertise—unless their project provides an exceptional opportunity to learn from more knowledgeable developers.

The rate at which someone learns a new technology is one of the most important factors in recognizing top programmers. Even if you could write the perfect job description and pin down the exact combination of knowledge and abilities for the current project, that mixture will be out of date when the next project comes along. It’s not uncommon for technologies to change even during the scope of a single project. When it does, what good was that carefully crafted combination of skills?

Yes, Experience Does Matter

This doesn’t mean that experience is unimportant. Experience allows us to learn from mistakes. It provides opportunities for problem solving. Just as important as the knowledge of how to do something is your trouble-shooting ability when things go wrong. This knowledge is often best gained through experience.

Even so, you can’t measure this experience solely in terms of time. It has more to do with the number of lines of code you’ve written and the number of unique technical challenges you have faced than the number of years you’ve done something. Given two programmers with 5 years of experience, the one who has written hundreds of thousands of lines of code will generally be more experienced than the one who has written tens of thousands. Likewise, a programmer who has worked on several projects in that timeframe will likely be more experienced than one who has worked on the same project.

When writing job descriptions, I use years of experience only for the overall number of years as a programmer. This is an effort to factor in the lessons learned from experience. Even then, I’m always watching out for the highly talented person who has come by their knowledge and experience faster than their peers.

What Does This Mean for Me?

In your career, don’t assume that years of experience is enough. Push the learning curve as hard as you can. When you are learning a new technology, seek out the books or other materials in that area that can give you knowledge that would take years to learn on your own (if ever). Use the work you do each day to drive your learning. For example, if you’re writing a new C++ class and you’re unsure whether to overload the copy constructor, use that as a reason to really learn the best approach. Don’t settle for mediocre knowledge.

Having pushed the learning curve, don’t be intimidated if the years of experience don’t match your background when you apply for a position. Stress to the recruiter that you have the knowledge that someone with the specified years of experience would have. Be sure you can back that up in interviews, though. Savvy hiring managers will quickly spot a pretender.

While this message is very friendly and informative, it doesn’t help you get your code compiled.

So, what do you do when your code masterpiece, for whatever reason, crashes the compiler. This typically happens, not when you are writing code, but when you update to a newer version of the compiler and amid your efforts to port your code base to the new compiler, the compiler dies.

Your first reaction will probably be to ask your boss if you can go home for the day — see if your luck will be better tomorrow. This is a good strategy, even though it seldom produces a solution to the problem (even if it does mysteriously solve the problem, it’s a little unnerving) — either way, you might get a half day off. Everyone needs a break once in a while.

Your second reaction will be to pass the problem on to someone more senior or someone more junior who might have more experience or time to deal with this kind of problem. Another good strategy — in the meantime, you can go get some coffee.

Your third reaction will be to find the person who wrote that code and ask him what he was thinking. “Do you have to solve everything with template meta-programming? Here, fool, let me show you how to write a for loop.” Another good strategy, it feels good to yell at people, it releases stress. And, if they are actually using template meta-programming where they could just use a for loop, they deserve it.

Your fourth and final reaction will be to dig into the code and solve the problem. You know which file it is crashing on, but since the compiler crashed, it doesn’t report a line number, so all you know is that you have a 20,000 lines of code with something the compiler doesn’t like buried in it. Here is a technique I learned from my brother who was writing code before most of you were wearing digital watches.

Put your computer science skills to use and do some divide and conquer. First #if 0 out the last 10,000 lines (breaking at a logical boundary, of course), see if it compiles without crashing. If it compiles, the problem is in the last half of the file so move your #if 0 somewhere around line 15000. If the compiler crashes, the problem is in the first half, move your #if 0 somewhere around line 5000. Repeat this binary search until you narrow down to a single statement or small section of code that causes the compiler to crash.

Then all you have to do is figure out the best way to restructure that code so the compiler can handle it. This could involve some guesswork and you should talk to the guy who you yelled at back in reaction three to get his opinion. By the way, you should probably remember this moment of humility the next time you hit stage three with a bug like this.

Now that you have solved this problem, you might want to narrow it down further and report the compiler crash to the compiler vendor or the gcc-bugs mailing list. Oh, yeah, and tell your boss what a tough problem you just solved — he might just give you another half day off.

I’d heard that before; resistance to change for fear of change. It’s like being told, “because I said so,” by your parents for lack of a good reason. I always saw that guy as being an obstacle in our software development, more interested in not rocking the boat than improving the process.

6 years later…

When I started at SlickEdit, I was introduced to a lot of firsts. I’m a Microsoft programmer [ducks behind cube wall]. I’ve been programming with Microsoft’s APIs, toolkits, libraries, IDEs, etc for a long time. When it comes to Microsoft technology, I’ve done a lot, and it’s a world that I feel completely at home in.

My first assignment here was to test SlickEdit on AIX, having never used SlickEdit before and with virtually no UNIX background. I tried my best to learn my way around, but I never shook that feeling of being hopelessly lost. To make things worse, everyone else was using it at the Jedi Knight level. All of my creature comforts were gone during this testing cycle. As an experienced programmer, it’s humbling and flat out embarrassing to have to ask questions like, “How do I install this program?”, “Where is my personal directory?” and “How do I build with GCC?”

When I got to work on Tools, and got back to Visual Studio and Windows, I felt like I was back home. I knew where everything was; I knew which step was the creaky step, I knew how to use all the buttons on the TV remote control, I was completely familiar with my surroundings. This isn’t at all to say that Visual Studio is better than SlickEdit, because in a lot of ways, it’s not. The same could be said about Windows vs. UNIX or Linux.

After so many years, I’ve become a creature of habit, and I recognize that. Not because the task of learning a new OS or code editor is too difficult, but because I really hate the feeling of being lost. Even worse is thinking about the wasted hours learning to do the same thing I could do in my comfort space in five minutes. Most horrible of all is just feeling stupid about not knowing how to do the simplest things.

After all this time, I feel like I’ve gained a little perspective from my old manager’s side of the fence. “Don’t take away their console interface… they don’t want to learn, they just want to get their job done,” was what he was really trying to say. Starting from square one and leaving your areas of proficiency behind is scary. I’m always happy to learn new things, but it’s always nice to come back home. Now I see his point of view.

Anyone who knows me well, knows that I am really into programming languages. Like editors, a programming language is a tool which is supposed to make your programming task easier.

Unlike the typical programmer, I have experience designing and implementing languages. Slick-C, the macro language in SlickEdit, is the third fairly major programming language I’ve been involved with. My experience and passion for programming languages gives me quite a bit more insight into performance and elegance.

I started learning Python because SlickEdit needed better support for the language. Unlike Ruby, I never heard a lot of hoopla about Python. The programming world has many, many scripting languages, most of them poorly designed. I figured that Python was probably just another poorly designed language with little or no technical merits.

I’ve been puzzled by the popularity of Perl since it has so many flaws. By the way, just because I dislike some elements of Perl doesn’t mean I will direct SlickEdit not to provide great support for it. SlickEdit language enhancements are based on popularity, and currently Perl is more popular than Python. However, that doesn’t mean I won’t rank on it a bit. Heck, I rank on C/C++ and it’s the language I use most (beware of sharp knife). To date, we have spent WAY more time on Perl and Ruby support than Python. This will always be the case, because it is WAY easier to implement Python support.

I have done my best to make these comparisons unbiased. The only code that I have written for these languages is sample code to better understand these languages. I do have good experience in parsing these languages either for color coding or tagging symbols for SlickEdit. My opinions are based on what I think make a state-of-the-art language. My next article “A Deeper Look at Python”, will talk more about Python and what things make a language state-of-the-art.

This comparison of Python, Perl, and Ruby is from a grammar perspective and does not cover the run-times of these languages. There are often cases where you end up choosing a language due to its run-time features. I’m not going to cover run-time features here. A detailed chart of reasons to choose one or the other which includes run-times would be very useful though.

Python Versus Perl

Python beats Perl by a mile. In a nut shell, Python and Perl have similar language features but Python is way easier to learn and use than Perl. Here are some specific examples:

Perl

Python

Python is a big improvement over Perl. I would provide more reasons why Python is much better but I’d like to keep this article short. I will say that Perl has a very large and arguably cryptic grammar. Perl is hard for others to quickly read and understand. I find Perl hard to write as well. Because Perl has a lot of syntax which is so different from other programming languages, you quickly forget it and must read the language documentation again.

Python Versus Ruby

Comparing Python and Ruby is not as simple as showing some small snippets of code as I did above. For me, Python is clearly a better language than Ruby but I can understand why some people might like Ruby better. Ruby took the “kitchen sink” approach. It has language syntax for pretty much all the features of Perl, Python, and then some. Ruby embraces many of Perl’s complex features, which may make Ruby seem more familiar to the Perl fanatic than Python. There are loads of “shorthands” and optional syntax. To the naive person, lots of features give the false impression that the language is better or somehow more powerful.

While more can mean better, it definitely doesn’t in Ruby’s case. Due to the very large grammar, Ruby is way more difficult to learn than Python. The comment on the Ruby web site about Ruby being easy to learn is definitely not true. Reading Ruby code written by someone else is a problem too since others will use constructs you are not familiar with or don’t like. The ambiguities in the language make it much harder for editing tools to provide smart editing features. I’m not willing to give up smart editing features for shorthands which save a few characters. If you are using a dumb editor, then you might see this differently.

In order to keep this article short, I will not show all the constructs in Ruby which make it a large ambiguous grammar. So this may look like I’m nit picking.

Poor Ruby Syntax Choices

Here are some poor Ruby syntax choices:

Ruby

Python

Ruby should have chosen a syntax which allows for a more conventional single line statement like one of the following:

Function Pointers or Function Objects

Ruby really messed up with function pointers (or function objects):

Ruby

Python

Ruby

The first Ruby example uses syntax which none of us should ever have to learn. To make matters worse, the keyword “yield” is used which means something totally different in C# and Python. I haven’t yet found a way to pass a predefined function as an argument to a Ruby function. Ruby does support passing method pointers using the “method” feature (ex “variable.method(:methodName)”), but again Python provides a much simpler syntax. To pass a method pointer in Python you just specify “variable.methodName” like you would in C#.

Ruby’s grammar is riddled with multiple ways of doing things. Adding to the confusion, it often unnecessarily chooses syntax different from popular languages we are familiar with. As a result, Ruby is much more difficult to learn, read, and write.

Not everything is better in Python. For example, Ruby’s class/member syntax is better than Python. For me, this doesn’t help Ruby nearly enough to make it a better language.

Summary

Both Ruby and Perl have large ambiguous grammars (Perl’s grammar is worse). They often disambiguate syntax based on context. For example, the slash symbol (/) can mean divide or perform a search depending on the context. In the search case, the slash is used to delimit the string. This is an unnecessary complexity since this is just another way to call a function with string parameters.

There will never be a case where having a very large and ambiguous grammar makes a language better. This is one of the flaws in C++. I’m an excellent C++ programmer and I’ve had to stare at some C++ statements for way too long. Note that Java and C# have very small unambiguous grammars with only a few shorthands. Modern languages have so many features that they must keep the grammar small in order to reduce the learning curve. I guarantee that the best state-of-the-art languages of the future will do the same. For this reason alone, Python is clearly better than Ruby or Perl. Keep in mind that I have not compared the run-times of these languages and there are often cases where you end up choosing a language due to its run-time features.

There are other ways that scripting languages can speed the programming process without creating a large grammar with excessive shorthands. For example, unqualifying names with import/using statements and choosing short name space names for standard run-times (Python does both of these). Also, if the grammar is small and unambiguous, editing tools like SlickEdit can provide more assistance.

My next article “A Deeper Look at Python”, will talk more about Python and what things make a language state-of-the-art.