In programming, simple things often turn out to involve obscure problems and techniques. I run into these and spend hours tracking down what went wrong, then console myself with the thought that “Next time I’ll know how to do it.” Unfortunately, by the time next time rolls around, I’ve forgotten all about it. Leaving little text files filled with incomprehensible notations scattered all over my computer doesn’t seem to help. My great idea this time: post all this stuff on my blog. Then I can google it like I do everything else! Brilliant, eh?
Today’s trick combines two obscure problems for one common activity, a stupid twofer, so pay attention, you in the back. It is very common for my perl scripts to fail to work the way I expect. When this happens, there are usually helpful error messages (always, always use strict and warn) to tell me how I’ve screwed up. Unfortunately, these are not always as helpful as one might hope.
A typical case is doing something with a large database table, say ~25,000 rows. I want to modify one of the fields based on certain conditions, and it works except for 200 exceptions. For these, 200 error messages pop up, repeatedly informing me
Use of uninitialized value $var in concatenation (.) or string at document x line y.
So something is wrong with my condition. But what? Where? I quickly pull out a trick from my stupid programmer’s bag and add a line printing out the field “id” from my table to stdout to find out what records are failing the condition, then run the script again. This is not helpful, because the field “id” prints 25000 times, scrolling so rapidly that I can barely see the longer lines that tell me there was an error. Undaunted, I pull out another trick from the bag and redirect output to my favorite, temp.txt, like so:
badscript.pl > temp.txt
This helpfully swallows up all the ids, while my screen still fills up with all the error messages without any “id” field info to tell me where the mistake was. Ids in temp.txt, error messages on screen. Whoops. This is because perl uses stderr to print out error messages, not stdout, and this is not redirected by >.
Let’s recap here in case you’re getting as confused as I did: what do I want to do? I want to send both stdout output (ids) and stderr output (error messages) to the same file. According to microsoft, this can be done as follows:
cmd.exe 1> output.txt 2>&1
Now why didn’t I think of that? Finding this info is the real trick; if you don’t google the right combination of words, it takes a while. (hint: “how to redirect standard error stream in windows”)
Now for the twofer. When I open the file, I find things that I know can’t be right, for example:
….
id = xx
error msg
id = xx
error msg
error msg
error msg
id = xx
….
I am only doing one thing per row, so it is not possible for there to be multiple errors per row. Much pulling of hair and gnashing of teeth until I realize that this is a problem I met a year and a half ago, and has to do with perl’s use of buffering. (See here for an amusing account of just a few of the many problems that this can cause.) The solution is to turn off buffering in the script before going into the loop that prints to stdout, like so:
$| = 1;
Good luck on googling that one! In fact, it is impossible to find any of the special perlvars using google. The only place you will find them is in perlvar, located on your computer as well as the internet. But how do you know to look in perlvar? And in case you’re wondering why 1 turns off buffering instead of 0, it turns out we must think of this as turning on autoflush (turn on the pipe?). Anyway, the result of adding this gewgaw and then running the script again is that temp.txt finally contains what I want: id followed by error messages immediately following where something really went wrong.
Summary: to get a file with output and error messages as they actually appear on your computer screen, you must: 1) turn on autoflush in your script ($| = 1;) and then use the microsoft way to redirect: script.pl 1> output.txt 2>&1.
I tell you, when it comes to stupid programming tricks, they don’t come much stupider than mine~~
Bonus obscurantism! If you really want to dazzle the rubes, don’t do this:
$| = 1;
try this instead:
$|++;
For a nice discussion of this and other gobsmackers, go to www.perlmonks.org and search for “Perl Idioms Explained”. I immediately printed all of them and pasted them on my wall.