One of the main things I use Perl for is to manage my grading chores on Chi Nan University’s Moodle system. Moodle is a very useful teaching tool, one I’ve been using for over 10 years.
So if it’s such a great system, why use Perl? I’ve used Perl with Moodle for various reasons over the years, but the main reason I use it now is to download student work to my computer, process it using a variety of fast and dirty methods that are very specific to how I do stuff, and then upload the results, usually comments and grades, back to Moodle for students to review.
The problem with this is Moodle changes, often. Some of these changes are quite significant, requiring you to completely revise your workflow.
The university tries to avoid upgrading Moodle in the middle of the semester for just this reason, but when an upgrade includes important security and bug fixes, the school sometimes bites the bullet and upgrades, and as a result I’m stuck with broken scripts in the middle of the semester, as happened last week.
The way the script broke, however, helped me learn a little bit more about Perl and particularly its extremely useful WWW::Mechanize package, so I’m putting up this note to remind myself of what the deal was and to let the one or two stray readers out there know as well.
The main reason I use WWW::Mechanize is to put up comments/grades for my translation assignments. I generally have 8 to 10 single sentences for one translation assignment, and anywhere from 25 to 40 students, though I have had as many as 60 students in the past, meaning that for one assignment I might have to download 300-600 answers and upload 600-1200 grades and comments.
Why download answers? After all, Moodle has a grading interface for this kind of question. Well, I have my methods, as Holmes said, and they’re fast and efficient for what I do. Downloading is not a big problem, it’s just screen scraping. Uploading grades and comments, however, is a different story. Doing grades and comments on my home computer and then copying them manually to the Moodle interface is enormously time consuming and requires very close attention, or you will put the wrong scores in the wrong boxes, a highly embarrassing mistake.
This workload can negate all the time I save grading and commenting using my home methods, no matter how much more efficient they are than Moodle’s methods. To avoid this gigantic waste of time, I wrote a Perl script.
The script gets the comments from the database table where I put them after grading and commenting, and uses WWW::Mechanize to upload them to the school’s Moodle site; all I have to do is input the assignment number and the rest is…mechanized.
This means I don’t have to consider the time spent uploading to Moodle at all, and can put in all sorts of safety checks to make sure the right grade goes to the right person; these safety checks have meant that I often catch mistakes I might otherwise have missed. Anyway, Mechanize is the main workhorse in the upload process.
The main pain in writing this script was finding the parameters to upload the data through Moodle, but after a lengthy process of hair-pulling and many bad words, I finally got them all down. Unforunately, when Moodle’s grading interface changes even a touch, I’m screwed. Basically, I am screen scraping to get the parameters I need to upload my local data, and I have to put these in a properly formatted html form for Moodle to accept everything. Even minor changes in formatting can break my script, with the result that the data simply doesn’t go in.
This happened with the security upgrade just last week, a very typical example of breakable screen scraping. I ran the script to upload comments and grades for the latest assignment, and the grades went up, but the comments did not. Why? Why?? Why???
To find out, I turned on fiddler, an HTTP debugging proxy, and set mech to go through it as follows:
my $mech = WWW::Mechanize->new();
#$mech->proxy(['http'], 'http://localhost:8888/');
I also set Firefox to use fiddler and filled out the Moodle page there as well, then compared the results of ‘post’ through Perl and through Firefox to see what was going wrong. Here I omit several days of aggravation, pulling out what’s left of my hair, and many more bad words.
The short of the story is that the Moodle format did indeed change. My original script set the parameters for the original Moodle page as follows:
First I get my local data from an mysql table and format it using this loop:
while ( my ($cmt,$mark) = $select->fetchrow_array ) {
push(@CommentsGrades, [ textarea => $cmt ], [ text => $mark ]);
}
Then I use Mechanize to get the page and set the HTML form parameters like this (omitting where I input passwords and do sanity checks):
$mech->get($qpage);
$mech->set_visible(@CommentsGrades);
$mech->submit();
This is how simple Mechanize solutions can look.
The problem this time arose from the fact that the original Moodle page presented the student answer in a textbox, and then used a textarea form to get comments. The new version uses a readonly textarea form to present the student answer, and a second writeable textarea form to get comments. It also added a dropdown option box to let you decide how you want to format your comments (html, plain text or moodle enhanced). It put this between the comment box and the text box for the mark (grade). This confused the issue even more.
The solution for this change was to modify the array with the comments and grade like this:
push(@CommentsGrades,[ textarea => '' ], [ textarea => $cmt ], [ text => $mark ]);
Funky, isn’t it?
It turns out that the set_visible function keeps track of the parameters it sets by parameter type; my array had only one type of textarea variable in it, for comments, but the new moodle had two textarea variables in it, for answer display and comments; this made for confusions and set_visible tried to load comments into the readonly textarea, hence the comments were not accepted. I fixed this by adding a dummy textarea to my array before the comment textarea.
Why were the grades accepted? There is only one type of visible text box, the one for grades, so set_visible correctly inserted the grade value where it belonged.
As for the dropdown option box, you don’t even need to put it in the array (unless you want fancy formatting for your comments, I guess.) The missing option box will not confuse set_visible at all, and this surprised me; I thought a missing parameter like this would have a cascade effect, knocking everything off by one for each parameter, but since I didn’t put in any data marked as option => xxx, set_visible just ignored the option box.
And I learned all this at the cost of only a few thousand hairs, and however long I have to spend in purgatory for all those ‘sbloods! and ‘by the rood!’s.