Capturing system command output in perl

Running system commands from inside a perl script is one of the nastiest of all perl headaches. I suffered grievously when trying to use this method with some pdf file addons, and have scars to show for it. Lately, having failed to learn my lesson not to mess with this stuff, I have been working on something similar: trying to use the new version of 7-zip to get crc hashes for the excel files in a directory.

The idea is to use the hash checksums to see which files have been modified, compared to an earlier archived version of the files. 7-zip command line syntax is already tricky and I have no idea what I’m doing in most of this; I’m just following incantations found on various occult internet sites. Here is the command line version of the 7-zip command:

"c:\Program Files\7-zip\7za.exe" h -ir!d:\yumyum\*.xls -ir!d:\yumyum\*.xlsx > temp.txt

This gets the crc hashes for all the xls and xlsx files in the d drive directory yumyum. Doing this in a perl script is equally tricky; the key is putting the various segments into an array, as follows:

my @cmds = (
'c:\Program Files\7-Zip\7za.exe', 'h',
'-ir!d:\Weisberg\*.xls',
'-ir!d:\Weisberg\*.xlsx'
);

One then uses system to invoke the mystic phrases, and an ‘or’ clause to ask for an explanation if it fails:

system (@cmds) == 0
or die "system @cmds failed: $?\n";

The procedure described above works, but the mind boggling issue is how to get the output of this command (lengthy scrolling for about 20 seconds) into a perl variable where you can do something with it. This task completely defeated me after half a day. I then switched to trying to pipe the output into a file. This accounted for the second half of my day. In fact, there is a ready made solution to this problem: Capture::Tiny, written by David Golden and included in the ActivePerl distribution.

The final result is:

use Capture::Tiny ':all';
my ($stdout, $stderr, $exit) = capture {
system( @cmds );
};

The output of the system call is in $stdout, the error info is in $stderr, and the exit return code is in $exit.

All hail David Golden! Hail!!

Posted in Programming | Comments Off on Capturing system command output in perl

Welcome to Spring Semester!

Hello! Fall 2017 was a very quiet semester in the warren, hopefully there will be more coming up this spring. If you want to get in touch and you’re an NCNU student, NCNU’s Moodle site is probably the best way. If you’re not from NCNU and you’re interested in a post here, please try my regular email address, which is available from the DFLL webpage at National Chi Nan University.

Posted in school | Comments Off on Welcome to Spring Semester!

Welcome to the new semester at Chi Nan!

Another semester begins, and I hope all the students in the Department of Foreign Languages and Literature had a pleasant first day. I’m looking forward to the semester, and to seeing all of you.

Posted in school | Comments Off on Welcome to the new semester at Chi Nan!

“I have always had adequate sex that no one appreciated”: The best of Studies in Intelligence

Taking a 10 minute pause from grading final exams, I would like to announce that the Warren will begin a new series introducing government funded fun and enlightenment from Studies in Intelligence, the CIA’s in-house journal. While there is much food for thought in the more analytical articles, there is also plenty of idle-reading entertainment hidden in the shadowy depths of this mysterious but official publication. Fearlessly footpadding through its classified depths, I have located the best of these cloaked gems and will expose them to the world at large at irregular intervals. Today’s recommendation is, in my opinion, one of the most incisive pieces SI ever published. What are you waiting for? Click here and read on.

Posted in History | Comments Off on “I have always had adequate sex that no one appreciated”: The best of Studies in Intelligence

Chiang Kai-shek Meets Tu Yueh-sheng

This note picks up from my last post on the sources and claims of Sterling Seagrave’s 1985 book, The Soong Dynasty (SD).

SD enjoyed a fourteen-week run on the New York Times best-seller list and launched Seagrave’s career as a chronicler of Asian history. His books since then have  generally met with success, and in the case of SD, have even had some influence on more scholarly works. This is unfortunate, because SD is one of the least reliable books on history I have ever read.

Tu Yueh-sheng, Chiang Kai-shek, and the Soongs

One of the main themes of Seagrave’s book is the role of Tu Yueh-sheng, leader of Shanghai’s Green Gang, in the lives and fortunes of the famous Soong family, and in the career of Chiang Kai-shek, perhaps the most prominent politician and military leader of pre-Communist China.

According to Seagrave, Chiang Kai-shek’s relationship with Tu Yueh-sheng can be traced back to the the beginning of China’s Republican era (1911). Through this relationship, Chiang, fronting for the Green Gang, helped Tu over the Nationalist Party (the KMT), the ruling party of China from 1927 to 1949.  As Seagrave says:

Chiang’s direct connection with the notorious Shanghai Green Gang after the winter of 1926-27 has been known for many years, but there has been only a vague understanding that those links went back much earlier, and of how they affected his career. It is now possible for the first time to see the “Divine Skein” linking them all the way back to his youth, before 1910, and the manner in which the Green Gang leaders used Chiang decisively (and were used by him) to snatch the revolution from the hands of Dr. Sun Yat-sen’s coalition. (12)

In fact, Seagrave’s evidence for an earlier alliance between Chiang and Tu is non-existent, based on a single spurious citation.

A non-existent reference

Seagrave’s most specific claim for an early acquaintance between Chiang and Tu is given on page 152 of the Soong Dynasty:

Big-eared Tu enjoyed visiting the famous Blue Villa and cruising the other Green Gang brothels in the Blue Chamber District with a young, ill-tempered bravo by the name of Chiang Kai-shek.

My last post on Seagrave’s book attempted to find a source, any source, for the existence of a Shanghai brothel called the Blue Villa, staffed, according to Seagrave, by 121 prostitutes (SD 158).  As it turned, there was indeed a book which mentioned such a place: a 1977 novel by Alain Robbe-Grillet called Topology of a Phantom City.

The fictional provenance of this detail cannot inspire confidence in the reliability of Seagrave’s claims, but for the general fact of an early acquaintance between Chiang and Tu, Seagrave’s endnote gives what seems to be a non-fictional source: “The intimate brothel relationship between Tu and Chiang at this early stage is noted by Murphey, pp. 7-9.”

This is a reference to Rhoads Murphey’s 1953 book Shanghai: Key to Modern China.  Unfortunately, this reference is as fictional as the Blue Villa itself. Pages 7-9 of Murphey do not mention Tu or Chiang.  In fact Tu Yueh-sheng is not mentioned once in Murphey’s book. Chiang is mentioned, once, but it is a quote on Shanghai’s status in the 1943, with no relevance to the brothels of early 20th century China (Murphey, 25).

The spurious nature of Seagrave’s citation was first noted by C. Martin Wilbur, an important scholar of China’s Republican era, in a review of Seagrave’s book, which Wilbur titled “Fabricating History.” 1

Wilbur’s review examines only the first 200 pages of Seagrave’s book, compiling a long list of errors, exaggerations, “flourishes” and “embellishments” and several claims that Wilbur bluntly calls fabrications.

Discovering that Murphey’s book had nothing on Tu and Chiang at the Blue Villa, Wilbur went through some of Seagrave’s other materials and decided that this claim may have been based on a passage in Brian Crozier’s biography of Chiang, The Man Who Lost China.  This passage does indeed describe Chiang paying regular visits to the brothel district of Shanghai.  There is a major discrepancy with Seagrave’s claim, however; according to Crozier, Chiang’s companion in his brothel excursions is not Tu Yueh-sheng, but Ch’en Ch’i-mei (陳其美).

Ch’en was an important figure in the years immediately before and after the revolution of 1911.  He was also an important figure in Chiang Kai-shek’s life, supposedly responsible for introducing Chiang to Sun Yat-sen. The passage Wilbur found in Crozier reads:

“Chiang’s friend Ch’en Ch’i-mei was his mentor in other things besides revolution. Whenever he frequented the houses of prostitution, Chiang was with him” (44).

In fact, a second passage from Crozier is even closer to Seagrave:

His revolutionary mentor, General Ch’en Ch’i-mei, had introduced him to the “blue chamber” district, with its brothels for all purses. (58)

Here we encounter the “Blue Chamber district” Seagrave mentions. These are the only references to Chiang visiting brothels with a named person that I have found in any of the books Seagrave lists in his bibliography.

But why cite Murphey for Crozier? It is of course possible for Seagrave to mistakenly reference book A for information derived from book B. In fact, it is frequent; there are dozens of examples of this in SD.  But for Seagrave, based on Crozier’s book, to confound Ch’en Ch’i-mei with Tu Yueh-sheng is not possible.  So as Wilbur asks, “Why the switch of names? Could it be because Ch’en died in 1916, and so cannot fit the conspiracy theory?” (132)

This spurious reference is the only one in Seagrave’s entire book claiming to document a tie between Chiang and Tu predating 1927.  Perhaps it is appropriate, then, that Seagrave places their carousing in Robbe-Grillet’s Blue Villa, fictional icing on top of historical falsehood, and a strong justification for the title of Wilbur’s review.

Tu and Chiang’s First Meeting: Huang Chin-jung speaks

So when did Chiang and Tu first meet?  One possible answer is in Brian Martin’s 1996 book, The Shanghai Green Gang: Politics and Organized Crime, 1919-1937.  Martin cites no less a source than Huang Chin-jung (Huang Jinrong), known to Seagrave’s readers as Pock-marked Huang. The meeting took place following the defeat of the warlord forces in Shanghai in 1927. Martin claims that:

According to Huang Jinrong’s own account, as told to a senior Chinese detective in the French Concession police in 1939, he and Du Yuesheng [Tu Yueh-sheng] personally met Jiang’s [Chiang’s] airplane at Longhua Airfield [in Shanghai] on March 26, where he introduced Jiang to Du. (99)

Despite Martin’s enthusiasm, the source is not Huang himself, but a reminiscent account from the detective, Xue Gengxin (薛畊莘), which was published in a 1980s compilation.2 While it still might not be the petrified truth, at least it really appears in a book.

  1. C. Martin Wilbur, “Fabricating History,” Issues and Studies 22.5 (1986): 129–148.
  2. Jiu Shanghai de banghui 舊上海的幫會, 1986, pp. 87-107
Posted in History | 2 Comments

Updating WordPress

I finally got around to updating this site’s wordpress installation. The reason for the long delay was that the online updating process from wordpress was broken.  I had originally thought this was because I had used non-standard permissions in the wordpress directory (using ‘hardened’ permissions), but it turns out that this was not the case.

Instead, for some reason, during my last update to wordpress almost a year and a half ago, the ownership of some of the files was not assigned to the Apache webserver. The permissions were okay, but the owner was wrong. This prevents the online update from working. These mis-owned files were actually shown during the latest version of the update process. Is this a new display for the update process, or have I been missing this for the last 18 months??

In any case, the explanation of the failed update was still not very clear in describing the problem, nor was this page.  It took a fair amount of head-scratching and experimenting to finally realize that the answer is simply to chown -R the wordpress directory using the web server name, and the update will work.  Note that if you have any symlinks in your directory, the source file also needs to be chowned, not just the symlink. (If I have misstated this, please let me know!)

Having been put on full maintenance alert, I also took a look at my log files, which showed that my xmlrpc file is being regularly hammered by ‘guests’ who claim to be from the Ukraine. Who knows where they’re really from. This is a well-known problem, so I’ve simply blocked that functionality for now.

In addition, some bot claiming to be from Majestic 12 has been ringing my bell non-stop, so I’ve put up a robots.txt file as well.  Of course that won’t do any good if it’s just some hacker with a Majestic 12 text label on his bot. We’ll see.

Posted in software | Comments Off on Updating WordPress

Two sources for the Weisberg Collection

There are currently two places to get materials from the Weisberg collection. The main source is naturally the website that Hood College set up for the collection.  The second source is Archive.org. The Archive.org now has a “copy” of the Hood collection, and when searching for Weisberg materials on-line, both of these often turn up in search results.

Curious about this, I finally got around to downloading the Weisberg materials on Archive.org last week and have now had a chance to take a look at them.  They turn out to be significantly different from the materials on the Hood College website, so I’m posting a note on some of these differences.

Archive.org is a gigantic filing cabinet, and it can sometimes be quite difficult to track down the sources of the materials that are put up there.  In this case, it seems the Weisberg materials there were posted mostly by one Mike Best, archivist for the National Security Internet Archive. I haven’t quite figured out who or what NSIA is, except that it is not related to the National Security Archive at George Washington University.  The NSIA was registered at Archive.org in March 2015, and since then it has since posted a huge amount of materials.  NSIA began posting Weisberg Materials in August 2015, and apparently finished putting up what they had by the end of September.

The description of the Archive.org version is at “Complete Weisberg Archive on the JFK Assassination”, which says: “Harold Weisberg donated the world’s largest accessible private collection of government documents and public records relating to the assassination of President John F. Kennedy to Hood College and the Beneficial-Hodson Library at Hood College, which donated a copy to the National Security Internet Archive.”

So this is not just someone scraping the Hood collection, but a copy provided by Hood to NSIA. If you really want the whole thing, there it is: 29 compressed files, over 100 gigabytes even in the ultra-compressed 7z format.  It was quite a job getting all this stuff direct from archive.org. There is a torrent file that might be faster, but the word is that our school throttles torrents, so I did multi-day downloading through archive.org.

Having gotten the whole thing, I’ve had a chance to compare parts of it with the Hood College version, and they are indeed different.  The most important difference is that the Hood pdf files were run through OCR software (apparently mostly Omnipage 18) to convert them into searchable files.  There is a search interface for the OCR versions available at the Hood website, and this is by far the most convenient, effective way of accessing the Weisberg collection.  The Archive.org files have not been OCRed; they are simply images.

This is not the end of the story though.  After some rather hard poking through the NSIA materials, it seems that this is very likely a working copy of the Hood materials.  It’s most useful feature is that it includes excel files for the pdfs in each directory. These excel files have all kinds of important information, such as dates, to-from fields for letters etc, and comments and cross-references to related documents.  These excel files are mostly not available from the Hood website.

Unfortunately, the fact that these are “working” files also has another meaning.  The whole thing seems to have been simply yanked off a hard disk at some point. The most recent files in the NSIA materials are dated 2015-07-12, and there are a number of temporary excel files included in the archive which also have this date. So the backup was done without even closing the excel files that were being edited.  A number of these were clearly not yet done, with numerous inconsistencies in the files listed in the excel sheets and the files actually present in the directory. Some of the excel sheets are even in the wrong directories, with whole directories sometimes misplaced inside other directories as well.

This is not to dismiss the amazing amount of work done on the collection. The large majority of the files are listed, and the large majority of the information listed is accurate, but the Weisberg collection is so huge that “large majority” means there are still thousands of places where there are problems. It is not a trivial task to fix these problems.

It is also worth noting that there are tens of thousands of duplicate pdfs throughout the collection. These are not just duplicate files in the Weisberg collection; there are places where the exact same pdf file is present in multiple locations. Some of this is probably some sort of cross referencing system. An example is that in the giant C zip file, there are dozens of directories of the form CIA [someone’s name].  Most of these appear in other places, with the directory name in the form [someone’s name] CIA. In the second form, however, sometimes the pdf files in these directories are still named CIA [someone’s name]. In cases where they have been renamed, it is almost always the case that they are still the same pdfs, just with the names changed. Some of these duplicate directories also do not appear in the materials on the Hood website and it seems that NSIA copy may represent the Hood archivists’ current efforts in this area.

Despite these problems, the NSIA copy is a useful ancillary to anyone who wants to work with the collection as a whole As an example, the excel indices include a “date” field for much of the collection’s files. According to this, the earliest fully dated document in the collection is Weisberg’s birth certificate: April 8, 1913.  There are also a few documents from after Weisberg’s death in 2002, including the obituary of Weisberg’s wife Lillian, who died March 20, 2003.  The most recent document is a powerpoint file for a 2011 conference presentation by Clayton Ogilivie, the primary archivist for the Hood collection (Presentation-Canterbury 01.pptx, located in the P zip file, apparently not otherwise available either at Archive.org or Hood).  This gives a very useful overview of the collection and its history.  Everyone interested in Weisberg and his materials owes a huge thanks to Mr. Ogilivie and the others who have put so much time into this project.

Posted in History, Research methods | Comments Off on Two sources for the Weisberg Collection

The Weisberg Collection

While looking up some names from articles in Studies in Intelligence, I ran into another interesting on-line resource: the Harold Weisberg Archive, hosted at Hood College in Frederick, Maryland. When a number of CIA names kept turning up at the Archive, I decided to take a look at it.  It turns out to be a huge collection, with some very interesting material.

Harold Weisberg was one of the many people who quickly became convinced, following the assassination of President John Kennedy in 1963, that Lee Harvey Oswald might not have been the killer, and/or that the assassination might have involved more than one person.  Weisberg was apparently first off the mark in writing a book denouncing the Warren Commission’s report on the assassination.

His book, Whitewash, was published in March 1965, a remarkably fast 6 months after the Commission published the 26 volumes of evidence and testimony it collected.  At the same time, he began collecting a huge assortment of documents relating to the assassination, which now form the basis of the Archive site.

During the rest of the 1960s, Weisberg went on to write a whole “Whitewash” series, almost all self-published.  He was an enthusiastic participant in Jim Garrison’s investigation of the assassination, but later came to see Garrison as a fraud.  Moving on from Garrison, but not the assassination, Weisberg was an early advocate of using the Freedom of Information Act to get at restricted records, and the lawsuits he filed throughout the 70s and 80s make up a big part of the Archive.

Weisberg was considerably older than most (but not all) of the crowd looking into the assassination; he was 50 the year Kennedy was killed.  He thus came from a quite different background than the other writers on the assassination, such as Mark Lane and Edward Epstein, and it turns out that he had personal reasons to suspect government misconduct both in the assassination and in the investigation; he was fired from a State Department position in the late 1940s as a suspected Communist.

He also brought investigative skills and tactics to the JFK assassination which were honed in the 1930s, when he worked for the La Follette Committee investigating union busting activities, and in the 1940s, when he did free lance investigative writing for magazines such as Click and Picture.  Apparently he had a brief spell in the OSS as a Latin American analyst, which led to his State Department position, but I haven’t yet found the records for this part of the story.

In other ways, though, Weisberg seems to have been typical of the assassination buff. He was an inveterate newspaper clipper and letter writer, and these make up a huge part of the Archive. The newspaper clippings, in addition to assassination related material, focus on the CIA, Vietnam, and Watergate, which is why the Archive kept showing up in my searches.

The letters are often very cryptic to me, since my knowledge of the assassination is close to zero.  From what I do understand, Weisberg comes off as highly opinionated and irascible. His letters are full of blunt comments often aimed directly at his correspondents, who seem to have included most of the major assassination buffs.  His invective doesn’t have the cruel, multi-faceted edge of the “great masters of vituperation”, such as Baron Corvo, but if you’re looking for straight denunciation of hare-brained stupidity, he’s got it.  One thing you won’t find though; any admission of ever having been wrong himself.  Or maybe I just haven’t come to those letters yet.

Weisberg died in 2002. For several years before his death he had been negotiating with Hood College, located near his home in Frederick, Maryland, to donate his 60 (!) filing cabinets of assassination related material after his death.  The Archive website is proof that Hood did not just chuck everything in the library basement.

Posted in History | Comments Off on The Weisberg Collection

Notes on Studies in Intelligence

I’ve read a bit more from and about Studies in Intelligence, the CIA’s in-house intelligence journal (not that much, put down that blindfold and cigarette).  A few random notes on this for today.

The most interesting discovery was the SI article ‘Fifty Years of Studies in Intelligence by Nicholas Dujmovic.  This article answered many of the questions I had after careful research in SI‘s back issues (e.g., “Rita Kronenbitter” was a man, not a woman), but it also carelessly revealed some of the major results of my own in-depth research (the journal changed page format in 1972).  The list of chief editors, the overview of trends in article topics, and the description of the evolution of SI’s unclassified section, along with second thoughts about whether it’s a good idea for SI to develop a readership among the general public (my hand is up) are all well worth a read.

Another interesting article from a very different perspective is by Jeffrey Richelson at George Washington University’s National Security Archive, Studies in Intelligence: New Articles from The CIA’s In-House Journal.’ The Archive is an academic project by several researchers who have banded together to pry open as many government filing cabinets as the Freedom of Information Act allows. Richelson has been working on getting SI articles declassified for some time; this article presents some of his finds as well as some carefully documented complaints about CIA failures to comply with the spirit, and sometimes the letter, of the FOIA.  The article was originally posted June 4, 2013, with 19 newly declassified articles.  A revised version was posted November 20, 2014 with seven additional articles after the Jeffrey Scudder case concluded (see Washington Post, July 4, 2014, “CIA employee’s quest to release information ‘destroyed my entire career.'”)

If you are looking here for trivia on SI, an overview of its content, or a critique of its value, you are definitely looking in the wrong place; that is not the sort of thing the Archive usually does. Instead, the focus is on what articles SI has not been willing to release, and why their decisions are often arbitrary, inconsistent, and just plain wrong headed.  Fair enough, especially since Richelson, unlike Dujmovic, has only a carefully redacted copy of SI’s table of contents to date.  And as Richelson notes, the copy they gave him is about 130 titles shorter than the copy they gave another group, who used litigation to get it, rather than a polite FOIA request.

Certainly there is much confusion about what has been declassified and what has not. Tracing down when where and how articles were declassified and made publicly available might be interesting in this respect. Dujmovic has actually made a start on this, but for the fan of the truly trivial, no doubt there is a lot more to be done.

Posted in History | 1 Comment

Welcome to the 2016 spring semester

A new semester is starting, and I’ll be teaching some general education classes that I haven’t taught for a while.  Hope these are interesting for all of you who decided to enroll, I’m looking forward to meeting you!

Posted in school | Comments Off on Welcome to the 2016 spring semester