SquareCog's SquareBlog

Pig trick to register latest version of jar from HDFS

Posted in Uncategorized by squarecog on August 30, 2011


%default guavaJar `hadoop fs -ls lib/*guava*jar | awk '{print $8;}' | sort -n | head -1`
register 'hdfs://$guavaJar'

The same idea also works without HDFS being involved, of course.

Pig Variables and Semicolons

Posted in programming by squarecog on August 24, 2011

Pay attention to where you put your semicolons lest they become part of the value!

Pig Script:

%default QUOTED '20090101';
%default UNQUOTED $QUOTED;

l = load '/logs/$QUOTED';
l = load '/logs/$UNQUOTED';

After preprocessing with pig -x local -r script.pig:


;

l = load '/logs/20090101';
l = load '/logs/20090101;';

Tagged with:
%d bloggers like this: