Thursday, September 20, 2012

Installing Snappy codec


Following are the steps to install Snappy with Hbase
2.       Unzip and untar  snappy.
3.       Type following command under snappy-1.05 folder ./configure; make; make install
4.       Snappy would install lib*snappy.so files under /usr/local/lib directory by default. Copy all lib*snappy*.so* files to $HADOOP_HOME/lib/native/Linux-amd64-64 dir
5.       You should see following after running ls
[hadoop@test-cluster-1 Linux-amd64-64]$ pwd
/home/hadoop/hadoop-0.20.2-hdh3u3/lib/native/Linux-amd64-64
[hadoop@test-cluster-1 Linux-amd64-64]$ ls -lrt
total 1940
-rw-rw-r-- 1 hadoop hadoop 224086 Mar 25 23:15 libhadoop.so.1.0.0
-rw-rw-r-- 1 hadoop hadoop 224086 Mar 25 23:15 libhadoop.so.1
-rw-rw-r-- 1 hadoop hadoop 224086 Mar 25 23:15 libhadoop.so
-rw-rw-r-- 1 hadoop hadoop   1171 Mar 25 23:15 libhadoop.la
-rw-rw-r-- 1 hadoop hadoop 411654 Mar 25 23:15 libhadoop.a
-rwxrwxr-x 1 hadoop hadoop    957 Jun 29 11:55 libsnappy.la
-rwxrwxr-x 1 hadoop hadoop 369308 Jun 29 11:55 libsnappy.a
-rwxrwxr-x 1 hadoop hadoop 171844 Jun 29 11:55 libsnappy.so
-rwxrwxr-x 1 hadoop hadoop 171844 Jun 29 11:55 libsnappy.so.1
-rwxrwxr-x 1 hadoop hadoop 171844 Jun 29 11:55 libsnappy.so.1.1.3

6.       Add following in $HBASE_HOME/conf/hbase-env.sh ( change dir as per ur installation) 
export HADOOP_CONF_DIR=/home/hadoop/hadoop-0.20.2-hdh3u3/conf
export HBASE_CLASSPATH=$HADOOP_CONF_DIR:/home/hadoop/hbase-hdh3u3/conf/hbase-site.xml
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:/home/hadoop/hadoop-0.20.2-hdh3u3/lib/native/Linux-amd64-64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/hadoop/hadoop-0.20.2-hdh3u3/lib/native/Linux-amd64-64
export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/hadoop/hadoop-0.20.2-hdh3u3/lib/native/Linux-amd64-64
export LIBRARY_PATH=$LIBRARY_PATH:/home/hadoop/hadoop-0.20.2-hdh3u3/lib/native/Linux-amd64-64
Scp the libsnappy* to all your nodes in the cluster
7.       Restart   hbase and hadoop.
8.       In  hbase shell type following command :     create 'test', { NAME => 'column', COMPRESSION => 'SNAPPY' }
9.       If there is no error snappy is installed properly

No comments:

Post a Comment