Subscribe

ONTAP Recipe: Configuring NFS Connector with Apache Tez in HDInsight

 

Did you know you can...

 

Configure NFS Connector with Apache Tez in HDInsight       

   

 

This recipe describes how to configure Apache Tez with the Netapp NFS Connector. This guide is specific to HDInsight clusters in Microsoft Azure. The steps in this guide can be generalized to any Hadoop cluster running Tez.

 

This recipe is specific to HDInsight clusters in Microsoft Azure. The stepscan be generalized to any Hadoop cluster running Tez.

 

1. Configure the user “hive” and the group “hadoop” across all workers in your Hadoop cluster.

  sudo /netappnfs/usergroup.py -u hive -g hadoop -t /netappnfs

 

2. Copy the Netapp NFS Connector jar files to the fs.defaultFS as configured in Ambari.

  hdfs dfs -mkdir /{path to desired location}/aux-jars

  hdfs dfs -copyFromLocal /netappnfs/hadoop-nfs-2.7.1.jar /{path to desired location}/aux-jars

  hdfs dfs -copyFromLocal /netappnfs/hadoop-nfs-connector-2.0.0.jar /{path to desired location}/aux-jars

 

3. Add the new auxiliary jar location to the tez-site.conf file (reference).

4. Add the auxiliary jar path using Ambari:

 

tez.tif

 

5. Manually edit tez-site.conf, adding the following lines:

  <tez.aux.uris>

  /{path to desired location}/aux-jars

  </tez.aux.uris>

 

6.Restart the Tez, Hive, Oozie, etc. services