Hadoop Snippets

Listing Files by Date / Order

hadoop fs -ls /app/dev/SmartAnalytics/Source/SR-XML/15265 | sort -k 6

 

Secure Copy JAR from a Remote Machine to Local Machine (Linux)

scp -r cipadmin@cstg-sa-prd04:/home/cipadmin/nag/cand*.jar .

Password: xxxxxxx

 

Secure Copy JSON File from Local Machine to the Remote Machine (Linux)

scp -r /apps/pentaho_nfs_data/CAND/AIX_eg/aix.json cipadmin@cstg-sa-prd04:/home/cipadmin/nag/

Password: xxxxxxx

 

Command Line Options to Execute Spark Jobs

/opt/mapr/spark/spark-1.2.1/bin/spark-submit –class com.cisco.ic.datascience.analyzer.Identifiers –driver-memory 6G –driver-java-options -Dspark.akka.frameSize=512 –executor-memory 6G –num-executors 2 –executor-cores 8 –master yarn-cluster ./cand-miner-1.3.0-jar-with-dependencies.jar maprfs:////app/dev/SmartAnalytics/Apps/CAND/AIX/input/sr_files/WF_SR_XML_V1.xml maprfs:////app/dev/SmartAnalytics/Apps/CAND/AIX/output/ maprfs:////app/dev/SmartAnalytics/Apps/CAND/AIX/input/json_files/aix_input_WELLS_FARGO.json

 

Delete Files Recursively from HDFS

hadoop fs -rm -R folder_name

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s