Hadoop Snippets

Listing Files by Date / Order

hadoop fs -ls /app/dev/SmartAnalytics/Source/SR-XML/15265 | sort -k 6


Secure Copy JAR from a Remote Machine to Local Machine (Linux)

scp -r cipadmin@cstg-sa-prd04:/home/cipadmin/nag/cand*.jar .

Password: xxxxxxx


Secure Copy JSON File from Local Machine to the Remote Machine (Linux)

scp -r /apps/pentaho_nfs_data/CAND/AIX_eg/aix.json cipadmin@cstg-sa-prd04:/home/cipadmin/nag/

Password: xxxxxxx


Command Line Options to Execute Spark Jobs

/opt/mapr/spark/spark-1.2.1/bin/spark-submit –class com.cisco.ic.datascience.analyzer.Identifiers –driver-memory 6G –driver-java-options -Dspark.akka.frameSize=512 –executor-memory 6G –num-executors 2 –executor-cores 8 –master yarn-cluster ./cand-miner-1.3.0-jar-with-dependencies.jar maprfs:////app/dev/SmartAnalytics/Apps/CAND/AIX/input/sr_files/WF_SR_XML_V1.xml maprfs:////app/dev/SmartAnalytics/Apps/CAND/AIX/output/ maprfs:////app/dev/SmartAnalytics/Apps/CAND/AIX/input/json_files/aix_input_WELLS_FARGO.json


Delete Files Recursively from HDFS

hadoop fs -rm -R folder_name

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.