Monday 7 December 2015

For using AWS in Spark

Add following jars at runtime in spark-classpath :

1. hadoop-aws-2.6.0.jar
2. aws-java-sdk-1.7.3.jar
3.guava-12.0.1.jar

spark-shell --jars  hadoop-aws-2.6.0.jar,aws-java-sdk-1.7.3.jar,guava-12.0.1.jar  (for spark with scala)
pyspark --jars  hadoop-aws-2.6.0.jar,aws-java-sdk-1.7.3.jar,guava-12.0.1.jar  (for spark with python)


No comments:

Post a Comment