Add following jars at runtime in spark-classpath :
1. hadoop-aws-2.6.0.jar
2. aws-java-sdk-1.7.3.jar
3.guava-12.0.1.jar
spark-shell --jars hadoop-aws-2.6.0.jar,aws-java-sdk-1.7.3.jar,guava-12.0.1.jar (for spark with scala)
pyspark --jars hadoop-aws-2.6.0.jar,aws-java-sdk-1.7.3.jar,guava-12.0.1.jar (for spark with python)
1. hadoop-aws-2.6.0.jar
2. aws-java-sdk-1.7.3.jar
3.guava-12.0.1.jar
spark-shell --jars hadoop-aws-2.6.0.jar,aws-java-sdk-1.7.3.jar,guava-12.0.1.jar (for spark with scala)
pyspark --jars hadoop-aws-2.6.0.jar,aws-java-sdk-1.7.3.jar,guava-12.0.1.jar (for spark with python)
No comments:
Post a Comment