首页 > Spark的几种运行模式

spark的三种模式有什么区别,Spark的几种运行模式

互联网 2020-11-28 15:48:32

1.local单机模式,结果xshell可见:./bin/spark-submit --class org.apache.spark.examples.SparkPi --master local[1] ./lib/spark-examples-1.6.0-hadoop2.4.0.jar 1002.standalone集群模式之client模式:conf/spark-env.sh添加export JAVA_HOME=/root/install/jdk1.7.0_21export SPARK_MASTER_IP=spark1export SPARK_MASTER_PORT=7077export SPARK_WORKER_CORES=1export SPARK_WORKER_INSTANCES=1export SPARK_WORKER_MEMORY=1gvi slaves添加node2node3

rm -rf slaves.templaterm -rf spark-env.sh.template结果xshell可见:./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node1:7077 --executor-memory 1G --total-executor-cores 2 ./lib/spark-examples-1.6.0-hadoop2.4.0.jar 1003.standalone集群模式之cluster模式:结果spark001:8080里面可见!./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://node1:7077 --deploy-mode cluster --supervise --executor-memory 1G --total-executor-cores 1 ./lib/spark-examples-1.6.0-hadoop2.4.0.jar 1004.Yarn集群模式,结果spark001:8088里面可见:在conf/spark-env.sh里添加export HADOOP_CONF_DIR=/home/install/hadoop-2.5/etc/hadoop./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-memory 1G --num-executors 1 ./lib/spark-examples-1.6.0-hadoop2.4.0.jar 100com.spark.study.MySparkPi./bin/spark-submit --class com.spark.study.MySparkPi --master yarn-client --executor-memory 1G --num-executors 1 ./data/spark_pagerank_pi.jar 100

免责声明:非本网注明原创的信息,皆为程序自动获取自互联网,目的在于传递更多信息,并不代表本网赞同其观点和对其真实性负责;如此页面有侵犯到您的权益,请给站长发送邮件,并提供相关证明(版权证明、身份证正反面、侵权链接),站长将在收到邮件24小时内删除。

相关阅读