【Spark】Windows环境下在IDEA中运行Spark程序报错及解决方案

【报错1】
【报错描述】

19/01/24 13:28:10 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78)
	at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:77)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:257)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:234)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:749)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:734)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:607)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2160)
	at scala.Option.getOrElse(Option.scala:120)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2160)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:322)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount$.main(SparkWordCount.scala:12)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount.main(SparkWordCount.scala)

【报错原因】运行IDEA的Windows环境没有配置HADOOP_HOME环境变量,导致IDEA找不到Hadoop二进制文件。
【解决方案】
step1. 在Windows环境下解压hadoop安装tar包hadoop-2.5.0-cdh5.3.6.tar.gz到E:\Program Files (x86)\目录中,安装包下载地址:http://archive.cloudera.com/cdh5/cdh/5/。
step2. 设置环境变量HADOOP_HOME为E:\Program Files (x86)\hadoop-2.5.0-cdh5.3.6。
step3. 解压hadoop-common-2.2.0-bin-32.rar压缩包,将压缩包中的文件全部复制到${HADOOP_HOME}/bin文件夹中。
step4. 重新运行程序,报错即可解决。

【报错2】
【报错描述】

19/01/24 13:28:10 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount$.main(SparkWordCount.scala:12)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount.main(SparkWordCount.scala)
19/01/24 13:28:10 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount$.main(SparkWordCount.scala:12)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount.main(SparkWordCount.scala)

【报错原因】程序中没有设置Master的运行模式,即Master运行在什么地方,本地local,standalone,yarn,还是mesos上。
【解决方案】Master运行在本地设置:

object SparkWordCount {
  def main(args: Array[String]): Unit = {
    // 本地文件
    val path = "data/word.txt"
    val savePath = s"result/wc/${System.currentTimeMillis()}"
    // 1. 创建SparkContext上下文
    val conf = new SparkConf().setMaster("local")
    val sc = new SparkContext(conf)

【报错3】
【报错描述】

19/01/24 13:29:51 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: An application name must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:404)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount$.main(SparkWordCount.scala:12)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount.main(SparkWordCount.scala)
19/01/24 13:29:51 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: An application name must be set in your configuration
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:404)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount$.main(SparkWordCount.scala:12)
	at com.ibeifeng.bigdata.spark.app.core.SparkWordCount.main(SparkWordCount.scala)

【报错原因】程序中没有设置运行的任务名称。
【解决方案】AppName可如下设置:

object SparkWordCount {
  def main(args: Array[String]): Unit = {
    // 本地文件
    val path = "data/word.txt"
    val savePath = s"result/wc/${System.currentTimeMillis()}"
    // 1. 创建SparkContext上下文
    val conf = new SparkConf().setMaster("local").setAppName("wordcount")
    val sc = new SparkContext(conf)
  • 0
    点赞
  • 1
    评论
  • 1
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

打赏
文章很值,打赏犒劳作者一下
相关推荐
©️2020 CSDN 皮肤主题: 书香水墨 设计师:CSDN官方博客 返回首页

打赏

魏晓蕾

你的鼓励将是我创作的最大动力

¥2 ¥4 ¥6 ¥10 ¥20
输入1-500的整数
余额支付 (余额:-- )
扫码支付
扫码支付:¥2
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者