spark 2.1.1

一 问题重现

spark-submit --master local[*] --class app.package.AppClass --jars /jarpath/zkclient-0.3.jar --driver-memory 1g app.jar

报错

Java HotSpot(TM) 64-Bit Server VM warning: Setting CompressedClassSpaceSize has no effect when compressed class pointers are not used
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /export/Logs/hadoop/g1gc.log due to No such file or directory

18/09/14 16:24:38 336 WARN Utils66: Set SPARK_LOCAL_IP if you need to bind to another address
Exception in thread "main" java.lang.NoClassDefFoundError: org/I0Itec/zkclient/exception/ZkNoNodeException
at kafka.utils.ZKGroupDirs.consumerDir(ZkUtils.scala:727)
at kafka.utils.ZKGroupDirs.consumerGroupDir(ZkUtils.scala:728)
at kafka.utils.ZKGroupTopicDirs.consumerOffsetDir(ZkUtils.scala:733)
at app.package.AppClass.main(AppClass.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more

二 问题解析

ClassNotFoundException: org.I0Itec.zkclient.exception.ZkNoNodeException

1 --jars --jars /jarpath/zkclient-0.3.jar

/jarpath/zkclient-0.3.jar 通过 --jars 参数传递给 spark 应用

2 类 org.I0Itec.zkclient.exception.ZkNoNodeException 确实存在 /jarpath/zkclient-0.3.jar 中

javap -cp /jarpath/zkclient-0.3.jar org.I0Itec.zkclient.exception.ZkNoNodeException
Compiled from "ZkNoNodeException.java"
public class org.I0Itec.zkclient.exception.ZkNoNodeException extends org.I0Itec.zkclient.exception.ZkException {
public org.I0Itec.zkclient.exception.ZkNoNodeException();
public org.I0Itec.zkclient.exception.ZkNoNodeException(org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String, org.apache.zookeeper.KeeperException);
public org.I0Itec.zkclient.exception.ZkNoNodeException(java.lang.String);
}

3 --verbose 参数输出

Main class:
app.pacakge.AppClass
Arguments:

System properties:
spark.executor.logs.rolling.maxSize -> 1073741824
spark.driver.memory -> 1g
spark.driver.extraLibraryPath -> /export/App/hadoop-2.6.1/lib/native
spark.eventLog.enabled -> true
spark.eventLog.compress -> true
spark.executor.logs.rolling.time.interval -> daily
SPARK_SUBMIT -> true
spark.app.name -> app.pacakge.AppClass
spark.driver.extraJavaOptions -> -XX:+PrintGCDetails -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+HeapDumpOnOutOfMemoryError -XX:-UseCompressedClassPointers -XX:CompressedClassSpaceSize=3G -XX:+PrintGCTimeStamps -Xloggc:/export/Logs/hadoop/g1gc.log
spark.jars -> file:/jarpath/zkclient-0.3.jar,file:/jarpath/app.jar
spark.sql.adaptive.enabled -> true
spark.submit.deployMode -> client
spark.executor.logs.rolling.maxRetainedFiles -> 10
spark.executor.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
spark.eventLog.dir -> hdfs://hdfs_name/spark/history
spark.master -> local[*]
spark.sql.crossJoin.enabled -> true
spark.driver.extraClassPath -> /usr/lib/hadoop/lib/hadoop-lzo.jar:/export/App/spark/extlib/*
Classpath elements:
file:/jarpath/app.jar
file:/jarpath/zkclient-0.3.jar

可见 /jarpath/zkclient-0.3.jar 确实存在 classpath 中

综上 3 点，为什么还会报 ClassNotFoundException，这里需要两个背景知识：

一个是 spark submit 过程，详见 https://www.cnblogs.com/barneywill/p/9820684.html

一个是 java 的 classloader，详见 https://www.cnblogs.com/barneywill/p/10111633.html

三 问题回放过程

1 spark-submit
2 spark-class
3 org.apache.spark.launcher.Main
4 org.apache.spark.deploy.SparkSubmit
4.1 初始化 MutableURLClassLoader，将 --jars 和应用 jar 作为 classpath 添加到 MutableURLClassLoader 中（注意此时 zkclient-0.3.jar 在这里），然后通过 MutableURLClassLoader 来加载应用 class，也就是 AppClass，然后反射调用 main
4.2 AppClass.main 中间接调用 kafka.utils.ZKGroupDirs（kafka jar 位于 spark.driver.extraClassPath 中），ZKGroupDirs 由 MutableURLClassLoader 的父 classloader，也就是 AppClassLoader 来加载，kafka.utils.ZKGroupDirs 中依赖 org.I0Itec.zkclient.exception.ZkNoNodeException，也由 AppClassLoader 来加载，并不是由 MutableURLClassLoader 来加载，所以报加载不到，虽然看起来 zkclient-0.3.jar 确实在 classpath 中；

问题的解决需要将 zkclient-0.3.jar 添加到 AppClassLoader 的 classpath 中（即 spark.driver.extraClassPath）