How to Resolve ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: “5s” in Spark Submit

Another Issue as below while submitting my spark job to the yarn cluster.

R yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: java.lang.NumberFormatException: For input string: “5s”
java.lang.RuntimeException: java.lang.NumberFormatException: For input string: “5s”
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:117)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:165)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:163)
at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:360)
at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:205)
at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:204)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:204)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:72)
at com.csaa.horf.initialLoad.CasDS.CasDataStore$.main(CasDataStore.scala:32)
at com.csaa.horf.initialLoad.CasDS.CasDataStore.main(CasDataStore.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:486)
Caused by: java.lang.NumberFormatException: For input string: “5s”
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492)
at java.lang.Integer.parseInt(Integer.java:527)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1258)
at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211)
at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
… 19 more

How did i resolve this?

Modifying hive-site.xml file helped me resolving this issue.

<property>
<name>hive.metastore.client.connect.retry.delay</name>
<value>5s</value>
</property>

to

<property>
<name>hive.metastore.client.connect.retry.delay</name>
<value>5</value>
</property>

<property>
<name>hive.metastore.client.socket.timeout</name>
<value>1800s</value>
</property>

to

<property>
<name>hive.metastore.client.socket.timeout</name>
    <value>1800</value>
</property>

After modifying this hive-site.xml file, I added this new hive-site.xml file to the Spark Classpath as below.

spark-submit –master yarn-cluster –class com.test.package.program –jars /usr/hdp/2.3.2.0-2950/spark/lib/datanucleus-api-jdo-3.2.6.jar –jars /usr/hdp/2.3.2.0-2950/spark/lib/datanucleus-core-3.2.10.jar –jars /usr/hdp/2.3.2.0-2950/spark/lib/datanucleus-rdbms-3.2.9.jar –files  /user/giri/sparksubmit/hive-site.xml application-jar application.properties

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s