How to Execute Spark Scala Script File using Spark-shell

Assume you have a Spark Program written through Scala. Now you want to run this Scala program through Spark-Shell with some conf properties. Please see below how this needs to be run.

Spark-Shell comamnd:

spark-shell –master yarn-client  –conf spark.shuffle.spill=true  –conf spark.executor.extraJavaOptions=-XX:MaxPermSize=512m –conf spark.sql.planner.externalSort=true –conf spark.akka.frameSize=1024 –conf –conf spark.shuffle.memoryFraction=0.2 –conf –conf spark.shuffle.manager=sort  –conf spark.ui.port=40445 –conf spark.yarn.executor.memoryOverhead=4096 –executor-cores 1  –executor-memory 15g –driver-memory 20g –queue default –num-executors 30 -i /app/data/workspace/testprogram.scala

This will launch SparkContext(sc), sqlContext(ie.,HiveContext) in your spark-shell and load your script file and start executing it.

See below:

16/01/25 13:57:45 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
16/01/25 13:57:45 INFO SparkILoop: Created spark context..
Spark context available as sc.
16/01/25 13:57:45 WARN SparkConf: The configuration key ‘spark.yarn.applicationMaster.waitTries’ has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key ‘’ instead.
16/01/25 13:57:46 INFO HiveContext: Initializing execution hive, version 0.13.1
16/01/25 13:57:46 INFO metastore: Trying to connect to metastore with URI thrift://n01bdl600.aap.csaa.pri:9083
16/01/25 13:57:54 WARN metastore: Failed to connect to the MetaStore Server…
16/01/25 13:57:54 INFO metastore: Trying to connect to metastore with URI thrift://n01bdl601.aap.csaa.pri:9083
16/01/25 13:57:54 INFO metastore: Connected to metastore.
16/01/25 13:57:54 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
16/01/25 13:57:55 INFO HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
16/01/25 13:57:55 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
Loading /app/data/workspace/testprogram.scala…



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s