Spark Will Not Start with Spark Address already in use

The Problem

Sometimes Spark just won't start. This is common when you're working on a shared cluster with other people who are working as well. While it can one of numerous issues there are some causes that are more likely than others. The typical error messages is something like:
15/07/21 20:41:21 WARN component.AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@13c8152c: Address already in use Address already in use
        at Method)
        at org.eclipse.jetty.server.AbstractConnector.doStart(
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(
        at org.eclipse.jetty.server.Server.doStart(
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(
        at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:192)
        at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:202)
        at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:202)
        at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446)
        at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
        at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442)
        at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:202)
        at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
        at org.apache.spark.SparkContext.(SparkContext.scala:224)
        at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:972)
Which might be repeated some time or just once. The most likely culprit is that someone else is running a Spark program currently or someone forgot to log out of the the Spark Shell.

Investigating the Causes

We can investiage with the following commands.
ps aux | grep spark
This command (if you're not already familiar) will look across all the users logged into the system and see if anyone is running a process with spark in the command. More often that not, this will show up with a user that is indeed running Spark. Sometimes it may be you! If it is indeed you, what you're going to need to do is kill the process (unless you can just close out of the shell yourself). If it's someone else you'll need to reach out to that person to ask them if they perhaps improperly shutdown their Spark Shell or app. Once you've killed that process, Spark will release the port and you can get back to work! If that didn't resolve your problem feel free to comment below and I'd be happy to help you out!

Questions or comments?

comments powered by Disqus