A Simple Scala Spark Project Template and Guide

Spark in Production

Like anything else, when putting things into production it's important that behavior is well scoped and understood. Things like testing come to the forefront of what you're doing and it's important to make sure that things are well scoped. After writing a couple of different projects in Scala and Spark I saw some repeated approaches and patterns I was taking, especially with regard to testing. I wanted to write and share a simple base application that people can use for themselves. If you'd rather just take a look at the template you can find it on my github. (https://github.com/anabranch/Apache-Spark-Scala-Project-Template)


There are a couple of requirements that I wanted out of my project scaffold, I wanted logging that was simple to include, I wanted to make sure that testing was easy and extensible. Lastly, I wanted a simple example to help jog my memory of how I like to start my projects.


I wanted to make sure that testing was both easy and intuitive in the application. I wanted to make sure that testing involved the true Spark datatypes with everything from an RDD to a DataFrame. The framework I have chosen to build on is ScalaTest, as it's got a great way of writing Spark tests. You do have to jump through some hoops regarding parallel test cases as well as trying to prevent multiple Spark contexts from being created but this template makes it easy to at least make it so that too many don't overlap. The most important thing to do is to be sure to sub-class the ExampleTest.scala, you simply write your test as you would a normal ScalaTest.
package net.sparktutorials.scaffold

import org.scalatest._
import org.apache.spark._

class Example extends BaseSpec {
  private var dataCobbler: ExampleClass = _

  override protected def beforeAll(): Unit = {
    dataCobbler = new ExampleClass()

  "I Should be Able to Cobble a List" should "and it should have the right output" in {
    val toCobble = sc.parallelize(Seq("Shoe", "Boot", "Foo"))
    toCobble.map(dataCobbler.cobble).count should be (3)
    toCobble.map(dataCobbler.cobble).first should be ("Cobble the Shoe")

With this framework it's easy to build and execute tests and it always helps to get an understanding from the test suite about how well and extensively you've tested your code. By including the coverage tool, getting test coverage is as simple as:
sbt clean coverage test


My second major requirement was logging. I wanted it to be easy for me to remember how to include my logger when starting new projects. Luckily, it's incredibly simple to do! Hat's off to include your logger in a serializable way by use of the transientannotation in Scala. This annotation prevents the field from getting serialized.
@transient lazy val logger = Logger.getLogger(getClass.getName)
You can read more about annotations here.


While you may want to structure your projects different. This set up works and is the one that I will continue to update. If you want to contribute please feel free to submit a pull request!

Questions or comments?

comments powered by Disqus