spark unit testing using spark-testing-base

manpreet Best Answer 2 years ago

I am exploring on how to write unit tests for my spark scala code. I came across spark-testing-base and upon trying to run a few standard examples i get an error

import org.scalatest.FunSuite
import com.holdenkarau.spark.testing.{SharedSparkContext}

class AppTest extends FunSuite with SharedSparkContext {
  test("test initializing spark context") {
    val list = List(1, 2, 3, 4)
    val rdd = sc.parallelize(list)

    assert(rdd.count === list.length)
  }
}

The error is as follows:

Class 'AppTest' must either be declared abstract or implement abstract member 'withFixture(test:Any):Outcome' in 'org.scalatest.SuiteMixin'

I am using IntelliJ IDEA for the same and my build.sbt looks as follows

name := ""

version := "1.0"

scalaVersion := "2.11.8"

fork in Test := true

libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "2.0.0",
  "org.apache.spark"  %% "spark-mllib"             % "2.0.0",
  "org.apache.spark"  %% "spark-sql"             % "2.0.0",
  "org.scalatest" %% "scalatest" % "1.9.1" % "test",
  "com.holdenkarau" %% "spark-testing-base" % "0.7.2" % "test"
)

parallelExecution in Test := false

I have never done unit testing before. I would appreciate any help with this issue

manpreet 2 years ago

import org.scalatest.FunSuite

class AppTest extends FunSuite {
  test("test initializing spark context") {
    val myLocalInProcessSession = SparkSession
                                    .builder
                                    .master("local[*]") //This is the key config change to make it a local in process spark session.
                                    .appName("myApp")
                                    .getOrCreate() 
    val list = List(1, 2, 3, 4)
    val rdd = myLocalInProcessSession.sparkContext.parallelize(list)

    assert(rdd.count === list.length)
    myLocalInProcessSession.close()
  }
}