java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterUtils$

Question

I was building this small demo code for Spark streaming using twitter. I have added the required dependencies as shown by http://bahir.apache.org/docs/spark/2.0.0/spark-streaming-twitter/ and I am using sbt to build jars. The project build successfully and only problem seems to be is- it is not able to find the TwitterUtils class. The scala code is given below

build.sbt

name := "twitterexample"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion,
"org.apache.bahir" %% "spark-streaming-twitter" % "2.1.0",
"org.twitter4j" % "twitter4j-core" % "4.0.4",
"org.twitter4j" % "twitter4j-stream" % "4.0.4"
)

The main scala file is

TwitterCount.scala

import org.apache.spark.streaming._
import org.apache.spark.streaming.twitter._
import twitter4j.Status

object TwitterCount {
    def main(args: Array[String]): Unit = {

    val consumerKey = "abc"
    val consumerSecret ="abc"
    val accessToken = "abc"
    val accessTokenSecret = "abc"
    val lang ="english"
    System.setProperty("twitter4j.oauth.consumerKey", consumerKey)
    System.setProperty("twitter4j.oauth.consumerSecret",consumerSecret)
    System.setProperty("twitter4j.oauth.accessToken",accessToken)
    System.setProperty("twitter4j.oauth.accessTokenSecret",accessTokenSecret)

    val conf = new SparkConf().setAppName("TwitterHashTags")
    val ssc =  new StreamingContext(conf, Seconds(5))
    val tweets = TwitterUtils.createStream(ssc,None)
    val tweetsFilteredByLang = tweets.filter{tweet => tweet.getLang() == lang}
    val statuses = tweetsFilteredByLang.map{ tweet => tweet.getText()}
    val words = statuses.map{status => status.split("""\s+""")}
    val hashTags = words.filter{ word => word.startsWith("#StarWarsDay")}
    val hashcounts = hashTags.count()
    hashcounts.print()


    ssc.start
    ssc.awaitTermination()
    }

Then I am building the project using

sbt package

and I submitting the generated jars using

spark-submit --class "TwitterCount" --master local[*] target/scala-2.11/twitterexample_2.11-1.0.jar

Please help me with this. Thanks

Answer 1

--class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)

You are missing package name in your code. Your spark submit command should be like this.

--class com.spark.examples.TwitterCount

Answer 2

I found the solution at last. java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags

I have to build the jars using

sbt assembly

but I'm still wondering what's the difference in jars that I make using

sbt package

anyone knows? plz share