scala语言的spark实现wordcount

生成rdd

val lines = sc.parallelize(Array("Hello Spark","Hello hadoop","Hello hadoop"))

 或者从文件读

val lines = sc.textFile("//......")

 切分

val words= lines.flatMap(x=>x.split(" "))

 切分后是这样

写道

res0: Array[String] = Array(Hello, Spark, Hello, hadoop, Hello, hadoop)

 

然后开始变成键值对形式，并且计算

val result=words.map(x=>(x,1)).reduceByKey((x,y)=>x+y)

 计算出结果就是

写道

res1: Array[(String, Int)] = Array((Hello,3), (Spark,1), (hadoop,2))

 还有更省事的方法 

val words= lines.flatMap(x=>x.split(" ")).countByValue()

 结果

写道

words: scala.collection.Map[String,Long] = Map(Hello -> 3, Spark -> 1, hadoop -> 2)