Spark的flatmap与map.filter.map有什么异同

 A => Option[B] => B 这个过程

scala的  aList.map(B(_)).filter(!_.isEmpty).map(_.get)  和  aList.flatMap(B(_))  有什么差别？ 在 spark rdd 以及 sparksql 同样的操作又有什么异同呢？

--

• Scala

对Option而言，逻辑上flatMap就是map.filter.map的简写。

 

 

  def flatMap[B](f: A => GenTraversableOnce[B]): Iterator[B] = new AbstractIterator[B] {
    private var cur: Iterator[B] = empty
    private def nextCur() { cur = f(self.next()).toIterator }
    def hasNext: Boolean = {
      // Equivalent to cur.hasNext || self.hasNext && { nextCur(); hasNext }
      // but slightly shorter bytecode (better JVM inlining!)
      while (!cur.hasNext) {
        if (!self.hasNext) return false
        nextCur()
      }
      true
    }
    def next(): B = (if (hasNext) cur else empty).next()
  }

 

  def filter(p: A => Boolean): Iterator[A] = new AbstractIterator[A] {
    // TODO 2.12 - Make a full-fledged FilterImpl that will reverse sense of p
    private var hd: A = _
    private var hdDefined: Boolean = false

    def hasNext: Boolean = hdDefined || {
      do {
        if (!self.hasNext) return false
        hd = self.next()
      } while (!p(hd))
      hdDefined = true
      true
    }

    def next() = if (hasNext) { hdDefined = false; hd } else empty.next()
  }

 

  def map[B](f: A => B): Iterator[B] = new AbstractIterator[B] {
    def hasNext = self.hasNext
    def next() = f(self.next())
  }

 

 

• Spark RDD

和scala的类似，本身RDD的flatMap、map、filter的compute方法都是调用Scala集合类的对应方法。

 

• SparkSQL 

用explain查看执行计划，不理解的话打断点（打日志）查看生成的java类。

1. http://www.winseliu.com/blog/2016/10/12/sparksql-view-and-debug-generatecode/
2. https://www.zhihu.com/question/51544925