Spark operator: RDD action Action operation – aggregate, fold, lookup; the difference between reduce/fold/aggregate
class=”htmledit_views”> Keywords: Spark operator, Spark function, Spark RDD Action, aggregate, fold, lookup aggregate def aggregate[U](zeroValue: U)(seqOp: (U, T) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: ClassTag[U]): U Aggregate users aggregate the elements in RDD, first use seqOp to aggregate the T-type elements in each partition in the RDD into U-type, and then use combOp to aggregate the previously aggregated U-type elements in each partition into U-type, Pay special attention to both seqOp and combOp will use the value of zeroValue, the type of zeroValue is U var rdd1 = sc. makeRDD(1 to 10,2) rdd1. mapPartitionsWithIndex{ (partIdx, iter) => { var part_map = scala.collection.mutable.Map[String,List[Int]]() while(iter.hasNext){ var part_name = “part_” + partIdx; var elem = iter. next() if(part_map. contains(part_name)) { var elems = part_map(part_name) elems ::= elems part_map(part_name) = elems } else { part_map(part_name) = List[Int]{elem} } } part_map.iterator } }.collect res16: Array[(String, List[Int])] = Array((part_0,List(5, 4, 3, 2, 1)), (part_1,List(10, 9, 8, 7, 6))) ##The first partition contains 5,4,3,2,1 ##The second partition contains 10,9,8,7,6 var rdd1 = sc. makeRDD(1 to 10,2) rdd1.aggregate(1)( {(x : Int,y : Int) => x + y}, {(a : Int,b : Int) => a + b} ) res0: Int = 58 The result is 58, see…
