bigdata►spark

scala计算用户基站停留时间

测试数据

18688888888,20160327082400,16030401EAFB68F1E3CDF819735E1C66,1
18611132889,20160327082500,16030401EAFB68F1E3CDF819735E1C66,1
18688888888,20160327170000,16030401EAFB68F1E3CDF819735E1C66,0
18611132889,20160327180000,16030401EAFB68F1E3CDF819735E1C66,0

程序步骤

整理数据

1
2
3
4
5
6
7
8
9
//整理数据
val PhNoLacAndTime: RDD[((String, String), Long)] = baseLog.map(line => {
  val fields = line.split(",")
  val phoneNo = fields(0)
  val lac = fields(2)
  val eventType = fields(3)
  val time = if (eventType.equals("1")) -fields(1).toLong else fields(1).toLong
  ((phoneNo, lac), time)
})