-
Notifications
You must be signed in to change notification settings - Fork 241
private val fileSizes = inputFileScanRDD.filePartitions
.flatMap(_.files)
.groupBy(_.filePath)
.mapValues(_.map(_.length).sum)
.map(identity) // make this map serializable
```
fileSizes could be obtained from the underlying plan statistics ?
All reactions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment