How to filter "locally and remotely" in functional programming

Question

My example applies to reading and deleting files (I/O), but this is probably a common scenario (eg, keeping local and global state in sync in functional programming).

I am reading in files from a directory.

import { readdirSync } from 'fs'
const files = readdirSync('./myFolder')

I want to filter out the operating system file .DS_STORE from the local list, but not only do I want to filter it out of the list, I also want to delete it from the file system.

My first approach here looks like this:

import { readdirSync, unlinkSync } from 'fs'
const files = readdirSync('./myFolder')
const filteredList = files.filter(file => {
 if (file.endsWith('.DS_STORE')) {
 unlinkSync(file) // <-- side effect
 return false
 }
 return true
})

My problem here is that the filter is performing a side effect to keep local and global state in sync.

I could isolate the side effect like this:

import { readdirSync, unlinkSync } from 'fs'
const files = readdirSync('./myFolder')
const filteredList = files
 .map((file) => {
 if (file.endsWith('.DS_STORE')) unlinkSync(file)
 return file
 })
 .filter((file) => !file.endsWith('.DS_STORE'))

I've now isolated the side effect, but I now have to do the endsWith check twice, and I potentially would move the string .DS_STORE to a variable that is closured for both the map and the filter to use. I also feel weird about using a map to just perform a side effect and then pass along the data, although maybe I shouldn't worry too much about that.

Is there another approach here that is more elegant? Am I maybe overthinking this? From an FP practitioner's point of view, is the first filter-only example still an acceptable solution?

Answer 1

Use Map.groupBy to split the list into two parts and operate on the part you are interested in.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map/groupBy

const dsStore = { store = true } 
const notDsStore = { store = false}
const result = Map.groupBy(files, i =>
 i.endsWith('.DS_STORE')) ? dsStore : notDsStore,
);
result.get(notDsStore).map(i=>unlinkSync(i));

Kind of overkill for short lists though it's still O(n) just to loop through twice

const unlinked = files
 .filter((file) => file.endsWith('.DS_STORE'))
 .map(unlinkSync)
const linked = files
 .filter((file) => !file.endsWith('.DS_STORE'))

• rpivovar: I'm amazed how you found a more elegant solution using a js feature that just barely landed
• rpivovar: I'm also continuously amazed by the answers I get from the software engineering stack exchange
• rpivovar: yeah, I think what bothers me about the map + filter is that it just feels uncomfortable having the side effect in the map. I feel like the map should just be responsible for transforming the items in the list and that's it
• Ewan: tbh here I dont really think the side effect is the problem. I assume its necessary and you can't get rid of the need for it. The problem is function chaining doesn't play well with conditionals. Like imagine if you had a big switch statement with various stuff you wanted to do depending on some varied conditions in the file object. And then you had follow up things to do to some of the files. it would get crazy fast
• Ewan: ..you would have to add a lot of streams ;)

Answer 2

now have to do the endsWith check twice

That unconditional return file doesn't make sense for your use case.

You want the .map to return zero or more files which are not '.DS_STORE'. Return the value only if you aren't unlinking it. (And of course, this is not functional programming, given the FS side effect.)

Answer 3

Add a function with a file argument that calls unlinkSync just for .DS_STORE files and returns null otherwise returns the file argument it receives, then replace the second "ends with filter" with a "different than null filter". Untested code following:

import { readdirSync, unlinkSync } from 'fs'
const files = readdirSync('./myFolder')
var unlinkDsStore = (file) => { 
 if ( file.endsWith('.DS_STORE') ) {
 unlinkSync(file);
 return null;
 }
 return file;
};
const filteredList = files.map((file) => unlinkDsStore)
 .filter((file) => file != null)