Reading lines from a file and processing them concurrently

Question

I'd like to read lines from a file and process them concurrently. I came up with the following code to do this:

var wg sync.WaitGroup
func main(){
 file := "data/input.txt"
 reader, _ := os.Open(file)
 scanner := bufio.NewScanner(reader)
 for scanner.Scan() {
 wg.Add(1)
 go processLine(scanner.Text())
 }
 wg.Wait()
}
func processLine(line string) {
 time.Sleep(time.Duration(rand.Intn(5)) * time.Second)
 fmt.Println("line:", line)
 wg.Done()
}

I added the random sleep time in there to simulate potential differences in processing times.

Is there any potential drawbacks that I should be aware of with this method of concurrent processing? Are there any better ways that I should consider concurrently processing lines in a file?

• Mast: Did you try it? Does it work as intended?
• peterSO: This is artificial code. This is not "actual code from a project rather than pseudo-code or hypothetical code." codereview.stackexchange.com/help/dont-ask
• Abe Miessler: @PeterSO - Why do you say that? The thing I am asking about improving is exactly as it will appear in the program I'm working on. I left the processLine function empty because it's not relevant to the question i'm asking
• Abe Miessler: @Mast - it works exactly as intended. There is no question about that - only whether I am not following best practices some way or if there is some problem that I might be missing in non-trivial examples.
• Abe Miessler: @peterSO - to address the link you posted, it says: Questions should not contain purely generic, hypothetical code, such as This is clearly not "...purly generic, hupothetical code". The concurrency code (which is what I'm asking about) is what will be used in my production application

Answer 1

The general approach seems fine. You may want to benchmark your application based on the type of inputs you'll receive and the type of computation that's done. If the computation is CPU intensive, it may not make sense to have too many goroutines running in parallel since they can't all use the CPU at the same time.

If you find that's the case, it could be better to have a channel into which you send the lines that are being read and have a bunch of worker goroutines that read a line from the channel, process it, read another line and so on. Benchmarking should give you a good idea about the right approach.

If each goroutine makes a request to an external resource (such as a web service or a database) you'll have to think about ways to limit the rate of such requests.

Some general comments about the code which I think you should fix before putting it into production:

1. The code doesn't have any error checking or a way to report errors from the processing of each input line. You'll have to add those in.
2. You're also using a global variable to share the WaitGroup, which I think you should avoid. Either pass it in as a function parameter or use an anonymous function that calls processLine() and then wg.Done() (instead of calling wg.Done() in processLine())
3. As I mentioned above you may want to consider limiting the number of concurrent goroutines depending on your use case.