Read lines from file, calling a consumer function for each line

Question

I wrote a function for iterating over a large file with performance in mind. It takes in an InputStream and reads until it reaches the end of the file. Whenever it detects an end of line it will call the UnsafeConsumer#accept with the line.

• UnsafeConsumer is the same as Consumer<T> but has a throws declaration on accept function
• stream has to be closed outside the function

I am am not sure if this is the fastest way to do this. Is there anything different that I could do?

public static void fileLines(InputStream stream, UnsafeConsumer<String,IOException> cons) throws IOException{
 StringBuilder lineBuild=new StringBuilder();
 int character;
 while(true){ //read file
 while(true){ //read line
 character=stream.read();
 if(character==-1){
 cons.accept(lineBuild.toString());
 lineBuild.setLength(0);
 return;
 }
 if(character=='\n'){
 cons.accept(lineBuild.toString());
 lineBuild.setLength(0);
 character=stream.read();
 if(character==-1) return;
 else if(character!='\r'&&character!='\n') lineBuild.append((char)character);
 break;
 }
 lineBuild.append((char)character);
 }
 }
}

• Emily L.: Did you benchmark? What other variations did you try?

Answer 1

You are conflating bytes with characters. An InputStream produces bytes, but you implicitly interpret each byte as a character when you do character=stream.read() followed by the cast lineBuild.append((char)character). To be clear:

• Strings consist of chars, not bytes.
• InputStreams and OutputStreams deal with bytes.
• Readers and Writers deal with characters.

To read lines as strings, first convert stream to a Reader using InputStreamReader.

BufferedReader br = new BufferedReader(new InputStreamReader(stream, "ISO-8859-1"));

Then, BufferedReader offers a .readLine() method. Or, since you seem to be interested in a stream-based approach, call .lines(), which produces a Stream<String>.

Answer 2

You can use the LineIterator from Apache commons IO library.

For your case it may be like:

LineIterator it = FileUtils.lineIterator(file, "ISO-8859-1");
try {
 while (it.hasNext()) {
 cons.accept(it.nextLine());
 }
} finally {
 it.close();
}

---

UPDATE (follow up of comments)

In case when you don't have file instance use IOUtils:

LineIterator it = IOUtils.lineIterator(stream, "ISO-8859-1");

• 200_success: Where does file come from? And why did you choose UTF-8, breaking compatibility with the original code?
• Dmytro Maslenko: You are right, file is missed in the given question, but the author mentioned about some large file. I assume a file instance is presented as well. "UTF-8" is just an example, there is a method without this parameter.