Save list over many csv files each with given number of lines

Question

I wrote a little code for outputting to csv. It takes a list object named outtext and saves multiple .csv files. Each csv file contains cutoff elements of the list (except for the last), where cutoff is a specified number of elements/lines. This is useful when the user has to avoid writing to too large files (i.e. GitHub restricts file sizes too 100MB). The filenames are numbered from 0 to n, where n is the length of the output object divided by cutoff.

However, the code looks quite clunky and is quite long, given that it performs a relatively simple task:

import csv
import math
# dummy content
outtext = mylist = [None] * 300000
# Parameters specified by user
output_file = "path/name.csv"
cutoff = 150000
output_file_tokens = output_file.rsplit('.', 1)
num_files = int(math.ceil(len(outtext)/float(cutoff)))
for filenumber in range(num_files):
 counter = 0
 output_file = output_file_tokens[0] + str(filenumber) + "." + output_file_tokens[1]
 while counter <= cutoff:
 with open(output_file, 'wb') as f:
 writer = csv.writer(f)
 for line in outtext[:cutoff]:
 writer.writerow(line)
 counter += 1
 del outtext[:cutoff]
 print ">>> " + output_file + " successfully saved"

Is there room for improvement?

• SuperBiasedMan: Where does outtext come from? It's not defined in this script.
• MERose: outtext is just the text to write out. I thought it's not necessary to understand the code - sorry.

Answer 1

I don't know where outtext comes from, but this seems overly complicated. There's too much nesting, and there's not enough separation of concerns.

Let's back up. We want to (a) chunk our list into sizes of cutoff and (b) write each such chunk into a new file. For each chunk, we create a new file, and write all of the rows. Each of those problems is bite-size and manageable.

First, from How do you split a list into evenly sized chunks?:

def chunks(l, n):
 """Yield successive n-sized chunks from l."""
 for i in xrange(0, len(l), n):
 yield l[i:i+n]

That gives us the first half of the problem:

for chunk in chunks(outtext, cutoff):
 # write chunk

Now for the second half, we just need an index. But rather than keeping our own counter, let's use enumerate(). Additionally, note that csvwriter has a writerows() method - so we don't need a loop for that.

Putting it all together:

for index, chunk in enumerate(chunks(outtext, cutoff)):
 output_file = '{}{}.{}'.format(output_file_tokens[0], index, output_file_tokens[1])
 with open(output_file, 'wb') as f:
 writer = csv.writer(f)
 writer.writerows(chunk)
 print ">>> " + output_file + " successfully saved"

You don't need to del anything. That's going to be pretty slow too - everytime we delete a chunk we have to shift everything in memory. This way, we don't have to do that.

• MERose: I didn't like the counter as well and knew about writerows(), but I didn't know how to do it without counter. Thanks for the great explanation - I learned something!