Efficient way of reading in list of string csvs' into lists

Question

I have a list of strings. Each string is a comma separated list of numbers. I want to quickly convert each of these strings into a list. So far I have done the following:

My Solution So Far

def convertCommaSeperatedStringToList(data):
 s_lists = []
 for s in data:
 s_list = [int(x) for x in s.split(',')]
 s_lists.append(s_list)

Example Dataset

# Here we have 600 very long lists to process
import numpy as np
data = []
for x in range(0,600):
 a = (np.random.randint(1000,size=100000)).tolist()
 b = ','.join(str(x) for x in a)
 data.append(b)

Just some background if you're wondering why I would have such a wierd list, I am reading this list from a downstream database.

Answer 1

Fastest way would be to use a list-comprehension here. This would avoid the repetitive lookups to list.append and will use the LIST_APPEND byte code.

s_lists = [[int(x) for x in row.split(',')] for row in data]

Note that for function names underscores are preferred over camel-case.

• kPow989: Thanks that worked well.compression seems to be the trick

Answer 2

As mentioned in other answers, list comprehension is the fastest way to solve this. However, I'd just like to point out that you should also consider using a generator function if it suits your needs to prevent the need to load the entire dataset.

def convertCommaSeparatedStringToList(data):
 for s in data:
 yield [int(x) for x in s.split(',')]