Reading a file representing word frequencies in Wikipedia, for clustering analysis

Question

I am writing a system that will run Kmeans to determine the most used words in Wikipedia. I do this by building a core set with streaming data. Wikipedia is 50TB but after a year of processing I was given data of size 14 GB which I need to parse and I want it to run fast. From client I send server 10000 points which are processed.

CODE:

import gc
import time
class point:
 def __init__(self, x, y, w):
 self.x = x
 self.y = y
 self.w = w
def parse(pathToFile):
 myList = []
 with open(pathToFile) as f:
 for line in f:
 s = line.split()
 x, y, w = [int(v) for v in s]
 obj = point(x, y, w)
 gc.disable()
 myList.append(obj)
 gc.enable()
 return myList
if __name__ == "__main__":
 startTime = time.time()
 L = parse('C:/Users/user/workspace/finalSub/api/data100mb.txt')
 print("--- %s seconds ---" % (time.time() - startTime))

Parsing a 114MB file takes 130 seconds when I've been told it should take few seconds.

I tried splitting the data to multiple chunks and apply multiprocessing, but it ended up that reading from multiple files is bad and results in even longer time time to parse. Look here.

File Sample:

1 1 1
1 2 1
1 3 1
1 4 1
.
.
1 N 1
2 1 1
2 2 1
2 3 1
2 4 1
.
.
2 N 1

How should I parse the file correctly or access it in-order to make it faster?

• Mahmood Muhammad Nageeb: In Python a class's name should be capitalized. So your class's name should be Point. Variables and functions' names should be written in lower case with an underscore if the name consists of more than one word. So your variables' names should be: my_list, l and start_time, respectively.
• Tony Tannous: @MahmudMuhammadNaguib does that affect runtime ?
• Mahmood Muhammad Nageeb: I don't think so. It's about style. This is a comment, not intended to be an answer. See: python.org/dev/peps/pep-0008/#naming-conventions
• 200_success: What do you intend to do with the results? What is supposed to happen when the middle coordinate is N?
• Tony Tannous: @200_success yes sir, I am writing a system that will run Kmeans to determind the most used words in Wikipedia. I do this by building a coreset with streaming data. Wikipedia is 50TB but after a year of processing I was given data of size 14 GB which I need to parse and i want it to run fast. From client I send server 10000 points which are processed.

Answer 1

Avoid reinventing-the-wheel, especially with common tasks that are likely to have optimized implementations. In this case, I recommend trying pandas.read_csv() to read the file. Then, the matrix can be fed to sklearn.cluster.KMeans().

• Tony Tannous: It is a problem. I must have the points in a list, or the 14GB will grow to 50TB. A Wikipedia article contains 5000 words for example, w is the weight which is always 1. each point corresponds to a (x, y) where x is the wikipedia article and y corresponds to the word. The word else might be 20000 If I turn this into a matrix I will have a huge space waste. Or am I missing something, sir. Each word is assigned a number id. i.e Else --> 20000
• 200_success: Why would the resulting matrix from read_csv() be any larger than your list of points? Either way, it's the same two-dimensional table containing the same data.
• Tony Tannous: The bottleneck ended up being the data being sent to server. Sending objects was costly! moved to binary and it went 40 times faster. :)

Answer 2

I would try the following things:

• remove the gc manipulation code

• use __slots__ for the Point class - should result into memory handling and performance improvements

  class Point:
   __slots__ = ['x', 'y', 'w']
   def __init__(self, x, y, w):
   self.x = x
   self.y = y
   self.w = w
  

• use a list comprehension instead of appending (generally faster):

  def parse(path):
   with open(path) as f:
   return [Point(*(int(s) for s in line.split())) 
   for line in f]
  

• try the PyPy Python implementation (with the latest PyPy on my machine the 100k line file is parsed about 4 times faster than the Python 3.6)

• Tony Tannous: Up from 130seconds to 170 seconds. :-(
• alecxe: @TonyTannous what about the PyPy?
• Tony Tannous: I need to try it. Though I rather a software gain and not some other technique from outside code.