Does ArcPy leak memory when importing CSV files?

Question

I'm trying to import a great number of csv files of about 250MB each. I wrote a python script for that as follows:

import gc
import glob
tables = glob.glob("C:\\Users\\emportella\\Documents\\ArcGIS\2517円")
for table in tables:
 arcpy.TableToGeodatabase_conversion(table, "C:\\Users\\emportella\\Documents\\ArcGIS\2517円\\Tables.gdb")
 gc.collect()

It works for the first 4 to 6 files and without giving any warning is closes. i've payed attention to memory usage in Windows Task Manager and the process cosumes more and more memory until it craches.

I've already tried using some different aproches like using:

env.workspace = "C:\\Users\\emportella\\Documents\\ArcGIS\2517円\\*.csv"
tables = arcpy.ListTables()
arcpy.TableToGeodatabase_conversion(tables, "C:\\Users\\emportella\\Documents\\ArcGIS\2517円\\Tables.gdb")

The TableToGeodatabase_conversion accepts list of tables but it craches the same way.

So there is the question is there a way to manage the memory used in arcpy framework?

• Tom: are you cleaning your vars as you iterate? ie "del tables" Is the code you posted the entire script?

Answer 1

Another approach might be to use the subprocess module to farm out each iteration to a separate process, each with its own memory space that is freed up as soon as the process exits. See this answer for an explanation: Work-around for slow processing times in ArcGIS 9.3 with Spatial Analyst functions

Answer 2

I can't speak to the memory consumption / crashes without actually profiling it. I can offer you an alternative workflow which works quite well for me in arcpy 10.1 (10.1 as it uses da.InsertCursors). The following code is meant to be used as a script tool, but it looks like you know enough about Python to modify it to loop through files and output unique ones.

import arcpy, os, csv
infile = arcpy.GetParameterAsText(0)
outtable = arcpy.GetParameterAsText(1)
outpath = os.path.dirname(outtable)
outname = os.path.basename(outtable)
temptable = "in_memory\\tempcsv"
temppath = os.path.dirname(temptable)
tempname = os.path.basename(temptable)
if arcpy.Exists(temptable):
 arcpy.Delete_management(temptable)
arcpy.CreateTable_management(temppath, tempname)
rfile = open(infile, 'rb')
csvreader = csv.reader(rfile)
index = 0
fieldnames = []
oidfieldnames = ["OID","FID","OBJECTID"]
for csvrow in csvreader:
 if index == 0:
 for column in csvrow:
 if column in oidfieldnames:
 column = column + "_ORIG"
 arcpy.AddField_management(temptable, column, "TEXT")
 fieldnames.append(column)
 arcpy.CreateTable_management(outpath, outname, temptable)
 arcpy.Delete_management(temptable)
 tablerows = arcpy.da.InsertCursor(outtable, fieldnames)
 else:
 tablerows.insertRow(csvrow)
 index = index + 1
rfile.close()