Processing many NetCDF files using ArcPy gives Serious Application Error

Question

I am processing more than 5k NetCDF files in ArcPy, using ArcGIS Desktop 10.5. I am doing it using the Python console of ArcMap. The process runs well, but when around 113 files are done, the process breaks down. The ArcMap gives an ERROR(attached at the end).

Does anyone know if there is a limitation with the number of files to process?

The process is:

1. Read the netCDF files --> arcpy.MakeNetCDFRasterLayer_md

2. Apply Zonal Statistics as table -->arcpy.gp.ZonalStatisticsAsTable_sa

3. Export the results to a CSV

The code is:

in_fd="is the netCDF files directory. More than 5k files."
shape= "is the shapefile where the zonal statistics will be applied."
names=[]
path=[]
dic={}
for root,folder,files in os.walk(in_fd):
for file in files:
 if os.path.isfile(os.path.join(root,file)) and file.endswith('.nc'):
 path.append(os.path.join(root,file))
 names.append(os.path.basename(file))
arcpy.env.workspace = r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany5円_Process3円_Task21円_Data4円_FWI\MIKEL\FWI_mv.gdb"
arcpy.env.overwriteOutput = True
for i in range (0,len(path)):
 print("#############")
 print("Empieza el: " + names[i])
 ras=arcpy.MakeNetCDFRasterLayer_md(in_netCDF_file=path[i], variable="dc", x_dimension="longitude", y_dimension="latitude", out_raster_layer=names[i][:-3], band_dimension="", dimension_values="", value_selection_method="BY_VALUE")
 table=arcpy.gp.ZonalStatisticsAsTable_sa(shape, "Id", ras, os.path.join(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany5円_Process3円_Task21円_Data4円_FWI\MIKEL\Statistics_DC","id_"+str(names[i][13:21])), "DATA", "ALL")
 with arcpy.da.SearchCursor(table,"*") as cursor:
 for row in cursor:
 dic[names[i][13:21]]=row[4],row[5],row[7],row[8]
 ls=[]
 vls=[(k,)+v for k,v in dic.items()]
 for row in vls:
 ls.append(row)
 if os.path.exists(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany5円_Process3円_Task21円_Data4円_FWI\MIKEL\CSV_mv\FWI_DC.csv"):
 os.remove(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany5円_Process3円_Task21円_Data4円_FWI\MIKEL\CSV_mv\FWI_DC.csv")
 with open(r"Z:\EMSV-064_IM_for_EMSN071_ForestFireHazard_Germany5円_Process3円_Task21円_Data4円_FWI\MIKEL\CSV_mv\FWI_DC.csv","wb")as out:
 csv_out=csv.writer(out)
 csv_out.writerow(['Fecha','MIN','MAX','MEAN','STD'])
 for row in ls:
 csv_out.writerow(row)
 arcpy.Delete_management(ras)
 arcpy.Delete_management(table)

The error is:

enter image description here

• Vince: SAE messages are really an issue for Esri Tech Support. You didn't specify the version in use, or give an indication of trying a standalone script, or trying the 64-bit Background Geoprocessing environment, so there isn't much GIS SE can offer.
• PolyGeo: You say this happens around the 113th file. What happens when you process the 112th to 115th files individually? And the the same four files using the same code tweaked to only do those four?
• MikV89: @PolyGeo. If I process the files individually, it works well, and if I run the process with 112th, 113th, 114th and 115th files also. I suggest that It is a space issue. I sent the error report to Esri, asking about the problem.

Answer 1

Your issues sounds very similar to a problem I had.

I very recently had to do a large amount of processing using the zonal stats as table tool and was hitting a catastrophic error that was completely bombing out my code, no error trapping could capture it.

I discovered the issue was with the zonal stats tool unable to process zones that were overlapping polygons. Rather failing gracefully and allowing me to trap the problem it just killed the main code.

With some research on the internet I finally managed a work around which I document here on another Q&A. I spawn subprocesses and run the zonal stats code in the subprocess, when it inevitably failed it returned to the calling code which was able to skip and continue.

• MikV89: Thanks, @Hornbydd, it's very useful.