3

I am doing a batch watershed and analysis process on 135 points. This script worked great the first time I ran it.

Ever since the first time, however, I have had the script stop at random moments. CPU usage remains high, memory remains allocated, yet nothing is happening. Sometimes it will get through 100 features, sometimes only 15. It never has a specific geoprocessing function that trips the process up (hence all my print messages!) Also of note, unless I end the python.exe process via task manager, a lock remains on the feature class that the script was last working with. I don't know if that is helpful.

I've tried using del and delete_management on temp files and in scratch workspace but the same issue appears. I've manually deleted all files and folders after it doesn't work. I've even tried migrating to the local C:\ drive rather than external storage. Nothing works. Any ideas?

Here is my code (with sensitive info ##### -ed out). It may seem a little redundant in the beginning but I was trying to hash out any potential errors:

import os
from subprocess import Popen
import arcpy
arcpy.CheckOutExtension("spatial")
import arcpy.sa
arcpy.env.workspace = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\outputs"
arcpy.env.scratchWorkspace = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\scratch"
arcpy.env.overwriteOutput = True
arcpy.env.outputCoordinateSystem = arcpy.SpatialReference(102685)
OutputFolder = "C:\\Users\\####\Documents\\ArcGIS\\Watersheds\\outputs\\"
ScratchFolder = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\scratch\\"
ParcelsFinal = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\inputs\\ParcelsFINAL.shp"
Imperv_Merged = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\inputs\\Imperv_Merged.shp"
flowDir = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\inputs\\dir"
pointInput = "C:\\Users\\####\\Documents\\ArcGIS\\Watersheds\\inputs\\pourPoints.shp"
c = int(arcpy.GetCount_management(pointInput).getOutput(0))
pointRow = arcpy.SearchCursor(pointInput)
fieldList = arcpy.ListFields(pointInput)
try:
 for row in pointRow:
 feat = row.Shape
 outFeat = OutputFolder + row.Name.replace("-","_")
 outScratch = ScratchFolder + row.Name.replace("-","_")
 print "Starting watershed analysis for " + row.Name + "."
 print ""
 outWater_r = arcpy.sa.Watershed(flowDir,feat)
 print "Watershed raster for " + row.Name + " created succesfully."
 Watershed = arcpy.RasterToPolygon_conversion(outWater_r, outFeat + "_watershed.shp", "SIMPLIFY")
 print "Watershed polygon for " + row.Name + " created succesfully."
 tempParcelClip = arcpy.Clip_analysis(ParcelsFinal, Watershed, outScratch + "tempClip1.shp")
 print "Temp parcel clip complete"
 ParcelERA = arcpy.Dissolve_management(tempParcelClip, outFeat + "_parcels.shp", "SWM_ERA", "", "MULTI_PART", "DISSOLVE_LINES")
 print "Parcels clip complete"
 tempImpervClip = arcpy.Clip_analysis(Imperv_Merged, Watershed, outScratch + "tempClip2.shp")
 print "Temp impervious clip complete"
 tempIntersect = arcpy.Intersect_analysis([tempImpervClip,ParcelERA], outScratch + "tempIntersect.shp", "ALL")
 print "Temp intersect complete"
 ImpervERA = arcpy.Dissolve_management(tempIntersect, outFeat + "_impervious.shp", "SWM_ERA", "", "MULTI_PART", "DISSOLVE_LINES")
 print "Impervious clip complete"
 print "Clipping analysis for " + row.Name + " complete." 
 arcpy.AddField_management(Watershed, "W_ACRES", "DOUBLE", "", "", "")
 arcpy.CalculateField_management(Watershed, "W_ACRES", "!" + arcpy.Describe(Watershed).shapefieldname + ".AREA@ACRES!", "PYTHON", "")
 arcpy.AddField_management(ParcelERA, "P_ACRES", "DOUBLE", "", "", "")
 arcpy.CalculateField_management(ParcelERA, "P_ACRES", "!" + arcpy.Describe(Watershed).shapefieldname + ".AREA@ACRES!", "PYTHON", "")
 arcpy.AddField_management(ImpervERA, "I_ACRES", "DOUBLE", "", "", "")
 arcpy.CalculateField_management(ImpervERA, "I_ACRES", "!" + arcpy.Describe(Watershed).shapefieldname + ".AREA@ACRES!", "PYTHON", "")
 arcpy.AddField_management(Watershed, "VISTA_NUM", "TEXT", "", "", 10)
 arcpy.CalculateField_management(Watershed, "VISTA_NUM", '"' + row.Name.replace("-","_") + '"', "PYTHON", "")
 arcpy.AddField_management(ParcelERA, "VISTA_NUM", "TEXT", "", "", 10)
 arcpy.CalculateField_management(ParcelERA, "VISTA_NUM", '"' + row.Name.replace("-","_") + '"', "PYTHON", "")
 arcpy.AddField_management(ImpervERA, "VISTA_NUM", "TEXT", "", "", 10)
 arcpy.CalculateField_management(ImpervERA, "VISTA_NUM", '"' + row.Name.replace("-","_") + '"', "PYTHON", "")
 print "Calculations complete"
 c -= 1
 print "Watershed analysis for " + row.Name + " complete."
 print "There are " + str(c) + " points remaining."
 print ""
 print ""
except:
 print arcpy.GetMessages()
p = Popen("E:\####\GIS Data\Code\Scripts\upper.bat", cwd=r"E:\####\GIS Data\Code\Scripts")
stdout, stderr = p.communicate()
print "Uppercase conversion complete"
try:
 flist = arcpy.ListFeatureClasses()
 MergeSHP = arcpy.Merge_management(flist, "E:\\####\\####\\Modeling\\Watersheds\\MassWatersheds\\MergeWatersheds.shp")
 arcpy.AddField_management(MergeSHP, "CALC_ACRES", "DOUBLE", "", "", "")
except:
 print arcpy.GetMessages() 
PolyGeo
65.5k29 gold badges115 silver badges350 bronze badges
asked Sep 11, 2014 at 21:02
3
  • Are you trying to run multiple instances of python? Is this being executed from command line or from ArcCatalog? Commented Sep 11, 2014 at 21:25
  • Its being executed through IDLE. Commented Sep 12, 2014 at 11:11
  • 1
    Same as command line, just with more user friendliness. Are you trying to run this process with multiple instances of IDLE? arcpy processes have a tendency to collapse randomly if they aren't the only process using the system temp/tmp directory. ArcCatalog/ArcMap get around this by creating their own temp/tmp folder in the temp/tmp folder so each one has a unique 'scratch' space. If you are trying to only run one instance then we'll have to dig deeper. Commented Sep 14, 2014 at 22:09

2 Answers 2

2

The symptoms you describe are to me typical of running a script/tool with insufficient RAM available for the size of input datasets, in conjunction with anything else you are doing on your PC/laptop.

I recommend:

  • Close everything and reboot your machine
  • Run your script/tool without anything else running
  • (if you wish, watch your memory usage via Task Manager, and I suspect it will be creeping up)

If it still does not complete, then repeat the procedure using a PC/laptop with more RAM.

I had a lot of compute intensive models/scripts that worked great on all but my largest input datasets on a machine with 4Gb RAM, but on my largest datasets they threw random hangs and errors. As soon as I ran them on a machine with 12Gb RAM they ran fine.

answered Sep 11, 2014 at 21:52
2
  • Thanks for the suggestion. I mean my machine has 32gb of RAM so I'm not sure if that is the problem, but I tried running it overnight without anything else running and it got through 118 of them so that is good I suppose. Will keep looking... Commented Sep 12, 2014 at 11:06
  • Check to see if you're running it on 32-bit python or 64 bit python. I had a similar issue with large datasets that was fixed with using the 64 bit geoprocessing libraries. blogs.esri.com/esri/arcgis/2012/11/12/… Commented Sep 28, 2015 at 15:17
1

If you do not really need to print all those messages, then consider that printing messages are using a lot of RAM and I think disabling/reducing the amount of print commands will let you run the script.

So what you need to do is to comment all the print commands and run it again.

PolyGeo
65.5k29 gold badges115 silver badges350 bronze badges
answered Jan 4, 2016 at 10:04

Your Answer

Draft saved
Draft discarded

Sign up or log in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

Post as a guest

Required, but never shown

By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.