Spurious ".tif Exists" errors on QGIS Python script processing

Question

I'm trying to automate geoprocessing by editing simple python scripts generated from "Save as Python" and running them within the QGIS script editor framework. I am using QGIS v2.18.7 my Mac (OS X Yosemite 10.10.5).

I've noticed that when I process multiple files (currently using gdalogr:warpreproject, with 12 layers run sequentially), I always get errors of the type "ERROR 1: Output dataset /path/file.tif exists" for at least some of the operations. The files in question do not actually exist as far as I can tell (I have tried using the temporary file option as well as writing to specific files using various strange file names to be sure). Furthermore, when I rerun, it tends to alter which layers are generating the error (e.g. months 01, 04, 05 on one run and months 03, 08, 09 for the next run). There is also an error on the last layer merge operation, which I am at this point guessing is a consequence of the errors in the input layers.

What might be causing these apparently spurious tif exists errors? Is there a fix or work-around? Is there a way to specifically flag -overwrite within the runalg() framework? (as suggested here: Can't clip vector layer from a raster layer I created)

---

Example code:

##WarpStack12=name
##m01=raster
##m02=raster
##m03=raster
##m04=raster
##m05=raster
##m06=raster
##m07=raster
##m08=raster
##m09=raster
##m10=raster
##m11=raster
##m12=raster
##YearDat =output raster
outputs_GDALOGRWARPREPROJECT_1=processing.runalg('gdalogr:warpreproject', m01,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None,None)
outputs_GDALOGRWARPREPROJECT_2=processing.runalg('gdalogr:warpreproject', m02,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_3=processing.runalg('gdalogr:warpreproject', m03,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_4=processing.runalg('gdalogr:warpreproject', m04,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_5=processing.runalg('gdalogr:warpreproject', m05,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_6=processing.runalg('gdalogr:warpreproject', m06,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_7=processing.runalg('gdalogr:warpreproject', m07,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_8=processing.runalg('gdalogr:warpreproject', m08,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_9=processing.runalg('gdalogr:warpreproject', m09,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_10=processing.runalg('gdalogr:warpreproject', m10,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_11=processing.runalg('gdalogr:warpreproject', m11,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRWARPREPROJECT_12=processing.runalg('gdalogr:warpreproject', m12,'EPSG:4326','EPSG:32616',None,500.0,0,'-87.75,-83.1,10.65,15.1','EPSG:4326',1,0,75.0,6.0,1.0,False,0,False,None, None)
outputs_GDALOGRMERGE_1=processing.runalg('gdalogr:merge', [
 outputs_GDALOGRWARPREPROJECT_1['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_2['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_3['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_4['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_5['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_6['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_7['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_8['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_9['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_10['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_11['OUTPUT'],
 outputs_GDALOGRWARPREPROJECT_12['OUTPUT'],
 ],False,True,1,YearDat)

Example error output (in this example, error shows up on 1st, 4th, and 11th layers. If I rerun, it may show up for different layers):

2017年06月15日T13:15:05 0 GDAL execution console output ERROR 1: Output dataset /var/folders/8y/6924gdvx3jg6k2rz5yrzwz100000gr/T/processing1524180f5e1443d3a74b4fdc0a8bd484/558b56639752426697b3345fd326a7c3/OUTPUT.tif exists, but some command line options were provided indicating a new dataset should be created. Please delete existing dataset and run again.

 2017年06月15日T13:15:05 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001032.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001060.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output ERROR 1: Output dataset

/var/folders/8y/6924gdvx3jg6k2rz5yrzwz100000gr/T/processing1524180f5e1443d3a74b4fdc0a8bd484/7fa79d45d41e40cdb0651f2242a00699/OUTPUT.tif exists, but some command line options were provided indicating a new dataset should be created. Please delete existing dataset and run again.

 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001121.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001152.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001182.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001213.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001244.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:06 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001274.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:07 0 GDAL execution console output ERROR 1: Output dataset

/var/folders/8y/6924gdvx3jg6k2rz5yrzwz100000gr/T/processing1524180f5e1443d3a74b4fdc0a8bd484/ee42542a5c0e434ea3f5c6156f9935bf/OUTPUT.tif exists, but some command line options were provided indicating a new dataset should be created. Please delete existing dataset and run again.

 2017年06月15日T13:15:07 0 GDAL execution console output Creating output file that is 1003P x 991L.
 Processing input file /QGISWork/MODFireBurned/Tiffs/2001/MCD64monthly.A2001335.Win04.006.burndate.tif.
 
 0...10...20...30...40...50...60...70...80...90...100 - done.
 2017年06月15日T13:15:07 2 Uncaught error while executing algorithm Traceback (most recent call last):
 File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/core/GeoAlgorithm.py",

line 203, in execute self.processAlgorithm(progress) File "/Applications/QGIS.app/Contents/MacOS/../Resources/python/plugins/processing/script/ScriptAlgorithm.py", line 378, in processAlgorithm exec((script), ns) File "", line 42, in NameError: name 'YearDat' is not defined

• Steven Kay: not sure if a typo, but there's a spurious space in your ##YearDat line. Try removing that in case the extra space has been added to the input name. Might explain the error you're getting (" NameError: name 'YearDat' is not defined")
• Tom: @StevenKay Eliminating the space fixed all the problems. The code now executes correctly. I have to admit to being surprised that this was the cause, but I'm often surprised when debugging anything. I'm happy to post an answer for the community's benefit, but invite you to since you answered the question in a comment.

Answer 1

there's a typo in this line...

##YearDat =output raster

the extra space before the equals sign seems to be significant. It looks as if processing includes the space in the input name, which is why you see the error

NameError: name 'YearDat' is not defined

Processing is looking for a field called "YearDat " (with a trailing space) rather than "YearDat"

I'd also recommend not using spaces in filenames or layer names, and using "_" (underscore) instead, when using GRASS or SAGA algorithms, for the same reason :)