Is it possible to use memory layers in QGIS processing algorithms?

Question

I am building a QGIS plugin that connects to a MySQL database in the local network, and then adds a subset of one of the tables to an in-memory layer; the subset is based on data currency (only taking the most recent observation for each location where measurements are made). This memory layer is successfully created.

However I then want to run some geoprocessing algorithms, and I'm having trouble using the in-memory layer in any of them.

 self.stationuri = "point?crs=epsg:4326&field=id:integer&field={}:double&index=yes".format(self.cb_field.currentText())
 self.vlayer = QgsVectorLayer(self.stationuri,"scratch","memory")
 if not self.vlayer.isValid():
 raise Exception("Failed to create in-memory layer")
 self.vlayer.startEditing()
 for i,r in enumerate(result): # Result is row-by-row result of SQL query
 # Add features
 ...
 self.vlayer.commitChanges()
 self.vlayer.updateExtents()
 # Add layer to map
 QgsMapLayerRegistry.instance().addMapLayer(self.vlayer)
 # Layer is successfully added to map with all features and geometry
 # BELOW IS WHERE IT FALLS APART
 try:
 processing.runandload("gdalogr:gridinvdist",self.vlayer,self.cb_field.currentText(),2,0,0,0,0,0,0,0,'Float32',None) # None = in-memory output; I get the same error if I specify a string path and filename.
 except Exception, e:
 raise e

No exception is raised, but no output is produced or added to the TOC, but the following log is made in processing.log:

INFO|Mon May 04 2015 11:28:23|GDAL execution console output|/bin/sh: 1: /tmp/processing/bbebe7599c83446d9c2b03a251879657/OUTPUT.tif: not found|/bin/sh: 1: -zfield: not found||FAILURE: Source datasource is not specified.|Usage: gdal_grid [--help-general] [--formats]| [-ot {Byte/Int16/UInt16/UInt32/Int32/Float32/Float64/| CInt16/CInt32/CFloat32/CFloat64}]| [-of format] [-co "NAME=VALUE"]| [-zfield field_name] [-z_increase increase_value] [-z_multiply multiply_value]| [-a_srs srs_def] [-spat xmin ymin xmax ymax]| [-clipsrc <xmin ymin xmax ymax>|WKT|datasource|spat_extent]| [-clipsrcsql sql_statement] [-clipsrclayer layer]| [-clipsrcwhere expression]| [-l layername]* [-where expression] [-sql select_statement]| [-txe xmin xmax] [-tye ymin ymax] [-outsize xsize ysize]| [-a algorithm[:parameter1=value1]*] [-q]| <src_datasource> <dst_filename>||Available algorithms and parameters with their's defaults:| Inverse distance to a power (default)| invdist:power=2.0:smoothing=0.0:radius1=0.0:radius2=0.0:angle=0.0:max_points=0:min_points=0:nodata=0.0| Moving average| average:radius1=0.0:radius2=0.0:angle=0.0:min_points=0:nodata=0.0| Nearest neighbor| nearest:radius1=0.0:radius2=0.0:angle=0.0:nodata=0.0| Various data metrics| <metric name>:radius1=0.0:radius2=0.0:angle=0.0:min_points=0:nodata=0.0| possible metrics are:| minimum| maximum| range| count| average_distance| average_distance_pts|

The important part seems to be FAILURE: Source datasource is not specified. However self.vlayer.isValid() == True, so I don't see what is wrong with my input. I have tried substituting self.vlayer with 'memory:scratch' in the call to processing.runandload, but then I get the following error printed to the console (but not raised): Error: Wrong parameter value: memory:scratch.

I get the same issue when running this through the QGIS GUI, and using the dropdown menu to select my scratch layer which is in the TOC. This occurs whether I specify the output raster as in-memory or specify a location on disk.

This question seems similar, but their solution was to add the memory layer to the TOC before using it. I am already doing that and yet the error persists.

I thought that this was a general issue with memory layers and the QGIS geoprocessing algorithms, but the following works without issue:

processing.runandload("qgis:fixeddistancebuffer",self.vlayer, 500, 5, True, "output_buffer.shp")

What am I doing wrong? Why can't my memory source dataset "be specified" in some of the processing algorithms?

EDIT: here's the source code of gdalogr:gridinvdist if that's useful.

Answer 1

It seems like memory layers cannot be used as input for GDAL/OGR processing scripts because Processing fails to properly prepare the data for use with ogr2ogr. That's why, for example, the QGIS buffer tool works but the GDAL/OGR buffer tool fails:

Algorithm Buffer vectors starting...
GDAL command:
cmd.exe /C ogr2ogr.exe "C:\Users\anita\AppData\Local\Temp\processing70e5e0852cb9456ba2e3780f8386122e86円d237c8f41443f58a230a8133172047\OUTPUTLAYER.shp" point?crs=EPSG:4326&memoryid={6772bccd-f55d-461d-aff6-6271ded02eea} point?crs=EPSG:4326&memoryid={6772bccd-f55d-461d-aff6-6271ded02eea} -dialect sqlite -sql "SELECT ST_Buffer( geometry , 1000 ),* FROM 'point?crs=EPSG:4326&memoryid={6772bccd-f55d-461d-aff6-6271ded02eea}' " 
GDAL command output:
FAILURE: 
Unable to open datasource `point?crs=EPSG:4326' with the following drivers. 
-> JP2ECW 
-> OCI 
-> SOSI 
...

Processing would have to somehow prepare the data (save it to a file) and then feed it to the GDAL/OGR tool.

I opened a ticket: Cannot use memory layers with OGR tools

Answer 2

That is the correct way, as explained in the documentation http://docs.qgis.org/2.14/es/docs/user_manual/processing/console.html

the next code work with in memory all except the last that it is load

MDT=path/mdt.tif
drain=processing.runalg("grass:r.drain",MDT,"",(pun),False,False,False,"%f,%f,%f,%f"% (xmin, xmax, ymin, ymax),0,-1,0.00100,None)
vect=processing.runalg("grass:r.to.vect",drain['output'],0,False,"%f,%f,%f,%f"% (xmin, xmax, ymin, ymax),0,None)
bu=processing.runalg("qgis:fixeddistancebuffer",vect['output'],Metros_afecta,1,False,None)
buf=bu['OUTPUT']
bufe= QgsVectorLayer(buf,"area", "ogr")
#the last load the layer 
QgsMapLayerRegistry.instance().addMapLayers([bufe])

the processing.runalg return a dictionary in this case bu['OUTPUT'] OUTPUT IS THE KEY, and the value is a temp path you can see the key with processeing.alghelp("name processing") as processing,alghelp("grass:r.drain")

return

processing.alghelp("grass:r.drain")
ALGORITHM: r.drain - Traces a flow through an elevation model on a raster map.
input <ParameterRaster>
coordinate <ParameterString>
vector_points <ParameterMultipleInput>
-c <ParameterBoolean>
-a <ParameterBoolean>
-n <ParameterBoolean>
GRASS_REGION_PARAMETER <ParameterExtent>
GRASS_REGION_CELLSIZE_PARAMETER <ParameterNumber>
GRASS_SNAP_TOLERANCE_PARAMETER <ParameterNumber>
GRASS_MIN_AREA_PARAMETER <ParameterNumber>
output <OutputRaster>

in this case the key is output , take care with capital letter you must writte in capital or with out capital, in this case not capital

• bugmenot123: Memory layers are a special thing and I think you are talking about TEMPORARY_OUTPUT files here, so files written to a temporary directory as files. Memory layers do not exist as files.

Answer 3

And what about to look at GDAL Virtual File Systems (http://www.gdal.org/gdal_virtual_file_systems.html)

There is vsimem prefix for in-memory file-like paths...