In Python, reading a GDAL raster from memory instead of a file

Question

I'd like to analyze some raster data in Python using NumPy.

This data doesn't necessarily exist as a file on the filesystem, it could be the result of a query against PostGIS, or something transferred over the network.

Is it possible to get GDAL to read data from a buffer in memory, instead of requiring a filename?

I realize one solution could be to open up a named temp file, write the data to that, and pass it as a filename to GDAL, but that seems inefficient, and more likely to incur I/O bottlenecks. Additionally, in the case of PostGIS, it would be nice to read the native format directly, as opposed to converting it to some other file format and reading that.

Edit: to clear up a bit of confusion, just using the PostGIS driver directly won't work either, since the query could be for a raster analysis, rather than just data stored in the DB.

Answer 1

If you have your own bytes, and need a GDAL Raster object, just use the MEM driver. No files required.

from osgeo import gdal
import numpy as np
driver = gdal.GetDriverByName('MEM')
src_ds = driver.Create('', 100, 200, 1)
band = src_ds.GetRasterBand(1)
# Create random data array to put into the raster object
ar = np.random.randint(0, 255, (200, 100))
band.WriteArray(ar)

Answer 2

GDAL has a Memory Driver and a PostGIS driver.

You can use the following to create a raster in memory:

# open postgis data
databaseServer = "<IP of database server OR Name of database server"
databaseName = "<Name of database>"
databaseUser = "<User name>"
databasePW = "<User password>"
connString = "PG: host=%s dbname=%s user=%s password=%s" %(databaseServer,databaseName,databaseUser,databasePW)
src_ds = gdal.Open(connString)
#create Memory driver
format = " MEM "
driver = gdal.GetDriverByName( format )
#copy data from PostGIS to Memory
driver.CreateCopy('', src_ds ) 

• YenTheFirst: How would you actually load existing data into this data set? i.e., I have the results of a query, or data copied into memory from a non-file source, and I need GDAL to parse it into something usable.
• ustroetz: In what kind of format is your data (i.e. array?)
• RyanKDalton: I'm not positive since I am just starting to use Python/GDAL drivers too, but it looks like you might use driver.CreateCopy() method. See How to apply "Band" settings using gdal Python bindings?
• YenTheFirst: my data could be an array of bytes, or a string buffer. the format of these bytes could be a GDAL raster format, or postgis's raster representation, or something similar. My problem is how to turn that into src_ds in the first place.
• ustroetz: Sorry, that's all I know (I edited my post). I never connected PostGIS with GDAL. But the documentation says it is possible.