homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Allow objects implemented in pure Python to export PEP 3118 buffers
Type: Stage:
Components: Interpreter Core Versions: Python 3.6
process
Status: open Resolution:
Dependencies: 10181 Superseder:
Assigned To: Nosy List: Arfrever, Jacob.Holm, alex, eric.snow, kermode, mark.dickinson, martin.panter, ncoghlan, pitrou, rhettinger, sbt, scoder, siemer
Priority: normal Keywords:

Created on 2012年01月16日 10:29 by ncoghlan, last changed 2022年04月11日 14:57 by admin.

Files
File name Uploaded Description Edit
llindstrom-newbuffer-0dd8ba1c2c2c.tar.gz kermode, 2013年08月16日 22:53 Extension module containing BufferMixin and Py_buffer. Source.
Messages (11)
msg151347 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012年01月16日 10:29
Currently, there's no straightforward way to create new classes in Python that delegate the PEP 3118 buffer APIs to another object that supports them (e.g. bytes, bytearray, memoryview, array.array, a NumPy array).
I see a few possible ways to go about this:
1. Add a single new special method, __buffer__. This creates an entry in the underlying getbuffer slot, but leaves releasebuffer empty.
The getbuffer wrapper then calls __buffer__ and delegates the getbuffer call to the returned object.
Any created Py_buffer instances refer directly to the underlying PEP 3118 supporting object, not the original object that defines __buffer__
The downside of this approach is that you can't implement a completely new PEP 3118 supporting object in pure Python (since you don't get access to the flags)
2. As above, but any Py_buffer structs returned are updated to refer to the original object, not the underlying one. A releasebuffer wrapper is also created, that calls back into __buffer__ to retrieve the object again. This approach has problems if __buffer__ can change to refer to different objects over time.
3. Define separate __getbuffer__ and __releasebuffer__ special methods. All C types implementing the PEP 3118 slots get these special methods automatically (as with any other slot). The flag variables are passed into __getbuffer__, and it is expected to return a memoryview object defining the view. The underlying getbuffer slot wrapper stores the current object in the Py_buffer "obj" slot, and the returned memoryview in "internal". The rest of the Py_buffer fields are populated based on the memoryview contents.
The releasebuffer wrapper then reverses this process, by passing the memoryview instance from the internal slot into the __releasebuffer__ call.
msg151351 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012年01月16日 11:47
Reviewing Stefan's work on #10181 suggests to me that option 3 is the only feasible approach.
(Allowing memoryview subclasses will permit types to pass their own state between __getbuffer__ and __releasebuffer__)
msg154683 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012年03月01日 09:33
I'm trying to understand what you want to be able to write. Do you
perhaps have a short example? Also, to get the bigger picture: Is
this related to your strview proposal?
http://mail.python.org/pipermail/python-ideas/2011-December/012993.html 
msg154684 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012年03月01日 09:48
Consider a Python wrapper around a bytes object, or mmap or similar that
wants to pass PEP 3118 buffer requests through to the underlying object.
Currently, there's no way to write such a delegation - the delegating class
has to be written in C.
msg154692 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012年03月01日 13:19
To answer your other question, no, strview isn't related - that's strictly a PEP 3118 *consumer*, which is well supported from the Python side now that memoryview is fixed.
The trick will be to allow a Python implemented object to be a PEP 3118 exporter *without* having to inherit from a C implemented type that does the slot mapping. Since PEP 3118 didn't describe a Python level API for the protocol, it may actually require a new PEP.
One example for what you could do with it: use the new memoryview.cast() to provide multidimensional views on an exporter that only supports 1D exports natively.
msg154855 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2012年03月03日 21:24
FWIW, Cython lets user code implement the buffer interface for extension types using the special methods "__getbuffer__()" and "__releasebuffer__()", so providing the same methods (although with a different signature) also for normal Python types would IMHO fit nicely.
msg156440 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2012年03月20日 17:42
FWIW pypy has an __buffer__ method (used exclusively internally, AFAIK), which has semantics similar to your first proposal.
msg156479 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2012年03月21日 06:59
Ok, just for the record: a single __buffer__() special method with delegation-only semantics would also work for Cython. Taking this path would provide a cleaner separation of the (then delegation-only) Python level protocol and the (all purpose) C level protocol. I think I'd prefer that.
I assume __buffer__() takes no arguments, right?
msg156485 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012年03月21日 10:16
The reason I don't particularly like the "delegation only" API is that the combination of the new memoryview implementation and bytes/mmap/etc to get a flat region of memory to play with means you could do some quite interesting things entirely at the Python level.
If you couldn't even use memoryview.cast() to change the shape of the exported memory, it makes the Python level API clearly inferior in power compared to what you can do in C.
msg163058 - (view) Author: Alyssa Coghlan (ncoghlan) * (Python committer) Date: 2012年06月17日 11:43
I suggest a PEP for 3.4 as the best way forward for this.
msg195436 - (view) Author: Lenard Lindstrom (kermode) Date: 2013年08月16日 22:53
A fourth way to add __getbuffer__ and __releasebuffer__ special methods to a Python class is through a new base class/mixin. The Py_buffer struct pointer passed to __getbuffer__ and __releasebuffer__ is wrapped with another special object type, which exposes the C struct fields as attributes. The base class is a direct subclass of object. Both the base and wrapper classes are implemented in an extension module. 
The use case is unit testing. A memoryview object may be adequate for simple C-contiguous arrays, but fails with F-contiguous or any non-contiguous array. It cannot export any arbitrary view, especially a deliberately invalid view, useful for testing error handlers.
My implementation is at https://bitbucket.org/llindstrom/newbuffer . It is used in Pygame 1.9.2 unit tests: https://bitbucket.org/pygame/pygame . The newbuffer module exports two classes, BufferMixin and Py_buffer. The BufferMixin class adds bf_getbuffer and bf_releasebuffer slot functions to base class PyObject, nothing more. The Py_buffer class is low level, exposing pointer fields as integer addresses. It is designed for use with ctypes and is modelled on ctypes.Structure.
Some limitations. In a class declaration, BufferMixin must be first in the inheritance list for the subclass to inherit the PEP 3118 interface. A buffer interface cannot be added to a builtin.
I suggest this is a practical alternative to the three previously proposed solutions to this issue. This option takes its precedence from the ctypes module, which adds dangerous memory level access to Python, but optionally. It does not require modification of the base code, only an addition to the standard library. Finally, this approach has been demonstrated in a real-world application.
History
Date User Action Args
2022年04月11日 14:57:25adminsetgithub: 58006
2020年07月15日 17:38:42skrahlinkissue41285 superseder
2016年03月23日 04:03:54siemersetnosy: + siemer
2015年05月16日 08:28:58ncoghlansetversions: + Python 3.6, - Python 3.5
2014年10月14日 16:38:56skrahsetnosy: - skrah
2013年12月04日 10:58:28martin.pantersetnosy: + martin.panter
2013年12月01日 05:02:26ncoghlansetversions: + Python 3.5, - Python 3.4
2013年08月16日 22:53:36kermodesetfiles: + llindstrom-newbuffer-0dd8ba1c2c2c.tar.gz
nosy: + kermode
messages: + msg195436

2013年06月16日 00:09:20Jacob.Holmsetnosy: + Jacob.Holm
2012年07月07日 22:28:58Arfreversetnosy: + Arfrever
2012年06月17日 11:43:28ncoghlansetmessages: + msg163058
versions: + Python 3.4, - Python 3.3
2012年05月27日 16:26:26sbtsetnosy: + sbt
2012年03月21日 10:16:02ncoghlansetmessages: + msg156485
2012年03月21日 08:00:58mark.dickinsonsetnosy: + mark.dickinson
2012年03月21日 06:59:12scodersetmessages: + msg156479
2012年03月20日 17:42:40alexsetnosy: + alex
messages: + msg156440
2012年03月03日 21:24:25scodersetnosy: + scoder
messages: + msg154855
2012年03月02日 03:33:36eric.snowsetnosy: + eric.snow
2012年03月01日 13:19:37ncoghlansetmessages: + msg154692
2012年03月01日 09:48:56ncoghlansetmessages: + msg154684
2012年03月01日 09:33:13skrahsetmessages: + msg154683
2012年01月23日 10:40:57rhettingersetnosy: + rhettinger
2012年01月16日 11:47:46ncoghlansetmessages: + msg151351
2012年01月16日 10:30:14ncoghlansettitle: Add a Python level special method to retrieve a PEP 3118 object -> Allow objects implemented in pure Python to export PEP 3118 buffers
2012年01月16日 10:29:30ncoghlansetdependencies: + Problems with Py_buffer management in memoryobject.c (and elsewhere?)
title: Add a __buffer__ special method to retrieve a PEP 3118 object -> Add a Python level special method to retrieve a PEP 3118 object
2012年01月16日 10:29:10ncoghlancreate

AltStyle によって変換されたページ (->オリジナル) /