BTW, I hope you app server is in the same AWS data center as your S3 bucket. If those are on different GEO locations, the IO can become painfully slow.

Hi Joe

Thanks for the response! (btw: I'm the same person as the post above; just used the wrong login).

I played around further and found the main issue (I believe). The thing was that the save command of AWSS3Storage actually copies the file through the server making the call (I haven't checked the network but it took about the same amount of time like the upload itself). I fixed it for my case that it is actually using the boto copy command instead of the upload_fileobj (including a seek operation) command.

So in general the save method does what it's supposed to do and it wouldn't make a difference if the file would not be there already but since your library does directly upload it, I kind of expected a move command somewhere. But S3 doesn't even have such a command so I hoped copying a bucket does it in a way that doesn't require to stream the file content through my server and it seems to work like that.

So all in all I now use an own subclass for the storage like that and it works as expected.

from storages.backends.s3boto3 import S3Boto3Storage
class AWSS3Storage(S3Boto3Storage): # pylint: disable=abstract-method
 default_acl = 'private'
 file_overwrite = True
 custom_domain = False
 def _save(self, name, content):
 # Basically copy the implementation of _save of S3Boto3Storage
 # and replace the obj.upload_fileobj with a copy function
 cleaned_name = self._clean_name(name)
 name = self._normalize_name(cleaned_name)
 params = self._get_write_parameters(name, content)
 if (self.gzip and # pylint: disable=no-member
 params['ContentType'] in self.gzip_content_types and # pylint: disable=no-member
 'ContentEncoding' not in params):
 content = self._compress_content(content)
 params['ContentEncoding'] = 'gzip'
 obj = self.bucket.Object(name)
 #content.seek(0, os.SEEK_SET) # Disable unnecessary seek operation
 #obj.upload_fileobj(content, ExtraArgs=params) # Disable upload function
 # Copy the file instead uf uploading
 obj.copy({'Bucket': self.bucket.name, 'Key': content.obj.key}, ExtraArgs=params)
 return cleaned_name

Of course there's still the possibility that I missed something to configure right with your library but with the own storage implementation it works now for me and it makes sense that those commented out parts actually led to the issue I had. Even if I hadn't done anything else with the file that could cause a download of it (the code above really was all I have done to that point with it).

Regarding your PS: I can't control it too far but I think the same GEO should be doable. Regardless it should only be an issue when initially uploading the file and later basically only URL signing should be quick. There should be no "user facing IO" happening.

Thanks again for your answer, help and of course the neat library!

Best,
Christian

Hi @drakon, interesting. That's an excellent find, maybe we should include your solution in this library, what do you think? Would you be interested in providing a patch? Best, Joe

Hi Joe, yeah, sure, happy to try it. Honestly I'm not quite an experienced OS contributor but happy to try to do it. Just give me some time (I'm now on holidays) and patience when I put up the PR. :)

Sure, take your time. Consider this a great start to get into OSS. And, I've been doing this for a while, patience is what you need the most ;)