-
Notifications
You must be signed in to change notification settings - Fork 1.6k
#857 Support multipart files using InputStream from source file #1593
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
zbum
commented
Oct 26, 2018
👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Could you please address comments?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small nit: could you please add a constructir without contenLength that sets the value as -1? There's a chance most users won't know the stream length beforehand.
client/src/test/java/org/asynchttpclient/request/body/multipart/MultipartUploadTest.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please test with and without Content-Length prior knowledge?
- Added tests for all combinations of known/unknown contentLength and enabled/disabled zeroCopy for `InputStreamPart` - Fixed how `length()` is computed for `MultipartPart` - returns -1 if contentLength < 0 - Added condition in `NettyBodyBody` to use `BodyChunkedBody` for "zeroCopy" of `RandomAccessBody` if contentLength is -1
I ran into some trouble with the Netty HttpObjectEncoder class when I tried to get zero-copy with -1 contentLength to work. Here's what happens:
- In
org/asynchttpclient/netty/request/body/NettyBodyBody.java#write(final Channel channel, NettyResponseFuture<?> future)we setmsg = new BodyFileRegion((RandomAccessBody) body); - In
io/netty/netty-codec-http/4.1.30.Final/netty-codec-http-4.1.30.Final.jar!/io/netty/handler/codec/http/HttpObjectEncoder.class#encode(ChannelHandlerContext ctx, Object msg, List<Object> out)this causes us to hit the following block
case 2: if (buf != null) { out.add(buf); } this.encodeChunkedContent(ctx, msg, contentLength(msg), out); break;
- Taking a look at the
contentLength(msg)method, this returns -1 whenmsgmatchesFileRegion
private static long contentLength(Object msg) { if (msg instanceof HttpContent) { return (long)((HttpContent)msg).content().readableBytes(); } else if (msg instanceof ByteBuf) { return (long)((ByteBuf)msg).readableBytes(); } else if (msg instanceof FileRegion) { return ((FileRegion)msg).count(); } else { throw new IllegalStateException("unexpected message type: " + StringUtil.simpleClassName(msg)); } }
- In
encodeChunkedContent(ChannelHandlerContext ctx, Object msg, long contentLength, List<Object> out):
private void encodeChunkedContent(ChannelHandlerContext ctx, Object msg, long contentLength, List<Object> out) { ByteBuf buf; if (contentLength > 0L) { String lengthHex = Long.toHexString(contentLength); buf = ctx.alloc().buffer(lengthHex.length() + 2); buf.writeCharSequence(lengthHex, CharsetUtil.US_ASCII); ByteBufUtil.writeShortBE(buf, 3338); out.add(buf); out.add(encodeAndRetain(msg)); out.add(CRLF_BUF.duplicate()); } if (msg instanceof LastHttpContent) { HttpHeaders headers = ((LastHttpContent)msg).trailingHeaders(); if (headers.isEmpty()) { out.add(ZERO_CRLF_CRLF_BUF.duplicate()); } else { buf = ctx.alloc().buffer((int)this.trailersEncodedSizeAccumulator); ByteBufUtil.writeMediumBE(buf, 3149066); this.encodeHeaders(headers, buf); ByteBufUtil.writeShortBE(buf, 3338); this.trailersEncodedSizeAccumulator = 0.2F * (float)padSizeForAccumulation(buf.readableBytes()) + 0.8F * this.trailersEncodedSizeAccumulator; out.add(buf); } } else if (contentLength == 0L) { out.add(encodeAndRetain(msg)); } }
- We basically miss all the blocks since the
encodeChunkedContentmethod doesn't handlecontentLength < 0L.
@slandelle : Not sure if my solution is acceptable, I'd be open to better ideas.
samridh90 I don't think it's worth trying to contribute chunked transfert encoding support for Netty's FileRegion. We can just disable bypass zero-copy when content-length < 0.
InputStream isn't zero-copy anyway, such it would only be suboptimal when sending a multipart that mixes a local file and an InputStream.
Then, don't bother implementing zero-copy for InputStreamPart as it shouldn't be used.
WDYT?
@slandelle : Sounds good to me. I'll get rid of the zero-copy for InputStreamPart.
Edit: Done, latest update removes zero-copy for InputStreamPart and adds tests that assert that it should not be used.
- Fix tests, add additional tests to cover various combinations
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was some misunderstanding. In my previous comment, I pointed to this line. If you check the content-length here, you can force to the BodyChunkedInput branch if it's < 0. Hence, There's no need to advice the user to disable zero-copy globally, it can be completely transparent.
README.md
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please revert.
There was some misunderstanding. Please check my comment above: #1593 (comment).
There's no need to ask the user to explicitly disable zero-copy globally.
You can just check the content-length and go with BodyChunkedInput if it's < 0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for clearing that up.
I've reverted the changes and added a check to BodyChunkedInput if contentLength 0
client/src/main/java/org/asynchttpclient/request/body/multipart/InputStreamPart.java
Show resolved
Hide resolved
.../src/main/java/org/asynchttpclient/request/body/multipart/part/InputStreamMultipartPart.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please fix header: Sonatype is no longer involved in this project and new code should be donated to AHC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
One last nit and we'll be good :)
Thanks!
Merged. Great job, thanks!
@slandelle: Happy to contribute! I'm not familiar with AHC's release cadence, when do you expect to include this feature in a release?
2.6.0 should be available on maven central in 15 min.
Motivation:
InputStreamto a file instead of theFileobject itself for a multipart file part.Changes:
InputStreamParttype for use in theaddBodyPart(...)APIInputStreamMultipartPartto transferInputStreamto a NettyByteBuforWritableByteChannelprotected long transferContentTo(WritableByteChannel target) throws IOExceptionfor zero-copy uploads.FilePartResults:
InputStreamas a multipart body partRelated Issue:
#857