-
Notifications
You must be signed in to change notification settings - Fork 330
Define a custom buffer size to streaming data to SQL Server #3788
-
Hey, i have a question: Is there a way to define a custom buffer size to streaming data to SQL Server?
The code below uses a Stream to insert a large file into column called Data. The file size is 800MB .
const string Statement = """ INSERT INTO [dbo].[Stream] (Id, Data) VALUES (@Id, @Data); """; using var input = File.OpenRead("/tmp/dfs/data/1/sample.txt"); using var connection = new SqlConnection(ConnectionString); connection.Open(); using var cmd = new SqlCommand(Statement, connection); cmd.Parameters.AddWithValue("@Data", input); cmd.Parameters.AddWithValue("@Id", 1); cmd.ExecuteNonQuery();
I noticed when i run, the chunk size used to streaming is 4096 and cannot find a way to set a different size. Debugging the code i found the line where is create the buffer with a constant size:
Can you help me ?
Thanks
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 4 comments
-
@AugustoDeveloper There is no API that allows configuring this buffer size. You can refer to the streaming support documentation here.
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
Yeah, i already readed this doc... no much helpful for this case. Is there any chance in future to create some api for this?
@AugustoDeveloper There is no API that allows configuring this buffer size. You can refer to the streaming support documentation here.
Beta Was this translation helpful? Give feedback.
All reactions
-
Have you benchmarked it and seen that the buffer size is a bottleneck? If so can you share the bench setup?
I've have thought that the limit would be the network for this since it'll be sending to the server over the network. Unless the packet size is slightly larger than the buffer it'd complicate the packetization slightly but not enough to cause a performance problem.
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
Ok, I'll benchmark and bring the results.
Have you benchmarked it and seen that the buffer size is a bottleneck? If so can you share the bench setup?
I've have thought that the limit would be the network for this since it'll be sending to the server over the network. Unless the packet size is slightly larger than the buffer it'd complicate the packetization slightly but not enough to cause a performance problem.
Beta Was this translation helpful? Give feedback.