Write large byte arrays on RAID0 virtual disk

Question

I have a camera that produces 20 frames per second. I made an app in C# that grabs the frames and put them in a concurrent queue. I have another thread that writes the frames on filesystem. Each frame is an array of 134184960 bytes that I must write on filesystem. The filesystem is an array of 20 SSD in RAID0 with theoretical speed of 500 MB/s * 20 with the bottleneck of PCiExpress 8x (7,7 GB/s) The production rate is about 2,5 GB/s, but my writing speed doesn't go over 900 MB/s I am using Filestream.Write (link)

What can I do to increase my writing speed?

private ConcurrentQueue<CapturedFrame> cq = new ConcurrentQueue<CapturedFrame>();
....
try
{
 while (cq.TryDequeue(out BufferToWrite))
 {
 byte[] bufferInBytes = new byte[134184960];
 //Matrox function to extract bytes
 MIL.MbufGetColor(BufferToWrite.IDFrame, MIL.M_PLANAR, MIL.M_ALL_BANDS, bufferInBytes);
 
 using (FileStream fileStream = new FileStream(Path.Combine(AcquisitionFolder, DateTime.Now.ToString("yyyy-MM-dd-HHmmss-fffffff") + ".raw"), FileMode.Create))
 {
 fileStream.Write(bufferInBytes, 0, bufferInBytes.Length);
 fileStream.Close();
 }
 } 
}
catch (Exception ex)
{
 log.Error("Exception: " + ex.Message);
}

changed the FileStream to this with no improvement 65536 = 64K is the stripe size of the RAID0 array, I've set the lenght of the stream

using (FileStream fileStream = new FileStream(Path.Combine(AcquisitionFolder, DateTime.Now.ToString("yyyy-MM-dd-HHmmss-fffffff") + ".raw"), FileMode.Create, FileAccess.Write, FileShare.Read, 65536, FileOptions.SequentialScan))
 {
 fileStream.SetLength(3 * 8192 * 5460);
 fileStream.Write(bufferInBytes, 0, bufferInBytes.Length);
 fileStream.Close();
 }

• Matthew Watson: .net 6 does support WriteFileGather() which allows you to supply several buffers at once to write - that might provide a small speed improvement, but I think it's very unlikely to make much difference.
• Danilo7: I've found this paper (Sequential File Programming Patterns and Performance with .NET microsoft.com/en-us/download/details.aspx?id=52334) online with samples of code attached, but still cannot find a way to speed up my writing
• CodeCaster: Is your code running on the machine that hosts those disks. or are you connected through a gigabit network? Have you determined it's C#/.NET that's limiting the throughput? Have you considered playing with buffer sizes of the stream? And so on.
• Danilo7: 1) My code is running on the machine that hosts the RAID0 array of disks 2) I am not sure if the problem is C# or else. I chose the RAID0 configuration expecting a much higher write speed. How can I determine if C# is the limiting factor? 3) I've already tried many Filestream buffers sizes with no big improvements
• Danilo7: I've used a tool to test disk speed (roylongbottom.org.uk/diskgraf%20results.htm) and it gave me a speed of around 4 GB/s imgur.com/a/zBNFpil

Answer 1

I was able to reach the 2,5 GB/s write speed with two steps:

• accessing the frames directly in memory, using byte* pointers
• writing the frames using Span structure and FileStream "unbuffered" as described in the paper i linked before
  https://www.microsoft.com/en-us/download/details.aspx?id=52334