Design guideline for saving big byte stream in c#

Question

I have an application where I am receiving big byte array very fast around per 50 miliseconds.

The byte array contains some information like file name etc. The data (byte array ) may come from several sources.

Each time I receive the data, I have to find the file name and save the data to that file name.

I need some guide lines to how should I design it so that it works efficient.

Following is my code...

public class DataSaver
{
 private Dictionary<string, FileStream> _dictFileStream;
 public void SaveData(byte[] byteArray)
 {
 string fileName = GetFileNameFromArray(byteArray);
 FileStream fs = GetFileStream(fileName);
 fs.Write(byteArray, 0, byteArray.Length);
 }
 private FileStream GetFileStream(string fileName)
 {
 FileStream fs;
 bool hasStream = _dictFileStream.TryGetValue(fileName, out fs);
 if (!hasStream)
 {
 fs = new FileStream(fileName, FileMode.Append);
 _dictFileStream.Add(fileName, fs);
 }
 return fs;
 }
 public void CloseSaver()
 {
 foreach (var key in _dictFileStream.Keys)
 {
 _dictFileStream[key].Close();
 }
 }
}

How can I improve this code ? I need to create a thread maybe to do the saving.

• CloneXpert: Consider using [this][1] approach, I believe you have a similar situation. [1]: stackoverflow.com/questions/6203836/…

Answer 1

You need to ask yourself 4 questions to start with:

1. How "big" is "big byte array"?
2. How fast is "very fast" ? Can we take 20 Hz as a design assumption?
3. Should data be appended in-order?
4. What is the expected overall size for a given file?

Just for the taste of it, appending 4MB to a file, on my 10 YO machine, takes less than 10 ms. A modern SSD and CPU will outdo this by an order of magnitude.

Anyway, knowing nothing, I would start with the most simple imaginable scheme, which is completely synchronous, and let it fail:

public static class FileSaver
{
 public static bool TrySaveData(byte[] data)
 {
 int dataOffset = 0;
 var file = GetFileName(data, out dataOffset);
 return TryAppend(file, data, dataOffset);
 }
 private static string GetFileName(byte[] data, out int offset)
 {
 string res = ... // get the file name
 offset = ... // depends on your format 
 return res;
 }
 private static bool TryAppend(string file, byte[] data, int offset = 0)
 {
 try
 {
 using (var stream = new FileStream(file, FileMode.Append))
 {
 stream.Write(data, offset, data.Length - offset);
 }
 return true;
 }
 catch (Exception ex) { /* log some error */ }
 return false;
 }
}

Why is this good? Because by the time it fails, you'll be smart enough to answer 1-4, and then we would design something else based on real understanding of the problem.

Answer 2

Careful about how many files you keep open, OS handles are not free. See http://msdn.microsoft.com/en-us/library/yz8tx0w3.aspx