In memory data access for CSV files that may have changed

Question

I'm reading data from CSV files and store the result in a simple DataContext instance that gets injected throughout my application.

public class DataContext
{
 private static readonly IDictionary<Type, IEnumerable<BaseEntity>> EntityMap = 
 Assembly.GetExecutingAssembly().GetTypes()
 .Where(t => t.IsClass && !t.IsAbstract && t.IsSubclassOf(typeof(BaseEntity)))
 .ToDictionary(t => t, t => Enumerable.Empty<BaseEntity>());
 public IEnumerable<T> Get<T>() where T : BaseEntity
 {
 return EntityMap[typeof(T)].Cast<T>();
 }
 public void Set<T>(IEnumerable<BaseEntity> value) where T : BaseEntity
 {
 EntityMap[typeof(T)] = value.ToList();
 }
}

I listen for changes in a folder. As soon as a CSV file gets updated, I'll update the corresponding list. Something like this:

void OnFileChange(string fileName)
{
 // handle in a separate thread to decrease the load of FileSystemWatcher
 Task.Run(() => {
 var parsedFileData = parser.Parse<DataType>(fileName);
 dataContext.Set<DataType>(parsedFileData);
 });
}

I never do a delta update of entities. It's always a complete replacement of the collection, which is by intention.

Some thoughts:

• What do you think of this in memory approach with variables vs something like Redis?
• Any other comments?

Answer 1

It's always a good idea to try to design LINQ compatible classes. Your DataContext can be designed this way.

• first you need to un-static-fy the field, it shouldn't be static (rarely anything has to be)
• implement the ILookup interface as it behaves exactly as your data-context, this is, return an empty collection if the key hasn't been found. Internally you can use a dictionary
• there is no need to scan the types and fill the dictionary with empty collections

Example:

public class DataContext : ILookup<Type, BaseEntity>
{
 private readonly IDictionary<Type, BaseEntityGroup> _items = new Dictionary<Type, BaseEntityGroup>();
 public bool Contains(Type type) => _items.ContainsKey(type);
 public int Count => _items.Count;
 public IEnumerable<BaseEntity> this[Type type]
 {
 get => _items.TryGetValue(type, out BaseEntityGroup result) ? result : Enumerable.Empty<BaseEntity>();
 set => _items.Add(type, new BaseEntityGroup(type, value));
 }
 public IEnumerator<IGrouping<Type, BaseEntity>> GetEnumerator() => null;
 IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

This requries another helper class for storing the items, so let's create one. It needs to implement the IGrouping interface:

public class BaseEntityGroup : List<BaseEntity>, IGrouping<Type, BaseEntity>
{
 public BaseEntityGroup(Type type, IEnumerable<BaseEntity> items) : base(items) => Key = type;
 public Type Key { get; }
}

Now you can work with it like this:

var dc = new DataContext();
dc[typeof(string)] = new BaseEntity[] 
{
 new BaseEntity2(),
 new BaseEntity2()
};

but you want to use some generics? So why not create a two extensions for it:

public static DataContext Add<T>(this DataContext context, IEnumerable<BaseEntity> items) where T : BaseEntity
{
 context[typeof(T)] = items;
 return context;
}
public static IEnumerable<T> Get<T>(this DataContext context) where T : BaseEntity
{
 return context[typeof(T)].Cast<T>();
}

Should you now need to query your data-context you can easily do it with LINQ.

Answer 2

1. If Set method is only called by the service that watches files and should not be used by consumers, then it should not be available to them as public method. The easiest way to achieve this is to inject interface instead of implementation:

   interface IReadonlyDataContext
   {
    IEnumerable<T> Get<T>() where T : BaseEntity;
    //no set method here, it is hidden from consumers
   }
   class DataContext : IReadonlyDataContext {...}
   

   Alternatively you can make Set method private and inject file watcher inside DataContext.

   class DataContext
   {
    class DataContext(IFileWatcher watcher)
    {
    watcher.FileChanged += OnFileChanged;
    }
    public IEnumerable<T> Get<T>() {...}
    private void OnFileChanged(...)
    {
    //call Set here
    }
    private void Set<T>(...) {...}
   }
   

2. Dictionary is not thread-safe and write operation always happen on new thread with zero synchronization. Looks like a recipe for run-time crashes to me.

• Johan: Good points. Are you suggesting a ConcurrentDictionary?
• Nikita B: @Johan, either that, or wrap those methods in lock statement.
• Johan: Allrighty. Regarding your first point, I don't really see how I could implement that. Could you show me an example?
• Nikita B: @Johan, I have updated my answer.