What approach would I be using for multiple DLLs using the same data but processing it with different classes

Question

**** EDIT: See below for "working" code based on the discussion.

I'm re-writing my software from scratch and instead of hardcoding the processing libraries I decided to take a look at dynamic interfaces. But even the examples I found were not explicit in what I wanted to do. I'd like to write the management thread processes once then load each of the libraries on a thread dynamically using the same method calls in each of the libraries. That way I wouldn't have to duplicate code. I was thinking of using Generic <T> but that doesn't appear to be compatible with dynamic interfaces.

How it's used...

I have a set of data that I pull from a DB. I need to push that data to different systems using different processes, but the same method names.

So I would have common methods between methods, two of which would look like this:

InitializeConnector();
ProcessData(MyDataType incomingdata);

So in practical usage (prototyped):

REST Service 1 data processing

public class RestService1
{
 public bool InitializeConnector()
 {
 do stuff here...
 }
 public bool ProcessData( MyDataType incomingDataType)
 {
 ... process data here
 }
 }

REST Service 2 data processing (not all connection services are REST but this is the example)

public class RestService2
{
 public bool InitializeConnector()
 {
 do stuff here...
 }
 public bool ProcessData( MyDataType incomingDataType)
 {
 ... process data here
 }
 }

Master thread class:

foreach (data in dataList)
{
 if (data.type == dataType.rest1)
 dynamicClass.ProcessData(data);
 ....
 // or better yet something like this
 dataType.rest1.ProcessData(data);
 ... 
}

Am I better off hard coding the implementation or is there a dynamic implementation that can be used?

I will be spinning up multiple threads to process the data. So each of these processes would be running on its own thread, and I'll have a master thread to "schedule" the processing in each of the threads.

The whole idea is to drop in new processing DLLs to link to other systems, but all using the same calls and the same.

*********************** EDIT / UPDATE: *************************

In working with Doc Brown's comments (thank you for your patience BTW), I came up with the following working code with caveats I'll highlight after the code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
//using Test.Library1;
//using Test.Library2;
namespace CustomInterfaceTest
{
interface CustomInterface<T>
{
 T GetQuery(string value);
}
public class StartHere : IDisposable
{
 private List<CustomInterface<string>> interfaceList = new List<CustomInterface<string>>();
 private bool disposedValue;
 public void Runme()
 {
 var lib1 = new TestLibrary1();
 var lib2 = new TestLibrary2();
 interfaceList.Add(lib1);
 interfaceList.Add(lib2);
 int nCount = 0;
 foreach (CustomInterface<string> interf in interfaceList)
 { 
 nCount++;
 Console.WriteLine( interf.GetQuery("Test in loop " + nCount));
 }
 }
 protected virtual void Dispose(bool disposing)
 {
 if (!disposedValue)
 {
 if (disposing)
 {
 // TODO: dispose managed state (managed objects)
 }
 // TODO: free unmanaged resources (unmanaged objects) and override finalizer
 // TODO: set large fields to null
 disposedValue = true;
 }
 }
 // // TODO: override finalizer only if 'Dispose(bool disposing)' has code to free unmanaged resources
 // ~StartHere()
 // {
 // // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
 // Dispose(disposing: false);
 // }
 public void Dispose()
 {
 // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
 Dispose(disposing: true);
 GC.SuppressFinalize(this);
 }
}
public class TestLibrary1 : IDisposable, CustomInterface<string>
{
 private bool disposedValue;
 string CustomInterface<string>.GetQuery(string value)
 {
 return "library 1 " + value;
 }
 protected virtual void Dispose(bool disposing)
 {
 if (!disposedValue)
 {
 if (disposing)
 {
 // TODO: dispose managed state (managed objects)
 }
 // TODO: free unmanaged resources (unmanaged objects) and override finalizer
 // TODO: set large fields to null
 disposedValue = true;
 }
 }
 // // TODO: override finalizer only if 'Dispose(bool disposing)' has code to free unmanaged resources
 // ~Library1()
 // {
 // // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
 // Dispose(disposing: false);
 // }
 public void Dispose()
 {
 // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
 Dispose(disposing: true);
 GC.SuppressFinalize(this);
 }
}
public class TestLibrary2 : IDisposable, CustomInterface<string>
{
 private bool disposedValue;
 string CustomInterface<string>.GetQuery(string value)
 {
 return "library 2 " + value;
 }
 protected virtual void Dispose(bool disposing)
 {
 if (!disposedValue)
 {
 if (disposing)
 {
 // TODO: dispose managed state (managed objects)
 }
 // TODO: free unmanaged resources (unmanaged objects) and override finalizer
 // TODO: set large fields to null
 disposedValue = true;
 }
 }
 // // TODO: override finalizer only if 'Dispose(bool disposing)' has code to free unmanaged resources
 // ~Library2()
 // {
 // // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
 // Dispose(disposing: false);
 // }
 public void Dispose()
 {
 // Do not change this code. Put cleanup code in 'Dispose(bool disposing)' method
 Dispose(disposing: true);
 GC.SuppressFinalize(this);
 }
}
}

The above provides the expected results where I can store the different objects derived off the master interface. However there is an issue: This is me knowing the DLL's (library classes currently) to add to instantiate the objects. What process is used to dynamically pull the classes?

• Filip Milovanović: "using the same method calls in each of the libraries. That way I wouldn't have to duplicate code." - suppose you didn't have to have separate DLLs; do you know how to achieve this in that case (no duplication of the master code, same method calls)?
• Gratzy: I have this hardcoded in my current application. But each of the scheduling, etc... is duplicate code and I'd like to move this to a single scheduler, not hardcoded to fixed libraries. And yes, they would have to be separate DLLs, they don't cross paths in processing data; there's very little similarity between them other than providing the root data to be processed.
• Filip Milovanović: I understand that they should be separate DLLs, that's OK; but I'm trying to assess how in-depth the answer should be. What I'm asking is, if all the code was in the same project, would you know how to create a shared interface for all the libraries, change your master scheduler to work through this interface, and have each of your processing components implement this interface? Because, that achieves what you want, and, once you do that, separating things into DLLs isn't hard.
• Gratzy: I have a thread scheduler for each of the data processing types\ right now. In starting to rebuild this, I built out a newer hybrid processor that uses threads and async methods to spin up tasks to manage the individual datastreams. In looking at the new scheduler closer, if I were to have a dynamic set of classes, I could easily pass the same data off to all the different streams to be processed. But I've never done that type of interface before so I'm not sure what to even search on to figure out.
• Doc Brown: Zonus, from what you wrote, I fail to see where the actual problem is. Can't you simply derive your services from a common interface with methods InitializeConnector and ProcessData? And which part of the code precisely is repeated, and where?

Answer 1

Let's say there is a folder, path, where you expect all these dynamically-loaded DLLs to be deployed. First, you need to get all the files:

var files = Directory.GetFiles(path, "*.dll");

Then from the files you need to load assemblies:

var assemblies = files.Select( file => Assembly.LoadFrom(file) );

Then from the assemblies you get the types:

var types = assemblies.Select( assembly => assembly.GetTypes() );

Then from the types you select the ones that have the interface:

var filteredTypes = types.Where( type => typeof(CustomInterface<string>).IsAssignableFrom(type) );

or possibly:

var filteredTypes = types.Where( type => type.GetGenericTypeDefinition() == typeof(CustomInterface<>) );

Then if desired instantiate them, and store them in a list:

var processors = filteredTypes.Select( type => Activator.CreateInstance(type) ).ToList();

• Gratzy: That's it?? If so, I was really making this to be a lot more difficult than I thought... :O
• John Wu: If I understand your requirements correctly (not sure), then yes, that is all there is to it... this is a common pluggable architecture.
• Gratzy: I'll work on this tomorrow. Thanks!
• Doc Brown: +1, well explained. However, I think there is one thing to be mentioned here: CustomInterface must be part of a separate DLL which must be the same for each of the plugin DLLs as well as for the processing code. This usually leads to the requirement of DLL / interface versioning.
• Gratzy: Thanks people! This helped a LOT. Sometimes when you don't know the process, I tend to make things more complex than they should be. This is going to save me a LOT of code, a LOT of time, and make it possible to add in additional libraries later without having to touch the main code.