Teams Bot Integration Sample #185

Have you taken a look at the Python Interop Walkthrough? Perhaps you could begin with and build on that. Otherwise, could you give more specifics on problems you're hitting?

Yep, so with that walkthrough, it's using a MediaCapture object to get frames from a webcam.
var webcamFrames = mediaCapture.Out.Resize(640, 480);
which eventually is sampled to get byte arrays to post to NetMQ.
...

// create down-sized, down-sampled, sparse, compressed, stream of frames
var webcamBytes =
 webcamFrames
 .Sample(TimeSpan.FromMilliseconds(200))
 .EncodeJpeg(90, DeliveryPolicy.LatestMessage)
 .Select(jpg => jpg.Resource.GetBuffer());
webcamBytes.Write("WebcamBytes", store);

...

// send frames over ZeroMQ
var frameWriter = new NetMQWriter<byte[]>(
 pipeline,
 "frames",
 "tcp://127.0.0.1:30000",
 MessagePackFormat.Instance);
webcamBytes.PipeTo(frameWriter);

In the teams sample, video frames are aggregated like so:

var video = this.videoInConnector.Aggregate(
 new Dictionary<string, Shared<PsiImage>>(),
 (aggregate, frames) =>
 {
 // aggregate dictionary of participant ID -> video frame
 foreach (var frame in frames)
 {
 if (aggregate.TryGetValue(frame.Key, out Shared<PsiImage> old))
 {
 old.Dispose();
 aggregate.Remove(frame.Key);
 }
 aggregate.Add(frame.Key, frame.Value.Item1.AddRef());
 }
 return aggregate;
 });

But I'm at a loss as to how to get this into the write shape to pipe bytes over to ZeroMQ.

I see what you're running up against. The video is a stream of Dictionary<string, Shared<Image>>. It turns out that this cannot be serialized by MessagePack. Let's try simplifying and reducing the type, both to save bytes over the wire and to give MessagePack a better shot at serializing. For example:

var encoder = new ImageToJpegStreamEncoder() { QualityLevel = 90 };
var simpleVideo = video.Select(dict =>
{
 return dict.Select(kv =>
 {
 var img = kv.Value.Resource.Encode(encoder).GetBuffer();
 return (kv.Key, img);
 }).ToList();
});

This converts video to a stream of List<(string, byte[])>. Note that the first Select is a \psi operator, while the second inner one is a LINQ operator over the Dictionary. Encoding the image saves space, but also Shared<Image> was the type that MessagePack couldn't handle (because of pointers within). Also notice the .ToList() to convert the lazy IEnumerable, which holds unserializable closures, to a concrete, reified collection.

The simple List of (string, byte[]) tuples should now be no problem.

var videoWriter = new NetMQWriter<List<(string, byte[])>>(
 pipeline,
 "frames",
 "tcp://127.0.0.1:30000",
 MessagePackFormat.Instance); ;
simpleVideo.PipeTo(videoWriter);

It unpacks on the Python side as a list of lists:

input = zmq.Context().socket(zmq.SUB)
input.setsockopt_string(zmq.SUBSCRIBE, u"frames")
input.connect("tcp://127.0.0.1:30000")
[topic, payload] = input.recv_multipart()
message = msgpack.unpackb(payload, raw=True)
for x in message[b"message"]:
 key = x[0]
 frame = x[1]
 # do whatever with them

I hope this helps!

Thank you, AshleyF (@AshleyF) , I couldn't get Python to work on windows - did on bash. Had to install the MediaPack on the win11 OS.
Now, with Python up, I'm getting this exception:

System.InvalidOperationException
Message=Receiver cannot subscribe to an emitter from a different pipeline. Use a Connector if you need to connect emitters and receivers from different pipelines.

(Stacktrace omitted).

This error (cross-pipeline subscription) indicates that an emitter in one pipeline is being piped to a receiver in another pipeline. \psi restricts this from happening accidentally but does allow such a structure by way of a Connector between. A common cause is when creating composite components as subpipelines and accidentally exposing inner component emitters/receivers directly on the outer component. If you could share your code or a description of the component structure, I'd be happy to take a look.

Also, here's a bit on the wiki talking about composite components and connectors: https://github.com/microsoft/psi/wiki/Writing-Components#4-composite-components

Hi Rich, we haven't tried deploying the bot in an Azure VM ourselves, so I don't have too much insight I can provide, unfortunately.

However, I am aware of a project in which someone else used the Teams bot integration sample to build their own psi bot for Teams, and they seem to have figured out how to deploy to the cloud. Maybe this doc will give you some ideas?
https://github.com/AI4Bharat/INCLUDE-MS-Teams-Integration/blob/main/docs/deploy.md

Hi there,
When I was exploring how to use PSI with this sample, it was all done via a VM in Azure. In my proof of concept using this, I was able to get it to render in a teams meeting - after some trials, here: https://www.youtube.com/watch?v=wjQmDisBg5U&t=9491s
I did use things like ngrok to tunnel into the VM, and it worked fine.
As it turns out, my unfamiliarity with some of the streams in PSI led to some confusion (as you can see in the above call), but that got eventually sorted out.
I don't think you should have too many problems in PoC context. And as long as you get more experience with how everything fits together, you should find alternative approaches to getting things deployed.