Edit - Stack Overflow

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Interprocess Communication between two python scripts without STDOUT

I am trying to create a Monitor script that monitors all the threads or a huge python script which has several loggers running, several thread running.

From Monitor.py i could run subprocess and forward the STDOUT which might contain my status of the threads.. but since several loggers are running i am seeing other logging in that..

Question: How can run the main script as a separate process and get custom messages, thread status without interfering with logging. ( passing PIPE as argument ? )

Main_Script.py * Runs Several Threads * Each Thread has separate Loggers.

Monitor.py * Spins up the Main_script.py * Monitors the each of the threads in MainScript.py ( may be obtain other messages from Main_script in the future)

So Far, I tried subprocess, process from Multiprocessing.

Subprocess lets me start the Main_script and forward the stdout back to monitor but I see the logging of threads coming in through the same STDOUT. I am using the "import logging " Library to log the data from each threads to separate files.

I tried "process" from Multiprocessing. I had to call the main function of the main_script.py as a process and send a PIPE argument to it from monitor.py. Now I can’t see the Main_script.py as a separate process when I run top command.

Answer*

Normally, you want to change the child process to work like a typical Unix userland tool: the logging and other side-band information goes to stderr (or to a file, or syslog, etc.), and only the actual output goes to stdout.

Then, the problem is easy: just capture `stdout` to a `PIPE` that you process, and either capture `stderr` to a different `PIPE`, or pass it through to real `stderr`.

---

If that's not appropriate for some reason, you need to come up with some other mechanism for IPC: Unix or Windows named pipes, anonymous pipes that you pass by leaking the file descriptor across the `fork`/`exec` and then pass the fd as an argument, Unix-domain sockets, TCP or UDP localhost sockets, a higher-level protocol like a web service on top of TCP sockets, `mmap`ped files, anonymous `mmap`s or pipes that you pass between processes via a Unix-domain socket or Windows API calls, …

As you can see, there are a huge number of options. Without knowing anything about your problem other than that you want "custom messages", it's impossible to tell you which one you want.

While we're at it: If you can rewrite your code around `multiprocessing` rather than `subprocess`, there are nice high-level abstractions built in to that module. For example, you can use a `Queue` that automatically manages synchronization and blocking, and also manages pickling/unpickling so you can just pass any (picklable) object rather than having to worry about serializing to text and parsing the text. Or you can create shared memory holding arrays of int32 objects, or NumPy arrays, or arbitrary structures that you define with `ctypes`. And so on. Of course you could build the same abstractions yourself, without needing to use `multiprocessing`, but it's a lot easier when they're there out of the box.

---

Finally, while your question is tagged `ipc` and `pipe`, and titled "Interprocess Communication", your description refers to _threads_, not processes. If you actually are using a bunch of threads in a single process, you don't need any of this.

You can just stick your results on a `queue.Queue`, or store them in a `list` or `deque` with a `Lock` around it, or pass in a callback to be called with each new result, or use a higher-level abstraction like `concurrent.futures.ThreadPoolExecutor` and return a `Future` object or an iterator of `Future`s, etc.

Draft saved

Draft discarded

Edit Summary*

Cancel

Add a comment |

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. python-3.x), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »

lang-py

CollectivesTM on Stack Overflow

Interprocess Communication between two python scripts without STDOUT

Answer*