1

Several processes are each writing a file to a directory. The goal is to control the size of the directory such that whenever it reaches a size (S), all processes stop writing to the directory and discard the file they are about to write.

If the size then becomes lower than S because some of those files were removed, the processes will resume writing files.

It seems that I need inter-process locking to achieve this design. However, I thought maybe there's an easier way, since inter process locking is not readily available in python and obviously there's contention between processes.

Python 2.7 Platforms (Win, Mac, Linux)

asked Jan 21, 2014 at 16:21

1 Answer 1

1

Using lock files may be an option. For example, each process checks for a file like "/target_dir/lock" before write. If file exists, process will not write anything. So you have to run separate monitor process, which checks directory size, and creates or deletes lock file.

answered Jan 21, 2014 at 16:28
Sign up to request clarification or add additional context in comments.

3 Comments

I don't have the luxury of having a monitor process. Without a monitor process, each process needs to update a size written in a file as it adds another file (or removes). It creates a lot of contention, but doesn't have a monitor.
You can add checks in your writer processes. So each process before write check the dir size. If it reaches some threshold, process will create lock file and stop writing. If it doesn't reach one but lock file exists, process will delete lock file and start writing. IMHO, it is possible but unclean solution, separate monitoring process would be better.
Checking dir size can be very expensive on each write. On Windows, you literally have to enumerate each file and add up the sizes.

Your Answer

Draft saved
Draft discarded

Sign up or log in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

Post as a guest

Required, but never shown

By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.