homepage

mailbox.mbox parser is splitting mbox files by "^From " pattern, which is wrong , in fairy it should split mbox by "\nFrom ".
Illustration:
------
From bla-blah@localhost
Header1
Header2
body1
body2
From blah-blah2@localhost
Header1
body1
From your dear friend
body3
------
This mbox would be splitted in 3 messages instead of 2

All the references I could find talk about triggering the match without the proceeding newline. That is, it is not certain that a blank line will precede the 'From ' header, and the typical quoting rules for mbox format call for any 'From ' at the start of a line (whether preceded by a blank line or not) to be quoted. This might have something to do with the fact that otherwise you have to special case the first line of the mbox, but I don't really know.
What tool are you using that is producing the unquoted 'From ' lines in your mbox? I know there are variants on the mbox format, so if one of them has the format you propose, this would become a feature request to support that variant mbox format.

On 2011年3月31日 14:13:50 +0000
"R. David Murray" <report@bugs.python.org> wrote:
> 
> R. David Murray <rdmurray@bitdance.com> added the comment:
> 
> All the references I could find talk about triggering the match
> without the proceeding newline. That is, it is not certain that a
> blank line will precede the 'From ' header, and the typical quoting
> rules for mbox format call for any 'From ' at the start of a line
> (whether preceded by a blank line or not) to be quoted. This might
> have something to do with the fact that otherwise you have to special
> case the first line of the mbox, but I don't really know.
> 
> What tool are you using that is producing the unquoted 'From ' lines
> in your mbox? I know there are variants on the mbox format, so if
> one of them has the format you propose, this would become a feature
> request to support that variant mbox format.
> 
> ----------
> nosy: +r.david.murray
> 
Hello, David !
This is an email from netcraft mailing list - the host which accepted
it is running sendmail with some antivirus software on top -
mimedefang + spamassassin from what I know.
Could be tat something is broken in that chain, I've spotted the error
when I was writing the script for mailbox --> maildir conversion,
while migrating this server.
So I had to inherit mailbox.mbox and fix as I need, I'll investigate
further what lead to such behaviour. 
Nevertheless, here is snippet from rfc4155 - 
In order to improve interoperability among messaging systems, this
 memo defines a "default" mbox database format, which MUST be
 supported by all implementations that claim to be compliant with this
 specification.
 The "default" mbox database format uses a linear sequence of Internet
 messages, with each message being immediately prefaced by a separator
 line, and being terminated by an empty line.
---
So I think assuming that there should be an empty line before
"From " separator line is fine (for the second email and further) and
would help to deal with all kinds of mbox mailboxes, fix is rather
trivial.
Best regards,
Valery Masiutsin

Hello Valery Masiutsin, i recently stumbled over this while searching
for the link to the standart i've stored in another issue.
(Without being logged in, say.)
The de-facto standart (http://qmail.org/man/man5/mbox.html) says:
HOW A MESSAGE IS READ
 A reader scans through an mbox file looking for From_ lines.
 Any From_ line marks the beginning of a message. The reader
 should not attempt to take advantage of the fact that every
 From_ line (past the beginning of the file) is preceded by a
 blank line.
This is however the recent version. The "mbox" manpage of my up-to-date
Mac OS X 10.6.7 does not state this, for example. It's from 2002.
However, all known MBOX standarts, i.e. MBOXO, MBOXRD, MBOXCL, require
proper quoting of non-From_ "From " lines (by preceeding with '>').
So your example should not fail in Python.
(But hey - are you sure *that* has been produced by Perl?)
You're right however that Python seems to only support the old MBOXO
way of un-escaping only plain "From " to/from ">From ", which is not
even mentioned anymore in the current standart - that only describes
MBOXRD ("(>*From )" -> ">"+match.group(1)). 
(Lucky me: i own Mac OS X, otherwise i wouldn't even know.)
Thus you're in trouble if the unescaping is performed before the split..
This is another issue, though: "MBOX parser uses MBOXO algorithm".
;> - Ciao, Steffen

It seems to me that "^From " is the correct way to match the start of each message. This is also what the qmail manual page (linked in the previous message) says. So closing as invalid.

Hello Petri
Qmail manpage does not sound as a valid reference for me, I've pointed
 relevant RFC (which dictates correct behaviour) as a reference, python
mbox parser does not conform to it.
Best regards,
Valery Masiutsin
On Sun, Jun 24, 2012 at 6:41 PM, Petri Lehtinen <report@bugs.python.org>wrote:
>
> Petri Lehtinen <petri@digip.org> added the comment:
>
> It seems to me that "^From " is the correct way to match the start of each
> message. This is also what the qmail manual page (linked in the previous
> message) says. So closing as invalid.
>
> ----------
> nosy: +petri.lehtinen
> resolution: -> invalid
> stage: test needed -> committed/rejected
> status: open -> closed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11728>
> _______________________________________
>

Actually, you're right. Sorry for overlooking the RFC. But that said, the RFC itself refers to the same manpage as a reference that's "mostly authoritative for those variations that are otherwise only documented in anecdotal form". So I guess it's quite a good reference after all :)
In Appendix A, RFC 4155 defines a set of rules for a "default" mbox format that maximizes interoperability between different mbox implementations.
The important things in the RFC concerning this issue are:
* There MUST be an empty line after each message.
* The RFC does not specify any escape syntax for message body lines starting with "From ". It says: "Recipient systems are expected to parse full separator lines as they are documented above."
Because the RFC states that there must be an empty line after each message, and it aims for maximum interoperability, I think we can assume that there always is an empty line there. But looking for "\n\nFrom " is not enough for finding the starting points of messages. We should actually parse the whole separator line which consists of "From ", an email address (addr-spec in RFC 2822), a timestamp (in UNIX ctime format without timezone), and a newline character.
I think this should be the default mode for reading mbox files. See #13698 for adding support for other formats.

Some thoughts on doing "clever tricks" to enhance mbox parsing:
 http://www.jwz.org/doc/content-length.html