Skip to content

Navigation Menu

Sign in
Sign up

Work around crazy emails with non-base64 encoded attachments - #283

Open
eliask wants to merge 2 commits into
gaubert:master from
eliask:master
Open

Work around crazy emails with non-base64 encoded attachments #283
eliask wants to merge 2 commits into
gaubert:master from
eliask:master

Conversation

@eliask

@eliask eliask commented Jan 1, 2017

Copy link
Copy Markdown

GMail IMAP push will get in a weird state if it encounters an email
which is encoded in this manner. The connection will be reset after a
long timeout after encountering this binary crap.

I suspect an earlier version of GMail server software, or some non-IMAP
import allows or allowed this in at some point, and yet it is not
possible to upload it back verbatim.

For my purposes, it suffices to skip these rare species but in general
the email parts should be re-encoded as base64, or the problematic
invalidly encoded attachments should be removed.

The illegally encoded binary emails are detected here by looking for the
PNG magic string with bytes that SHOULD NOT appear in any legitimate
email data.

A screenshot of one of the affected emails is at
https://imgur.com/a/i09Xa.

GMail IMAP push will get in a weird state if it encounters an email
which is encoded in this manner. The connection will be reset after a
long timeout after encountering this binary crap.
I suspect an earlier version of GMail server software, or some non-IMAP
import allows or allowed this in at some point, and yet it is not
possible to upload it back verbatim.
For my purposes, it suffices to skip these rare species but in general
the email parts should be re-encoded as base64, or the problematic
invalidly encoded attachments should be removed.
The illegally encoded binary emails are detected here by looking for the
PNG magic string with bytes that SHOULD NOT appear in any legitimate
email data.
A screenshot of one of the affected emails is at
https://imgur.com/a/i09Xa.

gaubert commented Jan 2, 2017

Copy link
Copy Markdown
Owner

Thanks eliask but this seems to be just a particular case ? I see only a unicode character tabulation with justification and the start of a PNG file ? This seems to solve your particular use case but not others ?
What do you think ?

eliask commented Jan 2, 2017 via email

Copy link
Copy Markdown
Author
Yeah, it's true. It only solves this particular case but I am not sure can there be non-base64/non-ASCII/printable characters in email bodies in general. In case it they are illegal in general, it would suffice to look for the bytes with values outside the legal range (maybe just >0x7f ?) But yeah, I don't know if that is the case. Email is complicated. For this particular problem, it would also suffice to parse the MIME parts and see if base64 encoded data is ACTUALLY base64 or not. But then again this introduces some extra complexity. Worth it or not, I don't know. And this also introduces uncertainty about whether Gmail dies on binary data in general or just inside this context.
...
On Mon, 2 Jan 2017, 12:32 Guillaume Aubert, ***@***.***> wrote: Thanks eliask but this seems to be just a particular case ? I see only a unicode character tabulation with justification and the start of a PNG file ? This seems to solve your particular use case but not others ? What do you think ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#283 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAiZGBk-pGNavz6huLGOOhYpXn_liVEFks5rORiigaJpZM4LYuTa> .

Copy link
Copy Markdown

I have encountered same issue. Email source tells me that it contains based64 encoded PDF document attachment, but when I look at it, i see that there is binary. Yet somehow gmail browser client manages to open such file correctly for viewing or download it to PC, yet if I use API - I fail to read such attachment, because base64 encoded claim is false.

This commit tests all base64-encoded parts of emails to push.
This generalizes the previous ad hoc method which only worked for
detecting the PNG magic header anywhere in the message.
If a email parts/payloads claim to be base64 encoded but contain
invalid characters outside the base64 alphabet, the email is never pushed
to server. The erroneous emails are NOT sent to quarantine since they
are assumed/observed to be both rare and can be detected client
side (and notably CANNOT be detected server-side as such because they
break the protocol contract).

eliask commented May 4, 2017

Copy link
Copy Markdown
Author

@BangisBanginis thanks for the heads up. I got motivation to generalize this patch to test all base64-encoded data in emails.

@gaubert now that the patch is "complete", could you merge it to master?

NOTE: it is theoretically possible that invalid non-base64 encoded data may cause similar issues, but on balance, I believe only base64 is in a privileged position and other data would probably not cause protocol issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /