Account registration attempts with scraped emails potentially lead to blacklisting Codeberg emails #298

hw commented 2020年10月02日 09:58:54 +02:00

In the last few days we received reports from Cloudmark, Yandex, Microsoft that they are flooded with Codeberg activation emails users flag as spam. We see in our logs many accounts with random usernames that never active, so the reports seem to have a foundation. It seems to work like this:

• evil guy get some lists with email accounts,
• evil guy tries to register (manually or even semi-automated) codeberg.org accounts with these addresses,
• email address owners get unsolicited activation email and hit the spam button.

Question: As we are not involved in the loop, what can we do to stop this problem?

In the last few days we received reports from Cloudmark, Yandex, Microsoft that they are flooded with Codeberg activation emails users flag as spam. We see in our logs many accounts with random usernames that never active, so the reports seem to have a foundation. It seems to work like this: - evil guy get some lists with email accounts, - evil guy tries to register (manually or even semi-automated) codeberg.org accounts with these addresses, - email address owners get unsolicited activation email and hit the spam button. Question: As we are not involved in the loop, what can we do to stop this problem?

martijndeb commented 2020年10月02日 10:23:29 +02:00

Is it possible to provide an extra security challenge step if certain conditions are met?
Because there are email reputation services out there, haveibeenpwnd could be checked, things like that. It's quite likely that if the address was scraped it's on haveibeenpwnd.

Instead of blocking (possibly legitimate) users, one could then pose another step. "I hereby declare that I'm a real user" and post a second captcha or similar solution. That would most likely stop automated registering?

Is it possible to provide an extra security challenge step if certain conditions are met? Because there are email reputation services out there, haveibeenpwnd could be checked, things like that. It's quite likely that if the address was scraped it's on haveibeenpwnd. Instead of blocking (possibly legitimate) users, one could then pose another step. "I hereby declare that I'm a real user" and post a second captcha or similar solution. That would most likely stop automated registering?

daniele commented 2020年10月02日 14:29:07 +02:00

At least for the microsoft "galaxy" you could request access through this page:
https://sendersupport.olc.protection.outlook.com/snds/index.aspx

This will get you a warning when someone marks emails coming from IPs you control as spam.

It's mostly a way to get informed AFAIK, it does not allow to appeal against any decision nor ask to be removed from any spam list.

The registration is (convoluted but) free as in gratis the person registering needs a microsoft account.

At least for the microsoft "galaxy" you could request access through this page: https://sendersupport.olc.protection.outlook.com/snds/index.aspx This will get you a warning when someone marks emails coming from IPs you control as spam. It's mostly a way to get informed AFAIK, it does not allow to appeal against any decision nor ask to be removed from any spam list. The registration is (convoluted but) free as in gratis the person registering needs a microsoft account.

tklein23 commented 2020年10月03日 22:37:21 +02:00

Do we keeps logs about registration attempts and the IPs they came from?

Do we keeps logs about registration attempts and the IPs they came from?

tklein23 commented 2020年10月03日 23:27:02 +02:00

I like @sexybiggetje to re-think the registration process. How to take this a step further:

1. Manual approvals of registrations. This might annoy users and introduce additional work for us, but it would avoid scripted account creations.
2. Invite-based registrations and limit the number of invites per user per time unit.
3. Close "free" registration for a bit.

I like @sexybiggetje to re-think the registration process. How to take this a step further: 1. Manual approvals of registrations. This might annoy users and introduce additional work for us, but it would avoid scripted account creations. 2. Invite-based registrations and limit the number of invites per user per time unit. 3. Close "free" registration for a bit.

hw commented 2020年10月04日 16:29:29 +02:00

Do we keeps logs about registration attempts and the IPs they came from?

7 days as described in Terms of Use

> Do we keeps logs about registration attempts and the IPs they came from? 7 days as described in [Terms of Use](https://codeberg.org/codeberg/org/src/branch/master/TermsOfUse.md)

martijndeb commented 2020年10月08日 17:26:54 +02:00

@tklein23 I'm unfamiliar with the codebase (and go). I have tried looking at the gitea source to see what options there are. Looks like they don't have a system of hooks there.

Option 1;
Is making multifactor authentication required an option? Or doesn't Gitea provide that as a mandatory option during registration?

Option 2;

What's involved with manual approving? Because we might turn on manual approving, and if possible create a system account that approves accounts on request via the API. (If the API provides an endpoint for this).

That we one could fire up a cronjob that does several things:

1. Check the e-mail address for reputation
2. Check the domain for reputation
3. Approve the request, or leave it open for manual tasks
4. Send custom mail to "support@codeberg" and "enduser" with a message akin to "We have choosen not to accept your registration because of the reputation services we use flagged your e-mail. If this is in error, please contact xyz"

This would still trigger some emails to those addresses, so this might not be the best alternative. But it could provide an extra step.

@tklein23 I'm unfamiliar with the codebase (and go). I have tried looking at the gitea source to see what options there are. Looks like they don't have a system of hooks there. Option 1; Is making multifactor authentication required an option? Or doesn't Gitea provide that as a mandatory option during registration? Option 2; What's involved with manual approving? Because we might turn on manual approving, and if possible create a system account that approves accounts on request via the API. (If the API provides an endpoint for this). That we one could fire up a cronjob that does several things: 1) Check the e-mail address for reputation 2) Check the domain for reputation 3) Approve the request, or leave it open for manual tasks 4) Send custom mail to "support@codeberg" and "enduser" with a message akin to "We have choosen not to accept your registration because of the reputation services we use flagged your e-mail. If this is in error, please contact xyz" This would still trigger some emails to those addresses, so this might not be the best alternative. But it could provide an extra step.

lhinderberger commented 2020年10月09日 12:24:33 +02:00

@sexybiggetje For the manual approval process you have described, could we prevent sending out mails if we modify the process to check the email reputation right when sending the registration form?

Because that way we could confine accounts with scraped email-adresses to "quarantine" and inform the user about that in the response page after sending their registration form instead of per mail triggered by cronjob. Accounts with good email reputation would not be sent to quarantine and instead would follow the regular registration-with-douple-opt-in process.

@sexybiggetje For the manual approval process you have described, could we prevent sending out mails if we modify the process to check the email reputation right when sending the registration form? Because that way we could confine accounts with scraped email-adresses to "quarantine" and inform the user about that in the response page after sending their registration form instead of per mail triggered by cronjob. Accounts with good email reputation would not be sent to quarantine and instead would follow the regular registration-with-douple-opt-in process.

martijndeb commented 2020年10月09日 14:35:33 +02:00

I think that could be possible but requires patching on Gitea. Or one would need to "trap" those emails in the SMTP server to prevent them from going out. But no clue how to achieve that.

I think that could be possible but requires patching on Gitea. Or one would need to "trap" those emails in the SMTP server to prevent them from going out. But no clue how to achieve that.

6543 commented 2020年10月09日 18:20:25 +02:00

I think this is a nice feature for gitea ... so all can benefit, if somebody has time to implement it

EDIT: I also like to have this for my instance ...

I think this is a nice feature for gitea ... so all can benefit, if somebody has time to implement it EDIT: I also like to have this for my instance ...

hw commented 2020年10月09日 21:08:44 +02:00

Option 1;
Is making multifactor authentication required an option? Or doesn't Gitea provide that as a mandatory option during registration?

I like the idea of advertising the use of 2FA more prominently.

Option 2;

What's involved with manual approving? Because we might turn on manual approving, and if possible create a system account that approves accounts on request via the API. (If the API provides an endpoint for this).

That we one could fire up a cronjob that does several things:

1. Check the e-mail address for reputation

Problem is that these registrations use valid email addresses of random people (who never heard about Codeberg or Gitea and thus think this is spam).

> Option 1; > Is making multifactor authentication required an option? Or doesn't Gitea provide that as a mandatory option during registration? I like the idea of advertising the use of 2FA more prominently. > Option 2; > > What's involved with manual approving? Because we might turn on manual approving, and if possible create a system account that approves accounts on request via the API. (If the API provides an endpoint for this). > > That we one could fire up a cronjob that does several things: > 1) Check the e-mail address for reputation Problem is that these registrations use valid email addresses of random people (who never heard about Codeberg or Gitea and thus think this is spam).

hw commented 2020年10月23日 22:13:28 +02:00

It seems we got whitelisted in this case, email checker tools report green again. (the underlying mechanism itself ofc might get exploited again in the future).

Maybe one step forward: https://github.com/go-gitea/gitea/issues/13283.

It seems we got whitelisted in this case, email checker tools report green again. (the underlying mechanism itself ofc might get exploited again in the future). Maybe one step forward: https://github.com/go-gitea/gitea/issues/13283.

6543 commented 2020年11月19日 18:33:36 +01:00

UPDATE: I'll focus on "[Feature] block-/allow-list for emails (#13628)" ... it might be an impruvement

UPDATE: I'll focus on "[[Feature] block-/allow-list for emails (#13628)](https://github.com/go-gitea/gitea/issues/13628)" ... it might be an impruvement

BoilingOil commented 2020年11月19日 19:27:36 +01:00

If there is a connection with my issue #191, then it must be the Outlook server.
I was waiting for an activation email which the server had unilaterally flagged as spam. But even when I reversed that flag (to make the link clickable), the link only caused the server to respond with "Bad Request". Clicking the same link from within gmail, later, caused no problems.

The only link I see between the two issues, is Outlook and Codeberg. Outlook's server marking the email as spam is one part. But that isn't what causes the "Bad Request" response when clicking the link.

If there *is* a connection with my issue #191, then it must be the Outlook server. I was *waiting* for an activation email which the server had unilaterally flagged as spam. But even when I reversed that flag (to make the link clickable), the link only caused the server to respond with "Bad Request". Clicking the same link from within gmail, later, caused no problems. The only link I see between the two issues, is Outlook and Codeberg. Outlook's server marking the email as spam is one part. But that isn't what causes the "Bad Request" response when clicking the link.

hw commented 2020年11月19日 21:30:11 +01:00

But that isn’t what causes the "Bad Request" response when clicking the link.

Could it be that this was outdated? Does this also happen for a second re-activation attempt?

> But that isn’t what causes the "Bad Request" response when clicking the link. Could it be that this was outdated? Does this also happen for a second re-activation attempt?

BoilingOil commented 2020年11月20日 03:59:39 +01:00

But that isn’t what causes the "Bad Request" response when clicking the link.

Could it be that this was outdated? Does this also happen for a second re-activation attempt?

Outdated my foot! And yes, it happened for every activation attempt, as long as I tried to click the link on live.com.

I'll recount the events in order:

1. I registered here
2. I immediately went to see if the activation email had appeared in my gmail box. It hadn't.
3. I went to live.com, where I found it in spam (hence why it hadn't appeared in gmail)
4. I removed the spam-flag, clicked the link, and got "Bad Request". I waited till the following moring, perhaps, there was some problem at live.com...
5. Next day, the email was outdated, so...
6. I came back here, requested a second activation email.
7. I went DIRECTLY to live.com, where I found it in spam again, unflagged it, clicked the link: again "Bad Request".
8. I waited for the second email to appear in my Gmail account, clicked the link, and it worked!

For reference, reread issue #191 !!

> > But that isn’t what causes the "Bad Request" response when clicking the link. > > Could it be that this was outdated? Does this also happen for a second re-activation attempt? Outdated my foot! And yes, it happened for every activation attempt, as long as I tried to click the link on live.com. I'll recount the events in order: 1. I registered here 2. I immediately went to see if the activation email had appeared in my gmail box. It hadn't. 3. I went to live.com, where I found it in spam (hence why it hadn't appeared in gmail) 4. I removed the spam-flag, clicked the link, and got "Bad Request". I waited till the following moring, perhaps, there was some problem at live.com... 5. Next day, the email was outdated, so... 6. I came back here, requested a second activation email. 7. I went DIRECTLY to live.com, where I found it in spam again, unflagged it, clicked the link: again "Bad Request". 8. I waited for the second email to appear in my Gmail account, clicked the link, and it worked! For reference, reread issue #191 !!

hw commented 2020年11月20日 10:08:11 +01:00

Outdated my foot!

Outdated in this context is if the activation takes more than 3 hours.

For reference, reread issue #191 !!

thank you!

> Outdated my foot! Outdated in this context is if the activation takes more than 3 hours. > For reference, reread issue #191 !! thank you!

BoilingOil commented 2020年11月20日 11:01:39 +01:00

Outdated in this context is if the activation takes more than 3 hours.

That much was obvious, thank you.

thank you!

You're welcome.

> Outdated in this context is if the activation takes more than 3 hours. That much was obvious, thank you. > thank you! You're welcome.

fnetX commented 2021年04月08日 19:12:46 +02:00

Having an eye on email reputation is an ongoing and repeated task of course. Better spam protection is tracked in other issues, so I'll close this. Thank you for the discussion.

Having an eye on email reputation is an ongoing and repeated task of course. Better spam protection is tracked in other issues, so I'll close this. Thank you for the discussion.