Can and should Stack Overflow automatically rewrite bit.ly links? [duplicate]

Question

This question already has answers here:

Ban URL shortening services (13 answers)

Closed 11 years ago.

Quite a few people paste in bit.ly links into comments or answers, sometimes even for linking to other pages (or answers within the same page) on Stack Overflow. This way, you do not see where the link is pointing to.

Assuming that there is an API to do this, should the site automatically rewrite the links to point at the original URL?

Are there any benefits to leave them at bit.ly?

Why are people pasting them through bit.ly in the first place? Stack Overflow displays shortened URL anyway, so formatting should not be a concern.

• tomjedrz: What is the problem with the bit.ly links? More to the point .. what is the problem you want to solve?
• Thilo: The problem is that I cannot see where the link is going before clicking. Also Stackoverflow has (or could) handle internal links differently (add more info to them) which will not work if it cannot recognize the URL.
• Thilo: I am mostly concerned with internal links, pointing back into Stackoverflow. But even for external links I do not like the obfuscation.
• Brad Gilbert: This would be fixed if we had an official shortened URL service meta.stackexchange.com/questions/23834/…
• random: @Brad That would still be the same problem as discussed, not knowing where the actual link goes.
• dlamblin: You may assume there is an API; it involves actually requesting each pasted link and confirming that it results in a status 200, or following it through to that status from the 300 family.
• George Stocker: But why do we need shortened URLs on Stack Overflow? This isn't twitter.
• Ether: FWIW, url shorteners make the ban on lmgtfy links totally pointless: meta.stackexchange.com/questions/15650/…
• Arjan: Just for the record: it seems that in the latest data dump there are 621 posts at SO with one or more bit.ly links (and 39 posts referring to tr.im, wich will soon RIP).
• Arjan: Hmmm, the very first tr.im URL I decided to save from the RIP was spam (to a blog post, but still...)
• apaul: @Duncan This post was written nearly a year before the supposed dup.
• Duncan Jones: @apaul34208 I didn't think that chronological order was important for duplicates. Rather, I voted to close because the other question is a superset of this question and contains an authoritative response from Jeff.

Answer 1

Eschew obfuscation.

I would not be opposed to an outright ban on obfuscated and shortened URLs. If one is entered into a question, answer, or comment, the system should simply reject the post with a link to the FAQ. There's no reason to shorten a URL that doesn't need to be typed out, and some people do care about the destination of a URL before clicking on it (for example, I have no wish to test my company's proxy or URL logging by visiting a NSFW site).

One other situation, which the site operators should care about: using an URL shortener allows using amazon.com URLs that are not rewritten by the system to insert SO referral information.

Another case is lmgtfy links, which are banned from the site but I have seen users using URL shorteners to get around this restriction.

• tomjedrz: I actually OPPOSE adding the SO referrer to the amazon.com links. The site should not be changing the messages from what the poster submitted. And, frankly, if it NOT SO that is suggesting the Amazon book, it is the poster. At the very least, this behavior should be transparent and told to the users as it is happening.
• user50049: +1, I will very seldom click a link if I can't tell where its pointing.
• Jared Harley: For us Greasemonkey users, there's always TinyURL Decoder: userscripts.org/scripts/show/40582
• Paul Nathan: I do not like tinyurls. who knows when Goat Se is lurking...
• Pekka: Where is goatse these days, anyway? Anybody know a stable mirror?
• quack quixote: @Pekka - I'm militantly agnostic about goatse's current location. I don't know and you don't either!!
• Mick MacCallum: Well, it was only a matter of time... stackoverflow.com/questions/14268669/…
• user215114: Not only should URL shorteners be banned, but ALL URLs that redirect to a different domain. Someone could potentially get around a URL shortener ban and continue to obfuscate links by using their own domain to perform a redirect. Proof of concept: gparyani.com/stackoverflow uses my own domain to redirect to the Bing homepage.

Answer 2

If you have the rep to do so, rewrite those links out into the actual.

Unless the problem comes from some quirk with colons, other punctuation marks or something else like too many characters to escape for.

Though the encoding continues to be improved on the links, so even that argument is falling away.

The problem with URL shorteners is that if the service fails you have no idea where that link went to and don't have an idea as to where to find the replacement. If the site just changes how it does its URLs, then you at least have the domain and the old URL slug to work off of.

• Thilo: Good hint about using my rep to rewrite. Will do that. Does not work with comments, though (at least not at my rep level).
• random: Yeah, you can't touch comments, just muck around with the posts themselves.
• Zoredache: You could add another comment with the correct link.
• Gnome: @Thilo: You can flag comments.

Answer 3

Yes, this should be done automatically in the background, all the shortened links will rot long term and become useless.

Answer 4

Yeah!

When it comes to the internet, one of my fears is reliance on technologies that can't sustain themselves. URL shorteners can't sustain themselves by themself. By definition, they can't show ads (If it's not silent, nobody would use it), and if it dies (like tr.im almost did), the links are worse than useless, even if the original pages remain.

Google's service gives me hope (If google go down, we have bigger worries than their URLs not working), but they still have no place on M?S[OFU] (If we're going to regex-refer to the trilogy, we have to include meta!), and if at all possible, automatic de-shortification should occur.

More to the point, who the hell is shortening links here? Surely that takes more effort for no gain at all?

(On a side note, I entirely blame twitter for all of this. If they'd simply not counted URLs in character limits, this entire scary system would never have been created)

• Kip: tinyurl predates twitter...
• Phoshi: @Kip; Wow, quite significantly, too! I wasn't aware of that. I still blame twitter for popularizing them, though ;)
• Caleb: I often shorten links in comments or edit notes on SO because they count towards the character limit and long ones make the messages really hard to read. A good example would be the popular link to Jon Skeet's question writing hints page: tinyurl.com/so-hints

Answer 5

First, I agree with Joel C. in that shortened links are occasionally a necessary evil:

There are a few cases where the shortened links are necessary. For example, try linking to anything in the Internet Archive's wayback machine or a screenshot on browsershots.org. The markdown chokes and it just won't work without an intermediary. This makes an outright ban problematic.

Second, this is a fairly challenging technical problem, in that it requires us to go through all posts and follow all links -- how do we know which ones are "url shorteners"? The ones on a whitelist? And when do we do this? On a scheduled task or cron job? Every time a post is saved?

I think it might make sense in the short term to just do this the old-fashioned way, with tiny slivers of fractional human effort -- if you see a shortened link that has no reason to be shortened, then un-shorten it.

• alexanderpas: a blacklist of url-shortners (like bit.ly) combined with a whitelist of destinations (like the Internet Archive's wayback machine or a screenshot on browsershots.org) - when a blacklisted link does not lead to a whitelisted destination (based on domain) they will be unwrapped, unless they contain specific problematic characters.
• Ether: Isn't "lmgtfy . com" blacklisted (it must be, because I can't even post a comment with that URL)? Simply expand the blacklist to a few other sites that we know aren't welcome.
• Jeff Atwood: @ether such as.....?
• user154510: I don't think following the shortlinks is necessary. Just provide a stack.ly shortener that only allows links to IA or browsershots :P

Answer 6

I changed my mind about url shortening services, they indeed do more harm than good in SO ecosystem. I still think it doesn't make sense to call someone as a spammer just by using it, but I do believe it's bad for "link health" in long term.

So, I would like to do a suggestion in this answer: SO could just rewrite these links, maybe using an API from longurlplease.

(I don't know if SO team are really going to depend in a third party service like this, but since url shorteners are a problem from every site that supports user generated content with links, I still think this is a valid and useful concept)

• Arjan: As for going to depend in a third party service -- if the service stops working, then getting the true URLs might stop working. But any URL that has been rewritten up till then, will be fine forever though.
• Paŭlo Ebermann: Looks like longurlplease stopped working since then.

Answer 7

There are actually some legitimate uses for shortening services.

The examples in my answer here would not have been possible without a shortening service. SO was misinterpreting the URL because it contained a regular expression as part of the query string:

http://www.strfriend.com/vis?re=<([A-Z][A-Z0-9]*)\b[^>]*>(.*%3F)</1円

However in the answer I explicitly stated the reason for the shortened URLs.

• Shog9: If SO translated them server-side, it could certainly do so after recognizing them as links and avoid the problem.
• Arjan: Was SO misinterpreting, or did you forget to encode reserved characters?

Answer 8

There are a few cases where the shortened links are necessary. For example, try linking to anything in the Internet Archive's wayback machine or a screenshot on browsershots.org. The markdown chokes and it just won't work without an intermediary. This makes an outright ban problematic. I can't find the link now, but in the bug reports from one of those Jeff himself suggested a url shortener as a work-around.

That said, unless it's one of those rare exceptions you probably shouldn't do it, and if you have the rep go ahead and clean it up.

• Mechanical snail: Nonsense: web.archive.org/web/20110728211623/http://stackoverflow.com
• Joel Coehoorn: In 2009, it was a problem. It seems they fixed it... or at least part of it.

Answer 9

If you are that concerned about shorten urls you should be using a tool on your browser to expand them on all sites. Both Google Chrome and Firefox have tools that let you expand short URLs. As Jeff pointed out they are sometimes a necessary evil, and I don't think it is work fixing when there are browser based workarounds.

Answer 10

I would rather vote to infer the title from the posted link

• Paŭlo Ebermann: That is independent, I would say. This one is for replacing the URL, the other for putting a useful title.

Answer 11

People use bit.ly because using it you can track how many clicks, twitter and other social media mentions. It's not just about shortening links, is about trying to measure it's impact. You have a panel in bit.ly for link statistics.

I don't mind having bit.ly links. They try to prevent spam from their links, so to me it's even safer than have a http://codertutorials.example.com link for example - I can read the link, but have no idea what's waiting for me there. In bit.ly at least they are providing a layer of protection.

Seriously, I want a good answer. And url shorteners are here to stay. Even google sort of has/had its own.

• random: URL shorteners are here to stay. Just like tr.im never shut down and broke (and obfuscated to no good) thousands of links.
• GmonC: And google created an url shortening service. What do you think it's really going to happen, all url shortening services are ceasing to exist, or are they going to improve? I have a lot of old bookmarks without url shortening services that are broken. As every action you do on the internet, you have to choose a reliable service. We can't predict what's is going to happen with url shortners in a long term. And trim still exists. blog.tr.im/post/160697842/tr-im-resurrected. Are we going to be all paranoid or just try to get advantage of these services since they are here to stay?
• random: If you're trying to track the clicks through a link you post on SOFU, that sounds like you're spamming.
• GmonC: You can't say I'm spamming because I just want to know how many people are clicking in provided links. If I wanted to track clicks from websites I own, I would just use google analytics. bit.ly is useful to tracking links out of my control. It's silly to say that a useful link to an answer, just because it's using a url shortening service that gives statistics, is spamming.
• Gnome: You already have a measure of the impact and usefulness of your answers: the voting score. For what purpose do you want the additional tracking?
• GmonC: I don't use bit.ly links in SO. What I'm saying is that I can see a reason in someone using bit.ly links. I don't know why someone wants to track down some links, and I cant judge someone as a spammer just because she wants to track something. Seriously, you people are overeacting to something so simple. If you don't like url shortening services, just don't use. But don't judge people that uses them. It's the same as judging someone based on tools she uses.
• Arjan: To extend on @random's comment a bit: tr.im has now announced they completely would like to shut down the API and redirection service by the end of 2010. Thanks a lot, not. (Even their own blog.tr.im is no longer there.)
• Arjan: (I guess I'll have some editing to do: it seems just 39 posts on SO are using tr.im.)
• GmonC: I changed my mind. It's doing more harm than good having url shorteners in a system like SO, it really doesn't make sense to have these links here. And now that twitter created it's own url shortening services (help.twitter.com/entries/109623), we can expect a lot of third party services to die too.