UseMod Wiki: WikiBugs/DownloadPageRedirectsToMSWithWget

Not a bug, see explanation below. --CliffordAdams

UseModWiki has moved at the end of april 2005 to another provider. The "problem" with wget should be solved now.

I just got these strange results using wget:

patsplat@prince:~$ wget http://www.usemod.com/wikicode/usemod092.tar.gz
--18:57:10--  http://www.usemod.com/wikicode/usemod092.tar.gz
           => `usemod092.tar.gz'
Resolving www.usemod.com... done.
Connecting to www.usemod.com[63.151.147.62]:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.microsoft.com/ [following]
--18:57:10--  http://www.microsoft.com/
           => `index.html'
Resolving www.microsoft.com... done.
Connecting to www.microsoft.com[207.46.249.190]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31,399 [text/html]

100%[===================================================>] 31,399       378.56K/s    ETA 00:00

18:57:11 (378.56 KB/s) - `index.html' saved [31399/31399]

patsplat@prince:~$

Moved from the UseModWiki/Download page:

Q: When I wget that tarball, I get www.microsoft.com/index.html. Why is that?

A: (June 3rd. 2005) This does not seem to be at problem any more, at least not with wget 1.8.2

A: use wget -U "stupid admins" http://www.usemod.com/wikicode/usemod10.tar.gz the -U set the user-agent changeing this will "fool" the hosting firm and allow you to download. This is a feature for the hosting firm but a BIG nuisance for us.

The server might not like the User-Agent header the client you are using is providing. I've had the same problem when trying to download the tarball with wget; since Netscape seemed to work fine, though, I tried supplying a custom User-Agent to wget ("Mozilla/3.0", for example), and it worked indeed.

: This web site (thinks it) needs everyone to use a HTTP/1.1 browser, but wget only speaks HTTP/1.0, i.e. an older version of the protocol (this is true as of wget version 1.5.3). It would be useful if somebody could put a mirror of the script on a site which does not have this restriction. As for why on Earth Apache will redirect to Microsoft's site if and only if the User-Agent identifies Wget specifically, that's anybody's guess ... If you give the server the exact same headers as wget produces, but without the User-Agent (yes, even with a HTTP/1.0 request), it will gladly serve you the file. --EraEriksson?

: I guess this is some sort of weird anti-spam feature.. There was a text somewhere with a list of User-Agents used by spambots. Unfortunately spammers use Wget too, so Wget is in this list as well. :-( -- Wilmer van der Gaast

Well, it prevents people from slamming the site with wget all the time. My roommate once tried to wget all of WikiWiki, and then print it, as a "present" for me. Sure, it was merely 8000 pages at the time, but you might appreciate how that's bloody huge. No, he's none too swift. ;) -- SunirShah

Not to promote people slamming websites, but if you DO want to use wget - have a look at the wget man page and read up on "-U agent-string" or in its long form "--user-agent=agent-string" - I use: wget -N -U "<insert whatever you want to call it here>" <insert url you want to download here> or put something like this in a file called .wgetrc in my home directory:

 user-agent="l33tw4r3l33cht00l0wnzj00/1.0 (incompatible; GNU Wg3t 1.8.1; DebianLinux [Woody]; i686)"

: Just to be clear, this wget-avoidance feature is a "feature" of my current hosting provider (futurequest.net), and not anything specific to UseModWiki or even usemod.com. Futurequest has provided amazingly good service for the low price I pay ($420/year for their "Gold" package: 150mb disk and 20Gb bandwidth/month), so I'm not complaining. Remember that usemod.com shares a machine with at least 100 other domains--some limits help ensure that no single site causes problems for the rest. --CliffordAdams

: If you really want to fetch the tar.gu frm a shell, curl works fine. The command would be:

$ curl -O http://www.usemod.com/wikicode/usemod092.tar.gz

Another way might be lynx:

$ lynx --source http://www.usemod.com/wikicode/usemod092.tar.gz > usemod092.tar.gz

Personally I'll have to decline using UseModWiki due to this 'feature'.. --stain

Huh? You don't want to use the UseModWiki software because the company that hosts this site has made some silly choices in configuring their web servers? What kind of a reason is that? --DavidWall

Perhaps stain wants to sponsor you a dedicated server :-) --absi

I like -U "stupid admins". I hope they notice in their logs!

looks like this problem is solved now?! I could wget the file without trouble, and didnt need to specify the user-agent.

wget http://www.usemod.com/wikicode/usemod10.tar.gz
--10:18:29--  http://www.usemod.com/wikicode/usemod10.tar.gz
           => `usemod10.tar.gz'
Resolving www.usemod.com... 63.151.147.62
Connecting to www.usemod.com[63.151.147.62]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 64,038 [application/x-tar]

100%[====================================>] 64,038        88.33K/s

10:18:31 (88.07 KB/s) - `usemod10.tar.gz' saved [64038/64038]

I use GNU Wget 1.9.1 , maybe it is due to the wget version? Or did you get your provider to fix this annoying "feature" ? --mutante