10 Things which make Wget Swiss Army Knife of Network Downloaders
I am frequently asked questions about wget by my friends who have recently joined the Linux world about Wget. All of them asking what wget has and other download managers (Windows based) don't.
SO here is this post illustrating some powers of wget which make it a Swiss Army Knife of Network Downloaders.
Wget is a well known and very powerful tool made available by GNU for downloading things via command line. It is puffed with features and since it is a command line utility, it can be easily coded into scripts for advanced use.
Reliability Wget is famous for its reliability. Wget is designed for being a robust application, for successfully downloading large files over non-reliable internet connections. Doesn't matter if you are on a slow connection which breaks the link with no warnings, wget can successfully download anything for you.
It is a non-interactive utility Wget is not on the mercy of user interface (say your shell, bash for example) or the user himself to keep itself running successfully. When you run wget it can keep itself running even when you ( accidentally or intentionally) close the Terminal windowm untill the download is complete. Web browsers require constant presence of user which can be a great hindrance in when transferring a lot of data. But once wget has started downloading, you can forget it untill wget completes the downloading and you are assure that it will get successfully untill something exceptional happens. It means that it can work in background even when user has logged off the system.
Supports different protocols Different protocols supported by wget include HTTP, HTTPS, and FTP, and retrieval thorough HTTP proxies is also possible.
Wget uses single threaded downloading Well !! what do this mean now? This means that if you are expecting wget to increase your downloading speed like other downloading managers available in Windows (IDM for example), which use segmented downloading, then sorry, wget can't do that. Though their are other tools like Axel and Aria2c (Aria2 console utility), which use segmented downloading to increase speed (we will talk about them in later articles). Single Threaded approach has its own pros. It can carry on the downloads at full speed without compromising with your browser's web surfing speed.
Graphical front ends for wget There are many GUIs available for wget like Gwget (for Gnome) and Kget for KDEm for those who don't like Command Line or can't remember commands. But if you know the basic use of wget with Terminal, you can use any GUI frontend to its maximum.
Recursive downloading Wget can follow links given in HTML, XHTML and CSS pages and downloading everything (or specific type of files) present on web pages. This feature is of great use. It means that you can give wget with a link to an HTML page and set the recursive key (-r) and it will follow all the links given within that page and download everything until maximum recursion depth is achieved (read more about this in this article about advanced usage of wget).
Mirroring Sites You can mirror entire websites or parts of websites with wget for backup purposes (or if you have other reasons for doing this). You can even convert all the links given in the html pages for suitable offline reading. It changes not only the visible hyperlinks, but any part of the document that links to external content, such as embedded images, links to style sheets, hyperlinks to non-HTML content, etc. Again you can read more about this in this article. Creating offline versions of websites or online ebooks is a piece of cake with wget.
Control on Everything When downloading with wget, you have complete control on everything. From downloading through a Http Proxy, naming the user agent, to assigning the bandwidth for wget. You can change the user agent string shown to the servers when downloading from them. By default it is shown the it is "wget" accessing files on server, which can be changed to anything with a single switch (-A). And you can also limit maximum bandwidth allotted to wget for downloading. When downloading multiple files, wget can be tuned to start new downloads after a specific time or after a random time period. This prove to be very useful when you don't want the server to notice you for suspicious activity.
There is much more customization possible. Refer to wget man pages for complete list of cammands or read articles on Beat Of The Geek for easy introduction on using wget.
Script-able Wget is in strong coordination with the Unix theory, "Do one thing and do it very good". It do one thing and is very good at it, i.e. downloading. Since it is a command line utility, it can be easily scripted using shell scripting (bash, sh etc), or using any other scripting language like Python, Perl etc. It is purely on your imagination and creativity, how you identify the potential of wget and how you utilize it.
Platform Independent Wget is quite a platform independent tool. If you are using a Linux machine, then maximum chance is that it is already present in your arsenal. It can be downloaded for for Windows from here OR if you have installed Cygwin for emulating Linux applications, it is automatically installed in your Windows machine (most probably). You can get it for your Mac from here.
About Auther:
Charanjit Singh blogs at channikhabra.blogspot.com, on Open Source Technology and Linux.
You can visit his blog for information rich articles and tutorials.
Facebook Link: Beat Of The Geek
Twitter: Beat Of The Geek
Latest update from Charanjit Singh:
1.How to backup your installed software on Ubuntu using Aptoncd
2.Why do I hate Linux ?
Labels: Linux, Tech News
