1

Like in curL, we have a bind address option, we can download using a certain interface..

For ex, to download from eth0:0 we use $curl --interface eth0:0 http://www.google.com

This way, my aim is to download files but since curL doesnt support segmented downloading, i dont want to use it. We have a speed restriction on every IP. To overcome this restriction, I can start many interfaces like this, and download some part of the file from each interface.

How do i make any other download manager like AXEL to bind to a particular address? I am asking for some idea like changing variables in BASH shell, so that the whole shell binds to a particular interface, lets say eth0:1...

Improve this question

4 Answers 4

Reset to default
4

It sounds like what you really want is an application that can download a file in parts from different interfaces and join them together at the end. Eg, if you knew your file was 100MB, and you wanted to grab chunks of roughly 10MB, you'd want to do:

1. start a download of bytes 0..10000000 on eth0, saving to filename.part1
2. start a download of bytes 10000001..20000000 on eth0:0, saving to `filename.part2
3. start a download of bytes 20000001..30000000 on eth0:1, saving to filename.part3
...
N-1. wait for all downloads to complete
N. join all filename.part* together to get filename.complete

I know wget can resume a partially-downloaded file. I'm pretty sure that works by reading to the end of the existing file and then requesting the file, starting from the next byte, from the server.

It looks like curl supports partial downloads like this using the --range <byterange> option. So you could script the above steps like so:

1. curl --interface eth0 --range 0,10000000 http://some.server.com/bigfile -o bigfile.part1
2. curl --interface eth0:0 --range 10000001,20000000 http://some.server.com/bigfile -o bigfile.part2
... 
N. cat bigfile.part* > bigfile

Caveats: this doesn't always work; if the HTTP/1.1 server doesn't have the partial-download feature enabled, you'll get the whole file on each call. See man curl for details on the --range option.

Edit: fixed byte ranges in examples

Improve this answer
1

Here is a blog post about splitting a download with curl. I tried in Windows, but couldn't get it to work, but you might have success with Linux.

Improve this answer
0

At the outset, let me correct few things that you said.

1) curl wont support segment downloading -- Curl supports downloading a particular range of the file. But first you need to verify whether server supports range download and if it supports you can request a particular range of the file by passing an HTTP header Range: bytes=start-end with option -H of the curl.

2) I am asking for some idea like changing variables in BASH shell, so that the whole shell binds to a particular interface -- For most of the variables, setting the variables wont mean anything unless the application reads them using some functions like getenv() (in C).

A probable solution

One thing you can definitely do is writing a shell script using curl, you can use --interface option to bind to specific interface and --header option to request a particular range. Then you can schedule your download among interfaces that are available. But wait! You need to do a lot of parsing to implement it. You may probably want to parse the ifconfig output to get the interfaces, and you may want to parse the curl's output to implement such a downloader. Moreover, the application has high risk of failure as it is completely dependent on some specific applications output. By that I mean, ifconfig and curl has no standards on how they produce output, but HTTP has fixed standards (RFC's). So parsing HTTP messages has low risk of failure then compared to some non standardized applications.

Any projects like Axel ?

I started working on a project whose use case is same as you asked. Its completely written in C. Here is a link to my project, Multiple Interface Downloader (MID) https://github.com/m0hithreddy/MID .

  • MID uses SO_BINDTODEVICE socket option to bind to a specific interface, so in most of the cases you require super user permissions and CAP_NET_RAW capability (root user has).

  • MID determines the network interfaces to be use in the download and adopts two step split for downloading the content.

    • First step: The file is divided among network interfaces (in real time).
    • Second step: Further the file is divided among several HTTP range requests that arise from that particular interface (NOTE: Server should support them at the first place to make all of this possible)

MID supports HTTP and HTTPS protocol.

Cheers :)

Improve this answer
-1

Make eth0:0 the default route instead of eth0. The command:

netstat -rn

will show you what your current default interface is. If it is not eth0:0, delete it with

route delete 0.0.0.0

then add it with something like

route add default eth0:0
Improve this answer
2
  • 1
    I want to use all the interfaces simultaneously. Like in one terminal, i use eth0:0 and in the other i use eth0:1 .. Now, are u getting it what i actually intend to do.
    – Abhijeet Rastogi
    Commented Sep 17, 2009 at 12:52
  • 1
    You have different terminals but only one kernel. The kernel uses one set of rules for routing, no matter what application the traffic comes from. If you set up static routes to different subnets or individiual hosts, the kernel can route the traffic through different ports to get there. Or you can set up Xen which will give you a whole new IP, port range, and routing table.
    – kmarsh
    Commented Sep 18, 2009 at 12:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .