I am looking for an FTP client that works on Windows 7 and can open a folder with millions of files quickly (e.g. 50 millions). It should be GUI and list all the files. Both FileZilla and WinSCP takes forever to display the folder's content.
asked Dec 7, 2015 at 15:20
-
1If you care about performance, you should specify the time and your hardware specs (especially the network connection speed) to get comparable results. FTP over a 56k modem will definitely take forever. It's much faster if you use 10000BaseT fiber. Scale with hardware :-)– Thomas WellerCommented Dec 7, 2015 at 20:06
-
250 Million File names at - if DOS 8.3 ASCII = 11 bytes = 55 MB of data to transfer for the file names alone - if there is 1MB link anywhere on the path that is a 50 second transfer for that alone. Also with 50M GUI objects you are likely to run out of GUI IDs on windows systems.– Steve BarnesCommented Dec 7, 2015 at 20:47
-
2@SteveBarnes: Your calculation should result in 550 MB, not 55 MB. And FTP also transfers modification time, owner information and user/group info, ~96 bytes per file. It does that with MSLD, which already switches to an optimized transfer mode. It's 4.8 GB (!) to transfer. On a 1 Gbps link, this alone takes 35 seconds. Keeping the 4.8 GB of data in RAM will require a 64 bit application.– Thomas WellerCommented Dec 7, 2015 at 21:20
-
1Besides that, 4.8 GB on Ethernet (packet size 1514 bytes) will require 3.1 M packets to be sent. At 1ms ping time, 2ms round trip time, this accounts for another 1:45 hours!– Thomas WellerCommented Dec 7, 2015 at 21:23
-
1If it's a local Intranet, you could try to enable Jumbo packets which have a size of 4096 bytes. This might bring it down to 0:40 hours. You have to do it in Windows (Registry) and all switches/routers need to support it.– Thomas WellerCommented Dec 7, 2015 at 21:27
|
Show 5 more comments
1 Answer
I already replied to your similar question here:
https://softwarerecs.stackexchange.com/a/27511/2336
Specifically in this question (FTP client), also network data transfer speed comes in as significant factor – see comments of Thomas Weller under your question. But you can still attempt to put files on remote side into single archive (TAR, ZIP) and after transferring the archive to your local machine, you can unpack it and go with one of solutions sketched in the answer linked above.
