How do I get better USB transfer speeds in (X)ubuntu?

Question

When I connect my USB 2.0 drive to Xubuntu and try to transfer large files, transfer speeds are good at first but drop after a few seconds to 1-2 Mib/s. From what I read, the fast transfer at the beginning is just until the cache is full, then the real USB transfer speed is used.

In Windows, speed is constant at about 25Mib/s, same stick, connected to same port.

This is the output from dmesg when connecting the stick:

[ 5237.580084] usb 1-4: new high speed USB device using ehci_hcd and address 6
[ 5237.714318] scsi4 : usb-storage 1-4:1.0
[ 5238.713909] scsi 4:0:0:0: Direct-Access     SanDisk  Cruzer           8.02 PQ: 0 ANSI: 0 CCS
[ 5238.715264] sd 4:0:0:0: Attached scsi generic sg2 type 0
[ 5238.727225] sd 4:0:0:0: [sdb] Attached SCSI removable disk
[ 5242.308981] sd 4:0:0:0: [sdb] 31301631 512-byte logical blocks: (16.0 GB/14.9 GiB)
[ 5242.309589] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[ 5242.311228] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[ 5242.311238]  sdb: sdb1

The stick is automounted, here is the output of "mount":

/dev/sdb1 on /media/B82C-6B07 type vfat (rw,nosuid,nodev,uhelper=udisks,uid=1000,gid=1000,shortname=mixed,dmask=0077,utf8=1,showexec,flush)

I have to use vfat because I want to transfer files to my Playstation3 system. Althouth the test below shows, that this is not the main cause.

It seems this is a common issue in (X)ubuntu. I haven't found a clear solution yet. It seems that the stick needs to be mounted as async instead of flush but I'm not exactly sure how to achieve this. I don't mind if I have to unmount the stick each time before disconnecting it as long as the transfer speeds are better.

Any ideas?

Nov 2: This bug report seems to be related: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/182931. What worries me is that the problem seems to be over 2 years old and still isn't solved.

Nov 10: I started a read-test with palimpstest (from gnome-disk-utility project). This gives me a minimal transfer rate of 23 MBit/s. Also tried it in krusader now, the read speed is constantly > 20 MBit/s. So the problem really is just writing to the stick!

Here is an image of a read-write test in the same program:

I had to delete the filesystem completely before the test which shows that vfat is not the main cause as mentioned above.

Another hint: If I transfer a 100 MB file, once it's at 100% it takes an unusual long time to actually finish the transfer. This again looks to me that the file is indeed in a cache but writing to the usb device is very slow.

Nov 12: "Good news". The problems seems to be memory stick itself. Trying an external harddrive (My Book Elite 1TB) gives me constant write speed of about 20 MiB/s. I first tried to reformat my memory stick so I can really compare the two (file sytems were different). After reformatting, both drives are mounted exactly the same way:

/dev/sdb1 on /media/My Book type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)
/dev/sdc1 on /media/cruzer_ type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096,default_permissions)

Cruzer is the stick (speed still dropping after a few seconds), My Book is the external harddrive with a constant write speed of 20 MiB/s.

Hard to say what the problem really is then. As I said, in Windows I get equally fast write speeds with the SanCruzer too.

Edit at end of bounty: Unfotunately, the problem with the stick is still not solved. However, after 7 days I want to award the bounty of course. The comment from aking1012 was really helpful as it shed a new light on the matter. However, I feel that I have to give the bounty to harrymc as he showed the most effort of getting the problem solved. Thanks all.

Answer 1

The article Slow USB 2.0 mass storage transfer rate suggests that this might be a problem of the device getting mounted by default with the sync option :

Do you have a "Devices" Icon on your Desktop?
If so, open it. If not, try opening media:/ in konqueror.

Selct the device then -> Properties -> Mounting(tag) and uncheck the Synchronous box. (This gets remembered, so you should only need to do it once per-device).

---

Some more ideas come from the bug-report Write through on external hdd making writing very slow

• you change to async at runtime by 'sudo mount -o remount,async /dev/sda1' (or whatever your mounted partition(s) is/are)
• you create an fstab entry for sda which mounts it async (this requires that the device is always plugged in at boottime)
• you change 'sync' to 'async' in the pmount source and rebuild the package

See also the discussion in the article as regarding pmount, if this applies to your system.

---

Another magic solution comes from USB 2.0 slow write but fast read, solution yet? and involves re-mount:

When I mount a USB drive on /dev/sdc1 with the following line

/dev/sdc1       /mnt/sdc1     auto    sync,noauto,user,exec     0       0

in /etc/fstab, and try to copy a 5.2 MB file to the drive from the local hard disk, then after an entire minute, 244 KB have been transferred. Copying a 29 MB file from the external drive to the local drive takes about 10 seconds.

Then if I unmount it, change the line in /etc/fstab to

/dev/sdc1       /mnt/sdc1     auto    noauto,user,exec     0       0

then copying the 29 MB file to the external disk takes an unnoticeable fraction of a second. Same with copying to the local disk from the external disk.

---

See this article and the reference to "usbtree". It tells how to check if the usb is functioning as 1.1 or 2.0 :

Linux and USB 2.0.

Answer 2

So I came to this page through a google search of the same issue ("ubuntu 15 speed up usb transfer") because my transfers were running at 75kb/s.

This was probably only the solution for my setup but my 16GB USB 3 drive was formatted to GPT fat32, I deleted all partitions, initialized to MBR, made a single NTFS partition, plugged it back into my Ubuntu PC and it's now transferring a 10GB file at over 50MB/s. For me, problem solved.

Answer 3

Access to the drive is usually buffered in async mode. The initial fast burst is likely to to memory cache. Slow speed is actual write speed.

If you have a USB stick which flashes during transfer try some tests. I found the writes can continue for quite a while after the copy is completed.

Mounting in async mode instead of sync mode allows the copy to appear to proceed faster, but doesn't give you the same security that copied data is really transferred.

Check the options in man mount. Some formats allow metadata (directory, file allocation table, etc,) updates to be deferred. This will cut down the amount of data that needs to be copied. This increases risk of data loss slightly.

Answer 4

We need to make sure that the system tries to write in program/erase block chunks. With modern flash memory they are usually 1M-4M in size. To do this you want to make sure yours FS is aligned to erase blocks, and that interface allows entire block transfer in one go (for cheap sticks). Otherwise we would get write amplification, as the system tries to write in chunks that are smaller, than erase block (doing read/mod/write) + block misalignment.

To check yours current setting do:

cat /sys/block/sd**X**/device/max_sectors

You can tune hall rules for those devices. See https://unix.stackexchange.com/questions/67719/change-value-of-usb-max-sectors-for-an-entire-family-of-devices.

In this case I had replaced max_sectors for all devices, that used default of 240 (USB storage) to 32K sectors or 2K sectors:

(use == to check, and = to assign):

On my system (Mageia 4, 3.14.24 core i7) I had to do this due terribly slow write speeds (2MB/sec) on Kingston DT101 G2 16GB:

vi /usr/lib/udev/rules.d/81-udisks_maxsect.rules and add:

SUBSYSTEMS=="scsi", ATTR{max_sectors}=="240", ATTR{max_sectors}="32678"

And the dd write speed went up 3x times :-) mc cp probably 10-20x up (after I had started first partition @8192'th sector and reformatted with 64k aligned clusters):

fdisk -u /dev/sdh (make DOS compat off if on),

mkfs.vfat /dev/sdh1 -n KINGSTON16G -s 128 **-R 4592** and use fsck.vfat -v /dev/sdh1 to check alignment (check [data start sector] should be multiple of 128 (cluster size)). Adjust number of the reserved sectors (-R) if needed.

Default max_sectors (240) seem to cause high write amplification on some of the cheap new drives. But be very careful with such high setting, the similar effect is achieved at 2048 sectors (probably 1M erase blocks:

SUBSYSTEMS=="scsi", ATTR{max_sectors}=="240", ATTR{max_sectors}="2048"

Test all yours old USB devices, that they still work well. Use vendor/model attributes in the rules files to be more specific.

PS: If using ext3/4:

Disable Journaling and set stride/stripe_width for mkfs.ext3:

mkfs.ext3 -O ^has_journal -E stride=[read page fs blocks] -E stripe-width=[erase block fs blocks]

Answer 5

I had some problems also with transfer rate on a WD external disk, after opening it in a windows SO, i always used LINUX, after that the transfer rate was like 1.5mb/s than i unmount the external hard drive, ran dmesg there it was saying that sdb1 it was unporperly unmounted, ran a fsck, that made a few repairs and after that 20mb/s of transfer rate again when copiyng from sda to external disk.
" fsck is always a risk if you have data, but it worked for me, with no data loss "

Answer 6

Check, really check, that your device is plugged into a USB port that supports full speed.

Although this may sound silly, sometimes things get mixed up. Some motherboards have high- and low-speed ports. Use a tool like usbview (note to execute it with superuser rights, sudo usbview) to verify that your device is operating at 480Mb/s (USB 2.0) or more.

For example, when you usually plug your device in via some extension cord, a hub on your display etc., it might happen - after re-pluggin cables on your deskside computer or so - that you accidentally plugged in the hub into a low speed (USB 1.) port. Then, any devices plugged into the hub will operate at low speed, while others, plugged in somewhere else, operate at high speed. Can be very confusing - and is a simple hardware thing... Happened to me once, and is not related to any USB kernel modules, mount options or so.