10

I have two supposedly identical 3TB Western Digital USB hard disk drives. One contains nearly the full 3TB of data. I want the other to be a backup of that data. There are lots of small files, so running cp would be even slower than copying 3TB over USB using dd. Both drives were bought at the same time, both use MSDOS partition table, but one shows logical/physical sector sizes of 4096/4096 and the other shows 512/4096.

My problem is that I’m trying to copy to the 512/4096 drive, but I can’t create a large enough partition because of the limits of the MSDOS partition table with 512 byte (logical) sectors. I've tried various commands (dd of the first few sectors to partition table, parted, fdisk, gparted, Windows XP “Manage” and Windows 7 “Manage”) but end up with either 2TB max limits or ~300GB partitions that thought they were 3TB when they were created.

Given that one of my drives is running at 4096/4096 then this question doesn’t hold true for my case.

Using GPT is my fall-back option, but it creates a partition that is slightly smaller than the NTFS partition, so I wouldn't be able to just dd the filesystem on the partition.

Is there a way to force 4096/4096 for an MSDOS partition table (either from Linux or Windows) so that I can create a 3TB partition that perfectly matches my source disk?

Improve this question
5
  • 1
    Couldn't you use software to simply duplicate the entire disk sector by sector? I know Acronis and CloneZilla have this ability.
    – Ramhound
    Commented Aug 20, 2012 at 15:04
  • 1
    Why can't you create a FAT32 partition with a sector size greater than 512? That's what is limiting the size of the volume. FAT32 maximum volume sizes range from 2 TB for a sector size of 512 bytes up to 16 TB for a sector size of 4,096 bytes.
    – martineau
    Commented Aug 20, 2012 at 15:18
  • 2
    @Ramhound - I've used dd if=/dev/sdb of=/dev/sdc, which is a Linux command that does sector by sector copying of an entire device (not just a partition), but the partition on the second device comes out at ~300GB despite being a perfect copy. dd is one of Clonezilla's 'last ditch' options when it can't do anything more intelligent.
    – IBBoard
    Commented Aug 20, 2012 at 18:27
  • @martineau: I'm using NTFS, because that's what the source partition was. I've mounted the drive in Win7, opened "Manage" and if it is on "MBR" partitioning (which the first disk appears to be) then even Win7 won't let me create a partition larger than 2TB. When printing the disk info in Linux using fdisk (IIRC) then "Sector size (logical/physical)" definitely differs between devices, which screws the partitions I have made.
    – IBBoard
    Commented Aug 20, 2012 at 18:31
  • Western Digital has free disk utilities for partitioning their drives -- you could try using them. Personally I use Acronis Disk Director -- a commercial program -- to do virtually all my partition manipulations.
    – martineau
    Commented Aug 20, 2012 at 19:28

4 Answers 4

Reset to default
4

I just had a similar issue. I had a Seagate 3TB USB Desktop Expansion drive that I wanted to move into my PC for faster access (I only had USB 2.0 on the PC). Once I did that I couldn't access the filesystem. Returning it to the USB to SATA controller made it work again. Comparing the two I found that in the external case (using the USB to SATA controller) the drive had 4096/4096 logical/physical sector sizes and when connected to the internal motherboard SATA, it had 512/4096 logical/physical sector sizes. Much like the OP sugguests, a 4096 logical sector size allows for > 2TiB partitions, but at the 512 sector size we get the 2TiB limit. I didn't want to copy stuff off (slowly over USB) to somewhere, repartition (GPT), reformat, and copy back, so I kept at it. My ultimate solution on Linux was to:

Delete the original MBR partition table which uses 4k sectors. Use gdisk (GPT variant of fdisk) to create the partition using 512 byte sectors spanning the exact same area. This basically amounted to taking the sectors for start/end of the partition in 4K-land and multiplying by 8. I did have an off-by one on the end, so I added 8 more there and so far so good (e2fsck alerted me to that.) The e2fsk is still going, so I'm not 100% sure I'm safe, but it seems all good in theory and I did mount the files for a bit and they looked fine.

Improve this answer
7
  • 2
    It's a GPT partition table versus a MSDOS partition table. No one mentioned filesystems. You can do a conversion from MSDOS/MBR to GPT using gdisk relatively safely.
    – justbrowsing
    Commented Sep 7, 2013 at 22:25
  • 1
    There are many other reasons to want to move to a GPT, even if you have a disk smaller than 2TiB, safety and redundancy chief among them. You may wish to read this excellent article, ibm.com/developerworks/linux/library/l-gpt/index.html
    – MariusMatutiae
    Commented Nov 7, 2013 at 7:35
  • 2
    the limit does not related to Linux or Windows. It's just because MBR doesn't support drives larger than 2TB
    – phuclv
    Commented Jan 10, 2014 at 10:57
  • 1
    I can confirm this approach works. This was my partition table after a night-long rsync and after having moved the disk from a USB-to-SATA adapter to a proper SATA connector (if you do the sums, you'll notice a perfect factor of 8). I've first tried a few times with a partial dd image and then I took the plunge. BTRFS scrub was totally happy with the values. Hope I'll never have to do this again, but it has been a didactic moment.
    – Avio
    Commented Nov 30, 2018 at 14:39
  • 1
    Converting MBR->GPT can solve the problem of accessing a >2 TB drive originally partitioned under a different sector size, but be cautioned that not all MBR drives have enough free space at the start and end for the GPT data structures: rodsbooks.com/gdisk/mbr2gpt.html Using gdisk on the block device directly in this scenario isn't safe because gdisk will have incorrect info about where the original partition data was due to the sector size mismatch & it could clobber the partition data. Maybe use losetup --sector-size to emulate the original sector size and gdisk the loop device.
    – balazer
    Commented Dec 27, 2019 at 8:15
4

Many Western Digital external USB drives over 2 TB (including at least some My Book, My Book Essential, Elements, and Easystore drives) can be configured for either 512 byte logical sectors or 4096 byte logical sectors using the WD Quick Formatter tool. When configured for 4096 byte logical sectors, the USB-to-SATA bridge in the enclosure does a translation between 512 byte logical sectors at the SATA interface to the internal drive and 4096 byte logical sectors at the USB interface to the host computer.

With 512 byte logical sectors, an MBR partition table could only use up to 2 TB of a drive. That's because MBR table entries are 32 bits with a max of 2^32 or 4,294,967,296 sectors. 2^32 sectors x 512 bytes/sector is 2 TB. With 4096 byte logical sectors, an MBR partition table can use up to 16 TB of a drive. (2^32 sectors x 4096 bytes/sector is 16 TB) Windows XP only supports MBR partition tables, and so 4096 byte logical sectors are the only way to use all of the space on a drive over 2 TB in Windows XP. The newer GPT partition table format supported in Windows Vista and later does not have the 32-bit limitation, and can support disks larger than 2 TB regardless of the sector size.

The WD Quick Formatter tool (version 2.0.0.65 available for download as of this writing) can enable or disable the sector size translation. This version of the tool does not work correctly under Windows XP, so I recommend running the tool in Windows 7 or later. When run in Windows 7, or in later version of Windows but with Windows 7 compatibility mode, the tool will present two configuration options on the "Format your WD external drive" screen:

WD Quick Formatter "Format your WD external drive" screen

  • XP Compatible: This option configures the drive for 4096 byte logical sectors, and creates an MBR partition table.
  • Most Compatible (Vista or later required): This option configures the drive for 512 byte logical sectors, and creates a GPT partition table.

Both options also create a single partition filling the entire drive, and quick format it in NTFS.

If you run the tool in Windows 8 or later without putting it in Windows 7 compatibility mode, the tool will not present a compatibility option on the formatting screen and it will format in the "most compatible" mode (512 byte logical sectors).

WD Quick Formatter 2.0.0.65 doesn't work correctly in Windows XP: it successfully configures 4096 byte logical sectors, but fails to format correctly. Another tool can subsequently be used to partition and format the drive.

WD Quick Formatter 1.2.0.10 works correctly in Windows XP, but is not available for download from Western Digital anymore.

When the target drive is configured for the same logical sector size as the source, you can copy to it using dd and the copy will work without any need to alter the partition table.

I can confirm that these Western Digital external USB drives can be configured for XP compatibility:

  • Easystore 14 TB (WDBCKA0140HBK)

  • Easystore 12 TB (WDBCKA0120HBK)

  • Easystore 8 TB (WDBCKA0080HBK)

  • My Book Essential 1140, 3 TB (WDBACW0030HBK)

  • Elements 1021, 3 TB (WDBAAU0030HBK)

Improve this answer
8
  • This is a great and highly overrated answer. I was especially curious about the numbers, in particular how 2^32 translates to a 2TB limit.
    – Hashim Aziz
    Commented Dec 25, 2019 at 15:59
  • 1
    That's just for external drives though (the quick formatter is in fact probably just adjusting the bundling feature in the USB bridge rather than the disk itself). For internal ones, while everybody and their cousin seems to have a tool (from Seagate and HGST, to Intel) WDC seems to leave you in the lurch.
    – mirh
    Commented Jan 11, 2021 at 2:04
  • This is high level formatting NOT low level. ./smartctl -i /dev/sdc will still say: 512 bytes logical, 4096 bytes physical
    – Zibri
    Commented Jan 19, 2023 at 17:04
  • Interesting. Do you know if WD formatter changes some firmware "flag" in enclosure or disk itself? I guess in enclosure, but maybe you know for sure.
    – Alex Martian
    Commented May 31, 2023 at 16:22
  • @HashimAziz, 2^32 * 512 bytes (one of typical sector sizes) = 2 TiB. Sector size is reported by SATA interface (or USB one) per superuser.com/a/1478374/607929.
    – Alex Martian
    Commented May 31, 2023 at 16:24
2

Your issue is likely similar to How to correct 512-byte sector MBR on a 4096-byte sector disk?. In short, it's the limitation of the USB enclosure's firmware by not understanding the 4K sector format

The physical sector size is the HDD's native sector size, so there's no way to "force" it to change that value unless you find someway to update the drive's firmware and do a low-level format

Some (old) 4K HDD may support emulated 512-byte sector mode (called 512e, which is what your 512/4096 drive does) in which physical size is still 4K, but it reports as having a 512-byte sector which is the logical size. That means partitions must be aligned properly to 4K or performance will be affected greatly. Technically that reported logical value may be changed by some disk-configuring software or by updating firmware if the manufacturer supports.

However I'm not aware of any such software. If you see different logical values for the same disk then it's because the old disk controller in the external enclosure doesn't understand the new 512e or 4K format. That happens a lot to USB HDDs, like mentioned in Shane Anderson's answer and:

In fact many old USB enclosures don't even support drives larger than 2TB

Worse yet, some enclosures do the reverse thing by combining 512-byte sectors to a 4K one

Complicating matters is certain USB disk enclosures. Some of these enclosures do the reverse of what AF does: They take eight disk sectors and bundle them into one new 4096-byte sector. I'm not sure what the reasoning is behind this move, but one practical advantage is that disks larger than 2TiB can be used with the old MBR partitioning system. One major disadvantage is that a disk partitioned in one of these enclosures can not be used directly or in an enclosure that doesn't do this type of translation. Likewise, a disk prepared without this translation can't be used when it's transferred into such an enclosure.

How to correct 512-byte sector MBR on a 4096-byte sector disk?

Similar issues:

Besides, you shouldn't use dd for copying disks. Instead, use some disk cloning tools, they'll smart enough to skip blank clusters and are also capable of cloning between disks with different sizes

Also, there's no "DOS partition table" but MBR and GPT (and some other uncommon formats). Unfortunately gparted and many Linux tools call that "msdos" which is incorrect

With MBR you can only addressed 232 sectors (2TB with 512-byte sector and 16TB with 4K sector). That's why your 4096/4096 drive works properly. I suppose Shane Anderson's workaround works, but if it doesn't the only way is to use GPT. In fact GPT would be recommended even if you have only one partition because it supports checksumming and a backup partition table, which significantly reduces the chance of accidental corruption/deletion

Improve this answer
4
  • 1
    If there's no way to force it, how did it change? That is the crux of the problem. An identical disk (make, model, etc) shows 4K sectors. The 512b sector disk used to report the correct size, but now reports a much smaller size. /Something/ changed, so unless it is a hardware failure of some intermediary level that faked the size then it should be possible to change it back. As for the "msdos" partition table - tell that to GParted, because that's what they give you as an option. I used dd because the disk was 95% full, so ghosting apps won't be much more efficient.
    – IBBoard
    Commented Jan 18, 2014 at 12:02
  • it doesn't change. It's the USB disk controller that reports a different value. Disk cloning tools may not be much more efficient, but they can handle the difference in partition sizes gracefully. See my edit
    – phuclv
    Commented Sep 8, 2019 at 2:05
  • Thanks for the edits, @phuclv. The link may be relevant and misaligned partitions could have caused the change in reported sector sizes, but I don't think the other edits would have solved my specific problem (the disks are long gone, so I can't check). My issue was two supposedly identical disks, both on USB, not one disk in two caddies (where you might expect firmware to change).
    – IBBoard
    Commented Sep 8, 2019 at 9:20
  • Take note that there may be even more to consider than just the "currently reported" sector size. At least when handling a NTFS partition, that is also determining the MFT record size Windows expects. If that doesn't match the scheme already on disk the boot sector will result corrupted.
    – mirh
    Commented Sep 6, 2020 at 16:10
-2

Yes and no.

You cannot change the physical sector size, and few drives will let you change the transport layer sector size, so you will always be writing a multiple of 4K at a time.

However, Logical sector size has nothing to do with physical sector size, and the Operating system can handle such conversions by combining writes.

The drive will ALWAYS be accessed 4096 bytes at a time. That said, any modern operating system's block device driver can convert between sector sizes, better known these days as Block Sizes, and thus partition block sizes don't have to match the block size of the underlying device.

You can easily use 4K logical sector partitions on a 512 byte logical sector device, or a 512 byte logical sector partition on a 4K logical sector device.

Additionally, partition level logical sector does not have to match the sector block size used in the filesystem.

Linux and by extension Android, uses Block size (which is filesystem level logical sector sizes) of 1024, 2048 or 4096 on ExtFS. (On 512 byte sector drives it will use multiple physical sectors to form a logical block).

On FAT32, DRDOS 6.x supported larger disks by using 8K logical Sectors, and Linux can use still larger logical sectors. This allows much larger FAT32 partitions to be created.

Similarly, many software programs when managing very small filesystems that contain a large number incredibly small files have used programs with logical sector sizes as small as 64KB.

References:

Linux MSDOSTOOLS man pages

Linux mke2fs man page

Wikipedia pages on FAT:

https://en.wikipedia.org/wiki/File_Allocation_Table

https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system

Improve this answer
5
  • This answer is mostly wrong and would seem to confuse a drive's logical sector sizes with a filesystem's allocation unit size (a.k.a. cluster size or block size). The question is asking about a drive's logical sector size, and not about a filesystem's block size. A drive's logical sector size is not under the control of the operating system.
    – balazer
    Commented Dec 26, 2019 at 22:22
  • No, BLOCK conversion handling is done on the DEVICE DRIVER layer, not the FILESYSTEM layer. It is NOT the same as Clusters, which are the Filesystem layer. Read my citations. Logical blocks are what the device driver reports to the filesystem driver, clusters or filesystem blocks are what the filesystem manages on it's formatting. They are completely different things.
    – Robert Wm Ruedisueli
    Commented Dec 27, 2019 at 1:08
  • Here is another citation from a quick google: thomas-krenn.com/en/wiki/…
    – Robert Wm Ruedisueli
    Commented Dec 27, 2019 at 1:11
  • Alright, then how do you enable block conversion? You said it's easy but you didn't say how. The question is asking how to change a target drive with 512 byte logical sectors to have 4096 byte logical sectors, so an MBR-partitioned source drive with 4096 byte logical sectors can be copied to the target and used in Windows.
    – balazer
    Commented Dec 27, 2019 at 6:13
  • I already told you. It's done automatically. The formatting utility can specify the block size. If it's set to a different size, it's converted by the OS. There is nothing to enable.
    – Robert Wm Ruedisueli
    Commented Jan 10, 2020 at 20:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .