Hello,
I've just bought a new 3TB hard disk (Verbatim Store 'n' Save), and I sadly find out that my system only sees 2.2 TB (2199 GB).
Of course I made a GPT and I have GPT support in the kernel.
The way I have always done >2TB partitions is (but never used in the system I'm refering to here):

But this time, as I said, it fails (only 2.2TB detected).
Now, I'm asking myself if it is a problem with BIOS. Actually I was looking at the BIOS (with hard disk plugged in and on), but couldn't find any relevant option to enable. Perhaps I need a BIOS update, but as I don't have experience with this stuff I'm a bit reluctant.

Could you please help me to determine what is the source of the problem?
As I said, if I can avoid it, I would prefer to not update the BIOS. Unless it is stricly necessary.
Any insight on this is appreciated.
TIA.

Regards_________________Computers are like air conditioners, they stop working properly if you open Windows

Last edited by majoron on Sun Sep 29, 2013 8:43 pm; edited 1 time in total

Some cheaper/older usb<->sata link have this too. Including older RAID cards. I'm sorry but I'm afraid it's the board and not the os or disk.

Thanks for your comment.
If you are right, I'm a bit worried. The BIOS update offered by Asus (the manufacturer of my motherboard) doesn't seem to include any change relevant for my problem, unless "Improve memory compatibility" implies some sort of solution._________________Computers are like air conditioners, they stop working properly if you open Windows

I calculated that the available space for a 3 TB disk should be 2.59 TiB (if you exclude reserved blocks). Anyway you can check the total (unformatted) disk size with parted. Assuming /dev/sda is your disk:

I calculated that the available space for a 3 TB disk should be 2.59 TiB (if you exclude reserved blocks). Anyway you can check the total (unformatted) disk size with parted. Assuming /dev/sda is your disk:

Code:

parted /dev/sda unit TiB print

It should return a value of around 2.73 TiB.

I agree.
I expect a raw size of 2794 GiB. And I've been reported by parted that the disk is 2199 GiB large (if I remember correctly).
Now, I'm sorry if the words in my original post are misleading (or if my usage of the units was sloppy), but what I'm talking about in this thread is getting the extra 595 GiB (2794-2199) that are physically in my device.

Or are you suggesting to forget about those 595 GiB and, for the sake of safety, do not update the BIOS?

Regards_________________Computers are like air conditioners, they stop working properly if you open Windows

You may need to install the "gptfdisk" package to run the second command. Change "/dev/sdc" to whatever the disk's device filename is. Also, please specify whether this is an internal or an external disk, and what type of interface it uses (PATA, SATA, eSATA, USB, etc.).

The motherboard's firmware (BIOS or EFI) should not be limiting how the disk is "perceived" by Linux, because Linux accesses the disk directly, without involving the firmware. This could be a parted bug, a limitation in an interface (USB interfaces are particularly prone to this sort of thing), or incorrect setting of the Host Protected Area (HPA) feature on the disk.

You may need to install the "gptfdisk" package to run the second command. Change "/dev/sdc" to whatever the disk's device filename is. Also, please specify whether this is an internal or an external disk, and what type of interface it uses (PATA, SATA, eSATA, USB, etc.).

The motherboard's firmware (BIOS or EFI) should not be limiting how the disk is "perceived" by Linux, because Linux accesses the disk directly, without involving the firmware. This could be a parted bug, a limitation in an interface (USB interfaces are particularly prone to this sort of thing), or incorrect setting of the Host Protected Area (HPA) feature on the disk.

Thank you for the interesting answer.

It is an external hard disk. The interface is USB. In principle it should be compatible with USB 1.1, 2.0 and 3.0 (or that is said in the docs).

BTW, the disk comes with a warning. It is said that the unit is splitted into two pieces because some OSs do not support >2TB partitions. However I didn't see a second partition never.

I don't have experience with multi-terabyte hard disks, but maybe you should check for a second device (such as /dev/sdd) instead of a second partition. It may be helpful to use the following command after inserting the hard disk:

Code:

watch -n 1 "dmesg | tail"

EDIT: Also it seems you have an HPA which is not visible to the operating system. This howto could be useful for disabling it. I would back up any data before trying that.

This last info is odd. buggy kernel driver?. Am I missing some kernel option? Defective disk?

This has my alarm bells ringing. My suspicion is that the manufacturer has done something weird with the USB interface in order to work around the 2TiB limit of MBR.

fturco wrote:

majoron wrote:

BTW, the disk comes with a warning. It is said that the unit is splitted into two pieces because some OSs do not support >2TB partitions. However I didn't see a second partition never.

I don't have experience with multi-terabyte hard disks, but maybe you should check for a second device (such as /dev/sdd) instead of a second partition. It may be helpful to use the following command after inserting the hard disk:

This is worth trying -- just doing "ls /dev/sd?" will show you all your disk devices. You can try unplugging the disk, issuing that command, plugging it back in again, waiting a few seconds, and then issuing the command again to see if more than one device shows up.

If that doesn't help, or if it does and you want to use the disk as a single unit, I recommend returning it to the store and buying another one. If I'm right and the manufacturer is doing weird things in its firmware, then the odds are that it will continue to cause problems in the future. Historically, products that try to be "clever" in this sort of way end up being more trouble than they're worth. It's better to use something that just presents itself as a full-sized drive, which Linux can handle just fine, even if, say, Windows XP couldn't use it.

BTW, the disk comes with a warning. It is said that the unit is splitted into two pieces because some OSs do not support >2TB partitions. However I didn't see a second partition never.

I don't have experience with multi-terabyte hard disks, but maybe you should check for a second device (such as /dev/sdd) instead of a second partition. It may be helpful to use the following command after inserting the hard disk:

Code:

watch -n 1 "dmesg | tail"

Nope. Only /dev/sdc and /dev/sdc1 appear (of course, sdc1 is the partition I made).
However, another devide shows up: /dev/sg3, which I doubt is another hard disk (I tried to open it with parted and fdisk, and failed).

Quote:

EDIT: Also it seems you have an HPA which is not visible to the operating system. This howto could be useful for disabling it. I would back up any data before trying that.

I'm still investigating this. I'm inclined to think that this is what is going on, or some sort of variation, as srs5694 said.
However, I read in the man page of hdparm the following (when talking about the -N option):

Quote:

Host Protected Area (HPA). This area is often used by computer makers to hold diagnostic software,
and/or a copy of the originally provided operating system for recovery purposes. Another possible use is to hide the true capacity of a
very large disk from a BIOS/system that cannot normally cope with drives of that size (eg. most current {2010} BIOSs cannot deal with
drives larger than 2TB, so an HPA could be used to cause a 3TB drive to report itself as a 2TB drive).

So I'm afraid that
1) I need to remove the HPA (which I still couldn't do, or that is what I think), and
2) most likely update the BIOS anyway because my motherboard was bought in 2010 and actually I think it is older than that.

At least this is what I understood. Other insights?_________________Computers are like air conditioners, they stop working properly if you open Windows

Last edited by majoron on Thu Sep 26, 2013 8:11 pm; edited 1 time in total

This last info is odd. buggy kernel driver?. Am I missing some kernel option? Defective disk?

This has my alarm bells ringing. My suspicion is that the manufacturer has done something weird with the USB interface in order to work around the 2TiB limit of MBR.

fturco wrote:

majoron wrote:

BTW, the disk comes with a warning. It is said that the unit is splitted into two pieces because some OSs do not support >2TB partitions. However I didn't see a second partition never.

I don't have experience with multi-terabyte hard disks, but maybe you should check for a second device (such as /dev/sdd) instead of a second partition. It may be helpful to use the following command after inserting the hard disk:

This is worth trying -- just doing "ls /dev/sd?" will show you all your disk devices. You can try unplugging the disk, issuing that command, plugging it back in again, waiting a few seconds, and then issuing the command again to see if more than one device shows up.

If that doesn't help, or if it does and you want to use the disk as a single unit, I recommend returning it to the store and buying another one. If I'm right and the manufacturer is doing weird things in its firmware, then the odds are that it will continue to cause problems in the future. Historically, products that try to be "clever" in this sort of way end up being more trouble than they're worth. It's better to use something that just presents itself as a full-sized drive, which Linux can handle just fine, even if, say, Windows XP couldn't use it.

Thank's for the reply. I'm afraid that the option of returning the disk is becoming more solid. In that case the questions would be, what else to buy? and, will my motherboard support it?_________________Computers are like air conditioners, they stop working properly if you open Windows

and your comment about the disk being split into two gives me a pointer.

Your drive my be programmed to lie about it real size by having the missing space allocated to a Host Protected Area, which is what the HPA reference in your message above is. This is, or was, a small area at the end of a drive, hidden from the operating system, often used for a system restore area. Special software is needed to access it, as the Host cannot see the space - hence Host Protected Area.

It should be listed in dmesg if its there. If you do have a HPA, you may use Linux to remove it to get all of your space back.
Put your dmesg onto a pastebin site and post a link if you want us to review it.

Nope. Only /dev/sdc and /dev/sdc1 appear (of course, sdc1 is the partition I made).
However, another devide shows up: /dev/sg3, which I doubt is another hard disk (I tried to open it with parted and fdisk, and failed).

If there's a /dev/sg3, there should also be a /dev/sg, which would be the part of the "disk" that holds the partition table. Ordinarily, /dev/s? devices (/dev/sda, /dev/sdb, and so on) point to a whole disk, whereas /dev/s?# devices (/dev/sda7, /dev/sdb2, etc.) point to partitions.

Quote:

1) I need to remove the HPA (which I still couldn't do, or that is what I think), and

Adjusting the HPA might work, and it might be worth trying this; however, HPA is a feature that's built into the hard disk itself. In order to manipulate the HPA, you've got to be able to send the appropriate commands to the disk, via hdparm. The problem is that USB disk interfaces add a layer to the interactions between the OS and the disk, and I don't know offhand what is typically done with the HPA data, much less what your specific enclosure does. Adjusting the HPA values might work; or it might do nothing; or it might do something weird, up to the point of rendering the disk unusable. If you plan to return the disk to the store if nothing else works, taking the risk might be worthwhile, and is even ethically justified IMHO. (The manufacturer has provided something non-standard and non-functional, and they'll need to deal with the return at the factory anyhow. They almost certainly know how to correct any issues that an adjustment of the HPA might cause.)

Quote:

2) most likely update the BIOS anyway because my motherboard was bought in 2010 and actually I think it is older than that.

The BIOS is a red herring. BIOS limitations are important only in the early stages of booting from a hard disk; once Linux takes control, the kernel communicates directly with the disk, bypassing the BIOS and any limitations it might have. Even if you're booting from the disk (which I'm guessing you're not, since it's an external disk), Linux can cope fine by putting a small /boot partition near the start of the disk, which won't be affected by the BIOS limitations.

To be clear: Your problem is not a motherboard BIOS issue, and upgrading your motherboard's BIOS will not fix the problem.

I've never before heard of a motherboard's BIOS setting the HPA on a hard disk unbidden. Of course, that's no guarantee that it never happens; my knowledge is necessarily limited. My hunch is that if it does happen this way, the BIOS would be setting the HPA on RAIDed disks as a way to store RAID metadata. (In this sense, the setting of HPA data wouldn't really be "unbidden;" it would be implicit in the request to set up the disk for use in a RAID array.) If I'm right, it's unlikely that your single external USB disk would have an HPA set for such a purpose; it's more likely that the external enclosure's firmware would be setting the HPA as a way of splitting a single over-2TiB disk into two virtual disks to help OSes that can't handle over-2TiB disks. This is particularly true because you paraphrased a notice from the disk's manufacturer to the effect that it's split into two parts. Overall, the evidence points strongly toward the enclosure's USB interface firmware being the culprit, not your motherboard's BIOS.

It can also be that the SATA/USB interface in the enclosure is brain dead and is only sending the 32bit LBA data to the PC.
Thats game over, the most that can hold is 4G sectors, or your 2.2Tb.
The 48bit LBA data is sent in a different location in the response to the identify command.

What does lsusb say about your drive/enclosure?
In particular, the vendor and device ID._________________Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

It can also be that the SATA/USB interface in the enclosure is brain dead and is only sending the 32bit LBA data to the PC.
Thats game over, the most that can hold is 4G sectors, or your 2.2Tb.
The 48bit LBA data is sent in a different location in the response to the identify command.

What does lsusb say about your drive/enclosure?
In particular, the vendor and device ID.

Thats just silly - even WinXP has an update to deal with HDD bigger than the MSDOS Partition Table Limit.
Its called NTFS.

NTFS is normally used on partitioned devices, and if the partitioning system in question is MBR, NTFS will be limited to 2TiB partitions. That said, it is possible to use NTFS on an unpartitioned disk, which would enable it to be used on larger disks. I don't know offhand how Windows XP would cope with this, though. NTFS can also, of course, be used on GPT-partitioned disks, but most versions of Windows XP can't handle GPT.

As a practical matter, some USB drive enclosures convert 512-byte sectors to 4096-byte sectors. This raises the MBR limit from 2TiB to 16TiB. Some BIOSes can't boot from such disks, though, and the disk will become unreadable if it's removed from the enclosure and connected directly to a disk. (I've even heard of enclosures with both USB and eSATA interfaces, which present the disk differently depending on which interface is used. That's a serious problem, of course.)

Quote:

It would be good to know the vendor and device IDs, so we know exactly whats in the USB enclosure.

I agree this might be helpful, since it would enable doing some Internet searches, or checking vendor technical documents, to discover what's actually going on with the specific disk in question.