| File Allocation Table |
Article Index for File |
Website Links For File |
Information AboutFile Allocation Table |
| CATEGORIES ABOUT FILE ALLOCATION TABLE | |
| disk file systems | |
| dos on ibm pc compatibles | |
| windows disk file systems | |
| windows components | |
|
The FAT file system is relatively uncomplicated, and is supported by virtually all existing Operating System s for Personal Computer s. This ubiquity makes it an ideal format for Floppy Disk s and Solid-state Memory Card s, and a convenient way of sharing data between disparate operating systems installed on the same computer (a Dual Boot environment). The most common implementations have a serious drawback in that when files are deleted and new files written to the media, directory fragments tend to become scattered over the entire media, making reading and writing a slow process. Defragmentation is one solution to this, but is often a lengthy process in itself and has to be performed regularly to keep the FAT file system clean. __TOC__ HISTORY The FAT filesystem was created by . The name originates from the usage of a table which centralizes the information about which areas belong to files, are free or possibly unusable, and where each file is stored on the disk. To reduce the management complexity, disk space is allocated to files in contiguous groups of hardware sectors called '' Cluster s''. The maximum possible number of clusters has dramatically increased over time, and the number of bits required to identify a cluster is used to name the successive major versions of the format. The FAT standard has also evolved in several ways where backward compatibility with existing software has been preserved. FAT12 This initial version of FAT is now referred to as FAT12. As a filesystem for floppy disks, it had a number of limitations: cluster addresses were "only" 12 bits long (which limited cluster count to 4096 and made FAT manipulation a bit tricky) and the disk size was stored as a 16-bit count of Sector s, which limited the size to 32 MB. FAT12 was used by several manufacturers with different physical formats but a typical floppy diskette at the time was 5.25", single-sided, 40 Track s, with 8 sectors per track, resulting in a capacity of slightly less than 160 KB. The FAT12 limitations exceeded this capacity by one or more orders of magnitude, the limits were successively lifted in the following years which increased storage capacity dramatically but eventually rendered FAT12 obsolete. By convention all the control structures were organised to fit inside the first track, thus avoiding head movement during read and write operations, although this varied depending on the manufacturer and physical format of the disk. Since when FAT12 was introduced DOS had no support for hierarchical directories the maximum number of files was typically limited to a few dozen. A limitation which was not addressed until much later was that any bad sector in the control structures area, track 0, could prevent the diskette from being usable. The DOS formatting tool rejected such diskettes completely. Bad sectors were only allowed in the file area, where they made the entire holding cluster unusable as well. Directories In March 1983, IBM launched the PC XT Computer , which featured a 10 MB hard disk. MS-DOS/PC-DOS 2.0 was released simultaneously, and introduced hierarchical directories to properly support the "massive" capacity the new medium provided. Apart from allowing for better organization of files, directories allowed it to store many more files on the hard disk, as the maximum number of files was no longer constrained by the (still fixed) root directory size. This number could now be equal to the number of clusters (or even greater, given that zero-sized files do not use any clusters on FAT). The format of the FAT itself did not change. The 10 MB hard disk on the PC XT had 4 KB clusters. If a 20 MB hard disk was later installed, and formatted with MS-DOS 2.0, the resultant cluster size would be 8 KB, the boundary at 15.9 MiB. Initial FAT16 In 1984 IBM released the PC AT , which featured a 20 MB hard disk. Microsoft introduced MS-DOS 3.0 in parallel. Cluster addresses were increased to 16-bit, allowing for a greater number of clusters (up to 65,517) and consequently much greater file-system sizes. However, the maximum possible number of sectors and the maximum (partition, rather than disk) size of 32 MB did not change. Therefore, although technically already "FAT16", this format was not yet what today is commonly understood under this name. A 20 MB hard disk formatted under MS-DOS 3.0 was not accessible by the older MS-DOS 2.0. Of course, MS-DOS 3.0 could still access MS-DOS 2.0 style 8 KB cluster partitions. MS-DOS 3.0 also introduced support for high-density 1.2 MB 5.25" diskettes, which notably had 15 sectors per track, hence more space for FAT. This probably prompted a dubious optimization of the cluster size, which went down from 2 sectors to just 1. The net effect was that high density diskettes were significantly slower than older ''double density'' ones. Extended partition and logical drives Apart from improving the structure of the FAT filesystem itself, a parallel development allowing an increase in the maximum possible FAT storage space was the introduction of multiple FAT partitions. Originally partitions were supposed to be used only for sharing the disk between operating systems, typically DOS and Xenix at the time, so DOS was only prepared to handle one FAT partition. It was not possible to create multiple DOS partitions using DOS tools, and third party tools would warn that such a scheme would not be compatible with DOS. Simply allowing several identical-looking DOS partitions could lead to naming problems: should C: be the first FAT partition on disk, for simplicity, or rather the partition marked as ''active'' in the partition table, so that several DOS versions can co-exist? And which partition should be C: if the system was booted from a diskette? To allow the use of more FAT partitions in a compatible way, a new partition type was introduced (in MS-DOS 3.2, January 1986), the '' Extended Partition ''; which was actually just a container for additional partitions called ''logical drives''. Originally only 1 logical drive was possible, allowing the use of hard disks up to 64 MB. In MS-DOS 3.3 (August 1987) this limit was increased to 24 drives; it probably came from the compulsory letter-based disk naming (A and B being reserved for the two floppy drives). The logical drives were described by on-disk structures which closely resemble the Master Boot Record (MBR) of the disk (which describes the ''primary'' partitions), probably to simplify coding. Though some believe these partitions were ''nested'' in a way analogous to Russian Matryoshka Doll s, that wasn't the case. They were always stored on disk like a row of separate blocks within a single box; these blocks are often referred to as being ''chained'' together, by the ''links'' in their EBR sectors. Only one extended partition was allowed. Logical drives were not bootable, and the extended partition could only be created after the primary FAT partition (except with third party formatting tools), which removed all ambiguity, but also the possibility of booting several DOS versions from the same hard disk. A useful side-effect of the extended partition scheme was to significantly increase the maximum number of partitions possible on a PC hard disk, beyond the 4 which could be described by the MBR alone. Prior to the introduction of extended partitions, some hard disk controllers (which at that time were separate option boards, since the IDE standard did not yet exist) could make large hard disks appear as two separate disks. Final FAT16 Finally in November 1987, Compaq DOS 3.31 introduced what is today called the ''FAT16'' format, with the expansion of the 16-bit disk sector index to 32 bits. The result was initially called the ''DOS 3.31 Large File System''. Although the on-disk changes were apparently minor, the entire DOS disk code had to be converted to use 32-bit sector numbers, a task complicated by the fact that it was written in 16-bit Assembly Language . In 1988 the improvement became more generally available through MS-DOS 4.0 and OS/2 1.1. The limit on partition size was now dictated by the 8-bit signed count of sectors-per-cluster, which had a maximum power-of-two value of 64. With the usual hard disk sector size of 512 bytes, this gives 32 KB clusters, thereby fixing the "definitive" limit for the FAT16 partition size at 2 Gibibyte s. On Magneto-optical media, which can have 1 or 2 KiB sectors, the limit is proportionally greater. Much later, Windows NT increased the maximum cluster size to 64 KB by considering the sectors-per-cluster count as unsigned. However, the resulting format was not compatible with any other FAT implementation of the time, and it generated massive Internal Fragmentation . Windows 98 also supported reading and writing this variant, but its disk utilities didn't work with it. The number of root directory entries available is set at formatting time, and is stored in a 16 bit signed field setting an absolute limit of 32767 entries (32736, a multiple of 32, in practice). For historical reasons, FAT12 and FAT16 media generally use 512 root directory entries on non-floppy media, and other sizes may be incompatible with some software or devices (entries being file and/or folder names in the old 8.3 format).2 Some third party tools like mkdosfs allow the user to set this parameter.3 Long File Names (VFAT, LFNs) One of the " User Experience " goals for the designers of Windows 95 was the ability to use Long Filename s (LFNs - up to 255 UTF-16 characters long), in addition to classic 8.3 Filename s. LFNs were implemented using a Work-around in the way directory entries are laid out (see below). The version of the file system with this extension is usually known as VFAT after the Windows 95 VxD device driver, also known as "Virtual FAT" in Microsoft's old document. Interestingly, the VFAT driver actually appeared before Windows 95, in Windows For Workgroups 3.11, but was only used for implementing 32-bit File Access , a higher performance Protected Mode file access method, bypassing DOS and directly using either the BIOS , or, better, the Windows-native protected mode disk drivers. In Windows NT, support for long filenames on FAT started from version 3.5 . OS/2 added long filename support to FAT using Extended Attributes (EA) before the introduction of VFAT; thus, VFAT long filenames are invisible to OS/2, and EA long filenames are invisible to Windows. FAT32 In order to overcome the volume size limit of FAT16, while still allowing DOS real-mode code to handle the format without unnecessarily reducing the available Conventional Memory , Microsoft decided to implement a newer generation of FAT, known as FAT32, with cluster counts held in a 32-bit field, of which 28 bits are currently used. In theory, this should support a total of approximately 268,435,456 (228) clusters, allowing for drive sizes in the range of 8 Terabyte s with 32K clusters, but the boot sector uses a 32 bit field to limit volume size to 232 sectors (2TB on a hard disk with 512 byte sectors). See this article by Raymond Chen . On Windows 95/98, due to the version of Microsoft's ScanDisk utility included with these operating systems being a 16-bit application, the FAT structure is not allowed to grow beyond 4,177,920 (< 222) clusters, placing the volume limit at 127.53 Gigabyte s.4. A limitation in original versions of Windows 98/98SE's Fdisk causes it to incorrectly report disk sizes over 64GB.5 A corrected version is available from Microsoft. These limitations do not apply to Windows 2000/XP except during Setup, in which there is a 32GB limit.6 Windows ME supports the FAT32 file system without any limits.7 FAT32 was introduced with Windows 95 OSR2, although reformatting was needed to use it, and DriveSpace 3 (the version that came with Windows 95 OSR2 and Windows 98) never supported it. Windows 98 introduced a utility to convert existing hard disks from FAT16 to FAT32 without loss of data. In the NT line, native support for FAT32 arrived in Windows 2000 . A free FAT32 driver for Windows NT 4.0 was available from Winternals , a company later acquired by Microsoft. Since the acquisition the driver is no longer officially available. Windows 2000 and 's opinionNorton, Peter (2002); ''Peter Norton's New Inside the PC'', Sams Publishing, ISBN 0-672-32289-7 (p. 428: “Microsoft has intentionally crippled the FAT32 file system”) is that “Microsoft has intentionally crippled the FAT32 file system.” The maximum possible size for a file on a FAT32 volume is 4 GB minus 1 Byte (232−1 bytes). For most users, this has become the most nagging limit of FAT32 as of 2007 , since video capture and editing applications and some other software can easily exceed this limit. Most new Windows machines now ship with NTFS and thus avoid these problems, but until mid-2006, those who run dual boot systems or who move external data drives between computers with different operating systems had little choice but to stick with FAT32. Since then, full support for NTFS has become available in Linux and many other operating systems, by installing the FUSE Library (on Linux) together with the NTFS-3G application. Data exchange is also possible between Windows and Linux by using the Linux-native Ext2 or Ext3 file systems through the use of external drivers for Windows, such as ext2 IFS ; however, Windows cannot boot from ext2 or ext3 partitions. Fragmentation The FAT filesystem does not contain mechanisms which prevent newly written files from becoming scattered across the partition. Other filesystems, like HPFS, use free space bitmaps that indicate used and available clusters, which could then be quickly looked up in order to find free contiguous areas (improved in ExFAT ). Another solution is the linkage of all free clusters into one or more lists (as is done in Unix filesystems). Instead, the FAT has to be scanned like an array in order to find free clusters, which can lead to performance penalties with today's large hard disks and consequently their large FAT sizes. In fact, computing free disk space on FAT is one of the most resource intensive operations, as it requires reading the entire FAT linearly. A possible justification suggested by Microsoft's Raymond Chen for limiting the maximum size of FAT32 partitions created on Windows was the time required to perform a simple "DIR" operation, which always displays the free disk space as the last line. Displaying this line took longer and longer as the number of clusters increased. The High Performance File System (HPFS) divides disk space into ''bands'', which have their own free space bitmap, where multiple files opened for simultaneous write could be expanded separately. Some of the perceived problems with fragmentation resulted from operating system and hardware limitations. The single-tasking DOS and the traditionally single-tasking PC hard disk architecture ( Only 1 Outstanding Input/output Request At A Time , No DMA Transfers ) did not contain mechanisms which could alleviate fragmentation by asynchronously prefetching next data while the application was processing the previous chunks. Similarly, write-behind caching was often not enabled by default with Microsoft software (if present) given the problem of data loss in case of a crash, made easier by the lack of hardware protection between applications and the system. MS-DOS also did not offer a system call which would allow applications to make sure a particular file has been completely written to disk in the presence of deferred writes (cf. Fsync in Unix or DosBufReset in OS/2 ). Disk caches on MS-DOS were operating on disk block level and were not aware of higher-level structures of the file system. In this situation, cheating with regard to the real progress of a disk operation was most dangerous. Modern operating systems have introduced these optimizations to FAT partitions, but optimizations can still produce unwanted artifacts in case of a system crash. A Windows NT system will allocate space to files on FAT in advance, selecting large contiguous areas, but in case of a crash, files which were being appended will appear larger than they were ever written into, with dozens of random kilobytes at the end. With the large cluster sizes, 16 or 32K, forced by larger FAT32 partitions, the ''external'' fragmentation becomes somewhat less significant, and ''internal'' fragmentation, ie. disk space waste (since files are rarely exact multiples of cluster size), starts to be a problem as well, especially when there are a great many small files. Third party support The alternative IBM PC operating systems—such as Linux , FreeBSD , BeOS and JNode —have all supported FAT, and most added support for VFAT, FAT32, JFAT shortly after the corresponding Windows versions were released. Early Linux distributions also supported a format known as UMSDOS , which was FAT with Unix file attributes (such as long file name and access permissions) stored in a separate file called “--linux-.---”. UMSDOS fell into disuse after VFAT was released and is not enabled by default in Linux Kernel s from version 2.5.7 onwards.8 The Mac OS X operating system also supports the FAT filesystems on volumes other than the Boot Disk . The Amiga supports FAT through the CrossDOS filesystem. FAT and Alternate Data Streams The FAT filesystem itself is not designed for supporting Alternate Data Streams ( ADS ), but some operating systems that heavily depend on them have devised various methods for handling them in FAT drives. Such methods either store the additional information in extra files and directories (Mac OS), or give new semantics to previously unused fields of the FAT on-disk data structures (OS/2 and Windows NT). The second design, while presumably more efficient, prevents any copying or backing-up of those volumes using non-aware tools; manipulating such volumes using non-aware disk utilities (e.g. defragmenters or CHKDSK ) will probably lose the information. Mac OS using PC Exchange stores its various dates, file attributes and long filenames in a Hidden File called FINDER.DAT, and Resource Forks (a common Mac OS ADS) in a subdirectory called RESOURCE.FRK, in every directory where they are used. From PC Exchange 2.1 onwards, they store the Mac OS long filenames as standard FAT long filenames and convert FAT filenames longer than 31 characters to unique 31-character filenames, which can then be made visible to Macintosh applications. Mac OS X stores metadata (Resource Forks, file attributes, other ADS) in a hidden file with a name constructed from the owner filename prefixed with "._", and Finder stores some folder and file metadata in a hidden file called " .DS_Store ". OS/2 heavily depends on Extended Attribute s (EAs) and stores them in a hidden file called "EA DATA. SF" in the root directory of the FAT12 or FAT16 volume. This file is indexed by 2 previously reserved bytes in the file's (or directory's) directory entry. In the FAT32 format, these bytes hold the upper 16 bits of the starting cluster number of the file or directory, hence making it difficult to store EAs on FAT32. Extended attributes are accessible via the Workplace Shell desktop, through REXX scripts, and many system GUI and Command-line utilities (such as 4OS2 ).9 To accommodate its OS/2 subsystem, Windows NT supports the handling of extended attributes in HPFS , NTFS , and FAT. It stores EAs on FAT and HPFS using exactly the same scheme as OS/2, but does not support any other kind of ADS as held on NTFS volumes. Trying to copy a file with any ADS other than EAs from an NTFS volume to a FAT or HPFS volume gives a warning message with the names of the ADSs that will be lost. Windows 2000 onward acts exactly as Windows NT, except that it ignores EAs when copying to FAT32 without any warning (but shows the warning for other ADSs, like "Macintosh Finder Info" and "Macintosh Resource Fork"). Future Microsoft has recently secured patents for VFAT and FAT32 (but not the original FAT), which is causing concern that the company might later seek royalties from Linux distributions and from media vendors that pre-format their products (see FAT Licensing below). Despite two earlier rulings against them, Microsoft prevailed and was awarded the patents. Since Microsoft has announced the discontinuation of its MS-DOS -based consumer operating systems with Windows Me , it remains unlikely that any new versions of FAT will appear. For most purposes, the NTFS file system that was developed for the Windows NT line is superior to FAT from the points of view of efficiency, performance, and reliability; its main drawbacks are the size overhead for small volumes and the very limited support by anything other than the NT-based versions of Windows, since the exact specification is a Trade Secret of Microsoft. The availability of NTFS-3G since mid 2006 has led to much improved NTFS support in Unix-like operating systems, considerably alleviating this concern. It is still not possible to use NTFS in DOS-like operating systems, which in turn makes it difficult to use a DOS floppy for recovery purposes. Microsoft provided a Recovery Console to work around this issue, but for security reasons it severely limited what could be done through the Recovery Console by default. The movement of recovery utilities to boot CDs based on BartPE or Linux (with NTFS-3G) is finally eroding this drawback. FAT is still the normal filesystem for removable media (with the exception of CDs and DVDs), with FAT12 used on floppies, and FAT16 on most other removable media (such as , do make use of it. FAT16 is used on these drives for reasons of compatibility and size overhead. The FAT32 formatting support in Windows 2000 and XP is limited to volumes of 32 GB, which effectively forces users of modern hard drives either to use NTFS , to partition the drive into smaller volumes (below 32 GB), or to format the drive using third party tools. exFAT See Also: exFAT exFAT is an incompatible replacement for FAT filesystems that is expected to be introduced with Windows CE 6.0 . It is intended to be used on Flash Drives , where FAT is used today. Windows XP and Vista file system drivers will be offered by Microsoft shortly after the release of Windows CE 6.0. exFAT introduces a free space bitmap allowing faster space allocation and faster deletes, support for files up to 264 bytes, larger cluster sizes (up to 32 MB in the first implementation), an extensible directory structure and name hashes for filenames for faster comparisons. It does not have short 8.3 filenames anymore. It does not appear to have security Access Control List s or File System Journaling like NTFS, though device manufacturers can choose to implement simplified support for transactions (backup file allocation table used for the write operations, primary FAT for storing last known good allocation table). DESIGN The following is an overview of the order of structures in a FAT partition or disk: A FAT file system is composed of four different sections. # The Reserved sectors, located at the very beginning. The first reserved sector is the Boot Sector (aka ''Partition Boot Record''). It includes an area called the '' BIOS Parameter Block '' (with some basic file system information, in particular its type, and pointers to the location of the other sections) and usually contains the operating system's boot loader code. The total count of reserved sectors is indicated by a field inside the Boot Sector. Important information from the Boot Sector is accessible through an operating system structure called the ''Drive Parameter Block'' in DOS and OS/2. For FAT32 file systems, the reserved sectors include a ''Backup Boot Sector'' at Sector 6. # The FAT Region. This contains two copies of the ''File Allocation Table'' for the sake of redundancy, although the extra copy is rarely used, even by disk repair utilities. These are maps of the Data Region, indicating which clusters are used by files and directories. # The Root Directory Region. This is a ''Directory Table'' that stores information about the files and directories located in the root directory. It is only used with FAT12 and FAT16 and means that the root directory has a fixed maximum size which is pre-allocated at creation of this volume. FAT32 stores the root directory in the Data Region along with files and other directories instead, allowing it to grow without such a restraint. # The Data Region. This is where the actual file and directory data is stored and takes up most of the partition. The size of files and subdirectories can be increased arbitrarily (as long as there are free clusters) by simply adding more links to the file's chain in the FAT. Note however, that clusters are allocated in their entirety, and so if a 1 KB file resides in a 32 KB cluster, 31 KB are wasted. FAT uses Little Endian format for entries in the header and the FAT(s). Boot Sector Common structure of the first 36 bytes used by all FAT versions: Same value of media descriptor should be repeated as first byte of each copy of FAT. Certain operating systems ( MSX-DOS version 1.0) ignore boot sector parameters altogether and use media descriptor value from the first byte of FAT to determine filesystem parameters.
|