Windows NTFS filesystem internals

NTFS is the preferred file system on Windows NT family, including Windows 7, XP, and 2000. It is efficient only on hard drives. It does not perform well (in other words it is too slow) on flash drives. In this case FAT32 or ExFAT should be used. 

NTFS is derived from OS/2 filesystem called HPFS. It was written by Gordon Letwin and others at Microsoft and added to OS/2 version 1.2, at that time still a joint undertaking of Microsoft and IBM.

NTFS  address many of the limitations of the earlier FAT16 and FAT32 file systems. The most important features are as follows:

• Data recoverability. NTFS limits the possibility of data corruption by organizing I/O operations by transactions. Transactions are atomic, which means that either the entire I/O operation must complete or none of it can complete. If anything interrupts the transaction in-progress, such as loss of power to the computer or a cancellation of the I/O operation, NTFS does everything possible to guarantee that any changes made to the file system as part of the I/O operation are undone, or rolled back, returning the file system to its condition before the I/O operation began.

  Also, NTFS is a fully recoverable file system. It is designed to restore consistency to a disk after a CPU failure, system crash, or I/O error. NTFS allows the operating system to recover without your having to use disk-checking utilities. However, NTFS provides some disk utilities in case recovery fails or corruption occurs outside the control of the file system.
   

• Storage fault tolerance. Data-redundant storage methods can be used with NTFS to ensure that if data is corrupted on one physical disk, an intact copy can be retrieved from the disk mirror. NTFS always uses data redundancy to protect internal data structures containing metadata vital to the integrity of the file system.
   
• Data security. NTFS implements files and directories as securable objects according to the Windows object security architecture. Access to file and directory objects in NTFS can be restricted to specific users and groups under this architecture. For more information on this, see File Security and Access Rights. Data security features for files and directories are not included in the FAT file systems.

Other advanced features provided by NTFS are the following:

• Multiple data streams. As mentioned in the Files section, NTFS files can consist of more than one stream. The additional streams may contain any kind of data, although typically it is data describing the file, or metadata. For more information about multiple data streams in NTFS files, see the Files section.
   
• Unicode names. Unicode is the standard character set used in NTFS and replaces the older single-byte ASCII character set. Every character used in every major natural language is represented by a unique double-byte number in the Unicode set.

  For more information about Unicode, see About Unicode and Character Sets.
   

• Improved file attribute indexing. NTFS includes the ability to index file attributes as a means of locating and sorting multiple files that share similar data quickly. You can index file names in FAT32 and FAT16 file systems, but not attributes. Also, the file system does not have the functionality to sort the indexed FAT32 and FAT16 file names.
• Dynamic bad-cluster remapping. When a read operation on an NTFS volume that is not fault tolerant encounters corrupted data on a cluster of sectors, each sector in the cluster is flagged as bad, and subsequent attempts to perform read operations on that sector will result in the error being returned. In the same scenario on FAT file systems, the file system itself does not flag bad sectors—the user must run the Chkdsk.exe utility to do this.

  When this scenario occurs on fault-tolerant NTFS volumes, the file system will do the following for each bad sector it encounters:
   

  1. Recover the uncorrupted data from a secondary source on the volume.
  2. Locate a good sector and write the recovered data to it.
  3. Remap the bad sector to the new good sector so that all subsequent attempts to perform I/O operations on the bad sector will be automatically redirected to the new one.
      
• Hard links and junctions. Hard links and junctions are two ways that storage objects can be linked under NTFS. For more information on both, see Hard Links and Junctions.
   
• Compression and sparse file support. NTFS volumes support file compression on an individual file basis. The file compression algorithm used by NTFS is Lempel-Ziv compression. This is a lossless compression algorithm, which means that no data is lost when compressing and decompressing the file, as opposed to lossy compression algorithms such as JPEG, where some data is lost each time data compression and decompression occur.

  For more information on NTFS file compression, see File Compression and Decompression.

  A file in which most of the data is zeros is called a sparse file. NTFS implements a form of file compression specifically for sparse files in which only nonzero data is written to the file and the file system provides the correct amount of zero data to applications at need. For more information on sparse file support, see Sparse Files.
   

• Change journals. NTFS creates and maintains change journals for each volume, to track all changes made to it. For more information on change journals, see Change Journals.
   
• Distributed link tracking. The Windows shell allows a user to create files on their desktop that link to applications that reside in another location in the volume. The Start menu, which the user can configure, contains many instances of this kind of link. Also, the object linking and embedding, or OLE, technology allows applications to embed links to external files within the files they create and maintain. The components of the Office 2000 suite—Word, PowerPoint®, and Excel—are examples of applications that use OLE technology.

  A problem arises in the previous examples when the file being linked to (the link source) is moved, making it inaccessible through the link—also referred to as the link client. Distributed link tracking was first introduced in the version of NTFS shipped with Windows 2000 to enable client applications to track link sources that have been moved. As a result, applications and users that create links do not have to maintain the linkage themselves when the link source is moved.

  For more information on distributed link tracking, see Distributed Link Tracking and Object Identifiers.
   

• Encryption. NTFS provides the Encrypted File System, or EFS, for cryptographic protection of files and directories. For more information on EFS, see File Encryption. For more information on cryptography in general, see Cryptography Reference.
   
• POSIX support. The following POSIX functionality was introduced in Windows 2000:
  1. Files can be accessed in NTFS file systems according to POSIX naming conventions. POSIX conventions allow file names that have trailing spaces, file names that have trailing periods (.), and file names that are identical except for the case of the characters.
  2. Traversal permissions, where the security attributes of each parent directory in the path of a file or directory is used in determining whether a specific user has access to it.
  3. "File change time" time stamps.
  4. POSIX-style hard links. For a code example that shows how to back up and restore these type of links, see Backing Up and Restoring POSIX File Links.
      
• Defragmentation API. A file is stored on a disk drive and other storage media in one or more clusters. Clusters are the atomic unit of data allocation, made up of one or more sectors. Sectors are physical storage units.

  As a file is written to the disk, the file may not be written in contiguous clusters. Noncontiguous clusters slow down the process of reading and writing the file. The farther apart on the disk the noncontiguous clusters are, the slower the process because of the increased time it takes to move the hard drive's read/write head. A file with noncontiguous clusters is said to be fragmented. To optimize files for fast access, a volume may be defragmented.

  Defragmentation is the process of moving a file's clusters on the disk to make them contiguous. NTFS does not perform defragmentation, but with version 5.0 it provides the ability for applications to perform defragmentation by calling an API. This API consists of functions that allow applications to obtain a map of the clusters that are in use and clusters that are not being used, obtain a map of how a file is using its clusters, and move a file.

  For more information on defragmentation, see Defragmenting Files.

• Reparse points. Under NTFS, a file or directory can contain a reparse point, which is a collection of user-defined data. For more information on reparse points, see Reparse Points.
   
• Directories as volume mount points. Volume mount points are directories on a volume that an application can use to "mount" a different volume, that is, to set it up for use at the location a user specifies. In other words, you can use a volume mount point as a gateway to a volume. When a volume is mounted at a volume mount point, users and applications can see the mounted volume by the path of the volume mount point or a drive letter. For example, with a volume mount point set, the user might see drive D as "C:\mnt\Ddrive" as well as "D:".

  Using volume mount points, you can unify into one logical file system disparate file systems such as NTFS, a 16-bit FAT file system, an ISO-9660 file system on a CD-ROM drive, and so on. Neither users nor applications need information about the volume on which a specific file resides. All the information they need to locate a specified file is a complete path. Volumes can be rearranged, substituted, or subdivided into many volumes without users or applications needing to change settings.

  For more information on volume mount points, see Volume Mount Points.

Recommended Links

Services for UNIX - Interoperability

GID on NTFS File System

Can you set group on a file or folder on NTFS file system? - No, ugh... Yes.

This question puzzled me for a long time but since it never really made it to my top priorities, I didn't look up for information on this. I thought of exploring this area more while I was researching something about NFS server.

I started with looking for the utilities (obviously, Windows-based ones) which can set this information for me. My search ended quickly with the chown.exe and chgrp.exe which you can install with Interix/SUA Base Utilities.

Using Process Monitor (replacement of Filemon and Regmon utilities) revealed that group information gets stored on the file system with an IRP_MJ_SET_SECURITY request -

The other interesting fact is this request originating from the POSIX subsystem (psxss.exe) which makes sense because chown.exe and chgrp.exe utilities are POSIX subsystem utilities.

This KB Article says -

In the Windows NT and Windows 2000 NTFS file system, each file also has an owner and a primary group. The primary group of a file is not used by the Win32 subsystem, but is present for programs that make use of the POSIX subsystem. When a file is created, the user who created the file becomes its owner and that user's primary group becomes the file's primary group. Access Control Entries (ACEs) are then added to the DACLs to assign permissions.

That makes it clear that none of other utilities I tried to use, could set this information because they were basically Win32 binaries and Win32 subsystem does not, in anyway, uses this information.

So the best practice would be to set correct primary groups for your users and then use Interix/SUA chown.exe and chgrp.exe utilities to manage them the way you want them to be seen by your UNIX clients.

Additional Note: The ls.exe and chmod.exe are other utilities which can help you do things the UNIX way.

NTFS5 vs. FAT32

Win2K File-System Facts
Compatibility with legacy file systems is an important aspect of OS upgrades. Win2K supports several file systems, and its support is essentially a superset of NT 4.0's support. (For more information about file systems, see "File-System Resources," page 104.) Win2K still supports FAT12, FAT16, and the CD-ROM File System (CDFS), which is the International Organization for Standardization (ISO)-9660-compliant NT file system you use to read CD-ROMs. However, Microsoft doesn't plan to add features or upgrade these systems, except for a possible future addition of support for the UNIX-centric Rock Ridge format in CDFS. NTFS, which is the premier Win2K file system, continues to exist but sports a new file-system structure and new capabilities. Microsoft also included in Win2K a Universal Disk Format (UDF) file system that the company introduced in Windows 98. Currently, Win2K's UDF version, which complies with the ISO-13346 standard, mainly supports DVD-ROM media. However, Microsoft plans to eventually replace the CD-ROM-centric CDFS specification with UDF. Win2K also supports a slightly more recent version of UDF (i.e., UDF 1.50) than Win98 supports (i.e., UDF 1.02).

NTFS.com NTFS File System General Information. Data Recovery

Exploring NTFS On-disk Structures

NTFS FAQ (en)

Microsoft/NTFS overview

NTFS Read-only NTFS support.

Systems Internals -- very useful site with a lot of utilities with source code. Makers of NTFSDOS -- NTFS driver for DOS and other interesting tools for NT recovery

• NTFSInfo v1.0
• NTFSDOS Tools v2.0
• NTFSDOS v2.0
• NTLocksmith v1.0
• Bluescreen of Death Screen saver  -- neat trick

File System Usage in Windows NT 4.0 - by Wemer Vogels - 2.15M .PDF

This paper discusses real world traces of filesystem usage. It shows how certain real world behaviors are important to optimize for and shows how NT succeeds (partially) at providing good filesystem performance.

PartitionMagic - Product Information -- nice utility to convert NTFS into something else

Tutorials

[PPT] Windows Kernel Internals NTFS

ntfsdoc - What is ntfsdoc

NTFS - Wikipedia, the free encyclopedia

Windows NT File System Internals: A Developer's Guide: Chapter 4. ...

Inside NTFS

NTFS streams

NTFS File System Internals - DECUS San Diego

Inside Win2K NTFS, Part 2

NTFS.com NTFS Format. Master File Table (MFT). General Information ...

The Delphi Magazine Issue 98 Sample Article/NTFS Compression And Sparse Filesby Marcel van Brakel This article first appeared in The Delphi Magazine Issue 98 (October 2003)

How could space not be infinite? Is there a sign somewhere
saying ‘Space Ends Here – Mind the Gap’?      (Max Tegmark)

Hard disks keep getting larger every year, but after a while even the largest hard disks just don’t cut it any more. My first PC had a 425Mb hard disk, and I was sure I’d never run out of space. Two years later I found myself looking for a second disk. My second PC had a 4Gb hard disk, and this time I just knew I’d never run out of disk space. Two years later I found myself looking for a second disk. My current PC has a 40Gb hard disk, and now, a year after purchase, it’s 75% full. At most companies I’ve worked for the exact same thing happens. When purchased the storage devices seem to have infinite space, but before long the limits of infinity are reached. Buying additional, or larger, disks or moving unused files to offline storage, are two possible ways to solve this problem. Compression is another.

Ever since Windows NT 3.51, the Windows NT File System (NTFS) has had native support for file compression. While very useful, this generic compression is not without drawbacks. Compression and decompression consume precious processor cycles and, because it uses a generic compression algorithm (Lempel-Ziv), it may not guarantee you the best possible compression ratio. With NTFS version 5 (introduced with Windows 2000) Microsoft has added another, specialised means of compression known as sparse files. In this article we’ll look at both of these compression methods and how you use them from a programmer’s perspective.

NTFS Internals

To fully understand how compression and sparse files work we need to take a peek inside NTFS. NTFS uses disk clusters as its basic unit of storage. Depending on how the volume was formatted, a cluster may span one or more disk sectors. This means that when NTFS reads from or writes to a disk, it always does so using at least one cluster. Even if you create a one-byte file, the file will occupy a full cluster on the disk. Each cluster is identified using a Logical Cluster Number (LCN). The first cluster has LCN 0, the second LCN 1, and so forth.

A file’s data will span one or more clusters on disk, which may not be physically contiguous but could be viewed as such logically. NTFS numbers these logical clusters belonging to a file using a Virtual Cluster Number (VCN). The first logical cluster of a file has VCN 0, the second VCN 1, and so forth. NTFS doesn’t keep track of each individual cluster that belongs to a file but instead manages them using the concept of a run (sometimes called an extend). A run is just a set of one or more clusters that have contiguous LCNs. Each run can be managed by storing its starting LCN and the count of clusters in that run. NTFS does exactly that: for each file it maintains a list of LCN and count pairs identifying the runs in which the file data is stored. The only missing piece is that NTFS needs to be able to translate an offset into a file into the LCN where it’s stored. For this purpose NTFS also stores a VCN to LCN mapping (an offset into a file can be converted to a VCN simply by dividing the offset by the cluster size). The whole concept of LCNs, VCNs and mapping structures is summarised in Figure 1. The file described therein has 8 clusters stored in two runs: four cluster each (runs aren’t required to be four clusters, it’s just an example).

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019