Reliable UDP (RUDP): The Next Big Streaming Protocol?

 

5

 All too often we shy away from the depths of IP protocols, leaving the application vendors such as Microsoft; Wowza Media Systems, LLC; RealNetworks, Inc.; Adobe Systems, Inc.; and others with more specific skills to deal with the dark art of the network layer for us, while we just type in the server name, hit connect, then hit start.
Those who have had a little experience will probably have heard of TCP (transmission control protocol) and UDP (user datagram protocol). They are transport protocols that run over IP links, and they define two different ways to send data from one point to another over an IP network path. TCP running over IP is written TCP/IP; UDP in the same format is UDP/IP.
TCP has a set of instructions that ensures that each packet of data gets to its recipient. It is comparable to recorded delivery in its most basic form. However, while it seems obvious at first that "making sure the message gets there" is paramount when sending something to someone else, there are a few extra considerations that must be noted. If a network link using TCP/IP notices that a packet has arrived out of sequence, then TCP stops the transmission, discards anything from the out-of-sequence packet forward, sends a "go back to where it went wrong" message, and starts the transmission again.
If you have all the time in the world, this is fine. So for transferring my salary information from my company to me, I frankly don't care if this takes a microsecond or an hour, I want it done right. TCP is fantastic for that.
In a video-centric service model, however, there is simply so much data that if a few packets don't make it over the link there are situations where I would rather skip those packets and carry on with the overall flow of the video than get every detail of the original source. Our brain can imagine the skipped bits of the video for us as long as it's not distracted by jerky audio and stop-motion video. In these circumstances, having an option to just send as much data from one end of the link to the other in a timely fashion, regardless of how much gets through accurately, is clearly desirable. It is for this type of application that UDP is optimal. If a packet seems not to have arrived, then the recipient waits a few moments to see if it does arrive -- potentially right up to the moment when the viewer needs to see that block of video -- and if the buffer gets to the point where the missing packet should be, then it simply carries on, and the application skips the point where the missing data is, carrying on to the next packet and maintaining the time base of the video. You may see a flicker or some artifacting, but the moment passes almost instantly and more than likely your brain will fill the gap.


If this error happens under TCP then it can take TCP upward of 3 seconds to renegotiate for the sequence to restart from the missing point, discarding all the subsequent data, which must be requeued to be sent again. Just one lost packet can cause an entire "window" of TCP data to be re-sent. That can be a considerable amount of data, particularly when the link is known as a Long Fat Network link (LFN or eLeFaNt; it's true -- Google it!).


All this adds overhead to the network and to the operations of both computers using that link, as the CPU and network card's processing units have to manage all the retransmission and sync between the applications and these components.


For this reason HTTP (which is always a TCP transfer) generally introduces startup delays and playback latency, as the media players need to buffer more than 3 seconds of playback to manage any lost packets.
Indeed, TCP is very sensitive to something called window size, and knowing that very few of you ever will have adjusted the window size of your contribution feeds as you set up for your live Flash Streaming encode, I can estimate that all but those same very few have been wasting available capacity in your network links. You may not care. The links you use are good enough to do whatever it is you are trying to do.
In today's disposable culture of "use and discard" and "don't fix and reuse," it's no surprise that most streaming engineers just shrug and assume that the ability to get more bang for your buck out of your internet connection is beyond your control.

For example, did you know that if you set your maximum transmission unit (MTU) -- ultimately your video packet size -- too large then the network has to break it in two in a process called fragmentation? Packet fragmentation has a negative impact on network performance for several reasons. First, a router has to perform the fragmentation -- an expensive operation. Second, all the routers in the path between the router performing the fragmentation and the destination have to carry additional packets with the requisite additional headers.

Also, in the event of a retransmission, larger packets increase the amount of data you need to resend if a retransmission occurs.
Alternatively, if you set the MTU too small then the amount of data you can transfer in any one packet is reduced and relatively increases the amount of signaling overhead (the data about the sending of the data, equivalent to the addresses and parcel tracking services in real post). If you set the MTU as small as you can for an Ethernet connection, you could find that the overhead nears 50% of all traffic.
UDP offers some advantages over TCP. But UDP is not a panacea for all video transmissions.
Where you are trying to do large-video file transfer, UDP should be a great help, but its lossy nature is rarely acceptable for stages in the workflow that require absolute file integrity. Imagine studios transferring master encodes to LOVEFiLM or Netflix for distribution. If that transfer to the LOVEFiLM or Netflix playout lost packets then every single subscriber of those services would have to accept that degraded master copy as the best possible copy. In fact, if UDP was used in these back-end workflows, the content would degrade the user's experience in the same way that historically tape-to-tape and other dubbed and analog replication processes used to. Digital media would lose that perfect replica quality that has been central to its success.
Getting back to the focus on who may want to reduce their network capacity inefficiencies: Studios, playouts, news desks, broadcast centers, and editing suites all want their video content intact/lossless, but naturally they want to manipulate that data between machines as fast as possible. Having video editors drinking coffee while videos transfer from one place to another is inefficient (even if the coffee is good).
Given they cannot operate in a lossy way, are these production facilities stuck with TCP and all the inherent inefficiencies that come with the reliable transfer? Because TCP ensures all the data gets from point to point, it is called a "reliable" protocol. In UDP's case, that reliability is "left to the user," so UDP in its native form is known as an "unreliable" protocol.
The good news is that there are indeed options out there in the form of a variety of "reliable UDP" protocols, and we'll be looking at those in the rest of this article. One thing worth noting at the outset, though, is that if you want to optimize links in your workflow, you can either do it the little-bit-hard way and pay very little, or you can do it the easy way and pay a considerable amount to have a solution fitted for you.
Reliable UDP transports can offer the ideal situation for enterprise workflows -- one that has the benefit of high-capacity throughput, minimal overhead, and the highest possible "goodput" (a rarely used but useful term that refers to the part of the throughput that you can actually use for your application's data, excluding other overheads such as signaling). In the Internet Engineering Task Force (IETF) world, from which the IP standards arise, for nearly 30 years there has been considerable work in developing reliable data transfer protocols. RFC-908, dating from way back in 1984, is a good example.
Essentially, RDP (reliable data protocol) was proposed as a transport layer protocol; it was positioned in the stack as a peer to UDP and TCP. It was proposed as an RFC (request for comment) but did not mature in its own right to become a standard. Indeed, RDP appears to have been eclipsed in the late 1990s by the Reliable UDP Protocol (RUDP), and both Cisco and Microsoft have released RUDP versions of their own within their stacks for specific tasks. Probably because of the "task-specific" nature of RUDP implementations, though, RUDP hasn't become a formal standard, never progressing beyond "draft" status.
One way to think about how RUDP types of transport work is to use a basic model where all the data is sent in UDP format, and each missing packet is indexed. Once the main body of the transfer is done, the recipient sends the sender the index list and the sender resends only those packets on the list. As you can see, because it avoids the retransmission of any windows of data that have already been sent that immediately follow a missed packet, this simple model is much more efficient. However, it couldn't work for live data, and even for archives a protocol must be agreed upon for sending the index. It responds to that rerequest in a structured way (which could result in a lot of random seek disc access, for example, if it was badly done).
There are many reasons the major vendor implementations are task-specific. For example, where one may use UDP to avoid TCP retransmission after errors, if the entire data must be faultlessly delivered to the application, one needs to actually understand the application.
If the application requires control data to be sent, it is important for the application to have all the data required to make that decision at any point. If the RUDP system (for example) only looked for and re-requested all the missing packets every 5 minutes (!) then the logical operations that lacked the data could be held up waiting for that re-request to complete. This could break the key function of the application if the control decision needed to be made sooner than within 5 minutes.
On the other hand, if the data is a large archive of videos being sent overnight for precaching at CDN edges, then it may be that the retransmission requests could be managed during the morning. So the retransmission could be delayed until the entire archive has been sent, following up with just the missing packets on a few iterations until all the data is delivered. So the flow, in this case, has to have some user-determined and application-specific control.
TCP is easy because it works in all cases, but it is less efficient because of that. On the other hand, UDP either needs its applications to be resilient to loss or the application developer needs to write in a system for ensuring that missing/corrupted packets are retransmitted. And such systems are in effect proprietary RUDP protocols.
There is an abundance of these, both free and open source, and I am going to look at several of each option (Table 1). Most of you who use existing streaming servers will be tied to the streaming protocols that your chosen vendor offers in its application. However, for those of you developing your own streaming applications, or bespoke aspects of workflows yourselves, this list should be a good start to some of the protocols you could consider. It will also be useful for those of you who are currently using FTP for nonlinear workflows, since the swap out is likely to be relatively straightforward given than most nonlinear systems do not have the same stage-to-stage interdependence that linear or live streaming infrastructures do.
Let's zip (and I do mean zip) through this list. Note that it is not meant to be a comprehensive selection but purely a sampler.
The first ones to explore in my mind are UDP-Lite and Datagram Congestion Control Protocol. These two have essentially become IETF standards, which means that inter-vendor operation is possible (so you won't get locked into a particular vendor).

Table 1: A Selection of Reliable UDP Transports 

DCCP

Let's look at DCCP first. DCCP provides initial code implementations for those inclined. From the point of view of a broadcast video engineer, this is really deeply technical stuff for low-level software coders. However, if you happen to be
(or simply have access to) engineers of this skill level then DCCP is freely available. DCCP is a protocol worth considering if you are using shared network infrastructure (as opposed to private or leased line connectivity) and want to ensure you get as much throughput as UDP can enable, while also ensuring that you "play fair" with other users. It is worth commenting that "just turning on UDP" and filling the wire up with UDP data with no consideration of any other user on the wire can saturate the link and effectively make it unusable for others. This is congestion, but DCCP manages to fill the pipe as much as possible, while still inherently enabling other users to use the wire too.
Some of the key DCCP features include the following:

• Adding a reliability layer to UDP
• Discovery of the right MTU size is part of the protocol design (so you fill the pipe while avoiding fragmentation)
• Congestion control

Indeed, to quote the RFC: "DCCP is intended for applications such as streaming media that can benefit from control over the tradeoffs between delay and reliable in-order delivery."

UDP-Lite

The next of these protocols is UDP-Lite. Also an IETF standard, this nearly-identical-to-UDP protocol differs in one key way: It has a checksum (a number that is the result of a logical operation performed on all the data, which if it differs after a transfer indicates that the data is corrupt) and a checksum coverage range that that checksum applies to, whereas vanilla UDP -- optionally in IPv4, and always in IPv6 -- has just a simple checksum on the whole datagram and if present the checksum covers the entire payload.
Let's simplify that a little: What this means is that in UDP-Lite you can define part of the UDP datagram as something that must arrive with "integrity," i.e., a part that must be error-free. But another part of the datagram, for example the much bigger payload of video data itself, can contain errors (remain unchecked against a checksum) since it could be assumed that the application (for example, the H.264 codec) has error handling or tolerance in it.
This UDP-Lite method is very pragmatic. In a noisy network link, the video data may be subject to errors but could be the larger part of the payload, where the important sequence number may only be a smaller part of the data (statistically less prone to errors). If it fails, the application can use UDP-Lite to request a resend of that packet. Note that it is up to the application to request the resend; the UDP-Lite protocol simply flags the failure up and the software can prioritize a resend request, or it can simply plan to work around a "discard" of the failed data. It is also worth noting that most underlying link layer protocols such as Ethernet or similar MAC-based systems may discard damaged frames of data anyway unless something interfaces with those link layer devices. So to work reliably, UDP-Lite needs to interface with the network drivers to "override" these frame discards. This adds complexity to the deployment strategy and certainly most likely takes the opportunity away from being "free." However, it's fundamentally possible.
So I wanted to see what was available "ready to use" for free, or close to free at least. I went looking for a compiled, user-friendly, simple-to-use application with a user-friendly GUI, thinking of the videographers having to learn all this code and deep packet stuff just to upload a video to the office.

UDPXfer

While it's not really a protocol per se, I found UDPXfer, a really simple application with just a UDP "send" and "listener" mode for file transfer.
I set up the software on my laptop and a machine in Amazon EC2, fiddled with the firewall, and sent a file. I got very excited about the prompt 5MB UDP file transfer taking 2 minutes and 27 seconds, and I then set up an FTP of the same file over the same link but was disappointed that the FTP took 1 minute and 50 seconds -- considerably faster. When I looked deeper, however, the UDPXfer sender had a "packets per second" slider. I then nudged the slider to its highest setting, but it was still only essentially 100Kbps maximum, far slower than the effective TCP. So I wrote to the developer, Richard Stanway, about this ceiling. He sent a new version that allowed me to set a 1300 packets-per-second transmission. He commented that it would saturate the IP link from me to the server, and in a shared network environment a better approach would be to the tune the TCP window's size to implement some congestion control. His software was actually geared to resiliency over noisy network links that cause problems for TCP.
Given that I see this technology being used on private wires, the effective saturation that Stanway was concerned about was less of a concern for my enterprise video workflow tests, so I decided to give the new version a try. As expected, I managed to bring the transfer time down to 1 minute and 7 seconds.
So while the software I was using is not on general release, it is clearly possible to implement simple software-only UDP transfer applications that can balance reliability with speed to find a maximum goodput.

Commercial Solutions

But what of the commercial vendors? Do they differentiate significantly enough from "free" to cause me to reach into my pocket?
I caught up with Aspera, Inc. and Motama GmbH, and I also reached out to ZiXi. All of this software is complex to procure at the best of times, so sadly I haven't had a chance to play practically with these. Also, the vendors do not publish rate cards, so it's difficult to comment on their pricing and value proposition.
Aspera co-presented at a recent Amazon conference with my company, and we had an opportunity to dig into its technology model a bit. Aspera is indeed essentially providing variations on the RUDP theme. It provides protocols and applications that sit on top of those protocols to enable fast file distribution over controlled network links. In Aspera's case, it was selling in behind Amazon Web Services Direct Connect to offer optimal upload speeds. It has a range of similar arrangements in place targeting enterprises that handle high volumes of latency-sensitive data. You can license the software or, through the Amazon model, pay for the service by the hour as a premium AWS service. This is a nice flexible option for occasional users.

Aspera provides variations on the RUDP theme, including fasp 3, which the company introduced at this year's IBC in Amsterdam. 
I had a very interesting chat with the CEO of Motama, which has a very appliance-based approach to its products. The RUDP-like protocol (called RelayCaster Streaming Protocol or RCSP) is used internally by the company's appliances to move live video from the TVCaster origination appliances to RelayCaster devices. These then can be hierarchically set up in a traditional hub and spoke or potentially other more complex topologies. The software is available (under license) to run on server platforms of your choice, which is good for data center models. They have also recently started to look at licensing the protocol to a wider range of client devices, and they pride themselves in being available for set-top boxes.

Motama offers an appliance-based" approach to its RUDP-like protocol, which it calls RelayCaster Streaming Protocol and which is available for set-top boxes and CDN licensing. 

The last player in the sector I wanted to note was ZiXi. While I briefly spoke with ZiXi representatives while writing this, I didn't manage to communicate properly before my deadline, so here is what I know from the company's literature and a few customer comments: ZiXi offers a platform that optimizes video transfer for OTT, internet, and mobile applications. The platform obviously offers a richer range of features than just UDP-optimized streaming, and it has P2P negotiation and transmuxing so you can flip your video from standards such as RTMP out to MPEG-TS, as you can with servers such as Wowza. Internally, within its own ecosystem, the company uses its own hybrid ZiXi protocol, including features such as forward error correction, combining applications layer software in a product called Broadcaster that looks like a server with several common muxes (RTMP, HLS, etc.) and includes ZiXi. If you have an encoder with ZiXi running, then you can contribute directly to the server using the company's RUDP-type transport.

In addition to UDP-optimized streaming, ZiXi offers P2P negotiation and transmuxing, similar to servers from RealNetworks and Wowza. 

Worth the Cost?

I am aware none of these companies licenses their software trivially. The software packages are their core intellectual properties, and defending them is vital to the companies' success. I also realize that some of the problems that they purport to address may "go away" when you deploy their technology, but in all honesty, that may be a little like replacing the engine of your car because a spark plug is misfiring.
I am left wondering where the customer can find the balance between the productivity gains in accelerating his or her workflow with these techniques (free or commercial) against the cost of a private connection plus either the cost of development time to implement one of the open/free standards or the cost of buying a supported solution.
The pricing indication I have from a few undisclosed sources is that you need to be expecting to spend a few thousand on the commercial vendor's licensing, and then more for applications, appliances, and support. This can quickly rise to a significant number.
This increased cost to improve the productivity of your workflow must be at some considerable scale, since I personally think that a little TCP window sizing, and perhaps paying for slightly "fatter" internet access, may resolve most problems -- particularly in archive transfer and so on -- and is unlikely to cost thousands.
However, at scale, where those optimizations start to make a significant productivity difference, it clearly makes a lot of sense to engage with a commercially supported provider to see if its offering can help.
At the end of the day, regardless of the fact that with a good developer you can do most things for free, there are important drivers in large businesses that will force an operator to choose to pay for a supported, tested, and robust option. For many of the same reasons, Red Hat Linux was a premium product, despite Linux itself being free.
I urge you to explore this space. To misquote James Brown: "Get on the goodput!"
This article appears in the October/November, 2012, issue of Streaming Media magazine as "Get on the Goodput."

NTFS or FAT32?

NTFS is the recommended file system for Windows XP and provides a number of benefits in terms of functionality, security, stability, availability, reliability, and performance. There are very few reasons to persist with FAT32.

If your PC has an earlier version of Windows on it in a dual-boot or multi-boot configuration with Windows XP then do not convert the system/boot partition to NTFS because if you do, the older version of Windows will not boot.

If you are looking for hard disk tools, you will find formatting tools and instructions on formatting hard disks here, data recovery tools here, and manufacturer's disk utilities here.

Page Index

• About FAT32
• NTFS & FAT32 Red Herrings
• Converting FAT32 to NTFS
• NTFS Troubleshooting

About FAT32

	Limitations of FAT32 - A Microsoft knowledge base article for Windows XP Home Edition, Professional, and 64-Bit Edition. This article discusses the limitations of the FAT32 file system in Windows XP. Description of the FAT32 File System in Windows XP - This article describes the FAT32 file system that is included with Microsoft Windows XP.
	Reasons to Retain FAT32 Dual and Multi-booting: If you want to install Windows 95, 98 or Millennium Edition with Windows XP using NTFS then the boot volume must be formatted as FAT32, not NTFS; this is because the older OSen must be installed on the boot volume. FAT32 is the only file common file system that the older OSen support that will also work with XP using NTFS. Windows 95 OSR2, Windows 98, Windows 2000, and Windows XP support FAT32 volumes. Small Disks: If you have old, low capacity drives of 20GB or less, FAT32 may afford a performance increase. For partitions of 2GB or less, FAT is recommended. Features not needed: If you do not need NTFS security, encryption, NTFS performance enhancements or support for large disks then there is no real reason to use NTFS. If you later decide to use NTFS, you can convert FAT32 to NTFS. The conversion process is discussed later in this article.

	Benefits of NTFS Support for large hard drives in excess of 127GB. Support for large files. NTFS in Windows XP supports a maximum file size up to the capacity of the disk. FAT32 supports a maximum file size of only 4 GB. Simple management of single disk partitions. No reboot is required between creating a new partition and formatting it. Improved performance, optimised for general performance and boot times. The size of the Master File Table (MFT) and its location are optimised based on the hard drive  characteristics. DISKPART and FORMAT takes about 90 seconds on a large hard drive. NTFS incorporates advanced file system features such as security, transacted operations, large volumes, and better performance on larger disks. These are not available on either  FAT or FAT32 disks. File compression and encryption. Third-party tools are not required. Volume shadow copy backup enable backups to be made without rebooting. A local hard drive can be mounted to a folder on an NTFS volume. NTFS is a journaling file system. It writes a log of changes being made to the files on disk.  This offers significant benefits in cases where a system is susceptible to power loss, experiences an unexpected reboot, or a crash. NTFS can quickly return a disk to a consistent  state without running CHKDSK, whereas FAT32 always requires CHKDSK to be run effect  recovery if a failure occurs. CHKDSK can take a very long time to complete on larger FAT32 drives but is very quick on NTFS.Both FAT and FAT32 have scaling and compatibility limitations that NTFS does not have. An NTFS volume is capable of scaling on very large disk sizes with a single partition and supports software RAID.
	For more technical information on the benefits of NTFS, read the Microsoft Windows Hardware  and Driver Central article on NTFS Preinstallation and Windows XP.
	For information on how Windows XP dynamically self-tunes, read the Windows Hardware and Driver Central article about Benchmarking on Windows XP Home Edition and Professional.  The article also covers disk efficiency optimisations, boot and application-launch prefetching,  as well as idle task scheduling.

NTFS & FAT32 Red Herrings

	If you don't know what a red herring is, read this.
	DOS FDISK Does Not Support NTFS FDISK Can Format NTFS Partitions Both of those statements are false. Uninformed rumours and unfounded statements about Windows 98 or DOS 6 FDISK make up one of the fishiest smelling red herrings about NTFS. The fact of the matter is, FDISK does not format any kind of partition. FDISK is a rudimentary partition manager. Furthermore, FDISK can work with NTFS partitions but it cannot delete NTFS partitions that are logical drives on extended partitions. Read Formatting / Partitioning Hard Disks and Installing XP for more information.
	Windows 98 Cannot Read NTFS This statement is false. It is only true insofar as Windows 98 cannot natively read NTFS disks. If you need to access a NTFS partition via Windows 98, you accomplish this with a third-party driver. Sysinternals once offered a free, read-only driver that allowed Windows 98 to read a NTFS partition. A full read/write access version was also available for purchase, however Sysinternals were absorbed into Microsoft in 2007, who then took over support of the Administrator's Pak, which contained the driver. You may still be able to find both the read-only and full version of the driver here, however Avira, the antivirus company, have made NTFS4DOS available, which enables access to NTFS drives for MS-DOS. The product has been discontinued and is unsupported by Avira but it is still available and free for personal use. You can get NTFS4DOS here.
	I want to Network XP with Windows 98 so I can't use NTFS This statement is false. The file system is independent of the network. If two machines are communicating over a network, the machines will not care about the file system, even if one of them is using clay tablets for mass storage.
	FAT32 Has Better Performance and is More Efficient Than NTFS This statement is false. NTFS is much more efficient than FAT32. Larger disks that are formatted FAT32 require far bigger File Tables. Larger file tables take longer to read. It is this for reason that XP will not allow you to create a FAT32 volume greater than 32GB in size.
	XP Does Not Support FAT32 Drives Larger than 32GB This statement is false. It does not follow that XP does not support FAT32 drives greater than 32GB just because XP will not allow you to create a FAT32 volume greater than 32GB. If a disk is preformatted with FAT32, right up to the theoretical limit for FAT32 disks, XP will support it.
	NTFS Does Not Fragment This statement is false. File fragmentation is a fact of life, irrespective of the underlying file system. Files increase and decrease in size with use, or are created, deleted and recreated over and over. Operating Systems attempt to keep parts of files in their place as new clusters are added as and when they are needed. The allocation of additional clusters cannot be guaranteed to follow on sequentially from where the file currently resides and are almost always allocated from a completely different location on the disk. It is this allocation process that causes fragmentation. If it is true that NTFS does not fragment, why does XP ship with a defragmenter?
	There are Very Few Recovery Tools for NTFS This statement is false. Not only does Windows XP ship with some very extensive system recovery and protection tools, it includes an impressive range of disaster recovery tools, far beyond what earlier versions of Windows ever provided. Plus there are a great number of third-party tools available, a lot of them free, to supplement those that come with XP. Windows XP provides advanced disk and maintenance tools you can use to prevent problems from occurring. Some of the most useful tools are discussed in the Data Corruption and Hard Disk Troubleshooting article on this site. The disk-related tools allow you to view disk information and correct a problem before it becomes a serious issue.

Converting FAT32 to NTFS

Windows XP comes with the tools necessary to convert your FAT16 or FAT32 file system to NTFS, however there are a number of issues that you must consider before taking the conversion route. The problems to contend with range from permissions issues to performance degradation.

	Permissions If you use the convert utility on an existing installation of Windows XP Professional or on Windows XP Home Edition, the default security settings installed on for FAT32 are not appropriate for NTFS. The result is that the All Users folder and all its subfolders have faulty permissions, for example you may notice that your virus scanner locks or gets stuck trying to access System Volume Information: To work around this problem, consult the File security issues after converting FAT32 partitions to the NTFS file system knowledgebase article.
	Performance On volumes that are created (not converted) as NTFS volumes, clusters start at sector zero, therefore every cluster is aligned on what is known as the cluster boundary. If the FAT32 partition was not created by Windows XP or Windows 2000 then the presence of the earlier OS' FAT/FAT32 reserved structures means that a FAT/FAT32 format cannot guarantee that data clusters will be aligned on a cluster boundary. In turn, this causes the conversion tool to use 512k clusters, which can potentially cause serious degradation in disk performance. To put that another way, earlier FAT/FAT32 OSen use a different reserved structure format to win2k and winXP, which means that there is a difference in the offset to the data clusters. To make up for this difference, the conversion tool uses a smaller cluster size. To avoid this problem, you must convert your FAT16/FAT32 disk to 4k aligned clusters. BootIt NG from TeraByte has an "Align for NTFS only" option that will convert your FAT clusters to 4k aligned. BootIt NG has a free trial period and is highly recommended. Caution: You should backup your important files before performing the alignment. Caution: The 4k alignment process may take several hours to complete. Caution: The 4k alignment task is not for the faint-hearted. Consider formatting with NTFS instead. To Use BootIt NG to align the clusters you will need a floppy drive. After extracting the files from the download archive: Run BOOTITNG.EXE to create a boot floppy Don't allow BootIt to install on your hard disk; do this by clicking the "Cancel Install" button, which will take you straight into the Maintenance screen Select "Partition Work" and choose the partition to be aligned Click "Slide" Check "Align for NTFS only" Click "OK" Once the alignment is complete, reboot and defragment the hard disk. How to Defragment Your Disk Drive Volumes in Windows XP This article describes how to defragment your disk drive volumes and describes the limitations of using Disk Defragmenter Microsoft Management Console (MMC) that is included with Windows XP. The Free Space That Is Required to Convert FAT to NTFS Check that you have sufficient free space to perform the conversion. Then do the FAT to NTFS conversion: How To Convert a FAT16 or FAT32 Volume to NTFS in Windows XP kadaitcha.cx recommends that you consider not converting your drive from FAT16/FAT32 to NTFS, and instead consider backing up your important data, do a NTFS format and clean install your OS. "Insufficient Disk Space for Conversion" Error Message When You Convert a Disk from FAT to NTFS in Windows XP File security issues after converting FAT32 partitions to the NTFS file system The "Drive Properties" Dialog Box May Display the Wrong File System Type After a Drive Is Converted to NTFS After you convert a drive to the NTFS file system by using the Convert.exe tool, the file system type may still be displayed as file allocation table (FAT) or FAT32 on the dialog box of the drive.

NTFS Troubleshooting

	"Missing or corrupt Ntfs.sys" error message when you restart Windows XP after you convert your hard disk to the NTFS file system You can use the following command to convert your hard disk from the FAT 32 file system to the NTFS file system: convert drive: /fs:ntfs When you use the command, after the computer completes the conversion and you restart Windows XP, you may receive an error message similar to the following error message: Windows could not start because the following file is missing or corrupt: System32\Drivers\Ntfs.sys
	How to Disable the 8.3 Name Creation on NTFS Partitions The creation of 8.3 filenames and directories for all long filenames and directories on NTFS partitions may decrease directory enumeration performance. This article describes a method of disabling the 8.3 name creation on all NTFS partitions. NOTE: Although disabling 8.3 name creation increases file performance under Windows NT, some 16-bit applications may not be able to find files and directories with long filenames.
	Chkdsk in Read-Only Mode Does Not Detect Corruption on NTFS Volume When you run the Chkdsk utility program in read-only mode on a volume that uses the NTFS file system, Chkdsk may not detect corruption in the disk structure.
	Error Message: "Access Is Denied" When You Try to Open NTFS File System Folders When you install Microsoft Windows XP Home Edition, and then you try to open folders on a different logical drive on your computer, you may receive the following error message: Access is denied.
	Windows Installer Error 1619 When You Install from NTFS-Protected Directories When you install a Setup program that is based on Windows Installer Setup technology, you may receive the following Windows Installer error message: This installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Installer package. If you have a verbose log file of the Setup program, the following information also appears at the end of the log file: MSI (c) (6C:54): Package name extracted from package path: 'somesetup.msi' MSI (c) (6C:54): Could not create LFN path for package: 'H:\tmp\apps\coreapps\somesetup\somesetup.msi' This installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Installer package. MSI (c) (6C:54): MainEngineThread is returning 1619.
	You cannot delete a file or a folder on an NTFS file system volume This article describes why you may not be able to delete a file or a folder on an NTFS file system volume and how to address the different causes to resolve this issue.
	How to use Xcacls.vbs to modify NTFS permissions There is an updated version of the Extended Change Access Control List tool (Xcacls.exe) that is available as a Microsoft Visual Basic script (Xcacls.vbs) from Microsoft. This step-by-step article describes how to use the Xcacls.vbs script to modify and to view NTFS file system permissions for files or for folders. You can use Xcacls.vbs from the command line to set all the file system security options that are accessible in Microsoft Windows Explorer. Xcacls.vbs displays and modifies the access control lists (ACLs) of files.
	Cannot View NTFS Logical Drive After Using Fdisk If you start Windows XP in a dual-boot environment with Windows 95 or Windows 98, use the Fdisk tool to delete a logical drive using the File Allocation Table (FAT) file system, and then restart Windows XP, you may no longer see logical drives within the Logical Disk Manager in Windows XP. For example, this behavior may occur if you do the following: You configure your computer to dual-boot between Windows XP and Windows 95 with a primary FAT file system partition as drive C. In Windows XP, you configure two logical drives: Drive D using NTFS Drive E using the FAT file system When you run Fdisk, you can view only the logical drive using the FAT file system (which is labeled drive D by Fdisk but is drive E in Windows XP). When you attempt to delete drive D, you delete the NTFS logical drive instead.
	Error message when you try to open files on an NTFS file system volume on a Windows XP-based computer: "Stop 0x0000008E" You have a computer that is running Microsoft Windows XP Service Pack 2. You have one or more NTFS file system volumes on the computer. You have a third-party driver installed on the computer. When you try to open files on a NTFS file system volume, you may experience the following symptoms: Your computer automatically restarts. After you log on, you receive the following error message: Microsoft Windows The system has recovered from a serious error. A log of this error has been created. Please tell Microsoft about this problem. We have created an error report that you can send to help us improve Microsoft Windows. We will treat this report as confidential and anonymous. To see what data this error report contains, click here. If the error message still appears, and if you want to see the data that the error report contains, click the click here link at the bottom of the message box. Then, you see error signature information that may resemble the following: BCCode : 0000008E BCP1 : c0000005 BCP2 : f83ef720 BCP3 : a91e8844 BCP4 : 0 OSVer : 5_1_2600 SP : 2_0 Product : 256_1 You receive the following "Stop" error message: A problem has been detected and Windows has been shut down to prevent damage to your computer Technical information: *** STOP: 0x0000008E (c0000005, f83ef720, a91e8844, 0) KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
	"Windows could not start because the following file is missing: \system32\drivers\ntfs.sys" error message in Windows XP Service Pack 2 If you install the 826939 Update Rollup 1 for Windows XP on a Windows XP SP2-based computer, the computer may not always start. You may receive a message that says that the \System32\Drivers\Ntfs.sys file is missing.
	The Default Cluster Size for the NTFS and FAT File Systems This article describes and lists the default values that Windows XP uses to format a volume. The article lists default values for both the NTFS file system and the file allocation table (FAT) file system.
	The partition size is extended, but the file system remains the original size when you extend an NTFS volume n Microsoft Windows XP and in Windows Server 2003, after you use the Disk Management snap-in or the Diskpart.exe command-line utility to extend a basic or dynamic NTFS file system volume, the partition size is extended, but the file system remains its original size. You do not receive an error message, but when you view the disk information in the Disk Management snap-in, the volume appears as the extended partition size, but the value in the Capacity column still shows the original size. If you view the properties of the volume in My Computer, or if you run the Chkdsk.exe tool against the NTFS volume, both items report the file system size as it was before the extension.
	You receive a "This folder already contains a file named ''" message when you copy an NTFS encrypted file to a new location on a Windows XP Service Pack 2-based computer Consider the following scenario: You have installed file security products on a Microsoft Windows XP Service Pack 2 (SP2)-based computer. You are using drives that are formatted to use the NTFS file system. You copy an NTFS encrypted file to a new location on the computer. In this scenario, you receive the following message: This folder already contains a file named 'filename'. Would you like to replace the existing file filesize modified date and time with this one? filesize modified date and time You receive this message even though you copied the encrypted file to a folder that does not already contain the encrypted file.
	Extending NTFS volume fails but appears to be successful When you try to extend an NTFS file system volume on a basic disk, or when you try to extend a dynamic NTFS volume, the operation is not successful. In Disk Management, the partition appears to be extended and appears to use the additional free space, showing that the volume was successfully spanned across disks. However, the NTFS on that volume is still the original size that it was before the extension. In My Computer, the properties of the volume show that the volume is the same size that it was before the extension. If you run Chkdsk on the NTFS volume, it reports the volume size that existed before you tried to perform the extension. This problem may also occur if you use the unattended parameter extendoempartition=1 during an unattended Windows setup and the boot disk is larger than approximately 200 gigabytes (GB). Setup successfully extends the partition but fails "silently" to increase the NTFS. After Setup has completed, the system volume properties as shown in My Computer show that the volume is the same size as it was before the installation.
	You Can View an NTFS Encrypted File in Thumbnail View If you encrypt a file by using the NTFS file system, and the view is set to Thumbnail, the contents of the file may still be visible. For example, if you encrypt an image file or an HTML file, and then you set it to the Thumbnail view, you will be able to see the image or HTML image on the thumbnail.
	Computer Stops Responding (Hangs) When It Writes Encrypted Data to an NTFS Partition In certain situations, your computer may stop responding, or hang, when it writes files or folders that are encrypted with the Encrypting File System (EFS) to a partition that is formatted with the NTFS file system. When your computer hangs, you cannot access the contents of the partition, and you have to restart your computer to restore functionality. This problem has been reported to occur when your computer restores data to encrypted files by using VERITAS Backup Exec version 8.6 or later or by using NTBackup.exe. This problem may also occur when your computer writes EFS data with other programs.
	Intermittent memory corruption occurs in the binary data that is stored on a compressed NTFS file system drive on a Windows XP-based computer Intermittent memory corruption occurs in the binary data that is stored on a compressed NTFS file system drive on a Windows XP-based computer. The computer cannot run programs that use this binary data.
	How to locate and correct disk space problems on NTFS volumes in Windows XP The NTFS file system supports many volume- and file-level features that may cause free disk space to be either misreported or reported as lost. You may notice this behaviour if an NTFS volume suddenly becomes very full, and you cannot find the cause or locate the folders and files that cause the NTFS volume to become full. This behaviour may occur if a user gains malicious or unauthorized access to an NTFS volume on which either very large files or a high quantity of small files are secretly copied, and then removes or restricts NTFS permissions on these files. This behaviour may also occur after a system malfunction or a power outage that causes volume corruption to occur. This article describes how to check NTFS disk space allocation to either discover offending files and folders or locate volume corruption. This article is intended for users of Windows XP operating systems that support advanced storage features and troubleshooting methods.
	How To Disable the NTFS File System Tracking of Broken Shortcut Links If you disable a shortcut, the NTFS File System in Windows XP and Windows 2000 automatically attempts to locate the shortcut destination by searching all paths that are associated with the shortcut. This step-by-step article describes how to prevent this behaviour from occurring.
	How to create and use NTFS mounted drives in Windows XP and in Windows Server 2003 This article describes how to create a mounted drive by using Disk Management in Microsoft Windows XP and in Microsoft Windows Server 2003. A mounted drive is a drive that is mapped to an empty folder on a volume that uses the NTFS file system. Mounted drives function as any other drives, but they are assigned drive paths instead of drive letters. When you view a mounted drive in Windows Explorer, it appears as a drive icon in the path in which it is mounted. Because mounted drives are not subject to the 26-drive-letter limit for local drives and mapped network connections, use mounted drives when you want to gain access to more than 26 drives on your computer. For example, if you have a CD-ROM drive with the drive letter E, and an NTFS volume with the drive letter F, mount the CD-ROM drive as F:\CD-ROM. You can then free the drive letter E, and gain access to your CD-ROM drive directly by using F:\CD-ROM. You can also use mounted drives when you need additional storage space on a volume. If you map a folder on that volume to another volume with available disk space (for example, 2 gigabytes), you extend the storage space of the volume by 2 gigabytes (GB). With mounted drives, you are not limited by the size of the volume in which the folder is created. Mounted drives make your data more accessible and give you the flexibility to manage data storage based on your work environment and system usage. These are additional examples by which you can use mounted drives: To provide additional disk space for your temporary files, you can make the C:\Temp folder a mounted drive. When space starts to run low on drive C, you can move the My Documents folder to another drive with more available disk space, and then mount it as C:\My Documents. Use the Disk Management snap-in to mount a drive on a folder on a local volume. The folder in which you mount the drive must be empty, and must be located on a basic or dynamic NTFS volume.
	You may receive an error message when you try to rename a folder on an NTFS partition in Windows XP When you try to rename a folder on an NTFS file system partition in Microsoft Windows XP, you may receive an error message that is similar to the following: Cannot rename foldername: Access is denied. Make sure the disk is not full or write-protected and that the file is not currently in use.
	Error message and events are logged in the System log when you try to compress a large file on an NTFS volume in Windows XP, in Windows 2000, or in Windows Server 2003: "Delayed Write Failed" Consider the following scenario. You have a computer that is running Microsoft Windows 2000, Microsoft Windows XP, or Microsoft Windows Server 2003. On this computer, you have a very large file on an NTFS file system volume. You try to compress the file by using NTFS File Compression, or you try to copy the file to an NTFS compressed folder. In this scenario, you may receive the following error message: Delayed Write Failed Note: This article applies to Microsoft Windows XP Home Edition, Microsoft Windows XP Professional (32-bit), and Microsoft Windows XP Professional x64 Edition.

Creating the first Windows Server 2003 Domain Controller in a domain

Preface:

One of the greatest features of Windows Server 2003 is its ability to be a Domain Controller (DC). The features of a domain extend further than this tutorial ever could, but some of its most well known features are its ability to store user names and passwords on a central computer (the Domain Controller) or computers (several Domain Controllers). In this tutorial we will cover the "promoting" (or creating) of the first DC in a domain. This will include DNS installation, because without DNS the client computers wouldn't know who the DC is. You can host DNS on a different server, but we'll only deal with the basics.

Method:

Click Start -> Run...

Type "dcpromo" and click "OK"

You will see the first window of the wizard. As it suggests, I suggest reading the help associated with Active Directory. After this, click "Next"

Click "Next" on the compatibility window, and in the next window keep the default option of "Domain Controller for a new domain" selected, and click "Next"

In this tutorial we will create a domain in a new forest, because it is the first DC, so keep that option selected

Now we have to think of a name for our domain. If you own a web domain like "visualwin.com", you can use it, but it isn't suggested because computers inside of your domain may not be able to reach the company website. Active Directory domains don't need to be "real" domains like the one above - they can be anything you wish. So here I will create "visualwin.testdomain"

Now in order to keep things simple, we will use the first part of our domain ("visualwin"), which is the default selection, as the NetBIOS name of the domain

The next dialog suggests storing the AD database and log on separate hard disks, and so do I, but for this tutorial I'll just keep the defaults

The SYSVOL folder is a public share, where things like .MSI software packages can be kept when you will distribute packages (as I said, AD has a lot of different features). Once again, I will keep the default selection but it can be changed if you wish to use the space of another drive

Now we will get a message that basically says that you will need a DNS server in order for everything to work the way we want it (i.e., our "visualwin.testdomain" to be reachable). As I mentioned earlier, we will install the DNS server on this machine as well, but it can be installed elsewhere. So keep the default selection of "Install and configure", and click "Next"

Because, after all, this is a Windows Server 2003 tutorial website, we'll assume there are no pre-Windows 2000 servers that will be accessing this domain, so keep the default of "Permissions compatible only with Windows 2000 or Windows Server 2003 operating systems" and click "Next"

The restore mode password is the single password that all administrators hope to never use, however they should also never forget it because this is the single password that might save a failed server. Make sure it's easy to remember but difficult to guess

Now we will see a summary of what will happen. Make sure it's all correct because changing it afterwards can prove to be difficult

After the previous next was clicked, the actual process occurs. This can take several minutes. It's likely that you will be prompted for your Windows Server 2003 CD (for DNS) so have it handy

If your computer has a dynamically assigned address (from DHCP) you will be prompted to give it a static IP address. Click ok, and then in the Local Area Connection properties, click "Internet Protocol (TCP/IP)" and then "Properties"

In the next window select "Use the following IP address" and select the information that you will use for your domain (and 127.0.0.1 for the primary DNS, because your computer will host DNS. I still suggest setting up an alternate as well.) Click "OK" and then "Close" on the next window

And after a while you will see

And we're finished. You may also want to see the other Active Directory tutorials on the main page.