RAID5/6 performance and reliability impact

I know you don't want to hear it, but a fact is a fact: the number one reason for loss of electronic data is user error. Inadvertent deletion, accidental formatting, hot coffee spilled on your laptop, the dog ate the homework, etc. Spilled coffee and accidental formatting happen way more often than hard disk crashes, at least where I hang out. (Why is there a "dental" in "accidental formatting"? It can't be a coincidence with the two of them causing most of the world's pain.)

There is really no foolproof 100% protection from user errors. Users are just too imaginative and sneaky. No matter how well their computers try to protect themselves against their owners, the owners find these inconceivably spectacular ways to lose their baby photos and tax records. Yes, tax records fall victim to more hard disk crashes than there are hard disks in existence, and the IRS is investigating this matter. The dogs are planning a class action suit, too.

There actually is a way to protect against user error: frequent backups. Do it yourself with utter frequency and consistency. Also, backup to the cloud.

The number two reason: hardware error, such as hard disk failure, or some sort of a crash that mangles or destroys your data. Here, there are two ways to protect yourself: automatic data redundancy and backups. The first option will not save you from user error; only from hardware errors and crashes - and even then - not always. The latter option is the only relatively foolproof to protect your data. Backup your data and backup often.

Now that I am done with the Computer Consultant's Number One Mantra, I will concentrate on data redundancy in the form of RAID or Redundant Array of Independent Disks.

This is going to be boring.

The two most common desktop RAID levels are 0 and 1 where The Zero isn't redundant at all and I have no clue why it is still called "redundant" but it's a long running tradition and I like to follow traditions with the exception of popcorn at the movies. I neither understand nor follow this strange tradition of drowning the 24-speaker surround sound exquisitely crafted by Hollywood, with popcorn crunching right between your ears. You do? Write a comment.

With RAID levels 0 and 1 being the most common, there is a growing trend of RAID levels 5 and 6 taking ground on home computers used for storing photos and videos. The reason is simple: 0 and 1 are not good enough: 0 is not protected, 1 isn't very efficient. 0 simply distributes bytes onto several drives storing only one copy of each byte. This makes the set faster but completely unprotected: one disk fails and your entire data stored on the set is gone. Not only Zero is unprotected, it puts your data at a higher risk than a single drive. The Mantra above is especially important with the Zero, please repeat after me:

Backup your data and backup often.

I warned you it was going to be boring.

RAID level 1 puts each byte onto two drives simultaneously. Each byte is stored twice, so if one drive fails, there is still a copy on the other drive. The data is protected right up until one of the drives fails. Once it fails, and eventually, they all do, the data is not protected until the RAID1 set is rebuilt again, i.e. the failed drive is replaced with a new one, and the set is restored to the "healthy" status. Which means RAID1 not only takes two drives to protect one, it is also not a 100% protection against drive failure. (Repeat after me...) In the end, the cost of protection is 50% of the total capacity: in a RAID Level 1 set of two 1TB disks, the total usable capacity of 1TB is half of the total capacity of 2TB.

With gigabytes and terabytes getting cheaper, lighter and physically smaller, it's not far-fetched to put 4 or 5 of them in a computer and try to protect them against a failure of one. Or two. Enter RAID levels 5 and 6: that's exactly what they do. Level 5 protects against a failure of a single hard disk in any RAID5 set, and Level 6 - against two. The cost of this protection is in a loss of a capacity of one or two disks, respectively. In other words, a RAID Level 5 set of five 1TB disks will have a usable capacity of 4TB, and a RAID Level 6 - 3TB. This is more efficient than RAID Level 1 as the usable capacity is more than 50%. There are also other RAID levels we will not touch in this article, as they are far less common than 0, 1, 5 and 6.

That said, RAID Levels 5 and 6 put a much heavier load on individual hard disks compared to other levels, notably 0 and 1, and heavier duty (enterprise level) hard disks are recommended for these configurations. Individual hard disks in RAID5 and RAID6 arrays have two parts: data and parity. Each individual write operation to an array will consist of two resultant write operations to each hard disk: to its data portion, and to its parity portion. The potential performance penalty is significant and can range from 10-15% to 90%, depending on an application, drive and controller characteristics. While disk and controller caches often reduce the penalty and improve RAID5/6 write performance, the fact that each drive has to do a double duty for each write operation, still remains. It is thus recommended to use heavy duty hard drives with longer MTBF numbers, designed for enterprise applications, in RAID5 and RAID6 arrays, and their derivatives.

Examples of "lighter duty" desktop drives not recommended for RAID5/6 arrays with moderate to heavy performance loads:

• Seagate Barracuda 7200.11 ST31500341AS 1.5TB 7200rpm: 3-year warranty, 700K Hrs MTBF
• Western Digital Caviar Black WD1001FALS 1TB 7200rpm: 5-year warranty, 750K Hrs MTBF

Examples of "heavy duty" enterprise hard disks recommended for high performance RAID5/6 applications:

• Seagate Barracuda ES.2 ST31000340NS 1TB 7200rpm: 5-year warranty, 1.2m Hrs MTBF
• Western Digital RE4-GP WD2002FYPS 2TB 5400-7200rpm: 5-year warranty, 1.2m Hrs MTBF

Note: I am not including enterprise class hard drives with 10,000rpm or higher rotational speeds, or SSD models, because they are still quite a bit more expensive than their 7200rpm counterparts, and are usually cost-prohibitive for mainstream video editing applications.

Sources:

• LSI/3Ware RAID 6 Tech Brief (PDF)
• Treadlayers' Review of LSI 3Ware 9690SA-8I RAID controller (Aug 17, 2009)

Liquid Cooling Goes Mainstream with HP Z800

Quietly and without much fanfare, Hewlett Packard began offering liquid cooling on its Z-series workstations. Liquid cooling kits have been around for years, usually as after-market options for overclockers. With HP offering it as a standard item, liquid cooling becomes less of "living on the edge" gaming affair, and more of a mainstream option, albeit for expensive top shelf models of their workstations.

While there seem to be no pre-configured models with liquid cooling kits installed, the Z800 "configure your own model" page has two options under "Thermal Kit": "HP Air Cooling Solution [Add $1], and "HP Liquid Cooling Solution [Add $250]". The latter is only supported with "High Power" processors, Intel models W5580 (3.2GHz) and W5590 (3.33GHz).

A brief trip to Google uncovered a great article on ComputerWorld that was just posted yesterday. Besides reducing power-sapping heat, liquid cooling serves another purpose that is critical for workstations: reducing fan noise. According to the article, the noise is cut down by 8 dB under heavy load (from 38 to 30 dB), and this is a big deal for those of us who don't enjoy monotonous and often annoying fan humming.

A well configured base liquid-cooled Z800 workstation will only run you $10,698.00. Peanuts for those of us who appreciate quietness. Here are the specs:

HP Z800 Workstation FF825AV-PR800
• Genuine Windows Vista® Business 64-bit
• HP Z800 1110W 89% Efficient Chassis
• 2 Intel® Xeon® W5580 3.20GHz 8MB/1333 QC CPUs
• HP Liquid Cooling Solution
• HP 24GB (12x2GB) DDR3-1333 ECC RAM (Dual Processor)
• HP NVIDIA Quadro FX3800 1GB Graphics
• HP 300GB SAS 15K 1st HDD
• HP SATA Blu-Ray Writer
• HP 3 year standard warranty

$10,698.00 (buy it from DV411)

Availability: HP usually ships "configure your own" models within 1-2 weeks.

Additional information:

• Listen to samples of noise produced by air- and liquid-cooled Z800 workstations in this review: Liquid-cooled HP Z800 Workstation Test Drive (Jan Ozer, Aug 10, 2009)
• HP Z800 home page
• Asetek Technology Helps Improve Acoustics on Ultra Quiet Workstations from HP, Aug 09, 2009

HP Z800: The BMW M6 of Workstations

Under the hood, Z800 looks just like BMW M6 10-cyl 500hp GT cruiser: you can see the oil dipstick, everything else is shiny plastic and metal. To access slots, drives, memory and CPU, just lift off plastic panels: no tools required. The workstation can be taken apart and put back together in seconds with bare hands.

Speaking of bare hands, the workstation now features case handles. Unboxing, boxing, and simply moving the rather heavy system (45 lbs net) suddenly became so much easier. Unlike IBM designs, Z800 has two handles, and unlike Mac Pros, the handles are cylindrical and will not cut your hands.

Even the "high power" models (3.2GHz and up) are nearly silent. The fans kick up just a bit under full CPU load, yet it's quieter than xw8600 under load - and much, much faster.

Up to six SAS or SATA 3.5" drives can be fit inside, 4 in pluggable drive cages, 2 more - in optical bays, using special "optical bay mounting kits". All installed without a single screw.

My favorite configuration for a top-of-the line video editing system: 300GB 10K rpm SAS system drive, five 2TB 7200rpm SATA drives in RAID0 mode on the LSI controller, which gives a total of 10TB capacity and about 500MB/s transfer rates, enough for realtime uncompressed HD and 2K editing and previewing.

Fast, quiet, stable, well designed, easily expandable and upgradeable - what more can one ask for? It's the best workstation HP has come up with.

Eager to get one configured for video editing or visualization applications? See Z800 on DV411.

The broadband water for your business

They say, a person can live just a few days without water. How long will our business live without broadband Internet service? Today, I found out: not long. AT&T DSL had an area outage, and we were without Internet for three hours. No email. No web access. Not even a phone service: we have VoIP.

The business stops dead in its tracks, without Internet access. I spent two hours on my cell phone calling AT&T technical support. I even spoke to the Tier 2 Tech Support people (gasp!). They are notoriously great yet usually impossible to get to, thanks to the lovely AT&T hold music. They all suspected that the problem was on my side. Three hours later, the service is back on, and I find out that it was an area-wide outage. Knowing that three hours earlier would have been helpful - but thanks anyway, AT&T.

If I wasn't too motivated to order a backup broadband service before, I sure was now. Time to call my old friends at LaunchNet, Inc., who installed and serviced two broadband connections for us in the past, and did it quicker and less expensively than their major telecom siblings.

AT&T's best "business DSL" offer for our location is a 6Mbs/768Kbs service with 8 static IPs, for $70 a month plus taxes and fees. LaunchNet offers the same for $59.95 a month, with a choice of two DSL modems. One is a "bridge" that has very little IQ to it, and simply acts as an adapter between the DSL service and your existing router. The other choice is Zoom X6 DSL modem and wireless router, with many more configuration, diagnostics and monitoring features, and this is the one I ordered.

In short, over the two years that we have been using LaunchNet as our broadband service provider, it proved to do the same as primary telecoms like AT&T, Covad and Verizon, only for less money and with much, much better customer service.

If broadband to your business is like water to a human being, it makes sense to have a reliable supplier. For us, it's LaunchNet:

http://www.launchnet.com/

Unless, of course, AT&T hold music tickles you pink. :)

Security Tips from the Dark Side

Never, ever, connect strange computers to your network. I did, and however briefly, involuntarily became a spammer. While our network is well protected and safe, the client's heavily infested box managed to fire a salvo or two of spam out of it. Since then, we've taken extra measures to ensure this never happens again, and even if it does, to ensure no spam ever comes out of our domain - by closing the standard SMTP port to anyone except the server. Still, this incident clearly demonstrated the importance of basic protection measures.

1. Do not connect strange computers to your network. Fire them up first and scan them for malware. If the computer is infected especially badly, consider taking its hard disks out and scaning them on a separate, well protected machine.
   
2. Scan them regularly with a good anti-malware agent. We use AVG, Microsoft Malicious Software Removal Tool, and now testing Microsoft Security Essentials, an amazingly lightweight anti-malware agent.
   
3. Do not use unknown anti-malware tools. Do not click on "free virus scan" ads the web is peppered with. Use only well-known tools from the likes of Microsoft, AVG, Symantec, etc.
   
4. Make sure your administrative accounts are password-protected. If the administrator account has no password, make one and do it now. Even if it's just two characters, this is the very first step in protecting your computer and saving yourself from embarrassment. Better yet, make it a 6-8 character password that has lower case letters and capitals, as well as a digit or two, and a special character like "{" or "$". Having a password in plain view on a stickie on your monitor is bad, but way better than having no password at all.
   
   Why: an infested computer will commonly try to infest other computers on a network by probing open shares. If your computer has an "open" (no password) administrative account, the chances of getting infested are much higher.
   
   How: in most Windows versions, log into your administrative account, hit Ctrl-Alt-Del, click on "change password". In XP and Vista, you may have to go to Control Panel, User Accounts, and change password(s) that way. Go through all user accounts with administrative privileges and ensure they all have passwords.
   
   


5. Quit using your "administrator" account. After you created a decent password for your "administrator" account, quit using it. Administrative accounts are primarily for administrative tasks: major software and hardware installations, other users' password resets, joining and disjoining domains and workgroups, diagnostics and trouble-shooting. Do not use them for general and day-to-day tasks.
   
   How: create a new account, call it "Hugo" (if that's your name), assign it administrative privileges if you must, set a decent password. Thank you. You just made my life, or a life of your "computer guy" much easier. He will thank you, but most importantly, you will thank yourself later, more than once.
   
   


6. Never ever under any circumstances share a whole drive, and specifically, your "Windows" or "Program Files" folders. XP, Vista and Windows 7 already have "Shared Documents" or "Public Folders" that are shared among users on the same computer, and can also be shared on the network. It is a good practice to only use that folder for shared docs and files.
   
   


7. Have a knowledgeable computer person check your computer every few months. Ensure it's patched, virus-free, lint and dust-free inside (saves electricity - serious!), junk-free: old temporary files - purged, applications you no longer use - uninstalled.

These simple steps, together with regular maintenance, will make your computers and network much less vulnerable to malware, and nearly eliminate the chances of becoming an involuntary spammer.

Happy Friday! :)

NAB and other news I think is cool!

NAB News

Lets start with today's good news on the Apple front. As many of you share my frustration with apple's hardware shortsightedness over the years, the latest graphics cards choices of "Crap and Crappier" were offered with the latest "Nehalem" systems in late march. I am now happy to say that Nvidia will be releasing a Mac version of the Quadro 4800! probably available in May or June. All you guys who bought the new Mac Pro and found your AFX/Shake/Color performance wanting will be very happy with this model. It's going to be expensive, but far less than the 5600 that apple offered with it's "Harperetown" systems, I am thinking around $2K.

Head over to the Nvidia website for specs and yummyness.

As you all know, I was working in the AJA booth at NAB this year which was quite an experience. Crazy is the best term to describe it. I heard rumors and garnered some hints about what was coming but I was abjectly refused an NDA no matter how much I begged. So literally a day before the show I head over to the booth to find out what I'll be talking about for the next week hoping I won't make a fool of myself to the masses. When I got there all I could say was "AWWWW YEAAHHHH" (obviously I'm talking about the Ki Pro DDR; the other new products are good but... come on!) From the mind of John Thorn comes this little recorder that really covers almost every production problem I have ever encountered. Right, I need a new paragraph.

OK let me start with the problems my clients have always encountered when shooting. There are 80 million different cameras with different acquisition codecs that are all equally a pain in the ass to edit. From high quality long GOP formats to low bitrate anamorphic crap, no matter what.. you were crashing if you tried to edit the stuff natively. One always had to convert it to something while usually destroying it's initial quality. One of everyone's' favorites being Prores HQ. Now 2 years ago I had my worries about it, and who could blame me... seriously 10-bit 4:2:2 codec at about 220 Mb/s in 1080p (not to mention my variable bit rate transcoding worries). However 2 years later, most of my worries have been unfounded. Enough on that, and on to the mind of Thorn and the very very talented engineering team at AJA.

So the KI Pro is essentially a DDR that bridges the gap between production and post, all in an easy to use lightweight low power kind of way. In a nutshell, you have every input source you can currently use (HD/SD SDI with embedded audio, HDMI v1.3, Component, XLR, etc.) and every output source (those plus Composite SD and Headphones) with hardware up/down/cross conversion. This means "In" from any camera and "out" to any monitor, so no matter what camera(s) you have this unit will work with it. Add the optional (and also lightweight)
Exo-skeleton surrounding chassis, to mount the Ki Pro between a camera and a tripod, and rod kits and you're good for virtually any tripod/steadicam/handheld setup.

On to recording. The Ki Pro records data into ProRes or ProRes HQ (around 100 and 220 Mb/s respectively) onto 250GB magazines (which will be priced around $300, more for the SSD version out later). This means not matter what camera you use, from your $500 AVCHD b-roll camera to your F35, you are going to instantly access a 4:2:2 10-bit file that you can edit frame accurately and in full raster and frame rate (up to 1080p 50/60) right then and there. The magazines are as easy to remove and replace as tapes and have a FireWire-800 port on the back so you can plug them right into you laptop (they will auto mount like any regular disk).

And the really cool part! You an control it wirelessly for anything that can open a webpage. It come with a built-in wireless Ethernet. All you have to do is connect it to your router by typing in your WEP or WPA password (your router dooes have a real wireless password, doesn't it?), then watch it shows up among your network devices, click on it, and up pop all your controls. You can even do it from your i-Phone you mac hungry gadget heads!

Right, it's time to go home and I'm rambling. There is soooooo much more cool stuff you can do with the Ki Pro that I'm not getting in here. Read up on it at the AJA website.

More NAB news tomorrow... Maybe the next day.

Optimizing HD uploads for YouTube

YouTube has become a very popular video sharing site, and recently, it started to let users upload HD videos. While these HD videos won't match Blu-ray or Dish HD quality, they still look fantastic compared to standard YouTube videos, which, to be honest, look crappy at best.

The questions often comes up, how to best save an HD video, to upload it to YouTube?

While YouTube's "Optimizing your video uploads" help article does recommend a resolution of 1280x720 for 16x9 HD videos, and lists recommended codecs (H.264, MPEG-2 or MPEG-4 preferred), it does not recommend a specific bitrate. What's a man to do, when a 5-minute video can be as big as 700MB in HDV format, and take many hours to upload?

Experimenting with various bitrates and types of encoding helped me narrow down the optimum settings for YouTube's HD videos.

YouTube re-compresses the videos anyway, so bigger files will not always produce better quality. I.e. uploading raw HDV videos (at 19-25Mbs) will be horrendously slow over a standard DSL connection, and the quality will not be visibly higher vs. videos encoded at 3Mbs H.264.

Certainly, a 25Mbs video will look better than a 3Mbs one, before it's uploaded to YouTube. Once it's uploaded, however, YouTube will "process" it, making it as small as possible while trying to maintain its quality, and once processed, a 25Mbs video is unlikely to look any better than a 3Mbs one.

I did a number of test encodes and uploads, starting at 500Kbs and ending at 7Mbs. After a 3Mbs "sweet spot", all tests looked very similar on YourTube, in HD mode. There were still barely noticeable differences between a 3Mbs and a 4Mbs test videos, but they were insignificant to my eye, and I did look really hard in full screen mode, on a 30" professional LCD monitor. After 4Mbs, these differences disappeared altogether. In other words, a 7Mbs video looked exactly the same as a 4Mbs one.

The encoding was done with a Adobe Media Encoder (part of Adobe Premiere Pro CS3), using Mpeg4 H.264 codec, single pass VBR (Variable Bit Recording).

To summarize, 3Mbs Mpeg4 (H.264) VBR (Variable Bit Recording) seems to be the optimum setting for most videos, to encode for YouTube HD. Any higher, and it's unlikely that YouTube HD viewers will feel it. At 2Mbs, it's not bad but you will see more compression artifacts than at 3Mbs, and 1.5Mbs will make YouTube think it's HD, but the quality will not be that great.

The resultant files are not exactly small, about 22MB per minute of video, but still far smaller than raw HDV, which is 142 to 187MB per minute. At 3Mbs, a 5-minute video will be about 110MB in size, and take about 25 minutes to upload over a 3Mb/768Kb DSL connection. A similar HDV video will take close to 3 hours to upload.