Saturday, January 26, 2008

Modern disk drives will automagically reallocate bad sectors on the fly, as soon as they encounter some kind of R/W/ECC error. But in order for this to happen, it must first access that sector. This is why you never see surface errors on modern disks.

Modern hard drives (ATA and SATA) have S.M.A.R.T. - Self-Monitoring, Analysis, and Reporting Technology. Once you have that enabled in BIOS (assuming you have a S.M.A.R.T. capable disk and controller) you can monitor a number of disk health and performance parameters.

What you should keep an eye on is the Reallocated Sectors Count (if the drive has a problem with a R/W/ECC error it will mark the sector "Reallocated" and transfer the data to a spare area on the disk). This will result in some performance decrease, and is a sign of imminent disk failure.

Monitoring S.MA.R.T.

ATA and SATA disks:To monitor S.M.A.R.T. data you can use HDTune on Windows or SmartMonTools (smartd, smartctl) on Darwin (Mac OSX), Linux, FreeBSD, NetBSD, OpenBSD, Solaris, OS/2, or eComStation systems. If you're up to it, you can also use SmartMonTools on Windows.

USB Enclosures:While in most cases you should have no trouble using HDTune or SmartMonTools, some USB drive enclosures may be resilient to monitoring with S.M.A.R.T. programs and will require vendor software. In such cases, you can download vendor software to perform monitoring, like "Western DigitalData LifeGuard Diagnostics".

iPods:You can also get S.M.A.R.T. info on your iPod. You can either configure it to act as a pass through device (regular USB media) or boot your iPod in diagnostic mode. You can check S.M.A.R.T. disk data and perform more test on your iPod. To do so, you must reset your iPod and hold REW + Select (5G) at the Apple boot menu. For other iPod models, see here (or Google Apple Diagnostic Mode your iPod Model).

Forcing the disk to remap damanged sectors

Now you should know that if you see any problems with Reallocated Sector Count, Reallocated Event Count, Seek Error Rate, Offline Uncorrectable, UDMA CRC Error Count, Multizone Error Rate, Hardware ECC Recovered values, you should consider getting a new disk. These are all signs of a failing disk. Learn more about S.M.A.R.T. attributes and their meaning here. Note that depending on vendor, there may also be enhanced or propriotary S.M.A.R.T. attributes. Read your HDD vendor documentation.

But sometimes you just need to get a bit more life out of a disk, and force the disk to reallocated damaged sectors. You can do so easily by performing a full raw disk read and write operation. For this, you can use the UNIX "dd" tool. Make sure your target disks aren't mounted (Type "mount" to list mounted disks then use "umount disk").

You can perform a disk read operation (reading the whole disk) using a syntax similar to:

# dd if=/dev/disk of=/dev/null bs=2048

You can perform a disk write operation (zero out the disk, this WILL result in data loss) using syntax similar to:

# dd if=/dev/zero of=/dev/disk bs=2048

Now you may wish to perform both a read and write at the same time, and not wipe out your disk data (zero it out). You can perform such a "disk refresh" using syntax similar to:

# dd if=/dev/disk of=/dev/disk bs=1m

This will read and rewrite the data to disk in 1MB chunks to prevent presently recoverable read errors from progressing into unrecoverable read errors.

Of course, you should read the dd manpage for your OS (on Windows you could use a dd for Windows implementation or resort to some sort of Linux or BSD LiveCD). Replace /dev/disk with your disk (make sure you're using the right disk). On Linux you can find out what disk you need to use from "dmesg" or /proc/partitions:

# cat /proc/partitions

You can also use "fdisk -l" to list partitons on your disk, see if that's the right disk

# fdisk -l /dev/hda

Do note that you need root permissions for all of this activity, so on some Linux systems you may need to use "sudo -i" to get a root shell, or precede all operations with "sudo".

While you're doing this rewrite operation, you should monitor the kernel log (dmesg). You can monitor /var/log/messages for this:

# tail -f /var/log/messages

You usually watch out for "DriveReady SeekComplete Error status=0x51 DriveStatusError error=0x04" or some other error.

You should also keep an eye on the Reallocated Sectors and other Interesting Parameters in smartctl:

# smartctl -A /dev/hda

Do this every now and then, and note the values before you've started the operation.

Once you begin the "dd" operations you can send dd a SIGINFO signal (use pkill / kill / whatever) to make it print out I/O information (progress). Some shells / TERMS also respond to Ctrl-T by sending SIGINFO.

# pkill -SIGINFO dd

Once you're done with dd and S.M.A.R.T. tools you should also perform a filesystem check (fsck / chkdsk / whatever).

Conclusions:

Monitor S.M.A.R.T. data with smartclt, keep an eye on Reallocs. Consider getting a new disk if you see reallocated sectors

Perform a disk refresh with dd in order to prevent recoverable read errors from progressing into unrecoverable errors. You don't need fancy tools like SpinRite.

You can use a simple Linux or BSD LiveCD to perform the disk refresh.

This is NOT a data recovery procedure. If you're doing data recovery, use something like dd_recover to a separate media.

This is NOT a step by step tutorial. Read your OS manpages to make sure you're not wiping out the wrong disk or something.

Always monitor S.M.A.R.T. parameters in order to spot disk failure before it happens.

Thursday, January 24, 2008

Security professional Jeff Jones published a Windows Vista security report for vulnerabilities between Nov. 2006 to Nov. 2007. It also does a side by side comparison with Windows XP, RedHat and Ubuntu Linux and MacOS 10.4.

Some people on the other hand aren't too excited about the Sun - MySQL AB deal. Some even think this is a plot mastered by Larry Ellison to destroy MySQL (tinfoil hat, anyone?). Here's what Marketwatch has to say on the subject:

"BERKELEY, Calif. (MarketWatch) -- Sun Microsystems Inc. gobbling up MySQL is perhaps the worst single event I have ever witnessed in the history of tech mergers and acquisitions."

I'm also not quite sure on how will this impact Sun's long term policy on PostgreSQL. Sun offers PostgreSQL support (PostgresSQL on Solaris even has native DTrace probes), and was a big contributor to the project. I don't think they would drop PostgreSQL support though. Maybe port PostgreSQL to MySQL as a storage engine? :-).

I'd still like to know how Sun expects to make money off MySQL. Well, enough money to justify the 1 billion investment. Sun does tend to make money off support contracts, services, hardware and all that, but MySQL AB hardly managed to pull in $50 million last year. And they have around 350 employees. And besides, nothing stopped Sun from selling MySQL support and all that before.

"Sun expects to report revenues for the second quarter of fiscal 2008 of approximately $3.60 billion, an increase of approximately 1 percent as compared with $3.57 billion for the second quarter of fiscal 2007. Net bookings for the second quarter of fiscal 2008 were approximately $3.85 billion, an increase of approximately 7% year over year."

As of January 2005, Oracle provides Critical Patch Updates on Metalink every quarterly schedule to address significant security flaws and recommended updates (required for security fixes). So, how is this all working out? Well, see for yourself...

"Complexity of task makes admins not want to bother":This research shows that "Two-thirds of Oracle DBAs don't apply security patches"

2/3? IMHO it's more like 9 out of.. 8.

""In fact, a good two-thirds of all Oracle DBAs appear not to be installing Oracle's security patches at all, no matter how critical the vulnerabilities may be, according to survey results from Sentrigo Inc., a Woburn, Mass.-based vendor of database security products.""

Even with development tools, GCC has ProPolice / SSP, Visual Studio has the /GS switch to protect against buffer overruns.

It's pretty clear, the security features are there, it's up to LAYER 8 (you!) to put it in practice.

The key idea here is *mitigation*. Don't abuse the Administrative accounts, read and apply those security guides and above all, use common sense. After all, Microsoft runs Windows on their servers (they even run 2008 while it's still in release candidate stage), and they're one of the biggest targets for abuse. There goes the argument that you "can't secure Windows".

What about the whole "open source" - many eyes concept? Doesn't this mean Microsoft is horribly insecure? What about 3rd party code reviews?

That whole concept is highly overrated. 99% of open source users never seen a line of code in their lives. Simple as that. Just because you can install Ubuntu doesn't make you a kernel developer. Don't get me wrong, I love Open Source software, I'm just not rushing to make any claims about how the opens source development model adds security (remember, you can have a whole lot more malicious people look at the code then developers).

Anyway, Enterprise customers can still get access to Windows and other Microsoft sources through various Shared Source programs:

"The ESLP allows eligible enterprise customers access to Microsoft Windows source code for internal development and support purposes, including debugging. This enables customers to develop and support their internally deployed applications and solutions that run on Windows."

Saturday, January 12, 2008

This is pretty interesting considering your basic CPU does something like 30GFLOPS (something around 16 GFLOPs per POWER 6 cores, 10GFLOPS for a Itanium cores). A cell board like this does 180GFLOPs.(Don't take this is a benchmark or anything. This is just some RAW data).

Some NVDIA something like 500 (technically the G80 has 128 fp32 ALUs @ 1350MHz with MADD - about 350 GFLOPs), a R600 is supposed to have like 500 and a Realizm 800 (Dual Wildcat VPUs) about 700 GFLOPS :-). So yeah, with 16 or so of these cards used right, you could score yourself a place on TOP500 SuperComputers. "Hey, my 4 graphic stations can beat your 1000-node Xeon cluster!".

And this is no joke, since GF8 series and the whole NVIDIA CUDA thing, NVIDIA has also started making... erm.. servers.

There's an awesome potential HPC market here... GPUs, Playstation 3s with Cells, Cell PCI-E cards... exploited properly, it can make some pretty fast clusters. See Folding@Home for example where where GPUs count for 58.3 and PS3's count for 18.1 average computations per client.

A quick look on netcraft reveals that Microsoft is running Windows 2008 / IIS 7 on their main webservers. I must say, they must have a great deal of confidence in the thing... microsoft.com is a *very* loaded website, and tends to be a big target for various attacks.

Friday, January 11, 2008

eComStation is an operating system based on IBM's OS/2 Warp. They are soon about to launch the 2.0 version containing bootable JFS and SMP support. It also contains interesting open source software such as OpenOffice 2 and Firefox 2.

Overall, it is a very polished OS/2 with new features to bring it in the 21st century. They also have a server version of eComStation.

Just like OS/2, it seems to dislike VMware or VirtualBox, so it runs a bit better in Microsoft's VirtualPC (which seems to have been made for OS/2 in the first place).

Wednesday, January 09, 2008

LogParser is an awesome Windows application for parsing through system logs (event viewer, Registry, Active Drectory, IIS, etc) using a somewhat SQL-like syntax. It can also generate detailed reports, graphs, etc. Also makes a nice starting point for various Data Mining operations on your logs.

VisualLogParser is a GUI for LogParser. It is available on Microsoft's Open Source project page, Codeplex.

Tuesday, January 08, 2008

Windows Home Server is a new Microsoft "family" friendly media and backup server. 120 trial available, so worth a check out for setting up a _very_ simple to use server you can't afford to administer or support yourself (like for a non-technical family member or friend).

It's got an interesting feature called "addons". They are like "packages" for various servers (or templates): DHCP, uTorrent, Wake on LAN, iTunes streaming, etc.

There is also AIDA32 and SiSoft Sandra which give detailed system information and can perform various benchmarks.

BgInfo from SysInternals is another interesting tool that displays systems information as part of the wallpaper background. It's quite nice if you're part of a large network and need to quickly identify what system you're on and easily view various network settings. It's somewhat similar to torsomo on various *NIX systems.

Magic Rescue scans a block device for file types it knows how to recover and calls an external program to extract them. It looks at "magic bytes" in file contents, so it can be used both as an undelete utility and for recovering a corrupted drive or partition. As long as the file data is there, it will find it.

Monday, January 07, 2008

Performing digital forensics or data recovery can sometimes be problematic, and you will eventually need to resort to file carvers.

Revit (Revive IT) is an advanced file carver thta uses file structure based carving, originally developed for DFRWS 2006. This means you can carve files right out of raw disk data, even in the lack of a filesystem or partition table.

"Provides a dynamic virtualized grid of language-level virtual machines (VMs), networks, and storage. The principal VM is the Java Virtual Machine (JVM), but others can also be supported, such as for Perl, Python, and Ruby."

"The system makes use of multiple features of the Solaris Operating System (Solaris OS): Zones are used for VM isolation and network binding control. Resource pools and the Fair Share Scheduler are used to manage processor allocations. IP Filter is used in network connectivity control. Extended accounting facilities are used to account for CPU time and network traffic. ZFS file system features such as snapshots and clones are available to developers."(Sun)