This is the second installment of the "Linux on the Opteron, are we ready?" article. Basically, it is a "where are we now?" article, noting that what once did work now does not, and others that did not work now do. The first article was published on OSNews almost three months ago. Since that time not too much has happened publicly in regards to the amd64 Linux situation, but a lot of people mailed to tell me that I should have checked out SuSE or the new Mandrake which was "about to be released" at that time. Also since that time I have upgraded the RAM and acquired a larger hard disk for the machine. I will give a brief rundown of the system as it stands now, what I tried to install on it, and what works.

I have an asus sk8n opteron system. I'm running gentoo on it 32bit only and after a small struggle (nearly) everything is working. The only thing that doesn't work is the onboard NIC. I've got x and dri opengl working with my firegl card. I'm not brave enough to try a 64bit install yet. One question, can someone expand on the mentioned idebus=50 kernel option? what does it do and is it safe?

One last thing, the driver (talking kernel 2.6 test 11) for the promise sata controller is still beta and causes my drives to dma-timeout sometimes.

Well, the idebus argument is passed to the kernel to force
the PCI IDE speed. Notice on your gentoo/opteron system that
if you check the dmesg output you will see that the drives
are falling back to PIO mode at 33MHz. The idebus=50 kernel
arg will up this to 50MHz... or you can take rac's patch
for the nforce3 chipset and apply it to the kernel to make
full use of the Nforce3 chipset and possibly go UDMA.

Yes, I have seen the promise driver have problems... I would
wait on it.

I have also gotten the opengl/etc to work on the firegl
in 32bit land on the opteron, but not in 64 bit under
gentoo.

My guess is it sets the bus speed to 50 Mhz for the IDE controller, possibly fixing some timing problems which shouldn't be there in the first place.

I have a motherboard with some ITE chipset IDE RAID and serial ATA, but I didn't have any drivers for any of that in any of the Linux distros I had handy at the time so I threw in a PCI promise IDE controller and was able to quickly build a RAID this morning. I wish chipset manufacturers could offer some backwards compatibility or Linux drivers offerred more forward compatibility or something.

I agree with the post above, I don't want to be a beta tester, again. I just want things to work out of the box. Is that so much to ask for? I mean we give everyone full access to the source code, including drivers for all their old hardware. What more do they need? And who's fault is this? Ours? For running any OS other than Microsoft Windows?

I'd guess all the problems experienced are due to the ASUS motherboard and its shoddy BIOS/poor engineering.

Every ASUS motherboard I have owned (ASUS A7Pro, CUPLE-VM, A7N8X) has required BIOS upgrades to work correctly, onboard peripherals diabled, exhibited lockups and have just generally been a pretty poor experience 'out of the box'

Sure, after you put in 20 hours work or so hunting for information on how to work around these issues and upgrading BIOSes multiple times, the boards can be made to run relatively stably.

But ASUS, as far as I am concerned, don't produce quality linux-ready boards, so if you buy their rubbish, expect to have to waste hours fixing their problems.

My latest board, the A78NX 'Deluxe' gives me massive disk corruption under a 2.4 kernel and locks up at least once a day under either 2.4 or 2.6 if APIC/ACPI is enabled.

booting the kernel with 'noapic nolapic acpi=off' seems to give me stability, but why should I have to run with these features diabled - ASUS are clowns.

There is a Preview Release of Fedora Core 1 for AMD64 systems available now (actually since Dec 1). It certainly represents another option for a cost effective Linux Opteron solution. Check it out at: http://www.linuxtx.org/amd64faq.html

Hi,
We have been testing Linux (both Redhat and Suse) in Opteron chip and we have got really good performance boost when applications are re-compiled for Opteron. Avegare is about 30% faster than using 32-bit binaries. After testing Opteron with Linux, I never wanna go to Intel world.

4. RedHat 8.0 (32bit)
Worked fine on HDAMA, but was slower than GINGIN64 Beta

5. Windows 2003 (32bit)
Loaded w/o problems on HDAMA (use current BIOS). Nice to see windows start in 18 seconds.
Does not load on e325. Had to load Windows 2000, and in place upgrade to 2003 (then worked fine). I assume some BIOS issue that IBM will need to address.

Although some of our applications are restricted to 32-bits because 64-bit builds of some 3rd party libraries are not available yet, we still gain a handsome 35% speed up when running our 32-bit apps on a 64-bit OS, like SuSE's 64-bit Linux OS.

Preliminary tests show that we gain another 35% when our apps are 64-bit; for a total speed up of ~70% over pure 32-bit.

[quote]I'd guess all the problems experienced are due to the ASUS motherboard and its shoddy BIOS/poor engineering.

Every ASUS motherboard I have owned (ASUS A7Pro, CUPLE-VM, A7N8X) has required BIOS upgrades to work correctly, onboard peripherals diabled, exhibited lockups and have just generally been a pretty poor experience 'out of the box'

Sure, after you put in 20 hours work or so hunting for information on how to work around these issues and upgrading BIOSes multiple times, the boards can be made to run relatively stably.

But ASUS, as far as I am concerned, don't produce quality linux-ready boards, so if you buy their rubbish, expect to have to waste hours fixing their problems.

My latest board, the A78NX 'Deluxe' gives me massive disk corruption under a 2.4 kernel and locks up at least once a day under either 2.4 or 2.6 if APIC/ACPI is enabled.

booting the kernel with 'noapic nolapic acpi=off' seems to give me stability, but why should I have to run with these features diabled - ASUS are clowns. [/quote]

Sorry to hear you have issues with your boards, but I wish to point out that along with Tyan, Asus has and continues to produce the best consumer boards on the market. I strongly suggest you look in the mirror for the source of your problems.

32-bit Suse worked perfectly on the Opteron/SK8N. I was able to find drivers for everything and they worked (I didn't bother with the Promise card). I ran FFTW benchmarks. I then installed the 64-bit Suse 9.0. There is no comprehensive driver package. Following instructions at Suse's website I was able to patch, compile, and install Nvidia's 3d drivers. I think they are working but the performance is about halfway between the 32-bit 2D only drivers doing 3D emulation, and Nvidia's 32-bit 3D drivers. The sound system worked without any extra help. However I cannot get the integrated network controller to work. I ended up throwing an old 3Com NIC in and that worked. I haven't tried to configure the Promise card yet.

My SK8N is bios 1003. I cannot flash it to 1004 for some reason....

I compiled and ran the FFTW benchmarks under the 64-bit Suse and the results showed a solid 20% increase in speed. FFTW is Fast Fourier Transform code....

I was wondering if anyone knows of any functioning workstations based on Tyan K8W (dual opteron monster board) running 64 bit Linux (any variety, but I'd be particularly interested in Gentoo specimens) with SMP enabled? Our lab is thinking of acquiring such a beast.

Anyone who knows anything about Linux is already aware of the problems with the file system, kernel panics, segmentation faults and lack of applications. It is no big deal, that is why no one wants to use it and it is free. It is not worth anything and the source code is written by the 'sand colored people'.

[quote]I'd guess all the problems experienced are due to the ASUS motherboard and its shoddy BIOS/poor engineering.

Every ASUS motherboard I have owned (ASUS A7Pro, CUPLE-VM, A7N8X) has required BIOS upgrades to work correctly, onboard peripherals diabled, exhibited lockups and have just generally been a pretty poor experience 'out of the box'

Sure, after you put in 20 hours work or so hunting for information on how to work around these issues and upgrading BIOSes multiple times, the boards can be made to run relatively stably.

But ASUS, as far as I am concerned, don't produce quality linux-ready boards, so if you buy their rubbish, expect to have to waste hours fixing their problems.

My latest board, the A78NX 'Deluxe' gives me massive disk corruption under a 2.4 kernel and locks up at least once a day under either 2.4 or 2.6 if APIC/ACPI is enabled.

booting the kernel with 'noapic nolapic acpi=off' seems to give me stability, but why should I have to run with these features diabled - ASUS are clowns. [/quote]
Well, first of all, A78NX board are based on the nVidia chipset, which bloody sucks under Linux. I think 2.6 has support for it, but for 2.4, you have to use nVidia's proprietary drivers.

As for the other two boards, well, all I have to say is that my A7V8X-X has worked just fine out of the box : no disk corruption, no lockups, and only a weird memory problem (wouldn't take a DDR400 memory stick for some obscure reason, so I just exchanged it for a DDR333 stick and it works fine - considering the XP2500+ I've got in there is a FSB333 CPU, I don't mind =))

I've got a quad processor HP rx8620 itanium2 box at work with (a measly) 16GB of RAM in it. It's my new testbed system. Runs 64 bit Linux (debian distro, with some stuff re-compiled in 64 bit). The db is postgres (32 bit), so the big win is just flat memory addressing and those sexy big caches.

The windows weenie has a quad processor Xeon with 16GB RAM running Win2K DataCenter and MS SQL Enterprise that's been kicking my little workstation install of postgres for a couple of months. That's some hella expensive software and hardware. And he sure didn't have any trouble reminding me of it.

Anyway, the db's performance on linux64/postgres flat out blows the windows system out of the water. Seriously, I'm getting performance gains in excess of an order of magnitude. First we thought there was a mistake in the testing but... not that I'm rubbing it in or anything.

Of course this isn't exactly the kind of gear you put on a desktop. It lives in a rack in the land-o-unpleasant air-conditioning. Shame really since it's got some sexy blinkenlights.

I have had same problems, but managed to overcome them.
Main headaches were:

'I had to use stage3 tarball (instead of stage1), emerge the things I wanted and then reemerge everything again with "emerge -e world"

Only periphery that doesn't work on my Dual Opteron is sound. Everything else works very well.

From the software perspective, many packages are still masked, but main things work (X, KDE, Gnome, some games, and yes-OpenGL works too). On non-working list are OpenOffice and AbiWord, for example.
I could probably make them work under 32-bit emulation, but I didn't want to fudge with this, I will rather wait to see 64-bit version.

Opteron works very nicely under Win2000/Win XP, but with Gentoo in 64-bit mode it really flies.

I have bought two identical dual Opteron machines and I'm so pleased with performance that I'm buying me a third one.
One machine works on Win XP, one is dual boot Win2000/Gentoo and third will replace existing fileserver/firewall/pribteserver/etc.

All machines are on 1 Gbit Ethernet switch (i have found quite nice and still on the cheap side Linksys SD2008)

It's fun to see hard disk as a bottleneck when copying files through the LAN ;o)

Also, PCI-X slots absolutely rock. Old PCI on my dual P3 board camo on the verge of saturation when trensferring files to/from disks. With a nice RAID card (I'm waiting for 3Ware 7506-12) this problem should be mnitigated for quite a few years to come...

Since my awesome 'Linux' machine just recovered from a segmentation fault, I had trouble getting back online, or the 'information super highway'. It was created by Al Gore the loser of the 2000 election. I hope everyone is doing well tonight, as I am enjoying a hot cup of tea, sitting in my pajamas, ahhh....

Don't necessarily blame the BIOS upgrade for the disk problems you experienced with Mandrake.

I have an old Compaq Deskpro 4000 which I upgraded the CPU on with an Evergreen 400MHz AMD 6 CPU upgrade kit and a Maxtor 30GB hard drive and the original Maxtor 2GB drive. I can install Red Hat 7.0 on it with absolutely no problems. When I tried to install Red Hat 7.3, Red hat 8.0 and Mandrake 9.0 on it, ALL THREE of them barfed on the Maxtor 30GB drive, claiming they could not read the partition table even after ALL THREE of them themselves created and initialized the partition table. Whether with fdisk or Partition Magic or their own tools, NONE of them could handle the drive.

This on the same machine and drive that Red Hat 7.0 blows right on with not one single complaint about the drive or anything else.

This tells me that sometime between RH 7.0 and 7.3 somebody fscked up the IDE kernel code so that certain drives simply cannot be handled. Either that or certain BIOS's like the Compaq (which is an oddball BIOS anyway) cannot be handled. My understanding is that after the Linux loader is done, Linux does not use the BIOS for accessing the hard drive geometry, which seems to me to leave it as the fault of the kernel IDE code.

Somebody needs to look into this since who knows where and on what drives this problem will bite.

We have a K8W with 8 GB RAM, and 2 Opteron 242's. We got it from Monarchcomputer.com. They were very responsive, and did a good job. It cost about $4,500 without the monitor (8x 1GB PC2700 doesn't come cheap). This system has been completely stable.

I put Redhat 3.0 AS beta (Taroon) on it with only one hitch. It didn't recognise the fancy dual-DVI Matrox video card. A plain text screen also is odd; It is stretched two lines too long and I couldn't adjust the monitor enough to fix it. I downloaded the X source and compiled it, and X runs now, but with just a plain window manager, so I just dropped that. I have mostly been accessing the machine remotely with ssh and VNC, and that works just great for me.

We were able to obtain the source code for the gene fragment assembly software CAP3 (many thanks to the author Xiaoqiu Huang) and easily recompile it. The largest job so far took about 6.8 GB RAM, and took over 30 hours to run. This same job was impossible on our dual Athlon. We can finally do some really big stuff and are very happy so far.

We are still waiting for a good non-beta 64-bit os (something like WhiteBox), but can get by for now. I don't intend to get _trapped_ into anything RedHat ever again. (Yes, I know they have an academic version. No, I don't want Fedora64. I'm not bitter, see?).

Regards,

Steve Wanamaker
Department of Botany and Plant Sciences
University of California, Riverside

I've been very interested in finding out generally how well SuSE 9.0 runs on AMD 64 bit systems. My most major concern has been their implementation of a 64 bit architecture - looks great on paper, but I'm much more interested in the capabilities in practice over paper. Planning for the future is great - but with the majority of apps run on the AMD64 chips being 32 bit, the AMD64 is largely unproven on the grand scale in my eyes. Ever the skeptic:)

That said, I have a great deal of faith in AMD for producing a solid product. I guess it's just that the 64 bit side of the new chips are something of a hocus-pocus black box sales pitch at this very moment... and I'm very, very curious to find out what advancements will come from 64 bit computing over 32 bit with regard to AMD's 64 bit architexture. After all - according to paper anyway - they did much more than just extend the 32 bit instruction set. Can anyone comment on applications compiled in 64 bit for AMD on a somewhat technical level? Especially under SuSE 9.0? I'm curious about any latency issues - positive or negative, how well GCC takes advantage of the new registers, any issues with ram < 4gb, any heat/power issues in 64 bit vs. 32 bit mode, overall stability of the system, etc... In a nutshell - anything wrong with the chip itself, or unexpected or extreme boosts/slumps in performance.

Ahem:) Atm my focus is the chip. Motherboard/Driver issues I expect will get ironed out quickly enough - but anything inherently wrong with the processor itself, or even just not up to par with expectations (and thus no real incentive to migrate) is of great concern to me. That's why so much interest in SuSE's performance -as afaik it is the best running OS on AMD's 64 bit architecture.

Anyway, sorry for rambling. Thanks in advance for any information you can provide, and thanks again for the article - which provided me with some very helpful information in itself.

I'm guessing that your lockups and Mandrake problems stem from the same thing. Either your disk is actually faulty, or Linux's driver for it is not very good at this point. If it weren't for the fact that your hardware is rather new, I'd be very worried about your hardware. But in your case it may indeed just be driver issues, especially since a lot of them might not be well tested in 64 bit mode.

First thing I did with my MSDN subscription was download the WXP 64-bit evaluation and take it for a test drive. It loaded right up with 64-bit drivers for everything except my Quatro and Fortissimo. NVidia has a wxp 64-bit driver for the Quatro which runs very nicely in dual-headed mode, but unfortunately there are no 7.1 drivers for my sound card so I was stuck with the onboard AMD sound. No *huge* deal but it would be nice to use my 5.1 speakers.

I quickly loaded up the Visual Studio .NET 2003 installation (the only reason to run windows to begin with) and I found that the setup MSI wrapper does not allow installations on "64-bit operating systems". I know this is a load of bull because it will work fine with the dual Opty's, but I they will probably fix this when Wxp64 becomes retail and a future patch to VS.NET 2k3 installer gets applied. With sole exception of VS.NET2k3 everything worked GREAT.

So after that fiasco I pulled Wxp64 off there and loaded up Wxp32 pro. NOTHING was recognized. I to used my wife's computer to go get the 32-bit drivers for everything and entered the usual windows install/reboot loop for about the next 2 hours.

Once all the drivers were in place, I have a fully operational 32-bit workstation with VS.NET2K3 and all the trimmings of 5.1 sound, but that's NOT why I bought dual Opterons. Even though the dual procs where faster than my old workstation, I needed to feel 64-bit day to day computing.

Enter Linux. I've been using Gentoo for quite some time now, and of course that's the first distro I went to when I needed 64-bit linux. I found that the x86_64 stage's were experimental and I wasn't very excited about a crippled workstation. I tried to install gentoo64 on a single ext3 partition (from reading the docs they warned of Rieser and XFS problems) so I wanted to play it safe. Fast-forward about 3 days later and I had a completely non-functional system. Something happenend and even Windows was crapping all over the place.

So my experience with Gentoo64 wasn't so sweet, but I didn't give up. After my entire system was hosed I decided to rethink my partition setup. Ultimately I decided to dedicate the 36gb drive and a portion of the 120gb drive to windows, and do soft RAID-0 on the two 9gb's for linux. Getting the MD's to work under Gentoo64 was too much of a headache for even ME, so I decided maybe it was time to rethink my distro.

I switched to Suse 9.0. I've wanted to try out Suse for a long time now but I loved portage so much I never switched. Finding that Suse 9 reports a "production ready" x86_64 distribution I was thrilled at not being labeled "experimental" any more. The installation was awesome, it found everything in my system (including 64-bit drivers for my sound card and all my usb stuff!). This kicked Wxp64 out of the water.

Unfortunately when it came to actually copying the files I ran into some problems. Pretty much every file off the DVD's were "corrupted" when they were copied to disk and failed MD5 checksumming. I found that if I clicked retry enough times (read: about 20) things would install. I experienced this through both the Sony on the IDE channel and the Pioneer on the SCSI channel. I ended up having one of the kids build me some Lego hackery to hold down the Alt+E key. I set a spool of blank dvd's on top of it and left it "retrying" overnight on all the packages.

The next morning I awoke to a fully functional Suse 9.0 system. I just got Xinerma support working under KDE last night with the nVidia GLX and kernel modules (by default Suse didn't see my dual-headed setup bitching about not having two video cards). Everything is working great, and I'm very happy with my new system. Suse 9.0 was the easiest installation I've ever used.

Special thanks to all the part-time hackers that gave me 64-bit desktop computing! Happy holidays!

[quote]
Enter Linux. I've been using Gentoo for quite some time now, and of course that's the first distro I went to when I needed 64-bit linux. ....
From reading the docs they warned of Rieser and XFS problems) so I wanted to play it safe.
[/quote]

We strongly recommend sticking with ext2/3 for now. We have had random reported problems with reiserfs on amd64, and have heard of major problems with running JFS on 64bit systems (which definately seems odd considering it was designed for 64bit systems initially). Though we have had successful reports of XFS, and no negative reports at all.

--end gentoo AMD64 filesystem snippet----

They explicitly state that they have heard only positive
reports from XFS users.

Had the same good experience with SuSE Linux Enterprise Server 8 (SLES8) and the YaST2 tool makes administration chores a simple enough that a junior admin can do it with little supervision. I work for a large Brokerage house on Wall Street and we were looking to explore the AMD processor for some of our work. Called SuSE since as the article stated they have supported 64 bit computing on the Opteron the longest. We were informed by SuSE that they had a business partner, Systems Solutions in the Downtown NYC area, when we contacted them it turned out they are a couple of blocks form our location. Upon arriving we signed the obligatory non-disclosure agreement, we informed the group from Systems Solutions that we were interested in the AMD version of SLES 8 and were particularly looking to run it in a clustered environment. They asked if we had decided on which hardware platform we wanted to run, which we hadnít yet. They invited us into their server room where they had 3 IBM e325 with 2 GHz Opterons running and then offered to provide the lead developer in our group a workstation and desk to do his coding and use their 2 of their severs as a proof of concept. Naturally we were surprised by this offer as was the manager from the Line of Business and decided to accepted their offer.

Systems Solutionsí offer saved considerable time in the initial development and testing and while our developer was working at their site their staff came to our location to create the lab to house the future cluster. As it turns out they are an integrator with a vast array of resources at their disposal they even have an electrical firm that they are partnered with that specializes in Data-Center wiring and power and they are also dealer for enterprise level UPS and cooling. After a successful testing of the application we purchased and had Systems Solutions install a 24 node 48 CPU cluster of IBM e325 servers and the configuration has broken every benchmark we have thrown at it. We had the added pleasure of informing the senior management of a rival group within our company that our AMD cluster easily beat their HP Superdome which uses 64 875MHz PA-8700+ processors with 12 less processors and at a fraction of the cost.

While the group was celebrating at the project closing party the EVP from the Line of Business jokingly said to the Project Manager form Systems Solutions, if they could fly him and 5 of his direct reports to a meeting in Florida then they would truly be a full service integrator. Everyone from our company was laughing then the Systems Solutions Project Manager took out his wallet as presented the EVP with his FAA pilotsí license and informed him that he is also the Director for the Flight Department and could have one of the 6 corporate jets ready within an hour. Then the Systems people started laughing, just a scary group of people, and a few days latter the EVP used their Flight Department and said it was the best service he ever experienced.

It was just an all around great experience with a professional vendor and we are looking forward to working with them again.