The only project I had left over (from that bunch) was how to extract the binary contents of Solaris Unix datastream pkg files.

Unfortunately, unlike the "rpm2cpio" command that Solaris includes, they have no regular program for converting a datastream package to an unpackable file (like a cpio or tar), unless you count all the basic pkg commands. What I was looking for was a way to extract a datastream pkg file, and get all the binary contents, without actually installing it. I got no love from Solaris, but I got some free time for myself and eventually figured it out ;)

I've included a script, below, to rip apart a Solaris datastream pkg and create a subdirectory (named the name of the pkg) into which I dump the pkginfo, pkgmap, pkgproto (which I derive from the pkgmap) and all the binary files that the pkg contains. The script, itself, is far from perfect, but if you check it out you can see, fairly easily, how you can manually pull apart a Solaris datastream pkg file and get to the binary contents that you really want (At least, I'm assuming you do - just like me ;)

Note that the version of cpio that comes with Solaris (in the SUNWcsu coreutils package) will complain about garbage bits in your cpio header, when you've clipped off the top of the pkg file. There are really only 2 garbage bits you have to delete (which you can do with vi), but I prefer to use the Gnu version of cpio which will "magically" ignore garbage bits and extract the cpio file no matter what. This version of cpio comes standard on Linux and can be downloaded (in Solaris datastream pkg format) from sunfreeware.com or (in source format, with links to all sorts of OS ports) Gnu's CPIO Download site - just in case you don't want to have to rip apart your Solaris pkg's on Linux (or don't have both OS's at your workplace).

Another strange thing is that Sun's version of cpio won't make directories, by default, that don't exist when you unpack a cpio archive (???) All I'm really saying is: Get the Gnu version - It's free and it'll save you many a gray hair ;)

I've tested today's script on multiple packages from sunfreeware.com and had great success with it working right the first time (assuming that I have Gnu's cpio). The script has broken on a few custom pkg's I've put together myself and can use a lot of work in the "additional" areas (like ensuring pre and post-install scripts get accounted for, depend files get tagged, checkinstall programs are properly extracted, etc). Still, it's a good beginning :)

Hopefully this will be of help to you, if you've been looking all over for something like this, and, almost definitely, can be the starting point for a fully functional Solaris datastream pkg ripper :)

If you want to do this manually, in short terms, just vi the pkg file you're interested in and delete everything up to the final instance of "pkginfo" in the file (after the final "pkgmap" line) and save the rest into another file, which you can manipulate with cpio. Who'd have thought it would end up being that simple? :)

Sunday, March 30, 2008

Note that this patch was created with "diff -c," so I've been sure to specify both the file I want to patch and the patch file in my arguments to "patch" (whereas I didn't need to in our post on easily patching multiple files). The exact command line to create the patch was:

host # diff -c eraser eraser.new > eraser.patch

All you'll need to do to update yesterday's script to include today's modifications is put the patch file (tagged on to the end of this post) in the same directory as the original script (or wherever you want, assuming you know how this all works and my explaining it is trivial ;) and name it something like "eraser.patch" - Then do the following:

As I noted before, you'll want to be specific in your arguments to "patch" when you run this, or be careful of "patch"'s default behaviour and don't choose to do a reverse patching, like this (BTW, even though my new script "eraser.new" is listed in the patch file, you don't actually need to have it for all of this to work):

Saturday, March 29, 2008

Today's script is a twist on a previous script we posted to do network wide updates. The essence is still the same. This should basically save you the hassle of having to log in to a million different servers to find and remove users who you know shouldn't have accounts on your boxes.

Again, you can also delete the "Weed out the undesirables pronto" section if you don't want to minimally secure the script from unauthorized usage by doing a logname check.

We've left out the login functionality for Linux this time, but we'll add it tomorrow as a "diff -c" patch file and add it to this file with "patch" to give the week an appropriate finish :)

There's a section I wrote specifically for an Informix sql database, that you can modify (or remove) if you like, to include an example of how to check for users (or anything, really) in a database. If you don't want or need that functionality, just delete the "$userlogin mail server sql user report:" section. All the sections are highlighted with tons of pound signs (#)

Once you edit this Expect script to suit your needs, you can run it easily, like so (with -h if you want to see the help screen):

Friday, March 28, 2008

Today we're going to continue with yesterday's post on manually mass patching files on Linux and Unix, but come at it from a different, and much simpler, angle.

First things first; you can delete the ed patch files from yesterday :) When you look at them, you can see that using "diff -e" to create an ed patch file creates a patch that contains absolutely no information that would allow you to reverse a change that you made, as opposed to a patch file made from diff straight-up. The small comparison below points this up more clearly:

The ed patch file type that we used yesterday, although simpler to understand, doesn't make it possible to reverse your changes without keeping backup copies of your files.

The good news is that patch files created with diff can still be used to patch massive amounts of files and recover them even if the original, unpatched, files get lost or deleted. The method used to obtain these results is simpler than what we walked through yesterday and only requires the use of the "diff" and "patch" commands. Both of these commands should come standard with your OS. They've been on Linux for a long time and have been on Solaris Unix since, at least, release 2.6 (probably earlier).

So, let's get started patching lots of files and then backing out those patches. Since yesterday's post was so long (and this one has the same potential), I'm going to use only 2 files as the base number of files (although the number of files can be however large you want) and grep out the relevant information for display purposes. Hopefully it will save us all some eye strain ;)

The first thing we'll want to do is copy all the scripts we want to patch into a new directory to work on them. We'll also put the new scripts (that we'll need to create our diff patches) in yet another directory. We'll never work directly on the scripts in their native directory until we're sure they're patched correctly. This isn't absolutely necessary, but is generally good practice. No sense in letting a simple mistake cost you any more time, or grief, than it has to. Again, the only thing that is different between our existing scripts and the new scripts is that the HOMEDIR variable has been changed to BASEDIR.

host # cp scriptdir/* tmpdir1/

<--- This could be any number of files.

The following is our setup, with only one difference between the files, as noted above:

Next, we'll use diff to create a patch file. Although, rather than just doing diff straight-up, we're going to run it in "contextual" mode. When you invoke "diff -c" it creates a contextual diff which, literally, means that it puts the diff output in context (so you can see the lines before and after the lines that differ). The main reason I like to use this option is that the output works with "patch" when patching multiple files from multiple directories. The output from a standard diff of multiple files in multiple directories doesn't work well for this (mostly because it puts all the file names on one line and "patch" attempts to find a file named "diff tmpdir1/file tmpdir2/file" - literally. And that file can never exist (I hope ;)

Now we'll create the multiple file patch and examine its contents, created at the directory level directly above tmpdir1 and tmpdir2, so we can get all the files in both directories (Note that only the lines beginning with the exclamation point (!) in the diff output are different. The lines above and below only serve to showcase the line in its context within the file):

Now we're ready to patch all of the files in tmpdir1 at once, using the simple form of the command (patch -i PATCHFILE), and receive an error for doing so. Note that we're running this from the directory above tmpdir1 and tmpdir2; exactly where we were when we used "diff -c" to create the patch file:

host # patch -i patchfile.patchcan't find file to patch at input line 4Perhaps you should have used the -p or --strip option?The text leading up to this was:--------------------------|diff -c tmpdir1/file1 tmpdir2/file1|*** tmpdir1/file1 Wed Mar 26 14:36:05 2008|--- tmpdir2/file1 Wed Mar 26 14:54:10 2008--------------------------File to patch: ^C

<--- Type the control (ctl) key + C, or any other escape/control key combination to break out of this prompt

This error killed us before it actually made any changes because of the way "patch" works. When run without any options (other than -i, which we used to indicate the name of our patch file), "patch" parses the patch file and strips the file names down to the base, in much the same way the "basename" command does. So, even though the file name in the patch file is "tmpdir/file1," the "patch" program is looking for a file named "file1" and it's looking for it in the directory we're in which, unfortunately, isn't where the file is.

Luckily, this little setback is easy to remedy. Using the -p option to "patch" we can instruct "patch" how to interpret the file names in our patch file. As we noted, when the option isn't present, "patch" reverts to "basename" type behaviour. If we used -p1, we would be instructing "patch" to remove the leading slash (/) from the file name. We're going to use -p0, which instructs "patch" to not interpret the file name and just take it as it is (In this case "tmpdir/file1," which is relative, but just fine considering where we're running the command from).

Excellent! HOMEDIR is now BASEDIR. Alas, as I intimated in our post yesterday on mass file updating, our boss has, only minutes later, decided that the BASEDIR variable really should be HOMEDIR after all. He's not going to explain why, we just need to switch everything back now ;)

And this is where our method of execution really pays off. Assuming we held on to that patch file, we can now use "patch" to put everything back the way it was in short order, by simply adding the -R (to reverse the patch operations) to the command line and running it again, like so:

And we're all set :) Hopefully this follow-up tutorial was easy enough to follow, and you can find some good use for it in your work routine. BTW, don't forget to copy the changed scripts back to the real script directory, but only after copying that directory off somewhere else, again. But only if you're as paranoid as I am ;)

Today, I thought we'd look at diff's usage in a, generally, manual process and peek at the first step toward automation. Everything in this post can be scripted out to provide more granular control, and done much more easily. I thought doing it in a laborious tutorial style to start out with would be best to get the major parts of the process laid out simply and, hopefully, written in such a way that they're simply understood. After reading that last sentence, I'm wishing myself lots of luck ;)

For purposes of our example today, we're going to assume that we have a directory in which we keep 3 executable files. All of these files are approximately the same, because they all do approximately the same thing, just for different programs. For simplicity, they've been named: file1, file2 and file3, as shown in this directory listing:

host # ls. .. file1 file2 file3

All of their contents are also almost totally identical:

host # cat file1 file2 file3#!/bin/bash# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

#!/bin/bash# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

HOMEDIR="/usr/binky"

For today (and this would work if we had 5000 files, but this post is going to be long enough ;) all we need to do is change the HOMEDIR standard variable in all of our scripts to BASEDIR. I'm not exactly sure why. Someone with more authority than me or you decided it would be a good idea. They will probably change their mind and have us switch it back next week ;)

The first thing we should do is create a backup directory and do our work there.

Then we'll just use a simple while loop and feed each file to sed and make the substitution, which we'll dump into another file called FILENAME.diff (So, file1 will get the substitution made on it by sed, and that difference will be put into a new file called file1.diff). Within that same loop, we'll be using the actual command "diff -e" to create an "ed" (A very basic Unix and Linux editor) patch file. Note that, for file1, this new file will be called file1.patch and that we echo a "w" and a "q" (on separate lines on purpose) into the bottom of the patch file.

Now, let's delete the *.diff files and take a look at the patch files. These are all considered "ed" scripts. This means that we can feed them to the "ed" Linux/Unix editor and have it execute commands for us automatically. This will come in handy in a second :)

The next step in the process is to apply those patches. We're going to be doing this all in our test directory, so there's no need to tar up all the patch files for a move to our "live" script directory

host # ls. file1 file2 file3.. file1.patch file2.patch file3.patch

So, now we're ready to test our mass change and pray all goes well. Note that, since we're still in our backup directory, we can do lots of damage to the scripts in here and it shouldn't make a difference to anyone. Only we will have to live with the shame ;)

Now, we'll do another fancy command line and pipe the output of "ls -1d *", grepping out all the patch files and then using "ed" to patch each file in the directory. Notice that the syntax for patching a file with an "ed" diff script is "ed FILENAME <FILENAME.patch" - This would normally hang and wait for us to send it a control-D or some other signal, but, since we echoed the "w" and "q" characters into the patch files, "ed" knows to write (w) and quit (q) - At the end of the command pipe, we'll purposefully not delete all the patch files. You can remove them if you like, but we'll be looking at using them to undo changes in a future post!

Okay, we're all done and now we can check the results. Looks like everything in our (potentially huge amount of) files got patched appropriately :) This shouldn't be a surprise since we checked our sed output already, but you never know.

host # cat file1 file2 file3

#!/bin/bash# Start File1 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash# Start File2 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

#!/bin/bash# Start File3 Process

COMMAND_ARGS="-d --takeforeverandaday"

BASEDIR="/usr/binky"

In a future post we'll look at a script that will allow us to do what we've just done today and make life a little easier when doing mass updates or patching :)

Wednesday, March 26, 2008

Today's we're going to go off on a tangent and take a look at color and the Linux and/or Unix console. I first noticed the use of color in the shell while using Linux a long long time ago, and it seemed really cool back then. Every now and again it bugs me, but I must admit that it does come in handy for a lot of things; editing code or simple file type identification are two great examples.

The script attached to this post was written mostly to illustrate the concept that you can see in the picture attached to this text. More to the point, I wrote it so that we could see the colors I'm writing about :) Admittedly, I used every possible combination of color and modification for completeness, which has the nasty side effect of making some of the output invisible (You can't see the black foreground on the black background unless reverse video or bolding are turned on), but that's the nature of the beast. You probably won't ever need or want to print black on black, unless you're writing a menuing system for someone you can't stand ;)

One thing to note is that the colors and modifications in this script are limited to the ECMA-compliant codes. Linux and the Bash shell, for example, have a lot more color options, but I wanted to keep this post as fairly balanced as possible. These codes should work on almost any Unix or Linux console and most SSH or Telnet shell clients today probably support them as well. Still, not everything will work everywhere. For instance, the blinking text code does not work on my PuTTY terminal, although it does work on my SecureCRT. Since I'm not paying for the color, I guess I can't complain ;)

Getting these colors to show on your terminal is actually fairly simple. First, you just need to know what all the codes stand for. Below is a quick listing. The 30's are foregrounds, the 40's are backgrounds and the remaining set are modifications (any and all of which can be combined, if you so choose):

Now, if you wanted to print a color to your screen, all you need to do is use the echo command (Note that -e tells echo to interpret the backslashed characters, so "\033[30m" doesn't show up literally). For instance, to print "HI" in red font, you could type:

host # echo -e "\033[31mHI"

The "m" character following the number code 31 (Indicating that you want to print in red) completes the instantiation of the color. Without it, your text would remain whatever color it already was. The incorrect echo statement may also have other bizarre side effects, like repositioning your cursor or resetting your terminal. The final "m" is very important :)

You can also combine different colors, as well as modifications, by using the semicolon. Below, the first line writes "HI" in light blue on a green background (Uggh) and the second does the same in BOLD :)

host # echo -e "\033[36;42mHI"host # echo -e "\033[36;42;1mHI"

One thing to note is that all of these commands would leave your terminal stuck in whatever state you instructed on the command line. That is, if you used the color commands to type with a green background, your terminal's background would remain green after you typed the line. In order to return everything to normal, you can use the modification number "0" - This will reset your terminal to the default color settings. If you only want to print one line in red font, the line shown below would work better for you than the original (the first example, above)

host # echo -e "\033[31mHI\033[0m"

Note, also, that (for the color code commands) there can't be any spaces. "\033[31m" has to be one unit. The text you put in between this command and the closing "\033[0m" (should you elect to go back to the way things were ;) can have spaces anywhere, just like any other block of text.

Hope you have some fun with shell colors. Enjoy the script. If nothing else, it should serve as a good reminder of the syntax and a good indicator of the breadth of options ECMA compliant colors give you. Linux, as I mentioned, has even more variety, but using those specific codes can make your scripts less portable. Good or bad? You decide ;)

Actually, I've been meaning to follow up on the Linux counterpart of this issue for a week or so now, but got caught up trying out various things. For instance, one of the more unusual (and interesting things) I ran across was a Perl project called alien that endeavours to pretty much translate between almost all package formats (dpkg, rpm, pkg, etc), but it didn't really address this specific issue. Still, pretty neat and in the experimental stage.

I also followed a lot of trial and error using rpmrebuild and standard rpm with the --repackage option, but found that both of these methods required the additional execution of at least one thing I didn't need to do to achieve my goal. For instance, using rpm with the --repackage option requires that you be upgrading, re-installing, deleting, etc, the RPM in question. All I want to do is create a package from existing source, and I'd rather not have to upgrade, re-install or delete the software on a live system to do it :)

Luckily, in the end, it boiled down to a fairly simple process. A lot of confusion came from various interpretations of the rpmbuild "spec file" and how, and why, it should be used. Many of the ideas I got that led me to an eventual solution came off of message boards rather than official sources. To put it somewhat humorously, a lot of those guys may not know what they're talking about, but they sure do know what they're doing ;) Seriously, if it weren't for the folks who strayed from the info-docs, this solution might have taken even longer to come around to.

You can call today's command (which I've name "rpm_remk" to remind me of my similar Solaris pkg project), very simply from the command line, like so:

host # ./rpm_remk mutt <--- If, for example, you wanted to create an RPM of the installed version of "mutt" on your system.

You can be as specific or general as RPM will let you when you use this command. For instance, on my box, I only have one instance of mutt:

host # rpm -qa|grep muttmutt-1.5.6i-64.9

So that simple command line suffices. For another program like, say, xaw3d, I would have to be more specific, because the underlying RPM system wouldn't know which package I was referring to. For example:

host # rpm -qa|grep xaw3dxaw3d-1.5E-216.3xaw3d-32bit-9-200407011229

Shows two different selections. RPM generally only picks the first one, like this:

host # rpm -q xaw3dxaw3d-1.5E-216.3

So, if we ran the command with only "xaw3d" as the argument, we'd create that 1.5E RPM, which would be great unless we wanted the 32bit version ;) Specifying "xaw3d-32bit-9-200407011229" as the argument would guarantee that the intended RPM got built!

Here's hoping this script helps you out and makes it a little easier for you to retrieve some of those old installed software packages for which the original RPM's have long since disappeared :)

Monday, March 24, 2008

It's been a while since we took a look at Expect on this blog, so I'm throwing a monster out there today ;)

Today's script is a bit long-in-the-tooth, and may be complicated looking, but it's basically a simple script to enable you to do updates across your network all at once; only having to login once (or twice, but all at the same time ;)

I put a small section in the beginning that checks whether or not a user invoking the script is allowed to use it (although this script doesn't allow anyone to do anything that they don't have the privilege or access and knowledge to do already). It's a simple check against an exec of the logname command, to make sure that the user's logname is among the list of lognames allowed to use the script. This whole section can be removed without affecting the script's functionality. Simply delete the section with the header: "Weed out the undesirables pronto" and it'll no longer be an issue.

I also added separate login functions for Solaris, Linux, HP-UX and SCO Unix for variety (Note that you may need to tweak these Expect routines, in case the regular expression matching doesn't suit your flavor). You may not need all of these functions, and you may only need one. In any case, you can either call the script specifically to use it the way in which you desire, or edit it so that it will only run the way you desire. If you call it with the "-h" flag it should give you more help than you could ever possibly want ;)

Sunday, March 23, 2008

Today, we're following up on yesterday's post of our common PKG and RPM command cross reference. Hopefully, this will enhance our previous post by providing the equivalent apt and yum commands for the Solaris Unix pkg and RedHat Linux RPM commands (where they exist) that we went through yesterday..

The format for the list below, will be the same as our PKG and RPM reference and we'll present the actions in the exact same order, so you can put the posts side by side and translate between all four package manipulation methods at a glance. Again, the format is:

What We Want To Do: APT command - YUM command

<--- with PKG being the package name.

Enjoy your Sunday and hope this helps you out. If you know of any way to do a lot of these things with apt or yum that I don't, please feel free to email me and I'll be glad to update this page :)

Show package info: apt-cache show PKG - yum info PKGInstall package: apt-get install PKG - yum install PKGRemove package: apt-get remove PKG - yum remove PKGList the package that owns a particular FILE: Can't be done to my knowledge - yum provides PKGList all files in an already installed package: Can't be done to my knowledge - Can't be done to my knowledgeList all files in an uninstalled package FILE: Can't be done to my knowledge - Can't be done to my knowledgeList all packages installed on your system: Can't be done to my knowledge - yum list installedCheck that all files from a package are installed correctly: Can't be done to my knowledge - Can't be done to my knowledgeCheck that all files from all packages are installed correctly: Can't be done to my knowledge - Can't be done to my knowledgeCheck that a package FILE is OK before installing: Can't be done to my knowledge - Can't be done to my knowledgeFix an altered package installation: Can't be done to my knowledge - Can't be done to my knowledge

Saturday, March 22, 2008

Today, I thought I'd put out a little summary, if you will, to wrap up all the pkg-to-rpm-to-pkg type articles we've been dealing with for the last few weeks and try to boil it down to a quick cross reference. This way you can have, at a glance, the basic information you need to manipulate both Solaris Unix pkg files and Linux RedHat RPM's.

You can find more detailed information, if you prefer, in our previous posts on working with Linux RPM's and building Solaris packages. I have yet to write a post on working specifically with pkg files (just lots of different aspects), but I suppose I ought to now, huh? ;)

This will be pretty straightforward, with a format of:

What We Want To Do: RPM command - PKG command

>--- With PKG and RPM being the actual pkg and RPM names

Enjoy, and I hope this helps save you some time digging around We will most likely look at the apt and yum equivalents of these commands very soon, as well :)