Primary Menu

In the first part of the series I wrote about basic console commands. Those are very useful. But besides the plain commands you can also use aliases or even functions to enhance your command line workflows. For more complex tasks various build systems can be used.

Aliases

Every bash offers the possibility to adapt commands to your personal needs by using aliases. An alias is a symbol for a original command that can be called instead of the original command.

The first and most important alias for me is the alias that lets me open the file which defines the aliases (~/.bash_profile on MacOs):

alias aliases=’subl ~/.bash_profile’

The line starts with the string alias. This tells the bash that you want to define a new one. The name of your new alias comes next, in this case aliases. After an equal sign I define that I want to call “subl ~/.bash_profile” which will call my favourite text editor Sublime Text 2 and open the file .bash_profile. This way I can easily edit my aliases afterwards.

As I use GIT via the command line I created short aliases for commands I use frequently:

Functions

Aliases are great, but they are just aliases. If you need even more advanced functionality and more flexibility you can use functions. As software developer you should be familiar with functions. They have a name, a signature and offer a certain functionality. So you can incorporate complex logic in bash scripts and use the similar to aliases. I’m not an expert on complex functions, but those simple functions help a lot in my git workflow:

function gco() { git checkout $@; }
gco my-branch-or-commit

function gc() { git commit -m “$@” ;}
gc “my commit message”

Build Scripts/Automation

Automation is the key to remove cumbersome work which thwarts other more important tasks. A good way to automate tasks like copying, combining, compiling or even deploying files is to use a build system. There are endless numbers of build systems out there, you will easily be able to find one for your favourite runtime environment and language. Personally I love to use rake, grunt and gulp. All three help to create tasks in a very clean and declarative way without removing the flexibility of using a custom script. Also there a many gems (Rake) and plugins (grunt, gulp) out there, so you probably find one which solves your automation problem.

Conclusion

Automation is what technology is all about. There are many tools out there – especially for developers – that help you to automate repetitive tasks.

Nevertheless I think that automation needs time. Maybe you ask yourself how often you will reach the break-even-point at that the time invested on creating the automation is less than the time spent by manually doing a task. I think that often you won’t reach this point.

But I personally think that even if an automation project will never reach this point it helped you to keep your routine (so the next automation can be implemented faster) and also it saves you from repetitive tasks that can be frustrating and demotivating.

Learn how to automate, keep yourself trained and you will be more productive.

As software developer you need to be as productive as possible. Even if you use an IDE you might spent a lot of time in the command line, also called terminal/bash/shell.

So one idea is to improve your command line abilities.

Those commands are your friends. Instead of moving forward and backward from UI to the shell all time you might accomplish simple tasks in the shell. Why? Less distraction. More productivity. I assume that you know how to navigate with cd. Other simple commands are (most of them are Unix based):

man [command]man mvman cp
Should you ever forget what a command does, call the manual pages with the man command. Not that easy to read, but you will get used to. Quit by pressing [q] on your keyboard.

Move – mvmv file1.txt ./folder/file1.txtmv file1.txt file2.txt
Use this to move your file from one place to another. Or to rename a file.

Copy – cpcp file1.txt file2.txtcp file1.txt ./folder/file1.txt
cp -r folder1 folder2Us this command to copy one file to another location. Use the -r flag to copy a whole folder hierarchy.

Delete – rmrm file1.txtrm -r folderrm -rf folder
Use this command to remove files. Use rm -r to delete recursively in folders. -f means force, so the command does not ask you if files should really be deleted. Combine ‘em and be careful***, cause there is no undelete!***

Find – findfind . -name “*.txt”
Use find to locate files you are searching for. In the example above we search in the current directory (.) with the parameter -name and set the name to “*.txt”. * is used as wildcard. This will show all *.txt files in the current folder and subfolders.

Show output – catcat file.txt
Cat can be used to output the contents of files. Makes sense for e.g. small text or configuration files.

Show output -lessless file.txtLess is also showing the contents of a file. But it is not printing them to the console like cat. This is useful for longer files, since the terminal has a limited number of rows. Press [q]to quit the program.

Create files – touchtouch file.txtSimply create a new file without content. For example useful for .gitkeep files, which will prevent empty folders from getting deleted.

Open files – openopen file.txtopen file.html
With this command you can open files with the default program. Use to open images or html-files in the browser.

Pipes with grepls -l | grep txtcat file.txt | grep John
Pipes are a unix thing. They connect two data streams. E.g. it could connect the output of a list command with the grep command that filters output as seen in the first example. Or take the output of the cat command and filter all lines with John in the line content.

Grepgrep -nr some_function ./src
Grep can be used for many things. I found it useful to search for strings all files recursively. The example command searches for some_function in ./src recursively (-r) and displays the line number (-n).

I/O Redirectionecho “node_modules/” >> .gitignore
With >> you can redirect the standard output to a file instead of the bash. That means you can echo strings which are then appended to the file you name after the >>. Can be used to append single lines to text or configuration files. Careful: The single > does nearly the same, but instead of appending a line it overrides the file contents. That is only the tip of the iceberg, read more about it here.

If you get used to those commands you can drastically improve your productivity. Try it!

I’ve had to create some very similar events in my calendar, but they did not match a valid recurrence pattern. As I find it quite painful to create multiple iCal events I thought I maybe should try out MacRuby to automate the process as far as I could. As the resources for this topic are rare on the net I thought I could share the basics. I’ll explain the details below the coding.

In the first step we have to get the default CalCalendarStore. This object contains all calendars we have configured in iCal.

The next few lines query the calendar which the test event should be put in.

From line 17. on the CalEvent object is initialized with several values. Note, that the chosen calendar is a property of the event. In this test case the event is created at now + 2hours end in now + 4hours. You can see that Ruby data types blend wonderfully into Objective-C types, as usually startDate and endDate would need a NSDate object.

For the perfect topping an alarm is created from line 23. on. The CalAlarm object is initialized with several values and then added to the event. You could go forward and start to add more reminders, as e.addAlarm a can be called multiple times.

Until now the stuff is not yet saved. Apple designed the library so that the CalCalendarStore object is responsible to save the event. So we have to call defStore.saveEvent, with the event object itself, the span (relevant for recurring events), and an error pointer.

If you did everything correct, you should see a new event in your iCal!

Coffescript saves my day, as I don’t like javascript and the coffeescript syntax is more python/ruby based. Coffeescript itself compiles to javascript. It is very easy to learn, nevertheless it was not easy to create the basic code to register a new jQuery plugin. The following code serves as template:

If you are still waiting for the *Yeah, I have to use coffeescript!”-moment read this:

The lines of code of my jQuery plugin were reduced by 50% and the code is much more readable now! Don’t miss the coffeescript page which will give you much more arguments for coffeescript as I could ever do here.

This ruby script might be useful for people like me who just started using bundler to manage the gems of their webapp and all others, which want to clean up their gems. Caution: all installed gems are removed from rubygems!

The ftp.upload_files method takes a wildcard parameter, which is the same as you would feed into the Dir[wildcard] class. At the end, the FtpTools just use Dir to find all files. For the wildcard it is good to know, that a single * says “upload all files and folders”, but a **/* says “upload all files and folders in a recursive way, including subfolders and their files. Thus said, “./**/*” would upload everything from the base dir of your rakefile.

Extending Rake’s FtpTools for deletion
In my case I was using rake to upload files for a PHP project – written with the symfony framework. Symfony – as many other frameworks – has a cache which needs to be cleared if some settings for the view change. As the cache is simply a folder on the FTP server, I thought that rake could be used to easily clear the cache. Reminding myself of the fact that everything in Ruby is just an object, I just extended the FtpUploader class:

This snippet only deletes files, no folders. In my case this is wanted, as I’m not sure what the cache does if you destroy the folder structure, but nevertheless it could be easily extended. After all files of a folder are deleted, after the @ftp.chdir(‘..’) is executed, the folder could be deleted.Usage

Often I found myself checking the comments for my game iBox3D using http://www.appcomments.com. But the problem is that you can only display the comments by country and my app does so far only have comments in Germany, USA and UK. But the website does not show where you have comments or not. It was hard to say, if there were new comments or not. So I decided to write a small ruby script which should compile all comments and display them. The script evolved as I continued to optimize it:

v1: I used WATIR to automate the internet explorer and to get the HTML from the website

v2: I used the built in RSS lib to fetch the RSS

v3: I used simple-rss to fetch the RSS

v4: I used simple-rss to fetch the RSS and used threads to do that for all countries simultaneously

This post will handle following topics:

Ruby 1.9

Watir

RSS with the build-in RSS lib and simple-rss

utf8 vs. ASCII

Threading

JRuby and Ruby

Version 1: Watir

Watir is a lib that helps to automate the browser. For those who want to install Watir at Ruby 1.9 you need to install the ruby devtools and then install the gem with the –platform=ruby option. Here is the script:

require'watir'### hide or show the internet explorer$HIDE_IE = true# declare some variables
source_url = "http://appcomments.com/app/iBox3D"
comments = Hash.new
country_links = Array.new### open the internet explorer and navigate to the source url
ie = Watir::IE.new()
ie.goto(source_url)### fetch the links to the different countries
ie.links.eachdo|link|if link.href =~ /iBox3D\?country=(\d*)$/
country_links << link.hrefendend### loop over all country links
country_links.eachdo|link|# go to the country specific link
ie.goto(link)# due to unknown reasons sometimes the DIV was not there => reloaduntil ie.div(:id=>"review_dropdowns").exists? dosleep(1)
ie.goto(link)sleep(5)end# get the DIV with id = review_dropdowns
review = ie.div(:id=>"review_dropdowns")# get the country text from the SPAN in the DIV
country = review.divs.first.spans.first.text# loop over all DIVs
ie.divs.eachdo|d|# check the DIV class, if it has "comment", then proceedif d.class_name.downcase.match(/\bcomment\b/i)# initialize a hash for the comment information
inf = Hash.new# the first link in the div is the title/header
inf[:header] = d.links[1].text# the second link in the div is the user
inf[:user] = d.links[2].text# loop over all DIVs in the DIV and find the comment_right class (rating)
d.divs.eachdo|star_div|if star_div.class_name.downcase.match(/\bcomment_right\b/i)# count the start image tags
counter = 0
star_div.images.eachdo|img|
counter += 1end
inf[:stars] = counter
endend# get the description of the rating
inf[:text] = d.ps.first.text# store the information in the comments hash
comments[country] = inf
endendend### show the resultputs comments.inspect

To understand the coding I suggest that you use Firebug or the Internet Explorer Developer Tools to have a look at the HTML of the coding of http://appcomments.com/app/iBox3D. The comments should be enough to understand the coding – at least I hope so. Nevertheless I want to tell you why I did not stop here and continued to search for alternatives:

if the html structure changes, the script will not work any more (no interface contract)

it takes very long to start of the internet explorer and to visit each page

this script is not platform independent, for other platforms you would have to use e.g. firewatir

if I would use threads for parallelisation it would be very memory intense, because many IE instances take a lot memory

German umlauts! (ä,ö,ü) Somehow ruby 1.9, which claims to have UTF-8 support, introduces a lot problems. The german character ü cannot be parsed by the internal rexml. I found the simple solution to force ruby to think that the content string is utf-8, using the force_encoding method. Then parsing works.

Version 3: Simple-RSS

Before I found that out I tried the ruby lib simple-rss, which could parse the ü (german umlaut). Nevertheless I had to do the same trick as above when I wanted to access the parsed content. At this point I want to introduce the next evolution step of my script:

Besides the advantages listet below I also added a file output to an HTML file. This is because the RSS description tag contains HTML which can be easily dropped to a HTML file.

Advantages:

RSS is a standardized protocoll, so the structure won’t change in future (interface contract)

instead of opening the browser to perform the scraping open-uri is used, which is faster and not so memory consuming

this script should run on many platforms, including linux, mac os x and windows

the process of making the web request is way easier

Version 4: Threading

Still my script took very long. No wonder, it had to make a GET request per country and one additional for the overview site to get all the country codes. But using ruby built-in threads it is easy to make all requests parallel! This speeds up the whole script:

Ruby has a nice library for OLE automation, win32ole. I used this library to get appointments from shared outlook (exchange) calendars. The scripts are testen on Windows Vista with Outlook 2003. This is how it works:

# get the shared calendar of the recipient
if myRecipient.Resolved
calendar = mapi.GetSharedDefaultFolder(myRecipient, 9)
end

# write the calendar data to the output
if calendar
calendar.Items.each do |item|
puts "#{item.Start} - #{item.End}: #{item.Subject}"
end
end

Outlook has a mechanism which should protect the outlook data from unauthorized access, e.g. viruses and trojans. Everytime you want to access the data from outlook a popup opens which asks you, if the access should be granted. In case of automation this is very bad, because you usually don’t want to click anything if you automate a process.

So here the solution. I’ve created a ruby script which clicks the dialog for the user. Then I converted it into an executeable using RubyScript2exe (Ruby code gets compiled).

# try to activate the window with the Title 'Microsoft Office Outlook'
while !wsh.AppActivate('Microsoft Office Outlook') do
time_out = time_out + interval
sleep(interval)
if time_out > time_out_max
Process.exit!
end
end

# if the popup exists and the timeout is not reached some
# keys get send to the popup
wsh.SendKeys("{TAB}")
wsh.SendKeys("{TAB}")
wsh.SendKeys(" ") # this is "space"
wsh.SendKeys("{TAB}")
wsh.SendKeys("{TAB}")
wsh.SendKeys("{ENTER}")

To compile this ruby file into an executeable rubyscript2exe can be used. To install it you can use Ruby Gems (command prompt):

gem install rubyscript2exe

After installing you can open a command prompt, change to the directory your scripts are in and compile the clicker using the following command

ruby rubyscript2exe.rb outlook_clicker.rb

The executable will be created in the same directory. In the file outlook_reader.rb the following lines call the outlook_clicker.exe:
Thread.new do
system('outlook_clicker.exe')
end

In this three lines a new thread is created which calls the outlook_clicker.exe using the system method. Basically we reached that both scripts are executed in parallel.

If you execute the outlook_reader.rb you can see that the popup shortly opens and then automatically gets closed.