So, basically I am trying to take notes about some new tricks I learned over the past week. Hopefully, I will do the iterations and find even more efficient ways to do these in the future.

Quickly move around on a line.

0 (zero) moves the cursor to the beginning of a line (white spaces are considered as characters)$ moves the cursor to the end of a line (white space are considered)^ moves the cursor to the beginning of a line (white space ignored)g_ moves to cursor to the end of a line (white space ignored)

For example (using + to stand for space here for illustration purpose):

++a line++
- - - -
0 ^ $ g_

w, W move the cursor forward to the beginning of a word, the difference between the two is W uses white space as separator. So, for me w is quite useful in code, W is useful in the textual sentences.

e, E move the cursor forward to the end of a word, same difference as in w and W.

b, B move the cursor backward to the beginning of a word, difference is again the separator.

ge, gE move the cursor backward to the end of a word.

*, # to quickly move forward and backward to the same word under the cursor. This is faster than search using / and press N and n.

o adds a new line under current line and start insert.O adds a new line above current line and start insert.s deletes the character under cursor and start insert.S deletes the current line and start insert.D deletes from cursor to the end of line.C deletes from the cursor to the end of line and start insert.

Quickly move around in code.

% matches the parenthesis, bracket and curly brace. For example, put the cursor on a closing curly brace and % takes the cursor to the matching opening brace.

[{ and ]} to match the opening and closing curly braces. Place the cursor inside a code block, using these two to quickly move to the beginning and end of the closes curly braces pair.

gd jumps to the local variable definition when the cursor is on a variable name.gD jumps to the global variable definition.

You can also do N. to repeat the change N times. For example, dd command delete the current line, use . to delete another line, use 5. to delete additional 5 lines.

There is also a similar repeat trick @: for last command made on command line. For example, on the command line, we do :!ls to list files in the current directory. Then, we can do @: to repeat it again.

enter visual mode: use v to begin character-based selection, V to begin line based selection, ctrl-v to begin vertical block based selection (move the cursor will select the whole columns of text)

put cursor to the end of the desired text

use d to cut, y to copy

put the cursor to the paste position

use p to paste after the desired position, P to paste before the desired position

On the Vim twiki, there is also the tip to overwrite the same amount of text using the copied/cut text using Nvp, NVp, N^Vp depending on the selection mode (character, line or block).

Disable auto-indentation, manual indentation.

Sometimes, I had one annoying problem when I copy and paste formatted source code from another application - Vim will reformat the content and mess up with the indentations. To prevent Vim from retabbing, use :set paste. After you are done, use :set nopaste to diable paste. You can also add a key binding using set pastetoggle in your .vimrc file. See Vim Twiki "Toggle auto-indenting for code paste" for more details.

Instead, you want manual indentation. In normal mode, == indents the current line. In line-based visual mode (V or shift-v), select several lines and = indents them all.

Interestingly, there is another (probably less useful) jump list that records the default last 100 positions a user makes a jump. A jump is one of search, substitute and mark. See Vim Twiki "jumping to previously visited locations".

ctrl-O to jump back in jump list

ctrl-I to jump forward in jump list

:jumps to show the jump list

Auto expansion, adding comments, etc.

One common case when coding is to add comments. So, how do you add something like these lines:

Note that text object is more convenient than motion when you want to delete a chunk of text, you don't need to move cursor to particular position. For example, the difference between dw and diw is that when you want to use dw to delete a word, you must move the cursor to the beginning of that word first. For diw, you can place the cursor anywhere inside that word.

For example, I want to refactor code that has a string value hard coded to use a variable:

CSP (Content Security Policy): a W3C draft spec that defines fine grained security policies for resource loading to mitigate the risk of injection attacks such as XSS. The policy per resource representation is defined in new header is Content-Security-Policy. Firefox and Chrome is experimenting in headers X-Content-Security-Policy and X-WebKit-CSP respectively. The spec also defines a report-only mode without browser enforcing the policy but sending violation reports to the server, which is helpful for gradually enforcing the right policy.

It adds more syntax items. It is very interesting to read 9.1 section about the compatibility with existing cookie implementation. Basically, it should overwrite an existing cookie value set by Set-Cookie if domain and path both matches.

It admits the same confusion that I had, and deprecates the Cookie2 and Set-Cookie2 headers introduced in RFC 2965!

"Prior to this document, there were at least three descriptions of

cookies: the so-called "Netscape cookie specification" [Netscape],
RFC 2109 [RFC2109], and RFC 2965 [RFC2965]. However, none of these
documents describe how the Cookie and Set-Cookie headers are actually
used on the Internet (see [Kri2001] for historical context). In
relation to previous IETF specifications of HTTP state management
mechanisms, this document requests the following actions:
1. Change the status of [RFC2109] to Historic (it has already been
obsoleted by [RFC2965]).
2. Change the status of [RFC2965] to Historic.
3. Indicate that [RFC2965] has been obsoleted by this document.
In particular, in moving RFC 2965 to Historic and obsoleting it, this
document deprecates the use of the Cookie2 and Set-Cookie2 header
fields."

The actual format is getting updated as well. Notice that now both Expires and Max-Age are supported with Max-Age overriding Expires if both exist. If not, the cookie is kept by user agent till the current session ends.

Monday, November 14, 2011

Another spam I got today, it looks fairly serious ;-), maybe the "Dear Sirs" part sold itself. Anyways, here it is for fun:

Title: Please get a new EINFrom: Clementina EvansTo the ATTENTION of: Accounting DepartmentDear Sirs,please be informed that you need to get a new EIN in connectionwith the probability of fraudulent actions on your behalf. Weare sorry for the inconveniences that might be caused to youby this measure. Please use the following link for instructions:http://onesourceprocess.com/7wmzwt/index.html

Yours sincerely,Clementina EvansOffice of Professional ResponsibilityInternal Revenue Service

Tuesday, November 8, 2011

Just found it interesting to publish some lastest spams I got each day here, hopefully this would be helpful to non-tech people. Please don't try the links inside ;-)

Title: New EIN
From: Robert Glass
Content:
To the ATTENTION of: Accounting Department
Dear Sirs,
we would like to notify you that you are required to receive
a new EIN in order to prevent fraud. We are sorry for any troubles
that might be caused to you by this measure. Please use the following
link for instructions:http://web.knauf.es/5nse63/index.html

Yours sincerely,
Robert Glass
Office of Professional Responsibility
Internal Revenue Service

Sunday, October 9, 2011

After multiple occurrences of AT&T dropping connections over the past months, I finally decided to switch to Comcast (yes, I have heard many terrible stories about their billing department, sales fraud, etc.). Plus, I got a really good deal from my colleague ($19 per month for the first year, $35 per month for the second year for their performance package, 15MB upload and 5MB download.

Based on advice, I did not choose to rent their modem and bought my own from Amazon: Motorola SB6121 and it worked great. There is an older model SB6120 that is a bit cheaper, but my friend had problems. Please check with the technician and confirm that those models work in your area.

Today, they came in and installed it for me. The overall process worked fine except for three complaints:

1. They kept calling for confirmation. In the past week, they called so many times for confirmation, 3 days before, yesterday, and this morning. I understand that some people might be forgetting things very quickly, but I am not that type of people. Plus, I have confirmed at the first call already and they are still calling as early as 7am in the morning! Since I did not get this morning's phone call, they left a message and said I would have to reschedule it, WTF. So, I ended up calling them again so they can send the technician in without rescheduling.

2. Some technicians are un-professional. I am quite busy with work and life, so my appointment is 8-10am, earliest I could get so I don't miss the work. For the first appointment, the technician waited outside for 1 hour before he came in (I saw his van and went out to tell him I am ready, but he kept sitting in his var for another half an hour). When he came in, I showed him where I want the setup and the pole to pull the signal from. He shrugged his shoulder and said "it will take at least 4-5 hours". I said I thought it'd be only 1-2 hours and I had no choice but to reschedule. Somehow from the look on his face, I felt that he did not want to do installation (bad mood, lazy, etc.) and I am fairly pissed. If you don't want to work, then don't come to waste your time and my time.

Today, another technician came in and he is much better. He finished the whole job in 1 hr and told me the previous technician "must be lazy".

3. The installation process forces you to install the desktop installer to activate. After the hardware is set up, I hooked up my wireless router and was directed to an activation page, which asked me to create a new Comcast user account. But the last step is to install Comcast desktop installer and there seems to be no way around according to the technician. (Update: according to my colleague, you don't need to install it, just restart the modem, shutdown router, connect router with modem, start the router. But I did not try this personally, not sure if it would work.)

The installer file is "ComcastDesktopInstaller_MMF3.pkg" in my case. I uploaded it online if you are interested. Luckily I unchecked "install browser toolbar" option during installation. People are suffering from this toolbar since it hijacks browser preferences and homepage. If you are the victim, please follow instructions on this page to get your previous setting restored.

After the network is up and running, I want to uninstall it but could not find much info. I did my best, and my conclusion is it might not do as much harm. It executed some Ruby scripts and installed some crap on the desktop, which you can delete easily.

Cleanup actions in summary:

1. Delete the two website shortcuts installed on my desktop.2. Delete ~/.cim_install_log. 3. I am not familiar with Ruby, but the scripts seems to just create Xfinity bookmarks for Safari, Firefox and Chrome. For details in the script, see the gist below.4. Besides those, I did not seem to find other suspicious malware or unwanted junks.

Detailed checkups for the Comcast app:

1. Did a grep for "comcast" and found a log file in home directory: ~/.cim_install_log. It contains all the output during installation process.

2. Used AppCleaner to check installed applications, does not find any malicious app installed from Comcast installer.

3. Used pkgutil command to search for anything related to comcast in package receipt database. See pkgutil manpage for details how to use it. I could not find any new applications installed from this installer pkg.

4. Used Pacifist to check the content of the pkg file. Did not see much suspicious content except for several Ruby and shell scripts to run during installation.

Don't install Firefox Toolbar

By default, these website shortcuts are install on your desktop, remove them

Saturday, September 10, 2011

Quickly ran though "Getting Good with JavaScript", this is a very short and basic introduction to JavaScript for someone new to the language. Deep and difficult concepts are avoided or explained in an easy to understand way for the reader to explore further by him/herself.

I found these two resources in the "Further Study" section very helpful and interesting, both are from Rey Bango's blog:

Wednesday, August 31, 2011

And it actually is very straightforward using HTML character entities. The concept is well explained in this Wikipedia article. There are two forms: numeric character reference and character entity reference:

numeric character reference: either in decimal form (&#dddd;) or in hexadecimal form (&#xhhhh;), e.g. "less than" character can be expressed as &#60; or &#x3c;

character entity reference: in the form of &name; the same "less than" character can be expressed as &lt;

Now, it is very clear how to display these special Apple keys with the help of entity tables such as this unicode table. Somehow I could not find the corresponding character entity reference table for the same keys.

If you use TextMate, there is a convenient HTML bundle feature that helps to insert these special characters in an HTML document. Create a new HTML document, then goto Bundles menu => HTML => Entities => Pick the character you want. You can also use the autocomplete, e.g. type "command" and press tab.

Choosing text editor is like choosing your religion. One popular/natural choice on Mac is TextMate. However, since I do lots of work on the server/dev linux box directly, I had to switch back and forth from Vim. Probably it is time to migrate to use Vim on Mac as well, especially given the fact that there are nice distro made for Mac specifically: MacVim and a customized MacVim packaged with many useful plugins called Janus.

Spend 5 minutes to go through the brief introduction for all the plugins that ships with Janus. Janus also includes a handy way to allow you to create/edit ~/.janus.rake file to add more plugins and you can run rake inside ~/.vim to update these plugins.

Command-T: provides an efficient way to search for files in current working directory. Use ⌘+T to show/hide.

snipMate: provides the same Textmate Snippet feature. As you do in TextMate, you can simply type snippet keywords and press tab to expand to code snippets and use tab to jump through various locations in the snippets. You can find all snippet definitions for various languages in ~/.vim/snippets.

ctags: TagList plugin is shipped with Janus. Use :Tlist to show/hide it. But I like Tagbar better than TagList (see below for installation instructions).

Tagbar: displays tags of the current file in a sidebar, sorta like TagList, but more comprehensive and orders tags under correct scope/context.

doctorjs: previously known as jstags, provides a set of static analysis tools for JavaScript. Combined with Tagbar, it provides a more detailed code structure navigation than TagList. See this Stackoverflow post for instructions how to set it up. Note that you need to install NodeJS as a prerequisite.

There are two issues to get doctorjs set up properly that is not covered in that Stackoverflow post:

It seems that installing doctorjs needs sudo (sudo make install inside doctorjs git clone directory) to get around permission issue.

I also got errors about underscore module and ctags NodeJS module cannot be found. They are located at /usr/local/lib/jsctags/ after doctorjs is installed. But node cannot locate them somehow. To fix it, I installed underscore using npm (npm install underscore -g), then I updated my NODE_PATH environment variable to include /usr/local/lib/jsctags/.

The virtual file system is basically based on a big file connected as a loop device. This tutorial about setting up disk quota based on virtual file system has more details. For example, this is the basic steps to set up a virtual ext3 file system based on a sparse file:

The experiment is quite preliminary: basically, I copy over files in different sizes (1, 5, 10, 25, 50MB) into the mounted file system directory or the native file system directory to test write latency, then read the copied files to test read latency. The read/write latency is averaged over 10 iterations.

A couple of quick points from the experiments:

read latency is similar for all four cases.

write latency for native ext3 is around half of the virtual ext3 based either on a sparse file or a zero-filled file.

write latency for native ext3 is roughly the same as virtual ext2, maybe because ext2 gets rid of the journaling overhead.

creating a virtual file system based on a sparse file has many advantages over a zero-filled file: the creation time is way much faster 6-8 order of magnitude for the file sizes we tried, the bigger the file, the bigger the difference; the sparse file does not occupy physical disk space and grows when content is actually written to disc.

virtual ext3 has spikes for write, sometimes as big as 80x the average write latency, not exactly what caused it.

This is final the performance graph for comparison. I am using the average number here, notice that the high average latency for write on virtual ext3 is caused by the spikes mentioned above.

After removing the spikes, using virtual ext3 based on a sparse file or zero-filled file creates around 2X the latency for native ext2 or ext2 based on a sparse file, as show in the second graph:

Friday, July 22, 2011

I am not a frontend developer, just stumbled upon this problem helping out my friend to set up a website that is supposed to serve multi-language content based on the language setting of the user. This is actually an interesting problem (maybe already solved and I did not know).

There are several different solutions that I found through after doing some study:

For example, apache mod_negotiation allows you to define a set of html pages in different languages (index.html.en, index.html.zh, etc.) and based on the user preferences from Accept-Language header, apache picks the right index.html language page to return.

You can also use mod_rewrite to perform the similar trick. Just set up the rewrite condition based on the value of the AL header and map to the correct language.

- Client side detection and action using JavaScript and navigator object.

In IE, navigator.userLanguage returns the operating system region and language setting, in Firefox, Opera, Safari and Chrome, navigator.language returns the browser application language setting. So, we can use them to get the user language and decide what language content to return.

For example, this code snippet (borrowed from an example on jsfiddle) defines multi-language content on the same page and use JavaScript to show or hide it based on the detected language.

The drawback of this approach is that a large part of the content is not needed and invisible. It is a waste of time to download.

Another approach is to use JavaScript to redirect to individual language content pages using window.location.href="redirected_page_url" or window.location.replace("redirected_page_url"). See Location object for more details on how to use them. The downside is this approach may not be search engine friendly (spiders may not be able to follow redirection).

One last note how to test multi-lang website. I tried in Firefox, simple change the language ranking in Preferences->Content Tab->Language Choose button and FF will start sending different Accept-Language headers based on the ranking. You can also play with the operating system language settings.

Sunday, June 5, 2011

If you are annoyed that you have to write multiple nesting asynchronous callback functions when you work with Node.js, you are not alone. Getting used to the asynchronous programming style and always keep in mind everything runs in the event loop is a bit tricky at the beginning.

There are many libraries and articles that addresses this nesting callback issue. This should help you to avoid those ugly pyramid shaped code now ;-)

CommonJS Promises is another way trying to address this issue by defining a standard interface for interacting with the result object of asynchronous actions. Roughly speaking, a promise object is a placeholder for the value for the asynchronous action. And you can treat is as normal object, assign values to another variable, pass it around, etc. Right now, FuturesJS implements this and it is not solely for server-side JS. dojo.Deferred is another similar effort. This SitePen blog post has some easier to understand coverage about the promises.

StratifiedJS extends the JavaScript language to achieve similar goals of allowing asynchronous control flow to be expressed in a synchronous way. Very neat work indeed. This is a short overview presentation about it.

Friday, May 13, 2011

Just had this annoying experience of working on an NFS mounted file system with TextMate. Basically, it tries to store metadata such as cursor position, etc. into metadata files. Since these mounted file system does not support extended attributes, so OSX instead writes to ._filename, which is really annoying since the directory is under version control and you see many of these metadata files showing up.

After poking around, I figured out how to turn this metadata feature off, yes, I lost a bit convenience feature, but no more cleanup of these ._filename anymore!

Saturday, April 23, 2011

Contrary to most people's view on the Amazon cloud outage incident, I believe this actually shows the power and outreach of the cloud computing. The technology needs some time to mature and become reliable, let's just be patient. It is very exciting to see even big sites like reddit, quora was affected, and even the cloud service provider Heroku was one of the victims.

There are many new players in the cloud computing field and Cloud Foundry from VMWare is definitely a very disruptive force to the current market. Currently, it supports Spring for Java apps, Rails and Sinatra for Ruby apps, JVM frameworks like Grails and Node.js! Best of all, the whole platform is open sourced.

CF provides a Micro Cloud for individual developers, for free now ;-) This is how to get a helloworld node.js app running:

1. Preparation

- Sign up for the free Micro Cloud for Cloud Foundry (http://www.cloudfoundry.com/).
- Wait a few days (mine took 9 days) and you will receive an approval email.
- Cloud Foundry uses a CLI tool called vmc for interactions with your Micro Cloud instance. Follow the instructions and use the credentials in the approval email to set up vmc (see "vmc getting started guide", really wish it is available not only in PDF format).

2. Write a helloworld node.js app

As of this writing, CF supports node.js-0.4.5 (use "vmc runtimes" to find out). The only thing that is different in the CF environment is that the system will assign a host and port to your application. CF adds several environment variables accessible through process.env.

For our helloworld, we are interested in VCAP_APP_HOST and VCAP_APP_PORT (there are also VMC_APP_HOST and VMC_APP_PORT, which seems still exist but will be deprecated).

So, you basically create a directory (I use the same appid as the directory name) and create an app.js for hello world like this:

This simple app just prints out "Welcome to Cloud Foundry!" and a bunch of environment variables ;-) Note that we put localhost and 3000 there so that it is easier to debug the app on a local node.js installation.

3. Deploy using vmc

The very first time you deploy the app, you run "vmc push <appid>". Note that you need to pick an id that is not being used, otherwise, you will get an error. It would be really nice for CF to provide a Web UI to check what appid is still available. You just follow the instructions and see the output. If everything goes well, you will see your app running at http://<appid>.cloudfoundry.com/.

To update your app, you need to run "vmc update <appid>" instead of "vmc push <appid>". You can also delete your app using "vmc delete <appid>".

vmc CLI provides many operations, "vmc -h" is your best friend to find and try them out.

4. Add npm module dependencies

In most cases, you will need to use dependent NPM modules such as Connect, Express, etc. With the current node.js 0.4.5 supported on CF, it is very straightforward.

- You create a package.json that defines your app/module with dependencies.
- Then you run "npm bundle", which creates a node_modules directory with all the required NPM modules.
- Finally, you do "vmc update <appid>" and you are done.

I will try to update this helloworld guide and add a bit about bundling dependent npm modules. To learn more about Cloud Foundry, read the CF blog and code, since it is open sourced, you can simply install it from github.

Thursday, April 14, 2011

There are so many node modules out there to the point that sometimes you can have several different ones that appear to solve the same problem.

node.js on GitHub has a list of classified node modules, but still it is quite difficult to pick the best one (in terms of node version compatibility, easy to use, still actively developed, good performance, etc.).

It would be really cool to have some sites that actually give reviews and code samples of using these node modules. If performance comparison can be provided, that is even better.

Saturday, March 19, 2011

Since I got the new ASUS RT-N16, the old Linksys WRT54GL v1.1 router has been sitting there and collecting dust. There is only one phone line at home at one corner of the house, so I have been thinking of using the idle router as a WiFi repeater so that I can get decent WiFi coverage for each room.

Finally I found some time this weekend and studied about it. Many online tutorials cover how to do it using DD-WRT, another great customized firmware for WiFi routers. Since my RT-N16 already runs Tomato, I'd want to find a way to have Tomato on WRT54GL and work as the repeater as well.

After digging into the Web searches, finally I found Tomator author's FAQ for WDS, it is very straightforward. Basically, both both master and repeater routers need to set "Wireless Mode" into "AP and WDS". You also need to turn off dhcp and wan (turn WAN/Internet type into "disabled") on the repeater. Then you configure WDS to "Link with..." each other's MAC address (note this is the Wireless MAC address, not router's MAC address).

WDS stands for Wireless Distribution System, it enables interconnections between access points (AP). Wikipedia definition has more technical details. Ideally, using WDS we can chain more APs together, just make sure you don't keep any loop ;-)

The beautify of this is that it uses the same ssid, so when you move around, it switches to the router with the best signal strength automatically.

To test the signal strength, there is a nice free tool for Mac called AP Grapher, very handy indeed.

Saturday, March 5, 2011

wireshark is one of the best tool to deep network analysis. But due to the permission issue on BPF devices on Mac OSX (by default only root can read and write BPF devices), you won't see network interfaces in wireshare on Mac ;-(

There are two ways to fix:

- run sudo chown <your_user_id> /dev/bpf* on command line before you start wireshark, you will need to do it again after reboot

- add a startup script to fix this permanently (see instructions that comes with wireshark, D&D Utilities/ChmodBPF directory in the installation package into /Library/StartupItems

. You can adjust the script for your needs as well, for details, see the documentation that comes with wireshare.

Saturday, February 26, 2011

node.js has rapid growth and there are so many hosting services available. Ryan maintains a long list of providers on github already and new ones like NodeFu keeps coming out every day. I have personally tried two managed hosting services duostack and JSApp.US, both are very easy to set up and use. The only drawback is that they pose limitations on installed modules, etc.

Amazon is offering 1 year free trial for its cloud services, so called "AWS Free Usage Tier". I decided to give it a try and set up an EC2 instance for node.js.

instead of ssh to the vm using ec2-user@ec2-public-dns, I had to use ec2-user@<address you can find in AWS management console by right clicking on the instance and then clicking connect>, note that the instructions there told you to ssh as root@<address>, but it did not work for me, saying I had to use ec2-user@

instead of --without-ssl option to ./configure, I added openssl and openssl-devel

I did node and npm installation using the "node-and-npm-in-30-seconds.sh" script from this gist mentioned in the Joyeur post "Installing Node and npm"

Again, reading a lot is not as useful as coding a lot. But if you don't use it for day-to-day work, you might easily forget many tricks. So, some quick references are handy in this situation to bring you quickly up to speed.

Saturday, February 5, 2011

Today, a colleague asked about exactly what is the difference between using "new" and calling the function directly. Only thing I can think of is the meaning for "this" inside the function. With "new", this means the object being created, but without "new", this means the global object, either window or something else. And I am also not clear what exactly happens behind the scene.

So, did some search and found some good readings that explains this topic fairly clearly. Lots of related answers from Stack Overflow is also quite helpful:

Using "new" to instantiate a class has both pros (great performance) and cons (can cause nightmare if you forget to use new). Many people including Douglas Crockford discourages the use of new. A module pattern is a more preferable way. John Resig also has a simple solution to avoid using new directly without much performance sacrifice.