Congratulations folks! The tip is magnificent, although if the script gets much longer it will need to be made a plugin on vim.org (I'm resisting the tempatation to make it handle tab delimiters, which wouldn't be too hard now we have GetExpr()).

Yep, this tip has progressed very well, and it now has a quality that would warrant uploading to vim.org. The big disadvantage of vim.org is that only a single owner is able to upload new versions, and if that owner goes away, nobody can contribute any more. Here on the Wiki, everyone can collaborate on a script, and there doesn't need to be a single owner. How should this be handled? Maybe there could be one vim.org login VimTipsUser, with password and rules and guidelines posted somewhere here in the Wiki, that can then be used by anyone from the Wiki community to upload new versions to vim.org?! (Assuming the vim.org owners are okay with posting such an open login to their site, and the potential for spam and vandalism.)

What do you think? Have other tips from here already made the move to a vim.org script (without a single owner/adopter)? -- Inkarkat 11:04, 13 July 2008 (UTC)

That's an attractive idea. I vaguely recall (but have forgotten the details) that in recent months there was a tip that someone developed into a script on vim.org. In the past, that has happened several times. However, as you point out, the script then suffers bit rot because usually the author moves on to other things and no maintenance occurs.

We have no system for collaborative access to vim.org. At the moment, I'm inclined to leave things the way they are until we get a case where someone really wants to proceed. We could have a public name/password, or we could have a page explaining that there is a well-known account, and if someone wants it they can email [list of 3 or 4 users here]. For email, I would be inclined to NOT publish any email addresses (even if obfuscated). Instead, rely on the Wikia procedure whereby there is an "email this user" link (that requires the sender to have a Wikia account, and the recipient to have supplied an email address; the sender does not see that address unless the recipient replies).

Also, I'm somewhat selfishly wanting to keep this tip here, at least for the moment. In general, I wouldn't want to have that long script on a tip and a maintained plugin on vim.org. So, if you really want to make a plugin with yourself as the initial maintainer, let's sort out the details. Otherwise, let's wait until this tip has developed even further.

Another option: It is possible to upload binary files to the wiki. So, we could upload a zip (or text file, or whatever), and provide a link for a download. This procedure uses the "Image" feature of the wiki (whereby it is possible to upload images). I'm not sure what Wikia thinks of this, but I could easily find out (I don't think they would mind), and I have noticed it elsewhere (we haven't done it). --JohnBeckett 00:28, 14 July 2008 (UTC)

Just so everyone knows, we normally try to avoid using talk pages to comment on tips. Instead, if you see something that should be noted in a tip (an improvement, a defect, a question about the tip), please just add to the Comments section at the bottom (use a line consisting of only ---- to give a horizontal rule to separate each comment). We find that generally is easier for editors and readers (if there may be a problem in a tip, a reader should see the comment raising the problem, without needing to examine the talk page as well).

Note also that here we don't keep comments (we just want tips without too many distractions). As soon as a comment is dealt with, it should be deleted (although if a newcomer has a comment, I would generally keep the comment for at least a week, with some sort of reply to indicate why the comment was not applicable). Likewise, unless this discussion keeps going, I'll probably recommend deletion of this talk page in a couple of months (if there's nothing of long-term value remaining on it).

This is the first time that we have had a real reason for holding a discussion on the talk page of a tip, so go wild with any suggestions! --JohnBeckett 02:00, 13 July 2008 (UTC)

Question: Is it possible to highlight the column header on the status line to be the same color as the highlighted column? It is some small convenience, I think it would be nice to have although what we have now is pretty good already.

Yes, it's possible (after all). Good (though tiny) point! I've added it to the code; in the status line, the column heading is now highlighted the same way as the current column. -- Inkarkat 20:32, 20 July 2008 (UTC)

I just noticed the feature, while using the code. It makes using the code just a bit more pleasant. Thanks! -- Lpb612 19:02, 23 July 2008 (UTC)

When I get some quality time, I might try and better understand lpb612's amazing regexs. I suspect that some minor tweaks would give a pattern that you could use so :sort would sort based on the fields in a column! --JohnBeckett 12:02, 19 July 2008 (UTC)

Please see the main page. There is a new section explaining the regex. -- Lpb612 06:06, 20 July 2008 (UTC)

This feature has been added. See the main page. I used a function instead of a simple :sort call, because sort call with the "skip" feature (ie. sort /pattern/) can probably do correct sorting in most cases, by skip the first n-1 columns, but then the text in all remaining columns are ordered, instead of just the n-th column.

A "delete column" feature has also been added.
--Lpb612 20:40, 20 July 2008 (UTC)

Suggestion
Look at the form of the entries in the column and convert to numbers if that's what they look like (this will improve the sort when sorting tables of data). Something along the lines of:

I'm sure there's a better way to do this (and it probably has undefined results if columns have multiple types present), but it's a start. --Al 13:30, 1 August 2008 (UTC)

This is nice, but more than "return comp1 > comp2" is needed. Also, the automatic type detection makes me feel a little nervous. It might be better if it were done once, before the start. Anyway, I have preferred another approach in an update I've put in the tip.

I notice that :SC 2=john works very nicely, but it doesn't highlight the hits (I use :set hlsearch). The following line used instead of the current line seems to fix this issue. I used 'normal' (not 'normal!') so that the mapping for 'n' is used.

Expands undo_ftplugin so more changes are undone, including removal of the autocommands. That means set ft= really does turn off highlighting.

Uses 'normal n' in the SC command so the first search highlights hits (if 'hlsearch' is set).

Uses matchadd() rather than :match so a priority can be assigned to the highlight. That allows search hits to be highlighted in the current column (and leaves the :match command available for the user).

[RESPONSE] I checked the new version with matchadd. It works where matchadd is provided (new version of vim?). But it seems that not all VIM 7.1 has matchadd. Another solution, as you (JohnBeckett) mentioned is the following: use "2match Keyword" instead of "match Keyword" in the s:Highlight function, and add

execute 'match Search /\c' . @/ . '/'

at the end of SearchCol function. This way, the SC search has a higher priority than "Keyword" match. Thanks JohnBeckett for both pointing out the problem, and also providing (two!) solutions. Update: Now I have added both methods to the code so that if matchadd is missing there is no warning message, and still highlighting search is correctly done--- Lpb612 06:13, 20 July 2008 (UTC)

I guess everyone knows that you can use the 'history' tab on the wiki to see old versions. If you click the date for a particular version, that version if opened. If wanted, you can edit the old version and copy out all the wikitext to a local file. That's a good way to compare two versions, although the wiki difference tool on the history page is surprisingly good and is often all that you need. Be careful not to save an old version unless you want to wipe out changes since that time. --JohnBeckett 04:21, 20 July 2008 (UTC)

[RESPONSE] The b:undo_ftplugin didn't execute anything after the :call matchdelete() on plain VIM 7.0, because (I think) the | binds stronger than the :sil!, and so if matchdelete() fails, the following commands aren't executed, and the mappings aren't undone. I fixed that, and also added the 2match none undo for plain VIM 7.0. I also removed the nnoremap <buffer>n n, which is a no-op, isn't it?! At least, highlighting also works for me without this. -- Inkarkat 20:29, 20 July 2008 (UTC)

[DISCUSSION]I think the

nnoremap <silent> <buffer> n n

should not be removed. Otherwise, there will be a "Press Enter" message for a long regular expression. Try a CSV file with many columns (e.g, 5 or 6) and try to search in the last few columns, you should be able to notice the problem. The trick is the <silent> part, which suppresses the message. -- 12.202.152.182 00:36, 21 July 2008 (UTC)

I agree with lpb612, and have restored the mappings for n and N. BTW, if you use a secure computer (which only you access), you can select the "remember me on this computer" option when you log on. I think you then only need to log on once a month, and it won't log you off after a timeout.

Sorry, I cannot reproduce the "Press Enter" message for a long regular expression. The VIM 7.1 default 'n' command seems to automagically shorten a long regexp in order to avoid this. Do you have any plugin / custom mapping for 'n' which could interfere here? (Try with gvim -N -u NONE; filetype plugin on; syntax on; e test.csv; SC ...) -- Inkarkat 09:24, 21 July 2008 (UTC)

OK, I think I found where the problem was. I had "set shm=at" which is supposed to make message shorter. But that has the strange side effect of the Press-Enter message. Lpb612 18:10, 21 July 2008 (UTC)

Breathtaking work lpb612 and Inkarkat! I couldn't resist tweaking the code a bit, but I haven't changed what it does. Some questions on the "05:30, 21 July 2008" version follow. --JohnBeckett 08:19, 21 July 2008 (UTC)

By the way, the wiki way of responding is like this (start your response with a ':' to indent your reply).

If you want a line break (new para), also start the new line with ':'. Add your signature ~~~~ on the same line (not on a new line). JohnBeckett 08:19, 21 July 2008 (UTC)

And here is a reply to the response (this one uses '::' for more indent). JohnBeckett 08:19, 21 July 2008 (UTC)

Thanks for the help. I am learning to use wiki. Lpb612 17:17, 21 July 2008 (UTC)

Function SearchCol() finishes with:

execute 'match Search /\c' . @/ . '/'

I don't understand the point of this. If the user has 'hlsearch' then the match is redundant? If the user has 'nohlsearch' then they do not want all hits highlighted? Also, if you do a search, then use ':noh' or ':set ft=' to switch highlighting off, the :match is still active (i.e. highlighting is not switched off). Note also that a search like :SC a/b (embedded slash) gives an error with the match. So, I think this line should be deleted. --JohnBeckett 08:19, 21 July 2008 (UTC)

Even if hlsearch is on, the 2match still "masks" the search. So the priority seems to be like this (low to high): hlsearch,...,2match,match. Lpb612 17:17, 21 July 2008 (UTC)

Maybe we can check for hls before executing match Search? Lpb612 17:52, 21 July 2008 (UTC)

A test suggests that whichever of 'match' and '2match' is used last wins. On Vim 7.1.40 or later, enter ':echo getmatches()' to see the matches. They are listed in the order that the commands were entered, and it appears that the last in the list is applied last, and therefore "wins".

To expand on this, the :match family acts just like matchadd() with the default priority of 10. Search highlighting with hlsearch has priority 0. Highest priority wins, though as John has discovered, last defined will win within the same priority. --Fritzophrenic 12:41, 22 July 2008 (UTC)

Update I've spoken with Bram about this (because :help :3match contradicts the above). What I and Fritzophrenic describe above is a bug introduced in Vim 7.1.40 (I've suggested a patch to fix it). What is supposed to happen (as in help) is that :match always beats :2match, which always beats :3match. Prior to 7.1.40, that's what happened. --JohnBeckett 11:43, 24 July 2008 (UTC)

Are you saying that the purpose of the 'match Search...' is for people who have a version older than 7.1.40? If so, perhaps some further ugly 'if &hlsearch && !exists("*matchadd")' code could be used if you really need this statement for older Vim. In general, I would advise against adding complexity to add a feature for old versions. However, a lot of people will have versions older than 7.1.40 so I guess some complexity would be justified, but only if the feature is really needed.

I am also against making code ugly unless necessary. Removing the line would be fine with me. For "older vim" users, they will just not be able to see the highlighted search when the search is in the current column (not big deal). Lpb612 04:35, 23 July 2008 (UTC)

I have commented out the line pending your thoughts because the command gives problems for my newer Vim: I don't need it, and I can't easily remove its highlighting (I map Space to :noh to remove search highlighting; pressing Space does not remove the 'match Search...'). --JohnBeckett 10:48, 22 July 2008 (UTC)

I don't know the difference between execute and call (this is my first vim script). So I just tried the first one that worked. Please feel free to change it. In fact, after reading the help manual, I still don't quite get it. It seems that execute is like the eval in bash? Lpb612 17:17, 21 July 2008 (UTC)

Use 'call' to invoke a function (it's a syntax limitation that 'call' is required). Use 'execute' to execute an expression. --JohnBeckett 10:48, 22 July 2008 (UTC)

Sometimes the code displays a message like this:

echohl WarningMsg
echo "column number out of range."
echohl NONE

and other times like this:

echoerr "column number out of range"

Is there a reason for the difference? We could almost justify having a function to display a warning message. --JohnBeckett 08:19, 21 July 2008 (UTC)

There should be no difference. echoerr should be changed. I will change it. Lpb612 17:17, 21 July 2008 (UTC)

Removed the n and N mappings (Inkarkat was correct: they are not needed; I had jumped to some conclusions).

I think they are needed if the use happens to have set shortmess variable (at least if set shm=at) -- Lpb612 20:43, 22 July 2008 (UTC)

If shortmess is set like that, the user doesn't want truncation of such messages, and will deal with the "Hit ENTER" (e.g. by reducing its likelihood via a larger 'cmdheight'). csv.vim should be consistent with the default behavior of the '/' and 'n' commands. -- Inkarkat 21:21, 22 July 2008 (UTC)

Removed trailing space from message "No comma-separated columns were detected. ". If it is wanted, consider having the new Warn() function add it to each message (however, Vim does not add spaces around warnings).

In DeleteCol() I replaced 'let b:csv_column=b:csv_max_col' with 'call s:HighlightPrevCol()'.

Previously, using :DC on the last column left no column highlighted, yet another :DC would delete another column.

Problem (not critical): The nice PrintColInfo message is not appearing (instead, get result of the global substitute command). I inserted redraw + redrawstatus and one or two other things, but could not make the PrintColInfo message appear.

Another issue is that if you :DC then u to undo, csv_max_col is too small. Need a way to rescan (without too much complexity!). You can rescan by saving the buffer and using :e.

I used 'echohl Type' (not 'echohl Keyword') in PrintColInfo to match the attractive highlighting of the header row.

In a week, I will be featuring this tip on the Main Page. Therefore I think I should move the "Wish List" to the talk page (which I intend to keep, although I might delete some obsolete comments). --JohnBeckett 03:17, 24 July 2008 (UTC)

As the code gets longer, it is inconvenient to cut and paste code from wiki to the hard disk. The following shell script can be used to download and save the code on the main page. It depends on wget, lynx, and awk.

Thanks for the tip. There is just no end to improvements! -- 12.202.152.182 05:06, 25 July 2008 (UTC)

Hi lpb -- matter of fact, I'm thinking of uploading a tweaked version of csv.vim very soon. It's not a substantive change, just looks more elegant to me.

Suggestion: Use the Show preview button! Then you will notice things like the fact that wikia didn't think you were logged on. I think I mentioned that you can choose an option to be remembered when you log on (you can visit wikia next day, and still be logged on even after shutting down). --JohnBeckett 11:18, 25 July 2008 (UTC)

The header line (line 1) can be differently highlighted. Does anybody know how this can be done? I cannot think of a way of doing it. --Lpb612 19:05, 23 July 2008 (UTC)

How about this? (The main trick is the \%1l regexp item, which matches line 1; you know the rest of the regexp :-) Strictly, syntax highlighting shouldn't be done in the ftplugin, but in a separate syntax/csv.vim, though. -- Inkarkat 20:37, 23 July 2008 (UTC)

The original script called sort() with a compare function. However, that function tested only for >. A sort compare function needs to return negative/zero/positive for < == > (the docs actually specify -1/0/1).

I decided to use the built-in :sort command instead as it supports more options and has much better performance. However, :sort n ... sorts integers only, and regards any field that does not start with an integer as zero. Therefore, the script supports an f option that calls sort() using a compare function that handles floats and integers.

The sort code calls sort() if the f option is used (to sort floats), and uses the :sort command otherwise (for better performance and more options). The sort code works on a range of lines (default is whole file), and accepts ! to sort in descending order.

str2float() converts string fields to floating point:

It correctly converts numbers like 12, -1.2, 1e2, .5, 0.0.

It ignores trailing junk ("1.5abc" is 1.5).

It returns 0 for an invalid number ("abc1.5" is 0).

Rather than use a regex to restrict what a valid float should look like, the code calls str2float(). If the result is 0, a simple check is done to see if the field really is zero. If not, the field must be invalid, and the field is returned as a string. The CompareLines() function compares floats with floats and strings with strings. When comparing a float with a string, the string always comes first.

When comparing strings, if :set ic? shows that 'ignorecase' is set, then a string like "abc" will be regarded as equal to "ABC", and "abc" will sort before "XYZ".

I have enhanced the script with a command to specify the delimiter, and a global variable to set the default delimiter (see the "Features" documentation above script).

The delimiter has to be a single character because many of the regular expressions include things like '[^,]' (search for not the delimiter). However, you can use a multibyte character as the delimiter. For example, assuming UTF-8 or other suitable encoding, you can press Ctrl-k and type .M in the command line (:Delimiter ·) to specify a middle dot as the delimiter. The new CharLen() function uses a trick from Vim help to determine the number of characters in a multibyte string.

It is possible to use '/' as the delimiter (:Delimiter /). Therefore, I have called escape() a couple of times to put a backslash in front of any slash (there will only be a slash if the delimiter is a slash). That is necessary to make commands like :s/xxx/yyy/ work with an embedded slash.

I know a similar topic is posted elsewhere on the wiki, but I wanted to tell anyone looking at the CSV script and running into errors with MacVim or Vim that this DevDaily blog post solved my issues when copy and pasting the sync script above.

Following the procedure after copying and pasting the CSV script code works as well to alleviate any errors on my machine. YMMV. --129.174.97.34 17:58, September 17, 2009 (UTC) Jeffrey Horn

Thanks for info. I had a very quick look and it looks like it's just saying (badly) to do :%s/\r//g to remove extraneous ^M characters. Surely that would only be needed if you copied a script on (say) a Windows system, then tried to use the script on a different platform? Is there anything about this particular tip which requires this info? JohnBeckett 21:23, September 17, 2009 (UTC)

Change GetStr() so it works with delimiters that have special meanings (like '~' when using the pattern for searching, and '&' when using substitute() to replace ',' with '&').

Add a CopyColumn() function with :CC user command to copy a column.

I also did some minor refactoring. Given that the user can enter their own search pattern to search a column, I found it too tricky to use \V to turn each regex into "very nomagic", so some fairly heavy kludges were required to work with the special characters.

More testing is needed. Please add any comments here, or in the "Comments" section at the bottom of the tip. JohnBeckett 03:20, February 21, 2010 (UTC)

I am relatively new to VIM scripting, please apologize if I made a blatant mistake...

I have followed the installation instructions closely and have configured the csv.vim on both VIm 7.2 for Windows (gVIm) and for Mac OS. In both cases I have downloaded the most recent executable available from vim.org, just to make sure I hadn't miss something important.

In both cases I got consistent behaviour: opening a comma delimited csv file with header the first column will be correctly highlighted, many commands like 0, $ behave as expected, but cursor movements (h, j, k, l) do NOT move to columns. They continue to behave as in "regular" VIm.

I need to mention that VIm in both cases has recognized the filetype as csv.