Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.

Suppose you have a big futures and commodities portfolio and wish to update correlation data in the FuturesInfo.txt file. For the Euro Currency, the strong correlated symbol list looks something like this
AD
..
BAX
BO
CD
...
SF
...

How would you generate such a list with minimum effort?
My solution is to use CSI on-line "Correlation Lab" and start off with 15 year correlation study.

The Correlation Lab generates a lot of data so it needs to be stripped and formatted for FuturesInfo.txt consumption. Doing this in Microsoft Excel is a very time consuming task. Better solution is to automate the task with single line shell script. Here is how.

$ cut -d "," -f 5 cu_80.txt | sed s/[0-9]// | sort | uniq | tr -d \"

OK, this is the whole automation.
This will run in any shell. (While Microsoft includes Posix compliance for government customers--mandated--retail version of Windows lacks Unix tools.
For users with no prior experience, try

Back to the one line shell script: This program does is take the CSI Correlation lab listing as input. (The Euro Currency results were saved to a file cu_80.txt)

The -d defines Delimiter as , (comma) and -f selects Field 5 to cut.

At that point, we'd like to get rid of duplicate lines, but first need to remove numbers from symbol names. CSI will distinguish between SF, SF2 as electronic vs pit sessions. Pit and electronic correlation ought to be about equal, so root symbol is all we need. Therefore, next in the pipeline,

sed s/[0-9]//

The Stream Editor sed Subsitutes (s directive) any number [0-9] for nothing.
The [] square brackets allow pattern specification.
The // indicates the the new pattern is nothing since nothing was specified.

Now, we have just a list of symbol names except we may have duplicates AD, SF, etc., occurrences. To filter duplicates,