Nifty Analysis

Saturday, May 02, 2015

What's new in the latest version?

Highlights of version 5.90

In addition to completely new functionality this version focuses on speed improvements and enhancements of existing functionality. There are hundreds of new features and changes to existing functionality as compared to version 5.80, listed in detail in "Release Notes" document in AmiBroker directory. Below is just a short list of few of them:

Performance improvements

AFL Engine: custom memory allocator does not use Microsoft runtime lib for reference tracking anymore. Result - complexformulas with lots of loops and OLE (especially low-level custom backtests) run upto 3 times faster in 32 bit and 4 times faster in 64-bit

HTMLView - Backtest report viewer - added Edit/Copy, Edit/Select All and Edit/Copy TABLE. The last command transforms HTML tables into CSV format and copies it into clipboard so tables can be pasted easily to Excel. Also it divides Entry/Exit columns into separate Entry/exit date/price columns

new Bid/Ask trend indicator in Real-time quote window - a graphical indicator showing the direction of 10 most recent changes in real-time bid/ask prices The right-most box is most recent and as new bid/ask quotes arrive they are shifted to the left side.

User-definable HTML backtest reports

now it is possible to output HTML instead of graphics in report chart formulas using AFL: EnableTextOutput( 3 ) - HTML output to backtest report

rewritten 3. Profit Table.afl using HTML embedding features auto-scalable layout (so it enlarges when numbers are bigger), bold summary columns, negative values in red, boundary date changed to last day of year/month

Charting improvements

Left/right extended Trend lines and Rays now use user-definable Extension Factor (new field in Study properties) instead of always infinite extent. Ext. Factor equal to ZERO means INFINITE, other values 0.1 ... 26 define how far to the left/right line is extended

Max zoom achievable via View->Zoom Out is increased to 5 million bars, also Pref/Charting/Default zoom limit set to 5 million.

Line drawings now have user definable line width in pixels (new "Line width" field in Study Properties dialog). In addition to that "Thick line" box makes line twice as wide (so actual width of thick line is 2 * lineWidth instead of adding 1 pixel to width)

added ability to control number of decimals in chart value labes via GraphLabelDecimals variable (example, adding GraphLabelDecimals = 2; to the formula would give you value lables with 2 decimal places)

User Interface improvements

Parameter window look and feel improved. Item height is increased and slider thumb made wider for easier use on small size/high DPI screen

New Analysis UI refreshes faster

Colors, bold and italic styles are now added to Interpretation and Commentary windows

64-bit version has now call stack trace in the bug report for better debugging

added more memory checks, early warnings and error messages when running out of memory

added checks for unusual, yet potentially 'troublemaker' scenarios

Highlights of version 5.80

In addition to completely new functionality this version focuses on incremental improvements and enhancements of existing functionality. There are hundreds of new features and changes to existing functionality as compared to version 5.70, listed in detail in "Release Notes" document in AmiBroker directory. Below is just a short list of few of them:

New tabbed user interface with ability to work in both MDI and separate floating frame mode, can be moved behind main AmiBroker screen and brought back (Window->Toggle Frame) (NEW) or kept on top (Window->Keep on top)

In addition to completely new functionality this version focuses on incremental improvements and enhancements of existing functionality. There are 116 new features and changes to existing functionality as compared to version 5.60, listed in detail in "Release Notes" document in AmiBroker directory. Below is just a short list of few of them:

Multithreaded GDI (graphics) rendering - now all drawing (graphic rendering) is done in separate worker threads so the user interface is way more responsive and charts are updated faster and completely independently from each other.

Automatic Walk-Forward out-of-sample summary report - each out-of-sample step produces individual report now, plus there is a new summary report that covers all out-of-sample steps. It is visible in the Report Explorer as last one and has "PS" type.

New Analysis window introduced in version 5.50 brings the following improvements over old Automatic Analysis

multi-threaded operation = speed - new Analysis window uses all available CPUs/cores to execute formulas in many threads in parallel providing significant speed ups. For example on 4 core Intel i7 that can run upto 8 threads, it can run upto 8 times faster than old Analysis window. Exact speed up depends on complexity of the formula (the more complex it is, the more speedup is possible), amount of data processed (RAM access may be not as fast as CPU thus limiting possible speed gains).

non-blocking operation - you can now view, scroll and sort results of analysis while they are still generated, also as user interface thread is not used for processing for most part, charts and other GUI-driven program parts are way more responsive than with old automatic analysis

multiple instances - you can run more than one instance of New Analysis at a time, so you can run many scans/backtest/explorations/optimizations in parallel without waiting for one to complete

slicker user interface - New Analysis window can act as tabbed document, can be floated, buttons can be re-arranged for better workflow. There is way more space for the result list, extra information about execution is provided on the new "Info" tab. Also walk-forward results are now displayed within New Analysis window for less clutter.

Fully Multi-threaded charting. Massively parallel AFL execution (each chart pane runs in separate thread) allows to maximize speed and utilisation of modern multi-core / multi-CPU computers. For example on 8-core Intel i7 CPU your charts will run upto 8 times faster than in version 5.30. The AFL engine has been completely rewritten from ground up to allow multiple instances of the engine running simultaneously. This enables not only multithreading but also enhances responsiveness of entire application, as even badly-written user formula used in a chart is not able to lock or slow the rest of the program. Multi-threading is ON by default. It can be turned off by unchecking "Multi-threaded charts" box in Tools->Preferences, "AFL" tab but it is strongly discouraged. Multi-threading should be ON if you want AmiBroker to operate at full speed.

12 new AFL functions

DateTimeAdd - adds specified number of seconds/minutes/hours/days to datetime

Data Window and data tooltip readout is immediate and does not require extra AFL execution (values required to display them are stored in RAM during normal chart refresh and available without need to re-run the formula). Tooltip variable is now obsolete. To display custom values in tooltips without plotting a line you can use Plot() with styleHidden flag.

Interpreation display does not require AFL execution

Inserting indicator and resetting parameters are orders of magnitude faster

chart zoom setting is now saved in a layout file and restored when layout is loaded

QuickAFL implemented in the Automatic Analysis - speeds up backtests, optimization and explorations by factor of 2 or more (if range is less than all quotations). (Note: in order to enable it you need to check "Use QuickAFL" box in the Automatic Analysis setttings).

Chart template sharingnow you can save the chart as "Chart Template, Complete (*.chart)" that stores all layout AND referenced formulas in SINGLE file that can be sent to your friend and entire chart will be restored on any computer with ease, without need to copy individual formulas.

New-Look charts - divider lines between panes are now single pixel and no borders around charts giving cleaner, larger and more readable chart display and printout

added Text-To-Speech capability via Say() AFL function. Now AmiBroker can speak out loud any text, for example it can say "Buy 100 shares of AAPL at 91". This is controllable from formula level so you can make it to speak depending on market conditions, signals generated from your formula, etc.

Detailed Change Log

New Analysis: If current formula is open in the formula editor it is auto-saved prior to running any Analysis (as old AA did)

New Analysis: on Backtest/Optimization current formula is saved in @LastBacktestFormula registry key so it can be referenced by Individual Equity formula

New Analysis and Old AA: HoldMinBars > 0 can now be used together with AllowSameBarExit=True Settings: 'Stops' tab: N Bar stop has priority check box state was not saved. Fixed

New Analysis: on attempt to load analysis project file that contains formula path that does not exist or is not writable AmiBroker will create the formula in the "Formulas\Imported" folder instead of not creating it at all

New Analysis: WinXP-only - RMB click column descend/ascend menus were not working due to Windows XP bug. Fixed.

New Analysis: when formula is picked from the folder that is inside current working directory, then relative path is used and stored in project instead of absolute path previously used

New Analysis: SeparateLongShortRank required proper MaxLongPos and MaxShortPos definitions in order to work. Now if they are missing defaults are applied as in old AA.

Plugins: IQFeed: added "You can't close plugin window" on attempt to close working IQFeed window.

UI: Chart tree, Layout tree in "Explorer" theme

UI: Category list now features new icons and decreased indentation

UI: Owner-draw listviews now fully implement ellipsis item truncation mechanism, but using ExtTextOut only which is 6 times faster than DrawText that Windows uses

Charts: fixed small alignment problem on Y axis when using fixed (nonproportional) font

New Analysis: backtester object now has EquityArray property that returns entire equity array (not just current value).Please note that values are filled during backtest and only after backtest is complete, all values are valid. If you call it in the middle, it will contain only "upto given point" data. Avoid abusing this function and it is costly in terms of RAM/CPU.e).You may usebo.EquityArray instead of Foreign("~~~Equity", "C" )in custom backtester code.It accesses local copy of equity in New Analysis (unlike Foreign that accesses global symbol)

New Analysis: AddToComposite with atcDeleteValues | atcEnableInPortfolio did not delete previous values. Fixed

New Analysis: "Interval" menu item in the auto-repeat group renamed to "AR interval" because some users interpreted it wrong (as periodicity)

New Analysis: "Save" creates now a Analysis project (.apx) file that includes all settings and formula needed in single file. The file itself is in human readable XML format. When such file is loaded on somebodys' else computer it will recreate original formula if one is not present. It will warn if there is a name conflict and let you decide to keep existing formula or overwrite.

UI: File open dialog now has "All supported files" option in the "Files of type"

UI: PopupWindow is now resizable (see size gripper in the lower-right corner)

New Analysis: Errors are now displayed in a modern message bar instead of dialog box

UI: holding down Ctrl and Q keys simultaneously while AmiBroker is starting resets all menus/command bars to defaults

New Analysis: scroll bar properly works when "Walk forward" tab is active

New Analysis: Walk forward tab content can be now copied using Edit->Copy / Ctrl+C

New Analysis: when "wait for backfill" was turned on and external data source was very slow to repond (like IQFeed during backfill) subsequent calls to Status('stocknum') could give non-unique numbers. Fixed.

AFL: CategoryGetSymbols supports new categoryAll (all symbols in the database) and new Mode parameterCategoryGetSymbols( category, number, mode = 0 );Mode parameter decides what field is retrived:0 (default value) - ticker symbol1 - full name
Example: to get all symbols existing in the database simply callCategoryGetSymbols( categoryAll, 0 );to get full names of all symbols use:CategoryGetSymbols( categoryAll, 0, 1 );

AFL: Added Warning 503. Using OLE / CreateObject / CreateStaticObject is not multi-threading friendly. See Users' Guide ""Efficient use of multithreading"" for more details.

AFL: Added ThreadSleep( milliseconds ) function. It suspends current thread for specified number of milliseconds (maximum is 100 ms). Works only from NON-UI threads. When called from UI thread the function does NOTHING and returns immediatelly. Please do NOT abuse this function. Using it may negatively impact performance.

Parameters" varname"Specifies the name of the destination static variable. Static variable if exists must be scalar numeric type. If static variable is not initialized, the function assumes that it has value of zero.exchangeSpecifies the exchange value. Scalar numeric.ComperandSpecifies the value to compare to the destination static variable. Scalar numeric.Return ValuesThe return value is the initial value of the destination static variable. If variable did not exist, it returns zero.The StaticVarCompareExchange function performs an atomic comparison of the "varname" static variable value with the Comperand value. If the static variable value is equal to the Comperand value, the Exchange value is stored in the static variable. Otherwise, no operation is performed.
The function StaticVarCompareExchange provides a simple mechanism for synchronizing access to static variables that are shared by multiple threads. The following examples show how to implement semaphore and critical section in AFL using StaticVarCompareExchange function:// EXAMPLE 1 : Simple semaphore (no waiting)if( StaticVarCompareExchange( "semaphore", 1, 0 ) == 0 ) // obtain semaphore{// protected section here// Here you have exclusive access (no other threads that check for semaphore will enter simultaneously) /////////////////////////StaticVarSet("semaphore", 0 ); // reset semaphore} else{_TRACE("Can not obtain semaphore");}///////////////// EXAMPLE 2 HOW TO IMPLEMENT CRITICAL SECTION IN AFL///////////////function _TryEnterCS( secname ){global _cursec; _cursec= "";

New Analysis: Testing on unaligned data sometimes took longer than expected because of the work necessary to bring signals in order. Now a smart method avoids most of the memory transfers and can give 10x speedup on backtesting very long unaligned intraday data.

New Analysis: "Backtest started", "Exploration started" text in the info tab appeared twice per run (instead of once). Fixed

New Analysis: Backtest report did not include formula code. Fixed.

New Analysis: Fixed crash that could occur when running analysis with Range set to "N recent bars" and N was exceeding number of quotes.

New Analysis: Implement single-thread start when Status("StockNum") is detected in the code. Other threads are started when first symbol processing is completed.

New Analysis: Send to Analysis did not update formula combo box properly in certain scenarios: when formula X was selected, then formula Y, then formula X again. Fixed.

Old AA: status bar displays true timing for fair comparisons between old AA and new Analysis. Watching progress bar numbers was misleading as it closed before all processing was truly complete, so it looked like old AA completed sooner than it really did

UI: menu item for Old Automatic Analysis restored as apparently some users got lost. Also attempt to run old Automatic analysis displays an information about new Analysis window

CHANGES FOR VERSION 5.49.0 (as compared to 5.48.0)

New Analysis: when floating pane is made normal from/to dates were reset to today. Fixed.

New Analysis: when Apply To: Filter is defined so NO symbol matches the criteria, appropriate message is displayed when trying to run Analysis and action is terminated instead of displaying progress bar forever

New Analysis: when analysis window is "floating" double click/show arrows/sync affects active MDI chart just like old AA did, as opposed to last one (when analysis is not floating)

New Analysis: Optimization parameter setup stage uses now not more than 100 bars. This reduces setup time.

New Analysis: Implemented "Wait for backfill" option. This slows down analysis sequence so AmiBroker waits for external RT data source to deliver backfill dataWhen "wait for backfill" option is turned on, AmiBroker checks if data for given symbol are complete, if not, it sends request to external source to backfill the data and waits 1 second.After one second, it checks again, it data are complete, if not, it waits another 1 second, and so on. Once data are complete given symbol is used for formula execution.

UI: when focus is switched to floating document window, the application menu is switched to proper context

UI: When File->Close was used on floating pane, document was closed but docking pane was not. Fixed.

UI: The toolbar button and menu option to launch old Automatic Analysis window are now hidden (can be brought back by using Tools->Customize), old AA can also be launched from new Analysis via Analysis->Old Analysis menu

UI: New menu options and toolbar button to launch new analysis window

UI: Moved all menu items controlling visibility of docking windows to "Window" menu

UI: floating state of windows can now be turned on/off using right-click menu over tab / floating window caption

New Analysis: Signal heaps are separated now, allowing to run concurrent backtest/optimizations in many analysis windows

New Analysis: removed unnecessary refreshes of symbol list during optimizations that caused slowdowns when user had large database

New Analysis: The size of control fields is properly adjusted with regards to DPI

New Analysis: Status("StockNum") was always zero in 5.47 and earlier. Now it is set properly.Caveat: be careful when you run codes that use Status("StockNum")==0 for special processing in multithreading mode.Other threads with "StockNum" = 1, 2, 3,... will run simultaneously! If you store static variables in step 0 these variables may be empty if you access them from other parallel running threads.

Fixed Error 53 message that was giving an error when proper \" (quotation mark espace sequence) was used

CHANGES FOR VERSION 5.47.0 (as compared to 5.46.0)

New Analysis: last used ApplyTo/Range/From/to dates/filters and other settings are saved when Analysis window is closed

AFL engine: new error 53: incorrect \ espace sequence. Error is issued when user forgets that single slash must be written as \\

Default title for new analysis window is "AnalysisX" (instead of "Unnamed") where X is consecutive number 1, 2..

attempt to use Say() on computer without sound card or without driver installed does not result in throwing exception/crash report, but it is rather silently ignored

CHANGES FOR VERSION 5.46.0 (as compared to 5.45.0)

New Analysis: Exhaustive multi-threaded optimization implemented

New Analysis: Smart (non-exhaustive) multi-threaded optimization implemented in a way compatible with existing optimizer plugins

Note however that due to the fact that most non-exhaustive optimization plugins are not re-entrant, AmiBroker prevents from running more than one non-exhaustive multi-threaded optimization at a time. You can however, run many exhaustive optimizations in parallel, as they don't require external plugins.

Slight modification to the way how compressed bar charts are drawn (so bars with H==L are better visible)

CHANGES FOR VERSION 5.45.0 (as compared to 5.44.1)

64-bit version compiled with new version of C runtime (Microsoft security update)

Fixed date/time column sorting in virtual list views when regional date format was different than yyyy-MM-dd

AFL: ClipboardSet() returns True (1) when clipboard has been successfully set. Sometimes clipboard is locked by another applications and then write can fail (ClipboardSet() will return zero/False then)

New Analysis: Changed alignment of Date/Time column to left in backtest result list.

Most AFL engine setup moved to worker threads. Result - smaller UI thread load in MT charts (better scalability) and much faster (upto 2x) explorations/scans in New Analysis window

NOTE:End users should only look at "Completed in... sec" line that shows actual run time of the analysis (scan/exploration/backtest).The times shown in "timings" row are for Amibroker.com internal use. In development/testing they help us tweak the performance of various parts of analysis engine. The detailed timings do not sum up to actual run time. No further infomation is available at the moment. The "timings" row is subject to change/removal in the future versions.
If you have problems/questions with the performance of new Analysis window please copy (Ctrl+C) the contents of Info tab and send to support.

Note that only first phase of backtest is multithreaded (each symbol in separate thread). 2nd phase (custom backtest) is executed in main thread as there is only one symbol to work on (i.e. equity) and it needs to talk with OLE which is single threaded and GUI.

Load/Unload buttons removed from Plugins dialog. This was developer-only feature and it was abused by ordinary people who were unloading plugins that were actually in -use (such as data plugins when being connected to given data source)

ListViews: the last line (partially shown) in RT quote window was not easily selectable. Fixed

New Analysis window: Implemented Parameters dialog

AFL: GetCursorMouseButtons() sometimes missed clicks when multithreading was on and formula took ages to execute. Now it is now 100% reliable.

Auto-selection of proper layer based on selected viewing intervalNOTE: This feature can be turned off by click with RIGHT mouse button over layers window and UNCHECKING "Auto-select layer"

"SettlementDelay" option describes the number of days (not bars) it takes for sale proceeds to settle and be available for opening new positions.SetOption("SettlementDelay", 3 ); // this will cause that proceeds from sale are only available for trading on 3rd day after saleFor detailed tracking" Detailed log" report option now shows available and unsettled funds for T+1, T+2 and so onNote: when using this option it is recommended to use backtestRegularRaw instead of backtestRegular, otherwise some trades may not be enteredbecause funds are not settled immediately and you need to be able to enter not on first but subsequent buy signals and that is exactly what backtestRegularRaw offers.
Note2: old backtester (Equity() function) ignores settlement delay

Zoom in/out toolbar buttons zoom in finer steps (+/-10%)

Analysis ListView, RMB menu new option: "Add Rank column" - adds a column with ordinal rankings based on current sort or just row number column when list is not sorted

In 5.41.0 did not display Time&Sales in reverse order (newest on top). Fixed.

CHANGES FOR VERSION 5.41.0 (as compared to 5.40.3)

New Analysis window (File->New->Analysis) - featuring multithreaded Scan and ExplorationPerformance note:to fully benefit from multithreading it is best to use AmiBroker local NATIVE database (not external)as it is the only database that can deliver data quickly enough.Tests show that on AmiBroker native databases scans and explorations are 100% scalable to multiple cores- i.e. for example would run upto 8 times faster on 8 CPU/core machine.

IMPORTANT: This window is work-in-progress. Multithreaded Backtest and Optimization features will be added later.

Totally rewritten listview with very efficient owner draw code speeds up list view scrolling/resizing/repainting more than 10 times. Native system ListView control is terribly slow on Windows XP/Vista/7 (surprisingly it was many times faster in Win9x!)

AFL: Percentile() does a param check for range > 0 now

Mini-chart in RT Quote window - Mini chart shows where LAST price is within LOW-HIGH range.

Errors and warnings are now displayed in red color in the indicator pane

During actual GDI chart rendering AmiBroker measures time and if it exceeds 500ms timeout it stops rendering with Warning 901.This check is required because GDI rendering occurs in GUI thread and if GUI thread does not process messages for half second Windows will think that application is "not responding".This prevents making application stuck with some overly complex, possibly incorrect formulas that call hundreds of Plot()s

Warning 502: Calling Plot()/PlotOHLC over 500 times is displayed in indicator in runtime to prevent abuse

if data are missing in the in-memory cache, Foreign() calls from non-GUI threads use synchronous retrieval method for reliability.This provides reliable results in AA but may slow down the multi-threaded operations if cache size is too small and user uses more Foreign() calls than in-memory cache size.IMPORTANT:For optimum performance in-memory cache setting (Tools->Preferences->Data) should be GREATER than number of Foreign() calls in any single formula used.Values lower than that cause lots of disk activity and serialization of all Foreign calls that means that additional threads must wait for data retrieval to complete.

Old and new Analysis window: outputting millions of rows to listview is much faster now

All list views - Ctrl+A (Select All) made much faster

CHANGES FOR EARLIER VERSIONS ARE DOCUMENTED IN RELEASE NOTES DOCUMENT THAT YOU CAN FIND IN AMIBROKER INSTALLATION FOLDER.

About ME

Disclaimer

Though all care is taken in arriving at the recommendations given in this Blog, the readers are cautioned that the prices of equity shares/debentures may rise or fall in a manner not foreseen. All possible efforts have been made to present factually correct data. However the Blog is not responsible if despite this , errors may have crept in inadvertently or through oversight. The analyst or blog are active market participants, and, at the time of the recommendations put up on the site, may or may not hold any positions in the stocks mentioned. This is only for education.