Due to performance issues with the TreeView implementation in Windows, building a tree with thousands of nodes is too slow. We have several alternatives:

a) Replace the TreeView component with a faster one. We have Virtual TreeView (http://www.delphi-gems.com/VirtualTreeview/VT.php) by Mike Lischke. Previously we concluded that using this component w<as not possible due to critical licensing issues. (The problem was that the license was not clear, latter, when it was clear the package was LGPL, it required a non-open source package which we cannot use). Now we know we can use it. (The non-open source package is not really required, it can be disabled). Now we face the problem of the migration to Lazarus, we think VTV does not work in Lazarus. (We can fix it using conditional compilation).

b) Avoid reloading the full category tree. We can add the new nodes to the remote database, and then update locally the category tree, without downloading and rebuilding. However, this goes against the our design goals: to have the most updated information without assuming anything.

c) We can add a quick feeding option, to add categories quickly, plus a recursive category deletion option. It would allow specifying a list of categories to add and then adding them in a batch. The recursive deleting would allows the user to delete the specified category plus all the inner ones.

I was blaming the ttreeview performance for this problem, and I was right, it was a problem with ttreeview.

However, it was a single issue that was causing this: AbsoluteIndex (from treenode) is superslow. Very, greatly, super, mega, freakingly SLOW.

If we dismiss the use of AbsoluteIndex (which we called EVERYTIME we added a node to the tree), a 10,000 categories tree loads in seconds. (Otherwise, it takes MINUTES).

In the search for speed improvements, we also modified the PRopm_AddCatTreeNode procedure so that does not cross all categories with themselves while looking for child nodes. This was a O(n) operation, the needed iterations grown exponentially. Now, the procedure do a limited crossing: now it order the catlist by parent_id, do a binary search for the first item with the requested parent_id, then iterates until finishes with the requested parent_id. Now, it is not really a crossing. This reduced the count of iterations from (Cats * Cats) to (Cats * 2) [approx].

So, for example, if the user has 1000 categories, previously the application tested for a match between parent/child nodes 1000000 times before finishing building the tree. One million times. Now, AbsoluteIndex was used only when adding the node, not while testing. So, for 1000 nodes, if AbsoluteIndex took 0.15 seconds (one hundred fifty milliseconds) building the whole tree would take 150 seconds (2.5 minutes); which is unacceptable when adding, deleting or editing categories.

So, the suboptimal implementation of PRopm_AddCatTreeNode along with the use of AbsoluteIndex caused this slowness.