Manual Sorting Emulated

Manual sorting has been suggested by many people many times. Although I am personally an enormous fan of having a manual sorting order I have always argued that in Toodledo it is probably well nigh impossible to implement due to the open structure of its other sorting options - where and how would it fit in?

But now I have this "great" idea for the Toodledo team, a way to implement something that for all practical purposes looks and behaves like manual sorting but which is in fact just a convenient way of changing three fields simultaneously in one drag-like mouse gesture.

Imagine, for example, that you have your list sorted by Priority, then Start date in descending order (as in Michael Linenberger's MYN, where Start date is used as the semi-manual sorting order adjustment parameter).

Then imagine you grab a task somewhere and drag it to a position near a task with Priority 2, Start date June 25. When you release the task Toodledo would copy those 2-3 fields (the sorting order fields) from the list position into your task all at once, which means that from then onwards your task will appear in approximately that position in the list, simply because those fields now have those values. (No hidden manual sorting order is required under Toodledo's hood.)

I am sure Toodledo would need to complement the above with some simple rules for borderline cases etc. Examples of such complementary rules might be:
When dragging to a point between two categories, e.g. Prio 2 and three, then consider the divider position, if any; if no divider then copy fields from task above; if no task above then copy from task below. For dates, length and other "continuous" values I believe interpolation would probably be the most appropriate, wherever possible, e.g. when dragging to a point between start date Jun 25 and Jun 27, put Jun 26.

It might even be possible to do all this in a more precise manner if it is possible in the UI to distinguish where on a task your are pointing the mouse. If you are pointing in the upper or lower 25% of a task's height then this could be seen as a wish to have the moving task placed in between that task and its neighbor, and therefore to interpolate or reconcile the field values as described above. But if you point in the middle 50% of a task's height then Toodledo could simply copy all three fields from that task without any adjustment at all. In both cases it would be convenient to have some form of visual confirmation on exactly where you are actually pointing.

And in the end, if you are not entirely happy with where the task eventually landed, you can still always adjust any of the field values in the "good old" manual way, just like today. For the most part, though, that will not be necessary and this feature would save you a lot of work overall, quite possibly not only for manual sorting per se, but also for quick field entry in general when adding or moving or recategorizing tasks - be able to set three fields simultaneously in one simple operation.

It's possible that your idea has the seeds of a brilliantly simple approach to manual sorting, and it's possible that Jake can use your suggestions to take Toodledo to the next level, but it seems to me that it would be more difficult to implement "emulated manual sorting" than it would be to have actual manual sorting.

Right now, the "drag and drop" function is used in two ways: 1) to turn a task into a subtask, and 2) to manually sort a list of subtasks. You are suggesting that it could also be used to do this:

Then imagine you grab a task somewhere and drag it to a position near a task with Priority 2, Start date June 25. When you release the task Toodledo would copy those 2-3 fields (the sorting order fields) from the list position into your task all at once, which means that from then onwards your task will appear in approximately that position in the list, simply because those fields now have those values.
...
When dragging to a point between two categories, e.g. Prio 2 and three, then consider the divider position, if any; if no divider then copy fields from task above; if no task above then copy from task below. For dates, length and other "continuous" values I believe interpolation would probably be the most appropriate, wherever possible, e.g. when dragging to a point between start date Jun 25 and Jun 27, put Jun 26.

So, when you grab a task, Toodledo has to:
1. know the sort order of the current view
2. know the field values of a task that is "near" the cursor
3. copy the field values (from that task) that are used in the sort
4. edit the task that is being dragged so that it has those same field values
5. resort the list of tasks to display the new order

Toodledo has 19 ways to sort a list. Five of those ways involve fields that are not editable: Date Added, Date Modified, Date Completed, Distance, Importance. With most of the other fields, there is nothing to "interpolate or reconcile", e.g. Folder, Context, Goal, Repeat, Tag, Status, Star, Location.

I think that if you examine some of the other possible sort combinations (besides Priority and Start Date), you'll see that your approach will not work, or it will be counterintuitive, or it will be very difficult to implement.

There is a relatively simple (though not necessarily easy) way of implementing manual sorting:
1. Create a hidden field that is used to track the sort order.
2. Have an option for "Manual" in the first Sort dropdown. When "Manual" is chosen, the other two Sorts are no longer available.
3. Use "drag and drop" to change the value of the hidden field based on the position of the task in the list. (Basically the way that Subtasks can currently be sorted.)

Whenever you choose "Manual" sort, the tasks are sorted by the hidden field. Whenever you choose some other Sort, the View is sorted by whatever combination of fields you've chosen.

Thanks for the feedback. Manual sort has always been a popular suggestion. Because of the way we designed Toodledo (with powerful sorting and filtering), it is difficult to add manual sorting. I have not yet seen a to-do list that provides manual sorting and automatic sorting at the same time. Any way you do it, there are compromises and awkwardness. You kind of have to pick one or the other. We will keep thinking about it and if we find an elegant solution, we will do it.

Thanks for the detailed response, and for your very interesting countersuggestion.

Yes, you are absolutely correct that some fields are not editable (Date Added, Importance etc), and I must confess I had not even thought about that :-(
I suppose, if this were implemented, that those fields would then simply have to be left unchanged, or dragging could be auto-disabled when such sorting is used. A bit ugly, but probably no real dealbreaker. Most of the views I ever used were not sorted by these non-editable fields, so it would still be a very useful feature.

As for the "discreet" (irreconcilable) fields (Folder, Priority etc.) I think there is just a case of miscommunication. What I meant (and had actually said, but perhaps unclearly) was that there would need to be some default rules for which task to copy the value from if you place your task in between, for example to then copy from the upper of the two tasks (I gave some more examples in the first post).

As for possible conflict or confusion with the existing features for dragging subtasks, again I have to confess I had not thought about that either :-(. But I do not think this can be impossible to solve. If necessary, this new kind of dragging could be in conjunction with some suitable meta key (Shift etc).

Now to the really interesting thing - your countersuggestion:

I like it very much and I always have. I have contemplated this before, but I have always ruled it out. I am now hoping you can show me where my thinking drifted astray.

Let me begin by saying: Yes, certainly doable and quite simple. That is why I like it. And it gives an exact placement of the task, too, which my suggested method generally will not give you if all three fields match a neighboring task.

But here is one of my concerns: What happens if you work alternately in a manual view and one or more auto-sorted favorite views? What I am picturing then is that if at the outset you have your manual list in perfect order, and then work in an auto-sorted view for any length of time - add tasks, change tasks etc - and then revert to the manually sorted view, then it will be in disorder, worse and worse the longer you have left it. But maybe this is not as bad as it might seem? Maybe if you use the manual view often enough it will be OK? Especially perhaps if the yet-unsorted tasks (all new or modified tasks?) are collected at the top under a divider heading called Unsorted? Maybe that would work?

My other concern is this: What happens if you want to have different manual sorting for different views? Is it really sufficient to have just one global manual sorting order? What I am picturing is cases such as Waiting For items, which you may wish to see in some kind of manual "priority" order overall in one view, but when viewing those same tasks in their respective project you want to see them in the sequence in which they will happen in the project? There are so many possible views in Toodledo and so many ways that users use these, that I have (perhaps prematurely) always shunned away from these seeming complexities. But maybe this, too, is not as serious as I had always imagined it? Maybe one global sorting order will go a long way and cover at least the vast majority of use cases?

All in all, I think either of these two solutions would be a massive step forward for Toodledo.

Thanks for your response and I agree that an elegant solution is desirable. But either of these two fundamentally different solutions would bring a lot of value to Toodledo even if neither of them is perfect.

Purveyor's suggestion is a "true" manual sorting order, just one single global sorting order but still probably quite useful.

My own original suggestion is not even a "sorting order" at all. Technically speaking it is just a 3-fields-in-one-shot quick-edit operation :-)

Purveyor's suggestion is a "true" manual sorting order, just one single global sorting order but still probably quite useful.

Hey! I didn't say anything that's "true" ... ;)

Although I mentioned having just one hidden field, I was thinking of having a separate manual sort order for each View, but I realize now that it would involve a more complex database structure. Perhaps it's better to start with a single global sort order that would be available in all Views. It might even work if the manual sort order disappears when you go to another sort or another view -- the manual sort is a "one off" view.

if at the outset you have your manual list in perfect order, and then work in an auto-sorted view for any length of time - add tasks, change tasks etc - and then revert to the manually sorted view, then it will be in disorder, worse and worse the longer you have left it. But maybe this is not as bad as it might seem? Maybe if you use the manual view often enough it will be OK? Especially perhaps if the yet-unsorted tasks (all new or modified tasks?) are collected at the top under a divider heading called Unsorted? Maybe that would work?

I agree, especially if the new unsorted tasks remained at the top.

Anyway, you have some good suggestions and some good questions. :)

Edited to add:
I've done a bit more research and it seems to me that the manual sort order should be a temporary sort, a type of "fine-tuning". There's little reason to save the order.

For example, you have a list sorted by Due Date and you have several tasks due today. You want one of these tasks to be at the top of the list. You could "Star" the task, and sort the list first by Star and then by Due Date. That would work. (Actually, I use this method now.) But what if you have two Starred tasks and you want one of them at the top? (I don't think it matters, but I can see why some people would want it this way.) You could give the task a High Priority and sort your list by Priority, Starred, Due Date. But what if you still have a tie?

A manual sort would allow you to nudge up various tasks to put them in exactly the right position. But, as soon as you go to another view or you sort the current view by various fields instead of by "Manual", you lose your manual sort and you go back to the "Automatic" sort, which, if set up properly, should give you a pretty good idea of what to do next.

Now, if we could have separate sorts for each Tab in a View, and if we could save the sort order with Searches ...

Although I do not know quite what it was, there was something in your first post that made me think more clearly than ever before of one single global sorting order. I think this is what made it seem more feasible than it has ever seemed before. I think I have probably always got lost (and scared) in the wild forest of endless manual sorting orders everywhere :-)

But one global sorting order, an extra hidden field that can be enabled by the user, is very straightforward, and it will probably go a long way already. It would even be quite easy to have more than one - Man1, Man2, Man3 if users need more than just one for different categories of lists. Still simple and transparent. And those who use them probably use them a lot, so maybe the problem of degraded "unsorted" lists is not all that serious, after all. Very good.

I still like my own idea, too, though. I like them both, in fact, and for two different reasons. Mine is actually a super-quick quick-edit operation that you can use directly on all (well, most) auto-sorted lists. This is also very useful and simple.

As for your manual exact fine-tuning of an automatically sorted list, I would love to have it, but is it really doable? I would want to leave the sorting in place, not have it disappear every time. Is the disappearing act the price for making it feasible?

But one global sorting order, an extra hidden field that can be enabled by the user, is very straightforward, and it will probably go a long way already.

Perhaps it will, but that's different from what I was talking about. When I said "hidden" I meant hidden forever from the user. If the field is going to be edited by the user, then it might as well be visible. For example, there are some users who put numbers in fields like Tag or Duration to fine-tune the sort order, or they add a prefix to the task name.

I proposed the hidden field as a way of having the Toodledo program do the sort after a user moved a task manually so that the sort order could be saved and would be available even after using a field-based sort. All the re-numbering would happen internally, and the user would see the end result of a reusable sorted list. But now I wonder whether the extra field is necessary.

With three sorts and up to 19 fields, it should be possible to get very close to the ideal sort order for each user. The manual sort is only for those users who want the tasks in precisely a certain order that is unattainable with a field-based sort (or requires more than a three-level sort) and who want to make a fine adjustment to the current list. This could happen by turning on "Manual Sort" which would first keep all tasks in their current order and then allow the user to move the tasks around. But, once you go to a field-based sort, the manual order would be lost.

As for the "disappearing act", it does make things simpler from a database viewpoint (no extra field to store and track) but I'm not sure whether it makes things simpler from a programming viewpoint.

In any case, I'm not too keen on manual sorting because I would rather not have to add more mouse clicks to the several hundred that I have to use each day in Toodledo to navigate among the Views, the tabs within the Views, the sort order and filters within each View, the tasks within each tab, and the fields within each task. Using saved Searches is a partial solution to all this bouncing around but the sort order is not stored.

Hmm. Yes, we probably misunderstand each other here and there, but that's alright. (For example, I did not mean that the user should be able to edit the "hidden" field, but never mind.)

What I believe we are beginning to see clearly is this:

1) There seem to be at least a couple of approaches that would be directly doable, as well as approaches that would seem to require a "feasibility study".

2) It is becoming obvious that there are perhaps two quite different aims or main priorities with all this "manual sorting"

a) quick movement (recategorization, reprioritization etc) of a task - by just dragging it off to some other region of a list instead of having to manually adjust a bunch of fields

b) fine-tuning of the sorting order beyond what the auto-sorting permits

I like both of these. Obviously my original "3-in-1 quick-edit" suggestion was aimed at the first of these only. It would allow, for example, people who use Linenberger's method to avoid all the clicking involved in manually adjusting the Priority and Start date. Or moving a task from Hold into Next action etc.

As for the fine-tuning of an existing list, I must say I would appreciate that, too. Just like you, I can also find it disturbing to have "equal" tasks appear auto-sorted in the "wrong" order. Even though they are "equal" in many ways, they still "belong" closer to some other tasks than to others, or maybe there is a subtle sequence involved etc. In theory maybe such fine-tuning could be accomplished by other means, too, than just dragging? Perhaps some kind of adjustment field (+1, +2, -1, -2 etc)?

Since the two purposes are so different, it opens up the possibility, at least in theory, for having two different solutions in place.

Hmm. Yes, we probably misunderstand each other here and there, but that's alright. (For example, I did not mean that the user should be able to edit the "hidden" field, but never mind.)

Uh, you're right! (Of course.)

But, I don't understand what you meant by "enabled by user" and why you would want more than one hidden field. Ultimately, there is only one order for any list, no matter how many fields you used to arrive at that order. When a user chooses a Manual sort, the hidden field would have the numerical representation of that order, i.e. first task on list is "1", second task "2", and so on.

Obviously my original "3-in-1 quick-edit" suggestion was aimed at the first of these only. It would allow, for example, people who use Linenberger's method to avoid all the clicking involved in manually adjusting the Priority and Start date. Or moving a task from Hold into Next action etc.

Seems to me that this could be solved by having another feature that has been requested: The ability to edit multiple tasks without having to use a Search, e.g. click four tasks and change the Start Date for all of them to next week. This feature is available in Appigo's Todo (although only for one field at a time), and it also has the ability to sort tasks manually, although I see that it changes the values of the first sort column, which can be one of only two fields: Due Date or Priority.

But, now that you mention Linenberger, according to my reading of his method, there are only two fields to sort -- Priority and Start Date -- and I didn't see anything about manual sorting. If you are using his method, why are you "an enormous fan of having a manual sorting order"? Seems that I missed something.

Enabled by user:
What I meant was just that for those who do not even want to know about the manual sorting, and don't want to see it as a sorting option, maybe this should be switchable in the Settings just like the other fields. But I could be wrong. Nothing very important.

More than one global field:
What I meant was that if one single global order is not enough - I cannot really picture all this completely - then it would still be easily doable to have say two or three global sorting orders available and make it possible to select which one of these you want to use for a particular type of list. Still easy and clear. If it is needed or or not I cannot really tell, but it would cover the case where you have a Next list or Hotlist list ordered manually according to one kind of principle and a Waiting For list also ordered manually but according to some other principle.

As long as the number of manual sorting orders is limited and manageable I can grasp it in my my brain and see that it is doable, but when it begins to smell of having unlimited or totally hidden sorting orders for every variety of every main view I confess I get totally lost and cannot really say anything.

As for multi-edit without a search: Yes, that is very useful too, but I see that as a different thing. Multi-edit is for editing many tasks at the same time without repetitions. My 3-in-1 quick edit suggestion is for setting three values at the same time for one single task without having to even open any field values (just drag the task). These two features do not exclude or overlap each other.

Linenberger: I am not using his method. I am using Nirvana which has native drag-and-drop, which I actively use and depend on. I sometimes reassess my choice of tools, though (Nirvana, Zendone, Toodledo ...) and I am just trying to be clear about how I would best implement simple "manualish" dragging-sorting if I were to go back to Toodledo; I cannot stand having to fiddle with field values just to move a task up or down the ladder or not getting the tasks closely enough in the order I want them. Chances are quite low that I will go back to Toodledo, but the main attraction here is the Show filter. The others have extremely minimal quick filtering options. What they do have is a more "premeditated" usage system out-of-the-box - you do not have to create your own system using saved searches or anything; just sign up and go; you do not even need saved searches at all. And they both, especially zendone, have more powerful project handling.

Anything short of perfection will not be tolerated!! ;)
Thanks for the clarification. Makes sense to me.

What they do have is a more "premeditated" usage system out-of-the-box - you do not have to create your own system using saved searches or anything; just sign up and go; you do not even need saved searches at all. And they both, especially zendone, have more powerful project handling.

I agree. Toodledo is the most customizable online task manager that I've seen but it is not a "system". It is a collection of tools that allows a huge range of methods to be used but it is not optimized for any given method. Jack of all trades, and all that ...

I still use it, mainly because of the synchronization with my iPhone and iPad. But I have spent far too much time customizing it and trying to get around the many limitations. FWIW, at our company I have developed a fairly sophisticated task management system for our employees and it does a good job (it is directly linked to our customer database) but it some ways it is not as powerful as Toodledo (I am not a full-time programmer), it doesn't sync with mobile devices, and we don't have the resources to take the functionality to the next level. For now, I will continue to use Toodledo for my personal tasks and some of my work tasks, until something comes along that I think is better.

Well, it's obviously not important enough, or interesting enough, or do-able enough for Jake. He has said that he uses those three criteria in deciding what to do next. He also kind of, sort of takes into account how many requests he has received for a particular feature but he doesn't give any special weight to the requests in this forum.

In essence, Jake wants the Toodledo development process to be a Black box:

In science and engineering, a black box is a device, system or object which can be viewed solely in terms of its input, output and transfer characteristics without any knowledge of its internal workings, that is, its implementation is "opaque" (black).

We listen very carefully to our users, but you have to understand that user requests are only one variable in our prioritization. We also consider other things, such as our resources, skills, complexity of development, competitor actions, interestingness, business impact, etc.

One thing to keep in mind is that only about 3% of our users have ever set foot in the forums, and an even smaller fraction have posted a comment. So, what you read in the forums shouldn't really be used as a general barometer for the desires of the entire user base.

This is something that is on our to-do list, but the main purpose of this idea is to save 2-6 mouse clicks. While this is certainly important, it is not what we would call a critical feature, so we have been focusing more on things that cannot be done at all, like mobile access and sharing. We will get to it eventually.

Everything Jake says there makes perfect sense. And only he knows what can be read out from the overall user statistics.

On the other hand, nothing in what he says demonstrates why these features we are discussing in this thread would be any less critical or interesting for the general user than any of the improvements that actually have been made in the meantime. So it remains a bit of a mystery.

This is the beauty of free competition. It is not a democratic exercise where watered-down compromises is all that comes out. Companies need a strong vision and stick to it. Everybody has the right to try. Some survive, some don't. Jake must follow his vision, whatever it is. It is his enterprise. We are all free to go anytime we want.

Toodledo's limitations don't demonstrate the beauty of free competition. They demonstrate the reality of limited resources and limited capabilities, resulting in watered-down compromises. But, as long as those compromises lead to enough functionality, then the end result will have enough value that some people will use it. Some people will even pay to use it.

Jake has done a good job. We'll see whether the competition can do something better. So far, I think that they haven't.

You're right. The competitors generally speaking are no better overall. Each app has countless flaws from your own perspective. Toodledo is by far the best general toolbox and also has the best quick filter I have seen so far, and this is why I am still considering Toodledo as a possible alternative.

What I meant by watered-down compromises was that if you read the forum of any app - Toodledo or any of the others - you will probably notice that you agree with only maybe 10%-20% of all the other users' feature requests. If it were all based on voting or democracy, or obeying all the customers, you most likely would end up with some incongruous hodgepodge - a very much watered-down compromise.

Toodledo is not like that at all. It is the best todo toolbox available in the market as far as I know. And most likely that was what Jake intended. And as I understand it Toodledo has been commercially successful. And it is one of my top 5 picks, too.