Archive for the ‘Uncategorized’ Category

In what follows, I’ll talk a bit more about how the Activity Journal displays extended periods of your past computer life in terms of your activities, and thereby gives you access to the entities you directed these activities at.

Subsequently, activity browsing by time is contrasted to other means for relating specifically to information you’ve used in the past. These include file browsing by folders and by recently visited folders, access via recent documents lists, as well as file search with date or date range constraints (over ‘creation’, ‘last accessed’, and ‘last modified’ timestamps, as portrayed in 1, 2, 3, 4, 5), and, finally, searching with the Synapse launcher (1, 2, 3, 4, 5) or using the also Zeitgeist-powered Ubuntu Unity file browser (1, 2, 3) and Unity’s scopes and lenses. Please note, that the discussion of most-used access notions and of browsing and search in application-specific histories will be postponed.

With this outline and comments in mind, let’s now resume the recap of the Activity Journal.

Display & Access … of your day-by-day computer work

Just as the prototypes of the new Journal (depicted here and there), their ancestor Activity Journal shows your day-by-day computer work, thus, what you worked on, when you did so and, to some extent, also together with whom.

The Activity Journal can not only be used to display and browse a chronological record of the information you’ve considered in your computer work, but also to regain this very information, that is, first of all, your files. Surpassing display and access of file activity only, other relevant activities such as commits to code hosting platforms, chat conversations in channels and in in private, as well as web activity are also exposed and can be returned to.

To this end, the time-anchored data from your Zeitgeist activity log are presented to you via three separate views, which can be switched among. These views are the default three-days-in-columns view (called ‘multi view’) and the two single-day thumbs and timeline views. For illustration, see this video (multi view: 0:00 – 0:29, thumbs view: 0:30 – 0:52, timeline view: 0:53 – 1:01). Each of the views emphasizes different aspects of your prior activity.

Multi View and Thumbs View

The multi view shows three days in a row (e.g., ‘today’ and the two previous days), and so gives you simultaneous access to structured lists of what you then did (e.g., chatted with a colleague) and touched (e.g., documents and diagrams).

While the multi view also provides file preview on mouseover, the thumbs view brings you per-day overviews, particularly useful for file contents which are easy to recognize graphically.

In order to have both thumbs view and multi view better scale with the number of files and other information touched per day, these two views group your activities into categories by type, with numbers in brackets indicating the size of these groups.

Timeline View

A visualization of your activities, which focuses on their detailed distribution over time, can be achieved with the timeline view.

Recognition of Workflow. With the timeline view, it is particularly convenient to recognize your workflow, i.e. the progression and course of your activities over a day, overall as well as in detail: Overall processes, such as from ‘planning out a day’ to ‘diving into the planned work’, and onto later ‘sharing or reporting about it’ (as was discussed in our talk at the Gran Canaria Desktop Summit in 2009, slides 15f.) can be just as well recognized as a detailed sequence, like in ‘directing your attention away from a certain document’ to ‘finding and inserting an image’, and ‘back to the document’.

Simultaneous Activities. In addition to visualizing, which activities you performed one after another, in the timeline view, it is well visible, what you’ve done in parallel. This, for instance, shows you, which documents you used in preparing a presentation and makes it immediately apparent, which code files you edited, while chatting with a colleague. Likewise, it becomes straightforward to regain the music, you listened to during a chat with your loved one.

Time Tracking. Finally, the timeline view also holds information on how long you worked with a document or with a set of information over a day. Consequently, beyond tracking your activities with files and the like, the Activity Journal also provides basic support for time tracking.
However, please note, that information about duration of your activities can be imprecise at times. This is due to principal difficulties with the tracking of thereto relevant events. Although, the focus time extension for tracking window focus (geared to produce more reliable duration information) has received some attention in the past (1, 2, 3, 4, 5), there is still some hard road ahead.

In both timeline and thumbs views, zooming can be utilized for smooth transitions between broad visual scanning and more detailed inspection.

Activity Browsing … by time

In contrast to file browsing by folders, activity browsing by time helps youremember and re-find what you’ve done during daysin the past, rather than it supports the recollection of what you’ve explicitly stored and linked together, e.g., into folders, in your keeping, organization, and maintenance.

Referring to a broad classification of human memory, a time-wise representation of activities focuses on episodic relationships such as co-occurrence, ‘did this before’ and ‘that after’ (as emerging from your courses of action directed at information and people here and there). On the other hand, hierarchical and interlinked groups, and also tags, emphasize what belongs together topically or in terms of projects (i.e. designating more static, but still developing, semantic relationships).

In other words, activity browsing by time is less based on structures as expressed in manually kept information collections, but instead enabled by temporal structures (such as ‘on that day’, ‘in the morning’, or ‘at 5 p.m.’), which can be understood as labels, your memory often implicitly assigns to you’re doings (i.e. to your activities). These labels become reflected in your activity log and are surfaced (i.e. made available) by the Activity Journal.

Recognition over Recall … of what you’ve done

Re-membering with your prior activities and the involved information is supported via cued recall , which, in many situations, beats free recall in convenience and efficiency: often, it is simply easier to recognize something than it is to remember that thing off the top of your head, or to think up, how to describe or specify that thing, for example, via a query term.

In accordance with this observation, each of the three views of the Activity Journal enables you to recognize previous document and image use, chat conversations led, and the like, as well as other information and media use, which occurred around these example activities, and vice versa.

(Recent Documents) ++

The Activity Journal’s approach to overviewing, browsing, regaining, and resuming activities can be understood as bearing some basic resemblance to employing recent document lists, in that it frees you of having to remember files, their names and their locations. However, it goes beyond the limited time range and the confined number of entry types typically covered by these lists.

Plus, the Activity Journal pictures your return and reuse (i.e. repeated activity directed at a certain piece of information). The timeline view, for instance, shows multiple return and reuse occurrences per day, in one row per file. In multi and thumbs views, on the other hand, reuse within days is not displayed, but each day anew at the time when you last accessed a file.

Finally, the Activity Journal also facilitates text searchingthrough your past activities. This is in a way similar to what Synapse and some of the other means, mentioned in the second and third paragraphs of this post afford, but differs in so far as with the Activity Journal you can see your reuse and activity distribution over time, where results are highlighted within the context of your other displayed activities, which is again known to foster the understanding of search results.

Depiction of reuse over time in the Activity Journal specifically points out that the temporal maps of your personal information space displayed by the Activity Journal put you vis-à-vis with a re-narration of your activities rather than just providing menus or lists for accessing before used documents and media. Instead, what is shown, can be thought of as pictures of how your lines of action touched upon and (r)evolved around web pages, images, and other objects, which contain the information you work with.

What’s Next?

In the following posts, first, the conceptual placement of Zeitgeist log data (and thereby also of the thereon-based Activity Journal) between lists of recently used information and detailed version histories will be discussed. Second, I’ll review in more detail issues related to re-finding and will explain re-finding strategies, which are specifically enabled by the Activity Journal. Finally, third, the means for enabling user control (beyond view manipulations as to an otherwise sometimes considered ‘not manipulable’ log), implemented in the past two releases of the Activity Journal will receive some attention. See you soon!

Re-Introduction

This post is part one of a recap of the Activity Journal in its current status. A quick impression of the 0.8.0 release of the application can be gained from three of Stefano’s screencasts (1, 2, 3).

Looking Backward … and some more

In that incarnation, the Activity Journal is about looking backward in time. It’s a viewport into your past, which provides means for navigation, search, and view filtering and, in addition, gives you some preventive control over what goes into your Zeitgeist activity log , allows you to selectively clean it, as well as lets you annotate and pin activities.

Automatic Experiential Maps of Personal Information Space

The Activity Journal focuses on mapping out over time your past activity/experience with information (as, e.g., captured in web pages and your files) and with people (as, e.g., via instant messaging). It does so by presenting automatically generated, time-based views onto your personal information space, which are updated in near real-time.

This, on the one hand, reflects the idea that your personal information space is implicitly and gradually assembled by what you’ve experienced and interacted with (in analogy to your brain, keeping memories about what you’ve seen and done).

Time as Interaction Metaphor

On the other hand, the approach of automatically drawing personal time maps echoes the observation that time, though amongst the most natural reference figures and orientation aids to humans, is a currently underleveraged resource for improving user experience with personal technology. This seems contradictory to the fact that, likewise space, time is part of an unavoidable systematic framework for organizing our experiences.

Therefore, the Activity Journal employs time as a powerful unifying frame of reference for mapping out footprints/traces of your ‘digital’ activities.

This follows the line of argument that while computer systems have contributed to information fragmentation, they can also be used to put the pieces together again: Fueled by the per-device activity log of your net-/notebook or desktop as maintained by Zeitgeist, the Activity Journal displays a rather comprehensive account of your activities (across places/folders/collections and web, and across applications/types and other domain notions), by merging all of the corresponding footprints into temporally structured pictures of your doings.

As a side note, the feasibility of cross-device, mobile, and cloud coverage has been successfully prototyped, but that’s beyond the focus of this recap.

Overview of Targeted User Tasks

The Activity Journal enables the utilization of temporal navigation strategies, such as traveling backwards in time by cursoring over the line-up of your activity footprints as well as the navigation by temporal landmarks.

The deployed temporal navigation strategies not only allow you to re-find stuff in your personal information space by employing your remembrances of when (i.e. at what day or within some other time frame) you used, saw, or listened to that stuff, but the presented activity footprints also tell you about what else you then did, at the same time, before and after.

Consequently, the other way round, you can appropriate any of the displayed activity footprints as temporal landmarks, which allow you to narrow in on a looked-for activity: particularly, you might remember having performed a target activity simultaneously, before, or after a just encountered landmark activity.

So, with the Activity Journal, you can look for both your stuff and your previous activities. The latter effectively helps you remember (i.e. reminds you) about what you’ve done and experienced in the past, and thereby leads to a better overall awareness of your activities as well as of their distribution over time. This again contributes to an increased feeling of control as to reporting tasks and offers the opportunity to review and reflect upon your overall workflow. Finally, some basic time tracking support is given.

Referring to situations in your past (first and foremost defined by their placement and extent in time), still have another application domain, namely task switching. Navigating back to an anchored time frame, which shows you (embedded in an overall, structured-by-the-day activity picture), what you did before an interruption occurred, is particularly helpful with regard to reinstatement of complex information tasks: the context and associated resources, … all you need to resume the task is just there.

Towards Interlinked Complementary Maps of Personal Information Space

Hence, the time-based perspective of the Activity Journal is beneficially complementary to hierarchical and other information and access structures in your personal information space, as exemplified by folders, links, bookmarks and by metadata such as tags.

Beyond complementary co-existence, the Activity Journal also affords some basic interactions with the aforementioned structures, in the sense of ‘upward navigation’ (for gaining the contexts and corresponding navigation/orientation means they make available), and in terms of drag and drop for the creation of information collections from activity footprints.

Between Recent Documents and Versioning Histories

With respect to level of detail and temporal coverage, the activity log data shown by the Activity Journal ranges in between the somewhat imprecise and temporally very gappy activity metadata held by standard file systems (such as ‘created’ and ‘last modified’ timestamps), on the one hand, and the usually very precise and abundant metadata for retracing file development, which is captured (to varying, but often fairly fine-grained temporal granularities) by versioning file systems.

What’s Next?

In the following posts, I’ll first talk a bit more about the views provided by the Activity Journal and about how overviewing and browsing activities by time differs from using some other, currently more widely deployed means for relating back to information you’ve used in the past. Second, the conceptual placement of Zeitgeist log data (and thereby also of the thereon-based Activity Journal) between lists of recently used information and detailed version histories will be discussed. Then, I’ll review in more detail issues related to re-finding and will explain re-finding strategies, which are specifically enabled by the Activity Journal. Finally, fourth, the means for enabling user control (beyond view manipulations as to an otherwise sometimes considered ‘not manipulable’ log), implemented in the past two releases of the Activity Journal will receive some attention. See you soon!

As announced (here) by great hacker and Open Source enthusiast Stefano Candori, I am currently mentoring him in a revamp of the Activity Journal (Stefano is its current maintainer) towards an application simply called Journal. In fact, it’s one of this year’s GSoC projects for the GNOME free software desktop project, and here’s the proposal.

Point of Departure and Motivation

The Activity Journal project was started back in October 2008 (then still under a different name) and since April 2009 has been widely covered and discussed beyond the project’s contributor base in tech news, in interviews, and on various portals and blogs across open source communities. Some examples of the rather extensive echo with a mostly positive twist are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 (with corrections in 22), 23, 24, 25, 26, 27, 28, 29, 30 (list sorted by time).

More than 1700 users have downloaded the 0.8.0 release of the Activity Journal from Launchpad (only one way of getting it) and many of these users bless us with a whole bunch of very useful feedback. By the way, some other ways of getting the current release of the Activity Journal are via Ubuntu apps dir, Debian packages, Fedora packages, and RPM).

Issues:

Overall Flexibility of Display. As the number of external application plugins and Zeitgeist dataproviders grows (and thereby also the number and variation of activities to be shown in the Activirty Journal), there is increasing need of flexibility in presentation form. For example, right now, all of the following must be fitted in the same fixed-sized boxes in the default ‘multiview’: document editing, picture use, chat activity, and music listening. And, well, ‘one size fits all’ is at least questionable here.
As another example, consider calendar items: people use them to structure and orient within their days. So why not using them for re-orienting back to what users did during and around the events they stand for? However, again, calendar items and other such anchor elements for later return don’t really find their place in the Activity Journal’s current visualizations.

Presentation of Matches. The density of result presentation as to searching and filtering in the Activity Journal’s entries is limited, because of the predefined columns in multiview.

Personal Cues based on Analysis of User Action. Two of the three views currently provided don’t really allow to reflect, how much and in which patterns a user interacted with any of the information shown.

View Switching. Then again, switching among the views can be troublesome.

Consequently, in this project, beyond adding new features and types of activity footprints, we’re introducing a new way of showing the Activity Journal to its users.

The new Journal’s main view (early prototype): A two-column line-up of variably-sized Activity Bubbles on a timeline with continuous scrolling and a ‘logarithmic’ Timebar for navigation

We think that the above presentation form will allow us to clear the just discussed deficiencies of the former Activity Journal visualizations. They are taken up in the same order as they were brought up above:

Overall Flexibility of Display. With the two-column line-up of variably-sized activity bubbles, different information formats and user activity footprints can be differently presented.

Presentation of Matches. The above illustrated structure of mapping out user activities in time has the potential to be more densely populated with search and filter results than what’s currently possible, while we’re not losing the option to also present matches ‘in context’, so together with what happened around them.
Still, it seems evident that we’ll need to also revamp something like the visual guidance as provided by the highlighting of days in the histogram slider of the current Activity Journal. This will help users to better orient to search results, which occurred further in the past.

Personal Cues based on Analysis of User Action. Intensity, quality, distribution, and patterns of information use, re-use and other user activities can be reflected in more differentiated ways.

View Switching. Finally, by heading for a combination of the advantages of the Activity Journal’s thumbs and multiple-days/columns views, the amount of views to be switched among will be reduced.
However, for now, it remains undecided, how we’ll integrate the Activity Journal’s timeline view and its feature most often positively commented by users: bringing about a basis for time tracking support.

What’s Next?

While the implementation progress of Stefano’s GSoC project will be detailed on his blog, overview of relevant related material will be given on a GNOME wiki page about the revamp project. Here on this blog, I’ll accompany and flank Stefano’s work with related recaps, reflections, and thoughts about where the Journal’s journey might be taken. See you soon!

GNOME Activity Journal is an Activity Browser, which lets you quickly find what you did – with files, emails, and on the web. As opposed to File Browsers, it mainly focuses on when you did something rather than where the items you worked with are located. GAJ is powered by the awesome Zeitgeist Framework.

The paper presents an approach to reducing user efforts when reorienting to interrupted work. This features a time-centric Journal view of a user’s information-work activities, which allows going back to ‘task states’ for activity resumption. We further compute an activity-induced correlation function to reflect information coherence, as emerging and developing on the user’s side when ‘using information items together’. We thereby investigate the expressiveness of easy to understand correlation indicators, such as temporal proximity, window switching, and clipboard use. Finally, we present a list of most-used items, which is considered to provide suitable ‘entry points’ for work resumption.

ReflAction Journal

One approach for determining connection sets, is re-finding those

information items, which have been the object/s of a user’s activities

by the time when the interruption occurred (activity-phase

specific approach).

One approach for determining the information items needed to continue an interrupted activity, is re-finding those items, which have been the object/s of a user’s activities by the time when the interruption occurred (activity-phase specific approach). The ReflAction Journal supports this approach by presenting so-called activity objects to the user.

The ReflAction-Journal visualization is based on computer-observable fractions of information work. In the figure below, the doing pane shows item-related user activities (uA), which mirror that a user’s activities leave traces on those items. Accordingly, an activity object is composed of an information item (defining its name), and the traces related to the item, where the latter are sets of {focus, edit/view, visibility} intervals, representing, if so, interrupted item-related user activities.

The re-finding means enabled by the ReflAction Journal improve today’s common user experience that computers do not well support the “leaving the task (state) as it is” strategy, as they are not hampered by closed windows.

ReflAction Correlated & MostUsed

Imagine that a few days have passed since a user had extensively used some important information sources of a presentation (or any other) document. Even though she meanwhile proceeded with editing the presentation, she still perceives those sources as related to her presentation document a. Therefore, we provide a function called uA-related, which is used to determine correlated items within a time frame (across activity phases). Its user interface (upper part of the below figure) is an early prototype and doesn’t allow convenient control over time scoping yet.

The intuition behind the uA-related function is that information items ‘used together’ implicitly gain activity-induced coherence. Let uA-related(a, b) denote the function mapping the tuple (a, b) to the correlation strength of b to a. The construction of this function is illustrated in the paper. It is called for all b’s within the provided time frame. So, uA-related(Vortrag) for ‘today’ yields the list of items presented in the figure above.

The most-used list shown in the lower part of the figure above provides the main working documents to a user, which are considered suitable ‘entry points’ to resuming one’s work. Each entry of the most-used list is computed by evaluating the aggregated lengths of activity intervals per item within a time frame, where, in general, editing has more weight than viewing.

The Zeitgeist and Elementary teams are currently hacking on Sezen. It is the first user interface built on top of Zeitgeist, which has the new Zeitgeist ‘full-text search’ extension, provided by Mikkel Kamstrup and Canonical. Big thanks for providing this awesome extension!! … GAJ will get it too … So stay tuned!

On behalf of the Zeitgeist team, I am pleased to announce that we will present the latest cooking from the Zeitgeist kitchen at GUADEC 2010.

FYI, here is what we presented at GCDS/GUADEC 2009 (slides) and at a CHI 2010 workshop (slides, paper).

Topics of this year’s presentation will include the current developments of Zeitgeist and of the GNOME Activity Journal, how to work with and extend them, integration with the GNOME desktop, and cross-desktop development issues. If you want to know more, check out http://zeitgeist-project.com/.

Seif: “Hey computer, I want to see what I was doing when Ryan Paul wrote his tweet about how to turn on Gwibber’s multicolumn view!”

Computer: “Here you are!”

Seif: “Thanks!”

Implicit Relating via the WHILE-Operator in a User’s Query

Today, a user who wants to get back to what s/he was doing while perceiving or doing something else is given a hard time. Why? Well, because this user is not supplied with a direct link to the information X, which was used while Y happened, upon provision of that Y.

Here is how Zeitgeist solves the riddle: In the above video, Y is a tweet, which can be interpreted as an event because it has a timestamp. Zeitgeist takes the tweet, looks up the contemporaries, and provides them. Thereby, Zeitgeist enables the user to re-find X within a click from Y.

In the shown prototype implementation, Zeitgeist searches for the user’s activities within a five minute radius around the tweet’s timestamp and delivers those files, which have been involved in the activities. So Zeitgeist’s answer can be read like this: “While Ryan sent out his tweet, you were working on actions.py and listening to music.”

Stay tuned!

Seif: “Hey computer, I want to see what I was doing when Ryan Paul wrote his tweet about how to turn on Gwibber’s multicolumn view!”

Computer: “Well, tell me what files you were using then or where they are located!”

Seif: “Mmh, I was programming and listening to music.”

Computer: “That can be a lot of files, man. Tell me more!”

Seif: “OK, I just remembered that python file I was working on and I now also remember one artist and one song title of the music I was listening to at that time.”

Computer: “Great! Now tell this information to me, one at a time, and I’ll provide you the files.”

Seif: “Urgs!”

Ryan Paul and Seif Lotfy having Fun with Gwibber and Zeitgeist

The below video demonstrates that Seif’s computer can do better, if it has Zeitgeist running.

Today, a user who wants to get back to what s/he was doing while perceiving or doing something else is given a hard time. Why? Well, because this user is not supplied with a direct link to the information X, which was used while Y happened, upon provision of that Y.

Here is how Zeitgeist solves the riddle: In the above video, Y is a tweet, which can be interpreted as an event because it has a timestamp. Zeitgeist takes the tweet, looks up the contemporaries, and provides them. Thereby, Zeitgeist enables the user to re-find X within a click from Y.

In the shown prototype implementation, Zeitgeist searches for the user’s activities within a five minute radius around the tweet’s timestamp and delivers those files, which have been involved in the activities. So Zeitgeist’s answer can be read like this: “While Ryan sent out his tweet, you were working on actions.py and listening to these two songs.”

Albrecht Schmidt presented our work [4] at the Personal Informatics Workshop at CHI 2010 and blogged about it. It was him, who coined the distinction of implicit and explicit human-computer interaction (HCI) [5], which is now commonly used in HCI: “In contrast to explicit interaction, implicit interactions are based not on explicit actions by the user (directed at a computer), but more commonly on users’ existing patterns of behavior.” [6, p.191], referring to [5]. Differentiating between implicit and explicit HCI proves productive as an analogy for investigating different forms of relating a user’s information items, as enabled by experience-trace infrastructures, such as Zeitgeist and ContextDrive.

Implicit and Explicit HCI

“Implicit human-computer interaction is an action, performed by the user that is not primarily aimed to interact with a computerized system but which such a system understands as input.” [5]. This is opposed to explicit HCI, where “the user tells the computer at a certain level of abstraction (e.g., by command-line, direct manipulation using a GUI, gesture, or speech input) what she expects the computer to do.” [5]. An example of explicit HCI is using a direct-manipulation graphical user interface (GUI) to drag a file from one folder to another, in order to have the file become located in the target folder. Implicit HCI, on the other hand, requires computers to observe and interpret what the user is doing, via sensors, which may be implemented as physical sensors (e.g., GPS) or digital sensors (e.g., file usage monitor). Thereupon, a computer can offer information such as “you presented conceptReport-v1.1, while being at this customer’s site last time on april 12”. Consequently, monitoring user behavior is a basic prerequisite for implementing implicit HCI.

Implicit Creation of Working Sets

I’ll introduce implicit relating as a specialization of implicit HCI, which means it inherits the goals of implicit HCI. The purpose of implicit HCI is to reduce mental load on the user’s side by reducing attention-intensive efforts directed at explicitly handling a computer’s UI, in order to get something done. Similarly, implicit relating of a user’s information items aims at reducing explicit user actions towards grouping her information items, for example, in order to represent “the stuff she is currently working on” as working sets. Working sets, as an example of information collections of implicitly related items may be related to, but are still different from to be archived information collections [1]. This can be illustrated by comparing (a) paper piles on your desk with (b) folders in your rack: Grouping criteria and purpose of (a) and (b) are simply different.

Working Sets

In my last post (final section, 4.), I referred to re-finding working sets as one of the new user experiences enabled by experience-trace infrastructures. The notion of a working set is borrowed from Federico’s GNOME 2008 UX hackfest post and his October 2009 post on this topic. The motivation behind this notion is that working on rather complex topics usually involves multiple information items, such as code, emails, design documents, videos, and web resources. Plus, completing complex tasks is seldom achieved within a single uninterrupted time period of activity dedicated to that task. Hence, the need to again and again resume work, which often implies re-using at least some of the involved information items. Beyond, working sets commonly evolve over time, be it shrinking or expanding. Nowadays, it is left to the user to explicitly create and maintain folders to group working-set items, for the purpose of efficiently re-finding them, when resuming an activity.

Scenario

Consider the following common situation: When continuing work with others, one often needs to refer back to items used earlier in the common activity thread. This could be the UI mockup discussed two days ago, along with the IRC discussion related to it, and the existing code you looked at, which would need to be changed, in order to implement the discussed mockup.

As already said above, today, in order to conveniently go back to these items for task resumption, you would have needed to group them together before. So what? Well, first, this grouping is all too often “forgotten”, for various reasons. One of them is that it is perceived secondary to one’s actual task work, i.e. on top of “getting things done”. Let alone the fact that today’s means turn linking together an image, a chat discussion, and code pointers into a rather clumsy process. Additionally, anticipating what one might need in the future is hard in principle.

If not having grouped the needed working-set items beforehand, one is forced to hunt each of them down again, by means of explicit HCI, i.e. to re-find them by browsing and searching. Even if re-finding each relevant item might not be a big problem on its own, re-finding all of them consumes time and potentially interrupts your work flow. Plus, recalling all of the relevant items, if you achieve to do so, implies considerable effort, i.e. memory load.

Presentation of Working Sets

Instead, in the above scenario, your computer could unobtrusively present the relevant working-set items as soon as you are chatting with your collaborator again. Zeitgeist enables this by interpreting the collaborator match, and possibly more matches, between the before and now situations, as an indicator that you might want to continue working on one of the topics you had recently treated together. The presentation of an implicitly created working set, i.e. a set of implicitly related items, necessitates monitoring user behavior, just as is required to implement implicit HCI. In the short scenario above, this is monitoring of using images, of chat conversations, and of programming.

All of this serves the general goal of having information bits and pieces “at hand” when needed. That is the vision behind the early-prototype video by Seif Lotfy below, which implements the saying “stuff you use together belongs together”, which we coined at GUADEC 2009. The corresponding calculations in the Zeitgeist engine are for now performed ad-hoc in nearly real-time. One of the remaining challenges is to suitably draw borders among different working sets, accounting for their dynamic evolvement over time, and distinguishing working sets from what the user does “on the side”.

The more we achieve to solve these issues, the better the employment of such implicitly created working sets will work towards faster, less effortful, and so more convenient re-finding and access of previously used information items relevant to the user’s respective activity. Referring to paper-based work at your physical desk again, the above described implicit information organization can be understood as automatically putting paper documents into piles, informed by how each of them is used, by their different arrangements on your desk over time, and by the, possibly multiple, occurrences of the documents being involved in your workflow.

Design Issues

Automatically presenting a working set upon recognizing similarities between earlier situations and the current one, which, by the way, is a case of implicit output, releases the user from the need to explicitly browse or search for the items without requiring him to explicitly create and maintain working sets. Thereby, in the ideal case, the user doesn’t need to recall the relevant information items, but instead just needs to recognize them and choose accordingly from the presented working set. This needs to be complemented by means of querying for working sets, for example by defining a time scope.

Even though Zeitgeist provides support for the above discussed means for implicit creation of working sets, these dynamic collections might, in some cases, still become too large to be dealt with easily by the user. Therefore, our visualization of a working set will reflect a gradual and dynamic relevance notion, i.e. show how much a certain item belongs to a working set and/or to the currently recognized situation. In addition, we are experimenting with the concept of a life time of items with respect to inclusion into a working set, in combination with the relevance concept. This could be visually mirrored as having items fade out over time, if not being used, and fade in again, if being re-used. If users want more control, pinning items onto the working-set canvas will prevent fading behavior.

Beyond the implicit means of a gradual relevance notion and an item life-time, there are also explicit means of letting a user manipulate a working set and its visualization, in order to keep them treatable. These are blacklisting, filtering, and maybe defining size limits. Furthermore, we will investigate letting users partition a working set by arranging its items on a canvas or by employing tags or folders. The two latter naming means would also allow several working sets to be presented at once, reflecting, for example, the case of the same group of people working on different tasks “in parallel”.

Users often experience a loss of control when confronted with dynamic structures, such as dynamic menus. This needs to be avoided with respect to the rather dynamic concept of working sets. Users should not fear “losing” working-set elements or not being able to later re-find and access items they had earlier seen in working sets. This goal could be achieved by automatically turning working sets into persistent collections, for example, daily. These periodic backups would be “lossless” with respect to once included items. Working sets can even be re-computed at any time, as long as no events are deleted from the Zeitgeist log. By default, deletion of essential events therefore only happens upon user request. Consequently, users will not be required to explicitly keep working sets.

Additionally, we will try designing the computation and visualization of working sets such that users will not be forced to organize or re-organize these sets at specific occasions or times. Instead, users should feel and be in control over when to further classify information from working sets. This is because of the known negative effects of imposing interrupts on a user’s work flow, which are perceived as unnecessary overhead on the user’s part. Another rationale for the above design goal is the often reproduced finding of [2], that untitled “piles” are helpful means to represent working-set like information.

As opposed to the world of our physical desktops, the computer world enables us to have the same information item in multiple “piles”, so working sets, at the same time. Beyond, Zeitgeist can also keep track of the evolvement of working sets and the information they “contain”, in order to enable a user to refer back to different versions of them over time. A related challenge of making working sets usable is to design their behavior and user interface in a way, which lets the user distinguish among copy and reference semantics for the “contained” information items.

Albrecht Schmidt presented our work [4] at the Personal Informatics Workshop at CHI 2010 and blogged about it. It was him, who coined the distinction of implicit and explicit human-computer interaction (HCI) [5], which is now commonly used in HCI: “In contrast to explicit interaction, implicit interactions are based not on explicit actions by the user (directed at a computer), but more commonly on users’ existing patterns of behavior.” [6, p.191], referring to [5]. Differentiating between implicit and explicit HCI proves productive as an analogy for investigating different forms of relating a user’s information items, as enabled by experience-trace infrastructures, such as Zeitgeist and ContextDrive.

Implicit and Explicit HCI

“Implicit human-computer interaction is an action, performed by the user that is not primarily aimed to interact with a computerized system but which such a system understands as input.” [5]. This is opposed to explicit HCI, where “the user tells the computer at a certain level of abstraction (e.g., by command-line, direct manipulation using a GUI, gesture, or speech input) what she expects the computer to do.” [5]. An example of explicit HCI is using a direct-manipulation graphical user interface (GUI) to drag a file from one folder to another, in order to have the file become located in the target folder. Implicit HCI, on the other hand, requires computers to observe and interpret what the user is doing, via sensors, which may be implemented as physical sensors (e.g., GPS) or digital sensors (e.g., file usage monitor). Thereupon, a computer can offer information such as “you presented conceptReport-v1.1, while being at this customer’s site last time on april 12”. Consequently, monitoring user behavior is a basic prerequisite for implementing implicit HCI.

Implicit Creation of Working Sets

I’ll introduce implicit relating as a specialization of implicit HCI, which means it inherits the goals of implicit HCI. The purpose of implicit HCI is to reduce mental load on the user’s side by reducing attention-intensive efforts directed at explicitly handling a computer’s UI, in order to get something done. Similarly, implicit relating of a user’s information items aims at reducing explicit user actions towards grouping her information items, for example, in order to represent “the stuff she is currently working on” as working sets. Working sets, as an example of information collections of implicitly related items may be related to, but are still different from to be archived information collections [1]. This can be illustrated by comparing (a) paper piles on your desk with (b) folders in your rack: Grouping criteria and purpose of (a) and (b) are simply different.

Working Sets

In my previous post (final section, 4.), I referred to re-finding working sets as one of the new user experiences enabled by experience-trace infrastructures. The notion of a working set is borrowed from Federico’s GNOME 2008 UX hackfest post and his October 2009 post on this topic. The motivation behind this notion is that working on rather complex topics usually involves multiple information items, such as code, emails, design documents, videos, and web resources. Plus, completing complex tasks is seldom achieved within a single uninterrupted time period of activity dedicated to that task. Hence, the need to again and again resume work, which often implies re-using at least some of the involved information items. Beyond, working sets commonly evolve over time, be it shrinking or expanding. Nowadays, it is left to the user to explicitly create and maintain folders to group working-set items, for the purpose of efficiently re-finding them, when resuming an activity.

Scenario

Consider the following common situation: When continuing work with others, one often needs to refer back to items used earlier in the common activity thread. This could be the UI mockup discussed two days ago, along with the IRC discussion related to it, and the existing code you looked at, which would need to be changed, in order to implement the discussed mockup.

As already said above, today, in order to conveniently go back to these items for task resumption, you would have needed to group them together before. So what? Well, first, this grouping is all too often “forgotten”, for various reasons. One of them is that it is perceived secondary to one’s actual task work, i.e. on top of “getting things done”. Let alone the fact that today’s means turn linking together an image, a chat discussion, and code pointers into a rather clumsy process. Additionally, anticipating what one might need in the future is hard in principle.

If not having grouped the needed working-set items beforehand, one is forced to hunt each of them down again, by means of explicit HCI, i.e. to re-find them by browsing and searching. Even if re-finding each relevant item might not be a big problem on its own, re-finding all of them consumes time and potentially interrupts your work flow. Plus, recalling all of the relevant items, if you achieve to do so, implies considerable effort, i.e. memory load.

Presentation of Working Sets

Instead, in the above scenario, your computer could unobtrusively present the relevant working-set items as soon as you are chatting with your collaborator again. Zeitgeist enables this by interpreting the collaborator match, and possibly more matches, between the before and now situations, as an indicator that you might want to continue working on one of the topics you had recently treated together. The presentation of an implicitly created working set, i.e. a set of implicitly related items, necessitates monitoring user behavior, just as is required to implement implicit HCI. In the short scenario above, this is monitoring of using images, of chat conversations, and of programming.

All of this serves the general goal of having information bits and pieces “at hand” when needed. That is the vision behind the early-prototype video by Seif Lotfy below, which implements the saying “stuff you use together belongs together”, which we coined at GUADEC 2009. The corresponding calculations in the Zeitgeist engine are for now performed ad-hoc in nearly real-time. One of the remaining challenges is to suitably draw borders among different working sets, accounting for their dynamic evolvement over time, and distinguishing working sets from what the user does “on the side”.

The more we achieve to solve these issues, the better the employment of such implicitly created working sets will work towards faster, less effortful, and so more convenient re-finding and access of previously used information items relevant to the user’s respective activity. Referring to paper-based work at your physical desk again, the above described implicit information organization can be understood as automatically putting paper documents into piles, informed by how each of them is used, by their different arrangements on your desk over time, and by the, possibly multiple, occurrences of the documents being involved in your workflow.

Design Issues

Automatically presenting a working set upon recognizing similarities between earlier situations and the current one, which, by the way, is a case of implicit output, releases the user from the need to explicitly browse or search for the items without requiring him to explicitly create and maintain working sets. Thereby, in the ideal case, the user doesn’t need to recall the relevant information items, but instead just needs to recognize them and choose accordingly from the presented working set. This needs to be complemented by means of querying for working sets, for example by defining a time scope.

Even though Zeitgeist provides support for the above discussed means for implicit creation of working sets, these dynamic collections might, in some cases, still become too large to be dealt with easily by the user. Therefore, our visualization of a working set will reflect a gradual and dynamic relevance notion, i.e. show how much a certain item belongs to a working set and/or to the currently recognized situation. In addition, we are experimenting with the concept of a life time of items with respect to inclusion into a working set, in combination with the relevance concept. This could be visually mirrored as having items fade out over time, if not being used, and fade in again, if being re-used. If users want more control, pinning items onto the working-set canvas will prevent fading behavior.

Beyond the implicit means of a gradual relevance notion and an item life-time, there are also explicit means of letting a user manipulate a working set and its visualization, in order to keep them treatable. These are blacklisting, filtering, and maybe defining size limits. Furthermore, we will investigate letting users partition a working set by arranging its items on a canvas or by employing tags or folders. The two latter naming means would also allow several working sets to be presented at once, reflecting, for example, the case of the same group of people working on different tasks “in parallel”.

Users often experience a loss of control when confronted with dynamic structures, such as dynamic menus. This needs to be avoided with respect to the rather dynamic concept of working sets. Users should not fear “losing” working-set elements or not being able to later re-find and access items they had earlier seen in working sets. This goal could be achieved by automatically turning working sets into persistent collections, for example, daily. These periodic backups would be “lossless” with respect to once included items. Working sets can even be re-computed at any time, as long as no events are deleted from the Zeitgeist log. By default, deletion of essential events therefore only happens upon user request. Consequently, users will not be required to explicitly keep working sets.

Additionally, we will try designing the computation and visualization of working sets such that users will not be forced to organize or re-organize these sets at specific occasions or times. Instead, users should feel and be in control over when to further classify information from working sets. This is because of the known negative effects of imposing interrupts on a user’s work flow, which are perceived as unnecessary overhead on the user’s part. Another rationale for the above design goal is the often reproduced finding of [2], that untitled “piles” are helpful means to represent working-set like information.

As opposed to the world of our physical desktops, the computer world enables us to have the same information item in multiple “piles”, so working sets, at the same time. Beyond, Zeitgeist can also keep track of the evolvement of working sets and the information they “contain”, in order to enable a user to refer back to different versions of them over time. A related challenge of making working sets usable is to design their behavior and user interface in a way, which lets the user distinguish among copy and reference semantics for the “contained” information items.

Here is the abstract: In this paper, we present steps towards adequate support for answering the central questions of orienting oneself within one’s past, current, and anticipated future activities and experiences, along with application areas, which will benefit from such support. The experience-trace concept is introduced and it is shown how support for mental time-travelling within and via elements of one’s activities and experiences can be implemented as re-finding elements within one’s experience trace. To this end, two interactive journal user interfaces are proposed: ReflAction Journal and (GNOME) Activity Journal. They are enabled by the ContextDrive and Zeitgeist frameworks and their respective experience-trace implementations.

Personal Semantic Technology

Trends in technology development indicate that personal computers could develop into always-on companions, i.e. possibly wearable, personal helpmates. Anticipating this sketch of the future, we aim at designing what we call personal semantic technology. This technology will be capable of reflecting personal meaning of computer-represented objects, as emerging and developing from them being involved in a user’s activities and experiences. We investigate and test our approach with today’s mobile computers, which can be seen as forerunners of the above envisioned machinery. Digital technology is accompanying early adopters in an ever increasing amount of situations. On the one hand, this personal technology is tool and medium for a multitude of its users’ activities and experiences. On the other hand, in the future, it might serve as a personal witness of its users’ activities and experiences, possessing precise and reliable memory.

Personal Experience Trace

Personal experience traces are one of the core concepts enabling personal semantic technology: A user’s experience trace represents a consolidation of “computer-experienced” events related to this user’s activities and experiences. The experience-trace notion reflects that a user’s behavior leaves traces within his/her personal computing machinery, which can be exploited towards this user’s benefit. This is facilitated via logging events and marking them with a variety of labels, some computed and others user-indicated.
The purpose of a personal experience trace is to mirror what a person actually does and experiences by capturing a continuous, comprehensive, and coherent picture of it. This picture encompasses traces referring to objects of retrospective, current, and prospective character. It is continuously evolving as the user acts and experiences. This said, we see and accept the principal gap between what humans do/experience and what computers can sense and represent of it. We respectfully understand this gap as a design challenge and consider it mandatory that a user is and feels in absolute control of this representation.

Supporting Mental Time-Travel

Mental time-travelling is what people do when they use their “mental eye” to situate themselves in what happened in the past or in what they imagine happening in their personal future. Personal calendars are one example of a tool facilitating backward and forward mental time-travel. Additionally, they serve to schedule future “events”. Please note, that the “events” represented by calendar items do not denote the same thing as the events capturing a user’s doing and experiencing into the personal experience trace.
The experience-trace implementations ContextDrive and Zeitgeist can not only capture when a calendar item has been created or modified, but, more importantly, what time span it refers to. Thereby, calendar items are turned into labels and container items for what actually happens during the time span in question. Plus, calendar items can be appropriated, for example, to serve users in relating their preparatory and follow-up activities to the represented “events”. The ReflAction Journal facilitates this via drag’n’drop. Further, the analysis engines of both ContextDrive and Zeitgeist are currently expanded, in order to automatically achieve this relating of events to labels and other higher-level entities. This is brought forward by investigating experience-trace induced notions of relatedness, according to the above introduced personal-semantic-technology approach.
To conclude, experience-trace implementations add an application-independent time-bound layer to the experience of using computers. Users are thereby enabled to relate to and situate themselves within representations of their past, current, and future activities and experiences.

New User Experience: Re-Finding Documents, Re-Situating, and Beyond

Employing the above introduced concepts and means opens up new ways of going back to what one did and experienced.

In order to re-find a document D, users can employ their memories of situations in which they created, viewed, modified, or otherwise used this document.

Vice versa, users can re-find all situations in which they interacted with a certain document.

Users can re-find other items interacted with while, before, or after working with D.

Employing “user-language labels”, such as calendar items or task representations, users can re-find “collections of use”. Such a collection of use could, for example, be the working set of documents last used for working on a task, before having switched to another activity.

Experience-trace implementations support users in bringing to their minds what they did during the last month, which helps them with reporting tasks.

In referring to the ReflAction Journal as a visualization of their personal experience trace, users see their item-related interaction intervals composed from events. They also have available labels denoting, for example, their scheduled “events” and tasks. Furthermore, properties of these labels and of the item-related events can be employed to search for items and events as well as to filter and scope the result set displayed, according to the users’ needs.