Motivation behind CogInfoCom

Cognitive infocommunications (CogInfoCom) is an interdisciplinary research field that has emerged as a synergy between infocommunications and the cognitive sciences. One of the key observations behind CogInfoCom is that through a convergence process between these fields, humans and ICT are becoming entangled at various levels, as a result of which new forms of cognitive capability are appearing. Crucially, these capabilities are neither purely natural (i.e., human), nor purely
artificial; therefore, it is suggested that they should be treated in a way that unifies both engineering and human-oriented perspectives.

Key features of CogInfoCom include:

Focus on cognitive capability: rather than merely focusing on ways in which humans, devices and ICT infrastructures interact, the field acknowledges the notion of ‘cognitive capability’ – an abstraction which allows for the introduction of temporal and contextual considerations into the analysis and design of relevant phenomena;

... from a generic perspective: rather than restricting the notion of cognitive capability to humans alone, the field adopts the point of view that such capabilities are emergent properties of any continued interaction and communication that potentially involves both natural and artificial components and systems;

... at various (particularly at large) temporal scales: rather than merely focusing on individual goal-oriented interactions at clearly specified points in time, the field adopts the point of view that the entanglement between humans and ICT is resulting in the necessity to consider their co-evolution at larger scales of time;

... in the context of emergent functionality: rather than focusing exclusively on function-oriented interactions, the field also considers cases where functionalities

developed at one time to fulfill a specific goal acquire new roles and uses – a phenomenon that is caused by constant change and growth in complexity of the elements (and relationships thereof) defining human-ICT networks.

A detailed account of the background behind CogInfoCom can be found in various papers as well as in an upcoming book (to appear later 2015).[1][2][3].

Definition of CogInfoCom

The first draft definition of CogInfoCom was given in [4]. The definition was finalized based on the paper with the joint participation of the Startup Committee at the 1st International Workshop on Cognitive Infocommunications, held in Tokyo, Japan in 2010. A finalized version of the definition is reproduced here (from [1]):

"Cognitive infocommunications (CogInfoCom) investigates the link between the research areas of infocommunications and the cognitive sciences, as well as the various engineering applications which have emerged as the synergic combination of these sciences. The primary goal of CogInfoCom is to provide a systematic view of how cognitive processes can co-evolve with infocommunications devices so that the capabilities of the human brain may not only be extended through these devices, irrespective of geographical distance, but may also interact with the capabilities of any artificially cognitive system. This merging and extension of cognitive capabilities is targeted towards engineering applications in which artificial and/or natural cognitive systems are enabled to work together more effectively."

As discussed in the following section, the implicit and explicit assumptions underlying CogInfoCom together form a unique viewpoint. As a result, new notions and concepts capable of leading to new research directions are continuously emerging. In this section, two early concepts central to multi-sensory communication between various levels of cognitive capability are introduced: the mode of communication, and the type of communication. In the future, these concepts may be extended to provide a more detailed qualification (and in the long run: quantification) of cognitive capabilities independent of the exchange of communicational
messages.

The mode of communication characterizes the relative cognitive capabilities of the actors involved in communication:

Intra-cognitive communication: information transfer information transfer occurs between two cognitive entities with equivalent cognitive capabilities (e.g., between two humans, or between two humans in the same social-technological environment – as determined by what is relevant to the application).

Inter-cognitive communication: information transfer occurs between two cognitive entities with different cognitive capabilities (e.g., between a human and an artificially cognitive system, or between two humans in a different social or technological environment – as determined by what is relevant to the application).

The type of communication refers to the type of information that is conveyed between the two communicating entities, and the way in which this is done:

Sensor-sharing communication: cognitive entities on both ends use the same sensory modality to receive information.

Sensor-bridging communication: sensory information is not only transmitted, but also transformed to a different, more appropriate sensory modality of the receiving cognitive entity.

Representation-sharing communication: the same information representation is used on both ends of communication.

Representation-bridging communication: sensory information is filtered and/or adapted so that a different information representation is used on the two ends of communication.

Remarks

A sensor-sharing application brings novelty to traditional infocommunications in the sense that it can convey any kind of normally perceptible signal (i.e., a signal that could be perceived if there were no distance to communicate across) to the other end of the communication line. The key determinant of sensor-sharing communication is that the same sensory modality is used to perceive the information on the receiving end of communication as would be if there were no distance between the sending and receiving ends.

Sensor bridging can reflect both the way in which information is conveyed (i.e., by changing sensory modality) as well as the novelty of the information type that is conveyed. Whenever the transmitted information type is imperceptible to the receiving entity due to a lack of appropriate sensory modality, communication will necessarily occur through sensor bridging.

A CogInfoCom application can be regarded as an instance of representation sharing even if it bridges between different sensory modalities. By considering the general characteristics of a representation (e.g., its character-based, icon-based etc. nature) rather than the specific details of its physical or biological manifestation, it becomes possible to describe representations of different modalities in unified ways.

Related research

Convergence of Fields behind CogInfoCom

General perspective

It is a common phenomenon for newly established fields to go through a process of maturation and ultimate convergence. The evolution of informatics, media and communications is no different: although these fields initially had different goals and applied different methodologies, their maturation and growing pervasiveness has led to the emergence of new synergies. Thus the fields of infocommunications, media informatics and media communications appeared in the latter half of the 20th century. The subsequent evolution of these disciplines, in turn, has resulted in still newer patterns of convergence. As modern network services aim to provide an increasingly holistic user experience, the infocommunications sector now encompasses "all information processing and content management functions [...] of information technology and electronic media" [2].

Parallel to these developments, with the enormous growth in scope and technological relevance of the cognitive sciences, the new fields of cognitive media [5][6][7][8], cognitive informatics [9][10] and cognitive communication(s) [11][12];[13] are gradually emerging. By today, these fields have either fully made their way, or are steadily on their way into standard university curricula and will eventually become a natural part of collective awareness.

For example, a quick search reveals that several research groups and companies around the world have ‘cognitive media’ (sometimes together with the suffix ‘technologies’) in their name. While originally the field was strongly motivated by new prospects of virtually augmented, interactive education, today it is driven by a more general focus on how various forms of media can be analyzed in terms of their effects on human cognition, and how they can be applied to sharing information in ways that appeals to human cognitive capabilities. As a result, any research focused on interactive media, or interaction technologies in general will have strong relevance to the field. However, another factor of growing importance behind cognitive media is the growing prevalence of artificial sensory capabilities implemented
in media: in a way analogous to the human nervous system, the Internet as an interconnection of globally distributed devices and nodes, together with the media applications based on it can be seen as an artificial nervous system and an artificial set of sensory modalities. For example, data available on social network platforms such as Facebook and Twitter are increasingly used to predict and understand physical, mental and social processes taking place in the world. According to this view, cognitive media targets not only the cognitive effects that media has on humans, but also the cognitive capabilities of media itself.

Similarly, the terms ‘cognitive informatics’ and ‘cognitive communication’ / ‘cognitive communications’ are omnipresent in delineations of research focus. A definition of cognitive informatics can be found on the official web page of a number of research organizations (e.g., the Pacific Northwest Laboratory funded by the U.S. Department of Energy). Today there are several research institutes dedicated to cognitive informatics, such as the Institute for Cognitive Informatics (ICON) in Sheffield, UK, which has an annual budget of over 1 million GBP. An IEEE International Conference on Cognitive Informatics has been held every year since 2002, and several symposia with leading researchers invited as keynote lecturers have been held in the past few years. Cognitive communication(s), when used without the trailing ‘s’, can refer to the study of ways in which humans anticipate context as a contributor to the choice of communication style, and perceive the consequences of communication choices (in short, it is the study of human mental models of communication). On the other hand, with the traling ‘s’, cognitive infocommunications is used to refer to devices and networks which can assign resources and functionalities in dynamic and intelligent ways (as in e.g., cognitive radio or cognitive networks). It is possible that in the future, these two directions will reach common formulations for certain problems if user needs and application scenarios are considered as contributing factors to the adaptivity of cognitive radio.

Infocommunications perspective

Based on the previous subsection, there has been constant convergence between infocommunications, media informatics and media communications in the past two decades. As a result, infocommunications today takes a broader focus than before (while some aspects, naturally, are no longer considered to be part of infocommunications).

The three fields of media, informatics and communications originally created separate theories, but are converging towards each other today. From a research historical point of view, CogInfoCom is situated in the region between cognitive informatics and cognitive communications.

An up-to-date and comprehensive outline of how infocommunications has evolved to become the science it is today can be found in [14]. The author demonstrates convincingly that the convergence leading to the infocommunications of today (i.e., without the participation of cognitive sciences) was thoroughly analyzed in the mid-1990's, and was soon recognized by both by the European Commission and by the International Telecommunication Union. The consensus is that this convergence took place at three levels, manifesting itself in the unification of technologies, the integration of markets and the harmonization of regulation. [2] Thus, today the same devices that are used to communicate with friends, family and colleagues, can also be used to access and actively process digital media content (hence, both the technology and the market is available for unification). Further, it is due to the harmonization of regulation that the cost of using modern infocommunications devices is transparent with respect to the kind of information that they are used to process and transmit.

The convergence process and its future prospects can be summarized in the following 4 steps[2]:

Unified telecommunications. A partial unification was possible from a technological point of view with the rapid development of digital technology. Hence, it became possible to handle different kinds of content with the same devices. On the other hand, the electronic media production industry had yet to become involved in the convergence process.

Infocommunications. The electronic media content producing industry, as well as the appropriate institutions for regulation joined the convergence process to produce the technological and social-economic revolution that is today's infocommunications industry.

Cognitive infocommunications. A natural fourth step is the integration of cognitive sciences into the convergence process behind infocommunications. On the one hand, this involves an expanding content space, in which novel types of information are gathered, inferred and reasoned upon in novel ways. The reason why infocommunications is essential to this development is that the content must also be presented to the human users, in a way that appeals to their unique cognitive capabilities.

As a result of the synergy between the cognitive sciences and infocommunications, the content space that is accessed and manipulated by both users and ICT is expected to grow in size and richness of interpretation. If the potential behind this expanding content space is to be harnessed, it can be expected that the respective unique capabilities of humans and ICT will be applied through new, long-term interaction patterns, leading to the emergence of new cognitive entities. A significant challenge in CogInfoCom is how to ‘power’ these cognitive entities with the kinds of information and functionality that are suitable to their roles and requirements.

New conceptual view behind CogInfoCom

As mentioned earlier, one of the key observations behind CogInfoCom is that there is a merging process between humans and ICT that is resulting in increasingly complex forms of human-ICT entanglement, and is at the same time creating the necessity for an analogous convergence between technology and the human-oriented cognitive sciences. The phenomena of merging and entanglement in the context of ICT are clear not only from everyday experience, but have also been remarked and analyzed to various degrees and in various contexts by many authors.

CogInfoCom proposes a new and unified conceptual approach in which the process of merging is derived from the theoretically unified concept of different levels of cognitive capabilities co-existing in the information space (irrespective of whether they are natural or artificial capabilities, and whether they are individual capabilities or capabilities which emerge from a cloud of artificial and/or biological components). This derivation extends to a variety of aspects, including:

Low-level direct entanglement including those that rely on invasive and non-invasive forms of interface (as in e.g. brain-computer interfaces). Entanglement at this low level allows for direct sensing and control, however, it is also relatively cumbersome in that it requires sensors to be implanted or worn and is also difficult to operate at conceptually higher levels of command.

Entity-level multisensory entanglement at the level of personal informatics devices, in which communication and interaction occur through (human – but crucially not only human) sensory modalities. The question of what kind of "communication language" to use (i.e. in terms of message encoding) depending on the semantics of the information, as well as – among others – the modality to be used, the application environment, and the user’s individual cognitive capabilities are strongly relevant to this level of entanglement. It is important to note that the challenge consists not only in providing effective and ergonomic interface design, but also in accomodating the transfer of an expanding set of semantic concepts – relevant at large temporal scales, for instance in co-existive smart home and other augmented virtual reality applications – through the limited possibilities afforded by human sensory modalities.

Collective entanglement: which occur at the collective level of multi-user interactions. Applications in this layer can be linked to collective behaviors in two ways: by making use of collective behaviors in order to support individual users’ interaction with a system; or alternatively, by supporting the prediction or a posteriori analysis of collective events based on an analysis of past behaviors (both individual and collective). Such applications often rely on the mining and analysis of vast amounts of heterogeneous data sources – including e.g. activities on social communication platforms.

From a CogInfoCom perspective, any kind of hardware or software component that actively collects / stores / understands / communicates data can be seen as a component with a set of cognitive capabilities. Whenever users become entangled with a system of such capabilities, the border between natural and artificial gradually becomes vague. In other words, it is often the case that there is often no longer any objective border between purely human and purely artificial cognitive capabilities. For example, in a scenario where a user controls an industrial robot with one hand using knowledge obtained from a smartphone held in her other hand, the question immediately arises: should this interaction be characteracterized from the perspective of communication between three different entities, or is there benefit in viewing the user and the smartphone as one entity that is communicating with the robot? The answer to this question is important, if only for the fact that both the robot and the supporting smartphone application might be designed differently if it is known in advance that they will used together in this specific scenario, or if the cognitive effects that the smartphone application will have on the user – such as limited dexterity and attention, increased capabilities for information access, etc. – are known in
advance. To consider two another examples, the boundary between artificial and human capabilities would be equally blurred in a scenario where a user’s lower arm is augmented through a robot arm that is capable of downloading new ‘skills’ from an ICT network; or in a scenario where a pair of augmented glasses, or an augmented helmet is used to provide an industrial operator with real-time information feeds complementing the task at hand (such technologies are already present in industry, and are on the verge of commercial breakthrough).

The bottom line is not that one would be philosophically inclined to specify a boundary between entities, but that it is also necessary to specify such boundaries from the functional perspective of engineering design. On the one hand, in a domain where difficult problems of synthesis can be effectively tackled only by breaking them down into smaller components, glued together through some form of communication once they are complete, the functional boundaries at which this is done can make or break the tractability and sustainability of an implementation. On the other hand, once it is accepted that the boundaries between artificial and natural are not as clean as they were a few decades ago, unprecedented possibilities emerge for the development of new functionalities – even cognitive capabilities. Such capabilities
can be seen as implemented in the dependencies between components in much the same way as lower-level functionalities are created as a result of several different components working appropriately in mutually specified contexts. This hierarchical dependence among capabilities can be seen as leading to a hierarchical organization of cognitive capabilities.

Research Background of CogInfoCom

Based on trends in EU and other government-supported research, the early forms of CogInfoCom arised through the course of domestic and international projects. The term cognitive infocommunication appeared as early as in 2005, and was mentioned in several papers by the end of the first decade of the 21st century [15][16].

Although CogInfoCom itself is a newly established research discipline, its emergence is directly related to a past of several decades. During this time, both the cognitive sciences and infocommunications have seen considerable growth. As a result, it is now commonplace to talk about cognitive machines, cognitive robotics and cognitive informatics on the one hand, and about the infocommunications revolution and the convergence between informatics, media, infocommunications and regulation on the other. These trends are leading to the gradual appearance of artificial cognitive capabilities, i.e. capabilities that are directed towards a broadened scope of sensing and processing of unstructured data. Users are becoming accustomed to accessing these artificial cognitive capabilities through their infocommunications devices in a wide variety of contexts. However, more than just allowing users to access these capabilities, infocommunications devices of the future are also expected to allow users to extend their cognitive capabilities with artificial ones through extended periods of co-evolved interactions (such interactions can be referred to as 'tangleactions' due to their long-term, entangled nature). This synergy of natural and artificial cognitive capabilities will be applicable in flexible and novel ways, both in physical contexts and in virtual worlds.

In terms of motivation and possible application areas, CogInfoCom has common interests - from certain perspectives - with a number of other fields, as described here. However, in terms of studying long-term co-evolution at a higher abstraction level of cognitive capabilities, CogInfoCom has also motivated and given rise to a number of new research areas an initiatives, including CogInfoCom channels, speechability, socio-cognitive ICT, EtoCom, CogInfoCom-aided industrial engineering and mathability - as described here. Details on these fields can be found in contributions to the various CogInfoCom fora, as well as in an upcoming book (to appear later 2015 [3]
).