Articles

Evaluating the Network: A Workflow for Tracking Twitter Interactions Using Social Networking Analysis

Author:

Sarah Goodier

University of Cape Town, Cape Town, ZA

Abstract

Networking plays an important role in research projects to build a community and audience around a research area. Using social media is popular in project communication as it provides the ability to engage with a group of followers daily. Such online networking tools provide the advantage of providing near-realtime data, which can be used to evaluate communication and networking success. This does, however, present a problem of what data can be collected in a reasonable timeframe to usefully feed back into the project’s strategies. This paper presents the evaluation approach the Research on Open Educational Resources for Development in the Global South (ROER4D) project has applied to understanding the project’s reach and engagement on the social networking site Twitter. This evaluation considered: 1) To what extent has ROER4D built a network of OER scholars? and 2) Which of ROER4D’s actions have been most effective at growing the network? Using a Utilization Focused Evaluation framework, this evaluation established a workflow through which the project’s Twitter network growth and snapshots of key online interactions could be tracked over time. Systematic measuring systems were set up to collect, select and analyse the most useful data to inform the project from three sources: Twitter analytics, Twitter Archiving Google Spreadsheet, and NodeXL. The social networking analysis shows the extent to which the project is building a network of scholars and what may have contributed to this growth. The feasibility of this networking evaluation approach for use by other projects, especially in the Global South, as well as the considerations when analysing Twitter data and metrics from accounts that would generally appeal to, in this case, the open education and OER communities, are highlighted.

Introduction

A network is a social structure composed of people and/or entities linked by a given type of relationship (Haines et al., 2011). Building up a network is important in any large-scale research project as it helps to build a community and audience around the research area as well as encouraging connections and partnerships for collaboration. Social networking sites are defined by boyd and Ellison (2007: 211) as:

Web-based services that allow individuals to (1) construct a public or semi-public profile within a bounded system, (2) articulate a list of other users with whom they share a connection, and (3) view and traverse their list of connections and those made by others within the system.

Social networking sites can be relatively low-cost mechanisms for building up a group of followers interested in a project’s communications, and thus establishing new online links both locally and globally (Haythornthwaite, 2005). These sites, therefore, create an environment in which people and organisations can discover those they share common interests with and connect with them, across geographical boundaries. For projects that focus on a topic that already has active groups of social networking site users, for example education scholars on Twitter (Kimmons & Veletsianos, 2016; Li and Greenhow, 2015; Veletsianos & Kimmons, 2016), communications strategies that include these key platforms can leverage existing interested networks. Learning whether and to what extent projects using certain social networking sites are succeeding in reaching their audience and engaging online assists in project learning. In this case study, the platform being considered is Twitter.

The social networking site Twitter, in brief

Twitter is an online social networking site launched in 2006 that enabled Tweeters (the users) to send public 140-character messages (tweets) about anything that might be of current interest to them (Rogers, 2013; Twitter, 2016a). These tweets can be accessed in the Twitter feed of all the Tweeters who follow the user sending the message, and also found by other users through searching the Twitter platform or the Web. Hashtags (#) are often included in tweets and all the recent tweets including a specific hashtag can be viewed by those interested in the term. Apart from regular Tweets, there are several ways of communicating on Twitter including: 1) Retweets: sharing another user’s tweet with your followers; 2) @Replies: using the specific Twitter user’s username (e.g. @Twitter) at the start of your reply to a tweet, which can grow into a conversation; 3) Mentions: referencing a specific Twitter user’s username at any point in your tweet other than the start. These communications provide data about Tweeters and their relationships to one another. Many already available resources provide detailed explanations of the Twitter platform for those who are interested in learning more, some of which are mentioned in the following paragraphs.

In social networks the people and/or entities making up the network are called nodes and the relationship(s) between them are called ties (Haythornthwaite, 2005; Hogan, 2008; Marin and Wellman, 2011; Wasserman and Faust, 1994). On Twitter, the nodes are the Tweeters and the ties between them can represent various connections such as who is following whom or who is interacting with whom. Relationships can also include who is mentioning a specific word or phrase in conversation with others, such as a hashtag. Many studies have been published in the last five years analysing twitter data set patterns and communities that have emerged on Twitter, especially around a given hashtag (#) (e.g. Bruns & Stieglitz, 2012; Ford, Veletsianos and Resta, 2014). Identifying ties between individuals and entities is key to exploring network growth. While all of these approaches touch in some way on measuring success of an approach and what we can learn from it, none of these studies have explicitly linked Twitter metrics to a structured evaluation of a project.

Gathering useful social networking analysis data for evaluation

Visualising, exploring and understanding structural data about relationships between members within networks is known as social network analysis (SNA) (Durland and Fredericks, 2005; Freeman, 2011; Hoppe and Reinelt, 2010). Online networking sites, such as Twitter, are inherently more amenable to SNA given the potential for extracting and analysing large amounts of data in near-real time. Ford, Veletsianos and Resta (2014), for example, analysed the tweets containing #PhDChat sent out over one month, to gain insight into the network of tweeters using this hashtag. In approaching such a large pool of available data from a platform such as Twitter, the problem can be determining what to collect, select and analyse. This highlights the importance of an evaluative lens in helping to determine what data will be the most informative to focus on. Articulating a set of relevant evaluation questions can reveal what will be the most useful to inform a project of what is working and what can be improved in a workable timeframe. To this end, exploring the methods and tools available to conduct SNA of the “digital records of social relationships” (Smith et al., 2009: 1) and constructing a workflow is key in order to best address these questions.

The use of SNA in evaluation work has taken off since the early to mid 1990’s (Durland and Fredericks, 2005). SNA has been used in various network evaluations, including looking at leadership (Hoppe and Reinelt, 2010), investigating public health networks (Luke & Harris, 2007) and assessing interdisciplinary research collaborations (Haines et al., 2011). Using SNA has been noted as being particularly appropriate in evaluations of collaborations, as well as in those evaluations using a participatory framework (Durland and Fredericks, 2005). This is possibly due to this method’s flexibility in analysing available network data from a broad range of contexts, including self-reported networks and network data extracted from online platforms. SNA aids in examining and visually representing the network, showing which relationships exist (Durland and Fredericks, 2005). This can be useful in evaluation of a research project’s network. The network can be explored in this way in its entirety as well as investigating its composition at key time points.

New links, especially when confined to one social networking site or communication channel, can be called weak ties (Granovetter, 1973; Haythornthwaite, 2005). As a weak tie generally connects two clusters of nodes together, any information being disseminated through such a connection can potentially reach a larger number of nodes (Granovetter, 1973; Hogan, 2008). Thus, it could be important to identify weak ties within a network and actively incorporate them into the project’s communications strategy in order to facilitate strengthening of the ties and promote sharing of information with their networks (Haythornthwaite, 2005). As it is the relationships and interactions between the nodes that matter in a network evaluation (Haythornthwaite, 2005), using SNA can assist in identifying and targeting those weak ties at the periphery of the network that can to be strengthened (Hoppe and Reinelt, 2010). While there is much that can be learned using SNA, it is important to refer back to the evaluation questions to focus the data collection efforts to optimise time and resources.

The focus of the data gathering in this paper is not on Twitter users’ personal information but instead on their interactions (follows, retweets, @Replies, etc.) with the ROER4D twitter account and any use of this term. These interactions with an educational research project Twitter account are not sensitive in nature. None of the content of individual tweets was analysed as a part of this evaluation. The only use of account information was to classify the Tweeters into core groups such as open education community members. Twitter is a predominantly public platform – you can select to make your tweets private, but the majority of users do not. Tweets can be seen and searched by other Tweeters as well as those not on the platform, as can your list of followers and who you are following. This limits the expectation of privacy of the platform users (Townsend and Wallace, 2016) which is different to other platforms such as Facebook where many accounts are private, available only to selected networks of friends (Zimmer, 2010). All Twitter data collected for this evaluation was from tweets posted from public accounts and available to anyone through the Twitter application programming interface (API). As such it does not meet the criteria to be considered research on human subjects (Moreno et al., 2013), and obtaining informed consent is not considered necessary. Data presented here has been de-identified and is presented largely in an aggregated form, focusing on Tweeters and tweet relationships to ROER4D, limiting the risk of harm.

Evaluating the ROER4D network’s social network

This paper presents the evaluation approach the Research on Open Educational Resources for Development in the Global South (ROER4D) project has applied to understanding the project’s reach and engagement on Twitter. This project consists of 18 sub-projects, conducting research on multiple topics associated with adoption and impact of open educational resources (OER) across 26 countries (ROER4D, 2013). As such, social networking sites have played a large role in reaching out to and engaging with the many potential audiences for the project’s research. Public social networking sites in particular are a focus of the project’s communication strategy. This is related to open research, data sharing and open access publishing being the pillars of its dissemination plan. One of the primary social networking sites used by ROER4D from early on in the project has been Twitter. The main audiences for the outputs of this project are sub-projects and other projects with similar areas of research, educators working in K-12 and higher education, policy makers and the open education and OER communities on Twitter (ROER4D, 2013). Given the current resource-constrained environment in both South African higher education (Department of Higher Education and Training, 2015) as well as for higher education globally (Jongbloed and Vossensteyn, 2016), using social networking sites strategically is becoming vital to maximise budget and time allocations around communications work in these environments. As such, in evaluating the project’s networking, SNA contributes to investigating success and identifying areas where improvements can be made. This analysis includes determining key and potentially important peripheral network members to focus on. What an evaluative approach adds to the understanding of ROER4D’s reach and engagement on Twitter is two-fold. Firstly, this approach adds the clear articulation of the project’s outcomes that are planned to be achieved in relation to Twitter. Secondly, the explicit establishment of appropriate indicators and a baseline against which to assess the extent of project success in this area are included. Any research project or educator using Twitter as part of their communication strategy, or as a tool in their pedagogical approach, can benefit from taking a similar approach: identifying outcomes and by what indicators success will be measured. ROER4D’s evaluation approach to the project’s reach and Twitter engagement has been guided by the Utilization Focused Evaluation (UFE) framework (Patton, 2008).

The UFE framework

UFE is a 12-step participatory decision-making framework, involving many interactions and feedback loops, which aims to enhance the utility and actual use of evaluations (Figure 1). To achieve the promotion of use of the evaluation findings, this iterative framework forefronts the way real people would apply evaluation findings in the real world (Patton, 2008; Ramírez and Brodhead, 2013). The UFE framework allows the Primary Intended Users (PIUs), who are the real people who will benefit from the evaluation, to play a large role in determining what questions to ask and what information uncovered in the answering of the questions may be the most useful in the context of the project or programme being evaluated. As participatory evaluations have been noted to be particularly amenable to the use of SNA (Durland and Fredericks, 2005), it is an appropriate method to use in this evaluation context.

Steps 1 to 5 in Figure 1 lay the groundwork for a solid evaluation as this establishes readiness for the evaluation, outlines potential uses and provide the basic scope of the evaluation work. Steps 6 to 8 involve narrowing down the scope of the evaluation to meet the project and PIU needs, while taking into account the available resources and prioritising what is most critical to evaluate. This process results in the evaluation design, which is composed of key evaluation questions (KEQs) which are formulated to provide data that link to the PIUs uses. The methods are selected based on what data is required to answer the KEQs, and the methods inform which tools are appropriate.

The evaluation is then conducted as planned in the design, with data being collected and analysed in steps 9 and 10. Step 11 involves the evaluator facilitating the use of the evaluation findings, in this case working with the PIU to communicate the evaluation results. Step 12 focuses on the experience and usefulness of the evaluation process, consolidating what can be learnt from the evaluation for both the evaluator and the PIUs.

Steps 9 to 12 help to reflect back on whether the evaluation has met the needs of the PIU, which in this case includes whether further uses for the networking data can be brought to bear and where the evaluation design can be improved upon as the project moves forward.

Methods

One of the objectives of the ROER4D project is to “build a network of OER scholars” (henceforth called Networking) (Hodgkinson-Williams, 2013: 13), which led to the formulation of a KEQ considering to what extent this building of a network has occurred. This KEQ relates to all the project’s networking activities, of which the Twitter activity covered in this paper is only a part. The main PIU for the evaluation of this objective is the project Principal Investigator, with the project’s Communications Advisor taking an interest in the findings and recommendations.

A very broad definition of ‘OER scholars’ has been used with regard to the social networking sites, which includes any person or entity showing an interest in the project through interaction with ROER4D. This is due to the limitations on what information can be known about a person or entity using the platform from the public information they provide. As ROER4D hopes to: 1) improve the project’s online visibility, 2) increase engagement with the project and 3) broaden their audience and their network, all through communication about the project via Twitter, this broad definition is considered appropriate within the project’s context. These three networking outcomes, highlighted in Figure 2 below, are the focus of the evaluation work reported on in this paper.

From discussion with the PIU, the uses of evaluation results generated with regard to the Twitter activity for the Networking objective were established as being: 1) to improve ROER4D’s communications strategy and 2) to have proof of concept about ROER4D’s networking model on the social media channel Twitter (Figure 3).

From a more practical viewpoint, the Communications Advisor was interested in having information regarding the communications in order to refine the approaches taken. This included information on who was being reached in terms of patterns of interaction over time that could inform future decision-making within the project. After these discussions about the project’s purposes and process for using Twitter, the resources available for this component of the evaluation and what available data could be informative to the project team and PIU in addressing the KEQ, three appropriate tools were selected: Twitter Analytics, Twitter Archiving Google Spreadsheet (TAGS), and the Network Overview Discovery and Exploration Add-in for Excel (NodeXL). These tools could provide, visualise and/or analyse the relevant Twitter data freely extracted from Twitter via its APIs.

Twitter analytics

Now available for every Twitter account, the Twitter Analytics dashboard can be accessed when logged in to Twitter (Twitter, 2016b). The account’s entire tweet activity, including the number of engagements (mentions, retweets, etc.), can be downloaded. This makes further analysis in a programme such as Microsoft Excel possible. The number of followers, including their interests, location and demographics (where available), can be explored via the dashboard. For the ROER4D evaluation, the focus is on analysis of the ROER4D Twitter account’s tweet activity and engagements, which were analysed and graphed using Excel.

TAGS

TAGS (Hawksey, 2014) allows for an archive of Tweets containing a specific phrase to be accumulated for analysis. This tool, which creates the archive in Google Spreadsheets, runs a web-based script to retrieve the Twitter data around a certain word or phrase (Gaffney and Puschmann, 2013). As well as this archive, TAGS also provides a dashboard overview including the top Tweeters of the phrase. Due to its reliance on Google Spreadsheets, this tool requires the user to have a Google account. The data from the archive can also be explored graphically via TAGSExplorer (Hawksey, 2016). It is important to note for data analysis and interpretation that TAGS uses the Twitter Search API to extract Twitter data, which has been found to over-represent central Tweeters, thus not accurately capturing peripheral Tweeter activity (Gonzalez-Bailon et al., 2014). In the case of ROER4D, tweets containing the phrase ‘ROER4D’ have been collected from 11 November 2014.

NodeXL

NodeXL1 (Social Media Research Foundation, 2014) is an open-source Microsoft Excel template which will graph node and relationship data (Smith et al., 2010). While this tool will work with any network data consisting of nodes and the ties between them, a focus of NodeXL is social media networks (Hansen et al., 2010). NodeXL provides an SNA graphing and analysis tool that can be quickly mastered by those familiar with Excel (Smith et al., 2009). Networks of follow-following relationships and of users who tweeted a specific term can be visualised and analysed. In terms of Twitter data, NodeXL makes use of Twitter APIs to extract network data for further analysis from an approximate 7-day window preceding the request (Hansen et al., 2010; Rogers, 2013).

The SNA tools workflow

Making use of the tools described above, both the amount of engagement at various time-points and the breadth of the audience and the network could be assessed. Taking into account the data that could be extracted using each of these tools, the basic tools workflow, capitalising on the strengths of each tool, was produced (Figure 4).

Figure 4

Workflow of ROER4D’s Twitter tools and data collection.

Overall, these three sources provide data on audience as well as engagement (account activity and interactions), both over the history of the Twitter account (interactions – over time) as well as at key time-points (interactions – 7-day window), and enable an archive of tweets to be kept for any further analyses. Gathering Twitter data over the whole life of a Twitter account or at least over long stretches of time helps to ensure a more representative sample of the Tweeters and their interactions (Gaffney and Puschmann, 2013). The Twitter Analytics data were examined for growth in the number of Twitter followers and overall engagement with tweets, both over time. This analysis highlights key points of activity that the ROER4D project can learn from and capitalise on. Looking at the TAGS data helps to identify the key people involved in interactions over time. The NodeXL data provides a more detailed, relation-based lens focusing in on a window of Twitter activity which indicates key Twitter users who are currently talking about the project (using the phrase ‘ROER4D’) on Twitter and their @Replies and mentions in these tweets. This analysis highlights interested individuals, focusing on those not directly interacting with ROER4D who can then be drawn into discussions, strengthening the project’s network of interested Tweeters.

As part of the ROER4D evaluation, data is generally collected monthly or when the PIU requires an update, for example, during or after a conference.

Data use in ROER4D

Updates on findings and recommendations are presented to the ROER4D team periodically in order to best facilitate use of the evaluation results. The data included in this paper spans the 150 weeks of the project that were the focus of the evaluation work. How data gleaned from the tools workflow has informed the project team in the case of ROER4D is presented below. Several of the metrics reported are informed by the key Twitter communication metrics put forward in Bruns and Stieglitz (2012).

Twitter analytics

Over 150 weeks of the project, ROER4D has gained 825 followers. In total, ROER4D has sent 984 tweets over those 150 weeks. On average, 6.6 tweets per week were sent from the ROER4D Twitter account over this period, providing the project with regular online visibility. It was found from early on in the project that Twitter activity peaks during events which ROER4D attends and shares via social media (Figure 5). This engagement includes retweets, replies, likes and clicks on the tweets. This finding has resulted in social media being strategically used during events to increase project visibility and engage with interested parties, who are identified using the TAGS and NodeXL tools’ affordances.

Figure 5

Graph showing the ROER4D’s Twitter account engagements per week over the course of the project. Peaks over 50 engagements per week (above dashed line) correspond with conferences, workshops and event weeks.

TAGS

TAGS data shows that in total 2,449 tweets including the phrase ‘ROER4D’ were sent by 525 unique users (excluding the ROER4D Twitter account). The average number of tweets which included the phrase ‘ROER4D’ sent by each of these Tweeters over 150 weeks was 4.7 (Range: 1–197). Over the course of the project, a core group of 16 Tweeters who most frequently use the phrase ‘ROER4D’ in their tweets has emerged (Figure 6). Knowledge of who the members of this core group of Tweeters are has provided information about key interactions as well as interaction gaps with strategic groups identified by the project team in the existing social network. This audience analysis and response informed moves to cultivate both direct and outwardly visible communications with these groups, to increase active engagement and fill the audience gaps within the project’s online network. The result was an increase the number of key members of the open education and OER communities being engaged on Twitter (see Answering the Evaluation Questions section below).

Figure 6

Graph of number of tweets containing the phrase ‘ROER4D’ showing the core group of 16 highly engaged Tweeters (before dashed line), who tweeted the phrase more than 30 times.

NodeXL

Any time points of interest that are identified using the above two tools (Twitter Analytics and TAGS) were investigated using NodeXL to graphically represent the communication interactions between ROER4D and other Tweeters using the ‘ROER4D’ phrase over the preceding 7 days. An example of such a representation is shown in Figure 7. This time-bounded information provided insights into the network of Tweeters communicating with and/or about ROER4D within a timeframe that could facilitate ROER4D strategically responding. This network-level viewpoint provides a unique perspective on the sum of interactions between tweeters that would not be found from the Twitter platform alone. As such, ROER4D has engaged some of these Tweeters using topics that they are discussing online within 7 days of these discussions. Tweeters and networks that are isolated (using the phrase ‘ROER4D’ in a single tweet; blue and green, respectively, in Figure 7) and which are indirectly linked to the conversation (using the phrase ‘ROER4D’ in a discussion with a Tweeter directly connected to ROER4D; orange) are prime targets for this engagement.

Figure 7

Directed network diagram showing the replies and mentions relationships between the Tweeters who have used the phrase ‘ROER4D’ over a 7-day period. The distances between nodes are arbitrary.

Answering the evaluation questions

How the above data approach has addressed the evaluation questions is explored below, per question.

To what extent has ROER4D built a network of OER scholars?Using the broad definition of ‘OER scholars’ in the project context on Twitter, there are several useful metrics that address this evaluation question for the ROER4D project. Table 1 shows the measures at weeks 1, 75 and 150, and their relationship to each relevant project outcome.This data shows that the initial building of a network in the first half of the project began with a period of slower growth, as interested parties began to learn about, followed and started to engage with the project. This was followed by a large increase in the number of followers and @Replies during the second half of the project. This increase is as a result of interested parties becoming aware of ROER4D through their networks and connections, as well as through increased active engagement by ROER4D. There was, however, a slowing in the number of impressions. As this metric quantifies the extent of the audience the tweet has the potential to reach, this smaller potential audience indicates that the content is being served in the timeline of or searched for by a more specific audience over the second half of the project.Retweets remained relatively constant while @Replies were greater in number in the second half of the project compared to the first half. This points to an increase in direct and meaningful engagements and the start and continuation of online discussions. The @Replies, combined with mentions, were found to be largely from a core group of individuals who have either contributed to the project or have built up a connection with it over time. At week 75, 77% of engagements (61 @Replies or mentions) from the top ten entities engaging with the project on Twitter were from project team members and 23% were from the open education or OER community. At week 150, 64% of engagements (115 @Replies or mentions) from the top ten were from project team members, 24% were from the open education or OER community and 12% were from those affiliated with the project funders. These figures indicate a strong intra-project network has been built up on Twitter, with the project team members engaging with the ROER4D account. While the open education and OER community percentage of the engagement has not grown, the overall numbers of engagements from these entities has increased between the first and second half of the project. The funder also increased engagement with the project via Twitter in the second half of the project. This is linked to the increased production in outputs from the project that were being shared on Twitter.

Which of ROER4D actions have been most effective at growing the network?Increased levels of tweeting linked to events and active engagement with those mentioning and @Replying ROER4D have been the most effective means of building up a following of engaged tweeters.

Average number of engagements

All event weeks

110.9

Up to week 75

104.1

From week 75 to week 150

118.3

All normal weeks

13.0

Up to week 75

15.3

From week 75 to week 150

10.9

Peaks of engagement, which have on average been 7.5% higher than engagement happening at other times, have accompanied the increased number of tweets (see Figure 5). This increased engagement in turn, increases online visibility of the project’s Twitter account and tweets, resulting in increased network growth over those periods of time.

Table 1

Measures at weeks 1, 75 and 150 of the ROER4D project’s activity on Twitter.

Related project outcomes (see pg. 4)

Measure

Number in week 1

Number at 75 weeks

Average number per week (up to week 75)

Number at 150 weeks (cumulative)

Average number per week (from week 75 to week 150)

% change between week 75 and week 15 averages

1, 3

Number of followers

5

460

6.1

825

8.9

45.9%

1

Number of impressions

600

155, 350

2071.3

245, 525

1202.3

–42.0%

1, 2

Number of retweets

1

582

7.8

1 178

7.9

1.3%

2

Number of @Replies

2

36

0.5

113

1.0

100.0%

Discussion and conclusion

The approach that ROER4D has used, involving systematic measuring systems to collect, select and analyse Twitter data, allows for analysis that shows the extent of network building and what has worked to achieve this. This analysis shows that the project built up a network of 825 scholars over 150 weeks, with the project’s Twitter audience relatively more actively engaged in the second half of the project than the first half. A core of group of tweeters engaged with the project were identified, with the top ten engaged Tweeters being members of the project team, the funder and members of the open education and OER communities. Tweeting around conferences or other events attended by others in the OER and/or open education community has contributed to this growth and resulted in key nodes of interactions.

While the 150 week mark and the mid-point (75 weeks) was considered in this paper, this specific method enables the project to be able to quantify the interactions over any desired time period and gain specific insights into how these interactions came about and how relationships have been solidified or lost over time. This data collection is largely automated, meaning that it is a relatively low effort approach to gain an aggregated view of network-level interactions for a great return in terms of the insights it can generate. This allows for an evidence-informed approach around approaching future networking opportunities, based on what has worked well in the past. The basic notifications and alerts sent by Twitter do not allow for this level of data accumulation and would need to be manually tracked to allow for analyses. While the data gathered through Twitter’s API is not necessarily completely comprehensive, as it is open to technical glitches and may not pull through all of the tweets, automated data collection is more accurate than manual approaches. Approaches of collecting data relating to networking using manual counting, observation and self-report have issues of being expensive, open to large errors and recall bias, while being time-intensive and not necessarily comprehensive in coverage.

As SNA is focused on relationships, it is an appealing method for those working on projects or programmes that include a networking aspect (Fredericks, 2013). The evaluation work presented here has resulted in the establishment of a tools workflow (see Figure 4) that makes best use of the available SNA data to inform the ROER4D PIU about the project’s networking objective outcomes relating to Twitter. This workflow of the appropriate tools enables the project’s Twitter network growth and snapshots of key online interactions to be tracked over time. In terms of the project’s outcomes relating to Twitter, increases in engagement with the project Twitter account have been found at various time-points and a broader audience and network has resulted over time. This provides evidence to be considered in relation to the overall networking KEQ regarding the extent to which a network has been built up over the course of the project.

The SNA approach has provided the project with detailed data about its Twitter social interactions in a useful timeframe, as discussed with the PIU, increasing the utility of this aspect of the evaluation to ROER4D. The evaluation recommendations have leveraged this data to indicate potentially important nodes with which to connect (Smith et al., 2009). In addition, the PIU as well as the project’s Communications Advisor have found the information gained from tracking of the ROER4D Twitter network useful in continuing networking efforts via Twitter as the project has moved forward.

Durland and Fredericks (2005) noted that while the SNA methods can help to answer evaluation questions posed, the data collected may also provide a broader view of the complexity of the programme (or project, in this case) being evaluated within its context and furnish key information on what works. While the evaluation work described above considers only the overall Twitter account activity and engagement at key points over time, the data produced from SNA has the potential to provide additional key information to the ROER4D project, an aspect not covered in this paper. Three examples of how this SNA data could be further used and analysed are briefly described below.

Firstly, for strategic ties identified through the SNA of the Twitter account, an investigation into which other communication channels they engage with the project on (if any) could inform the project of where effort could be best placed to strengthen these ties (Haythornthwaite, 2005). A second point of investigation could be to analyse and consider the implications of directed ties, which indicate the direction of the relationship from one node to another (Marin and Wellman, 2011). This could inform the project about the direction of the relationships in terms of who is replying to whom, and who is mentioning whom using the phrase ‘ROER4D’ in their tweet. Finally, using SNA metrics that quantify various aspects of the ROER4D Twitter network could help to put a number on priority relationships, indicating which nodes may be key within the network to cultivate to reach the widest possible audience. One example of such a metric is clustering coefficients, which measures the degree to which nodes are directly linked to each other or tend to cluster together (Freeman, 2011). Targeting nodes which are not within the core cluster can help to reach a wider audience. Additionally, this can be useful in a project such as ROER4D which is attempting to cultivate audience for research outputs from sub-projects based in countries across the Global South as, when combined with the location of the Tweeters it can assist in geographical audience targeting.

In conclusion, SNA provides a robust approach to identifying and analysing online social ties (Hogan, 2008). Implementing a UFE approach to evaluating the Networking objective in the ROER4D project has allowed for the production of a useful tools workflow that has been put in place to enable the timely extraction, visualisation and analysis of appropriate data and answers to the evaluation questions. These answers, as reported back to the project by the evaluator, have helped to inform the project of what is working, who is responding and where strategic approaches can be made towards growing the network and improving engagement. The insights from the findings presented here have informed project future decision-making regarding both communications and networking strategies.

The same methods and the tools workflow as presented here could feasibly be applied to other research projects or programmes that are using Twitter. The tools used are available free of charge, which makes them ideally suited to a resource-constrained scenario. For someone using this workflow in higher education beyond research projects, the ability to set up low effort, automated workflows for tracking, for example, interactions and engagement for each of a variety of courses or workshops would be of use. The affordances of NodeXL in particular would provide snapshots of learner interactions, which could be used to inform grades linked to level of engagement on the platform. Having a data set which can be mined to address key learning questions around how using a platform such as Twitter has integrated into the pedagogical approach used is invaluable in the education context.

The data on various network relationships and node interactions, brought to bear on appropriate evaluation questions, can feed back into planning and implementation. This would strengthen education project communication and networking activities, using an evidence-based approach.

Acknowledgements

This work was carried out with the aid of a grant from the International Development Research Centre (IDRC), Ottawa, Canada. The author would like to thank the IDRC and the ROER4D project for supporting this evaluation

Darling, E, Shiffman, D, Cȏté, I and Drew, J (2013). ‘The role of Twitter in the life cycle of a scientific publication’. Ideas in Ecology and Evolution 6(1): 2–43, DOI: https://doi.org/10.4033/iee.2013.6.6.f

Department of Higher Education and Training (South Africa) (2015). Speech by the Minister of Higher Education and Training, DR BE Nzimande, to parliament, Oct 27 2015

Knight, C G and Kaye, L K (2016). ‘To tweet or not to tweet?’ A comparison of academics’ and students’ usage of Twitter in academic contexts’. Innovations in Education And Teaching International 53(2): 145–155, DOI: https://doi.org/10.1080/14703297.2014.928229

Veletsianos, G and Kimmons, R (2016). ‘Scholars in an increasingly open and digital world: How do education professors and students use Twitter?’. The Internet and Higher Education 30: 1–10, DOI: https://doi.org/10.1016/j.iheduc.2016.02.002