Why is it Difficult to Detect Sudden and Unexpected
Epidemic Outbreaks in Twitter?

Online message classification continues to be a complex and challenging task for long term EI surveillance and intelligence gathering in general. One reason for this is that given the evolution of real-world events, the variable to observe cannot always be known a priori. One such example is in the detection of food-borne illness, in which the contaminated food item is not known in advance.

More in detail, the main challenges faced in Stage I are as follows.

Feature Change Detection.  Detecting new relevant terms and filtering irrelevant tweets, are inter-related and impact each other. We need a way to: 1) dynamically detect when new and relevant terms appear in health-related tweets over time; and then 2) subsequently incorporate the tweets containing these terms as part of the data to train our classification models.

Dynamic Labeling. As terminology evolves, the criteria for defining a relevant tweet is likely to also change. However, expert labeling of classifier training instances is expensive and in practice difficult to obtain, especially for the rate and volume needed to build and maintain a good classifier.

In the rest of the section, we present the approach taken in M-Eco to address these challenges.

The architecture of M-Eco’s module encapsulating Stage I has been designed around the idea of novelty management. In particular, we incorporate a feature change detection mechanism into a framework for the adaptive classification of tweets.

Existing message classification techniques rely usually on supervised or unsupervised machine learning approaches [Fisichella et al. (2010), Paul and Dredze (2011a)]. In M-Eco, we defined a solution based on a semi-supervised approach to determine if tweets are relevant or not for outbreak detection and threat assessment.

The choice of features and classifier was driven largely by the fact that the M-Eco system runs continuously, so features and classifiers that are not time-consuming to extract or encode need to be used. In the course of our investigation, we found that a linear Support Vector Machine (SVM) model exhibits a favorable trade-off between classification performance and training time.

Our approach builds upon [Hido et al. (2008)] and includes three main novel contributions beyond the state-of-the-art by 1) modelling and describing the feature change over time using an orthogonal vector, which is learned by a SVM; 2) computing a novelty score that lets the system identify those tweets that contribute to the feature change, so that; 3) manual labels can be obtained, dynamically and on-demand, by asking a human judge as part of an active learning setting [Settles (2010)].

One aspect that we assess in the experiments for this stage, is to what extent the expertise of the annotators impact the classification quality. To this end, besides public health experts, we also consider crowd-sourced workers from the CrowdFlower platform¹¹1http://crowdflower.com/ as annotators.

Figure 2 presents an overview of M-Eco’s feature change detection and adaptive classification steps. The Message Filtering process runs continuously to classify all unlabeled tweets, which are arriving from the Twitter Streaming APIs.

All tweets are annotated with locations, medical conditions, and temporal expressions using a series of language processing tools, including OpenNLP²²2http://opennlp.apache.org/ for tokenization, sentence splitting and part-of-speech tagging; HeidelTime for temporal expression extraction [Strötgen and Gertz (2010)]; LingPipe dictionary-based entity extraction for medical conditions [Alias-i (2008)]; as well as entity extraction tools developed within the project for location entities [Otrusina et al. (2012)]. In this paper, we only focus on text-based analysis of Twitter messages.

Then, we use two types of classifiers: a Virtual Classifier (VC) and an Adaptive Classifier.

Positive keywords associated to diseases are pathogen (e.g., Streptococcus pyogenes) and symptoms (e.g., sore throat, fever, bright red tongue with a strawberry appearance, rash, bumps, itchy, and red streaks).

For filtering tweets irrelevant to the medical domain, we use a list of negative keywords associated to diseases from two freely-available resources: 1) MedISys³³3http://medusa.jrc.it/medisys/homeedition/en/home.html providing a list of negative keywords created by medical experts, and 2) Urban Dictionary⁴⁴4http://www.urbandictionary.com/, a Web-based dictionary of slang, ethnic culture words or phrases.

The Novel Tweet Detection process is run periodically to trigger an update on the current adaptive classification model. In our case, a period of one week was used to correspond with weekly reporting performed by the health agencies in our study.

The Novel tweet Detection phase is responsible to detect a feature change. To this end, the Virtual Classifier is used to compare incoming and unlabeled tweets against the set of existing tweets (those previously labeled during the Adaptive Classification step). If it is determined that feature change has occurred, novel tweets – those for which it is expected that the classifier will not be able to correctly label – are flagged and channeled to the Labeling Process so that manual labels can be obtained by a human judge.

Note that, regardless of whether feature change has been detected, all tweets are classified with a model that has been trained on the labeled instances that the Adaptive Classifier put into the Health-related Tweets storage. We apply the adaptive classification algorithm to all tweets presented to the system, even if feature change was detected, because the relevant tweets are needed by downstream components of our pipeline.

The labels for novel tweets are acquired using dynamic labeling, in which for-hire Human Intelligence Task (HIT) or crowdsourcing, is used to obtain labels on-demand. Until the system receives HIT labels for the novel tweets, it uses temporary (automatic) labels given by the current Adaptive Classification model.

After labeled tweets have been obtained from the HIT workers, they are used as input to the Adaptive Classifier; a new classification model is trained; and any temporary tweet labels are updated for improving and maintaining its accuracy over time. Unlike other work [Paul and Dredze (2011b), Demartini et al. (2012)], we also address the challenge of assessing the quality of the HIT labels against those of our domain experts (cf. Section 2.3).

The VC detects feature change through a scoring mechanism in which the most novel tweets in the data stream are identified and presented to a human for labeling. Our experiments (cf. Section 2.3) show that the selected novel messages reflect the events in the real world that cause feature change and that they are useful for training and maintaining an accurate classifier for EI.

The VC corresponds to a SVM model that maps a feature vector to a hypothetical class (i.e., novel/not-novel). The VC is trained to learn a decision boundary between the unlabeled incoming tweets and those existing in the collection. If this decision boundary is highly accurate, then it is very probable that feature change has occurred between the two data sets [Hido et al. (2008)].

Novel Tweets Detection. M-Eco assigns to each newly incoming tweet a score that represents how novel it is to the system. This score is given with respect to the distance to the separating hyperplane of the aforementioned (binary) VC, which is trained on the set of labeled tweets and the set of new incoming tweets. We assume that the greater the distance between the newly incoming tweet and the tweets known to the system, the more dissimilar they are.

Based on active learning principles, that is to label those instances that would most change the current model [Settles (2010), Georgescu et al. (2014)], we propose that obtaining labels for the most novel tweets will help the system to keep its accuracy over time, a hypothesis that we test in our experiments detailed in Section 2.3.

Labeling Criteria. It is very challenging to determine if a tweet is relevant or not for EI, even for human experts. This is due to the fact that contextual information within the short messages is very limited, unlike existing work in the domain of EI based on news articles [Collier (2010)].

In an effort to address the needs of our system for online and dynamic labeling of tweets, together with a team of epidemiologist, we created a set of simple annotation guidelines for determining the relevance of tweets for the task: a tweet is relevant if somebody reports himself or another person being ill while it is irrelevant if no one is suffering from symptoms; i.e., mentions refers to opinion, advertising, jokes, music, books, films, artists, landmarks, sporting events, slang, etc.; and for the given set of criteria, we examine crowdsourcing as an alternative to obtaining the correct labels of tweets by the experts themselves.

In the rest of the section, we present the results of the empirical evaluation to assess this approach.

The evaluation is composed of two separate experiments. In the first one, we were interested in evaluating the combined effectiveness of the Adaptive Classifier and of the Virtual Classifier over the time.

In the second experiment, we evaluated the quality of the labelling process carried out by crowdsourcing compared with the one obtained by public health experts.

Feature change detection. We design the experiments to evaluate the suitability of the proposed approach to determine if tweets are relevant for outbreak detection as follows.

We train and measure the performance of the Adaptive Classifier in a given time span and then use the model for prediction over a subsequent time span, with three different strategies to account for feature change:

1. 1.

   No feature change handling strategy in which a non-adaptive classifier and no retraining are used;

2. 2.

   Random selection of tweets used to retrain the Adaptive Classifier,

3. 3.

   Novelty selection, which uses the novelty scoring to select the tweets to retrain the Adaptive Classifier.

In detail, we conduct the evaluations using a dataset consisting of 6,625 tweets collected within a time period from week 12th through week 14th of year 2011. These tweets were randomly selected from those containing at least one word in the Health-related tweets database (cf. Figure 2). Then, we manually labeled them to determine if they were in fact health related or not, based on the guidelines outlined earlier in this section. Finally, we trained a SVM binary classifier using feature vectors derived from a bag-of-words representation combined with bi-grams.

We assess our results in terms of accuracy, i.e., the proportion of true results (both true positives ($TP$) and true negatives ($TN$) among the total number of tweets examined:

Here, $TP$ corresponds to the number of tweets classified and labeled as health-related, and $TN$ to the number of tweets classified and labeled as non health-related.

We then applied this classifier to the feature change detection task. We employed another dataset of 6,625 tweets, now randomly selected from the calendar weeks 15th through 19th of year 2011. We created a subset of about 1,100 positive and negative tweets for each week and manually labelled them as well.

We considered three different scenarios with a varying percentage, $q$, of the novel tweets. In particular we evaluated the scenarios where $q=1\%$, $q=5\%$ and $q=10\%$, using a significance level, $\alpha=0.01$. Results are reported in Figure  3.

From these results, we can derive the following conclusions:

1. 1.

   The strategy using the Virtual Classifier provides on average the best results, and the more novel tweets are, the better is the accuracy of this strategy.

2. 2.

   The strategy without any change provides the poorest results, highlighting that the dictionary evolves over the time (even over the limited timeframe we considered) and some mechanism to update the classifier is needed.

3. 3.

   The accuracy of the classifiers degrades for most strategies during week 17th, 2011. This time slot contains many tweets mentioning “royal wedding fever”, referring to the wedding of Prince William and Kate Middleton on April 29, 2011. Those tweets are sometimes misclassified as relevant.

Crowdsourcing versus Experts.  The second experiment explores the quality of the labelling process performed by human annotators. To this end, we employed a total of 1,500 tweets, by randomly sampling 500 tweets from each of the calendar weeks 15th, 16th and 17th, 2011. We presented them to 43 workers of the CrowdFlower platform to elicit their feedback.

To control the quality of the crowdsourced labels, two actions were taken. First, a set of “golden” tweets with known labels was added to the unlabeled tweets that were randomly shown to the workers. A trust value is computed for each worker based on the number of correctly labeled “golden” tweets. If this trust value is below a fixed threshold, the workers labels are removed from the task. Second, each tweet was labeled by a minimum of 3 workers for each task, and only those tweets corresponding to a majority agreement above 65% percent among the multiple annotators were used.

The average agreement among the 43 HIT workers on a given tweet was 93.89% with an average accuracy on the injected gold labeled tweets of 92%.

Out of the 1,500 tweets labeled by the crowd, 1,114 tweets had a perfect agreement of 100%; 295 tweets had an agreement between 66% and 100%; and 91 tweets had an agreement between 50% and 66%.

We chose 130 tweets according to the agreement of the crowdsourcing annotators to include a representative amount of “easy” tweets (perfect agreement) and more “difficult” tweets (low agreement). Then, we asked five public health experts to also annotate this subset and measured their inter-annotator agreement on the 130 instances, which achieved a 89.33%.

For the 130 tweets labeled by both the experts and the HIT workers, there was a percent agreement of 87.69%. The percent agreement is the ratio of number of agreements among crowd and experts to the total number of units judged by crowd and experts.

When measuring the classifier performance individually for each group, based on a 10-fold cross validation and equal percentages of positive and negative examples, the classifier performance in terms of accuracy was 75% for the experts and 83% for the HIT workers.

Our results suggest that people outside the public health domain are able to accurately judge the relevance of tweets when given a simple set of criteria. Thus, once the modules of Stage I have detected a feature change, it is also feasible to outsource the novel tweets as part of a separate feature change handling procedure, without the necessary involvement of a public health expert.

Although the classifier performance was less for the expert labels, than the crowd labeled data, we believe this is due to the fact that in practice, whether a tweet is relevant for an expert depends on several factors, such as different time periods of an outbreak (e.g., before, during or after); or on the task and role of the expert with respect to an epidemic investigation. Nonetheless, the crowd can still help to filter label instances that are clearly off topic.

To get a better understanding of the impact of detected feature change on the classification accuracy, a larger set of expert labeled tweets for experimentation would be useful to further improve the significance of the results. However, by doing so, it would still not address the need for experts to re-label each time feature change is detected and in practice, the overhead of such a task is too expensive and not timely enough. We propose instead, that after tuning expert labeled examples with a good inter-annotator agreement, such instances could be used as gold standard to filter out HIT workers whose trust value is below a threshold.

The majority of existing works center their studies on recurring diseases only, e.g., Influenza-like Illness (cf. Section 5). Moreover, many researches are either focused on a single nation and language, or the spatial dimension is not considered at all.

Our goal is to provide a broader EI system, able to detect and monitor outbreak events in Twitter, for multiple locations and for multiple diseases, including sudden and unexpected outbreaks (cf. Table 3.1). This goal is a challenging task, due to two main issues:

Spatio-Temporal Monitoring of Diseases. Location-awareness is one of the key starting points for any EI solution. Indeed, knowing where an outbreak is happening is naturally one of the most important pieces of information.

From this point of view, the typical use of Twitter is not helping. For instance in our experimentation, we observed that explicit coordinates where present in less than 1% of the collected tweets. As a consequence, other techniques, mainly based on Natural Language Processing, or on the analysis of user profiles of people tweeting are needed in order to infer the location of a tweet.

Temporal and Spatial Dynamics of Diseases. The characteristics of infectious diseases are highly dynamic in time and space, and their behavior varies greatly among different regions and the time periods of the year. E.g., some infectious diseases can be rare or aperiodic, while others occur more periodically. In addition, various diseases have different transmission rates and levels of prevalence within a region.

To get a deeper insight on this issue, we collected information about five outbreaks that occurred in 2011, which are detailed in Table 3.1.

By analyzing the time series of the tweets about these five different outbreaks, we found out that the Twitter data regarding the public health outbreaks can be characterized by two dimensions: (1) Oscillation, which is seen as the frequency at which the curve spikes, and (2) Magnitude (or volume) of daily count of tweets, sinks or slopes.

Figure 4 shows representative examples of the different outbreak dynamics from Twitter time series data. The gray areas in these plots represent the timeframe where an outbreak alert was broadcasted via ProMED-mail [ProMED (2015)], a global reporting system providing information about outbreaks of infectious diseases.

A time series with low oscillation indicates that the average daily number of tweets is more or less constant (eventually zero) but it noticeably peaks within the outbreaks period. Examples are provided by the two bottom frames of Figure 4. On the left there is the distribution of tweets for the outbreak of Botulism in France in 2011. We can see that the magnitude of the tweets is very low, both outside and inside the emergency time frame. This situation may occur in the scenarios where the diffusion of English tweets is limited.

On the other hand, the picture on the right is the case of the Escherichia Coli outbreak in Germany, 2011. Since it had a very high international and media coverage, due to the ease with which it could spread, we can see that the number of tweets during the peak is almost two orders of magnitude higher than the example of Botulism in France. In general, in presence of diseases leading to low oscillations of tweets, it is easy for surveillance algorithms to produce correct alerts.

A time series with a high oscillation means that the number of tweets varies often and greatly over the year. Considering the outbreak Anthrax in Bangladesh (Figure 4, upper left frame) as an example of high oscillation/low magnitude time series, we can observe a number of tweets per day ranging from 0 to 5 all over the year. Thus, a less sensitive surveillance algorithm is necessary to avoid to get continuously false positives.

A time series with high oscillation and high magnitude occurs when: 1) a disease occurs continuously in a country, such as Mumps or Leptospirosis and/or 2) the name of the disease is a highly ambiguous term, such as for Mumps, which is for instance also the name of a software system⁵⁵5MUMPS is also the acronym for Massachusetts General Hospital Utility Multi-Programming System – https://en.wikipedia.org/wiki/MUMPS.

In Figure 4, upper right frame, we report on the Mumps outbreak in the USA, which is not included in the evaluation since it was not possible to find a reliable ground-truth regarding the dates of the emergency.

In these situations, algorithm tuning is essential, since the dynamic of data is too large, and it is not easy to identify significant aberrations.

The Stage II of our EI system takes as input the dataset containing health-related tweets coming from the Stage I, and provides two outputs: (1) a collection of datasets containing health-related tweets, aggregated by the spatial dimension and (2) a set of alerts, triggered by surveillance algorithms analyzing the temporal dynamics of each of these datasets.

Thus, Stage II encompasses two main steps, context creation and biosurveillance, which are described as follows.

Context creation. We are interested in monitoring, for each location, a set of possible diseases. In order to infer the location associated to a tweet we define three rules, which are applied in order of importance, as follows:

1. 1.

   Mention of the location (city, country, etc.) in the text of the tweet.

2. 2.

   The tweet’s geo-location information (latitude and longitude), if present.

3. 3.

   Location indicated in the user profile of the author of the tweet.

Tweets not matching any of the above rules are discarded. Even if they are health-related, it is not possible to understand where the event is located. Thus, their informative contribution is negligible. Moreover, we do not consider the tweet language (provided as an attribute by the Twitter API) in determining location information, since it is too inaccurate.

In the end, the geographical granularity level we considered is the country. The geo-mapping between location or coordinates and the relative nation was performed using the Yahoo BOSS Geo Services APIs ⁶⁶6https://developer.yahoo.com/boss/geo/.

The output of this step is a collection of datasets, which are intended as a spatial partitioning of the original dataset coming from the Stage I. The total sum of contained tweets is most likely lower, since tweets without any information on the location are discarded.

Biosurveillance. A standard approach to detect anomaly in health-related time series data is to leverage state-of-the-art biosurveillance algorithms. Many of them can be found in the literature [Farrington et al. (1996), Hutwagner et al. (2003), Khan (2007)] and implemented in analysis tools like R [Höhle et al. (2015)].

In our study, all biosurveillance algorithms are included in the free package Surveillance for R, that implements multiple statistical methods for the “Temporal and Spatio-Temporal Modeling and Monitoring of Epidemic Phenomena” [Höhle et al. (2015)].

In particular, we assessed the following four algorithms: C1, C2, C3, and Farrington, which are described as follows.

The rationale behind the selection of these methods is that the EARS family require a very limited number of previous data to provide an alert, thus being potentially suitable for a new outbreak detection system. Moreover, it has basically no parameters to tune, make its applicability straightforward.

On the other hand, the Farrington algorithm is largely considered a robust and fast method. Thanks to these characteristics, currently it is the method used at European public health institutes [Hulth et al. (2010)]. This algorithm has a set of parameters to specify. In particular, among others, it requires to define the windows size, $w$, i.e. number of weeks to be considered for the alert generation. We performed our experimentations with $w$ = 2, 3, and 4. Nevertheless, since the choice of the parameters heavily depends upon the data, it is outside the scope of this research to investigate hyperparameter optimization techniques (e.g. as in [Corazza et al. (2013)]).

Further details on these algorithms and their implementation can be found in the R-Surveillance documentation [Höhle et al. (2013)].

Here we seek to address the following question: what are the most suitable surveillance algorithms for outbreak alert generation using Twitter data?

To perform our study, we analyze Twitter data collected from January the 1st, 2011 to December the 31st, 2011. The data was collected using the pipeline defined in the Stage I (Section 2), resulting in a total of 112,134,136 health-related tweets.

Ground Truth. Studying the usefulness of Twitter data in an early warning task requires real-world outbreak statistics. Therefore, we build a ground truth by relying upon ProMED-mail [ProMED (2015)]. An outbreak event is intended as a temporal anomaly found in time series data that occur when the impact of an infectious disease is above an expected level at a certain time.

We collected 3,056 ProMED-mail reports occurred during year 2011, and among them, we selected 5 different outbreaks according to two criteria: 1) a clearly identifiable starting date from the ProMED-mail, by considering the first ProMED-mail post on it, and 2) a representative distribution of tweets.

An important aspect of our work is that we consider the duration of each outbreak by manually analyzing the text of each ProMED-mail document, unlike previous work [Collier (2010)] that assumes the publication date of a document as the estimated relevant time of an outbreak.

In particular, we determine the starting date of a disease by looking at the text inside the first ProMED-mail post, and the ending date was associated to the text inside the last ProMED-mail publication, for that particular disease-location pair. One reason for doing this is that the events in ProMED-mail undergo moderation, so there is often a delay between the time of the actual outbreak and the publication date of the related report. However, it is worth noting that this strategy gives us a good confidence only on the beginning date of the outbreak. In fact, the absence of further ProMED-mail posts does not necessarily mean an end of the outbreak, but just that there was no significant news in which it was reported.

Evaluation Metrics. To assess the quality of the generated alerts, we use standard Information Retrieval metrics, namely precision, recall, and f-measure, which are defined as follows:

With our problem at the hand, a valid definition of the temporal granularity is not easy, due to the high temporal variability of the data and the lack of a proper ground truth. To clarify it, in case an algorithm generates an alert 5 days before the official communication in ProMED-mail, should it be considered a false positive or a true positive with a good timeliness? We opted for the second case, thus defining a $TP$ (True Positive) as an alarm that is raised within the time frame of the ProMED-mailalert or up to 10 days before it. A $FP$ (False Positive) is an alert generated outside this above defined time frame. A $FN$ (False Negative) is an alert not generate in this time frame.

In Figures 5 and 6, we present the results we got from the application of the 4 different surveillance algorithms to our dataset. In particular, as described before, for the three EARS algorithms, we used the default parameters, while for the Farrington algorithm, we tested the values 2, 3 and 4 as for the windows size. Consequently in Figures 5 and 6 we report 6 different results (namely EARS C1, C2 and C3, Farrington W2, W3 and W4) for the five evaluated diseases.

As mentioned earlier, our goal is to detect outbreak events for general diseases that are not only seasonal, but also non-recurring diseases. From the results we collected, we can report the following lessons learned:

– In general all the algorithms present a very high recall (very close to 1), meaning that they are able to detect very well the epidemic outbreaks. This is an expected result, given the high correlation that is clearly visible in our datasets between tweet peaks and outbreak timeframes.

– The real difference among the algorithms can be seen in terms of number of false alarms (i.e., precision). We can notice that in general the EARS family is providing a large amount of false positives. Probably this is due to the short time window used from these algorithms. On the other hand, Farrington algorithm performs better in terms of precision and f-measure, regardless of the window size. Thus we can in general confirm the effectiveness of the Farrington algorithm for this kind of task. As already reported in Section 3.2, this is the standard algorithm for epidemic outbreaks in many health institutions. In our case, the Farrington results are always better than those of the three EARS algorithms.

– The hyperparameter window size of the Farrington algorithm has an impact on the quality of the provided alarms, but no generalizable trend can be devised from our experiments. The window size should be defined according to the specific temporal dynamics of each considered disease. A solution to automatically find the best hyperparameter optimization would be highly recommendable to solve this issue.

– As expected, there is a noticeable difference between low and high oscillation diseases. In the latter case, all the algorithms perform significantly worse.

Even though algorithms for recommender systems and learning to rank are agnostic to the problem domain, their application for EI based on Twitter is not straightforward. We identified two major challenges faced within M-Eco:

Limited User Feedback Available for the Recommender System. Learning to Rank and Recommender Systems approaches have been successfully applied to address the growing problem of information overload in a broad range of domains, for instance, web search, music, news media, movies, and collaborative annotation [Ricci et al. (2011)]. Such approaches usually build models offline, in a batch mode, and rely upon abundant user interactions and/or the availability of explicit feedback (e.g., ratings, likes, dislikes). In the case of EI, experts’ interactions and explicit feedback are scarce, which makes it harder to build effective models for ranking or recommendation.

Dynamic Nature of Twitter. The real-time nature of Twitter, on the one hand, makes it attractive for public health surveillance; yet, on the other, the volume of tweets also makes it harder to: 1) capture the information transmitted, 2) compute sophisticated models on large pieces of the input, and 3) store the input data, which can be significantly larger than the algorithm’s available memory [Muthukrishnan (2005)]. One major challenge in monitoring Twitter for EI lies in capturing the dynamics of an outgoing outbreak, without which the time-sensitive intelligence for threat assessment would be rendered useless.

Our PTR4EI extends a learning to rank framework [Liu (2009)] by considering a personalized setting that exploits a user’s individual context. We consider such context as implicit criteria for selecting tweets of potential relevance and for guiding the recommendation process. The user context $C_{u}$ is defined as a triple $C_{u}=(t,MC_{u},L_{u})$, where $t$ is a discrete Time interval, $MC_{u}$ the set of Medical Conditions, and $L_{u}$ the set of Locations of user interest.

We consider an initial context $C_{u}$ specified explicitly by the user, which can be precise, but static and limited to the medical conditions or locations manually included by the user. Our goal is to automatically capture the dynamics of the outbreak as reported in Twitter. To this end, we expand the user context by including additional medical conditions and locations related to the ones she specified, and exploit the resulting and richer context for personalized ranking. Our approach expands the user context by using 1) latent topics computed with LDA [Blei et al. (2003)] based on an indexed collection of tweets for epidemic intelligence; and 2) hash-tags that co-occur with the initial context.

We use the terms in the expanded context that correspond to medical conditions, locations, and complementary context⁷⁷7Complementary Context corresponds to the set of nouns, which are neither Locations nor Medical Conditions. It may include named entities such as names of persons, organizations, affected organisms, expressions of time, quantities, etc. to build a set of tweets by querying our collection, which correspond to a subset of Tweets output by Stage II. This step helps us to filter irrelevant tweets for the user context.

Next, we elicit judgments from experts on a subset of the tweets retrieved in order to build a ranking function model. We then obtain for each labeled tweet a feature vector that help us training our personalized ranking function. Finally, we use the ranking function to rank new incoming tweets automatically. Please refer to [Diaz-Aviles et al. (2012a)] and [Diaz-Aviles et al. (2012b)] for further details.

In this section we review the experimental evaluation of our approach on the EHEC outbreak in Germany, 2011, as the real-world event of interest, and discuss the results we obtained.

To support users in the assessment and analysis during the German EHEC outbreak, we got an alert from Stage II starting on 2011-05-23. We monitored related tweets up to 2011-06-19. In this way, we are taking into account the main period of the outbreak,⁸⁸8Note that even though the main period of the outbreak is considered for the evaluation, nothing prevents us to build the model during the ongoing outbreak, and recompute it periodically (e.g., weekly). the disease of interest, and the location.

During the period of the EHEC outbreak (May and June, 2011) a total of 7,710,231 tweets related to medical conditions were collected by the M-Eco system (Stage I); of which, 456,226 were related to the EHEC outbreak in Germany. Individual judgment were solicited to three experts on a subset of 240 of these tweets. The experts were asked to provide a relevance judgment as to whether the tweet was relevant in supporting their analysis of the outbreak, or not; disagreement in the assigned relevance scores were resolved by majority voting.

For each tweet, we prepared five binary features: $F_{MC}$, $F_{L}$, $F_{\text{\#-tag}}$, $F_{CC}$, and $F_{URL}$. We set the corresponding feature value equal to true if a medical condition, location, hash-tag, complementary context term, or URL were present in the tweet, and false otherwise. For learning the ranking function, we used the Stochastic Pairwise Descent algorithm [Sculley (2010)].

We compared our approach that expands the user context with latent topics and social generated hash-tags, against three ranking methods:

We randomly split the dataset into 80% training tweets, which will be used to compute the ranking function, and 20% testing tweets. To reduce variability, we performed the experiment using a cross-validation with 10 different 80/20 partitions. The test set is used to evaluate the ranking methods. The reported performance is the average over the 10 rounds. For evaluation, we used precision at position $n$ ($P@n$) [Baeza-Yates and Ribeiro-Neto (2011)].

The ranking performance in terms of precision is presented in Figure 7. As can be seen, PTR4EI outperforms the three baselines. Local information helps RankMCL to beat RankMC. PTR4EI, besides local features, exploits complementary context information and particular Twitter features, such as the presence of hash-tags or URLs in the tweets, this information allows it to improve its ranking performance even further, reaching a P@10 of 96%. A similar behavior is observed for MAP and NDCGL [Diaz-Aviles et al. (2012a)].

After the detection of the outbreak, authorities investigating the cause and impact of the outbreak are interested in the analysis of micro-blog data related to the event. Millions of health-related tweets are produced every day, which make this task overwhelming for the experts. Yet, our approach, PTR4EI demonstrated a superior ranking performance and was able to provide users with a personalized short list of tweets that met the context of their investigation. PTR4EI exploits features that go beyond the medical condition and location (i.e., user context) and includes complementary context information, extracted using LDA and the social hash-tagging behavior in Twitter.

The main advantage of PTR4EI is that it can discover new relationships from the dynamic data stream based on a limited context in order to help filtering the large amount of data.

The method presented here requires labeled data to train the model; placing extra effort and burden on experts. We are currently considering crowdsourcing as a complementary means to obtain labeled data on demand, similar to the crowdsourcing approach discussed in Stage I (Section 2.2).