Dr Kim Luyckx is the author of 'Fine-Grained Emotion Detection in Suicide Notes: A Thresholding Approach to Multi-Label Classification', recently published in the Biomedical Informatics Insights Supplement.  We asked Dr Luyckx to tell us about the background of her paper.

To start please tell us about the challenge this year.  Why did you decide to become involved, and what goals did you and/or your team expect to accomplish?

Track 2 of the 2011 Medical NLP Challenge involved automatically classifying sentences from suicide notes according to the emotions expressed. This task struck us as particularly interesting as it was closely related to a number of projects we are involved in within the CLiPS Computational Linguistics Group at the University of Antwerp (Belgium).

Our domains of interest include techniques for authorship attribution on a large scale, emotion classification, and gender and age prediction on chat data. Techniques from any of these domains can easily be adapted and applied to suicide notes. As we usually do our research on an individual basis, the challenge was an excellent opportunity to focus our combined skills and expertise on a well-defined problem.

While overlapping interests and collaboration were our primary driving factors, the dataset was a secondary motivator. Emotion classification of text is a tricky task, since it is often hard – even for humans – to agree on the emotional contents of a piece of writing. The data set offered in the challenge, however, seemed like it might avoid these agreement problems for two reasons. First of all, we expected emotions to manifest themselves strongly in suicide notes, as taking the radical decision of ending one's own life is inspired by a lot of emotional turmoil. Secondly, the suicide notes had been annotated by clinicians, experts in the study of suicidal behavior. We trusted that they would find it easier to identify (and agree on) the emotions expressed in suicide notes.

By participating in the Medical NLP Challenge, we first of all hoped to learn from each other and from other participants. We also hoped to evaluate the techniques we use for emotion classification on this new benchmark data set.

In writing this paper what were the particular challenges you faced?  How did you overcome these challenges?

It turned out we were a bit optimistic concerning the clarity of emotions and the annotation agreement in the data set. Emotion classification is still a tricky task, even in such high-intensity material as suicide notes, and even for highly qualified annotators.

Inter-annotator agreement was modest, and due to a small decision, emotion-free sentences as well as sentences for which no agreement had been reached were grouped in the same subset. While this course of action may have made sense for clinical applications, it seemed less logical from a classification or machine learning point of view. As a result of this small decision, it was impossible for participants to discern training sentences truly devoid of emotion, turning a potentially valuable “no emotion” class with negative instances into a noisy “unannotated” class.

We attempted to solve this problem by training one of our systems on emotion-carrying sentences only, consequently discarding a large part of the training data. The problem with this approach was that the data set was already quite small and discarding half of it had a significant impact on classification performance. Systems that did use the unannotated sentences consistently outperformed the classifiers trained on emotion-carrying sentences only.

An alternative solution was to manually re-assign an emotion label to a sentence segment instead of assigning multiple labels to an entire sentence. Only when we agreed on the link between a label and a sentence segment, we assigned that label. We did not alter any labels or assign any additional emotion labels. Training on re-annotated data significantly increased our performance.

Another challenge was the fact that each sentence potentially carries multiple emotion labels. Due to the small size of the data set, predicting combined labels would lead to an even more substantial sparseness problem, so we had to think of a way to assign multiple labels to the same instance based on single-label predictions. We adopted the intuitive technique of thresholding, only including emotion labels if their probability exceeded an experimentally determined threshold.

Of course, the difficulties we encountered were not only technical. When you work in teams, things are bound to get a bit more complex. Each of us had different programming styles and different scripts that had to be adapted to interact with each other smoothly. The consequence of this time-consuming process was that we did not have the time to implement some of the more complex techniques we had hoped to evaluate.

As many of the articles appearing in the supplement are quick to acknowledge, suicide is a distressingly common cause of death particularly among younger people.  Has this work changed your view of suicide: do you find yourself more or less understanding or sympathetic of people who commit suicide and those they leave behind?

I think we mostly realized how diverse people's state of mind can be when contemplating suicide. There were of course people who seemed unhappy, angry, or desperate, but more suprisingly, others remained extremely composed, listing all the practical things that would need to be taken care of after their death. The range of emotions expressed is not restricted to negative feelings either, as most notes also contained expressions of love and happiness.

As for our understanding or sympathy towards people who commit suicide, we don't believe our feelings have changed much. People who are driven to such a definitive act must have withstood a lot of pain and pressure, and it is not up to us to judge on whether they should or could have made different decisions.

We do believe that some people can be helped and it is possible to show them that the road they are walking is not necessarily a dead-end. We hope that our contributions to the Medical NLP Challenge will help make it easier to assist these people before it is too late.

View more information about Dr Kim Luyckx and her co-authors: Frederik Vaassen, Claudia Peersman, and Dr Walter Daelemans

More information about CLIPS is available here

share on