This position paper outlines the various context factors to be considered in order to develop effective methods for summarization and its evaluation. A key argument is that we cannot develop useful summarization systems unless we pay close attention to both the context (where summarization is applied), and the purpose (why is it done).
The paper analyses three key factors: (1) the input to the summarization model, (2) the purpose of the output summaries, and (3) the output format of the summaries.
A summary is loosely defined as a reductive transformation of source text through content reduction by selection and/or generalization on what is important in the source. A possible three-step model to achieve this can be:
- I : source text interpretation (to source text representation)
- T : source representation transformation (to summary text representation)
- G : summary text generation (from summary representation)
Because we have to characterize a source text as a whole, define important content which is a matter of what is essential (fundamental) as well as what is salient (highlight), and capture this important content which is a matter of both information and expression.
Work done so far on capturing important content mainly falls under two categories: text extraction and fact extraction.
| Text extraction | Fact extraction |
|---|---|
| No prior assumption is made about what is important in the source text (open approach) | To-be extracted content is predefined by a set of notions (closed approach) |
| Output closely resembles the source text in terms of structure and presentation order | Output is more biased towards the chosen facts and may involve natural language generation |
| Output is general but low-quality due to weak and indirect methods of estimating importance | Output has better quality in substance and presentation, but needs more effort to explicitly specify the required information to be captured |
They are the factors which can be operationalized for developing an application. For summarization, identifying specific context factors is crucial as the idea of a general-purpose summary is an illusion. When a range of contexts is considered, it is evident that any one summary cannot meet all the requirements. Similarly, the idea of a basic summary also makes hidden assumptions, e.g., the subject knowledge of the summary's consumers is on par with that of the source text's intended readers. Thus, what summaries should be like is defined by what they are wanted for, as well as by what their sources are like.
There are three classes of context factors: input, purpose and output. The relation between these three factors defines a summary function as: given input factor data, satisfy purpose factor requirements, via output factor devices.
- Input factors deal with the form, subject and the number of units provided to the summarization system.
- Purpose factors comprise the different use-cases of the summaries such as where will they be used, by whom, and for what.
- Output factors deal with how much information should be covered by the summaries, their format and writing style.
It is impossible to properly evaluate the summaries without knowing their purpose. Conventionally, system summaries are either evaluated by comparing with their source texts, or human written summaries (a.k.a ground-truth/gold standard/references) of the same source text. Comparison with source text aims to answer the question: has the summary captured the important concepts and their relations in the source? Evidently, answering this question is highly subjective dependent on who is evaluating the summary. Given the difference in background knowledge among people, it is certainly possible that two people disagree on what is important in the source text.
Comparing with a human written summary is likewise influenced by the same subjectivity in that a summary agreed upon by a large group of people to be high-quality/ideal does not exist. Thus, it is suggested to collect multiple human summaries as references for comparing the system summaries which is evidently resource expensive. More importantly, the definition of a summary need not always be stated as a condensation of the source text but rather consider and clarify the purpose factors of the summarization system for a reliable evaluation.
Future research must focus on developing techniques for better representation of the source text (intermediate processing), taking source text analysis as far as possible on a linguistic basis, without a reference domain model. This shallow representation can be seen as a sequence of sentence representations which can then be linked by common elements leading to a better integrated representation that can be used for creating better summaries.
