PEAK: Pyramid Evaluation via Automated Knowledge Extraction




1. Introduction

This package provides a Java implementation of PEAK, the first tool to automatically generate the pyramid models that enables the whole automatical assessment of both human and machine summaries. For more details about the approach please refer to our paper:

PEAK: Pyramid Evaluation via Automated Knowledge Extraction. Online
Qian Yang, Rebecca J. Passonneau, Gerard de Melo.
In: Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI 2016). AAAI Press.

2. Installation

After installing Eclipse or another Integrated Development Environment (IDE), you can new a java project, for instance, with the name "PEAK". Then please put the source codes into your "workspace/PEAK/src". You may need ADW when running our source codes. If you are using Eclipse, you can download the jar package of ADW, and add it to your project as external jars, together with other supported files of ADW. To save your time, here are all the external JARs including ADW ones. You need to add them following the steps below.



For the supported files of ADW, you need to add them into your "workspace/PEAK" as follows.


3. Requirement

Java 8 (JRE 1.8) or higher

4. Quick Start

There are different aims to run PEAK. Model 1: Given model summaries, generate the pyramid; Model 2: Given a pyramid it already generated, score new summaries; Model 3: Given model summaries and summaries to score, generate the pyramid and the scores.

Here are the sample dataset including input and output. You can add it into "workspace/PEAK" as follows.


When you run "model1.java" in the picture below for Model 1, you can generate the pyramid given model summaries. Model 1 is supported by 11 small codes (from "step1.java" to "step11.java") and each has its own unique purpose. That is, the final result of "step11.java" is the pyramid, and all the other results of the previous steps are the temp results needed for generating the pyramid. "model2.java" is for Model 2, i.e., scoring new summaries given a pyramid you already generated. And "model3.java" is for Model 3.


Please note that, for other dataset that you want to use, you should replace the names of input files (model summaries and students’ summaries) according to your dataset. Here are four places that need your revisions: "step1.java", "step6.java", "step7.java", "step12.java".

Please also note that, you may need ClausIE to pre-process your input files. Here are the tutorial and the source code and binaries of ClausIE.

5. License

PEAK: Pyramid Evaluation via Automated Knowledge Extraction
Copyright (c) 2015 Qian Yang. All Rights Reserved.

If you use this system, please cite the following paper:

PEAK: Pyramid Evaluation via Automated Knowledge Extraction. BiBTeX
Qian Yang, Rebecca J. Passonneau, Gerard de Melo.
In: Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI 2016). AAAI Press.

For more information or bug reports please contact: laraqianyang[at]gmail[dot]com.

Version 2.0 -- February 8, 2016.