Scoring Software    
Description
The scoring software ('region_word_score') was implemented in C. It generates semantic scores to evaluate your region labeling algorithms based on a human labeled data set. Two evaluation methods, the semantic range oriented scoring and the frequency correct oriented scoring, are currently supported (see our paper for details).
Download
Executable files under different systems are available below,

How to Use it?
To use the software, please read the following instructions carefully,
  • Provide a directory with the following input files,
    1. words ---- the vocabulary words from the training data. Each line in the file is one word, sensed or not sensed. The sense of the word, if available, is parenthesized following the word. For example, boy(2).
    2. human_label --- includes a list of file names. Each file contains the ground truth labels for the ground truth regions segmented by human. A region can have multiple labels separated by ':'. An example can be found HERE.
    3. human_segmentation --- includes a list of file names. Each file provides a human segmentation mask for a ground truth image. An example can be found HERE.
    4. machine_segmentation --- includes a list of file names. Each file specifies for an image segmentation masks generated by machine. The segments should start from 1.
    5. machine_segmentation_index--- This file provides the indices of the machine segments, ranked by their size (# of pixels). We use this to drop some segments/regions that are too tiny to evaluate.
  • Specify parameters in a script file. See the example for details.
Now you are ready to run the program. Just do
   region_word_score < [script file]

The example included in the download may further clarify your questions regarding the program input.
What's the Output?
Depending on the evaluation method you specify, a sub-directory called range or freq will be created in the output directory. Within the directory, there are 5 files generated by the program,
  • xxx-semantic-score-mvp----This file includes a scoring matrix for each imagex ('xxx' is either 'range' or 'freq'). The matrix has one row per image region and one column for each vocabulary word. The matrix entries are the scores deserved for predicting that word for that region.
  • xxx-semantic-score-map-mp---- This intermediate score matrix provides a score for each vocabulary word with each ground truth word for a particular image, independent of segmentation.
  • word-base-score----The base score for each word computed from the semantic trees.
  • semantic-tree ---- The semantic tree for each word. (Not friendly readable)
  • semantic-tree.dot ---- An input file for the dot tool. With this, you can generate a nice graph to visualize the semantic relationship between the words in the vocabulary.