File: nedc_eval_eeg_v5.0.0/_AAREADME.txt
Package: nedc_eval_eeg
Version: 5.0.0

-------------------------------------------------------------------------------
Change Log:

(20220418) (JP): updated support for the ann tools library
(20200817) (LV): updated for new version of nedc_eval_eeg
(20200515) (JP): added check for duplicate and overlapping hypotheses
(20200417) (JP): made optional parameters work properly
(20200328) (JP): initial version
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Summary:
-------------------------------------------------------------------------------

This package contains the standard version of our scoring
software. v5.0.0 includes support for csv and xml files that contain
annotation information. Note that starting with v4.0.0, NIST integration
was obsoleted as it became too difficult for people to install and manage
the NIST scoring software.

This software is designed to score system output from machine learning
systems that operate on sequential data. To learn more about the
theory behind how we score seizure detection systems, please refer to
this document:

 V. Shah, M. Golmohammadi, I. Obeid, and J. Picone, “Objective
 Evaluation Metrics for Automatic Classification of EEG Events,” in
 Signal Processing in Medicine and Biology: Emerging Trends in
 Research and Applications, 1st ed., I. Obeid, I. Selesnick, and
 J. Picone, Eds. New York City, New York, USA: Springer, 2021,
 pp. 1–26.
 URL: https://www.isip.piconepress.com/publications/unpublished/book_sections/2021/springer/metrics/

This software can be retrieved from the following URL:

 https://www.isip.piconepress.com/projects/tuh_eeg/downloads/nedc_eval_eeg/

You will be prompted for a username and password that you received when you
registered on our web site. If you have not registered, please go here:

 https://www.isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml

and follow the instructions for registration. It is an automatic process and
you will receive the username and password immediately.

To install this software, download the tar file at the above URL, and
untar it:

 tar xvfz nedc_eval_eeg.v5.0.0.tar.gz

------------------------------------------------------------------------------
Running NEDC Eval EEG:
------------------------------------------------------------------------------

This software requires Python 3 and is written entirely in Python.
The software can be run in two ways: standard (supports all
NEDC annotation file formats), competition (supports a single csv file
containing all the annotations (used for the Neureka 2020 Epilepsy Challenge).

There is no installation of this software needed. Download and unpack
the software into a directory whose full pathname is <root>.

Edit this file: <root>/bin/nedc_eval_eeg

Change this line:

 NEDC_NFC=/data/isip/tools/master/nfc/util/python/nedc_eval_eeg/v5.0.0;

to the full pathname of your installation directory.

You can, of course, put the bin directory in your path. You can get information
about the available options by executing the script with the "--help" option:

 ../../bin/nedc_eval_eeg --help

After editing the driver script, Then go to the test directory:

 cd <root>/src/test/00

and execute the following command:

 ../../bin/nedc_eval_eeg ref.list hyp.list

If it ran successfully, you will see this output:

 nedc_000_[1]: ../scripts/nedc_eval_eeg.py ref.txt hyp.txt 
  ... creating the output directory ...
  ... executing NEDC DP Alignment scoring ...
  ... executing NEDC Epoch scoring ...
  ... executing NEDC Overlap scoring ...
  ... executing NEDC Time-Aligned Event scoring ...
  ... done ...

The output scoring files will be located in a directory called output.

The file output/summary.txt contains an extremely detailed analysis of the
performance of your system. The output directory also contains per-file
output so you can better understand the performance on individual files.
All this is described in the above publication.

To confirm that	your test was successful, run the following command in the
test case directory:

 diff answers/summary.txt output/summary.txt
  
Other than a difference in some header information, these two files should
be identical. A typical output that you might see is:

 nedc_000_[1]: diff answers/summary.txt output/summary.txt 
 4,5c4,5
 <  File: answers/summary.txt
 <  Date: Mon Aug 17 13:14:48 2020
 ---
 >  File: output/summary.txt
 >  Date: Tue Aug 18 11:27:08 2020
 476c476
 < NEDC EEG Eval Successfully Completed on Mon Aug 17 13:14:49 2020
 ---
 > NEDC EEG Eval Successfully Completed on Tue Aug 18 11:27:13 2020

This means everything ran fine and the only differences are the date and
location of the file.

------------------------------------------------------------------------------
Input Files:
------------------------------------------------------------------------------

STANDARD VERSION:

 The file "ref.list" represents a list of reference annotation files. 
 The format for these files is straightforward:

 nedc_000_[1]: more data/ref/00000258_s001_t000.csv 
 # version = csv_v1.0.0
 # bname = 00000258_s001_t000
 # duration = 1750.0000 secs
 # map file: /data/isip/tools/master/nfc/lib/nedc_eas_default_montage.txt
 # annotation label file: /data/isip/tools/master/nfc/lib/default_map.txt
 #
 channel,start_time,stop_time,label,confidence
 TERM,0.0000,14.3320,bckg,1.0000
 TERM,14.3320,163.0320,seiz,1.0000
 TERM,163.0320,251.9720,bckg,1.0000
 TERM,251.9720,317.4720,seiz,1.0000
 TERM,317.4720,481.7640,bckg,1.0000
 TERM,481.7640,598.1640,seiz,1.0000
 TERM,598.1640,710.8280,bckg,1.0000
 TERM,710.8280,878.3280,seiz,1.0000
 TERM,878.3280,1077.0480,bckg,1.0000
 TERM,1077.0480,1315.0480,seiz,1.0000
 TERM,1315.0480,1590.5840,bckg,1.0000
 TERM,1590.5840,1679.8840,seiz,1.0000
 TERM,1679.8840,1750.0000,bckg,1.0000

 "TERM" can be ignored - it refers to the type of annotation (TERM means
 the annotation applies to all channels).

 The next two fields are the start and stop time in seconds of the event. The
 next field is the event label followed by a confidence.

 The hypothesis files listed in hyp.list have the same format, but the
 information contained in them should be generated by your system that you
 are evaluating.

 Note that with v5.0.0 you only need to provide non-background events. We
 fill in the gaps with an event designated as background.

COMPETITION VERSION:

 You can also store all your hypotheses in one file. This is what we call
 the competition format. This was used in the Neureka 2020 Epilepsy Challenge.
 The single reference file contains entries of the form:
  
  00000258_s002_t000  0.0000 20.0000 bckg 1.0000
  00000258_s002_t002  0.0000 15.9375 bckg 1.0000
  00000258_s002_t002  15.9375 136.4375 seiz 1.0000
  00000258_s002_t002  136.4375 272.0000 bckg 1.0000
  ...

  The tag is a unique identifier for each EEG file (the basename). The
  next two fields are the start and stop times of the event respectively.
  The next field is one of two values: "bckg" or "seiz". The latter is used
  to identify a seizure event. The last field is a confidence value.

  The hypothesis file follows the same format.

OTHER IMPORTANT NOTES:

 This scoring software can be configured to do multiclass scoring. Instead of
 having only two events (bckg and seiz), you can edit the parameter file
 in the source directory (nedc_eval_eeg_params_v00.txt) to describe whatever
 classes you wish to score.

If you have any additional comments or questions about this software,
please direct them to help@nedcdata.org. We will do our best to answer
your questions promptly.

Best regards,

Joseph Picone
