File: nedc_eeg_resnet/v1.0.0/AAREADME.txt
Tool: The NEDC EEG ResNet System
Version: 1.0.0

-------------------------------------------------------------------------------
Change Log:

20230212 (JP): final review of the new instructions
20221217 (ML): added instructions for training/decoding/evaluation
20220613 (JP): initial version

-------------------------------------------------------------------------------

This tool has been used to annotate various NEDC EEG corpora. To
learn more about the origins of the tool, please see this publication:

 Khalkhali, V., Shawki, N., Shah, V., Golmohammadi, M., Obeid, I., &
 Picone, J. (2021). Low Latency Real-Time Seizure Detection Using
 Transfer Deep Learning. In I. Obeid, I. Selesnick, & J. Picone
 (Eds.), Proceedings of the IEEE Signal Processing in Medicine and
 Biology Symposium (SPMB) (pp. 1–7). Philadelphia, Pennsylvania,
 USA. https://isip.piconepress.com/publications/conference_proceedings/
 2021/ieee_spmb/eeg_transfer_learning/.

A. WHAT'S NEW

Version 1.0.0: 
  + Deleted the pickling process
  + Reorganized the system's structure
  + Collapsed Decode and Postprocess bash script to one script

B. INSTALLATION REQUIREMENTS

Python code unfortunately often depends on a large number of add-ons, making
it very challenging to port code into new environments. This tool has been
tested extensively on Windows and Mac machines running Python v3.9.x.

Software tools required include:

 o Python 3.7.x or higher (we recommend installing Anaconda)
 o Numpy: http://www.numpy.org/
 o Torchvision: https://pytorch.org/vision/stable
 #o Pillow: https://pillow.readthedocs.io/en/stable/
 o Scikit-learn: https://scikit-learn.org/stable/

There is a requirements.txt included in the release that helps you automate
the process of updating your environment.

C. USER'S GUIDE

C.1. WINDOW USERS

For Window users, we recommend users install Anaconda in order to run
a bash emulator.

Through the Anaconda prompt, create a new environment and specify the proper
python version:

 $ conda create -n <my_environment_name> python=3.9

Install a bash emulator that will allow running the annotation tool:

 $ conda install m2-base

Install the required packages:

 $ conda install numpy
 $ conda install torchvision
 $ conda install lxml
 $ conda install scikit-learn

The easiest way to run this is to change your current working directory
to the root directory of the installation and execute the tool as follows:

 $ cd <my_install_location>/nedc_eeg_resnet/v1.0.0
 $ ./bin/nedc_eeg_resnet
 
Once the software has been installed, you need to do the following things if
you want to run this from any directory:

 - set the environment variable NEDC_NFC to the root directory
   of the installation:

    $ export NEDC_NFC='<my_install_location>/nedc_eeg_resnet/v1.0.0'

 - put $NEDC_NFC/bin in your path:

    $ export PATH=$PATH:$NEDC_NFC
 
You should be able to type:

 $ which nedc_eeg_resnet

and see the command. Then you can simply type:

 $ nedc_eeg_resnet

C.2. LINUX/MAC USERS

For Mac users, since Mac OS X 10.8 comes with Python 2.7, you may 
need to utilize pip3 when attempting to install dependencies:

 pip3 install numpy   
 pip3 install scikit-learn
 pip3 install lxml
 pip3 install torchvision

The easiest way to run this is to change your current working directory
to the root directory of the installation and execute the tool as follows:

 cd <my_install_location>/nedc_eeg_resnet/v1.0.0
 ./bin/nedc_eeg_resnet
 
Once the software has been installed, you need to do the following things if
you want to run this from any directory:

 - set the environment variable NEDC_NFC to the root directory
   of the installation (e.g., <my_install_location>/nedc_eeg_resnet/v1.0.4)

 - put $NEDC_NFC/bin in your path
 
You should be able to type:

 $ which nedc_eeg_resnet

and see the command. Then you can simply type:
  
 $ nedc_eeg_resnet


+ Note:

  If you would like to change any bash script in /bin please go to /src/shell,
  make the changes there and type 'make' to move the changed scripts to /bin  

===============================================================================
===============================================================================

These kinds of complete machine learning systems are a bit complicated because
there are a lot of files involved. We have distributed example files with
this release so that you have a working example of the complete system.
Please make sure you run the complete system before you attempt to change
the code.

-------------------------------------------------------------------------------
A. Training

If you would like to run training please follow these steps:

1: set up datasets:

  training requires a training edf and csvbi list file and a development edf
  and csvbi list file, by default these lists are:

  $NEDC_NFC/test/list/train_edf.list
  $NEDC_NFC/test/list/train_csv.list
  $NEDC_NFC/test/list/dev_edf.list
  $NEDC_NFC/test/list/dev_csv.list

  if you want to change the name/path of these files please navigate to:
  $NEDC_NFC/src/shell/nedc_eeg_resnet_env.sh

  and change the following variables:

  TRAIN_EDF_LIST="...change to wanted filename and path..."
  TRAIN_CSVBI_LIST="...change to wanted filename and path..."
  DEV_EDF_LIST="...change to wanted filename and path..."
  DEV_CSVBI_LIST="...change to wanted filename and path..."
  
2. customizing training arguments:

  for most of our training arguments we use a parameter file, the arguments in
  this file can be customized to fit your particular dataset, the file is found
  here:

  $NEDC_NFC/lib/nedc_eeg_resnet_train_param.txt
  
  please look through this file and adjust the values to your dataset/wants
  
  the output model filename by default is:

  $NEDC_NFC/models/model.pckl

  if you want to change the name/path of the output model please navigate to:

  $NEDC_NFC/src/shell/nedc_eeg_resnet_env.sh

  and change the following variable:

  MODEL_FILE="...change to wanted filename and path..."

+ Note:

  if you set the save_model_every_epoch argument in the parameter file to true
  a model will be saved every epoch named:

  $NEDC_NFC/models/model_'epoch_num'.pckl

  ex:
  after first epoch: $NEDC_NFC/models/model_1.pckl
  after second epoch: $NEDC_NFC/models/model_2.pckl

3. running training:

  training can be run using this driver bash script:
  $NEDC_NFC/bin/nedc_eeg_resnet_train

+ Note:

  we recommend running training on GPU as it is much faster.

  In order to get the best results please make sure all of your input
  files have a sample frequency that is evenly divisible by 50. For example,
  100 Hz, 150 Hz, 200 Hz, etc. work well and can be handled by this package.

  If you want to use data that does not meet this constraint, we recommend
  you use an industrial strength downsampler to either convert the data
  to 50 Hz, or convert it to a multiple of 50 Hz. For example, for 256 Hz,
  we use a high performance C++ downsampler to convert the data to 250 Hz,
  and then use this package on the 250 Hz data. (Most of our technology
  is based on 250 Hz data.)
  
-------------------------------------------------------------------------------
B. Decoding

If you would like to run decoding please follow these steps:

1: set up datasets:

  decoding can be run on any combination of individual edf files or lists of
  edf files, by default decoding will be run on:

  $NEDC_NFC/test/list/train_edf.list  
  $NEDC_NFC/test/list/dev_edf.list
  $NEDC_NFC/test/list/eval_edf.list

  if you want to change the name/path of these files please navigate to:

  $NEDC_NFC/src/shell/nedc_eeg_resnet_env.sh

  and change the following variables:

  TRAIN_EDF_LIST="...change to wanted filename and path..."
  DEV_EDF_LIST="...change to wanted filename and path..."
  EVAL_EDF_LIST="...change to wanted filename and path..."

  you can change these variables to have multiple edf files or edf list files
  as well for example:

  TRAIN_EDF_LIST="$NEDC_NFC/test/list/edf.list $NEDC_NFC/test/list/my_edf.edf"
  DEV_EDF_LIST="/data/edf_files/test1.edf /data/edf_files/test2.edf"

  if you want to customize which datasets you run decoding on please navigate
  to the following file:

  $NEDC_NFC/src/shell/nedc_eeg_resnet_decode.sh

  locate the following lines of code:

  # decode the training dataset
  #
  echo "==== Decoding the Train Dataset ===="
  echo "Decoding the Train Dataset Started on: $(date "+%D %T")"
  $TRAIN_CMD
  echo "Decoding the Train Dataset Finished on: $(date "+%D %T")"

  # decode the dev dataset
  #
  echo "==== Decoding the Dev Dataset ===="
  echo "Decoding the Dev Dataset Started on: $(date "+%D %T")"
  $DEV_CMD
  echo "Decoding the Dev Dataset Finished on: $(date "+%D %T")"

  # decode the eval dataset
  #
  echo "==== Decoding the Eval Dataset ===="
  echo "Decoding the Eval Dataset Started on: $(date "+%D %T")"
  $EVAL_CMD
  echo "Decoding the Eval Dataset Finished on: $(date "+%D %T")"

  the following lines of code can be commented out by typing '#' in front, so
  that the command won't be executed, for example if you want to only decode
  the train dataset, you might do the following:

  # decode the training dataset
  #
  echo "==== Decoding the Train Dataset ===="
  echo "Decoding the Train Dataset Started on: $(date "+%D %T")"
  $TRAIN_CMD
  echo "Decoding the Train Dataset Finished on: $(date "+%D %T")"

  # decode the dev dataset
  #
  #echo "==== Decoding the Dev Dataset ===="
  #echo "Decoding the Dev Dataset Started on: $(date "+%D %T")"
  #$DEV_CMD
  #echo "Decoding the Dev Dataset Finished on: $(date "+%D %T")"

  # decode the eval dataset
  #
  #echo "==== Decoding the Eval Dataset ===="
  #echo "Decoding the Eval Dataset Started on: $(date "+%D %T")"
  #$EVAL_CMD
  #echo "Decoding the Eval Dataset Finished on: $(date "+%D %T")"

  to reverse this just simply delete all '#' added.

2. customizing decoding arguments:

  for most of our decoding arguments we use a parameter file, the arguments in
  this file can be customized to fit your particular dataset, the file is found
  here:

  $NEDC_NFC/lib/nedc_eeg_resnet_decode_param.txt
  
  please look through this file and adjust the values to your dataset/wants
  
  the input model filename by default is:

  $NEDC_NFC/models/model.pckl

  if you want to change the name/path of the model please navigate to:

  $NEDC_NFC/src/shell/nedc_eeg_resnet_env.sh

  and change the following variable:

  MODEL_FILE="...change to wanted filename and path..."  

  decoding produces a csvbi output file for every edf input file, the output
  directory for the csvbi files by default is:

  $NEDC_NFC/test/output
  
  if you want to change the output directories please navigate to:

  $NEDC_NFC/src/shell/nedc_eeg_resnet_env.sh                          
                                                                    
  and change the following variable:

  OUTPUT_DIR="...change to wanted path..."

  if you want to customize the output directories based on the dataset please
  navigate to:

  $NEDC_NFC/src/shell/nedc_eeg_resnet_decode.sh

  and change the following variables:

  TRAIN_ODIR="...change to wanted path..."
  DEV_ODIR="...change to wanted path..."
  EVAL_ODIR="...change to wanted path..."

  (see the note below for a description of rdir)
  by default, rdir is set as:

  $NEDC_NFC/test/data
  
  if you want to change rdir please navigate to:
  $NEDC_NFC/src/shell/nedc_eeg_resnet_decode.sh                          
                                                                    
  and change the following variable:
  RDIR="...change to wanted path..."

+ Note:

  For decoding it is important that the output csv_bi files are able to save
  your system's way of organizing your files, we use the argument rdir
  (replace directory) so that you can specify a common directory path
  present in all argument files that can be removed. For example if you
  have the following argument files:

  /home/files/edf/train/trainset1/0001.edf
  /home/files/edf/train/trainset1/0002.edf
  /home/files/edf/dev/devset4/0001.edf
  /home/files/edf/dev/devset4/0002.edf
  /home/files/edf/eval/evalset2/0001.edf
  /home/files/edf/eval/evalset2/0002.edf

  you might set the rdir argument to /home/files/edf that way the output csv_bi
  files retain all organizational directories (trainset1, evalset2, etc.), this
  is what the output files would be with this specific rdir argument:

  $NEDC_NFC/test/output/train/trainset1/0001.csv_bi
  $NEDC_NFC/test/output/train/trainset1/0002.csv_bi
  $NEDC_NFC/test/output/dev/devset4/0001.csv_bi
  $NEDC_NFC/test/output/dev/devset4/0002.csv_bi
  $NEDC_NFC/test/output/eval/evalset2/0001.csv_bi
  $NEDC_NFC/test/output/eval/evalset2/0002.csv_bi

  if rdir is not a valid filepath, by default it won't save any input file
  directories.
  
3. running decoding:

  decoding can be run using this driver bash script:
  $NEDC_NFC/bin/nedc_eeg_resnet_decode

+ Note:

  we recommend running decoding on CPU as running it on GPU will not make
  it run any faster
  
+ Note:

  In order to get the best results please make sure all of your input
  files have a frequency that is divisible by 50, for example, 100 Hz,
  150 Hz, 200 Hz, etc. See the comments about this under section A.

-------------------------------------------------------------------------------
C. Evaluation

If you would like to run evaluation please follow these steps:

1: set up datasets:

  evaluation needs a reference csvbi file (file with correct annotation) for
  each hypothesis csvbi file (file created from decoding that contains the
  model's annotation), by default the reference csvbi list files are:

  $NEDC_NFC/test/list/train_csv.list  
  $NEDC_NFC/test/list/dev_csv.list
  $NEDC_NFC/test/list/eval_csv.list

  if you want to change the name/path of these files please navigate to:
  
  $NEDC_NFC/src/shell/nedc_eeg_resnet_env.sh

  and change the following variables:
  
  TRAIN_CSVBI_LIST="...change to wanted filename and path..."
  DEV_CSVBI_LIST="...change to wanted filename and path..."
  EVAL_CSVBI_LIST="...change to wanted filename and path..."

  in order to set the hypothesis csvbi files a directory leading to these files
  must be specified, by default they are the decoding output directories:

  $NEDC_NFC/test/output/train
  $NEDC_NFC/test/output/dev
  $NEDC_NFC/test/output/eval

  if you want to change the name/path of these directories please navigate to:
  
  $NEDC_NFC/src/shell/nedc_eeg_score.sh
  
  and change the following variables:
  
  TRAIN_DECODE_OUT_DIR="...change to wanted path..."
  DEV_DECODE_OUT_DIR="...change to wanted path..."
  EVAL_DECODE_OUT_DIR="...change to wanted path..."

  if you want to customize which datasets you run evaluation on please navigate
  to the following file:
  
  $NEDC_NFC/src/shell/nedc_eeg_score.sh

  locate the following lines of code:

  # run scoring on the train dataset
  #
  echo "==== Scoring the Train Dataset ===="
  echo "Scoring the Train Dataset Started on: $(date "+%D %T")"
  $TRAIN_CMD
  echo "Scoring the Train Dataset Finished on: $(date "+%D %T")"

  # run scoring on the dev dataset
  #
  echo "==== Scoring the Dev Dataset ===="
  echo "Scoring the Dev Dataset Started on: $(date "+%D %T")"
  $DEV_CMD
  echo "Scoring the Dev Dataset Finished on: $(date "+%D %T")"

  # run scoring on the eval dataset
  #
  echo "==== Scoring the Eval Dataset ===="
  echo "Scoring the Eval Dataset Started on: $(date "+%D %T")"	
  $EVAL_CMD
  echo "Scoring the Eval Dataset Finished on: $(date "+%D %T")"

  the following lines of code can be commented out by typing '#' in front, so
  that the command won't be executed, for example if you want to only evaluate
  the eval dataset, you might do the following:

  # run scoring on the train dataset
  #
  #echo "==== Scoring the Train Dataset ===="
  #echo "Scoring the Train Dataset Started on: $(date "+%D %T")"
  #$TRAIN_CMD
  #echo "Scoring the Train Dataset Finished on: $(date "+%D %T")"

  # run scoring on the dev dataset
  #
  #echo "==== Scoring the Dev Dataset ===="
  #echo "Scoring the Dev Dataset Started on: $(date "+%D %T")"
  #$DEV_CMD
  #echo "Scoring the Dev Dataset Finished on: $(date "+%D %T")"

  # run scoring on the eval dataset
  #
  echo "==== Scoring the Eval Dataset ===="
  echo "Scoring the Eval Dataset Started on: $(date "+%D %T")"	
  $EVAL_CMD
  echo "Scoring the Eval Dataset Finished on: $(date "+%D %T")"

  to reverse this just simply delete all '#' added.

2. customizing evaluation arguments:

  for most of our evaluation arguments we use a parameter file, the
  arguments in this file can be customized to fit your particular
  dataset, the file is found here:
  
  $NEDC_NFC/lib/nedc_eval_eeg_params_v00.txt
  
  please look through this file and adjust the values to your dataset/wants
  
  evaluation by default produces 4 evaluation files based on the 4 evaluation
  methods (dpalign, epoch, overlap, and taes) and a general summary file, the
  output of these files by default is:
  
  $NEDC_NFC/test/output/train/report
  $NEDC_NFC/test/output/dev/report
  $NEDC_NFC/test/output/eval/report
  
  if you want to change the output directory of the evaluation files please
  navigate to:
  
  $NEDC_NFC/src/shell/nedc_eeg_score.sh

  and change the following variables:
  
  TRAIN_OUT_DIR="...change to wanted path..."
  DEV_OUT_DIR="...change to wanted path..."
  EVAL_OUT_DIR="...change to wanted path..."

3. running evaluation:

  evaluation can be run using this driver bash script:
  $NEDC_NFC/bin/nedc_eeg_score

-------------------------------------------------------------------------------
If you have any additional comments or questions about the data or software,
please direct them to help@nedcdata.org. We will do our best to answer them.
However, this distribution is fairly complex, so familiarity with Linux, shell,
and Python is required.

Best regards,

Joe Picone
