5
• TP: true positive where a pixel of a class of interest
(a rock pixel) is correctly classified as pixel of rock
region.
• TN: true negative where a pixel of a non-interest
class (a non-rock pixel) is classified correctly.
• FP: false positive where a pixel of a non-interest class
is incorrectly classified as pixel of a class of interest.
In autonomous roof bolting, a pixel of non-rock
region is classified as a pixel of a rock region.
• FN: false negative where a pixel of an interest class is
incorrectly classified as a pixel of a non-interest class.
It is desired to increase the true positive and true negative
metrics while it is required to reduce the false positive and
false negative metrics. The confusion matrix is used to show
these four variables.
From these four measurements, four metrics can be
calculated to evaluate the performance of a deep learning
model which are accuracy, precision, recall, and F1 score
which are given by equations 2–5 [30].
TP TN FP FN
TP TN Accuracy =+++
+(2)
TP FP
TP Precision =+(3)
TP
TP
FN Recall =+(4)
F1 score 2 *Precision Recall
Precision *Recall =+(5)
Each of these four metrics provides specific insights
into the performance of the deep learning model. For
example, the accuracy metric shows the correct prediction
among all classes. The precision metric indicates the correct
prediction of the positive class (rock) and used to minimize
the false positive while the recall metric indicate how the
machine learning model correctly identifies the true posi-
tive from all the actual positive samples and used to mini-
mize the false negative. The F1 score balances between the
precision and recall.
RESULTS
A. Model Performance
Figure 6 shows the performance of the deep learning based
binary semantic segmentation during the training and vali-
dation in terms of loss while Figure 7 shows the perfor-
mance in therms of accuracy.
B. Model Testing
The model has been tested on color images from a tradi-
tional camera and also on accumulated images from an
event camera.
DISCUSSION
From the results of the proposed model, the proposed
deep learning based semantic segmentation model affords
acceptable results for segmenting the input images into two
regions (binary segmentation) rock regions and non-rock
regions. As it can be seen from Figures 7 and 8, the learning
curves are smooth after seven training epochs. The train-
ing accuracy and validation accuracy of the proposed model
are around 98% at a relatively small epoch 14 as shown in
Figure 7 which means non-rock objects.
that the model could learn fast. From Figure 8, the
model could provide high performance when it has been
tested on new color images. The obtained evaluation met-
rics are:
• Accuracy =97.36%
• Precision =99.55%
• Recall =93.36%
Figure 5. Cameras Setup for Collecting Real Data. Two
Event Cameras and L515 LiDAR Camera.
Table 1. Deep learning Model Hyper-Parametrs
Model Hyper-Parameters Value
Image Size 256×256
No. Classes 2
No. Epochs 15
batch_size 16
Learning Rate 10−4
Buffer Size 1,000
Training Data 75%
Validation Data 15%
Testing Data 10%
• TP: true positive where a pixel of a class of interest
(a rock pixel) is correctly classified as pixel of rock
region.
• TN: true negative where a pixel of a non-interest
class (a non-rock pixel) is classified correctly.
• FP: false positive where a pixel of a non-interest class
is incorrectly classified as pixel of a class of interest.
In autonomous roof bolting, a pixel of non-rock
region is classified as a pixel of a rock region.
• FN: false negative where a pixel of an interest class is
incorrectly classified as a pixel of a non-interest class.
It is desired to increase the true positive and true negative
metrics while it is required to reduce the false positive and
false negative metrics. The confusion matrix is used to show
these four variables.
From these four measurements, four metrics can be
calculated to evaluate the performance of a deep learning
model which are accuracy, precision, recall, and F1 score
which are given by equations 2–5 [30].
TP TN FP FN
TP TN Accuracy =+++
+(2)
TP FP
TP Precision =+(3)
TP
TP
FN Recall =+(4)
F1 score 2 *Precision Recall
Precision *Recall =+(5)
Each of these four metrics provides specific insights
into the performance of the deep learning model. For
example, the accuracy metric shows the correct prediction
among all classes. The precision metric indicates the correct
prediction of the positive class (rock) and used to minimize
the false positive while the recall metric indicate how the
machine learning model correctly identifies the true posi-
tive from all the actual positive samples and used to mini-
mize the false negative. The F1 score balances between the
precision and recall.
RESULTS
A. Model Performance
Figure 6 shows the performance of the deep learning based
binary semantic segmentation during the training and vali-
dation in terms of loss while Figure 7 shows the perfor-
mance in therms of accuracy.
B. Model Testing
The model has been tested on color images from a tradi-
tional camera and also on accumulated images from an
event camera.
DISCUSSION
From the results of the proposed model, the proposed
deep learning based semantic segmentation model affords
acceptable results for segmenting the input images into two
regions (binary segmentation) rock regions and non-rock
regions. As it can be seen from Figures 7 and 8, the learning
curves are smooth after seven training epochs. The train-
ing accuracy and validation accuracy of the proposed model
are around 98% at a relatively small epoch 14 as shown in
Figure 7 which means non-rock objects.
that the model could learn fast. From Figure 8, the
model could provide high performance when it has been
tested on new color images. The obtained evaluation met-
rics are:
• Accuracy =97.36%
• Precision =99.55%
• Recall =93.36%
Figure 5. Cameras Setup for Collecting Real Data. Two
Event Cameras and L515 LiDAR Camera.
Table 1. Deep learning Model Hyper-Parametrs
Model Hyper-Parameters Value
Image Size 256×256
No. Classes 2
No. Epochs 15
batch_size 16
Learning Rate 10−4
Buffer Size 1,000
Training Data 75%
Validation Data 15%
Testing Data 10%