1
24-050
Hierarchical Training Pipeline for Event-Based Robotic
Perception Models for Autonomous Roof Bolting
Rik Banerjee
M3 Robotics Lab, Colorado School of Mines
Golden CO, USA
Akram Marseet
M3 Robotics Lab, Colorado School of Mines
Golden CO, USA
Andrew J. Petruska
M3 Robotics Lab, Colorado School of Mines
Golden CO, USA
ABSTRACT
Event cameras are used for their performance in high
dynamic-range lighting conditions which are canonical to
active mining environments. Direct labeling of event-based
image data to train a model to perform semantic segmenta-
tion using traditional methods is slow and error-prone. This
study proposes a framework to use roughly hand-labeled
color images from a mine as an input to an intermediary
probabilistic algorithm called alphamatting to generate a
ground-truth data set. These high-fidelity labels can be used
to train a semantic segmentation model to differentiate the
support strap from the roof. This model can then be lever-
aged to segment an event-based scene to enable autono-
mous roof bolting. This pipeline has been shown to achieve
an accuracy of 88% with a false positive rate of 3%.
INTRODUCTION
Roof bolting is commonly regarded as one of the riskiest
occupations in the United States [1]–[4]. This arises from
several factors: the operator faces the danger of sustaining
injuries from the bolting machine [5], and there exists a risk
of becoming a victim of a roof collapse [6]. Furthermore,
a significant portion of accidents occurs when operators
with less than one year of drilling experience are engaged
in bolting. Enhancing miner safety and productivity
could be achieved by relieving operators of the need to be
This work is supported by National Institute for Occupational
Safety &Health |NIOSH/Project 75D30121C12206.
underground to identify roof drilling areas, position, and
operate the drill. The low-light conditions in underground
mines have been studied to have a measurable adverse effect
on the health of the miners [7]. A robotic solution can be
used to automate this process.
As can be seen in Figure 1, automating this process
faces a two-fold challenge. Firstly, the autonomous system
needs to classify the visual scene into Rock and Not Rock.
This can be achieved by a process called semantic segmenta-
tion [8]. Secondly, a three-dimensional (3D) representation
of the surface needs to be computed by using either stereo-
vision or a depth sensor. This paper will present an archi-
tecture to solve the issue of semantically recognizing rock
in an underground mine during active drilling operations.
This means that the system needs to withstand vibrational
loads, dust clouds, and challenging lighting conditions [7].
These elements pose challenges for implementing ready-
made computer vision products to address these issues.
Additionally, this work will introduce the use of neuromor-
phic sensors called event cameras, to perform the semantic
segmentation task [9].
These cameras provide numerous benefits compared
to traditional ones, including low latency, high temporal
resolution, and an exceptionally high dynamic range [10].
These features enable effective information capture in a
dynamic mining setting, resisting issues such as motion
blur from vibrating sensor platforms, shadows from single-
point source lighting, and interference from dust clouds.
However, due to the novelty of this sensor, there is a lack of
24-050
Hierarchical Training Pipeline for Event-Based Robotic
Perception Models for Autonomous Roof Bolting
Rik Banerjee
M3 Robotics Lab, Colorado School of Mines
Golden CO, USA
Akram Marseet
M3 Robotics Lab, Colorado School of Mines
Golden CO, USA
Andrew J. Petruska
M3 Robotics Lab, Colorado School of Mines
Golden CO, USA
ABSTRACT
Event cameras are used for their performance in high
dynamic-range lighting conditions which are canonical to
active mining environments. Direct labeling of event-based
image data to train a model to perform semantic segmenta-
tion using traditional methods is slow and error-prone. This
study proposes a framework to use roughly hand-labeled
color images from a mine as an input to an intermediary
probabilistic algorithm called alphamatting to generate a
ground-truth data set. These high-fidelity labels can be used
to train a semantic segmentation model to differentiate the
support strap from the roof. This model can then be lever-
aged to segment an event-based scene to enable autono-
mous roof bolting. This pipeline has been shown to achieve
an accuracy of 88% with a false positive rate of 3%.
INTRODUCTION
Roof bolting is commonly regarded as one of the riskiest
occupations in the United States [1]–[4]. This arises from
several factors: the operator faces the danger of sustaining
injuries from the bolting machine [5], and there exists a risk
of becoming a victim of a roof collapse [6]. Furthermore,
a significant portion of accidents occurs when operators
with less than one year of drilling experience are engaged
in bolting. Enhancing miner safety and productivity
could be achieved by relieving operators of the need to be
This work is supported by National Institute for Occupational
Safety &Health |NIOSH/Project 75D30121C12206.
underground to identify roof drilling areas, position, and
operate the drill. The low-light conditions in underground
mines have been studied to have a measurable adverse effect
on the health of the miners [7]. A robotic solution can be
used to automate this process.
As can be seen in Figure 1, automating this process
faces a two-fold challenge. Firstly, the autonomous system
needs to classify the visual scene into Rock and Not Rock.
This can be achieved by a process called semantic segmenta-
tion [8]. Secondly, a three-dimensional (3D) representation
of the surface needs to be computed by using either stereo-
vision or a depth sensor. This paper will present an archi-
tecture to solve the issue of semantically recognizing rock
in an underground mine during active drilling operations.
This means that the system needs to withstand vibrational
loads, dust clouds, and challenging lighting conditions [7].
These elements pose challenges for implementing ready-
made computer vision products to address these issues.
Additionally, this work will introduce the use of neuromor-
phic sensors called event cameras, to perform the semantic
segmentation task [9].
These cameras provide numerous benefits compared
to traditional ones, including low latency, high temporal
resolution, and an exceptionally high dynamic range [10].
These features enable effective information capture in a
dynamic mining setting, resisting issues such as motion
blur from vibrating sensor platforms, shadows from single-
point source lighting, and interference from dust clouds.
However, due to the novelty of this sensor, there is a lack of