Stochastic computing (SC) with its stream-based, probabilistic number representation promises large area and power benefits as well as increased error tolerance compared to conventional binary computing. While SC is less precise, it is considered a promising option for implementing neural network inferencing in ultra-low-power edge devices. SC-based Neural Networks (SCNNs) typically combine stochastic and binary components for interfacing and to alleviate certain SC limitations. Moreover, ultra-low-power VLSI for edge computing is often less reliable due to noisy environments or deliberate power-reliability trade-offs. In this work, we present the first detailed investigation of the behavior of an SCNN and its individual components on hardware prone to timing errors. Our results show that robustness of SC is highly dependent on specific design choices and that biases in the error distributions may even cause SCNNs to perform worse under certain circumstances than comparable binary implementations. It shows that robustness should be treated as a design goal in SC rather than taken for granted.