Albert Einstein did not believe in a personal deity, but he was willing to talk about the ontological mystery (the mystery of being) in ways that bordered on the religious:
The most beautiful emotion we can experience is the mysterious. It is the fundamental emotion that stands at the cradle of all true art and science. He to whom this emotion is a stranger, who can no longer wonder and stand rapt in awe, is as good as dead, a snuffed-out candle. To sense that behind anything that can be experienced there is something that our minds cannot grasp, whose beauty and sublimity reaches us only indirectly: this is religiousness. In this sense, and in this sense only, I am a devoutly religious man.
[Quote source here.]
I think that, among contemporary agnostics and atheists, the use of the word “mystery” is almost a fright word. To admit that there is any mystery about the universe’s existence is to suggest that there is, ultimately, something incomprehensible and unsolvable about it (something that cannot be reduced to a function or a problem), and this is a “no-no.”
But Einstein’s emotionally open agnosticism strikes me as an anecdote to this form of contemporary reductionism.
Prometheus Unbound 