Classic programming assigns values to variables, something like “var x = 5;” and any other variable depending on this assignment is exact, it has a deterministic outcome. Real life is however more complex, human beings are used to deal with uncertainties; it might rain so I’ll take the umbrella with me. Uncertainties are everywhere and at some point any real-world situations has to deal with ‘noise’. Probabilistic programming takes uncertainty to the core and replaces the above assignment with “var x = Normal(5, 0.3);” stating that the value is normally distributed around 5 with variance 0.3. It means that it could be 4.5 or 5.1 or even 0.2 but much less likely than a value around 5.
Taking into account probabilistic distribution in computations and how to infer things from uncertainty is what probabilistic programming is all about. If in addition you consider the weights of a neural network as probabilistic you venture in deep probabilistic programming. It creates more ‘real’ AI in the sense that it reflects our own way of thinking and how life altogether is, in fact, statistical.
Much like quantum motion can be reduced to classical paths in the limit of singular (certain) distributions, so can probabilistic programming reproduce normal programming in the limit of zero variance.