From what I infer by reading your post very quickly (I will read it better as soon as I have more time), you are not using Mathematica in a very efficient way.
An example? You claim Mathematica is not using the 4 cores you have available. Try changing Map[] to ParallelMap[] and you win parallelism, for free. Oh btw, For[] loops in Mathematica are quite inefficient (on that, we agree).
Also, you use python+pgsql so why don't you try using Mathematica+pgsql? Needs["DatabaseLink`"] might be beneficial :)
