Answers:
The official NumPy book is available at the website (http://numpy.scipy.org/). It is an e-book, available for $40, or $20 if you can show that you are a college student. I have not read it, so I cannot attest to its quality. The tutorial on the SciPy wiki is at http://www.scipy.org/NumPy_Tutorial, though it appears unfinished, but could still be useful.
I believe there was also question about the size of matrices supported in NumPy in comparison to Matlab. Assuming Matlab is efficiently programmed, the size of a matrix should be pretty close to the size of an element * the number of elements. I’m quite certain this is the case with NumPy, so using the same type of elements, they should allow the same size. However, I believe NumPy allows more control over the memory each element takes, so you should be able to bring each element down to 1 byte, although there will be a very large drop in precision. I wouldn’t recommend going below a 4-byte float for scientific computing.
Thanks I will look over the wiki again, it has evolved since I first looked at it. :-)
As for the size question, I am working on problems that are getting close to the size of the whole human genome. We routinely end up with matrices of close to 100,000 columns, although the rows tend to be fewer (sometime on the order of 5,000, but sometimes hitting 20,000). We can afford lots of very fast boxen and memory, but MATLAB seems to crap out at about 100,000 by 1,000 due to index limits, rather than memory (although memory comes into play as soon as we have a couple of matrices).
So we are working on more efficient computations, but we would like to be able to do some of the naive phrasing in the meantime. But we can’t do it in MATLAB. :-(
This is the space I am coming from. Thanks for the pointer!
In a matrix that size, you have 500,000,000-2,000,000,000 elements. I was able to get a 100,000×5,000 array on my computer, but this takes 2 gigabytes using 32-bit floats. 100,000×20,000 would take 8 gigs. Unless you can do with only 8-bit precision, you’ll need a 64-bit processor and OS to support this, but it should work.
Hi again, thanks for poking around, I appreciate it a lot! :-) I don’t have a lot of local experts to bounce ideas off of…
Do you have any experience using 64-bit MATLAB? Does it allow the full addressing required? We have gotten mixed replies from various experts, but know no one who actually has run a 64-bit MATLAB and seen it work. We suspected that it might work.
Also, since you seem to be smart about this particular topic, do you know if the memory limits for MATLAB under Linux are less restrictive than under Windows? When we start to get somewhere with indices, we often hit problems at the OS level with memory.
Speed and memory are not problems price-wise, the granting agencies seem open to our expected prices.
Thanks for the feedback you’ve given me, it helps. Seriously!
