Why is SciPy better than Matlab or IDL? ;)

Why is SciPy better than Matlab or IDL? ;)

Hello everyone,

I’m an engineer at Rijnhuizen, which is a
research institute for plasma physics. We use Python to drive one of our main
research projects, however, in the scientific circles in this institute Python
(and SciPy) are largely unknown. Time to change this, I think J. However, since I am an engineer
and not a scientist I my viewpoint on Python and SciPy are more of an engineering
perspective, like its Open Source, Free, modern language, functional
programming, etc... I’m not entirely sure these are compelling arguments
for a scientist to start working with Python (or atleast not on it self). So I
was wondering, if I was to promote Python in the scientific community here at
Rijnhuizen. So I have a few questions:

-In what aspects does
SciPy excel, compared to say IDL or matlab?

-In what ways allows it a scientist
to be more effective?

-How usable is SciPy for Plasma physics,
molucular dynamics and nanolayer Surface and Interface Physics (the 3 main
areas at Rijnhuizen)?

Re: Why is SciPy better than Matlab or IDL? ;)

I would say that its main strength is being a library of a
full-featured, well-supported, cross-platform programming language. This
means you can build arbitrarily complex applications using Python and
Scipy, while it would be very painful with Matlab.

> - In what ways allows it a scientist to be more effective?

See above.

> - How usable is SciPy for Plasma physics, molucular dynamics and nanolayer Surface and Interface Physics (the 3 main areas at Rijnhuizen)?

Re: Why is SciPy better than Matlab or IDL? ;)

I’m an engineer at Rijnhuizen, which is a
research institute for plasma physics. We use Python to drive one of our main
research projects, however, in the scientific circles in this institute Python
(and SciPy) are largely unknown. Time to change this, I think J. However, since I am an engineer
and not a scientist I my viewpoint on Python and SciPy are more of an engineering
perspective, like its Open Source, Free, modern language, functional
programming, etc... I’m not entirely sure these are compelling arguments
for a scientist to start working with Python (or atleast not on it self). So I
was wondering, if I was to promote Python in the scientific community here at
Rijnhuizen. So I have a few questions:

-In what aspects does
SciPy excel, compared to say IDL or matlab?

-In what ways allows it a scientist
to be more effective?

-How usable is SciPy for Plasma physics,
molucular dynamics and nanolayer Surface and Interface Physics (the 3 main
areas at Rijnhuizen)?

Re: Why is SciPy better than Matlab or IDL? ;)

> - In what aspects does SciPy excel, compared to say IDL or
> matlab?

Vectorized operations are (almost) as fast. Depending on what BLAS you
use, linear algebra can be just as fast. String processing in Python
is so incredibly better than Matlab that the thought of doing string
operations in Matlab almost makes me cry. Python has bindings to
nearly everything -- a dozen different GUI toolkits to low-level
hardware interfaces. Likewise there are a handful of different ways
to interface your code to code written in other languages -- ctypes,
swig, boost::python, cython for C/C++, PyObjC for Objective C, f2py
for Fortran, scikits.mlabwrap for Matlab, rpy and rpy2 for R...

> - In what ways allows it a scientist to be more effective?

As has been echoed here, writing large programs will be easier. Python
is a real programming language and was designed that way from the
ground up. Matlab was designed as an interactive environment first and
programming language features were grafted on as an afterthought.

> - How stable is it compared to other packages. (bugs,
> computation)?

I won't lie in that I occasionally run into bugs in SciPy, but one of
three things usually happens:

- I look at the code myself -- all of which is available to view and
modify, not the case with Matlab -- and file a bug report and/or patch
- I look at the code and realize it's not a bug but me
misunderstanding something, at which point I might edit the
documentation at http://docs.scipy.org/scipy/ to make it clearer if
necessary.
- If the code is incomprehensible to me, I ask on the mailing lists
and (either already or shortly thereafter) it's fixed by someone else
in Subversion.

Re: Why is SciPy better than Matlab or IDL? ;)

On 07/07/2009 11:14 AM, David Warde-Farley wrote:

On 7-Jul-09, at 10:03 AM, Vincent van Beveren wrote:

- In what aspects does SciPy excel, compared to say IDL or
matlab?

Vectorized operations are (almost) as fast. Depending on what BLAS you
use, linear algebra can be just as fast. String processing in Python
is so incredibly better than Matlab that the thought of doing string
operations in Matlab almost makes me cry. Python has bindings to
nearly everything -- a dozen different GUI toolkits to low-level
hardware interfaces. Likewise there are a handful of different ways
to interface your code to code written in other languages -- ctypes,
swig, boost::python, cython for C/C++, PyObjC for Objective C, f2py
for Fortran, scikits.mlabwrap for Matlab, rpy and rpy2 for R...

I would add because it is not always obvious and overlooked is that
Python really integrates very different things into a single unified
yet simple interface. As an example, you can very easily 'bolt on' web
stuff to an existing peice of Python code to create an almost instant
web application. (I have mixed linear algebra with html processing
including parsing cgi and creating html tags and images to provide a
couple of web apps.) So suddenly instead of saying I can do solve this
problem (and perhaps here is the code), you can just say 'go to this
web site and do it'. Plus, I can easily fix bugs and add new code
without the user doing anything. This is great publicly to grant
agencies and such like because you can actually show what you have done
and also provide some outreach to the community.

Re: Why is SciPy better than Matlab or IDL? ;)

On Tue, Jul 7, 2009 at 3:03 PM, Vincent van
Beveren<[hidden email]> wrote:
> - In what aspects does SciPy excel, compared to say IDL or matlab?
>
> - In what ways allows it a scientist to be more effective?
>
> - How usable is SciPy for Plasma physics, molucular dynamics and
> nanolayer Surface and Interface Physics (the 3 main areas at Rijnhuizen)?
>
> - How stable is it compared to other packages. (bugs, computation)?

I think it's probably been covered by others but some of the key
things for me are:

- how easy it is to extend performance critical portions in C or
FORTRAN (cython, ctypes, f2py, weave etc.)
http://www.scipy.org/PerformancePython may be slightly out of date but
I think it's a great illustration.

- how easy it is (relatively speaking) to employ parralel processing.
http://www.scipy.org/ParallelProgramming . Using the multiprocessing
module (available as a backport for 2.5) it is literally 2 lines to
have a for loop parallellised over 8 cores (or 16 if I had them). In
MATLAB anything more than 4 cores gets into really silly money (not
usually included in site license). This is probably the major factor
driving interest in Python from colleagues in my lab. Also of course
there are parralel python, ipython etc. and python bindings to MPI,
cuda, opencl etc. which I think would usually allow much quicker
progress than working in C.

- just how nice it is to use; broadcasting... even just how nice it is
to handle default arguments in functions and update code with new
parameters in a backwards compatible was was a bit of a revelation
for me after matlab.

- slices as views - in my work I usually have one long data set which
I slice and view in different ways. In matlab I think slices make
copies (at least they used to - possibly its changed now, but another
thing is that you can never really know what matlabs going to do,
whereas in Python you really can control how memory is laid out and
manipulated at a low level) but in Python all my different views point
to the same data.

Re: Why is SciPy better than Matlab or IDL? ;)

> As an example, you can very easily 'bolt on' web stuff to an
> existing peice of Python code to create an almost instant web
> application. (I have mixed linear algebra with html processing
> including parsing cgi and creating html tags and images to provide a
> couple of web apps.) So suddenly instead of saying I can do solve
> this problem (and perhaps here is the code), you can just say 'go to
> this web site and do it'. Plus, I can easily fix bugs and add new
> code without the user doing anything.

Indeed, I've done this too. Getting matplotlib to render PNGs for
TurboGears/Django/your-favourite-web-framework is dead simple, too so
you can provide really quite fancy visualization in your web-based
interface without having to do anything extra.

Re: Why is SciPy better than Matlab or IDL? ;)

David Warde-Farley <dwf <at> cs.toronto.edu> writes:
> On 7-Jul-09, at 12:39 PM, Bruce Southey wrote:
> > As an example, you can very easily 'bolt on' web stuff to an
> > existing peice of Python code to create an almost instant web
> > application.
[...]
> Indeed, I've done this too. Getting matplotlib to render PNGs for
> TurboGears/Django/your-favourite-web-framework is dead simple, too so
> you can provide really quite fancy visualization in your web-based
> interface without having to do anything extra.

@Bruce, David
Do you have a particular site that is one-line and
available to the gereral public?
I am intending something similar.

Although it is not specially topic of this list, I am particulary interested in
how you generate the matplotlib graphics on the fly.
To you store them in a temporary directory or do create
the files somehow on the fly?

Re: Why is SciPy better than Matlab or IDL? ;)

> Hello everyone,
>
> I'm an engineer at Rijnhuizen, which is a research institute for plasma
> physics. We use Python to drive one of our main research projects,
> however, in the scientific circles in this institute Python (and SciPy)
> are largely unknown. Time to change this, I think :). However, since I am
> an engineer and not a scientist I my viewpoint on Python and SciPy are
> more of an engineering perspective, like its Open Source, Free, modern
> language, functional programming, etc... I'm not entirely sure these are
> compelling arguments for a scientist to start working with Python (or
> atleast not on it self). So I was wondering, if I was to promote Python in
> the scientific community here at Rijnhuizen. So I have a few questions:
>
> - In what aspects does SciPy excel, compared to say IDL or
> matlab?
> - In what ways allows it a scientist to be more effective?
> - How usable is SciPy for Plasma physics, molucular dynamics and
> nanolayer Surface and Interface Physics (the 3 main areas at Rijnhuizen)?
> - How stable is it compared to other packages. (bugs,
> computation)?
>
> Any comments and insights are welcome!
>
> Regards,
> Vincent
>
> PS. Posted in scipy-dev by accident.
> ___
> Ing. V. van Beveren
> Software Engineer, FOM Rijnhuizen
> E: [hidden email]<mailto:[hidden email]>

The real question is "why is Python better than Matlab or IDL?"

I don't have an opinion on Matlab since I never used it. However, I used
IDL professionally for about 20 years or so (just for reference, I am a
solar physicist).

A few years ago, having gotten some proficiency in Python, I pretty much
dropped my usage of IDL and now use Python. This means python, ipython,
numpy, scipy, matplotlib and whatever other modules I need. One basic
difference is that I can use it at home, on the road and at work without
need of an expensive license. Also, I enjoy using it.

I have seem some object-oriented implementations is IDL and they are
pathetic. It may just be the way it was done but I don't know. The use of
objects in Python is very straightforward.

Re: Why is SciPy better than Matlab or IDL? ;)

Thanks everyone for your comments. Because
of the many responses I can’t reply to everyone personally, but thank you
for your input. I have summarised your comments into a list, if you have any
corrections, or additions, I would like to hear:

-Completely
Free: No ridiculous licenses or restrictions

-Open
source:

oIf there
is a bug you can fix it yourself, or ask the community

oYou can
always go back to the source

-Stable:

oOriginally
written by commercial company, before released opensource,

oThe base
of EPD (Enthought Python Distribution)

-Large
user community:

owell
supported

oactively
developed

-Easy to
build arbitrary complex applications (IDL and Matlab make this more difficult)

I’m an engineer at Rijnhuizen, which is a
research institute for plasma physics. We use Python to drive one of our main
research projects, however, in the scientific circles in this institute Python (and
SciPy) are largely unknown. Time to change this, I think J. However, since I am an engineer
and not a scientist I my viewpoint on Python and SciPy are more of an
engineering perspective, like its Open Source, Free, modern language,
functional programming, etc... I’m not entirely sure these are compelling
arguments for a scientist to start working with Python (or atleast not on it
self). So I was wondering, if I was to promote Python in the scientific
community here at Rijnhuizen. So I have a few questions:

-In
what aspects does SciPy excel, compared to say IDL or matlab?

-In what ways
allows it a scientist to be more effective?

-How usable
is SciPy for Plasma physics, molucular dynamics and nanolayer Surface and
Interface Physics (the 3 main areas at Rijnhuizen)?

Re: Why is SciPy better than Matlab or IDL? ;)

Vincent van Beveren wrote:

>
> Thanks everyone for your comments. Because of the many responses I
> can’t reply to everyone personally, but thank you for your input. I
> have summarised your comments into a list, if you have any
> corrections, or additions, I would like to hear:
>
>
>
> - Completely Free: No ridiculous licenses or restrictions
>
> - Open source:
>
> o If there is a bug you can fix it yourself, or ask the community
>
> o You can always go back to the source
>
> - Stable:
>
> o Originally written by commercial company, before released
> opensource,
>

It depends on which parts of the stack you are talking about. AFAIK, the
original design for numpy was done ~15 years by Jim Hugunin, and scipy
was born as an open source project from the start. The "official"
history can be found here for scipy:

Re: Why is SciPy better than Matlab or IDL? ;)

Thanks everyone for your comments. Because of the many responses I can’t reply to everyone personally, but thank you for your input. I have summarised your comments into a list, if you have any corrections, or additions, I would like to hear:

-Completely Free: No ridiculous licenses or restrictions

-Open source:

oIf there is a bug you can fix it yourself, or ask the community

oYou can always go back to the source

-Stable:

oOriginally written by commercial company, before released opensource,

oThe base of EPD (Enthought Python Distribution)

-Large user community:

owell supported

oactively developed

-Easy to build arbitrary complex applications (IDL and Matlab make this more difficult)

-As a language it:

ohas many modules out of the box

ois very readable, and modular

oeasy parallel programming

owell featured:

§has support for many data structures (Lists, Tuples, Sets, Dicts)

§Object Orientated (OO in IDL is a pain, Matlab: what objects?)

actually, Matlab has had objects for some time, but it feels very tacked-on.

much of the functionality of this is already in Matlab (can't speak for IDL). That's actually one of the pains of Python, that you have to install a whole lot of 3rd party packages where Matlab, it is all there. The Enthought Python Distribution, and Python XY go a long way to solve that, although 3d plotting seems easier in Matlab.

I converted from Matlab to Python a few years ago. Much of it had to do with price and convenience (one example of Matlab craziness is the license manager, which always went berserk on me and kicked me out of my legitimate copy of Matlab). However, once I switched, there was another advantage that hadn't occurred to me. I could replace all of my shell scripts, perl scripts, etc... with python too. That way, I got a lot more fluid with the language with non-scientific apps. I could make a web-based gradebook, and easily manipulate directories of images files. I could parse html files (BeautifulSoup), and many other things. It opened up many more productive applications of the language than I had anticipated.

For scientific work, cython is an unbeatable tool for optimizing slow pieces of your code. It's like Matlab cmex files, but infinitely more convenient and powerful. I haven't run Matlab for over a year now, and I haven't really missed it. Off my website, I have a presentation for Matlab users wanting to switch to python. It might be useful for you.

Re: Why is SciPy better than Matlab or IDL? ;)

> On 8-Jul-09, at 7:48 AM, Tim Michelsen wrote:
>
>> Would be nice to see your implementation.
>
> The way I did it was to do the plotting onto a FigureCanvasAgg and
> then printing it to a fake file using cStringIO.
>
> from matplotlib.backends.backend_agg import FigureCanvasAgg
> import matplotlib.pyplot as plt
> import

Apologies since this is OT for scipy, so please followup to
matplolib-users if you have further questions or comments, but there
is a slight problem with the example above. If you are using mpl in a
web server, and want explicit control of the canvas, you can either
use pyplot and the canvas like so::

The code you posted is a hybrid of using pyplot and managing your own
Canvas creation, which can work under certain configurations but is
brittle and not supported. This is mainly because pyplot creates and
manages its own canvas instances.

Re: Why is SciPy better than Matlab or IDL? ;)

On 11-Jul-09, at 7:41 AM, John Hunter wrote:

> The code you posted is a hybrid of using pyplot and managing your own
> Canvas creation, which can work under certain configurations but is
> brittle and not supported. This is mainly because pyplot creates and
> manages its own canvas instances.

Oops. I suppose it was an instance where I fiddled until it worked and
never gave it a second look. Thanks for the clarification John.