Hi, all
I have been trying to install numpy.
Python is installed in my home directory
~/Python-2.2
and when I do
% python setup.py install
in the directory ~/Numeric-20.3, I get the following error message
distutils.errors.DistutilsPlatformError: invalid Python installation:
unable to open /usr/local/lib/python2.2/config/Makefile (No such file or
directory)
Can anybody figure out what I am doing wrong?
sungpil

Paul Dubois writes:
> I have verified that this package seems to work on Windows. I says seems
> only because I didn't try enough to uncover anything subtle.
Thanks! If you run maparray.py from the command-line, it runs a basic
regression test suite, which doesn't really try enough to uncover
anything subtle either.
> Unless or until we are convinced as a community that this is (a) the
> right way to do this and (b) that the package is portable, it would not
> be wise to put it in the main distribution.
Well, I'm not the community, but I'll state my arguments.
On (a): I don't know about the right way to do it; as I'm sure is
obvious, I'm new to extending Numerical Python in C, but I doubt
there's a simpler way to do it ("it" being seeing the contents of
files as Numeric arrays), and I think it does work as well as it's
possible to get it to work without major hacking of Numeric (to
support read-only arrays) or the mmap module (to prevent closing an
open object, to allow read-only mmapping on Windows). The other
things on the wishlist are basically features to add --- "start" and
"size" arguments to maparray(), a "create-a-file" argument to
maparray(), etc., and don't change the basic structure.
On (b): it depends on two-argument mmap.mmap(), open(),
<file>.fileno(), and os.fstat(). The major portability hurdle there
is probably mmap.mmap(), but that's OK.
> I would like to hear from the community about this so that I will know
> whether or not to add this package as a separate SourceForge 'package'
> within the Numerical Python area. Meantime I will add a link to the web
> page.
I would too. There have been downloads from something like 50 IP
addresses, but I've only heard from three people.

>
> Is there some support for random arrays in numarray? Or, if it becomes
> part of the standard library, will there be a separate module (such as
> Numeric's RandomArray)?
>
> --
> Magnus Lie Hetland The Anygui Project
> http://hetland.orghttp://anygui.org
>
Not at the moment, but I imagine we will add a separate module as it
is done now for Numeric.
While I'm replying, our short range development plans are (now
that we are back working on it):
1) adding properties for .shape, .flat, etc. This was done
last week but we are testing this and will put it out as soon
as possible (probably after the Python Conference). By the way,
to make it usable from Python <2.2, we will add accessor functions
with different names than we used previously (e.g., .getshape(),
setshape(), .getflat()...)
2) reworking numarray to make it safe against array overruns
and anything that might cause segfaults or buserrors (in principle
should only happen now if you twiddle directly with private attributes).
3) provide examples of how to add new ufuncs and interface C code and
such. We may decide to add a module or two in the process. And perhaps
some limited benchmarking.
4) rework the implementation of complex types to be in C.
5) do more extensive benchmarking and look at how to optimize smaller
arrays.
Perry Greenfield

Is there some support for random arrays in numarray? Or, if it becomes
part of the standard library, will there be a separate module (such as
Numeric's RandomArray)?
--
Magnus Lie Hetland The Anygui Project
http://hetland.orghttp://anygui.org

Numarray should soon support the following properties:
.shape
.flat
.real
.imag
.imaginary
We're testing them here at STSCI. If all goes well, there should be a new numarray release within 1-3 weeks.
Todd
--
Todd Miller jmiller@...
STSCI / SSG (410) 338 4576

According to the (probably outdated) user guide for numarray[1]
attributes have accessors such as
a.foo() --> get foo
a.foo(val) --> set foo to val
but this doesn't seem to be the case. I see that the shape() method
doesn't take any arguments; I guess the reshape method is the thing to
use here. How about other attributes? Will they have special names for
their setters too, or is this only true for the shape attribute? It
seems to me that if we are going to abandon plain Python attributes
(or 2.2 properties), it would be nice to handle the transition in a
consistent manner... Perhaps?
(But then again, I'm sure I'm missing something :)
[1] http://stsdas.stsci.edu/numarray/Userguide.html
--
Magnus Lie Hetland The Anygui Project
http://hetland.orghttp://anygui.org

I see in the docs that numarray uses accessor methods instead of
properties, mainly because of performance (to avoid __getattr__ and
__setattr__). With the new property type in Python 2.2, that shouldn't
be as important anymore... Perhaps it would be possible to use
properties where this type is available, and accessors elsewhere?
--
Magnus Lie Hetland The Anygui Project
http://hetland.orghttp://anygui.org

The O'Reilly Open Source Convention (July 22-26, 2002 -- San Diego,
CA) is accepting proposals for tutorials, talks, panels, and lightning
talks. See the Call for Participation in the Python and Zope track on
python.org. Proposals are due by March 1, so don't wait a moment
longer!
CFP URL: http://www.python.org/workshops/oscon2002/cfp.html
--david ascher

Thanks for your quick reply. This looks vaguely S-Plus-like, with which I am
familar. I wasn't too clear in my question however, I would actually like a
vector returned which would contain the values of c for the minimun value of
b, within each category of a. Something like:
for i in range(1:5000): # range of values of a
d = c[Numeric.argmin(Numeric.where(a==i, b, sys.maxint))]
Seems like I could hack a fix using such a vector, but I would guess that
speed would be an issue.
Essentially the functionality implied by
GROUP BY in sql
egen in stata
lapply() in S-Plus.
Paul F. Dubois wrote:
> import Numeric, sys
> a = Numeric.array([1,2,1,2,4,3,1,2,3,2,3,2])
> b = Numeric.array([4,3,5,2,4,5,3,6,3,2,5,6])
> c = Numeric.arange(len(b))
>
> print c[Numeric.argmin(Numeric.where(a==1, b, sys.maxint))]
>
>
> -----Original Message-----
> From: numpy-discussion-admin@...
> [mailto:numpy-discussion-admin@...] On Behalf Of
> Michaell Taylor
> Sent: Monday, January 28, 2002 12:59 PM
> To: numpy-discussion@...
> Subject: [Numpy-discussion] conditional array indexing
>
>
>
> Very new user to Numpy, so excuse the question. It relates to a
> fundamental
> issue which I have having trouble getting a grip on.
>
> Specifically, say I have the following:
>
> a = ([1,2,1,2,4,3,1,2,3,2,3,2])
> b = ([4,3,5,2,4,5,3,6,3,2,5,6])
> c = random variable of length=len(b)
>
> I would like to get the value of C associated with the lowest value of b
>
> within groupings of a. That is, there are three incidents of "1" in the
> a
> array. The lowest value of b associated with the value 1 in the a index
> is
> "3". This occurs in index value 6 on the a and b arrays. Now, I would
> like
> to then know the value of c[6].
>
> Obviously the real problem is far more complex with len(a)==5000.
> Any ideas?
>
> Thanks in advance.
>
> Michaell
>
> _______________________________________________
> Numpy-discussion mailing list Numpy-discussion@...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
--
=========================================
Michaell Taylor, PhD
Senior Economist, Reis, New York, USA

On Sun, 27 Jan 2002, Travis Oliphant wrote:
[...]
> If I understand what you are trying to do, there is a function called
> arraymap in the SciPy package (special module) (it was in the Numeric source
[...]
> I think that using this function and a combination of take and put can do
> what you describe.
I don't see how, though I may very well be missing something. To restate
the problem very quickly (though the Python function I posted is much
clearer -- it's only two or three lines of actual code): for every element
in a list of array indices, the element in an output array at that index
needs to be incremented; the tricky part is that this needs to happen once
for *every time the index appears in the list*. Eg., in a 1D example,
[0,1,1,1,4,5] might give you
[1,3,0,0,1,1]
(again, see my previous post for an actual tested example)
I don't see how you can do this with arraymap, take and put.
John

Magnus Lie Hetland wrote:
> I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but
> somehow the compilation gets stuck on this line:
>
> gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray
> -I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c
> -o build/temp.linux-i686-2.2/_ufuncmodule.o
>
> No error message, no crash; it just never finishes -- it freezes. I
> compiled/installed it on a Solaris box earlier, with no problems
> whatsoever.
>
> I've never seen gcc freeze like this; can there be some circular
> dependencies or something? (This machine had some problems with its
> clock earlier, which made make go in a loop...) I tried running touch
> on all files, but that didn't change anything...
>
> Any ideas, or tips for finding out what's wrong? (BTW, Numeric
> compiles and installs just fine...)
Yes, just wait several minutes for it to finish compiling. I had to wait over 5
minutes on 233 MHz i586.
--
Paul Barrett, PhD Space Telescope Science Institute
Phone: 410-338-4475 ESS/Science Software Group
FAX: 410-338-4767 Baltimore, MD 21218

import Numeric, sys
a = Numeric.array([1,2,1,2,4,3,1,2,3,2,3,2])
b = Numeric.array([4,3,5,2,4,5,3,6,3,2,5,6])
c = Numeric.arange(len(b))
print c[Numeric.argmin(Numeric.where(a==1, b, sys.maxint))]
-----Original Message-----
From: numpy-discussion-admin@...
[mailto:numpy-discussion-admin@...] On Behalf Of
Michaell Taylor
Sent: Monday, January 28, 2002 12:59 PM
To: numpy-discussion@...
Subject: [Numpy-discussion] conditional array indexing
Very new user to Numpy, so excuse the question. It relates to a
fundamental
issue which I have having trouble getting a grip on.
Specifically, say I have the following:
a = ([1,2,1,2,4,3,1,2,3,2,3,2])
b = ([4,3,5,2,4,5,3,6,3,2,5,6])
c = random variable of length=len(b)
I would like to get the value of C associated with the lowest value of b
within groupings of a. That is, there are three incidents of "1" in the
a
array. The lowest value of b associated with the value 1 in the a index
is
"3". This occurs in index value 6 on the a and b arrays. Now, I would
like
to then know the value of c[6].
Obviously the real problem is far more complex with len(a)==5000.
Any ideas?
Thanks in advance.
Michaell
_______________________________________________
Numpy-discussion mailing list Numpy-discussion@...
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Very new user to Numpy, so excuse the question. It relates to a fundamental
issue which I have having trouble getting a grip on.
Specifically, say I have the following:
a = ([1,2,1,2,4,3,1,2,3,2,3,2])
b = ([4,3,5,2,4,5,3,6,3,2,5,6])
c = random variable of length=len(b)
I would like to get the value of C associated with the lowest value of b
within groupings of a. That is, there are three incidents of "1" in the a
array. The lowest value of b associated with the value 1 in the a index is
"3". This occurs in index value 6 on the a and b arrays. Now, I would like
to then know the value of c[6].
Obviously the real problem is far more complex with len(a)==5000.
Any ideas?
Thanks in advance.
Michaell

I'm trying to compile numarray 0.11 with Python 2.2 on a linux box but
somehow the compilation gets stuck on this line:
gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -IInclude/numarray
-I/home/mlh/python/Python-2.2/include/python2.2 -c Src/_ufuncmodule.c
-o build/temp.linux-i686-2.2/_ufuncmodule.o
No error message, no crash; it just never finishes -- it freezes. I
compiled/installed it on a Solaris box earlier, with no problems
whatsoever.
I've never seen gcc freeze like this; can there be some circular
dependencies or something? (This machine had some problems with its
clock earlier, which made make go in a loop...) I tried running touch
on all files, but that didn't change anything...
Any ideas, or tips for finding out what's wrong? (BTW, Numeric
compiles and installs just fine...)
--
Magnus Lie Hetland The Anygui Project
http://hetland.orghttp://anygui.org

> In the process of 'porting' Konrad Hinsen's Histogram class (from the
> Scientific package) to N dimensions, I ran across the problem of how to
> apply a function to an array for a specified set of indices.
If I understand what you are trying to do, there is a function called
arraymap in the SciPy package (special module) (it was in the Numeric source
for a short time but I think we decided to take it out --- I can't remember
why).
I think that using this function and a combination of take and put can do
what you describe.
It is available as scipy.special.arraymap
Best regards,
Travis Oliphant

cavallo@... <cavallo@...>:
...
> The license should be BSD? GPL? LGPL? any comment is welcome, because i
> never understood these problem (it is my first project, didn't i tell
> you?).
If you don't specifically want to follow the philosophy of the GPL,
I'd suggest that you go with some BSD variant (e.g. MIT, which is even
simpler). That way, you place less restrictions on its
use/distributions, and more people can/will use it.
(YMMV, of course.)
--
Magnus Lie Hetland The Anygui Project
http://hetland.orghttp://anygui.org

Hy,
this is my first attempt to give some back to the python/numerical-python
community.
Here (http://kdfio.sourceforge.net) you can find an early release of my
I/O routine for reading and writing kdf file as in khoros environment.
Khoros (from http://www.khoral.com) is an interactive program for digital
image processing, we are using in our lab (university of heidelberg).
All the time i've found hard to use such environment because it
lacks some flexibility i need.
Now i've been using numerical python for some time and i would other will
use it as standard base for numerical computation, but for my purpose it
was lacking I/O with such an environment (khoros).
Khoros is not free, but anyway if you are using it i hope you will like
these routines to read and write their files.
These are in an early stage, so expect a lot of bugs, missing feature and
so on: it's only my first project so feedbacks, suggestions and complains
are welcome.
Ops, i've forgotten: i'm personally using this rotines for my PHD, so they
should be reasonably stable.
The license should be BSD? GPL? LGPL? any comment is welcome, because i
never understood these problem (it is my first project, didn't i tell
you?).
best regards to all GREAT folks in this group,
antonio cavallo
ps. you can download khoros if you are a student from:
http://www.khoral.com/khoros/kp2001_student

I've been asked to write a short column about Numpy and my web site in
IEEE AP, for the Educational Corner. I still consider myself a Numpy
neophyte, but perhaps thats the beauty of Python, being able to do neat
things without being an expert. I am pondering what to write about on
the Numpy side. I am soliciting any genious ideas for topics :) The
first thing I thought of is that I just plain enjoy being able to tweek
code without worrying about compilers. Second, the code ends up so much
more readable than Fortran or C. Thirdly, vectorizing the ported code
makes it look much more like the mathematical basis of the program.
Lastly, I think its the perfect language for students learning
algorithms. I'm open to any ideas. Thanks, Rob.
--
The Numeric Python EM Project
http://www.pythonemproject.com

Having written all this down, I'm half-expecting somebody is going to
point out some simple way of doing this with existing builtin functions,
but here goes...
In the process of 'porting' Konrad Hinsen's Histogram class (from the
Scientific package) to N dimensions, I ran across the problem of how to
apply a function to an array for a specified set of indices. In this
case, the problem is to increment an N-dimensional array for each of a set
of indices. In general, it occurred to me that it would be useful to have
a function (named 'amap' here) to loop over a specified set of indices,
applying a function to a set of arguments taken from a set of argument
arrays (similar to the map buitin function in this respect), and filling
in the corresponding elements in a target array with the results. For
example, here is a bit of the histogram problem:
[The original 1D Histogram class uses a clever array-broadcasting trick
which creates intermediate rank-2 arrays, feeding in the data in chunks to
avoid quadratic behaviour. This is fine with 1D arrays, but in poor taste
for the N dimensional case, if it works at all.]
#!/usr/bin/env python
import Numeric as N
def flatten_indices(shape, indices):
dims = indices.shape[0]
facs = pow(shape, N.arange(dims))[::-1,N.NewAxis]
return N.add.reduce(facs*indices)
def amap(target, function, indices, *arg_arrays):
indices = N.asarray(indices)
arg_arrays = map(N.asarray, arg_arrays)
if not isinstance(target, N.ArrayType):
raise TypeError, "target must be an array"
if not target.iscontiguous():
raise ValueError, "target must be contiguous"
if not N.alltrue([aa.iscontiguous() for aa in arg_arrays]):
raise ValueError, "all arg_arrays must be contiguous"
if not len(indices.shape) == 1:
raise ValueError, "indices must be rank-1"
tf = target.flat
for index in indices:
args = [arg_array.flat[index] for arg_array in arg_arrays]
tf[index] = function(*args)
# construct a 2 dimensional histogram given indices, ind, of data points
nbins = (5, 4) # 5 bins along one axis, 4 along the other
histo = N.zeros(nbins) # empty histogram
# these are the indices into the histogram at which data points lie:
# histo[1,0] should be incremented twice, histo[2,2] should be incremented
# once, etc.
ind = N.array([[1, 1, 2, 4],
[0, 0, 2, 3]])
flat_ind = flatten_indices(histo.shape, ind)
assert flat_ind == N.array([4, 4, 10, 19])
# Now we want to increment histo for each index pair ind[:,n] (in this
# 2D case)
def increment(x): return x+1
amap(histo, increment, flat_ind, histo)
print histo
Actually, I just realised that calling it something related to 'map' is
rather misleading, because the builtin Python map generates a list of the
same length as the sequence passed in; this function doesn't: it
'generates' (fills in, in fact) an array of the same size as the target
passed in. Can't think of a better name, though.
So: is there a way of doing this (implementing amap) efficiently already?
I don't think there is, but I'd be interested to be proved wrong.
I admit I don't like the relatively large number of arguments, but I still
think it is worthwhile. You could make the *arg_arrays a single argument,
and make target an optional argument, at the end of the arguments:
def amap(function, indices, arg_arrays, target=None):
so that by default, the target array is first created as zeros, then
filled in and returned, or perhaps even better:
def amap(function, arg_arrays, indices=None, target=None):
though the special case where you don't give the indices argument is
already achievable without this function, so it could be argued that
hiding away the indices argument as a mere optional argument obscures the
real point of the function. The last form also has the (debatable)
benefit of being similar to the built-in Python map:
def map(function, *args):
The example call above then looks like this:
histo = amap(increment, [histo], indices=flat_ind)
Another couple of tweaks might be to allow scalar values to replace the
arg_arrays argument, and to allow normal indices rather than flattened
ones (the only reason I started out with flat indices was that put() did
the same):
def amap(function, args, indices=None, target=None):
So that the example call looks like:
histo = amap(increment, [histo], indices=ind)
or
histo = amap(Numeric.add, [histo, 1], ind)
I like those last two. Even if there is already an efficient way of doing
this with existing Numeric functions, I think this is a natural way of
thinking about some operations on arrays -- this one, at least -- and
would be useful to have in Numeric.
For real efficiency, I suppose the right thing to do would be to have a
map method on ufuncs, as is the case for reduce:
histo = Numeric.add.map([histo, 1], ind)
but it would be more work to implement than just a standard function, and
less general.
I've made a first attempt at the function, in C (will upload diff against
20.2.0 if anyone so requests). It only accepts one arg array, probably
doesn't really do exactly the same as the Python function above (for
better or worse), and it's a mess of a function, since I rarely write
anything in C. It is also incorrect in at least some ways (most
obviously, it only deals with Python floats / C doubles / Numpy
PyArray_DOUBLEs), though there are some tests (which pass).
I'm wasn't sure how fast it would be, since after all it's only using
ordinary Python functions, not ufuncs, but my crude tests appeared to show
an improvement of about a factor of two over an optimised version of the
Python function above. The slow bit seems to be constructing the argument
list / passing arg list to function, so I removed that to make the
comparison, along with an unecessary attribute access in the loop.
Presumably the general (> 1 arg_array) version would show a bigger
improvement over the corresponding Python version.
By the way, does anyone understand what the reduceat ufunc method is
supposed to do? Seems to be the Beale cipher of Numeric -- having studied
its output to see if it was the function I was looking for, I can find no
sense in it and kind of suspect that it was constructed only to keep us
puzzling. ;) Is Jim Hugunin not available for comment on the matter, or is
this somebody else's dirty work?
John