Now that we've got a lot of contributors to Phobos and many projects in
the works, I decided to start a thread to help us make a rough plan for
Phobos's short-to-medium term development. There are three goals here:
1. Determine what's next in the review queue after std.csv (voting on
std.csv ends tonight, so **please vote**).
2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language than
it is now.
3. Figure out who's already working on what from the wish list and what
bottlenecks, if any, are getting in the way and what can be done about them.
The following is the wish list as I see it. Please suggest additions
and correct any errors, as this is mostly off the top of my head. Also,
status updates if you're working on any of these and anything
substantial has changed would be appreciated.
* Some higher level networking support, such as HTTP, FTP, etc. (Jonas
Drewsen's CURL wrapper handles a lot of this and may be ready for a
second round of review.)
* Serialization. (Jacob Carolberg's Orange library might be a good
candidate. IIRC he said it's close to ready for review.)
* Encryption and hashing. (This is more an implementation problem than
a design problem and AFAIK noone is working on it.)
* Containers. (AFAIK noone is working on this. It's tough to get
started because, despite lots of discussion at various times on this
forum, noone seems to really know what they want. Since the containers
in question are well-known, it's much more a design problem than an
implementation problem.)
* Allocators. (I think Phobos desperately needs a segmented
stack/region based allocator and I've written one. I've also tried to
define a generic allocator API, mostly following Andrei's suggestions,
but I'll admit that I didn't really know what I was doing for the
general API. Andrei has suggested that allocators should have
real-world testing on containers before being included in Phobos.
Therefore, containers block allocators and if the same person doesn't
write both, there will be a lot of communication overhead to make sure
the designs are in sync.)
* Streams. (Another item where the bottleneck is mostly at the design
level and people not really knowing what they want.)
* Compression/archiving. (Opening standard compressed/archived file
formats needs to just work. This includes at least zip, gzip, tar and
bzip2. Of course, zip already is available and gzip is supported by the
zlib module but with a crufty C API. At least gzip and bzip2, which are
stream-based as opposed to file-based, should be handled via streams,
which means that streams block compression/archiving. Also, since tar
and zip are both file based, they should probably be handled by the same
API, which might mean deprecating std.zip and rewriting it.)
* An improved std.xml. (I think Thomas Sowinski is working on a
replacement, but I haven't seen any updates in a long time.)
* Matrices and linear algebra. (Cristi Cobzarenco's GSoC project is a
good starting point but it needs polish. I've been in contact with him
occasionally since GSoC ended and he indicated that he wants to get back
to working on it but doesn't have time. I've contributed to it
sparingly, but find it difficult because I haven't gotten around to
familiarizing myself with the implementation details yet, and it's hard
to get into a project that complex with a few hours a week as opposed to
focusing full time on it.)
* std.database. (Apparently Steve Teale is working on this. This is a
large, complicated project because we're trying to define a common API
for a variety of RDBMSs. Again, it's more a design problem than an
implementation problem.)
* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve
Schveighoffer ported it to Windows, but issues with the DMC runtime
prevent it from working on Windows.)
* Parallel algorithms. (I've implemented a decent amount of these in
my std.parallel_algorithm Github project, but I've become somewhat
frustrated and unmotivated to finish this project because so many of the
relevant algorithms seem memory bandwidth bound and aren't substantially
faster when parallelized than when run serially.)
After writing this, the general pattern I notice is that lots of stuff
is blocked by design, not implementation. In a lot of cases people
don't really know what they want and analysis paralysis results.

I decided to start a thread to help us make a rough plan for
Phobos's short-to-medium term development.

Not too much time ago there was a Phobos wish list thread. I suggest you to
take a look at it (it contains my answer too, so I think there is no need to
repeat it here).

2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language than
it is now.

It's not just a matter of modules, but also of few functions to add to existing
modules. I have some of those.
Another thing that will improve D attractiveness for people that try D for the
fist time is to find ways to make more Phobos functions work with
const/immutable input data (like string functions).
Bye,
bearophile

2. Come up with a wish list of high-priority modules that Phobos is=20
missing that would make D a substantially more attractive language
than it is now.

That's probably not for the Phobos, but having clear roadmap for the
GUI bindings projects (dwt2, gtkd, qtd, wxd,...) would definitely, imho,
make D more attractive language by allowing developers to jump into
writing GUI apps.

* Containers. (AFAIK noone is working on this. It's tough to get=20
started because, despite lots of discussion at various times on this=20
forum, noone seems to really know what they want. Since the
containers in question are well-known, it's much more a design
problem than an implementation problem.)

+1

* Compression/archiving. (Opening standard compressed/archived file=20
formats needs to just work. This includes at least zip, gzip, tar
and bzip2. Of course, zip already is available and gzip is supported
by the zlib module but with a crufty C API. At least gzip and bzip2,
which are stream-based as opposed to file-based, should be handled
via streams, which means that streams block compression/archiving.
Also, since tar and zip are both file based, they should probably be
handled by the same API, which might mean deprecating std.zip and
rewriting it.)

+1

* std.database. (Apparently Steve Teale is working on this. This
is a large, complicated project because we're trying to define a
common API for a variety of RDBMSs. =20

+1
Sincerely,
Gour
--=20
A person who is not disturbed by the incessant flow of=20
desires =E2=80=94 that enter like rivers into the ocean, which is=20
ever being filled but is always still =E2=80=94 can alone achieve=20
peace, and not the man who strives to satisfy such desires.
http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810

Now that we've got a lot of contributors to Phobos and many projects
in the works, I decided to start a thread to help us make a rough plan
for Phobos's short-to-medium term development. There are three goals
here:
1. Determine what's next in the review queue after std.csv (voting on
std.csv ends tonight, so **please vote**).
2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language
than it is now.
3. Figure out who's already working on what from the wish list and
what bottlenecks, if any, are getting in the way and what can be done
about them.
The following is the wish list as I see it. Please suggest additions
and correct any errors, as this is mostly off the top of my head.
Also, status updates if you're working on any of these and anything
substantial has changed would be appreciated.
* Some higher level networking support, such as HTTP, FTP, etc.
(Jonas Drewsen's CURL wrapper handles a lot of this and may be ready
for a second round of review.)
* Serialization. (Jacob Carolberg's Orange library might be a good
candidate. IIRC he said it's close to ready for review.)
* Encryption and hashing. (This is more an implementation problem
than a design problem and AFAIK noone is working on it.)
* Containers. (AFAIK noone is working on this. It's tough to get
started because, despite lots of discussion at various times on this
forum, noone seems to really know what they want. Since the
containers in question are well-known, it's much more a design problem
than an implementation problem.)
* Allocators. (I think Phobos desperately needs a segmented
stack/region based allocator and I've written one. I've also tried to
define a generic allocator API, mostly following Andrei's suggestions,
but I'll admit that I didn't really know what I was doing for the
general API. Andrei has suggested that allocators should have
real-world testing on containers before being included in Phobos.
Therefore, containers block allocators and if the same person doesn't
write both, there will be a lot of communication overhead to make sure
the designs are in sync.)
* Streams. (Another item where the bottleneck is mostly at the
design level and people not really knowing what they want.)
* Compression/archiving. (Opening standard compressed/archived file
formats needs to just work. This includes at least zip, gzip, tar and
bzip2. Of course, zip already is available and gzip is supported by
the zlib module but with a crufty C API. At least gzip and bzip2,
which are stream-based as opposed to file-based, should be handled via
streams, which means that streams block compression/archiving. Also,
since tar and zip are both file based, they should probably be handled
by the same API, which might mean deprecating std.zip and rewriting
it.)

AFAIK almost all compression (and encryption) algorithms work like
streams. The directory structure is often simply added on top of the
streaming functionality (at least true for all .tar.* formats).
BTW: The unfinished stream API blocks lots of projects: Encryption,
Compression, the curl wrapper could be integrated with streams, http
servers and almost everything else related to I/O.

* An improved std.xml. (I think Thomas Sowinski is working on a
replacement, but I haven't seen any updates in a long time.)
* Matrices and linear algebra. (Cristi Cobzarenco's GSoC project is
a good starting point but it needs polish. I've been in contact with
him occasionally since GSoC ended and he indicated that he wants to
get back to working on it but doesn't have time. I've contributed to
it sparingly, but find it difficult because I haven't gotten around to
familiarizing myself with the implementation details yet, and it's
hard to get into a project that complex with a few hours a week as
opposed to focusing full time on it.)
* std.database. (Apparently Steve Teale is working on this. This is
a large, complicated project because we're trying to define a common
API for a variety of RDBMSs. Again, it's more a design problem than
an implementation problem.)
* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve
Schveighoffer ported it to Windows, but issues with the DMC runtime
prevent it from working on Windows.)
* Parallel algorithms. (I've implemented a decent amount of these in
my std.parallel_algorithm Github project, but I've become somewhat
frustrated and unmotivated to finish this project because so many of
the relevant algorithms seem memory bandwidth bound and aren't
substantially faster when parallelized than when run serially.)

Inter-process communication, but as we have thrift now, that might be
ok. (Extending std.concurrency message passing to interprocess
communication with a low overhead serializer like messagepack would
still be great though)
Logging: AFAIK the new log module should be almost ready. I already
used it a few times, but you have to a patch druntime to use
it.
High performance IO: A reactor / proactor framework
A replacement for std.json

After writing this, the general pattern I notice is that lots of stuff
is blocked by design, not implementation. In a lot of cases people
don't really know what they want and analysis paralysis results.

Now that we've got a lot of contributors to Phobos and many projects
in the works, I decided to start a thread to help us make a rough plan
for Phobos's short-to-medium term development. There are three goals
here:
1. Determine what's next in the review queue after std.csv (voting on
std.csv ends tonight, so **please vote**).
2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language
than it is now.
3. Figure out who's already working on what from the wish list and
what bottlenecks, if any, are getting in the way and what can be done
about them.
The following is the wish list as I see it. Please suggest additions
and correct any errors, as this is mostly off the top of my head.
Also, status updates if you're working on any of these and anything
substantial has changed would be appreciated.
* Some higher level networking support, such as HTTP, FTP, etc.
(Jonas Drewsen's CURL wrapper handles a lot of this and may be ready
for a second round of review.)
* Serialization. (Jacob Carolberg's Orange library might be a good
candidate. IIRC he said it's close to ready for review.)
* Encryption and hashing. (This is more an implementation problem
than a design problem and AFAIK noone is working on it.)
* Containers. (AFAIK noone is working on this. It's tough to get
started because, despite lots of discussion at various times on this
forum, noone seems to really know what they want. Since the
containers in question are well-known, it's much more a design problem
than an implementation problem.)
* Allocators. (I think Phobos desperately needs a segmented
stack/region based allocator and I've written one. I've also tried to
define a generic allocator API, mostly following Andrei's suggestions,
but I'll admit that I didn't really know what I was doing for the
general API. Andrei has suggested that allocators should have
real-world testing on containers before being included in Phobos.
Therefore, containers block allocators and if the same person doesn't
write both, there will be a lot of communication overhead to make sure
the designs are in sync.)
* Streams. (Another item where the bottleneck is mostly at the
design level and people not really knowing what they want.)
* Compression/archiving. (Opening standard compressed/archived file
formats needs to just work. This includes at least zip, gzip, tar and
bzip2. Of course, zip already is available and gzip is supported by
the zlib module but with a crufty C API. At least gzip and bzip2,
which are stream-based as opposed to file-based, should be handled via
streams, which means that streams block compression/archiving. Also,
since tar and zip are both file based, they should probably be handled
by the same API, which might mean deprecating std.zip and rewriting
it.)

AFAIK almost all compression (and encryption) algorithms work like
streams. The directory structure is often simply added on top of the
streaming functionality (at least true for all .tar.* formats).
BTW: The unfinished stream API blocks lots of projects: Encryption,
Compression, the curl wrapper could be integrated with streams, http
servers and almost everything else related to I/O.

* An improved std.xml. (I think Thomas Sowinski is working on a
replacement, but I haven't seen any updates in a long time.)
* Matrices and linear algebra. (Cristi Cobzarenco's GSoC project is
a good starting point but it needs polish. I've been in contact with
him occasionally since GSoC ended and he indicated that he wants to
get back to working on it but doesn't have time. I've contributed to
it sparingly, but find it difficult because I haven't gotten around to
familiarizing myself with the implementation details yet, and it's
hard to get into a project that complex with a few hours a week as
opposed to focusing full time on it.)
* std.database. (Apparently Steve Teale is working on this. This is
a large, complicated project because we're trying to define a common
API for a variety of RDBMSs. Again, it's more a design problem than
an implementation problem.)
* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve
Schveighoffer ported it to Windows, but issues with the DMC runtime
prevent it from working on Windows.)
* Parallel algorithms. (I've implemented a decent amount of these in
my std.parallel_algorithm Github project, but I've become somewhat
frustrated and unmotivated to finish this project because so many of
the relevant algorithms seem memory bandwidth bound and aren't
substantially faster when parallelized than when run serially.)

Inter-process communication, but as we have thrift now, that might be
ok. (Extending std.concurrency message passing to interprocess
communication with a low overhead serializer like messagepack would
still be great though)

It would be awesome if we could extend it enough to work over TCP or
something similar.

Logging: AFAIK the new log module should be almost ready. I already
used it a few times, but you have to a patch druntime to use
it.
High performance IO: A reactor / proactor framework
A replacement for std.json

After writing this, the general pattern I notice is that lots of stuff
is blocked by design, not implementation. In a lot of cases people
don't really know what they want and analysis paralysis results.

Oh that reminds me, I'd like a nice thing for networking too, where you
can do something like register callback functions for events and it
handles the rest for you.

A replacement for std.json

Blah, that will break a lot of stuff. There is one outstanding proposal in the
wild though, by Robert Jacques, and I think I'd like that.
But, at the same time, I actually use std.json and it's pretty frustrating to
have
the library developers break things so egregiously every other release.

Oh that reminds me, I'd like a nice thing for networking too, where you
can do something like register callback functions for events and it
handles the rest for you.

A replacement for std.json

Blah, that will break a lot of stuff. There is one outstanding proposal in the
wild though, by Robert Jacques, and I think I'd like that.
But, at the same time, I actually use std.json and it's pretty frustrating to
have
the library developers break things so egregiously every other release.

If you know a module will break you can keep your own copy and decide if
you want to migrate to the new module or keep the old one.
--
/Jacob Carlborg

Now that we've got a lot of contributors to Phobos and many projects in
the works, I decided to start a thread to help us make a rough plan for
Phobos's short-to-medium term development. There are three goals here:
1. Determine what's next in the review queue after std.csv (voting on
std.csv ends tonight, so **please vote**).

Isn't it better to add a list on the wiki so we don't have to do this
all over yet again.

2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language than
it is now.

I would really like to see high level networking support, something like
the CURL wrapper.

* Serialization. (Jacob Carolberg's Orange library might be a good
candidate. IIRC he said it's close to ready for review.)

This is ready for (pre-)review and have been that for quite a while now.

* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve
Schveighoffer ported it to Windows, but issues with the DMC runtime
prevent it from working on Windows.)

Wasn't there a fix for the DMC runtime? In that case, why isn't it
already applied?
--
/Jacob Carlborg

* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve
Schveighoffer ported it to Windows, but issues with the DMC runtime
prevent it from working on Windows.)

Wasn't there a fix for the DMC runtime? In that case, why isn't it
already applied?

There is a fix. I've sent it to Walter in the form of "replace this file
with the attached version". I tested the changes by replacing the object
file in the snn.lib library.
Walter has since given me github access to the DMC runtime source, and I
have not been able to successfully build the runtime due to dependence on
obscure assembler tools that Walter has not offered, nor can I download
them. I'm hesitant to submit a pull request for something I can't
successfully build or test from scratch, but I may just submit the
original changes as a pull-request anyways to get the ball rolling.
-Steve

* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve
Schveighoffer ported it to Windows, but issues with the DMC runtime
prevent it from working on Windows.)

Wasn't there a fix for the DMC runtime? In that case, why isn't it
already applied?

There is a fix. I've sent it to Walter in the form of "replace this file
with the attached version". I tested the changes by replacing the object
file in the snn.lib library.
Walter has since given me github access to the DMC runtime source, and I
have not been able to successfully build the runtime due to dependence
on obscure assembler tools that Walter has not offered, nor can I
download them. I'm hesitant to submit a pull request for something I
can't successfully build or test from scratch, but I may just submit the
original changes as a pull-request anyways to get the ball rolling.
-Steve

* Streams. (Another item where the bottleneck is mostly at the design
level and people not really knowing what they want.)

Steven Schveighoffer's new-stdio included a nice stream interface.
I don't know what the current state of the module is.

I'm slowly working on it. I've had very little time these last few months
to work on any extraneous projects.
However, I know the direction this needs to move, and the work is
straightforward. When I get something worth sharing, I'll share it.
Sorry I couldn't be more forthcoming. If you follow my github account,
you should see the new-io branch start to get new commits in the near
future. I really need to get working on this, before too much is built on
top of the existing std.stdio. The new stream interface should be a boon
to most libraries (such as std.xml).
-Steve

* Compression/archiving. (Opening standard compressed/archived file
formats needs to just work. This includes at least zip, gzip, tar and
bzip2. Of course, zip already is available and gzip is supported by the
zlib module but with a crufty C API.

Now that we've got a lot of contributors to Phobos and many projects in
the works, I decided to start a thread to help us make a rough plan for
Phobos's short-to-medium term development. There are three goals here:
1. Determine what's next in the review queue after std.csv (voting on
std.csv ends tonight, so **please vote**).
2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language than
it is now.
3. Figure out who's already working on what from the wish list and what
bottlenecks, if any, are getting in the way and what can be done about
them.
The following is the wish list as I see it. Please suggest additions and
correct any errors, as this is mostly off the top of my head. Also,
status updates if you're working on any of these and anything
substantial has changed would be appreciated.
* Some higher level networking support, such as HTTP, FTP, etc. (Jonas
Drewsen's CURL wrapper handles a lot of this and may be ready for a
second round of review.)

As I've mentioned in another thread it is ready for a second round of
review. We just need someone to step up and run the review since it
wouldn't be apropriate for me to do it myself. Anyone?

* Serialization. (Jacob Carolberg's Orange library might be a good
candidate. IIRC he said it's close to ready for review.)
* Encryption and hashing. (This is more an implementation problem than a
design problem and AFAIK noone is working on it.)
* Containers. (AFAIK noone is working on this. It's tough to get started
because, despite lots of discussion at various times on this forum,
noone seems to really know what they want. Since the containers in
question are well-known, it's much more a design problem than an
implementation problem.)
* Allocators. (I think Phobos desperately needs a segmented stack/region
based allocator and I've written one. I've also tried to define a
generic allocator API, mostly following Andrei's suggestions, but I'll
admit that I didn't really know what I was doing for the general API.
Andrei has suggested that allocators should have real-world testing on
containers before being included in Phobos. Therefore, containers block
allocators and if the same person doesn't write both, there will be a
lot of communication overhead to make sure the designs are in sync.)

I've though about doing some containers myself but have hesitated since
the general opinion seem to be that allocators need to be in place first.

* Streams. (Another item where the bottleneck is mostly at the design
level and people not really knowing what they want.)

What does streams provide that could not be provided by ranges?
/Jonas

* Containers. (AFAIK noone is working on this. It's tough to get started
because, despite lots of discussion at various times on this forum,
noone seems to really know what they want. Since the containers in
question are well-known, it's much more a design problem than an
implementation problem.)

* Allocators. (I think Phobos desperately needs a segmented stack/region
based allocator and I've written one. I've also tried to define a
generic allocator API, mostly following Andrei's suggestions, but I'll
admit that I didn't really know what I was doing for the general API.
Andrei has suggested that allocators should have real-world testing on
containers before being included in Phobos. Therefore, containers block
allocators and if the same person doesn't write both, there will be a
lot of communication overhead to make sure the designs are in sync.)

I've though about doing some containers myself but have hesitated since
the general opinion seem to be that allocators need to be in place first.

Yeah, this is problematic. In voting against my allocator proposal,
Andrei mentioned that he wanted the allocators to be well-tested in the
container API first. This means either we have a circular dependency or
allocators and containers need to be co-developed. Co-developing them
is problematic. If one person does containers and another allocators,
the project might be overwhelmed by communication overhead. If the same
person does both, then this is asking a pretty lot for a hobby project.
Of course, I hope to graduate in <1 year and will be looking for a job
when I do. Any company out there have a strategic interest in D and
want to hire me to work full-time on allocators and containers?

* Streams. (Another item where the bottleneck is mostly at the design
level and people not really knowing what they want.)

What does streams provide that could not be provided by ranges?

If I understand correctly, streams _would_ be a flavor of ranges. They
would just be ranges that are geared towards being stacked on top of one
another specifically for the purpose of I/O. They would typically be
design around the vanilla input range (not forward, random access, etc.)
or output ranges.
Traditionally, streams would also be class based instead of template
based. However, IMHO a good case can be made for template based stream
ranges in D because we have std.range.inputRangeObject and
std.range.outputRangeObject. This means that you can stack streams
using templates, with no virtual function call overhead, and then if you
need to respect some binary interface you could just stack an
inputRangeObject() or outputRangeObject() on top of all your other crap
and only have one virtual call. Example:
auto lines = lineReader(
gzipUncompresser(
rawFile("foo.gz")
)
);
// LineReader!(GzipUncompresser!(RawFile)))
pragma(msg, typeof(lines));
auto objectOriented = inputRangeObject(lines);
// InputRangeObject!(char[])
pragma(msg, typeof(objectOriented));

* Some higher level networking support, such as HTTP, FTP, etc. (Jonas
Drewsen's CURL wrapper handles a lot of this and may be ready for a
second round of review.)

As I've mentioned in another thread it is ready for a second round of
review. We just need someone to step up and run the review since it
wouldn't be apropriate for me to do it myself. Anyone?

If noone else wants to volunteer, I will again. Is there something I'm
missing? I find that being review manage takes very little effort:
Post an initial review announcement, post a reminder, maybe post a
summary/moderation message here and there, post a vote message, post a
vote reminder, tally votes. It really doesn't take much time.

On 11/20/2011 12:30 PM, Jonas Drewsen wrote:
If noone else wants to volunteer, I will again. Is there something I'm
missing? I find that being review manage takes very little effort: Post
an initial review announcement, post a reminder, maybe post a
summary/moderation message here and there, post a vote message, post a
vote reminder, tally votes. It really doesn't take much time.

I think there is at least one thing missing from that:
http://www.boost.org/community/reviews.html#Review_Manager
Making sure it is ready for review :D. There is also "If necessary, work
with the submitter to verify the code compiles and runs correctly on
several compilers and platforms." Not that you must do it yourself, or
that we are collaborating with GDC and LDC yet.

Which, by the way, is a shame. druntime may be compiler-specific, but
Phobos is the D standard library and, as such, should definitely compile
on all mainstream compilers. This ought to be part of the review
process. The problem with doing it, though, is that currently even
Phobos is merged manually into GDC, and LDC maintains a fork...
- Alex

On 11/20/2011 12:30 PM, Jonas Drewsen wrote:
If noone else wants to volunteer, I will again. Is there something I'm
missing? I find that being review manage takes very little effort: Post
an initial review announcement, post a reminder, maybe post a
summary/moderation message here and there, post a vote message, post a
vote reminder, tally votes. It really doesn't take much time.

I think there is at least one thing missing from that:
http://www.boost.org/community/reviews.html#Review_Manager
Making sure it is ready for review :D. There is also "If necessary, work
with the submitter to verify the code compiles and runs correctly on
several compilers and platforms." Not that you must do it yourself, or
that we are collaborating with GDC and LDC yet.

Yeah, this is problematic. In voting against my allocator proposal,
Andrei mentioned that he wanted the allocators to be well-tested in the
container API first. This means either we have a circular dependency or
allocators and containers need to be co-developed. Co-developing them
is problematic. If one person does containers and another allocators,
the project might be overwhelmed by communication overhead. If the same
person does both, then this is asking a pretty lot for a hobby project.

around some time ago.
If I remember correctly it was found to be a useful addition.
Why not adding the current allocator interface with say a GC-only
implementation there
so that container development can start off and provide the necessary
feedback.
martin

* std.database. (Apparently Steve Teale is working on this. This is a
large, complicated project because we're trying to define a common API
for a variety of RDBMSs. Again, it's more a design problem than an
implementation problem.)

agreement on design, and there are implementation difficulties caused by
licensing issues and the fact that some potentially useful approaches
don't appear to work - ODBC for example is a nightmare.
I am trying to sort out what is possible and what isn't as quickly as
possible. Then when the current massive web site debate has wound down I
might try to get people to return to the design issues.
Steve

Now that we've got a lot of contributors to Phobos and many projects in
the works, I decided to start a thread to help us make a rough plan for
Phobos's short-to-medium term development. There are three goals here:
1. Determine what's next in the review queue after std.csv (voting on
std.csv ends tonight, so **please vote**).
2. Come up with a wish list of high-priority modules that Phobos is
missing that would make D a substantially more attractive language than it
is now.
3. Figure out who's already working on what from the wish list and what
bottlenecks, if any, are getting in the way and what can be done about them.
The following is the wish list as I see it. Please suggest additions and
correct any errors, as this is mostly off the top of my head. Also, status
updates if you're working on any of these and anything substantial has
changed would be appreciated.
* Some higher level networking support, such as HTTP, FTP, etc. (Jonas
Drewsen's CURL wrapper handles a lot of this and may be ready for a second
round of review.)
* Serialization. (Jacob Carolberg's Orange library might be a good
candidate. IIRC he said it's close to ready for review.)
* Encryption and hashing. (This is more an implementation problem than a
design problem and AFAIK noone is working on it.)
* Containers. (AFAIK noone is working on this. It's tough to get
started because, despite lots of discussion at various times on this forum,
noone seems to really know what they want. Since the containers in
question are well-known, it's much more a design problem than an
implementation problem.)
* Allocators. (I think Phobos desperately needs a segmented stack/region
based allocator and I've written one. I've also tried to define a generic
allocator API, mostly following Andrei's suggestions, but I'll admit that I
didn't really know what I was doing for the general API. Andrei has
suggested that allocators should have real-world testing on containers
before being included in Phobos. Therefore, containers block allocators and
if the same person doesn't write both, there will be a lot of communication
overhead to make sure the designs are in sync.)
* Streams. (Another item where the bottleneck is mostly at the design
level and people not really knowing what they want.)
* Compression/archiving. (Opening standard compressed/archived file
formats needs to just work. This includes at least zip, gzip, tar and
bzip2. Of course, zip already is available and gzip is supported by the
zlib module but with a crufty C API. At least gzip and bzip2, which are
stream-based as opposed to file-based, should be handled via streams, which
means that streams block compression/archiving. Also, since tar and zip
are both file based, they should probably be handled by the same API, which
might mean deprecating std.zip and rewriting it.)
* An improved std.xml. (I think Thomas Sowinski is working on a
replacement, but I haven't seen any updates in a long time.)
* Matrices and linear algebra. (Cristi Cobzarenco's GSoC project is a
good starting point but it needs polish. I've been in contact with him
occasionally since GSoC ended and he indicated that he wants to get back to
working on it but doesn't have time. I've contributed to it sparingly, but
find it difficult because I haven't gotten around to familiarizing myself
with the implementation details yet, and it's hard to get into a project
that complex with a few hours a week as opposed to focusing full time on
it.)
* std.database. (Apparently Steve Teale is working on this. This is a
large, complicated project because we're trying to define a common API for
a variety of RDBMSs. Again, it's more a design problem than an
implementation problem.)
* Better support for creating processes/new std.process. (Lars
Kyllingstad wrote a replacement candidate for Posix and Steve Schveighoffer
ported it to Windows, but issues with the DMC runtime prevent it from
working on Windows.)
* Parallel algorithms. (I've implemented a decent amount of these in my
std.parallel_algorithm Github project, but I've become somewhat frustrated
and unmotivated to finish this project because so many of the relevant
algorithms seem memory bandwidth bound and aren't substantially faster when
parallelized than when run serially.)
After writing this, the general pattern I notice is that lots of stuff is
blocked by design, not implementation. In a lot of cases people don't
really know what they want and analysis paralysis results.

As a console/embedded games developer, my top 4:
*Allocators* - +1 .. GC is lovely to have, but in many cases, I need strict
control over my memory management. Any libraries that use the GC internally
are potentially unusable to me in some applications.
*Low level/hardware vector-maths library* - This is kinda blocked by the
lack of support for an intrinsic vector type in the compiler... the
language needs at very least a 128bit vector type to express the registers
and make it possible to allocate them in structures, and pass them between
functions. Architecture specific compilers/libraries can then build on top
of that, and then ideally, a low level maths library in Phobos might
collate various architectures vector hardware support into a common API.
I would happily volunteer to to handle this, but I need at least a 128bit
hardware vector type in the language first (GDC has this though proprietary
extension, rather, the language should define a standard). I guess I could
design the API without it, but the library would only really work
efficiently on flavours of GCC.
*Crypto *- +1 as mentioned above, blocked on streams...
*Higher level maths/signal processing *- The linear algebra library
mentioned in a previous post might be very useful, but also standard
algorithms like fourier transforms. Library implementations of these would
be amazing, potentially making use of hardware back ends where possible.
--0016e64ea22a21845404b267abcf
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div class=3D"gmail_quote">On 20 November 2011 05:02, dsimcha <span dir=3D"=
ltr">&lt;<a href=3D"mailto:dsimcha yahoo.com">dsimcha yahoo.com</a>&gt;</sp=
an> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;=
border-left:1px #ccc solid;padding-left:1ex;">
Now that we&#39;ve got a lot of contributors to Phobos and many projects in=
the works, I decided to start a thread to help us make a rough plan for Ph=
obos&#39;s short-to-medium term development. =C2=A0There are three goals he=
re:<br>
<br>
1. =C2=A0Determine what&#39;s next in the review queue after std.csv (votin=
g on std.csv ends tonight, so **please vote**).<br>
<br>
2. =C2=A0Come up with a wish list of high-priority modules that Phobos is m=
issing that would make D a substantially more attractive language than it i=
s now.<br>
<br>
3. =C2=A0Figure out who&#39;s already working on what from the wish list an=
d what bottlenecks, if any, are getting in the way and what can be done abo=
ut them.<br>
<br>
The following is the wish list as I see it. =C2=A0Please suggest additions =
and correct any errors, as this is mostly off the top of my head. =C2=A0Als=
o, status updates if you&#39;re working on any of these and anything substa=
ntial has changed would be appreciated.<br>
<br>
* =C2=A0Some higher level networking support, such as HTTP, FTP, etc. =C2=
=A0(Jonas Drewsen&#39;s CURL wrapper handles a lot of this and may be ready=
for a second round of review.)<br>
<br>
* =C2=A0Serialization. =C2=A0(Jacob Carolberg&#39;s Orange library might be=
a good candidate. =C2=A0IIRC he said it&#39;s close to ready for review.)<=
br>
<br>
* =C2=A0Encryption and hashing. =C2=A0(This is more an implementation probl=
em than a design problem and AFAIK noone is working on it.)<br>
<br>
* =C2=A0Containers. =C2=A0(AFAIK noone is working on this. =C2=A0It&#39;s t=
ough to get started because, despite lots of discussion at various times on=
this forum, noone seems to really know what they want. =C2=A0Since the con=
tainers in question are well-known, it&#39;s much more a design problem tha=
n an implementation problem.)<br>
<br>
* =C2=A0Allocators. =C2=A0(I think Phobos desperately needs a segmented sta=
ck/region based allocator and I&#39;ve written one. =C2=A0I&#39;ve also tri=
ed to define a generic allocator API, mostly following Andrei&#39;s suggest=
ions, but I&#39;ll admit that I didn&#39;t really know what I was doing for=
the general API. =C2=A0Andrei has suggested that allocators should have re=
al-world testing on containers before being included in Phobos. Therefore, =
containers block allocators and if the same person doesn&#39;t write both, =
there will be a lot of communication overhead to make sure the designs are =
in sync.)<br>
<br>
* =C2=A0Streams. =C2=A0(Another item where the bottleneck is mostly at the =
design level and people not really knowing what they want.)<br>
<br>
* =C2=A0Compression/archiving. =C2=A0(Opening standard compressed/archived =
file formats needs to just work. =C2=A0This includes at least zip, gzip, ta=
r and bzip2. =C2=A0Of course, zip already is available and gzip is supporte=
d by the zlib module but with a crufty C API. =C2=A0At least gzip and bzip2=
, which are stream-based as opposed to file-based, should be handled via st=
reams, which means that streams block compression/archiving. =C2=A0Also, si=
nce tar and zip are both file based, they should probably be handled by the=
same API, which might mean deprecating std.zip and rewriting it.)<br>
<br>
* =C2=A0An improved std.xml. =C2=A0(I think Thomas Sowinski is working on a=
replacement, but I haven&#39;t seen any updates in a long time.)<br>
<br>
* =C2=A0Matrices and linear algebra. =C2=A0(Cristi Cobzarenco&#39;s GSoC pr=
oject is a good starting point but it needs polish. =C2=A0I&#39;ve been in =
contact with him occasionally since GSoC ended and he indicated that he wan=
ts to get back to working on it but doesn&#39;t have time. =C2=A0I&#39;ve c=
ontributed to it sparingly, but find it difficult because I haven&#39;t got=
ten around to familiarizing myself with the implementation details yet, and=
it&#39;s hard to get into a project that complex with a few hours a week a=
s opposed to focusing full time on it.)<br>
<br>
* =C2=A0std.database. =C2=A0(Apparently Steve Teale is working on this. =C2=
=A0This is a large, complicated project because we&#39;re trying to define =
a common API for a variety of RDBMSs. =C2=A0Again, it&#39;s more a design p=
roblem than an implementation problem.)<br>
<br>
* =C2=A0Better support for creating processes/new std.process. =C2=A0(Lars =
Kyllingstad wrote a replacement candidate for Posix and Steve Schveighoffer=
ported it to Windows, but issues with the DMC runtime prevent it from work=
ing on Windows.)<br>
<br>
* =C2=A0Parallel algorithms. =C2=A0(I&#39;ve implemented a decent amount of=
these in my std.parallel_algorithm Github project, but I&#39;ve become som=
ewhat frustrated and unmotivated to finish this project because so many of =
the relevant algorithms seem memory bandwidth bound and aren&#39;t substant=
ially faster when parallelized than when run serially.)<br>
<br>
After writing this, the general pattern I notice is that lots of stuff is b=
locked by design, not implementation. =C2=A0In a lot of cases people don&#3=
9;t really know what they want and analysis paralysis results.<br>
</blockquote></div><br><div>As a console/embedded games developer, my top 4=
:</div><div><br></div><div><b>Allocators</b> - +1 .. GC is lovely to have, =
but in many cases, I need strict control over my memory management. Any lib=
raries that use the GC internally are potentially unusable to me in some ap=
plications.</div>
<div><br></div><div><b>Low level/hardware vector-maths library</b> - This i=
s kinda blocked by the lack of support for an intrinsic vector type in the =
compiler... the language needs at very least a 128bit vector type to expres=
s the registers and make it possible to allocate them in structures, and pa=
ss them between functions. Architecture specific compilers/libraries can th=
en build on top of that, and then ideally, a low level maths library in Pho=
bos might collate various architectures vector hardware support into a comm=
on API.</div>
<div>I would happily volunteer to to handle this, but I need at least a 128=
bit hardware vector type in the language first (GDC has this though proprie=
tary extension, rather, the language should define a standard). I guess I c=
ould design the API without it, but the library would only really work effi=
ciently on flavours of GCC.</div>
<div><br></div><div><b>Crypto=C2=A0</b>- +1 as mentioned above, blocked on =
streams...</div><div><br></div><div><b>Higher level maths/signal processing=
=C2=A0</b>- The linear algebra library mentioned in a previous post might b=
e very useful, but also standard algorithms like fourier transforms. Librar=
y implementations of these would be amazing, potentially making use of hard=
ware back ends where possible.</div>
--0016e64ea22a21845404b267abcf--

*Crypto *- +1 as mentioned above, blocked on streams...
*Higher level maths/signal processing *- The linear algebra library
mentioned in a previous post might be very useful, but also standard
algorithms like fourier transforms. Library implementations of these
would

The performance is actually quite good. I'd only wish for a real to
half-spectrum specialization (probably in-place) performance wise.

be amazing, potentially making use of hardware back ends where possible.

--001636833ee854059a04b26a2327
Content-Type: text/plain; charset=UTF-8
Oh wow, awesome! :)
I'm always surprised by the state of the D libraries.. There's already lots
of awesome obscure things are in there, but also completely obvious major
features blatantly missing :)
So how about a DCT? Image/video processing... Perhaps a simple adaptation
of FFT?
And it's not clear to me at all whether that class will work with fixed
point data... That's extremely important for a class like this (but
granted, not really in line with D's take on numeric precision).
Almost all audio and video data is in fixed point. Int->float conversion is
very expensive on many platforms.
On 23 November 2011 17:34, Martin Nowak <dawg dawgfoto.de> wrote:

*Crypto *- +1 as mentioned above, blocked on streams...

*Higher level maths/signal processing *- The linear algebra library
mentioned in a previous post might be very useful, but also standard
algorithms like fourier transforms. Library implementations of these would

The performance is actually quite good. I'd only wish for a real to
half-spectrum specialization (probably in-place) performance wise.
be amazing, potentially making use of hardware back ends where possible.

Oh wow, awesome! :)
I'm always surprised by the state of the D libraries.. There's already
lots
of awesome obscure things are in there, but also completely obvious major
features blatantly missing :)

So how about a DCT? Image/video processing... Perhaps a simple adaptation
of FFT?

And it's not clear to me at all whether that class will work with fixed
point data... That's extremely important for a class like this (but
granted, not really in line with D's take on numeric precision).
Almost all audio and video data is in fixed point. Int->float conversion
is
very expensive on many platforms.

implementation will be very tough, the log2N gain of calculating
intermediate
results is directly opposed to precision.
It would be interesting to offer vectorized 'Integral <-> Float'
conversions.

On 23 November 2011 17:34, Martin Nowak <dawg dawgfoto.de> wrote:

*Crypto *- +1 as mentioned above, blocked on streams...

*Higher level maths/signal processing *- The linear algebra library
mentioned in a previous post might be very useful, but also standard
algorithms like fourier transforms. Library implementations of these
would

The performance is actually quite good. I'd only wish for a real to
half-spectrum specialization (probably in-place) performance wise.
be amazing, potentially making use of hardware back ends where
possible.

Oh wow, awesome! :) I'm always surprised by the state of the D
libraries.. There's already lots of awesome obscure things are in
there, but also completely obvious major features blatantly missing
:)
So how about a DCT? Image/video processing... Perhaps a simple
adaptation of FFT?

Mostly interesting things like ZeroMQ, I also have a good Binding for
CZMQ that needs to get into Deimos. If you wanted something
higher-level than ZeroMQ, the project is here:
https://github.com/1100110/CZMQ
If you want me to consider working on a specific binding to a C
library, just drop me an email or something.
The small ones like ZeroMQ can be done in a few hours or so. A little
longer to test properly.
I'd rather work on something that I know will be used, than my little
pet projects which will probably not be used for awhile.

And it's not clear to me at all whether that class will work with
fixed point data... That's extremely important for a class like
this (but granted, not really in line with D's take on numeric
precision). Almost all audio and video data is in fixed point.
Int->float conversion is very expensive on many platforms.
On 23 November 2011 17:34, Martin Nowak <dawg dawgfoto.de> wrote:

*Crypto *- +1 as mentioned above, blocked on streams...

*Higher level maths/signal processing *- The linear algebra
library
mentioned in a previous post might be very useful, but also
standard algorithms like fourier transforms. Library
implementations of these would

half-spectrum specialization (probably in-place) performance
wise.
be amazing, potentially making use of hardware back ends where
possible.

I'm always surprised by the state of the D libraries.. There's already
lots
of awesome obscure things are in there, but also completely obvious major
features blatantly missing :)

So how about a DCT? Image/video processing... Perhaps a simple adaptation
of FFT?
Too many different DCTs, too many different needs I'd say.

Well... hardware accelerated video decoders have boiled it down to a
standard set of functions, so it's obviously not an endless amount, and I'd
suggest someone else (who wrote those apis) has already done the work of
identifying the ones that allow you to process most modern formats.

And it's not clear to me at all whether that class will work with fixed

point data... That's extremely important for a class like this (but
granted, not really in line with D's take on numeric precision).
Almost all audio and video data is in fixed point. Int->float conversion
is
very expensive on many platforms.
Too specialized. Given the obscurity of doing a fixed point FFT, the

implementation will be very tough, the log2N gain of calculating
intermediate
results is directly opposed to precision.

Obscurity? MP3 isn't obscure. I'm sure there are bunches of reference
implementations to draw from.
But again, I couldn't use the standard library on a PSP or iPhone for
instance if there is no fixed point version. And I'm left in a world where
I have to roll my own, just like I always do in C with respect to the the
CRT, because the standard library just isn't thorough >_<
It would be interesting to offer vectorized 'Integral <-> Float'

conversions.

No reason not to. If D ever gets a hardware vector type, I'll start writing
that library the same day...
That doesn't address this issue though, I'm concerned with a lot of
architectures that either don't have float units, or extremely slow float
processing. Hand held gaming systems and phones.
I do think fixed point versions of these data shovelling DSP functions is
quite important to many people, but I'm sure 3rd party libs will come.
--0015174c15b85c876c04b26d22dd
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div class=3D"gmail_quote">On 23 November 2011 21:35, Martin Nowak <span di=
r=3D"ltr">&lt;<a href=3D"mailto:dawg dawgfoto.de">dawg dawgfoto.de</a>&gt;<=
/span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class=3D"im">On Wed, 23 Nov 2011 18:20:35 +0100, Manu &lt;<a href=3D"m=
ailto:turkeyman gmail.com" target=3D"_blank">turkeyman gmail.com</a>&gt; wr=
ote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Oh wow, awesome! :)<br>
I&#39;m always surprised by the state of the D libraries.. There&#39;s alre=
ady lots<br>
of awesome obscure things are in there, but also completely obvious major<b=
r>
features blatantly missing :)<br>
</blockquote></div>
There are some real gems, e.g. findRoot.<div class=3D"im"><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
<br>
So how about a DCT? Image/video processing... Perhaps a simple adaptation<b=
r>
of FFT?<br>
<br>
</blockquote></div>
Too many different DCTs, too many different needs I&#39;d say.</blockquote>=
<div><br></div><div>Well... hardware accelerated video decoders have boiled=
it down to a standard set of functions, so it&#39;s obviously not an endle=
ss amount, and I&#39;d suggest someone else (who wrote those apis) has alre=
ady done the work of identifying the ones that allow you to process most mo=
dern formats.</div>
<div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex;"><div class=3D"im">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
And it&#39;s not clear to me at all whether that class will work with fixed=
<br>
point data... That&#39;s extremely important for a class like this (but<br>
granted, not really in line with D&#39;s take on numeric precision).<br>
Almost all audio and video data is in fixed point. Int-&gt;float conversion=
is<br>
very expensive on many platforms.<br>
<br>
</blockquote></div>
Too specialized. Given the obscurity of doing a fixed point FFT, the actual=
<br>
implementation will be very tough, the log2N gain of calculating intermedia=
te<br>
results is directly opposed to precision.<br></blockquote><div><br></div><d=
iv>Obscurity? MP3 isn&#39;t obscure. I&#39;m sure there are bunches of refe=
rence implementations to draw from.</div><div>But again, I couldn&#39;t use=
the standard library on a PSP or iPhone for instance if there is no fixed =
point version. And I&#39;m left in a world where I have to roll my own, jus=
t like I always do in C with respect to the the CRT, because the standard l=
ibrary just isn&#39;t thorough &gt;_&lt;</div>
<div>=C2=A0</div><div><br></div><blockquote class=3D"gmail_quote" style=3D"=
margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
It would be interesting to offer vectorized &#39;Integral &lt;-&gt; Float&#=
39; conversions.<br></blockquote><div><br></div><div>No reason not to. If D=
ever gets a hardware vector type, I&#39;ll start writing that library the =
same day...</div>
<div>That doesn&#39;t address this issue though, I&#39;m concerned with a l=
ot of architectures that either don&#39;t have float units, or extremely sl=
ow float processing. Hand held gaming systems and phones.</div><div>I do th=
ink fixed point versions of these data=C2=A0shovelling=C2=A0DSP functions i=
s quite important to many people, but I&#39;m sure 3rd party libs will come=
.</div>
</div>
--0015174c15b85c876c04b26d22dd--

* Compression/archiving. (Opening standard compressed/archived file formats
needs to just work. This includes at least zip, gzip, tar and bzip2. Of course,
zip already is available and gzip is supported by the zlib module but with a
crufty C API. At least gzip and bzip2, which are stream-based as opposed to
file-based, should be handled via streams, which means that streams block
compression/archiving. Also, since tar and zip are both file based, they should
probably be handled by the same API, which might mean deprecating std.zip and
rewriting it.)

* Streams. (Another item where the bottleneck is mostly at the design
level and
people not really knowing what they want.)

I'm not sure what the purpose of streams would be, now that we have ranges.

Right. As I mentioned in a previous post buried deep in this thread, I
think streams should just be a flavor of ranges that have most or all of
the following characteristics:
1. Live in std.stream.
2. Oriented toward I/O.
3. Heavy use of higher order ranges/stacking for things like
compression/decompression and encryption/decryption.
4. Mostly focused on input ranges as opposed to random
access/forward/bidirectional, since this is the best model for data from
a network or stdin.

* Streams. (Another item where the bottleneck is mostly at the design
level and
people not really knowing what they want.)

I'm not sure what the purpose of streams would be, now that we have
ranges.

think streams should just be a flavor of ranges that have most or all of
the following characteristics:
1. Live in std.stream.
2. Oriented toward I/O.
3. Heavy use of higher order ranges/stacking for things like
compression/decompression and encryption/decryption.
4. Mostly focused on input ranges as opposed to random
access/forward/bidirectional, since this is the best model for data from
a network or stdin.

Andrei's plan for streams was discussed a fair bit a while back. IIRC, it
essentially involved a lower level range API which took chunks of elements
rather than elements such that you were essentially dealing with a buffer that
was reading in the data. A higher level range would then be put on top of that
range which presented a more normal range API. So, a lower level stream of
ubytes would be return ubyte[] for its front (the buffer which was ubyte[]),
and then a higher level range which would return ubyte for its front would be
on top of that. I'd have to go digging through the archives though to see
exactly what Andrei was proposing, but that's more or less what I recall.
AFAIK, he still intends for something along those lines to be implemented, but
no one has done it yet.
- Jonathan M Davis

* Streams. (Another item where the bottleneck is mostly at the design
level and
people not really knowing what they want.)

I'm not sure what the purpose of streams would be, now that we have
ranges.

We still need streams for low-level i/o. e.g. FILE * is a stream.
ranges are good for high-level concepts, like a range of lines from a
stream. Or a range of XML nodes. Ranges are not good for reading N bytes
from a file descriptor.
-Steve

On 26 November 2011 23:39, Walter Bright <newshound2 digitalmars.com
<mailto:newshound2 digitalmars.com>> wrote:
On 11/26/2011 5:46 AM, Steven Schveighoffer wrote:
Ranges are not good for reading N bytes from a file
descriptor.
Why not? Isn't that exactly what a range is supposed to be good for?
It sounds like a bad idea to me... I can imagine ending up with a whole
pile of allocations of short ranges, and then people doing bunches of
range concatenations to re-assemble the buffer. That's a lot of
pointless allocation and memcopying. D's apis should discourage
inefficient usage like that.

I think ranges are here to avoid memcopying and pointless allocations.

e ending up with a whole pile of allocations of short ranges, and then peop=
le doing bunches of range=C2=A0concatenations to re-assemble the buffer. Th=
at&#39;s a lot of pointless allocation and memcopying. D&#39;s apis should =
discourage inefficient usage like that.</div>
--20cf300fb4eb24e41604b2b51f8b--

A range has a specific interface:
T front()
void popFront()
bool empty()
An input stream has a specific interface:
size_t read(ubyte[] buffer)
or alternatively:
ubyte[] read(ubyte[] buffer)
How does one map that interface into an input range? What is T? I
remember proposals where there was some extra function setFrontBytes, or
an additional parameter to front, that allows you to set the number of
bytes to read. This seems like trying to fit a square peg in a round hole.
Now, a range fits very well on top of an input stream with a given
context. That context defines *what* front() is going to return. For
example, the next line of text.
But you still need a well-designed buffered stream interface that a range
can use to do its work from. Streams are not going to be the main
interface for most users, but they will be the tools that range developers
use to create easily iterated streams. And when the user wants to read
data from a file that has no well-defined T that you can define a range
for, the stream should provide all the tools necessary to read that data.
Now, an output stream *does* fit quite well as a range, since an output
range's single put method coincides with an ouptut stream's write method.
-Steve

A range has a specific interface:
T front()
void popFront()
bool empty()
An input stream has a specific interface:
size_t read(ubyte[] buffer)
or alternatively:
ubyte[] read(ubyte[] buffer)
How does one map that interface into an input range? What is T? I
remember proposals where there was some extra function setFrontBytes, or
an additional parameter to front, that allows you to set the number of
bytes to read. This seems like trying to fit a square peg in a round
hole.
Now, a range fits very well on top of an input stream with a given
context. That context defines *what* front() is going to return. For
example, the next line of text.
But you still need a well-designed buffered stream interface that a
range can use to do its work from. Streams are not going to be the main
interface for most users, but they will be the tools that range
developers use to create easily iterated streams. And when the user
wants to read data from a file that has no well-defined T that you can
define a range for, the stream should provide all the tools necessary to
read that data.
Now, an output stream *does* fit quite well as a range, since an output
range's single put method coincides with an ouptut stream's write method.
-Steve

I was absent from range discussions, probably it has been said, sorry.
A ### range means, a structure which implements:
T front()
void popFront()
bool empty()
This matches with:
T peek()
void seek(1)
bool eof()
T is ubyte.
I believe we are asking the wrong questions, first question should be why
someone would like this simple range interface (3 functions) for streams?
Any use? Yes with these 3 we can do quite many things, not optimal but we
can. For the real use we need a new range.
T[] get(size_t)
bool put(size_t, T[])
bool seek(size_t)
...
Don't we need a range like this in any case? I doubt we can do much
without it.

A range has a specific interface:
T front()
void popFront()
bool empty()
An input stream has a specific interface:
size_t read(ubyte[] buffer)
or alternatively:
ubyte[] read(ubyte[] buffer)
How does one map that interface into an input range? What is T? I
remember proposals where there was some extra function setFrontBytes,
or an additional parameter to front, that allows you to set the number
of bytes to read. This seems like trying to fit a square peg in a
round hole.
Now, a range fits very well on top of an input stream with a given
context. That context defines *what* front() is going to return. For
example, the next line of text.
But you still need a well-designed buffered stream interface that a
range can use to do its work from. Streams are not going to be the
main interface for most users, but they will be the tools that range
developers use to create easily iterated streams. And when the user
wants to read data from a file that has no well-defined T that you can
define a range for, the stream should provide all the tools necessary
to read that data.
Now, an output stream *does* fit quite well as a range, since an output
range's single put method coincides with an ouptut stream's write
method.
-Steve

The task is not to see if a stream can implement a range, it's to see if a
range can implement a stream. The target interface is the read function I
mentioned.
Nobody wants to use the "one byte at a time" input range to interface with
streams. This just furthers my point.

I believe we are asking the wrong questions, first question should be
why someone would like this simple range interface (3 functions) for
streams?
Any use? Yes with these 3 we can do quite many things, not optimal but
we can. For the real use we need a new range.
T[] get(size_t)
bool put(size_t, T[])
bool seek(size_t)
...
Don't we need a range like this in any case? I doubt we can do much
without it.

Yes, but let's not call this a "range", since it does not work in
functions that use ranges or with foreach (this is Walter's goal).
Ranges on top of streams make sense. The most classic is byLine. But
byLine is not going to use a range interface to do it's underlying work.
IMO the layers from lowest to highest are:
device stream -> buffered stream -> range
In fact, that's *exactly* what phobos does today. buffered stream (and
the underlying device stream) is FILE *.
I'm working on a replacement for FILE * which allows much more
flexibility, performance, while std.stdio will maintain the exact same
interface.
-Steve