tcl-core

This is a Call For Votes on TIPs 400 and 405.
TIP #400: Setting the Compression Dictionary and Other 'zlib' Updates
http://tip.tcl.tk/400.html
Sometimes it is necessary to set the compression dictionary so that
a sequence of bytes may be compressed more efficiently (and
decompressed as well). This TIP exposes that functionality. It also
reduces the number of inconsistencies in the zlib command.
TIP #405: Add Collecting Loops, the 'lmap' and 'dict map' Commands
http://tip.tcl.tk/405.html
The lmap command is a collecting loop with the semantics of
foreach. When the loop begins an accumulator is set to an empty
list. In any iteration where the body of the loop completes
normally, the result of the body is appended to the accumulator
list. The return value for lmap is the contents of the accumulator.
The dict map command is an equivalent collecting loop with the
semantics based off dict for.
In each case, I think they're about as good as they're going to get.
Please send your votes to this mailing list by 12:00 UK time (11:00 UTC)
next Tuesday ([clock format 1350385200]) in the normal format. My vote
follows below:
TIP #400: YES
TIP #405: YES
Donal.

On 10/9/2012 6:19 AM, Donal K. Fellows wrote:
> This is a Call For Votes on TIPs 400 and 405.
TIP#400 YES
TIP#405 YES
>
> TIP #400: Setting the Compression Dictionary and Other 'zlib' Updates
> http://tip.tcl.tk/400.html
>
> Sometimes it is necessary to set the compression dictionary so that
> a sequence of bytes may be compressed more efficiently (and
> decompressed as well). This TIP exposes that functionality. It also
> reduces the number of inconsistencies in the zlib command.
>
> TIP #405: Add Collecting Loops, the 'lmap' and 'dict map' Commands
> http://tip.tcl.tk/405.html
>
> The lmap command is a collecting loop with the semantics of
> foreach. When the loop begins an accumulator is set to an empty
> list. In any iteration where the body of the loop completes
> normally, the result of the body is appended to the accumulator
> list. The return value for lmap is the contents of the accumulator.
> The dict map command is an equivalent collecting loop with the
> semantics based off dict for.
>
> In each case, I think they're about as good as they're going to get.
>
> Please send your votes to this mailing list by 12:00 UK time (11:00 UTC)
> next Tuesday ([clock format 1350385200]) in the normal format. My vote
> follows below:
>
> TIP #400: YES
> TIP #405: YES
>
> Donal.
--
Andreas Kupries
Senior Tcl Developer
Code to Cloud: Smarter, Safer, Faster™
P: 778.786.1122
F: 778.786.1133
andreask@...
http://www.activestate.com
Learn about Stackato for Private PaaS: http://www.activestate.com/stackatohttp://www.tcl.tk/community/tcl2012/ - Tcl'2012, Nov 12-16, Chicago, IL, USA.

On 10/09/2012 09:19 AM, Donal K. Fellows wrote:
> This is a Call For Votes on TIPs 400 and 405.
TIP #400: YES.
My first reaction: "Oh, $env(LC_DEITY), what a bloat of Core
functionality with this obscure operation." But following the logic:
1. We want zlib in the core so that we can eventually support a
core-level VFS that depends on it, and so that Tk can support
PNG natively.
2. Several things (like SPDY) need to prime the compression dictionary.
3. If the zlib in core doesn't export this functionality, then a
SPDY extension would have to redo the zlib support.
So, failing a vote of "IF YOU MUST", I'll vote YES.
TIP 405: YES.
The function is obvious to a Lisp programmer. And I don't know how
many times I've written:
set l {}
foreach elt $list {
lappend l [do-something-to $l]
}
return $l
(This, in turn, implies that we have the requisite bytecode gubbins
to compile these things. That's a relief.)
--
73 de ke9tv/2, Kevin
>
> TIP #400: Setting the Compression Dictionary and Other 'zlib' Updates
> http://tip.tcl.tk/400.html
>
> Sometimes it is necessary to set the compression dictionary so that
> a sequence of bytes may be compressed more efficiently (and
> decompressed as well). This TIP exposes that functionality. It also
> reduces the number of inconsistencies in the zlib command.
>
> TIP #405: Add Collecting Loops, the 'lmap' and 'dict map' Commands
> http://tip.tcl.tk/405.html
>
> The lmap command is a collecting loop with the semantics of
> foreach. When the loop begins an accumulator is set to an empty
> list. In any iteration where the body of the loop completes
> normally, the result of the body is appended to the accumulator
> list. The return value for lmap is the contents of the accumulator.
> The dict map command is an equivalent collecting loop with the
> semantics based off dict for.
>
> In each case, I think they're about as good as they're going to get.
>
> Please send your votes to this mailing list by 12:00 UK time (11:00 UTC)
> next Tuesday ([clock format 1350385200]) in the normal format. My vote
> follows below:
>
> TIP #400: YES
> TIP #405: YES
>
> Donal.
>
>
> ------------------------------------------------------------------------------
> Don't let slow site performance ruin your business. Deploy New Relic APM
> Deploy New Relic app performance management and know exactly
> what is happening inside your Ruby, Python, PHP, Java, and .NET app
> Try New Relic at no cost today and get our sweet Data Nerd shirt too!
> http://p.sf.net/sfu/newrelic-dev2dev
>
>
>
> _______________________________________________
> Tcl-Core mailing list
> Tcl-Core@...
> https://lists.sourceforge.net/lists/listinfo/tcl-core
>

On 10/15/2012 08:26 PM, Larry McVoy wrote:
> Before you get all excited about zlib in the core, let me point you at
> lz4. The tl;dr - zlib -6 compresses better but is slower in both
> directions (MUCH slower), zlib compresses about the same but is still
> slower. lz4 compresses faster than gig ether (about 300MB/sec) and
> uncompresses at about 1.5GB/sec.
Is it the same compression - that is, something compressed by zlib will
uncompress with lz4, and vice versa?
Because otherwise it fails to satisfy the chief reason for zlib in the
core, which is for data interchange.
--
73 de ke9tv/2, Kevin

On Mon, Oct 15, 2012 at 08:37:21PM -0400, Kevin Kenny wrote:
> On 10/15/2012 08:26 PM, Larry McVoy wrote:
> > Before you get all excited about zlib in the core, let me point you at
> > lz4. The tl;dr - zlib -6 compresses better but is slower in both
> > directions (MUCH slower), zlib compresses about the same but is still
> > slower. lz4 compresses faster than gig ether (about 300MB/sec) and
> > uncompresses at about 1.5GB/sec.
>
> Is it the same compression - that is, something compressed by zlib will
> uncompress with lz4, and vice versa?
>
> Because otherwise it fails to satisfy the chief reason for zlib in the
> core, which is for data interchange.
It's not compat at all. It's just compression that is faster than disk
so it's basically free. zlib is much slower than disk so it's not free.
--
---
Larry McVoy lm at bitmover.com http://www.bitkeeper.com

On 16/10/2012 01:49, Larry McVoy wrote:
> It's not compat at all. It's just compression that is faster than disk
> so it's basically free. zlib is much slower than disk so it's not free.
First off, I'm not writing a production-grade compression algorithm. I'm
only going to use others' libraries. (High-quality compression is nearly
as brain-bending as cryptography.)
The principal use cases for zlib support are:
* Compression over HTTP connections, where gzip is the most widely
supported compression format (for some ugly reasons).
* Handling PNG images, where compression is pretty much mandated
(given that we would want to load PNG images created by other tools;
that's actually more important than shrinking the data). I forget
whether it's the actual zlib format or the raw compressed data
format that is used; I was one of the people who hacked the code
together in Tk to do PNG read/write, but we did it in a great hurry
and were much more concerned about whether we were doing the right
Tcl C API. :-)
Getting maximal compression is actually not a critical part of either of
those use cases, but using the zlib formats *is* because others do. The
case for zlib being in the core is that the packages above (http, Tk)
really don't have that much to do with each other and that they're both
"core distributed"; given that, everything else is just about getting a
competent implementation.
We can do lz4 some other time/elsewhere if demand is enough. I think
there's enough experience now (there's some real subtleties) that we can
help people wanting to do it as a stacked channel transform in an
extension. That's the first step anyway for something like this that has
limited API impact on the rest of Tcl.
Donal.

On Tue, Oct 16, 2012 at 2:20 AM, Kevin Kenny <kkenny2@...> wrote:
> TIP 405: YES.
>
> The function is obvious to a Lisp programmer. And I don't know how
> many times I've written:
>
> set l {}
> foreach elt $list {
> lappend l [do-something-to $l]
> }
> return $l
>
> (This, in turn, implies that we have the requisite bytecode gubbins
> to compile these things. That's a relief.)
With due respect for the functional languages which enchanted my
teens, I can't help feeling that this TIP trades orthogonality for a
small speed improvement. Is there a reason why the bytecode for the
above couldn't somehow be optimized to become the one generated by the
TIPped function ? Granted, he fact that the accumulator var is unnamed
guarantees the absence of traces, but the core is already riddled with
special-casing of traceless vars (which "l" above most frequently is),
since it is a runtime-detectable attribute.
-Alex
PS: Sorry to chime in so late, I'd just love to hear how people come
to terms with this dilemma.

Alexandre Ferrieux wrote:
> Kevin Kenny wrote:
> > The function is obvious to a Lisp programmer. And I don't know how
> > many times I've written:
> > set l {}
> > foreach elt $list {
> > lappend l [do-something-to $l]
> > }
> > return $l
> > (This, in turn, implies that we have the requisite bytecode gubbins
> > to compile these things. That's a relief.)
>
> With due respect for the functional languages which enchanted my
> teens, I can't help feeling that this TIP trades orthogonality for a
> small speed improvement. Is there a reason why the bytecode for the
> above couldn't somehow be optimized to become the one generated by the
> TIPped function ?
The rationale for [lmap] is that this:
return [lmap element $list { do-something-with $element }]
is nicer to read and nicer to write than this:
set result {}
foreach element $list {
lappend result [do-something-with $element]
}
return $result
or this:
return [fmap [list apply {{elt} {do-something-with $elt}}] $list]
The rationale for putting [lmap] In The Core is that (a) it is
easily bytecodable and (b) it is profitable to do so.
> Granted, he fact that the accumulator var is unnamed
> guarantees the absence of traces, but the core is already riddled with
> special-casing of traceless vars (which "l" above most frequently is),
> since it is a runtime-detectable attribute.
I'd add: given the choice between a Sufficiently Smart Compiler
that can figure out how to optimize common idioms and a
Sufficiently Simple Primitive that's easy to optimize in
the first place, the latter is a clear win on every metric
I can think of. TIP#405's [lmap] is squarely in that category.
--Joe English

On 16/10/2012 23:48, Alexandre Ferrieux wrote:
> Is there a reason why the bytecode for the above couldn't somehow be
> optimized to become the one generated by the TIPped function ?
Yes.
Oh, you wanted to know the reason? :-) The problem is that we've never
done anything that optimizes bytecode across command boundaries; it is
terra incognita. What's more, it's going to be really hard to change
that with the current compiler/executor because of the way it works with
direct issuing of "virtual machine code" and how it uses variable width
instructions (especially variable width jumps). It's not that it can't
be done, but it's a definite PITA to work on things as they currently
stand as the data structures we use aren't fit for that sort of use.
One of the reasons I want to work on Tcl 9 is so that we can do much
better code generation.
Donal.

"Donal K. Fellows" <donal.k.fellows@...> writes:
> On 16/10/2012 23:48, Alexandre Ferrieux wrote:
> > Is there a reason why the bytecode for the above couldn't somehow be
> > optimized to become the one generated by the TIPped function ?
>
> Yes.
>
> Oh, you wanted to know the reason? :-) The problem is that we've never
> done anything that optimizes bytecode across command boundaries; it is
> terra incognita.
Yeah.
I wanna see
set foo [something with big $foo]
optimized a-la-K
Donald Arseneau asnd@...

Hi,
On 2012/10/17 12:48 AM, Alexandre Ferrieux wrote:
>> set l {}
>> foreach elt $list {
>> lappend l [do-something-to $l]
>> }
>> return $l
> With due respect for the functional languages which enchanted my
> teens, I can't help feeling that this TIP trades orthogonality for a
> small speed improvement. Is there a reason why the bytecode for the
> above couldn't somehow be optimized to become the one generated by the
> TIPped function ?
The TIP trades orthogonality for expressiveness, not for performance.
[lmap] succinctly expresses the intent to capture output on a 1:{0,1}
basis (typically 1:1).
Such trades are common. [foreach] iterates over a list in a manner more
expressive than for/lindex, and [for] provides a loop that is more
expressive than set/while/incr. [try/on/trap/finally] is more
expressive than catch/catch{switch}/(finaliser)/return. [lassign] is
more expressive than abusing [foreach].
In terms of performance [lmap] uses the same bytecode as [foreach] with
the exception of the unnamed accumulator, and has a miniscule
performance gain.
Regards,
Twylite

On Wed, Oct 17, 2012 at 1:27 AM, Trevor Davel (Twylite)
<twylite@...> wrote:
> Hi,
>
>
> On 2012/10/17 12:48 AM, Alexandre Ferrieux wrote:
>>>
>>> set l {}
>>> foreach elt $list {
>>> lappend l [do-something-to $l]
>>> }
>>> return $l
>>
>> With due respect for the functional languages which enchanted my
>>
>> teens, I can't help feeling that this TIP trades orthogonality for a
>> small speed improvement. Is there a reason why the bytecode for the
>> above couldn't somehow be optimized to become the one generated by the
>> TIPped function ?
>
> The TIP trades orthogonality for expressiveness, not for performance.
> [lmap] succinctly expresses the intent to capture output on a 1:{0,1} basis
> (typically 1:1).
OK, then I realize that the underlying feature I am after is a
preprocessor/inliner (possibly coupled with TAL), so that all the
variants mentioned in the tip discussion can be written in script.
Since none of this is readily available, I can see the light now: thanks !
-Alex

Hi,
On 2012/10/17 07:15 PM, Alexandre Ferrieux wrote:
> OK, then I realize that the underlying feature I am after is a
> preprocessor/inliner (possibly coupled with TAL), so that all the
> variants mentioned in the tip discussion can be written in script.
Yeah - that makes two of us ;) If you have any suggestions on how to do
it, I'm listening.
Regards,
Twylite

Hi,
On 2012/10/16 03:42 PM, Donal K. Fellows wrote:
>> This TIP was Accepted 5/0/0
> TIP #405: Add Collecting Loops, the 'lmap' and 'dict map' Commands
>
> Both now implemented and passing the test suite (well, for me anyway
> ;-)).
Thanks Donal. I haven't had a chance to test yet, but I've looked at
all of the changes and the implementation appears to match the TIP as I
understand/intend it.
Regards,
Twylite