-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
On 2009-01-28 01:18, ethridgela wrote:
> I just changed the type of one of my properties from nothing (implied Page
> type) to String, and back to Page. Afterwards, my Special:Properties page
> shows two instances of that property, with about half my data falling into
> each. But, the property has the same name, and same page.
>
> How can this happen? More importantly, how can I correct it?
I think you just need to wait a little longer. And if that doesn't help, run
the SMW_refreshData (iirc) maintenance script.
Patrick.
- --
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkmd7QwACgkQyYHmhobjRtTHYgCguN8CZFxKAv/4r65Fz8oiWvDJ
90oAoKR/0xDbHNL3T5v/AXfusLmEvMNV
=tvXA
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Thomas,
On 2009-02-20 00:01, Thomas Schweitzer wrote:
> Well, I'm about doing complicated stuff with templates, but maybe I can clarify
> my two points with two simpler use cases:
>
> 1. Derive values of properties: Imagine an article "super" that derives its
> property "prop1" in some way (i.e. a query) from the property "prop1" of
> the articles "sub 1" to "sub n". So, in "super" I could write
> [[prop1::{{#ask:...[[prop1::+]]|limit=1}}]]. Unfortunatly, the result of
> the query needs to be further evaluated by a template, so I have to assign
> the result of the query within the template ([[prop1::{{{1}}}]]).
Here I would be interested why it needs to be evaluated further. Maybe it's
possible to bring the query result into shape with other methods (parser
functions...)? Or maybe a new result format for your use case would be useful?
Without knowing the details that's impossible to know.
> And in
> general, within the template, I don't know the name of the property, i.e.
> prop1. Why? Because there is not only "prop1" but also "prop2"..."prop6"
> that are set by the same query. So I put the whole query into another
> template that is parameterized with the name of the property e.g.
> something like {{evaluate|prop1}}...{{evaluate|prop6}}.
If the problem was solved in the 'super' article which has different parsing
needs for results of different queries this would not be necessary. Also, the
queries that really belong to the 'super' article could stay there.
> 2. The use case for my second point is simpler: {{#ask:
> ...|format=template|template=colorizeResults}}. The template
> "colorizeResult" should highlight some of the results in a given color.
> But unfortunately, I can not pass the color. It would like something like
> |format=template|template=colorizeResults|templParams=red,blue. I think, I
> can't achieve this with inlcusions ot other templates.
I see. This is where I would call {{#ask: ... | format=template |
template=Query specific output template}}, where the 'Query specific output
template' (give it a useful name) contains something like that (amongst other
things that may be specific to just this query):
* {{colorize | red | {{{1}}} }}
* {{colorize | blue | {{{2}}} }}
If this doesn't make sense because your output templates really only differ in
one small detail (the colourisation), and the colours are really specific to
the query result, you should maybe think about putting the colour information
into a property that gets returned along with the query.
Patrick.
- --
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkmd7HMACgkQyYHmhobjRtQwKQCfdGnjmMA/bKqWkPdbKB+d6m5Z
lusAnj70LL0ROenu4beT9zOr9pgtCsKB
=9HAV
-----END PGP SIGNATURE-----

A few general responses:
0) SMW is often pitched as an extension Wikipedia should adopt. Wikipedia
is going on 3million english pages, and I can easily imagine queries
returning 10s of thousands of rows (return all species and geographic ranges
of butterflies).
SMW *is* incredibly powerful, but if it is seriously not meant to scale past
a few thousand pages, the general utility is going to be small.
1)Thanks for the good suggestions as to how to extract these large datasets
outside of SMW. I already make heavy use of mwclient, and have a shortcut
solution for crudely embedding results of large queries (ie: using the
embedded SQL extension in abusive ways). However, it seems a key feature of
SMW is the ability to query the semantic content stored in the wiki, and
present it in wiki pages. Using bots, and processing data dumps, will allow
me to answer the question "give me the average age of my 200,000 users",
but does not allow me to embed the result(in a clean fashion) in a page-
truth be told, nor does my inline SQL query.
Help SMW! you're my only hope.
2) I completely agree. It is immensely helpful to read the message boards,
documentation, and source code. However, this exact topic was not covered,
or at least I did not find a clear discussion. Hence my post, which I think
the discussion around will be useful to future hunters of information about
scalability issues surrounding SMW.
--
View this message in context: http://www.nabble.com/Odd-behaviour-of-SMW-query-results-in-CSV-format....-truncated-results%2C-memory-exceptions%2C-and-%27Are-there-streaming-output-types-%27-tp22088074p22107249.html
Sent from the Semantic Mediawiki - User mailing list archive at Nabble.com.

Hi Joel-
Thanks for the plotticus extension btw- we are already using it in fact,
and it's great!
I had noticed that the plotticus format was able to handle large limit sizes
and was going to poke around the code to figure out why- thanks for saving
me a lot of exploration!
John
johnmajor wrote:
>
> Hello-
>
> I have some queries which should return a CSV data table of 30,000+ rows,
> but either does not return the correct #, or crashes with a memory
> problem.
>
> The query looks like:
> {{#ask: [[Category:Person]] [[status::employee]]
> |mainlabel=-
> |?fname=
> |?lname=
> |?zip code=
> |sep=,
> |format=csv
> }}
>
> *This should return 32,674 rows, but the CSV file I get back has only 100
> entries.
>
> So, I modify the query to have "|limit=40000 ", and this causes a memory
> failure
> "Fatal error: Allowed memory size of 1996488704 bytes exhausted (tried to
> allocate 76 bytes) in
> /var/www/wiki/extensions/SemanticMediaWiki/includes/SMW_DV_Types.php on
> line 44 "
> (I already have a 2G memory limit... and will soon have queries which will
> return 100000+ rows... so increasing memory seems unpractical)
>
> Questions/Comments:
> 1) It seems non-intuitive that a CSV export would have a default limit of
> 100 rows.
> 2) When returning the complete result set, it appeats that SMW is trying
> to load the whole thing into
> memory before returning it... hence causing the memory fault.
> 3) Is there a way to return large numbers of query results? Perhaps using
> a streaming CSV method?
>
> Thanks!
> John
>
--
View this message in context: http://www.nabble.com/Odd-behaviour-of-SMW-query-results-in-CSV-format....-truncated-results%2C-memory-exceptions%2C-and-%27Are-there-streaming-output-types-%27-tp22088074p22106337.html
Sent from the Semantic Mediawiki - User mailing list archive at Nabble.com.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Patrick,<br>
thanks for your answer.<br>
<br>
Well, I'm about doing complicated stuff with templates, but maybe I can
clarify my two points with two simpler use cases:<br>
<ol>
<li>Derive values of properties: Imagine an article "super" that
derives its property "prop1" in some way (i.e. a query) from the
property "prop1" of the articles "sub 1" to "sub n". So, in "super" I
could write [[prop1::{{#ask:...[[prop1::+]]|limit=1}}]]. Unfortunatly,
the result of the query needs to be further evaluated by a template, so
I have to assign the result of the query within the template
([[prop1::{{{1}}}]]). And in general, within the template, I don't know
the name of the property, i.e. prop1. Why? Because there is not only
"prop1" but also "prop2"..."prop6" that are set by the same query. So I
put the whole query into another template that is parameterized with
the name of the property e.g. something like
{{evaluate|prop1}}...{{evaluate|prop6}}.</li>
<li>The use case for my second point is simpler: {{#ask:
...|format=template|template=colorizeResults}}. The template
"colorizeResult" should highlight some of the results in a given color.
But unfortunately, I can not pass the color. It would like something
like |format=template|template=colorizeResults|templParams=red,blue. I
think, I can't achieve this with inlcusions ot other templates.</li>
</ol>
Cheers<br>
Thomas<br>
<br>
Patrick Nagel schrieb:
<blockquote cite="mid:499D7257.20805@..." type="cite">
<pre wrap="">-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Thomas,
On 2009-02-19 22:16, Thomas Schweitzer wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I've got two questions concerning #ask: queries with format=template.
1. As template parameters {{{1}}},... I get the results of the query. But
how can I access the names of the properties (i.e. the "column headers")
within the template? This is very useful if the same template is used for
several differents queries.
2. How can I pass additional parameters to the template? I wrote a template
that behaves differently depending on a context that is given as parameter
of the template. However, with format=template and template=xxx this does
not seem to be possible.The very unpleasing alternative would be writing a
new template for each context.
</pre>
</blockquote>
<pre wrap=""><!---->
Sounds like you are putting a lot of complexity into one single template - why
not make multiple simpler templates that transclude each other? Specify the
specialised templates in the ask query and let these specialised templates
include the common stuff from another template.
I think the template result format should be kept stupid and simple as it is,
otherwise it will be just too confusing.
Just my 2分...
Patrick.
</pre>
</blockquote>
<br>
</body>
</html>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Thomas,
On 2009-02-19 22:16, Thomas Schweitzer wrote:
> I've got two questions concerning #ask: queries with format=template.
>
> 1. As template parameters {{{1}}},... I get the results of the query. But
> how can I access the names of the properties (i.e. the "column headers")
> within the template? This is very useful if the same template is used for
> several differents queries.
> 2. How can I pass additional parameters to the template? I wrote a template
> that behaves differently depending on a context that is given as parameter
> of the template. However, with format=template and template=xxx this does
> not seem to be possible.The very unpleasing alternative would be writing a
> new template for each context.
Sounds like you are putting a lot of complexity into one single template - why
not make multiple simpler templates that transclude each other? Specify the
specialised templates in the ask query and let these specialised templates
include the common stuff from another template.
I think the template result format should be kept stupid and simple as it is,
otherwise it will be just too confusing.
Just my 2分...
Patrick.
- --
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkmdclcACgkQyYHmhobjRtTtAwCbB0xzxoRT9ZwDWvh/55QuHAxy
dwcAoNLbe8yLaQkDglaiRjwRN9Vt0ntL
=wBTP
-----END PGP SIGNATURE-----

A possible virus was found in this message.
To her. 'it's no good stirring up trouble.' and growth and
development as a lawyer, we must remember everywhereand
the shockbut what i remember best in aid of it, and bind
her closer to you, if such to see. Of course there is much,
said her uncle,.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Jonathan,
On 18/02/09 18:31, Jonathan D. Greene wrote:
> Good point, but should both the source page AND the page into which it is
> transcluded register the same property value?
>
> (As the implementation currently does?)
>
> This state of affairs either makes for muddled queries and effectively excludes
> transclusion.
>
> It'd be great if there were a way to specify in a source page, "register this
> value here" or "register this value only when the host page is transcluded."
That's where the '<includeonly>' and '<noinclude>' tags in templates come in.
Put your property assignments between <includeonly>...</includeonly> and the
property will only get assigned in a page where the template actually gets
transcluded, but not in the template page itself.
Patrick.
- --
Key ID: 0x86E346D4 http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkmcu74ACgkQyYHmhobjRtTq3ACgyJ+1iiBwNxXDYpEI0gnbSpkh
wXIAn29TDgaq0ZA3ElxIvocryd9gfwEF
=MOkt
-----END PGP SIGNATURE-----

Comments inline...
On Wed, Feb 18, 2009 at 12:59 PM, johnmajor <john.e.major.jr@...> wrote:
>
> Hello-
>
> I have some queries which should return a CSV data table of 30,000+ rows,
> but either does not return the correct #, or crashes with a memory problem.
>
> The query looks like:
> {{#ask: [[Category:Person]] [[status::employee]]
> |mainlabel=-
> |?fname=
> |?lname=
> |?zip code=
> |sep=,
> |format=csv
> }}
>
> *This should return 32,674 rows, but the CSV file I get back has only 100
> entries.
>
> So, I modify the query to have "|limit=40000 ", and this causes a memory
> failure
> "Fatal error: Allowed memory size of 1996488704 bytes exhausted (tried to
> allocate 76 bytes) in
> /var/www/wiki/extensions/SemanticMediaWiki/includes/SMW_DV_Types.php on line
> 44 "
> (I already have a 2G memory limit... and will soon have queries which will
> return 100000+ rows... so increasing memory seems unpractical)
>
> Questions/Comments:
> 1) It seems non-intuitive that a CSV export would have a default limit of
> 100 rows.
I think the idea was to require people to explicitly ask for really
large result sets that might cause slow responses or use lots of
memory; the few result printers I looked at when doing the initial CSV
implementation all set a default limit if one wasn't specified.
> 2) When returning the complete result set, it appears that SMW is trying to
> load the whole thing into
> memory before returning it... hence causing the memory fault.
That's true. See below.
> 3) Is there a way to return large numbers of query results? Perhaps using a
> streaming CSV method?
I'm not aware of any way to stream results from the query engine to
the output formatters. The QueryPrinter base class expects the
format-specific code to return the result as a single string. I
suppose you can imagine an alternate implementation that allows for
streaming the results from query printers. So at this point, not that
I know of.
>
> Thanks!
> John
> --
> View this message in context: http://www.nabble.com/Odd-behaviour-of-SMW-query-results-in-CSV-format....-truncated-results%2C-memory-exceptions%2C-and-%27Are-there-streaming-output-types-%27-tp22088074p22088074.html
> Sent from the Semantic Mediawiki - User mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
> -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
> -Strategies to boost innovation and cut costs with open source participation
> -Receive a $600 discount off the registration fee with the source code: SFAD
> http://p.sf.net/sfu/XcvMzF8H
> _______________________________________________
> Semediawiki-user mailing list
> Semediawiki-user@...
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>