I would like to be able to enable the description of the change to no longer being optional but forced instead simply, when enabled, change to ikiwiki cannot be approved without description (commit message).

This was not a known bug before your report. It looks as though every
time we use Image::Magick->Read(":foo.png"), which is (or was)
ImageMagick's syntax for opening a file of unknown type without
interpreting a prefix containing : as a special directive instead
of part of the filename, it fails.

Please try re-running the test with better diagnostics using
commit 4ace7dbb7
and report what it says. --smcv

When I run ikiwiki with the --rebuild option (or only with the --setup file.setup option a map directive like [[!map pages="*" show=title]] generates a page map as if it didn't contain any show parameter. Only after I manually edit something which causes the page containing the map directive to be rebuilt is the page map regenerated without ignoring the show parameter.

I can't seem to do a password reset on this wiki. I am writing this
through the anonymous git push interface (phew for that!).

I have tried three times now to reset my password through the user
interface - my account name is anarcat, and when i do the password
reset, I get a token. I go visit the website, set a passphrase, click
Save Preferences and I end up on a login form. I enter my
passphrase, click Login and I get the error:

Multimarkdown footnotes are pretty useful. If they are enabled in a
wiki, they don't look so good with the default stylesheet, however, as
the references are in the same size and positioning as everything
else.

This particular wiki does not use multimarkdown, so there's no easy
way to demonstrate this here, you'll have to trust me on this.

I settled on xx-small because it's the only size that doesn't affect
line-height here. However, Bootstrap's way may be better.

At any rate, the correct way to fix this is to avoid custom styling
and use the <sup> tag for the footnote reference, as it has
meaning which is important to have proper semantic output (e.g. for
screen readers), as detailed in this Stack Overflow discussion.
--anarcat

ikiwiki code does not interpret Markdown or translate it into HTML.
If I'm interpreting what you say correctly, you seem to be implying
that you think Text::MultiMarkdown is producing incorrect
HTML for footnotes (is an <a> with a class, should be a <sup>).
If so, please report that as a MultiMarkdown bug, not an ikiwiki bug,
or alternatively don't use MultiMarkdown.

The recommended backend for the mdwn plugin is
Text::Markdown::Discount, which optionally implements
footnotes using the same syntax as MultiMarkdown (originating in
"PHP Markdown Extra"). However, ikiwiki doesn't currently enable
that particular feature. Maybe it should, at least via a site-wide
option.

What remains after eliminating the MultiMarkdown bug seems to be:
ikiwiki's default stylesheet does not contain the necessary styling
to work around the non-semantic markup produced by the non-default
Text::MultiMarkdown Markdown implementation. Is that an accurate
summary?
--smcv

That is an accurate summary.

I didn't realize that Discount didn't actually support footnotes in
Ikiwiki by default. I guess I enabled Multimarkdown exactly for that
kind of stuff that was missing... It seems to me it would be
reasonable to enable footnotes in Ikiwiki. There's already a lot of
stuff that Discount does that is way more exotic (e.g. tables) and
non-standard (e.g. abbr:).

Ideally, users would get to configure which
Discount flags
are enabled in their configuration, but I understand that makes the
configuration more complicated and error-prone.

Discount enables enough features by default that adding footnotes doesn't
seem bad to me. I'm also tempted by something like

mdwn_enable: [footnotes]
mdwn_disable: [alphalist, superscript]

where the default for anything that was neither specifically enabled
nor specifically disabled would be to enable everything that we don't
think is a poor fit for the processing model (pandoc-style document
headers) or likely to trigger by mistake (typographic quotes and
maybe alpha lists).
--smcv

That being said, Discount generates proper semantic markup when
footnotes, so this bug doesn't apply to the default Discount mode,
if we ignore the fact that it doesn't support footnotes at all.
Should I open a todo about this and the above?

In the meantime, wouldn't it be better to have some styling here to
workaround the problem in MMD?

Honestly, I'd rather have ikiwiki's level of support for the non-preferred
Markdown implementation be: if you are stuck on a platform with no C compiler
or Perl headers, you can use the pure-Perl Markdown flavours, and they
will sort of mostly work (but might not look great).

I'm a little concerned that styling these rather generically-named classes
might interfere with the implementations of footnotes in other Markdown
implementations, or indeed non-Markdown - I wouldn't want to style
a.footnote if the HTML produced by some other htmlize hook was
<sup><a class="footnote" ...>[1]</a></sup> for instance.
But they're probably harmless.

Alright, your call. At least this bug will be available as a workaround
for others that stumble upon the same problem! --anarcat

Note that I also make the bottom <div> small as well so that it has
less weight than the rest of the text. -- anarcat

I am not sure what is going on. It affects this wiki and the git-annex
wiki. I am editing this through the anonymous git push interface.

OpenID is failing on me. That is, sometimes it works, sometimes it
doesn't. For example, while writing this, I clicked the "Preferences"
link and I seemed to have been logged in automatically without
problem, even though I previously tried to login and failed with an
error similar to Error: OpenID failure: time bad sig:, which
of course I cannot reproduce anymore on ikiwiki.info now:

I can still reproduce this on the git-annex wiki, however, which is
odd. This could be because the OpenID host is screwing up, as I am
not directly responsible for that box anymore... but then why would it
work on one wiki and not the other?

But worse, I cannot login with my regular non-OpenID user, which I
started using more regularly now. When I type the wrong password, the
login form gives me "Invalid entry" next to the password field. But
then if I do a password recall and reset my password, I get a
different error:

Error: login failed, perhaps you need to turn on cookies?

That happens reliably on git-annex.branchable.com. ikiwiki.info seems
to be more stable: i can eventually login. i can login to my personal
wiki with OpenID fine. I can also login to branchable.com itself with
openid without issues.

So I guess the problem is mostly with git-annex.branchable.com? Not
sure how to debug this further.

Update: now I can't login to the ikiwiki.info site anymore, getting
the same errors as on the git-annex site.

Update2: it seems this is specific to the HTTP/HTTPS switch. If I use HTTPS, things work fine, but not with plain HTTP. So I'm moving this to the branchable wiki, as I am not having that problem on other ikiwiki sites. Maybe the bug specific to ikiwiki is the lack of clarity in figuring out wth is going on here... See http://www.branchable.com/bugs/login_failures_without_https

This seems to be a concacentation of multiple unrelated problems with
different stuff, which is not a good bug report technique. Then to add to
the fun, you filed the same bug on branchable too. Sigh!

The time_bad_sig problem with the perl openid library is a problem I am
aware of but it's not clear if the problem is clock skew, or a protocol
problem. At least one user to report it seemed to get it due to a http
proxy. I'm pretty sure it could also happen if multiple openid logins
were attempted at the same time (the consumer_secret which is stored
server-side is used). The problem is not specific to ikiwiki.

Ikiwiki says "login failed, perhaps you need to turn on cookies?" when
the user successfully logged in, but their cookie does not indicate why
they were logging in to begin with, so ikiwiki does not know what action
to continue to. One way to get this when cookies are enabled is to
re-post a login form after already using it, by eg using the back button
to go back to a previous login form and try to reuse it.

I am sorry. I thought the problem was originally related to ikiwiki
then figured it was only happening on branchable sites, so I figured
it was better to report it on the branchable.com forums.

I know that there's a OpenID-specific issue, but I had such issues in
the past and succesfuly solved those. Because the timing of the emergence
of the problem, i felt there was a correlation between the two issues.

And indeed, there seems to be a HTTPS-related issue: both login mechanisms
work fine when on HTTPS, and both fail on HTTP. So I don't see those things
as being necessarily distinct. -- anarcat

I've explained how the "login failed, perhaps you need to turn on
cookies?" can happen and what it means. Clearly nothing to do with
http; clearly not specific to branchable.

I just now logged into this site using openid over http, and it worked
ok. I think it's more likely that the time_bad_sig problem occurs
intermittently (which would make sense if it's a timing related issue),
and so you've just so happened to see it when logging in with
http and not https, than that there's actually a correlation.
--Joey

Steps

I've not signed in because the sign in page didn't come up, instead a simple (You might want to Signin first?) showed up, which I've haven't read and commented away.

Results

As a consequence of that, the user '' - that's a null character, have somehow logged in and it seems that there is no way to log it out.

None of this phantom user edits are being commited - this blog post was made with that user logged in via web.

It seems I can't log out from nowhere. I've rebuild the wiki from the command line and restarted the nginx server, the phantom user remains logged in and open to anyone willing to edit away the wiki.

I wonder whether this might be caused by the combination of the httpauth plugin
with the nginx web server. httpauth is known to work correctly with Apache,
but you might be the first to use it with nginx.

Specifically, I wonder whether $cgi->remote_user() might be returning the
empty string. Looking at the code, we expect it to be either a non-empty
username, or undef.

Please try installing this CGI script on your nginx server, making it
executable and accessing its URL without carrying out any special HTTP
authentication (you can delete the script immediately afterwards if
you want). If my theory is right, you will see a line REMOTE_USER= in
the output. Post the output somewhere, or mail it to smcv at
debian.org if you don't want to make it public.

If you do not intend to use
HTTP basic authentication,
please do not enable the httpauth plugin. That plugin is intended to be used
in conjunction with a web server configured to require HTTP basic authentication
with one of a limited set of authorized usernames.

Question

See inside dot ikiwiki. .ikiwiki/userdb is a Perl Storable file;
there are instructions for inspecting it on that page. .ikiwiki/sessions.db
is most likely a Berkeley DB file.

I would be interested to see the contents of these two files and the complete
.setup file. I would also be interested to see a tarball of the entire
wiki source directory, if it isn't excessively large. If you'd be willing to
share them, please contact smcv@debian.org. --smcv

I think I've sent right away when you asked, anyway I still have the tarball hanging around. The last iikb domains will expire next month though, the wiki will only be accessible by mirror https://notabug.org/iikb/dev.iikb.org.

I see from the tarball that you have a lot of uncommitted changes. This is
probably because whatever is causing the anonymous accesses to succeed is
breaking other code paths by giving them an empty username: in particular
it seems reasonably likely that the git plugin will refuse to commit
changes in that situation.

I would expect that you should be getting error messages on the ikiwiki
CGI script's stderr in this situation. In Apache they would normally
end up in error.log; I don't know how nginx does logging, but it is
probably something similar. Please check that log for relevant-looking
error messages. --smcv

I have an ikiwiki, where I activated the plug-in sidebar:
* enable sidebar? yes
* show sidebar page on all pages? yes

In order to create the page, I add a link to it on my entrance page: [[sidebar]].
I now can choose where to put it, I put it as "sidebar", not index/sidebar. Language is Markdown, nothing else installed on the system.
After saving I can create the page clicking on the "?" and I fill it with links and the plugin pagestats.
The sidbar appears properly on all pages, also the tag-cloud.

However, when I try to go into "preferences" I just get an error message, saying it has the format html: "Content-type: text/html "
The entire html is
\Content-type: text/html

po_translatable_pages: /index or /hugo or /hugo/keys or /about or /archive or /tips
or /talks or /imprint or /copyright or /blog or /posts or /law or /quotes or /quotes/*

Config problems in ikiwiki.setup should really not cause the whole site
build to crash; this can make it hard to recover. --Joey

Given who's reporting this, am I right in assuming that's with ikiwiki 3.20150614? --smcv

I try to setup a small site with the auto-blog.setup and played a bit with it:
If I activate the po plugin and set po_translateable_pages to something meaningful (like the example: * and !*/Discussion),
then I'll get the same error

syntax error in pagespec "\"page(./posts/*)"

but only after a second run of the ikiwiki --setup site.setup

My try to get a clue: deleting any po and pot files and run the rebuild again - works fine
run the rebuild a second time - error as above

tune any of the pagespec variables in the setup and at the inline directives of the blog or sidebar dosn't change anything
except leaving the po_translateable_pages empty, than the rebuild works and doesn't create any po files (as expected).

Is this helpful or have I done anything stupid ? -- Michael

This would be helpful if I could reproduce the crash from your instructions, but I couldn't
Which version of ikiwiki is this?
--smcv

It was version 3.20141016.2 as it is in debian stable / jessie
I tried again with version 3.20160121 as it is in debian sid
same behavior as described

I did setup a new blog with auto-blog.setup, activated the po plugin with the defaults
and get the error again (running ikiwiki --setup twice) --Michael

phil is kindly running ikiwiki on hands.com as http://rhombus-tech.net
we continuously run into "merge" issues which need recovering on a
near monthly basis.

i have a local checkout of the repository: i often need to upload images via
that, doing the usual "git pull", followed by "git commit -a", followed by
"git push", adding an HTML page that is edited by vim as well as the images.

i also often need to "recover" the wiki - for example by renaming pages that
users have erroneously added, deleting pages that they should not have made,
moving pages from locations that they should not have added or that i decide
should be restructured.

these are the operations where everything usually gets completely fscked.

the really weird thing is that when i know that things are out-of-sync,
a "git pull" gives a completely different head branch from the one shown
through the RecentChanges log!

phil has often had to recover an entire set of files that are completely out
of sync, that never enter the "git pull" stream onto my laptop, and are not
visible on the wiki itself either.

this is all incredibly strange and mysterious, but it basically means that
ikiwiki is not particularly robust and reliable for everyday use. i'd very
much like it to be!

Calling ikiwiki with a bunch of options, including the --dumpsetup somefile.setup option creates somefile.setup for later reuse with the --setup option. The wiki state dir however is not saved in the setup file, it has no wikistatedir at all.

Problem

Page foo contains the directive [[!rpcbug ]] (rpcbug being the name of the plugin). Calling proxy.rpc("srcfile", "bar") in the preprocess function seems to mess up the RPC communication between ikiwiki and the plugin, and the result is that the generate foo/index.html page is a text file containing the return value of the preprocess call.

What I do not understand is that disabling the format function (by commenting line 46 of the plugin) solves the problem. Half of an explaination is that I think that when the the proxy.rpc function is called, the RPC communication is messed up in such a way that the format function is not called, and the return value of preprocess is considered to be the whole html code of the resulting page.

I understood that: as the first rpc call messes up the communication, more rpc calls are messed up, but if there is only one rpc call, not that much is broken. -- Louis

I hope someone will understand the problem better than I do, because I have no idea about how to solve this.

The call to srcfile(foo) fails (because Ikiwiki thinks that page foo does not exist).

Ikiwiki thinks that processing of the directive is finished, whereas the plugin still waits for the answer of Ikiwiki.

Ikiwiki asks the plugin to render a new directive, but the plugin interprets the request as the return value for its previous request. Thus, the plugin thinks that srcfile(foo) is page (this page being a misinterpretation of the Ikiwiki request).

So, I think that this might be an error in the
rpc_call
function of the external plugin: when the called method fails, it should
return something (or raise an exception, if this is possible in RPC) to notify
the plugin that something went wrong.

Ikiwiki sends a methodCall message to the plugin (which is a call to the
preprocess function);

the plugin sends a methodCall message to ikiwiki (which is a call to the
srcfile function);

Ikiwiki answers with a methodCall message:

Ikiwiki answers this because the function call failed, and it is already
processing the next directive;

the plugin thinks that it is its request answer, and misinterprets it.

Thus, I think that the bug is in the proxy.py python file. On receiving a
methodCall (instead of a methodResponse) as an answer to a methodCall
request, proxy.py should notice the type of request, and call the requested function.

The current default behavior of the plugin implies having the additional
stylesheet not activated (if you don't set rel="stylesheet") or only
one of them activated (if you add two stylesheets and not set the same
title for both). This was hard to understand for two of us while working
on https://labs.riseup.net/code/issues/9314 and until we went and read
those W3C documents.

I think that to match better the description of that feature, and to be
easier to comprehend in its default setting, the meta plugin should by
default:

... and see if that fixes the problem. Then we can start looking at the release notes to figure out what change they did that broke us and upgrade. Or pin the version on our side. Or simply switch to something else. --anarcat

Now I know it's "bad" to rewrite history in git, but sometimes, and especially with public sites such as a wiki, if confidential information gets transmitted in the wiki, it can be pretty important to remove it, and the only way to do this on a public git repo is by rewriting history.

(This happened as part of my implementation of git-annex support to be honest, but i think it applies to other situations as well.)

The problem is that ikiwiki keeps track of the last commit it saw in $srcdir/.ikiwiki/indexdb. Then it uses this to infer which files changed. If history changed, this will fail with a fairly dramatic:

The workaround I have found was to remove the indexdb file, because that's apparently legit. But it would be nice to have (1) a proper error message (it had to dig around the error.log to understand what's going on), (2) to have a proper fallback if the git log fails and (3) to recover with the newer commit ID when we fallback. --anarcat

FWIW, I had a 500 Internal Server Error while submitting this bug at first.

I've found that switching my own non-ikiwiki projects to https://github.com/jgm/CommonMark has helped sort them out for the most part.

ikiwiki does not implement Markdown on its own: it uses one of several
third-party libraries, with the current recommendation being
Discount. Out-of-process implementations like
pandoc are not suitable to be the default for
performance reasons.

There seems to be a Perl binding for libcmark at
https://github.com/nwellnhof/perl-commonmark, but unfortunately
its README points out that the libcmark API is not stable,
which means libcmark and perl-commonmark would have to be upgraded
in lockstep: this makes them awkward to deal with in Linux
distributions. As a result I'm not going to look into this myself
until there is a stable API for Commonmark available in Debian.

However, if you want to add optional Commonmark support to the
mdwn plugin, I'd review a patch. --smcv

Behaviour: The 3rd entry doesn't show the next month, but the 1st month of the year (aka January).

Problem: Since there are no negative month numbers (unless someone starts with march because of Feb 29), –1 is interpreted correctly.
Explicitely positive numbers aren't recognized as being relative. Possibly it is the numerical interpretation of the value, there is no difference between n and +n.

Solution: treat the value as string, check for a leading +, set a relativeMonth flag (which then also should happen on negative values, if it does not happen yet). If then it is set for the month in question, first calculate month_year and then go on as usual.

Idea: since i mentioned gcal earlier, how about some of the shorthanded sytax as "." for this, ".-" for previous, ".+" for next month together with its neighbours?

On my blog, i have setup a simple calendar and sparkline on the sidebar, similar to joey's. Unfortunately, in my case it looks like all posts were done in february, date at which i converted from drupal.

Is it possible the meta(date) directives are being ignored by those plugins? --anarcat

For background, each page has two dates: creation date (ctime, meta(date))
and last modification date (mtime, meta(updated)). postsparkline
defaults to showing the ctime but can be configured to use the mtime
instead; calendar always uses ctime. So what you're doing should work
like you expect.

The plugins don't get to choose whether they ignore meta(date);
the effect of a meta(date) directive in $page is to set $pagectime{$page}
during scanning (overriding whatever was found in the filesystem), and
that data structure is what the plugins read from. So the first thing to
investigate is whether the ctime
in your .ikiwiki/indexdb is correct. --smcv

This is more complicated than it looked at first glance because both
jquery and jquery-ui have broken API since the version we embed,
and we also ship other jquery plugins for attachment.
Perhaps someone who knows jquery could check compatibility and
propose a branch? --smcv

What I expected

What happened instead

Likely fix

Looks like proxy.py needs the trick from Debian bug #637604 so
that it can defer a few imports (at least xml.parsers.expat and
the XML-RPC libs) until the methods using them are called. --schmonz

It's more complicated than I thought. Findings and questions so
far:

Failing to load an external plugin should be an error

When a typical Perl plugin fails to load (say, by failing to compile),
IkiWiki::loadplugin() throws an exception. For XML-RPC plugins
written in any language, ikiwiki assumes loading succeeded.

Let's take plugins/rst as an example. It's written in
Python and uses proxy.py to handle XML-RPC communication with
ikiwiki. Let's say that proxy.py compiles, but rst itself
doesn't. We'd like ikiwiki to know the plugin isn't loaded, and
we'd like an error message about it (not just the Python errors).

Now let's say rst would be fine by itself, but proxy.py doesn't
compile because some of the Python modules it needs are missing
from the system. (This can't currently happen on Debian, where
libpython2.7 includes pyexpat.so, but pkgsrc's python27
doesn't; it's in a separate py-expat package.) As before, we'd
like ikiwiki to know rst didn't load, but that's trickier when
the problem lies with the communication mechanism itself.

For the tricky case, what to do? Some ideas:

Figure out where in auto.setup we're enabling rst by default,
and stop doing that

In pkgsrc's ikiwiki package, add a dependency on Python and
py-expat just in case someone wants to enable rst or other
Python plugins

For the simple case, I've tried the following:

Available in a git repository branch.
Branch: schmonz/external-plugin-loading
Author: schmonz

In IkiWiki::Plugin::external::import(), capture stderr

Before falling off the end of IkiWiki::Plugin::external::rpc_call(),
if the command had been 'import' and stderr is non-empty, throw
an exception

In IkiWiki::loadplugin(), try/catch/throw just like we do with
regular non-external plugins

With these changes, we have a test that fails when an external
plugin can't be loaded (and passes, less trivially, when it can).
Huzzah! (I haven't tested yet whether I've otherwise completely
broken the interface for external plugins. Not-huzzah!) --schmonz

I don't think you're deriving much benefit from Markdown's table syntax
if you have to mix it with HTML::Template and ikiwiki directives,
and be pathologically careful with whitespace. "Right tool for the job"
and all that

When I edited this page I was amused to find that you used HTML,
not Markdown, as its format. It seems oddly appropriate to my answer, but
I've converted it to Markdown and adjusted the formatting, for easier
commenting.
--smcv

templates expose odd behavior when it comes to composing
links and directives:

the parameters are passed through the preprocessor twice, once on
per-parameter basis and once for the final result (which usually contains the
preprocessed parameters).

one of the results it that you have to write:

[[!template id="infobox" body="""
Just use the \\\[[!template]] directive!
"""]]

(that'd be three backslashes in front of the opening [.)

this also means that parts which are not used by the template at all still
have their side effects without showing.

furthermore, the evaluation sequence is hard to predict. this might or might
not be a problem, depending on whether someone comes up with a less contrived
example (this one assumes a [[!literal value]] directive that just
returns value but protects it from the preprocessor):

we can use [[!literal """[[!invalid example]]"""]], but we can't use
[[!template id=literalator value="""[[!invalid example]]"""]] with a
'literalator' template <span class="literal">[[!literal """<TMPL_VAR
value>"""]]</span> because then the invalid directive comes to action in
the first (per-argument) preprocessor run

links in templates are not stored at all; they appear, but the backlinks
don't work unless the link is explicit in one of the arguments.

[[!template id="linker" destination="foo"]]

with a 'linker' template like

Go to [[<TMPL_VAR destination>]]!

would result in a link to 'destination', but would not be registered in the
scan phase and thus not show a backlink from 'foo'.

this seems to be due to linkification being called before preprocess rather
than as a part of it, or (if that is on purpose) by the template plugin not
running linkification as an extra step (not even once).

(nb: there is a way to include the raw_ value of a directive, but that only
refers to htmlification, not directive evaluation.)

both those behaviors are non-intuitive and afaict undocumented. personally, i'd
swap them out for passing the parameters as-is to the template, then running
the linkifier and preprocessor on the final result. that would be as if all
parameters were queried raw_ -- then again, i don't see where raw_ makes
anything not work that worked originally, so obviously i'm missing something.

i think it boils down to one question: are those behaviors necessary for
compatibility reasons, and if yes, why?

The FormattingHelp link in the edit form of any page points to the same ikiwiki/formatting help text for Markdown, regardless of page type (which could be HTML, reStructuredText, etc.) On the wiki I run, this is confusing users.

What I would like is that either the FormattingHelp link changes with page type (requires Javascript, if one is going to change the page type for new pages), or that the ikiwiki/formatting page is an index of supported page types with a further link to help text for each one (less user-friendly but likely easier to implement).

Unfortunately, I can't think of a good way to solve this without
introducing a special case for enabled() in Render.pm, either a
new dependency type "enabled(smileys)" => $DEPENDS_ENABLED
or a special case that treats "enabled(smileys)" => $DEPENDS_PRESENCE
differently. --smcv

I assume this means Mail::Sendmail doesn't know how to send Unicode
strings, so any string passed to it (or any message body, or something?)
will need to be passed through encode_utf8(). It looks as though
Mail::Sendmail also defaults to

Content-Type: 'text/plain; charset="iso-8859-1"'

so it'll need a 'Content-Type' => 'text/plain; charset="utf-8"'
too.

I'm disappointed to see how many of the library modules used by ikiwiki
are not Unicode-clean... but then again, Mail::Sendmail was last released
in 2003 so it's hardly surprising. I wonder whether Email::Sender
is any better?

(If you know Python 2, the analogous situation would be "doesn't
know how to send unicode objects, so you have to get a str object
with a_unicode_object.encode('utf-8')".) --smcv

utf8 "\xEB" does not map to Unicode at /usr/share/perl5/IkiWiki.pm line 873, <$in> chunk 1.

lead me to a call to utf8::valid, which lead to http://perldoc.perl.org/utf8.html which says this is an "INTERNAL" function:

Main reason for this routine is to allow Perl's testsuite to check that operations have left strings in a consistent state. You most probably want to use utf8::is_utf8() instead.

Apparently the main point of the function is to emit the warning in unit tests - problem is, in the ikiwiki context, the only useful thing to warn about would be the name of the file you're trying to parse, not the name of the source code. Alternatively, since the code does continue on with the data, not whining about it might be an option but an actionable message would be better.

getsetup defines config options to be one of: boolean, string, integer,
pagespec, "internal" (non-user-visible string), ref to an array of one of
those scalar types, or ref to a hash { string => one of those scalar types }.
IkiWiki::Setup also appears to support regexps (qr//), although that's
not documented (presumably they're treated the same as strings).

Supporting arbitrary arrays/hashes as values would require some way to
untaint the values recursively.

Complex config data also can't be used with the websetup
plugin, which currently supports everything that IkiWiki::Setup does,
except for hashes. --smcv

Yet I am not sure how to fix that kind of problem in Perl... --anarcat

If I remove the "eval" above, I get:

Error: Wide character in syswrite at /usr/lib/perl/5.14/Sys/Syslog.pm line 485.

I have improved a little the error handling in log_message() so that we see something when syslog fails, see the branch documented above. I can also confirm that reverting syslog should show wiki name fixes the bug. Finally, I have a unit test that reproduces the problem in git, and a working patch for the bug, again in git.

One last note: I noticed that this problem also happens elsewhere in ikiwiki. For example, the notifyemail plugin will silently fail to send notifications if the pages contain unicode. The ?notifychanges plugin I am working on (in option to send only the diff in notifyemail) seems to be working around the issue so far, but there's no telling which similar problem are out there.

The caps in the image title were somehow converted to small letters and then the image is saved as a directory. Very puzzling.
I get the same error when image names are small letters.

The error also occurs with png images.

How do I fix this?

Later investigation ... I got around the problem by creating the mark-up in a new directory. However, if I try to create a new directory with the same name as the directory containing the problem code, the problem re-emerges -- the old directory is apparently not overwritten. Perhaps this is an issue with the git storage.

I turned on the sidebar plugin, with global_sidebars on (in the web setup page), created a sidebar page in the root, and edited the sidebar a few times.

I then noticed that all pages on the root had been updated with a sidebar, but no subpages (i.e. a/b). Only after editing a subpage did it get a sidebar. Editing sidebar itself only updated subpages with sidebars, the other subpages had not been refreshed (proven by their unchanged filesystem date)

After calling ikiwiki --setup on the command line all pages were updated. So this seems to be a difference between web-started --setup and command-line --setup. Or it just doesn't work the first time --setup is called after sidebars are enabled.

/home/b-fusioninventory/public_html/documentation/index.es.html independently created, not overwriting with version from documentation.es

I tried rebuilding it, and the rebuild failed like this:

building recentchanges/change_ef4b9f92821335d96732c4b2c93ed96bc84c2f0d._change, which depends on templates/page.tmpl
removing recentchanges/change_9ca1de878ea654566ce4a8a031d1ad8ed135ea1c/index.html, no longer built by recentchanges/change_9ca1de878ea654566ce4a8a031d1ad8ed135ea1c
internal error: recentchanges/change_9ca1de878ea654566ce4a8a031d1ad8ed135ea1c._change cannot be found in /home/b-fusioninventory/source or underlay

This internal error seems like the root cause of the original failure.
ikiwiki crashed and did not record that it wrote the index.es.html file.

The toc directive scrapes all headings from the page, including those in the sidebar. So, if the sidebar includes navigational headers, every page with a table of contents will display those navigational headers before the headers in that page's content.

I'd like some way to exclude the sidebar from the table of contents. As discussed via Jabber, perhaps toc could have a config option to ignore headers inside a nav tag or a tag with id="sidebar".

Well, you can adjust the inline's pagespec to exclude it, or even tag it
with a tag that the pagespec is adjusted to exclude. --Joey

I did it by making a new tag called "redir", tagging the redir page with it and then modifying the pages attribute of my inline to exclude pages with that tag. However, there is the same problem with the archives, probably the calendar if you use that and likely some other cases that I haven't thought about. In all these places you need to explicitly exclude redir pages. I think that ideally redir pages should have some special treatment that excludes them by default in most situations, because they are not real pages in a sense. They can have a body but if the browser is working properly it will never be shown.

How about adding a new PageSpec called redir(glob) and excluding such pages from the post(glob) PageSpec? I think this behaviour makes more sense and thus should be the default, but if a user wants the old behaviour that's still available as "page(glob) or redir(glob)".

Good URL redirections are important because they allow you to move things around without breaking incoming links from external sites and people's browsing history (which you can't fix, unlike internal links). --anton, 2016-01-31

For some time now, in circumstances that I've had enormous troubles
trying to track, I've seen feeds getting removed by ikiwiki when
apparently unrelated pages got changed, with the message:

removing somepath/somepage/somefeed, no longer built by some/unrelated/page

I've finally been able to find how and why it happens. The situation is
the following:

page A has an inline directive that (directly) generates a feed F

page B inlines A, thus (indirectly) generating F again

page B is rendered after page A

The feed removal happens when changes are made to prevent B from
inlining A; for example, because B is a tag page and A is untagged B, or
because B includes A through a pagespec that no longer matches A. In
this case, this happens:

page A is built, rendering F

page B is built, not rendering F, which it used to render

F is removed because it is not built by B anymore

Note that although this issue is triggered (for me) from the changes I
proposed last year to allow feed generation from nested inlines
coalescing it to be page-based instead of destpage-based
(bb8f76a4a04686def8cc6f21bcca80cb2cc3b2c9 and
72c8f01b36c841b0e83a2ad7ad1365b9116075c5) there is potential for it
popping up in other cases.

Specifically, the logic for the removal of dependent pages currently
relies on the assumption that each output has a single generator. My
changes caused this assumption to be violated, hence the error, but
other cases may pop up for other plugins in the future.

I have a [patch] fixing this issue (for feeds specifically, i.e. only
the problem I am actually having) on top of my mystuff branch, but
since that also has heaps of other unrelated stuff, you may want to just
pick it from my gitweb.

The patch changes the will_render() for feeds to be based on the page
rather than on the destpage, matching the fact that for nested inlines
it's the inner page that is ultimately responsible for generating the
feed.

I've noticed that it requires at least two full rebuilds before the
index is again in a sensible state. (On the first rebuild, all feeds
from nested inlines are actually removed.)

While the patch is needed because there are legitimate cases in which
nested feeds are needed (for example, I have an index page that inlines
index pages for subsection of my site, and I want those feed from
being visible), there are other cases when one may want to skip feed
generation from nested inlines.

If you look at org mode, the link to the Discussion page is not there (has a question mark), as if it didn't exist. But--through the search--I discovered that the Discussion page does exist actually: Discussion.

So, there is a bug that prevents a link to the existing Discussion page from appearing in the correct way on the corresponding main page. --Ivan Z.

I have heard repeated reports on http://mesh.openisp.ca/ that editing a page that has a waypoint in it will sometimes make that waypoint disappear from the main map. I have yet to understand why that happens or how, but multiple users have reported that.

A workaround is to rebuild the whole wiki, although sometimes re-editing the same page will bring the waypoint back on the map.

I have been able to reproduce this by simply creating a new node. It will not show up on the map until the wiki is rebuilt or the node is resaved. -- anarcat

The listdirectives` directive doesn't register a link between the page and the subpages. This is a problem because then the orphans directive then marks the directives as orphans... Maybe it is a but with the orphans directive however... A simple workaround is to exclude those files from the orphans call... --anarcat

There's a distinction between wikilinks (matched by link(),
backlink() etc.) and other constructs that produce a
hyperlink. Some directives count as a wikilink (like tag)
but many don't (notably inline, map, listdirectives,
and orphans itself). As documented in
orphans, orphans will tend to list
pages that are only matched by inlines/maps, too.

The rule of thumb seems to be that a link to a particular
page counts as a wikilink, but a directive that lists
pages matching some pattern does not; so I think
listdirectives is working as intended here.
orphans itself obviously shouldn't count as a wikilink,
because that would defeat the point of it

Anything that uses a pagespec to generate links,
like inline and map, can't generate wikilinks, because
wikilinks are gathered during the scan phase, and pagespecs
can't be matched until after the scan phase has finished
(otherwise, it'd be non-deterministic whether all wikilinks
had been seen yet, and link() in pagespecs wouldn't work
predictably).

I suggest just using something like:

[[!orphans pages="* and !blog/* and !ikiwiki/directive/*"]]

This wiki's example of listing orphans has a
more elaborate pagespec, which avoids bugs, todo items etc.
as well.

No follow-up or objection for a while, so considering this to
be working as designed. --smcv

Seems I'm a bit late to butt in, but would it be possible to have two
further phases after the scan phase, the first running map and inline
and the second orphan? Then map and inline could log or register their
links (obviously somewhere were it won't change the result of the link function)
and orphan could take them into account. This logging could be
turned on by parameter to not waste time for users not needing this and
make it tunable (i.e. so that the user can decide which map directives count and which don't)

For someone using map and especially autoindex the output of the orphans directive
is simply wrong/useless (at least it is for me). And there is no easy workaround like for listdirectives
-- holger

Hmm. I think this can be done without introducing any "phases",
even, but it would require each plugin that generates links according
to a pagespec to have either a conditional call into the orphans plugin,
or a call to a new core function in ikiwiki that exists solely to
support the orphans plugin. Something like this, maybe:

# in map.pm, inline.pm, pagestats.pm etc., at scan time
if (IkiWiki::Plugin::orphans->can("add_reachable")) {
IkiWiki::Plugin::orphans::add_reachable($page, $pagespec);
}
# in orphans.pm (pseudocode; note that this does not *evaluate*
# $pagespec, only stores it, so it's OK to do this at scan time)
sub needsbuild ($pages)
for each page in $pages
clear $pagestate{location}{orphans}{reachable}
sub reachable ($location, $pagespec)
add $pagespec to @{$pagestate{location}{orphans}{reachable}}
# in preprocess function in orphans.pm (pseudocode)
# executed at build time, not at scan time, so pagespecs work
for each maybe_orphan with no links to it
for each location with a list of reachable pagespecs
make the page with the orphans directive depend on \
the page that is the location
for each of those pagespecs
if pagespec matches orphan
take orphan off the list
go to next orphan
output list of orphans

(Maybe parentlinks should also annotate the parent/ancestors of
each page as reachable from that page.)

Do other people (mainly Joey) think that'd be acceptable, or
too intrusive?

Taking this off the list of resolved bugs again while we think about it.

I suspect that in the presence of autoindex, what you really want might
be less "there's a link to it" and more "there's a path to it from
the root of the wiki", which is why I called the proposed function
"add_reachable". On the other hand, maybe that's too computationally
intensive to actually do; I haven't tried it.
--smcv

(I'll interpet Joeys silence as a good sign ;-). Is there a difference between "link to it" and "path to it"? If we assume autoindex produces bonafide "first class" links there shouldn't be one!?

So far your idea sounds great, says me without any knowledge of the source. I'll try to grok it. Is there a medium for silly questions, a wiki seems not the right fit for that? -- holger

Yes, there has to be a difference between a first class wikilink
and the thing to which map and inline can contribute.
map and inline use a pagespec to decide what they include,
and pagespecs can't be evaluated and get a correct answer until the
set of links has been collected, because their results often depend
on the set of links. Otherwise, suppose you had a page foo whose only
contents were this:

[[!inline pages="!backlink(foo)"]]

If inline generated links, it would inline exactly those pages that
it doesn't inline. That's never going to end well --smcv

We have to differentiate between what users of ikiwiki consider first class links and what internally is happening. For the user any link contributing to the structured access tree is first class. The code on the other hand has to differentiate between the static links, then generated links, then orphan links. Three "passes", even your proposed solution could be seen as adding another pass since the orphan plugin has to run after all the plugins generating (first class user) links. -- holger

I think the difference between your point of view, and what ikiwiki
currently implements / what its design is geared towards, is this:
ikiwiki says A links to B if the source code of A contains an
explicit link to B. You say A links to B if the compiled HTML
of A contains a link to B.

Would you agree with that characterization?

I suspect that "link in the source code" may be the more useful concept
when using links for backlinks (I think the original implementation is
http://c2.com/cgi/wiki?BackLink) and as pseudo-tags
(http://c2.com/cgi/wiki?WikiCategories). The fact that this is what
link() and backlink() mean could be better-documented: it's
entirely possible that the author of their documentation (Joey?)
thought it was obvious that that's what they mean, because they
were coming from a compiler/source-code mindset.

Also, backlinks become rather all-engulfing if their presence in
the compiled output counts as a link, since after a render pass, they
would all become bidirectional; and as I noted previously, if pagespecs
can match by linkedness (which we want) and plugins can generate lists
of links according to pagespecs (which we also want), then links in the
compiled output can certainly get into Russell's paradox-like
situations, such as the page that links to every page to which it
does not link.

For the special case of deciding what is orphaned, sure, it's the
compiled HTML that is the more relevant thing;
that's why I talked about "reachability" rather than "links".

How does that look? I can provide a patch for the base wiki if you guys really want... -- anarcat

What you dislike seems to be the default rendering of definition lists by
browsers. I don't think it's ikiwiki's place to override browser defaults
for standard markup in the document body, at least not in the default
antitheme. --Joey

When I create a link like [[cmd_test]] , the link appears as 'cmd test'.

Expected behavior:

I would like to be able to create links with underscores. I realize this is a feature, and I searched for ways to escape the underscore so it would appear, but I didn't find any.

as a workaround, you can use [[cmd__95__test|cmd_test]] (which will link to a page named "cmd test" at the url location "cmd_test") or [[cmd__95__test]] (which will link to a page named "cmd_test" at the url location "cmd__95__test"). i would, from my limited understanding of ikiwiki internals, consider the bug valid, and suggest that

explicit link text be not subject to de-escaping (why should it; this would be the short term solution)

escaped page names never be used in user visible parts of ikiwiki (in my opinion, a user should not need to know about those internals, especially as they are configuration dependant (wiki_file_regexp))

note that in wikilink, that very behavior is documented; it says that "[[foo_bar|Sandbox]]" will show as "foo bar". (although you can't tell that apart from "foo_bar" easily because it's a hyperlink).

i assume that this behavior stems from times when wikilinks and directives were not distinguished by [[ vs [[! but by the use of whitespace in directives, so whitespace had to be avoided in wikilinks.

having hacked around in the link plugin, i can confirm that the link texts are explicitly de-escaped, and that when no pipe is inside the link (ie links like [[cmd_test]]), the string "cmd_test" is regarded as a link (that will subsequently be converted to a readable text) rather than as a readable text (for which a suitable link target is found automatically). --chrysn

When an ikiwiki instance is holding a lock, a web user clicking on "add comment" (for example) will have to wait for the lock to be released. However, all they are then presented with is a web form. Perhaps CGI requests that are read-only (such as generating a comment form, or perhaps certain types of edits) should ignore locks? Of course, I'd understand that the submission would need to wait for a lock. — Jon

Ikiwiki has what I think of as the Big Wiki Lock (remembering the "Big
Kernel Lock"). It takes the exclusive lock before loading any state,
to ensure that any changes to that state are made safely.

A few CGI actions that don't need that info loaded do avoid taking the
lock.

In the case of showing the comment form, the comments
plugin needs CGI session information to be loaded, so it can check if
the user is logged in, and so it can add XSRF prevention tokens based on
the session ID. (Actually, it might be possible to rely on
CGI::Session's own locking of the sessions file, and have a hook that
runs with a session but before the indexdb is loaded.)

But, the comment form also needs to load the indexdb, in order to call
check_canedit, which matches a pagespec, which can need to look things
up in the indexdb. (Though the pagespecs that can do that are unlikely
to be relevant when posting a comment.)

I've thought about trying to get rid of the Big Wiki Lock from time to
time. It's difficult though; if two ikiwikis are both making changes
to the stored state, it's hard to see a way to reconcile them. (There
could be a daemon that all changes are fed thru using a protocol, but
that's really complicated, and it'd almost be better to have a single
daemon that just runs ikiwiki; a major architectural change.)

One way that almost seems it could work is to have a entry path
that loads everything read-only, without a lock. And then in read-only
mode, saveindex would be an error to run. However, both the commenting
code and the page edit code currently have the same entry path for
drawing the form as is used for handling the posted form, so they would
need to be adapted to separate that into two code paths. --Joey

This is possibly/probably due to my weird setup, which is that I have apache behind nginx, with the result that apache sees the client's IPv4 address as having been mapped to IPv6. i.e. ::ffff:10.11.12.13. That being the case, I currently need to specify that (with the ::ffff: prepended) if I want to whitelist (or more importantly blacklist) and IPv4 address.

It strikes me that this is liable to become more of a problem as people finally start using IPv6, so it might be worth ensuring that the code that compares IP addresses be able to treat the two formats (with and without the ffff's) as equivalent. --fil

Can't appear to get 'wiki' functions (i.e. editing) running when ikiwiki is running on a port other than the default (port 80). Somewhere in the processing it considers the base URL to exclude the port number and the websever throws back an error finding the page.

For example if you run on 'http://my.gear.xxx:8080/' then after clicking login (using default password auth) it will process and try to redirect you to 'http://my.gear.xxx/cgi-bin/ikiwiki.cgi'. I'm assuming that somewhere we've used the 'path' and the 'host' and dropped the remainder. I can figure out where this is yet but I'll post back if I get lucky.

-- fergus

NB: both the 'url' and the 'cgiurl' include the port and removing the port element provides the expected functionality.

I tried to reproduce this by making my laptop's web server use port
8080. Set up ikiwiki to use that in cgiurl and url, and had
no problem with either openid or password auth login.

Ikiwiki has had some changes in this area in the past year; you don't say
what version you were using. It could also be a problem with your web
server, conceviably, if didn't correctly communicate the port to the cgi
program. --Joey

I did think of that so threw a 'printenv' script to check the port was arriving
right.

SERVER_PORT=8181
HTTP_HOST=zippy0.ie0.cobbled.net

[ ... ]

In apache, HTTP_HOST includes the port. This is not part of the CGI
spec it seems, but perl's CGI module seems to rely on it,
in virtual_port:

That'll be an interesting discussion as I'd suggest that HTTP headers are defined in the CGI specification as client headers and thus what thttpd is doing is wrong (i.e. mangling the client's own representation). Whether a CGI client should trust HTTP header over the server is probably already settled by convention.

I originally set up ikiwiki by using the debian package, but had some odd issues, so i figured i'd try installing from git. To do that i uninstalled the debian package and then did the Makefile dance from the git dir. In that process the original dirs configured in templatedir underlaydir in my wiki were deleted; HOWEVER when rebuilding the script just went ahead and did not even note the lack of those dirs. It would be nice if it threw errors if the dirs were configured, but non-existant.

Hmm. This behavior was explicitly coded into ikiwiki for underlay dirs:
commit.
Pity I didn't say why, but presumably there are cases
where one of the underlaydirs is expected to be missing, or where
this robustness of not crashing is needed.

The situation with missing templatedirs is more clear: When
it's looking for a given template file it just tries to open it in each
directory in turn, and uses the first file found; checking that a
directory exists would be extra work and there's a nice error message if
a template cannot be found. --Joey

I'd agree with the thought behind that ... if it actually had thrown an error. However it did not. How about just checking the config variables when the template and/or config is set up? --Mithaldu

I just tried to clone the git repo onto a windows machine to test things out a bit and it turns out i cannot even successfully checkout the code because of those colons. Would a patch changing those to underscores be accepted? --Mithaldu

Well, this is a difficult thing. Ikiwiki has a configuration setting to
prevent it writing filenames with colons, but for backwards compatability
that is not enabled by default. Also nothing would stop people from
making commits that added filenames with colons even if it were disabled
in ikiwiki. I don't know that trying to work around obscure limitations
in OSs that I've never heard of ikiwiki being used on is worth the bother
TBH, but have not really made up my mind. --Joey

I'm not trying to run it there. Ikiwiki is way too friggin' weird to try that. I just want to be able to check out the main repo so i can work in a native editor. Right now your core repository is downright hostile to cross-platform development in any way, shape or form. (Just plain splitting the docs from the code would work too.) --Mithaldu

Does(n't) cygwin handle the filename limitation/translations? If so, can you check out via git inside a cygwin environment? — Jon

That actually allows me to check things out, but the resulting repo isn't compatible with most of the rest of my system, so it's extremely painful. --Mithaldu

I'm using the most recent release of ikiwiki (3.20110905), the Perl shipped with SuSE 11.4 (v5.12.3), and built and installed xapian 1.2.7 from source, as it seems the current stable version that's encouraged for use by xapian.

After enabling the search plugin and pointing ikiwiki to the omega program, rerunning ikiwiki --setup, and attempting a search, all searches return 0 results. No errors are reported by omindex or ikiwiki while producing the indexes in .ikiwiki/xapian/*, and the files appear to contain the indexed data. I don't think it's a problem in indexing.

When running omega by hand in the .ikiwiki/xapian directory, providing queries on the command-line, runs correctly but again provides no results.

I found that Debian stable is currently shipping 1.2.3, and on a hunch, I built that version, and searching now works fine. This looks like the usage of xapian's query template has changed somewhere between 1.2.3 and 1.2.7. Someone more familiar with xapian's query template language should be able to figure out what needs to be changed more specifically.

Debian has 1.2.7 now, and I have it installed and searching is working
fine with it. --Joey

I have this same issue. I tried xapian version 1.2.5. 1.2.8, 1.2.13. I will try and see if installing 1.2.3 fixes this issue. --Ramsey

If I try to upload a second attachment (or remove the previously uploaded attachment), no upload happens. Instead the page gets created. No matter what I typed in, I just get a map to show the attachment. Now I can edit this page and everything is fine again.

Another workaround is to first save the text and then edit and upload the rest.

I mean the map directive.
It was ikiwiki 3.20110430.
Tried Firefox and uzbl (webkit) with or without javascript.

Just updated to 3.20110905. Now the problem has changed. Instead of saving the page with the second upload and leading me to it, it leaves me in the editform but creates the page anyway.
When saving I get informed, that someone else created the page. Obviously it was ikiwiki itself with the mentioned map:
[[!map pages="path/to/page/* and ! ...

This told me that autoindex is the bad guy. Deactivating this plugin helps out. Don't know if this is worth fixing... I can live without that plugin. --bacuh

The right fix would probably be for do=create to allow replacing a page
in the transient underlay without complaining (like the behaviour that
do=edit normally has).

It turns out that with autoindex_commit => 0, the failure mode is
different. The transient map is created when you attach the
attachment. When you save the page, it's written into the srcdir,
the map is deleted from the transientdir, and the ctime/mtime
in the indexdb are those of the file in the srcdir, but for some
reason the HTML output isn't re-generated (despite a refresh
happening). --smcv

When the aggregate plugin was used for a feed and this is removed (or the
same feed name given a different rss feed), the old entries don't
automatically vanish.

I think that if it was just removed, they are never GC'd, because the
expiry code works on the basis of existing feeds. And if it was replaced,
old items won't go away until expirecount or expireage is met.

To fix it probably needs an explicit check for items aggregated by feeds
that no longer provide them, Catching old items for feeds that were changed
to a different url may be harder yet. --Joey

Wikis are great tools for collaborative content of all types, but the choice for website creators who want a level of collaboration seem to have to choose between a static website, a wiki that anyone (or all members) can edit, or an overkill customized web app.

A simple innovation that needs to propagate through wiki software is adding the ability to suggest edits and accept those edits. Perhaps you want a wiki that anyone can suggest and edit, but only registered users can edit freely or accept edits. Or you want anyone, including members, to only be able to suggest edits, and only have moderators able to approve edits and edit freely. Etc, etc.

Ikiwiki always has some work in this area; there is
the moderatedcomments plugin and the checkcontent hook.
The hook allows, for example a plugin to reject changes
with spam links or swear words. A plugin could also use
it to save the diff for later moderation.

I think the difficulty
is in the moderation interface, which would need to apply the diff
and show the resulting page with the changes somehow evident (for users
who can't just read diffs), and would have to deal with conflicting
edits, etc. --Joey

Hi, I created [[sandbox/subpage]] then I deleted it with the "remove" button.
After confirmation there was a message about a xapian error (My bad, I did not write down the exact error message).
Now, accessing ?sandbox/subpage leads my browser complains about a redirect loop. JeanPrivat

Uh. Now the bug of redirect loop seems to have solved itself. However, I don't know if the xapian error need to be investigated. But I found another bug. JeanPrivat

Apache will return 403 (Forbidden) instead of 404 (Not Found) if the
Indexes option is turned off. This is because with Indexes turned on,
it considers it something it might be able to serve in the future. With
Indexes off, it will never serve that page in the future (unless
Indexes is turned back on).

The 404 plugin code only checks for 404, not 403. It should check for both.

There are plenty of reasons a web server might 403. In most of those
cases, trying to create a page where the forbidden content is is not the
right thing for ikiwiki to do. --Joey

The table plugin seems to be unable to read a CSV file that uses \r\n for line delimiters.
The same file with \r works fine. The error message is "Empty data".
--liw

I was seeing this as well on an Ubuntu 11.04 system with Ubuntu 11.04, Perl 5.10.1, IkiWiki 3.20110124ubuntu1, and libtext-csv-perl 1.21-1, all installed from APT. However, when I removed libtext-csv-perl from APT and installed it from CPAN, the problem went away. FWIW, what CPAN grabbed was MAKAMAKA/Text-CSV-1.21.tar.gz. --micahrl

The use of "DirectoryIndex index", when combined with multiviews, is intended
to serve up a localized version of the index.??.html file.

But, if the site's toplevel index page has a discussion page, that
is "/index/discussion/index.html". Or, if the img plugin is used to scale
an image on the index page, that will be "/index/foo.jpg". In either case,
the "index" directory exists, and so apache happily displays that
directory, rather than the site's index page!

Ack, we do have a problem. Seems like ikiwiki's use of index/ as
the directory for homepage's sub-pages and attachments makes it
conflict deeply with Apache's MultiViews: as the MultiViews
documentation
says, index.* are considered as possible matches only if the
index/ directory does not exist. Neither type maps nor
mod_mime config parameters seem to allow overriding this behavior.
Worse even, I guess any page called index would have the same
issues, not only the wiki homepage.

I can think of two workarounds, both kinda stink:

Have the homepage's targetpage be something else than
index.html.

Have the directory for the homepage's sub-pages and attachments
be something else than index.

I doubt either of those can be implemented without ugly special
casing. Any other idea? --intrigeri

I'm not sure how well that fits into IkiWiki's structure, though;
perhaps the master language could be responsible for generating the
type-map on behalf of all slave languages, or something?

Another possibility would be to use filenames like index.html.en
and index.html.fr, and set DirectoryIndex index.html? This could
get problematic for languages whose ISO codes conventionally mean
something else as extensions (Polish, .pl, is the usual example,
since many sites interpret .pl as "this is a (Perl) CGI").
--smcv

There is something to be said about "index/foo" being really ugly
and perhaps it would be nice to use something else. There does not
appear to even be one function that could be changed; "$page/foo" is
hardwired into ikiwiki in many places as a place to dump subsidiary
content -- and it's not even consistent, since there is also eg,
"$page.rss". I agree, approaching it from this direction would be a
mess or a lot of work.

Type maps seem like a valid option, but also a lot of clutter.

index.html.pl does seem to be asking for trouble, even if apache
can be configured to DTRT. It would make serving actual up perl scripts
hard, at least. But that is some good out of the box thinking..
perhaps "index.foo.pl.html"?

However, that would mean that
web servers need to be configured differently to serve translated
and non-translated sites. The current apache configuration for po
can be used with non-po sites and they still work. --Joey

I am vulnerable to the same problem because I use MultiViews, though I don't use the po module;
I have to serve both Australian English and American English for my company's website
(for SEO purposes; certain words that relate to our products are spelt differently in US and Australian English, and we need to be able to be googled with both spellings).
I'm just fortunate that nobody has thought to add attachments to the front page yet.
I raise this to point out that this is going to be a recurring problem that won't necessarily be fixed by changing the po module in isolation.

One could argue that "index" is already a special case, since it is the top page of the site.
Things like parentlinks already use a special case for the top page (checking the variable HAS_PARENTLINKS).
Likewise, when --usedirs is true, index is treated as a special case, since it generates "index.html" and not "index/index.html".

Unfortunately, I'm not sure what the best approach to solving this would be.
--KathrynAndersen

For security reasons, one of the sites I'm in charge of uses a Reverse Proxy to grab the content from another machine behind our firewall.
Let's call the out-facing machine Alfred and the one behind the firewall Betty.

For the static pages, everything is fine. However, when trying to use the search, all the links break.
This is because, when Alfred passes the search query on to Betty, the search result has a "base" tag which points to Betty, and all the links to the "found" pages are relative.
So we have

It works for me, but it has the odd side-effect of prefixing links with a space. Fortunately that doesn't seem to break browsers.
And I'm sure someone else could come up with something better and more general.

The <base href> is required to be genuinely absolute (HTML 4.01 §12.4).
Have you tried setting url to the public-facing URL, i.e. with alfred
as the hostname? That seems like the cleanest solution to me; if you're
one of the few behind the firewall and you access the site via betty
directly, my HTTP vs. HTTPS cleanup in recent versions should mean that
you rarely get redirected to alfred, because most URLs are either
relative or "local" (start with '/'). --smcv

I did try setting url to the "Alfred" machine, but that doesn't seem clean to me at all, since it forces someone to go to Alfred when they started off on Betty.
Even worse, it prevents me from setting up a test environment on, say, Cassandra, because as soon as one tries to search, one goes to Alfred, then Betty, and not back to Cassandra at all.
Hardcoded solutions make me nervous.

I suppose what I would like would be to not need to use a <base href> in searching at all.
--KathrynAndersen

<base href> is not required to be absolute in HTML5, so when
html5: 1 is used, I've changed it to be host-relative in most cases.
I think that at least partially addresses this bug report,
particularly if we generate HTML5 by default like I've suggested.

The <base> is there so we can avoid having to compute how to
get to (the virtual directory containing) the root of the wiki from
ikiwiki.cgi, which might well be somewhere odd like /cgi-bin/.
I think there are probably other things that it fixes or simplifies.
--smcv

On FreeBSD, perl defaults to installation in /usr/local/bin/perl since it is not a part of the base system. If the option to create symlinks in /usr/bin is not selected, > building and running ikiwiki will fail because the shebang lines use #!/usr/bin/perl [args]. Changing this to #!/usr/bin/env -S perl [args] fixes the issue.

I think this should be a concern of ikiwiki's official FreeBSD port.

At any rate, even if it is decided that ikiwiki should be fixed, then it is probably better to use
$installbin/perl from -MConfig and not the env hack.

In XML, fragment identifiers are of type ID, and there can only be a single attribute of type ID per element. Therefore, in XHTML 1.0 the id attribute is defined to be of type ID. In order to ensure that XHTML 1.0 documents are well-structured XML documents, XHTML 1.0 documents MUST use the id attribute when defining fragment identifiers on the elements listed above. See the HTML Compatibility Guidelines for information on ensuring such anchors are backward compatible when serving XHTML documents as media type text/html.

At least my setup on kapsi.fi always prints 404 Not Found after adding a page with non-ascii characters in name. But the page exists and is visible after the 404 with url encoding and the blog page is inlined correctly on the feed page.

Apparently ikiwiki.info does not complain with 404. Should the character encoding be set in wiki config?

Happens also after editing the page. Here's an example:

page name displayed in 404: http://mcfrisk.kapsi.fi/skiing/posts/Iso-Sy%F6te%20Freeride%202011%20Teaser.html?updated

page name in the blog feed: http://mcfrisk.kapsi.fi/skiing/posts/Iso-Sy%C3%B6te%20Freeride%202011%20Teaser.html

Difference is in the word Iso-Syöte. Pehaps also the browsers is part of
the game, I use Iceweasel from Debian unstable with default settings.

I remember seeing this problem twice before, and both times it was caused
by a bug in the web server configuration. I think at least one case it was
due to an apache rewrite rule that did a redirect and mangled the correct
encoding.

I recommend you check there. If you cannot find the problem with your web
server, I recommend you get a http protocol dump while saving the page,
and post it here for analysis. You could use tcpdump, or one of the
browser plugins that allows examining the http protocol. --Joey

Server runs Debian 5.0.8 but I don't have access to the Apache configs. Here's the tcp stream from wireshark without cookie data, page name is testiä.html. I guess page name is in utf-8 but in redirect after post it is given to browser with 8859-1.

It looks like there is no way to logout of ikiwiki at present, meaning that if you edit the ikiwiki in, say, a cybercafe, the cookie remains... is there some other security mechanism in place that can check for authorization, or should I hack in a logout routine into ikiwiki.cgi?

Click on "Preferences". There is a logout button there. --liw

It would be nice if it were not buried there, but putting it on the
action bar statically would be confusing. The best approach might be to
use javascript. --Joey

I agree that javascript seems to be a solution, but my brain falls
off the end of the world while looking at ways to manipulate the DOM.
(I'd argue also in favor of the openid_provider cookie expiring
in less time than it does now, and being session based)

(The openid_provider cookie is purely a convenience cookie to
auto-select the user's openid provider the next time they log
in. As such, it cannot be a session cookie. It does not provide any
personally-identifying information so it should not really matter
when it expires.) --Joey

It would be nice to move navigational elements to the upper right corner
of the page...

I was referred to this page from posting to the forum. I am also interested in being able to use user class and status to modify the page. I will try to put together a plugin. From what I can see there needs to be a few items in it.

It should expose a link to a dedicated login page that, once logged in, returns the user to the calling page, or at least the home page. I have started a plugin to do this: justlogin

it needs to expose a link to a little json explaining the type of user and login status.

it should expose a link that logs the person out and returns to the calling page, or at least the home page.

Then there would need to be a little javascript to use these links appropriately. I have little javascript experience but I know that can be done. I am less sure if it is possible to add this functionality to a plugin so I'll start with that. If no one objects I will continue to post here if I make progress. If anyone has any suggestions on how to modify my approach to code it in an easier way I'd appreciate the input. justint

I'd like the more plugin and RSS to play better together. In the case of the html generation of the main page of a blog, I'd like to get the first paragraph out, but keep RSS as a full feed.

Maybe there is a different plugin (I also tried toggle)?

I am not a fan of the more directive (thus the rant about it sucking
embedded in its example). But I don't think
that weakening it to not work in rss feeds is a good idea, if someone
wants to force users to go somewhere to view their full content,
they should be able to do it, even though it does suck.

The toggle directive will degrade fairly well in an rss feed to
display the full text. (There is an annoying toggle link that does
nothing when embedded in an rss feed). --Joey

I also note, that at least currently, more seems to break on a few pages, not being parsed at all when aggregated into the front page.

It's just a simple directive, it should work anywhere any directive will,
and does as far as I can see. Details? --Joey

Optimally, UTF-16 (which is ubiquitous in the Windows world) and UTF-32 should be fully supported, probably by converting to mostly-UTF-8 and using &#xXXXX; or &#DDDDD; XML escapes where necessary.

Suboptimally, UTF-16 and UTF-32 should be converted to UTF-8 where cleanly possible and a warning printed where impossible.

Reading the wikipedia pages about UTF-8 and UTF-16, all valid Unicode characters are representable in UTF-8, UTF-16 and UTF-32, and the only errors possible with UTF-16/32 -> UTF-8 translation are when there are encoding errors in the original document.

Of course, it's entirely possible that not all browsers support utf-8 correctly, and we might need to support the option of encoding into CESU-8 instead, which has the side-effect of allowing the transcription of UTF-16 or UTF-32 encoding errors into the output byte-stream, rather than pedantically removing those bytes.

An interesting question would be how to determine the character set of an arbitrary new file added to the repository, unless the repository itself handles character-encoding, in which case, we can just ask the repository to hand us a UTF-8 encoded version of the file.

So, in English, page text inside a cut directive will not be filtered.
Because the cut directive takes the text during the scan pass, before
filtering happens.

Commit 192ce7a238af9021b0fd6dd571f22409af81ebaf and
po vs templates has to do with this.
There I decided that filter hooks should only act on the complete
text of a page.

I also suggested that anything that wants to reliably
s/FOO/BAR/ should probably use a sanitize hook, not a filter hook.
I think that would make sense in this example.

I don't see any way to make cut text be filtered while satisfying these
constraints, without removing cutpaste's ability to have forward pastes
of text cut laster in the page. (That does seems like an increasingly
bad idea..) --Joey

OK -- so the FOO/BAR thing was only a very stripped-down example, of
course, and the real thing is being observed with the
getfield plugin. This one needs to run beforepreprocessing, for its {{$page#field}} syntax is (a) meant to be usable
inside ikiwiki directives, and (b) the field values are meant to still be
preprocessed before being embedded. That's why it's using the filter
hook instead of sanitize.

Would adding another kind of hook be a way to fix this? My idea is that
cut (and others) would then take their data not during scanning, but
afterfiltering.

If comments_allowdirectives is set, previewing a comment can run
directives that create files. (Eg, img.) Unlike editpage, it does not
keep track of those files and expire them. So the files will linger in
destdir forever.

Probably when the user then tries to save the comment, ikiwiki will refuse
to overwrite the unknown file, and will crash.
--Joey

I'd like a way to always ask the RCS (Git) to update a file's mtime in
refresh mode. This is currently only done on the first build, and later
for --gettime --rebuild. But always rebuilding is too heavy-weight for
this use-case. My options are to either manually set the mtime before
refreshing, or to have ikiwiki do it at command. I used to do the
former, but would now like the latter, as ikiwiki now generally does this
timestamp handling.

From a quick look, the code in IkiWiki/Render.pm:find_new_files is
relevant: if (! $pagemtime{$page}) { [...].

This could be done via a needsbuild hook. The hook is passed
the list of changed files, and it should be safe to call rcs_getmtime
and update the pagemtime for each.

That lets the feature be done by a plugin, which seems good, since
rcs_getmtime varies between very slow and not very fast, depending on
VCS.

AFAICS, the only use case for doing this is if you commit changes and
then delay pushing them to a DVCS repo. Since then the file mtime will
be when the change was pushed, not when it was committed. But I've
generally felt that recording when a change was published to the repo
of a wiki as its mtime is good enough. --Joey

Only way for this to be improved would be for the svn plugin to
explicitly check the file for conflict markers. I guess it could
change the error message then, but the actual behavior of putting the
changed file back in the editor so the user can recommit is about right
as far as error recovery goes. --Joey

Lighttpd apparently sets REDIRECT_STATUS=200 for the server.error-handler-404 page. This breaks the 404 plugin which checks this variable for 404 before processing the URI. It also doesn't seem to set REDIRECT_URL.

I was able to fix my server to check the REQUEST_URI for ikiwiki.cgi and to continue processing if it was not found, passing $ENV{SEVER_NAME} . $ENV{REQUEST_URI} as the first parameter to cgi_page_from_404. However, my perl is terrible and I just made it work rather than figuring out exactly what to do to get it to work on both lighttpd and apache.

This is with lighttpd 1.4.19 on Debian.

/cgi-bin/ikiwiki.cgi?do=goto also provides redirection in the same way,
if that's any help? You might need to set the lighttpd 404 handler to
that, then compose REDIRECT_URL from other variables if necessary.

I originally wrote the plugin for Apache; weakish contributed the
lighttpd docs and might know more about how to make it work there.
--smcv

As I said, I got it working for me, but somebody who knows perl should probably look at it with the aim of making it work for everyone.
I considered having lighttpd construct a proper url for the 404 redirect itself, but I don't know if it can do something like that or not.
For what it's worth, here's the change I made to the module:

It seems that rebuild a wiki (ikiwiki --rebuild) after changing the underlaydir config option doesn't remove the pages coming from the previous underlaydir.

I've noticed this with the debian package version 3.20100102.3~bpo50+1.

Perhaps it is possible to improve this or mention it in the manual page?

--prosper

--rebuild causes ikiwiki to throw away all its info about what it built
before, so it will never clean up pages that have been removed, by any
means. Suggest you do a --refresh, possibly followed by a --rebuild
if that is really necessary. --Joey

The problem with this patch is that, in a recent fix to the same
plugin, I made @pages come from $form->field("page"), and
that, in turn is already run through decode_form_utf8 just above the
code you patched. So I need to understand why that is apparently not
working for you. (It works fine for me, even when deleting a file named
"umläute" --Joey

Update, having looked at the file in the src of the wiki that
is causing trouble for remove, it is: uml\303\203\302\244ute.mdwn
And that is not utf-8 encoded, which, represented the same
would be: uml\303\244ute.mdwn

I think it's doubly-utf-8 encoded, which perhaps explains why the above
patch works around the problem (since the page name gets doubly-decoded
with it). The patch doesn't fix related problems when using remove, etc.

Apparently, on apoca's system, perl encodes filenames differently
depending on locale settings. On mine, it does not. Ie, this perl
program always creates a file named uml\303\244ute, no matter
whether I run it with LANG="" or LANG="en_US.UTF-8":

perl -e 'use IkiWiki; writefile("umläute", "./", "baz")'

Remains to be seen if this is due to the older version of perl used
there, or perhaps FreeBSD itself. --Joey

Fixing it is really a bit involved, see commit
f1ddf4bd98821a597d8fa1532092f09d3d9b5483. The fix I committed fixes
bestlink to not return deleted pages, but only after the needsbuild and
scan hooks are called. So I was able to fix it for every case except the
one you gave! Sorry for that. To fix it during beedsbuild and scan,
a much more involved approach would be needed. AFAICS, no existing plugin
in ikiwiki uses bestlink in needsbuild or scan though.

Cool that was fast! Well at least half the bug is solved For now I'll
probably try using a workaround if using bestlink within the needsbuild
or scan hooks. Maybe by testing if pagemtime equals zero. --harishcm

Yeah, and bestlink could also do that. However, it feels nasty to have
it need to look at pagemtime. --Joey

The map directive sort by pagename. That looks kind of odd, when used together with show=title. I would expect it to sort by title then.

This would be quite hard to fix. Map sorts the pages it displays by page
name, which has the happy effect of making "foo/bar" come after "foo";
which it has to do, so that it can be displayed as a child of the page
it's located in. If sorting by title, that wouldn't hold. So, map
would have to be effectively totally rewritten, to build up each group
of child pages, and then re-sort those. --Joey

Ok, you are right, that does would break the tree. This made me think that I do not
need to generate a tree for my particular use case just a list, so i thought i could use inline instead.
This created two new issues:

inline also does sort by pagename even when explicitly told to sort by title.

I cannot get inline to create a list when the htmltidy plugin is switched on. I have a template which is enclosed in an li tag, and i put the ul tag around the inline manually, but htmltidy breaks this. --martin

You might want to check if the report plugin solves your problem. It can sort by title, among other things. --KathrynAndersen

I'm trying to make a pretty theme for ikiwiki and I'm making progress (or at least I think I am :-). However I've noticed an issue when it comes to theming. On the front page the wiki name is put inside the "title" span and on all the other pages, it's put in the "parentlinks" span. See here:

I understand the logic behind doing this (on the front page it is the title as well as the name of the wiki) however if you want to do something different with the title of a page vs. the name of the wiki it makes things pretty tricky.

I'll just modify the templates for my own site but I thought I'd report it as a bug in the hopes that it will be useful to others.

aggregate takes a name parameter that specifies a global name
for a feed. This causes some problems:

If a site has multiple pages that aggregate, and they use the same
name, one will win and get the global name, the other will claim it's
working, but it's really showing what the other aggregated.

If an aggregate directive is moved from page A to page B, and the wiki
refreshed, aggregate does not realize the feed moved, and so it will
keep aggregated pages under A/feed_name/*. To work around this bug,
you have to delete A, refresh (maybe with --aggregate?), and then add B.

Need to find a way to not make the name be global. Perhaps it needs to
include the name of the page that contains the directive?

The remove plugin does not report an error if git rm fails. (It
probably doesn't if other VCS backends fail too). This can happen for example
if a page in your source directory is not a tracked file for whatever reason
(in my case, due to renaming the files and forgetting to commit that change).

I was just hovering over the '...' next to the backlinks on a page on
http://ikiwiki.info/. In terms of the size of my browser window, this was
towards the bottom-right of the screen.

When I hovered over the '...', the additional backlinks float appeared. This
caused the page length to grow down, meaning a horizontal scrollbar was added
to the page. This meant the text reflowed, and the '...' moved outside of my
mouse pointer region.

This caused an infinite loop of box appears... text moves, box disappears...
box re-appears.. which was not very visually pleasant.

In general I think that the onhover float is a bit of bad UI. Even a truncated
list of backlinks looks cluttered due to there being no delimiters. I moved to
having an always-complete list of backlinks and having them as LI elements
inside a UL to make it look neater, although I appreciate that would make some
pages very long indeed.

How about doing something a little like toggle for the excess
items instead?

An additional, related issue: if the box expands beyond the bottom of the
page, you might move your mouse pointer to the scrollbar in order to move
further down the list, but of course then you are outside the hover region.

the server is running gcc v3.3.5 (at this point, this is the main
difference between the working system and my box.)

I've removed the locale declarations from both the config file and
the environment variable.

I've also modified the page template and have my templates in a non
standard location. The wiki compiles fine, with the template, but
might this be an issue? The CGI script doesn't (seem) to load under
the new template, but I'm not sure how to address this issue.

All of the required/suggested module dependencies are installed
(finally) to the latest version including (relevantly)
CGI::FormBuilder 3.0501.

I'm running ikiwiki v3.08. Did I mention that it works perfectly in
nearly every other way that I've managed to test thusfar?

I suspect that your perl is too old and is incompatible with the version of CGI::FormBuilder you have installed.

Is so, it seems likely that the same error message can be reproduced by running a simple command like this at the command line:

In ikiwiki 2.66, SVG images are not recognized as images. In ikiwiki.pm,
the hardcoded list of image file extensions does not include ".svg", which
it probably should unless there's some other issue about rendering SVGs?

The 'img' plugin also seems to not support SVGs.

SVG images can only be included via an <object>, <embed>, or
<iframe> tag. Or, perhaps as inline SVG.
The htmlscrubber strips all three tags since they can easily
be used maliciously. If doing inline SVG, I'd worry that the svg file
could be malformed and mess up the html, or even inject javascript. So,
the only options seem to be only supporting svgs on wikis that do not
sanitize their html, or assuming that svgs are trusted content and
embedding them inline. None of which seem particularly palatable.

I suppose the other option would be converting the svg file to a static
image (png). The img plugin could probably do that fairly simply.
--Joey

This seems to have improved since; at least chromium can display svg
images from <img> tags. Firefox 3.5.19 did not in my testing.

So, svgs can now be included on pages by linking to them, or by using
the img directive. The most portable thing is to use the img directive
plus some size, which forces them to be resized and a png to actually
be displayed.

I'm working on inline SVG and MathML support in ikiwiki and I've
modified my htmlscrubber to sanitize SVG and MathML using the
whitelists from html5lib. Here's a patch. I've also made some
notes about this here: svg.

I suspect that this bug may have caught the eye of anyone interested
in this sort of thing. I'll elaborate a bit on my user page to avoid
getting off-topic here. --JasonBlevins, October 21, 2008

Yes, these need to be fixed. But note that the localised texts come back
into ikiwiki and are used in various places, including plugins.
Including, possibly, third-party plugins. So localising the buttons would
seem to require converting from the translations back into the C locale
when the form is posted. --Joey

Wouldn't it be more easy to change all calls to the corrects ones (including in plugins) ?
For instance in the same file (CGI.pm): elsif ($form->submitted eq gettext("Save Page")) {.
That way no conversion to the C locale is needed.
gettext use should just be publicized in documentation (at least in write). --bbb

It would be easy, but it could break third-party plugins that hardcode
the english strings. It's also probably less efficient to run gettext
over and over. --Joey

In standards templates things seems wrongly written too. For instance in page.tmpl line like:

with EDITURL_TEXT variable initialized in Render.pm through a gettext call.

Am I wrong ?

No, that's not a sane way to localise the templates. The templates can be
translated by making a copy and modifying it, or by using a tool to
generate .mo files from the templates, and generate translated templates
from .po files. (See l10n for one attempt.) But pushing the
localisation of random strings in the templates through the ikiwiki
program defeats the purpose of having templates at all. --Joey

If not I can spend some time preparing patches for such corrections if it can help.

A ?PageSpec that is entirely negated terminals, such as "!foo and !bar"
matches all other pages, including all internal pages. This can lead to
unexpected results, since it will match a bunch of recentchanges pages,
etc.

Recall that internal-use pages are not matched by a glob. So "*" doesn't
match them. So if the pagespec is "* and !foo and !bar", it won't match
them. This is the much more common style.

Indeed, it seems what would be best would be for "!foo" to not match any
pages, unless it's combined with a terminal that positively matches pages
("* and !foo"). Although this would be a behavior change, with transition
issues.

Another approach would be to try to detect the case of an entirely negated
pagespec, and implicitly add "and !internal()" to it.

Either approach would require fully parsing the pagespec. And consider cases
like "!(foo and !bar)". Doesn't seem at all easy to solve. --Joey

It occurs to me that at least one place in ikiwiki optimizes by assuming
that pagespecs not mentioning the word "internal" never match internal
pages. I wonder whether this bug could be solved by making that part of
the definition of a pagespec, rather than a risky optimization
like it is now? That seems strange, though - having this special case
would make pagespecs significantly harder to understand. --smcv

The Atom and RSS templates use ESCAPE=HTML in the title elements. However, HTML-escaped characters aren't valid according to http://feedvalidator.org/.

Removing ESCAPE=HTML works fine, but I haven't checked to see if there are any characters it won't work for.

For Atom, at least, I believe adding type="xhtml" to the title element will work. I don't think there's an equivalent for RSS.

Removing the ESCAPE=HTML will not work, feed validator hates that just as
much. It wants rss feeds to use a specific style of escaping that happens
to work in some large percentage of all rss consumers. (Most of which are
broken).
http://www.rssboard.org/rss-profile#data-types-characterdata
There's also no actual spec about how this should work.

This will be a total beast to fix. The current design is very clean in
that all (well, nearly all) xml/html escaping is pushed back to the
templates. This allows plugins to substitute fields in the templates
without worrying about getting escaping right in the plugins -- and a
plugin doesn't even know what kind of template is being filled out when
it changes a field's value, so it can't do different types of escaping
for different templates.

The only reasonable approach seems to be extending HTML::Template with an
ESCAPE=RSS and using that. Unfortunately its design does not allow doing
so without hacking its code in several places. I've contacted its author
to see if he'd accept such a patch.

(A secondary bug is that using meta title currently results in unnecessry
escaping of the title value before it reaches the template. This makes
the escaping issues show up much more than they need to, since lots more
characters are currently being double-escaped in the rss.)

Update: Ok, I've fixed this for titles, as a special case, but the
underlying problem remains for other fields in rss feeds (such as
author), so I'm leaving this bug report open. --Joey

I'm curious if there has been any progress on better RSS output?
I've been prototyping a new blog and getting good RSS out of it
seems important as the bulk of my current readers use RSS.
I note, in passing that the "more" plugin doesn't quite do what
I want either - I'd like to pass a full RSS feed of a post and only
have "more" apply to the front page of the blog. Is there a way to do that?
-- ?dtaht

To be clear, the RSS spec sucks to such an extent that, as far as
I know, there is no sort of title escaping that will work in all
RSS consumers. Titles are currently escaped in the way
that tends to break the fewest according to what I've read.
If you're unlucky enough to
have a "&" or "<" in your name, then you may still run into
problems with how that is escaped in rss feeds. --Joey

It would be nice if the aggregate plugin would try to
extract the m/ctime out of each post and touch the files on the filesystem
appropriately, so that ikiwiki reflects the actual time of the post via the
inline plugin, rather than the time when the aggregation ran to pull the post in. --madduck

When I add a blog post, I see it on the wiki but it doesn't appear on History or RecentChanges. If I run hg status on the wiki source dir, I see the new file has been marked as A (ie, a new file that has not been commited).

If I then edit the blog post, then the file gets commited and I can see the edit on History and RecentChanges. The creation of the file remains unrecorded. --?buo

Ikiwiki calls rcs_add() if the page is new, followed by rcs_commit().
For mercurial, these run respectively hg add and hg commit. If the
add or commit fails, it will print a warning to stderr, you might check
apache's error.log to see if there's anything there. --Joey

The problem was using accented characters (é, í) on the change comments. I didn't have
an UTF-8 locale enabled in my setup file. By coincidence this happened for the first time
in a couple of consecutive blog posts, so I was mistaken about the root of the problem. I don't know if
you will consider this behavior a bug, since it's strictly speaking a misconfiguration but it
still causes ikiwiki's mercurial backend to fail. A quick note in the docs might be a good idea. For my part, please
close this bug, and thanks for the help. --?buo

So, in a non-utf8 locale, mercurial fails to commit if the commit
message contains utf8? --Joey

(Sorry for the delay, I was AFK for a while.) What I am seeing is this: in a non-utf8 locale, using mercurial "stand-alone" (no ikiwiki involved), mercurial fails to commit if the commit message has characters such as á. If the locale is utf8, mercurial works fine (this is with mercurial 1.0).

However, the part that seems a bit wrong to me, is this: even if my locale is utf8, I have to explicitly set a utf8 locale in the wiki's setup file, or the commit fails. It looks like ikiwiki is not using this machine's default locale, which is utf8. Also, I'm not getting any errors on apache's error log.

Wouldn't it make sense to use the machine's default locale if 'locale' is commented out in the setup file?

Ikiwiki wrappers only allow whitelisted environment variables
through, and the locale environment variables are not included
currently.

But that's not the whole story, because "machine's default locale"
is not very well defined. For example, my laptop is a Debian system.
It has a locale setting in /etc/environment (LANG="en_US.UTF-8").
But even if I start apache, making sure that LANG is set and exported
in the environment, CGI scripts apache runs do not see LANG in their
environment. (I notice that /etc/init.d/apache explocitly
forces LANG=C. But CGI scripts don't see the C value either.)
Apache simply does not propigate its runtime environment to CGI
scripts, and this is probably to comply with the CGI specification
(although it doesn't seem to completly rule out CGI's being passed
other variables).

If mercurial needs a utf-8 locale, I guess the mercurial plugin needs
to check if it's not in one, and do something sane (either fail
earlier, or complain, or strip utf-8 out of comments). --Joey

As far as I can tell, ikiwiki is not checking the SSL certificate of the remote host when using openid authentication. If so, this would allow for man-in-the-middle type attacks. Alternatively, maybe I am getting myself confused.

Test #1: Enter URL as openid server that cannot be verified (either because the certificate is self signed or signed by an unknown CA). I get no SSL errors.

Test #2: Download net_ssl_test from dodgy source (it uses the same SSL perl library, and test again. It seems to complain (on same site ikiwiki worked with) when it can't verify the signature. Although there is other breakage with the version I managed to download (eg. argument parsing is broken; also if I try to connect to a proxy server, it instructs the proxy server to connect to itself for some weird reason).

For now, I want to try and resolve the issues with net_ssl_test, and run more tests. However, in the meantime, I thought I would document the issue here.

-- Brian May

Openid's security model does not rely on the openid consumer (ie,
ikiwiki) performing any sanity checking of the openid server. All the
security authentication goes on between your web browser and the openid
server. This may involve ssl, or not.

Note that I'm not an openid expert, and the above may need to be taken
with a grain of salt. I also can make no general statements about openid
being secure. --Joey

For example, my openid is "http://joey.kitenet.net/". If I log in with
this openid, ikiwiki connects to that http url to determine what openid
server it uses, and then redirects my browser to the server
(https://www.myopenid.com/server), which validates the user and redirects
the browser back to ikiwiki with a flag set indicating that the openid
was validated. At no point does ikiwiki need to verify that the https url
is good.
--Joey

Ok, so I guess the worst that could happen when ikiwiki talks to the http
address is that it gets intercepted, and ikiwiki gets the wrong address.
ikiwiki will then redirect the browser to the wrong address. An attacker could
trick ikiwiki to redirect to their site which always validates the user
and then redirects back to ikiwiki. The legitimate user may not even notice.
That doesn't so seem secure to me...

All the attacker needs is access to the network somewhere between ikiwiki
and http://joey.kitenet.net/ or the ability to inject false DNS host names
for use by ikiwiki and the rest is simple.

-- Brian May

I guess that the place to add SSL cert checking would be in either
LWPx::ParanoidAgent or Net::OpenID::Consumer. Adding
it to ikiwiki itself, which is just a user of those libraries, doesn't
seem right.

It's not particularly clear to me how a SSL cert can usefully be
checked at this level, where there is no way to do anything but
succeed, or fail; and where the extent of the check that can be done is
that the SSL cert is issued by a trusted party and matches the domain name
of the site being connected to. I also don't personally think that SSL
certs are the right fix for DNS poisoning issues. --Joey

"Using SSL with certificates signed by a trusted authority prevents these kinds of
attacks by verifying the results of the DNS look-up against the certificate. Once
the validity of the certificate has been established, tampering is not possible.
Impersonating an SSL server requires forging or stealing a certificate, which is
significantly harder than the network based attacks."

With regards to implementation, I am surprised that the libraries don't seem to
do this checking, already, and by default. Unfortunately, I am not sure how to test
this adequately, see Debian bug #466055. -- Brian May

If I try to authenticate using openid to my site, it tries to create a http or https connection to the openid server. This doesn't work, because the direct connection is blocked by the firewall.

It would be good if ikiwiki supported setting up a proxy server to solve this.

I have found if I add:

newenviron[i++]="HTTPS_PROXY=http://host.domain.com:3128";

to IkiWiki/Wrapper.pm it solves the problem for https requests, however it obviously would be preferred if the proxy name is not hard coded.

Also, the ability to set HTTPS_CA_FILE and HTTPS_CA_DIR might benefit some people. Then again, it I can't see any evidence that the SSL certificate of the server is being checked. See the bug report I filed on this separate issue.

Unfortunately, HTTP_PROXY doesn't work for http:// requests, it looks like that library is different.

Update 2008-10-26:

Better solution, one that works for both http and https, and uses config options. It appears to work...

Rather than adding config file settings for every useful environment
variable, there is a ENV config file setting that can be used to set
any environment variables you like. So, no changed needed.
--Joey

One thing I don't like about using ikiwiki for tracking bugs is I don't
get notified when changes are made :-(.

Anyway, if you look at the code I pasted above, the environment variables
do not work for http:// - you have to use $ua->proxy(...) for them.
This is significant, because all openid servers in my version appear to have been
defined with http:// not https:// in /usr/share/ikiwiki/openid-selector/ikiwiki/openid/openid-jquery.js

Use $ua->env_proxy() to get it to read the environment variables. Then http:// does work.

Unfortunately this breaks https:// even more - but nothing I do seems to make https:// work anymore.

LWP::UserAgent defaults to not caring about proxy settings in
the environment. (To give control over the result, I guess?)
To get it to care, pass env_proxy => 1 to the constructor. Affected
plugins: aggregate, openid, pinger. This probably wants to be on
by default, and might not need to be configurable. --schmonz

Okay, in a real-world scenario it does need to be
configurable. A working implementation (tested with aggregate,
not tested with the other two plugins) is in my git, commit
91c46819dee237a281909b0c7e65718eb43f4119. --schmonz

Oh, and according to the LWPx::ParanoidAgent docs, "proxy support is
explicitly removed", so if ikiwiki can preferentially find that
installed, even with the above commit, openid won't be able to
traverse a proxy. --schmonz

After installing IkiWiki 2.16 on Mac OS X 10.4 server I attempted to use "/Library/Application\ Support/IkiWiki/Working\ Copies" as the parent of my $SRCPATH and get "skipping bad filename" errors for any .mdwn file in that directory:

The brokenlinks plugin falsely complains that
formatting has a broken link to smileys, if the smiley plgin
is disabled. While the page links to it inside a
conditional, and so doesn't show the link in this case, ikiwiki scans for
links w/o looking at conditionals and so still thinks the page contains the
link.

In markdown syntax, none of the other special characters get processed
inside a code block. However, in ikiwiki, wiki links and
preprocessor directives still get processed
inside a code block, requiring additional escaping. For example, [links
don't work](#here), but a <a href="../ikiwiki/wikilink/">wikilink</a> becomes HTML. --JoshTriplett

Indented lines provide a good way to escape a block of text containing
markdown syntax, but ikiwiki links like [[this]] are still
interpreted within such a block. I think that intepretation should not
be happening. That is I should be able to write:

Has there been any progress or ideas on this bug recently? I use an
expanded CamelCase regexp, and without much escaping in freelink text, or
url links, or in codeblocks I get IkiWiki's attempt at creating a "link
within a link".

I have no ideas other than perhaps once IkiWiki encounters [[ or the
position is reset with a backreference from a CamelCased word, further
processing of wikilinks is disabled until the position is reset and a "no
not makelinks" flag or variable is cleared.

I've come up with some really ugly workarounds to handle case specific
stuff like codeblocks but the problem creeps up again and again in
unexpected places. I'd be happy to come up with a patch if anyone has a
bright idea on a nice clean way (in theroy) to fix this. I'm out of ideas.

--CharlesMauch

I've moved the above comment here because it seems to be talking about
this bug, not the similar Smileys bug.

In the case of either bug, no, I don't have an idea of a solution yet.
--Joey

I've now solved a similar bug involving the smiley plugin. The code used
there should give some strong hints how to fix this bug, though I haven't
tried to apply the method yet. --Joey

As far, as I can see, smileys bug is solved by checking for code/pre. In
this case, however, this is not applicable. WikiLinks/directives should be
expanded before passing text to formatter, as their expansion may contain
markup. Directives should be processed before, as they may provide partial
markup (eg template ones), that have no sense except when in the page
cotext. Links should be processed before, because, at least multimarkdown may
try to expand them as anchor-links.

For now, my partial solution is to restrict links to not have space at the
start, this way in many cases escaping in code may be done in natural way
and not break copypastability. For example, shell 'if [[ condition ]];'
will work fine with this.

Maybe directives can also be restricted to only be allowed on the line by
themselves (not separated by blank lines, however) or something similar.

The header of subpages always links to its "superpage", even if it doesn't
exist. I'm not sure if this is a feature or a bug, but I would certainly prefer
that superpages weren't mandatory.

For example, if you are in 'example/page.html', the header will be something
like 'wiki / example / page'. Now, if 'example.html' doesn't exist, you'll have
a dead link for every subpage.

This is a bug, but fixing it is very tricky. Consider what would happen if
example.mdwn were created: example/page.html and the rest of example/
would need to be updated to change the parentlink from a bare word to a
link to the new page. Now if example.mdwn were removed again, they'd need
to be updated again. So example/ depends on example. But it's even more
tricky, because if example.mdwn is modified, we don't want to rebuild
example/*!

ikiwiki doesn't have a way to represent this dependency and can't get one
without a lot of new complex code being added.

Note that this code has now been added. In new terms, example/* has a
presence dependency on example. So this bug is theoretically fixable now.
--Joey

For now the best thing to do is to make sure that you always create
example if you create example/foo. Which is probably a good idea anyway..

Note that this bug does not exist if the wiki is built with the "usedirs"
option, since in that case, the parent link will link to a subdirectory,
that will just be missing the index.html file, but still nicely usable.
--Joey

Has bugs updating things if the bestlink of a page changes due to
adding/removing a page. For example, if Foo/Bar links to "Baz", which is
Foo/Baz, and Foo/Bar/Baz gets added, it will update the links in Foo/Bar
to point to it, but will forget to update the backlinks in Foo/Baz.

The buggy code is in refresh(), when it determines what
links, on what pages, have changed. It only looks at
changed/added/deleted pages when doing this. But when Foo/Bar/Baz
is added, Foo/Bar is not changed -- so the change it its
backlinks is not noticed.

To fix this, it needs to consider, when rebuilding Foo/Bar for the changed
links, what oldlinks Foo/Bar had. If one of the oldlinks linked to
Foo/Baz, and not links to Foo/Bar/Baz, it could then rebuild Foo/Baz.

Problem is that in order to do that, it needs to be able to tell that
the oldlinks linked to Foo/Baz. Which would mean either calculating
all links before the scan phase, or keeping a copy of the backlinks
from the last build, and using that. The first option would be a lot
of work for this minor issue.. it might be less expensive to just rebuild
all pages that Foo/Bar links to.

Keeping a copy of the backlinks has some merit. It could also be
incrementally updated.

This old bug still exists as of 031d1bf5046ab77c796477a19967e7c0c512c417.

And if Foo/Bar/Baz is then removed, Foo/Bar gets a broken link,
instead of changing back to linking to Foo/Baz.

This part was finally fixed by commit
f1ddf4bd98821a597d8fa1532092f09d3d9b5483.

If a file in the srcdir is removed, exposing a file in the underlaydir,
ikiwiki will not notice the removal, and the
page from the underlay will not be built. (However, it will be if the wiki
gets rebuilt.)

This problem is caused by ikiwiki storing only filenames relative to
the srcdir or underlay, and mtime comparison not handling this case.

A related problem occurs if changing a site's theme with the
theme plugin. The style.css of the old and new theme
often has the same mtime, so ikiwiki does not update it w/o a rebuild.
This is worked around in theme.pm with a special-purpose needsbuild hook.
--Joey

Web browsers don't word-wrap lines in submitted text, which makes editing a
page that someone wrote in a web browser annoying (gqip is vim user's
friend here). Is there any way to improve this?

See "using the web interface with a real text editor" on the tips
page. --JoshTriplett

Would it be useful to allow a "max width" plugin, which would force on commit the split of long lines ?

Please, no. That would wreak havoc on code blocks and arguments to
preprocessor directives, and it would make bulleted lists and quoted
blocks look bogus (because the subsequent lines would not match), among
other problems. On the other hand, if you want to propose a piece of
client-side JavaScript that looks at the active selection in a text area
and word-wraps it, and have a plugin that adds a "Word-Wrap Selection"
button to the editor, that seems fine. --JoshTriplett