Author: TobyBartelsFormat: MarkdownI just wrote an addition to [[regular space]], and I can't even tell if it got saved properly. There's no response from the Lab (just a long wait, no error message, until it times out), even though I've restarted it twice.
Incidentally, it's been at least a week since I've been able to get more than two characters into the terminal at a time. Even to restart it with `~/x`, I have to log in twice!

I just wrote an addition to regular space, and I can't even tell if it got saved properly. There's no response from the Lab (just a long wait, no error message, until it times out), even though I've restarted it twice.

Incidentally, it's been at least a week since I've been able to get more than two characters into the terminal at a time. Even to restart it with ~/x, I have to log in twice!

Author: TobyBartelsFormat: MarkdownFrom `wget`, I get ‘ERROR 504: Gateway Timeout.’.
`/etc/init.d/instiki status` says ‘instiki dead but subsys locked’; anything besides `status` gives an error that `/usr/local/instiki/tmp/pids/server.pid` doesn't exist or that `instiki` is already running. But `ps -Al` gives nothing that looks relevant (except for `lighttpd`).
Also, apparently the reason why I could never get more than two characters in is that typing `~` will always freeze it! Otherwise, the `ssh` terminal is doing just fine. So `cd; ./x` is probably best from now on.

Also, apparently the reason why I could never get more than two characters in is that typing ~ will always freeze it! Otherwise, the ssh terminal is doing just fine. So cd; ./x is probably best from now on.

Author: Andrew StaceyFormat: MarkdownWhen loaded from /etc/init.d/instiki then it creates a lockfile. The lockfile
isn't created by instiki, but by the script in /etc/init.d. Thus if instiki
crashes, the lockfile isn't removed and the script won't restart. So it's
necessary to remove the lockfile before relaunching it.
rm /var/lock/subsys/instiki
But one should check that instiki isn't running before removing the lockfile
(via ps auxc).

When loaded from /etc/init.d/instiki then it creates a lockfile. The lockfile
isn't created by instiki, but by the script in /etc/init.d. Thus if instiki
crashes, the lockfile isn't removed and the script won't restart. So it's
necessary to remove the lockfile before relaunching it.

rm /var/lock/subsys/instiki

But one should check that instiki isn't running before removing the lockfile
(via ps auxc).

Author: Andrew StaceyFormat: MarkdownOh, and some implementations of ssh use ~ as an escape character. To quote:
> -e escape_char
> > Sets the escape character for sessions with a pty (default: ‘~’).
The escape character is only recognized at the beginning of a
line. The escape character followed by a dot (‘.’) closes the
connection; followed by control-Z suspends the connection; and
followed by itself sends the escape character once. Setting the
character to “none” disables any escapes and makes the session
fully transparent.
So you could avoid the problem by doing `ssh -e '#'` or something like that.

Oh, and some implementations of ssh use ~ as an escape character. To quote:

-e escape_char

Sets the escape character for sessions with a pty (default: ‘~’).
The escape character is only recognized at the beginning of a
line. The escape character followed by a dot (‘.’) closes the
connection; followed by control-Z suspends the connection; and
followed by itself sends the escape character once. Setting the
character to “none” disables any escapes and makes the session
fully transparent.

So you could avoid the problem by doing ssh -e '#' or something like that.

Author: TobyBartelsFormat: MarkdownOr just type `~` twice. Jeez, why didn't I ever try that? Even accidentally!
Thanks for `/var/lock/subsys/instiki`; I knew that there was a lock file somewhere, but I never knew where it was.

Author: UrsFormat: MarkdownWe need some method to archive important results of discussions here. In five month the nLab may halt while I am working on it and I will have to try to remember where it was that I saw you two chat about lockfiles.
A good idea might be to split off from the nLab's HowTo page an AdminHowTo page that lists answers to issues like these.

We need some method to archive important results of discussions here. In five month the nLab may halt while I am working on it and I will have to try to remember where it was that I saw you two chat about lockfiles.

A good idea might be to split off from the nLab's HowTo page an AdminHowTo page that lists answers to issues like these.

Author: Andrew StaceyFormat: MarkdownIt's a moot point, but I would put _this_ sort of stuff on the 'nlabmeta' web. Most people don't need to know about this stuff, and most people couldn't do anything about it even if they did know it.
But you're right. We do need to archive some of the stuff here. Some of it does get done (like how to do redirects) but it's not as simple as just copying the solution to a problem across from here to there. For example, take the 'downloading the n-lab' thread. There's a useful script there that if someone wants to download the n-lab then they should know. However, if someone hasn't yet had the inclination to download the entire n-lab then I'd rather not put the idea into their heads as if _everyone_ does it then there goes our bandwidth!
We probably have three categories of information:
1. Stuff everyone might find useful. Such as how to do redirects, or include SVGs.
2. Stuff that anyone could find useful, but isn't necessary for ordinary use of the n-lab. Such as downloading the whole lot.
3. Stuff that only the lab elves need to know. Such as how to reboot the server.
I'd recommend: n-lab HowTo for the first, n-labmeta HowTo for the second, and a lab elves technical page for the third.
I'm not saying that any information should be hidden or deliberately obscured, but that it is layered in such a fashion that the most likely and useful information is encountered first.

It's a moot point, but I would put this sort of stuff on the 'nlabmeta' web. Most people don't need to know about this stuff, and most people couldn't do anything about it even if they did know it.

But you're right. We do need to archive some of the stuff here. Some of it does get done (like how to do redirects) but it's not as simple as just copying the solution to a problem across from here to there. For example, take the 'downloading the n-lab' thread. There's a useful script there that if someone wants to download the n-lab then they should know. However, if someone hasn't yet had the inclination to download the entire n-lab then I'd rather not put the idea into their heads as if everyone does it then there goes our bandwidth!

We probably have three categories of information:

Stuff everyone might find useful. Such as how to do redirects, or include SVGs.

Stuff that anyone could find useful, but isn't necessary for ordinary use of the n-lab. Such as downloading the whole lot.

Stuff that only the lab elves need to know. Such as how to reboot the server.

I'd recommend: n-lab HowTo for the first, n-labmeta HowTo for the second, and a lab elves technical page for the third.

I'm not saying that any information should be hidden or deliberately obscured, but that it is layered in such a fashion that the most likely and useful information is encountered first.

Author: UrsFormat: MarkdownItexI have just restarted it once more.
I noticed the last two days, the nLab had a multitude of down-times of a few minutes, from which it did recover -- I believe these revoverys are due to Andrew's recent modification which makes the instiki software restart on a regular basis of minutes or something like this
So I am wondering about two things:
* even if it does recover, why is it down so many times per day?
* why does the automatic recovery still fail some time?

I have just restarted it once more.

I noticed the last two days, the nLab had a multitude of down-times of a few minutes, from which it did recover – I believe these revoverys are due to Andrew’s recent modification which makes the instiki software restart on a regular basis of minutes or something like this

Author: DavidRobertsFormat: MarkdownItexYeah, I was just about to say. Someone should host Jim's recently uploaded article in a stable place for now and change the link at the cafe, as it reflects badly on the cafe not being able to satisfy links.

Yeah, I was just about to say. Someone should host Jim’s recently uploaded article in a stable place for now and change the link at the cafe, as it reflects badly on the cafe not being able to satisfy links.

Author: UrsFormat: MarkdownItex> it reflects badly on the cafe not being able to satisfy links.
I find that the smallest problem. What worries me is the Lab being down. And what really worries me is: the $n$Journal-to-be not being up.
I wish I knew what we could do.

it reflects badly on the cafe not being able to satisfy links.

I find that the smallest problem. What worries me is the Lab being down. And what really worries me is: the nnJournal-to-be not being up.

Author: Andrew StaceyFormat: MarkdownItexWe've been getting some memory spikes recently, with the instiki processes going up to about 3 times their usual level. When I've spotted them, I've done a "soft reset" which has worked fine. When I get a bit of time, I'll track down what's causing them and report back to Jacques.

We’ve been getting some memory spikes recently, with the instiki processes going up to about 3 times their usual level. When I’ve spotted them, I’ve done a “soft reset” which has worked fine. When I get a bit of time, I’ll track down what’s causing them and report back to Jacques.

Author: UrsFormat: MarkdownItexIt might be a good idea if you (Tim) for instance also got access to the $n$Lab server. Since we are back to the point where it needs restarting once of twice a day, it would be good to have more people be able to do so. If you are willing to help with this, I suppose Andrew would be glad to create an account for you. (You need a public key, though. In case you are not familiar with the trouble one has to go through for this, I can send you a step-by-step list for what to do. )

It might be a good idea if you (Tim) for instance also got access to the nnLab server. Since we are back to the point where it needs restarting once of twice a day, it would be good to have more people be able to do so. If you are willing to help with this, I suppose Andrew would be glad to create an account for you. (You need a public key, though. In case you are not familiar with the trouble one has to go through for this, I can send you a step-by-step list for what to do. )

Author: Andrew StaceyFormat: MarkdownItexThe steering committee should decide on this, but certainly I have no problems with Tim having reboot access to the nLab.
What I'd really like is for someone who knows a thing or two about what _might_ be the problem to tell me what to look for!

The steering committee should decide on this, but certainly I have no problems with Tim having reboot access to the nLab.

What I’d really like is for someone who knows a thing or two about what might be the problem to tell me what to look for!

Author: UrsFormat: MarkdownItex> Is the n-lab set for automatic reboot still?
I think it is, but for some reason the automatic reboot doesn't always work. I suppose when it hangs it hangs so badly that it cannot do anything anymore, hence also not reboot itself.

Is the n-lab set for automatic reboot still?

I think it is, but for some reason the automatic reboot doesn’t always work. I suppose when it hangs it hangs so badly that it cannot do anything anymore, hence also not reboot itself.

Author: Andrew StaceyFormat: MarkdownItexActually, the automatic reboot stuff doesn't seem to work as I'd hoped it would. It seems that it goes in to the restart cycle, but the restart isn't in place by the time it next checks, so it assumes that the restart didn't wok and goes into a sulk. I need to investigate other systems.

Actually, the automatic reboot stuff doesn’t seem to work as I’d hoped it would. It seems that it goes in to the restart cycle, but the restart isn’t in place by the time it next checks, so it assumes that the restart didn’t wok and goes into a sulk. I need to investigate other systems.

Author: UrsFormat: MarkdownItexI have restarted it.
Tim, should we try to give you access to the nLab server? Would you be willing to? Given the number of your lab-down-reports, it would be really good for the $n$Lab community and for you, it seems, if you could restart the lab.

I have restarted it.

Tim, should we try to give you access to the nLab server? Would you be willing to? Given the number of your lab-down-reports, it would be really good for the nnLab community and for you, it seems, if you could restart the lab.

Author: UrsFormat: MarkdownItexI had restarted it, but before I saw your message here.
As long as myself I am working on the lab, I notice quickly when it goes down. Trouble begins when I am not myself working on it.

I had restarted it, but before I saw your message here.

As long as myself I am working on the lab, I notice quickly when it goes down. Trouble begins when I am not myself working on it.

Author: Tim_PorterFormat: MarkdownItexTopically as far as another thread goes, the Lab seems to be down. Although this does not apply this time, I seemed to notice that it crashes at weekends. Is this true? Is it internal to the software (e.g. a routine scheduled garbage collection or something like that) which overloads it, or is there some external source such as someone downloading new material to have a personal copy.

Topically as far as another thread goes, the Lab seems to be down. Although this does not apply this time, I seemed to notice that it crashes at weekends. Is this true? Is it internal to the software (e.g. a routine scheduled garbage collection or something like that) which overloads it, or is there some external source such as someone downloading new material to have a personal copy.

Author: UrsFormat: MarkdownItex> Topically as far as another thread goes, the Lab seems to be down.
I have now restarted it.
> Although this does not apply this time, I seemed to notice that it crashes at weekends. Is this true?
It crashes several times each day. But during the week I am usually online and notice it fairly quickly, and restart it without always dropping a note here. On weekends even I am offline for longer periods. I think that explains it.
But I will now try to record every single time that I restart the server, so that we get a better idea.

Topically as far as another thread goes, the Lab seems to be down.

I have now restarted it.

Although this does not apply this time, I seemed to notice that it crashes at weekends. Is this true?

It crashes several times each day. But during the week I am usually online and notice it fairly quickly, and restart it without always dropping a note here. On weekends even I am offline for longer periods. I think that explains it.

But I will now try to record every single time that I restart the server, so that we get a better idea.

Author: UrsFormat: MarkdownItexI have restarted the $n$Lab.
But then something curious happened: the pages that I was waiting for to display did display the instant that my finger touched the enter-key to send off the restart command.
Maybe a coincidence. But it did happen before to me. So I thought I'd mention it.

I have restarted the nnLab.

But then something curious happened: the pages that I was waiting for to display did display the instant that my finger touched the enter-key to send off the restart command.

Maybe a coincidence. But it did happen before to me. So I thought I’d mention it.

Author: zskodaFormat: MarkdownItexWell, I do not like common holidays for the plain reason that most of the things do not function, are closed or are forbidden to do during them. This time is the $n$Lab.

Well, I do not like common holidays for the plain reason that most of the things do not function, are closed or are forbidden to do during them. This time is the nnLab.

Author: Tim_PorterFormat: MarkdownItexSome of that may be due to lots of people downloading megalength films somewhere between you and the main 'superhighway'. (After one preview: That was not too bad.)

Some of that may be due to lots of people downloading megalength films somewhere between you and the main ’superhighway’. (After one preview: That was not too bad.)

Author: zskodaFormat: MarkdownItexNo there is some other problem. Somehow I can not get the double dollar sign work consistently. When it does not like it, like usual
<nowiki>$$ A $$</nowiki>
it runs it for minutes and returns at the end the page without he formula.
$$
A
$$
(Look at the source of this)

No there is some other problem. Somehow I can not get the double dollar sign work consistently. When it does not like it, like usual

it runs it for minutes and returns at the end the page without he formula.

Author: Andrew StaceyFormat: MarkdownItexWhen the lab is down, then the rendering of maths here won't work either. That's because the actual conversion takes place on the same server as the nlab due to the fact that it needs something beyond what I'm allowed to run here.
I've just restarted the lab. My apologies for the long delay in restarting.

When the lab is down, then the rendering of maths here won’t work either. That’s because the actual conversion takes place on the same server as the nlab due to the fact that it needs something beyond what I’m allowed to run here.

I’ve just restarted the lab. My apologies for the long delay in restarting.

Author: UrsFormat: MarkdownItexMaybe Tim's observation above was right after all: as far as I can see the lab was down consistently at the very end of the week, somewhere from Sunday to Monday during the last weeks, maybe longer.

Maybe Tim’s observation above was right after all: as far as I can see the lab was down consistently at the very end of the week, somewhere from Sunday to Monday during the last weeks, maybe longer.

Author: Andrew StaceyFormat: MarkdownItexWhat were the symptoms that time? I'm intrigued because I happened to be logged in as root when you restarted it and so when I noticed that you restarted it, I checked the logs and couldn't see any of the usual suspects.

What were the symptoms that time? I’m intrigued because I happened to be logged in as root when you restarted it and so when I noticed that you restarted it, I checked the logs and couldn’t see any of the usual suspects.

Author: UrsFormat: MarkdownItexIt's down again. I can't even restart it, because even the command line is not reacting. (Had this before throughout the mrning. It has been immensely slow ever since two hours ago or so).

It’s down again. I can’t even restart it, because even the command line is not reacting. (Had this before throughout the mrning. It has been immensely slow ever since two hours ago or so).

Author: UrsFormat: MarkdownItexNow I have restarted it again. I have received one response of one page, but am already waiting again for the second page.
By the way, it is this kind of very frustrating experience that made me ask last time: what is our perspective? Do we need to stick with this?

Now I have restarted it again. I have received one response of one page, but am already waiting again for the second page.

By the way, it is this kind of very frustrating experience that made me ask last time: what is our perspective? Do we need to stick with this?

Author: Andrew StaceyFormat: MarkdownItexIf the commandline is also slow then that's nothing to do with Instiki but is to do with the connection between your computer and the server.
In poking around just now I discovered that the daily updates weren't getting applied due to an issue with a dependency. I've just fixed that.

If the commandline is also slow then that’s nothing to do with Instiki but is to do with the connection between your computer and the server.

In poking around just now I discovered that the daily updates weren’t getting applied due to an issue with a dependency. I’ve just fixed that.

Author: UrsFormat: MarkdownItex> If the commandline is also slow then that's nothing to do with Instiki but is to do with the connection between your computer and the server.
Sure, but since at the same time my computer happily accesses all kinds of sites, it seems to indicate that the server that the $n$Lab is running on is busy with something else.. Might be a hint as to what the problem is, maybe, I thought.
By the way, I have just restarted once more. This time calling any page produced a Passenger error message saying that Ruby on Rails could not be started.

If the commandline is also slow then that’s nothing to do with Instiki but is to do with the connection between your computer and the server.

Sure, but since at the same time my computer happily accesses all kinds of sites, it seems to indicate that the server that the nnLab is running on is busy with something else.. Might be a hint as to what the problem is, maybe, I thought.

By the way, I have just restarted once more. This time calling any page produced a Passenger error message saying that Ruby on Rails could not be started.

Author: Andrew StaceyFormat: MarkdownItex> Sure, but since at the same time my computer happily accesses all kinds of sites, it seems to indicate that the server that the
nLab is running on is busy with something else.
Not necessarily. Remember that we're using a VPS on some system, so there are several links in the chain between your computer and the nLab server, any one of which might be causing the slowdown.

Sure, but since at the same time my computer happily accesses all kinds of sites, it seems to indicate that the server that the
nLab is running on is busy with something else.

Not necessarily. Remember that we’re using a VPS on some system, so there are several links in the chain between your computer and the nLab server, any one of which might be causing the slowdown.

Author: Tim_PorterFormat: MarkdownItexIt looks as if the Lab has been to a late night street party and has a hang over! It has not yet got up. Can someone turn on the alarm clock and wake it up. (In case non-Brits are unaware of the events in the UK yesterday, there was a royal wedding! As a further completely irrelevant fact, the newly weds live not far from me on Anglesey so there were `street parties'... in the streets! At least the weather was good.) End of irrelevant comments... can someone restart the Lab please.

It looks as if the Lab has been to a late night street party and has a hang over! It has not yet got up. Can someone turn on the alarm clock and wake it up. (In case non-Brits are unaware of the events in the UK yesterday, there was a royal wedding! As a further completely irrelevant fact, the newly weds live not far from me on Anglesey so there were ‘street parties’… in the streets! At least the weather was good.) End of irrelevant comments… can someone restart the Lab please.

Author: Andrew StaceyFormat: MarkdownItexSomething struck me this morning. One reason why it might go down so often on Sundays might be because that is the day when the system does a _full_ backup. When it does that, it _locks_ the database so that no more information can be written to it. Due to the size of the database, the time that it is locked might be significant. And if instiki tries to access it while it is locked, then that might cause Something Bad.
I'll investigate further.

Something struck me this morning. One reason why it might go down so often on Sundays might be because that is the day when the system does a full backup. When it does that, it locks the database so that no more information can be written to it. Due to the size of the database, the time that it is locked might be significant. And if instiki tries to access it while it is locked, then that might cause Something Bad.