This page is a reference for ProgClub system administrators. For information about ProgClub domains, see [[Domains]]. For information about member services, see [[Services]]. See [[Machines]] for information about hosts on the ProgClub network. See [[Projects]] for current projects or check out our [[Forums]] to get in touch.

This page is a reference for ProgClub system administrators. For information about ProgClub domains, see [[Domains]]. For information about member services, see [[Services]]. See [[Machines]] for information about hosts on the ProgClub network. See [[Projects]] for current projects or check out our [[Forums]] to get in touch.

= Administrative reference =

= Administrative reference =

+

+

The migration to [[Network v2]] is now in full swing!

If you're administering ProgClub assets, please document your actions on the wiki. See the relevant pages:

If you're administering ProgClub assets, please document your actions on the wiki. See the relevant pages:

Line 13:

Line 14:

See, or update, [[Network administration]] for work that needs to be done.

See, or update, [[Network administration]] for work that needs to be done.

+

+

== Denying hosts with UFW ==

+

+

So the new systems [[integrity]] and [[strength]] have been deployed as part of the [[Network v2]] migration. They are mostly managed via AWS and Salt. If you're watching the web-logs and see a brute force attempt against WordPress etc. then you can issue a "temporary" ban for the offending IP address (it's "temporary" because you should remove it after a day or two once the problem has gone away). The actual firewall for our services is with AWS and its security groups, so we leverage this fact to simplify our UFW rules, for example, by using a default allow incoming policy.

+

+

root@integrity:/# ufw default allow incoming

+

root@integrity:/# ufw default allow outgoing

+

root@integrity:/# ufw reset

+

root@integrity:/# ufw default allow incoming

+

root@integrity:/# ufw default allow outgoing

+

root@integrity:/# ufw deny from 127.206.172.17

+

root@integrity:/# ufw status

+

root@integrity:/# ufw enable

+

root@integrity:/# ufw status

+

+

== Deleting ufw rules ==

+

+

# ufw status numbered

+

# ufw delete 123

+

+

== Wiki maintenance ==

+

+

If you need to update the wiki sidebar see [[MediaWiki:Sidebar]].

== Code of conduct ==

== Code of conduct ==

Line 101:

Line 125:

crontab

crontab

If you see no output, there are no uncommited changes.

If you see no output, there are no uncommited changes.

+

+

== Mailman ==

+

+

ProgClub uses the Mailman software to manage its mailing lists. From time to time some spam makes it through the mail filters onto the list. Such spam then really needs to be removed otherwise it will become a part of the HTML list archives published on our site.

+

+

To remove spam from a mailing list, ssh to [[charity]] and then:

+

+

$ sudo -s

+

# cd /var/lib/mailman/archives/private

+

# vim list.mbox/list.mbox

+

+

Then from vim delete the offending messages. When you're done:

+

+

# cd ../..

+

# bin/arch --wipe list

+

# chown -R list:www-data archives/private/list

+

+

And then you're back in business.

== Setting up an Ubuntu server ==

== Setting up an Ubuntu server ==

When configuring a new server consider the following checklist:

When configuring a new server consider the following checklist:

+

+

=== SSH in as root ===

+

+

Your virtual server will be configured as an SSH system accessible by root. SSH to your new system as root and then change the password:

While in vim change the 127.0.0.1 address to the machine name, not 'hostname.localname' as in the template. Rather use something like 'charity.localdomain'. You can run hostname -f again to check that everything worked.

−

honesty.progclub.net

−

=== Fail2ban ===

+

cp /etc/hosts.master /etc/hosts

+

exit

+

hostname -f

−

# apt-get install fail2ban

+

E.g.

−

# vim /etc/fail2ban/jail.local

+

charity.progclub.org

−

Something like this:

+

=== Install Fail2ban ===

+

sudo apt-get install fail2ban

+

+

sudo -s

+

cat > /etc/fail2ban/jail.local <<EOF

[DEFAULT]

[DEFAULT]

ignoreip = 127.0.0.1

ignoreip = 127.0.0.1

Line 374:

Line 897:

enabled = true

enabled = true

[pam-generic]

[pam-generic]

−

enabled = false

+

enabled = true

[xinetd-fail]

[xinetd-fail]

enabled = false

enabled = false

Line 405:

Line 928:

[named-refused-tcp]

[named-refused-tcp]

enabled = false

enabled = false

+

EOF

+

exit

+

+

Then open in vim to adjust email address or other settings if necessary:

Or to disable entirely ([https://en.wikipedia.org/wiki/Swappiness apparently not optimal]):

+

+

vm.swappiness=0

+

+

=== Setup complete: reboot ===

+

+

And now you're done. Just reboot the box for good measure.

+

+

sudo reboot && exit

+

+

If you like now you can [[Admin_reference#Setting_up_an_Ubuntu_server|setup another server]].

== Allocating user IDs ==

== Allocating user IDs ==

Line 580:

Line 1,138:

== Adding a new user ==

== Adding a new user ==

+

+

=== Generating passwords ===

+

+

While setting up a new user account you will need to generate passwords for the user. To generate passwords use the [https://www.unconfusable.com/password-generator/ Unconfusable Password Generator].

=== Updating members database ===

=== Updating members database ===

−

At the moment member information is recorded in [[Members]]. A member should have a Wiki account to start with (create one for them if they haven't already created their own), then an entry in the [[Members]] page (where they are allocated a member number), then the "User info" on their user page should be filled out (leave fields as "Not disclosed" unless you have their permission to publish their details). After the [[Members]] page has been updated you will have:

+

At the moment member information is recorded in [[Members]]. A member should have a Wiki account to start with ([https://www.progclub.org/pcwiki/index.php?title=Special:UserLogin&type=signup create one] for them if they haven't already created their own), then an entry in the [[Members]] page (where they are allocated a member number), then the "User info" on their user page should be filled out (leave fields as "Not disclosed" unless you have their permission to publish their details). After the [[Members]] page has been updated you will have:

Latest revision as of 04:31, 1 March 2019

This page is a reference for ProgClub system administrators. For information about ProgClub domains, see Domains. For information about member services, see Services. See Machines for information about hosts on the ProgClub network. See Projects for current projects or check out our Forums to get in touch.

Denying hosts with UFW

So the new systems integrity and strength have been deployed as part of the Network v2 migration. They are mostly managed via AWS and Salt. If you're watching the web-logs and see a brute force attempt against WordPress etc. then you can issue a "temporary" ban for the offending IP address (it's "temporary" because you should remove it after a day or two once the problem has gone away). The actual firewall for our services is with AWS and its security groups, so we leverage this fact to simplify our UFW rules, for example, by using a default allow incoming policy.

Deleting ufw rules

Wiki maintenance

Code of conduct

As a ProgClub administrator you have a lot of power. You have the capability to destroy ProgClub's files and configuration, to access all of ProgClub's databases, to pretend to be other users, and to access other users' private data. We expect you won't abuse your power. Specifically, under no circumstances should you:

Copy data from any of ProgClub's administrative databases into your own system

Read or copy any data from members' databases that are not yours

Pretend to be another member

Publicly disclose member details, such as name or email address, without their permission

All of your systems administration activity should be documented, and you shouldn't be doing anything evil. If you're not sure what qualifies as evil, if you have any doubt at all, please ask.

Although ProgClub doesn't make any guarantees, members should be able to feel as though their privacy is respected at ProgClub, and they should be able to feel confident that administrators aren't spying on them by reading their email or their other data.

Where are the keys?

Your administrator login on charity and your member account in LDAP will give you sudo privs on all of the ProgClub machines, both administrative and user machines. Some tools require special purpose logins, and for those you will find login details in the /home/jj5/login_* files which you can access from any of the machines.

Administrative and user machines

As explained on the Machines page ProgClub separates its machines into two groups: administrative machines and user machines. There is only one administrative machine, charity. There are two user machines, hope and honesty. If you're doing sysadmin work on the user machines, make sure you duplicate your work and create an identical system configuration on each machine. Both user machines should have the same configuration. And, yes, this means that you have to duplicate your documentation too. If you're doing some experimental configuration I'd suggest doing it on honesty first, and then once you've got everything figured out and stable duplicating onto hope. You can use hope first though if it suits you. Try not to let too much time pass with the systems in different configurations, generally you should do your sysadmin work on both systems immediately one after the other.

Administrator logins

The way logins work for administrators (as opposed to normal users) is that you have two logins. One login is for the administrative server (currently there is only one of these, charity), and the other login is your Kerberos/LDAP login that gets you access to the user machines. You can use different passwords if you want, but the usernames (and UIDs) will be the same. All administrators are members of the 'sudo' group on charity and in LDAP. This means that you can use your sudo privileges on either administrative machines or user machines.

Etckeeper

after you're done with your changes. There's an auto commit every day, and an autocommit whenever you apt-get install something. You can manually commit your changes as above. To see the commit log for a particular file:

$ sudo bzr log /etc/passwd

To revert an unwanted or bad change, work out which revision you want to revert to (see log above) and run:

$ sudo bzr revert --revision <commit number> <file>

For example, to restore the /etc/passwd file to the state it was in in revision 3,

$ sudo bzr revert --revision 3 /etc/passwd

To check for uncommited changes, run (From inside /etc):

$ sudo bzr status

For example:

$ sudo bzr status
modified:
crontab

If you see no output, there are no uncommited changes.

Mailman

ProgClub uses the Mailman software to manage its mailing lists. From time to time some spam makes it through the mail filters onto the list. Such spam then really needs to be removed otherwise it will become a part of the HTML list archives published on our site.

While in vim change the 127.0.0.1 address to the machine name, not 'hostname.localname' as in the template. Rather use something like 'charity.localdomain'. You can run hostname -f again to check that everything worked.

Then open in vim to adjust email address or other settings if necessary:

sudo vim /etc/fail2ban/jail.local

Configure mail and forwarding

sudo apt-get install bsd-mailx postfix

Package configuration
┌────────────────────────┤ Postfix Configuration ├────────────────────────┐
│ │
│ Please select the mail server configuration type that best meets your
│ needs.
│ ▒
│ No configuration: ▒
│ Should be chosen to leave the current configuration unchanged. ▒
│ Internet site: ▒
│ Mail is sent and received directly using SMTP. ▒
│ Internet with smarthost: ▒
│ Mail is received directly using SMTP or by running a utility such ▒
│ as fetchmail. Outgoing mail is sent using a smarthost. ▒
│ Satellite system: ▒
│ All mail is sent to another machine, called a 'smarthost', for ▒
│ delivery. ▒
│ Local only:
│ The only delivered mail is the mail for local users. There is no
│ network.
│
│
│ <Ok>
│ │
└─────────────────────────────────────────────────────────────────────────┘

Package configuration
┌─────────────────────────┤ Postfix Configuration ├─────────────────────────┐
│ The "mail name" is the domain name used to "qualify" _ALL_ mail │
│ addresses without a domain name. This includes mail to and from <root>: │
│ please do not make your machine send out mail from root@example.org │
│ unless root@example.org has told you to. │
│ │
│ This name will also be used by other programs. It should be the single, │
│ fully qualified domain name (FQDN). │
│ │
│ Thus, if a mail address on the local host is foo@example.org, the │
│ correct value for this option would be example.org. │
│ │
│ System mail name: │
│ │
│ example.progclub.net_____________________________________________________ │
│ │
│ <Ok> <Cancel> │
│ │
└───────────────────────────────────────────────────────────────────────────┘

Setup complete: reboot

Allocating user IDs

If you need to allocate a user id for a member, use their member number from the Members page. Member numbers start at 1000 and increment from there.

If you need to allocate a user id for a system user, such as a user that a process will run as, check for UIDs starting from 500 and use the next available one. If you create a user on one machine with a particular UID you should create the same user with the same UID on all the other machines too.

Adding a new user

Generating passwords

Updating members database

At the moment member information is recorded in Members. A member should have a Wiki account to start with (create one for them if they haven't already created their own), then an entry in the Members page (where they are allocated a member number), then the "User info" on their user page should be filled out (leave fields as "Not disclosed" unless you have their permission to publish their details). After the Members page has been updated you will have:

{username}: the Unix/Kerberos username of the new member

{member number}: the member's User ID

{group}: whether the user has 'sudo' membership or not

there are two groups at ProgClub: 'sudo' and 'user'

everyone is in 'user' (gidNumber == 500)

administrators are in 'sudo' (gidNumber == 27)

And that's enough information to create their account as detailed below.

Kerberos administration

To create a new user in Kerberos:

SSH to charity.progclub.org

Login

Run kadmin addprinc:

$ sudo kadmin -p {your username} -q "addprinc {username}"

Note: use sudo so that the log file can be written

{your username} is your ProgClub username

{username} is the ProgClub username of the user your are adding

If the user is an administrator, add them to /etc/krb5kdc/kadm5.acl. E.g., at the end of the file,

username@PROGCLUB.ORG *

replacing 'username' as appropriate. You need to restart the Kerberos administration server after this change: