Ever since I’ve started hosting my own domains, I’ve searched for ways to automate the website provisioning.

Normally, the documentation includes the information on configuring your infrastructure based on roles. But if you have a webserver role for all your minions it doesn’t help with the individual provisioning I am aiming at.

Planing Phase

The first rule: plan for what you need with what you have.

For this example I’ll be using the following set-up:

e-tel.eu hosted on srv01.so.ai with SSL and HSTS enabled

www.bitleader.com hosted on srv01.so.ai with SSL enabled

bitleader.one hosted on srv02.so.ai without SSL

so.ai hosted on srv03.so.ai without SSL

all the hostnames are unique (not the fqdns)

all the SSL key files are found in /etc/ssl/salt-managed/keys/website.key

all the SSL certificate files are found in /etc/ssl/salt-managed/certs/website.crt

the SSL chain file is located in /etc/ssl/salt-managed/bundle.crt

all the servers run either Ubuntu 14.04 or Ubuntu 16.04

Data Required

The main idea is to split the provisioning data in two main categories:

Identical across servers and websites

Website and/or server specific

The website and server specific data will be stored in pillars, the rest directly in the state files (in the sls files).

Salt-Master Configuration

pillar_source_merging_strategy

To be able to overwrite part of the pillars with data from other pillar files, you need to set it on recurse. See this link on information about it).

file_roots

Make sure you include a filebase folder. If you don’t, you’ll need to change the examples accordingly.

/etc/salt/master.d/local.conf

YAML

1

2

3

4

5

6

7

8

9

10

11

12

pillar_source_merging_strategy: recurse

file_roots:

base:

-/srv/salt

-/srv/salt/filebase

fileserver_backend:

-roots

pillar_roots:

base:

-/srv/pillars

Pillars

pillars/top.sls

In your pillars/top.sls you should have:

pillars/top.sls

YAML

1

2

3

base:

'*':

-apache

pillars/apache/init.sls

In the folder pillars/apache you create the file init.sls:

Shell

1

mkdir-ppillars/apache&&touchpillars/apache/init.sls

In order for you to be able to load pillars with different content for the different hosts, you need to specify this in the pillars/apache/init.sls file. The common setting are the only ones that will be directly in this file.

pillars/apache/init.sls

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

{%set localhost=salt['grains.get']('host')%}

include:

-apache.{{localhost}}

apache:

default_host_root:/var/www

default_host_log:/var/log/apache2

default_host_enable_ssl:True

default_virtual_base:/www# the website name will be appended to this

default_virtual_log:logs# the folder will be in form /www/f.q.d.n/logs

default_virtual_root:htdocs# the folder will be in form /www/f.q.d.n/logs

default_virtual_tmp:tmp# the folder will be in form /www/f.q.d.n/tmp

document_root_options:'-Indexes +FollowSymLinks -MultiViews'

server_admin:root@localdomain

server_signature:'off'

server_tokens:'Prod'

ssl_enable_bundle:True# If True it will manage the /etc/ssl/bundle.crt file

force_ssl will be used to redirect HTTP traffic to HTTPS (redirect 301)

hsts will be used to enable HSTS for the specified hsts_max_age

Note! You can overwrite all the settings from init.sls in the host settings file (for example, hsts_max_age).

pillars/apache/srv01.sls

YAML

1

2

3

4

5

6

7

8

9

10

11

12

13

apache:

hsts_max_age: 15768000 # about half a year

websites:

- name: e-tel.eu

aliases:

-www.e-tel.eu

ssl: True

force_ssl: True

hsts: True

- name: www.bitleader.com

aliases:

-bitleader.com

ssl: True

pillars/apache/srv02.sls

YAML

1

2

3

4

5

6

apache:

default_virtual_base: '/sites'

websites:

- name: bitleader.one

aliases:

-www.bitleader.one

pillars/apache/srv03.sls

YAML

1

2

3

4

5

apache:

websites:

- name: so.ai

aliases:

-www.so.ai

States

You need to create the folder and all the files. For a better structure, the following are split into separate state files: package dependencies, states for the default website and states for the other websites.

Shell

1

2

3

4

mkdirsalt/apache&&

foriinfolders files packages init;do

touchsalt/apache/${i}.sls;

done

salt/top.sls

You include the Apache state in the salt/top.sls file:

An error has occurred. Please try again later.

salt/apache/init.sls

The folder, package and file states will be included in this file:

salt/apache/init.sls

YAML

1

2

3

4

include:

-apache.packages

-apache.default

-apache.websites

salt/apache/packages.sls

All the package dependencies, as well as the Apache service, are monitored here. Depending on the Apache modules you need, this is also where you can enable or disable them.

Note! You’ll need to change the package names and add a service name if you plan on using RH/CentOS!

Note! If you’re using salt < 2016.3 you need to replace apache_module.enabled with apache_module.enable and apache_module.disabled with apache_module.disable

salt/apache/packages.sls

YAML

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

apache2:

pkg.installed:

- pkgs:

-apache2

-apache2-utils

service:

-running

-watch.pkg:

- pkgs:

-apache2

- require:

- pkg: apache2

mod_rewrite:

apache_module.enabled:

- name: rewrite

- require:

- pkg: apache2

- listen_in:

- service: apache2

mpm_event:

apache_module.disabled:

- name: mpm_event

- require:

- pkg: apache2

- listen_in:

- service: apache2

mpm_worker:

apache_module.disabled:

- name: mpm_worker

- require:

- pkg: apache2

- listen_in:

- service: apache2

mpm_prefork:

apache_module.enabled:

- name: mpm_prefork

- require:

- pkg: apache2

- apache_module: mpm_event

- apache_module: mpm_worker

- listen_in:

- service: apache2

The Default Website

The last state file is salt/apache/default.sls. But before we start with it, some more conventions:

all the configuration files, that will be deployed on the server, are located on the Salt-Master under /srv/salt/filebase/apache/.

all the SSL certificates are located on the Salt-Master under /srv/salt/filebase/apache/ssl/certs/*.crt

all the SSL keys are located on the Salt-Master under /srv/salt/filebase/apache/ssl/keys/*.key

the SSL certificates are named either fqdn.crt for the default server certificate, or website.crt for the hosted virtual servers

the same applies for SSL keys, their form will be fqdn.key and website.key

Note! This guide doesn’t cover generating your certificate. Information on how to do that can be found on DigitalOcean for example.

salt/apache/default.sls scale with your setup

For the actual Apache configuration files also wanted to automatically make the tuning of the prefork module. Wish I could link the source of the formulas, but alas, I have lost it.

salt/apache/default.sls

YAML

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

{%setfqdn=salt['grains.get']('fqdn')%}# the servers fqdn

{%setmem=salt['grains.get']('mem_total')%}# the servers total memory

{%setsettings=salt['pillar.get']('apache')%}# as defined in pillars/apache/

{%setram_base=(((mem/1000)|round)+1)|int%}

{%setmax_spare_servers=(ram_base+1)|int%}

{%setserver_limit=(50+((ram_base**2)*10)+((ram_base-2)*10))|int%}

{%setmax_requests_per_child=(2048+(ram_base*256))|int%}

include:

-apache.packages

{{settings.default_host_root}}:

file.directory:

- user: root

- group: root

- dir_mode: '0755'

- makedirs: True # Like running makedir -p

- require:

- pkg: apache2

{{settings.default_host_log}}:

file.directory:

- user: root

- group: adm

- dir_mode: '0750'

- makedirs: True # Like running makedir -p

- require:

- pkg: apache2

/etc/apache2/mods-available/mpm_prefork.conf:

file.managed:

- source: salt://filebase/apache/mpm_prefork.conf

- user: root

- group: root

- mode: 0644

- template: jinja

- listen_in:

- service: apache2

- defaults:

mem: {{mem}}

ram_base: {{ram_base}}

max_spare_servers: {{max_spare_servers}}

server_limit: {{server_limit}}

max_requests_per_child: {{max_requests_per_child}}

/etc/logrotate.d/apache2:

file.managed:

- source: salt://filebase/apache/logrotate # will generate the logrotate file also for the websites

- user: root

- group: root

- mode: '0644'

- require:

- pkg: apache2

- file: {{settings.default_host_log}}

- template: jinja

- defaults:

settings: {{settings}}

fqdn: {{settings}}

/etc/apache2/sites-available/000-default.conf:

file.managed:

- source: salt://filebase/apache/site_available_template.conf # will also be used in salt/apache/websites.sls

- mode: '0644'

- user: root

- group: root

- template: jinja

- listen_in:

- service: apache2

- require:

- pkg: apache2

- file: {{settings.default_host_root}}

- file: {{settings.default_host_log}}

- defaults:

fqdn: {{fqdn}}

settings: {{settings}}

website: False

/etc/ssl/salt-managed/certs:

file.directory:

- user: root

- group: root

- dir_mode: '0755'

- makedirs: True

/etc/ssl/salt-managed/keys:

file.directory:

- user: root

- group: root

- dir_mode: '0700'

- require:

- file: /etc/ssl/salt-managed/certs

{%ifsettings.ssl_enable_bundle%}

/etc/ssl/salt-managed/bundle.crt:

file.managed:

- source: salt://filebase/apache/ssl/bundle.crt

- require:

- file: /etc/ssl/salt-managed/certs

- listen_in:

- service: apache2

{%endif%}

{%ifsettings.default_host_enable_ssl%}

/etc/ssl/salt-managed/certs/{{fqdn}}.crt:

file.managed:

- source: salt://filebase/apache/ssl/certs/{{fqdn}}.crt

- user: root

- group: root

- mode: 0644

- listen_in:

- service: apache2

/etc/ssl/salt-managed/keys/{{fqdn}}.key:

file.managed:

- source: salt://filebase/apache/ssl/keys/{{fqdn}}.key

- user: root

- group: root

- mode: 0400

- listen_in:

- service: apache2

{%endif%}

salt/apache/websites.sls

All states related to a hosted website are in this file.

salt/apache/websites.sls

YAML

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

{%setsettings=salt['pillar.get']('apache')%}# as defined in pillars/apache/

include:

-apache.default

{%forwebsiteinsettings.websites%}

{%setbase=settings.default_virtual_base+'/'+website.name%}

{%setlog=base+'/'+settings.default_virtual_log%}

{%setroot=base+'/'+settings.default_virtual_root%}

{%settmp=base+'/'+settings.default_virtual_tmp%}

{{base}}:

file.directory:

- name: '{{base}}'

- user: root

- group: root

- mode: '0755'

- makedirs: True # Like running mkdir -p

{{ root }}:

file.directory:

- name: '{{root}}'

- user: www-data # you could have here a custom username or uid, stored under website.uid for example

- group: www-data

- dir_mode: '0770'

- recurse:

- user

- group

{{ tmp }}:

file.directory:

- name: '{{tmp}}'

- user: www-data

- group: www-data

- dir_mode: '0770'

- require:

- file: {{ root }}

- recurse:

- group

{{ log }}:

file.directory:

- name: '{{log}}'

- user: www-data # has to have with the same username as the web server

- group: adm

- dir_mode: '0770'

- require:

- file: {{ root }}

/etc/apache2/sites-available/{{ website.name }}:

file.managed:

- source: salt://filebase/apache/site_available_template.conf # will also be used in salt/apache/websites.sls