The master plan to conquer the world

The goal of GigaFox is to deploy a global infrastructure involving multiple resources spread in different regions and connected to each other. Their fittings plan is the biggest of all.

Fittings plan

Copy the text below and put it in a text file named fittings.yaml:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
 ---

 actions:
   - ansible:
       output: gigafox_ansible.yaml
   - inventory:
       output: gigafox_inventory.yaml
   - rub:
       key: ~/.ssh/id_rsa.pub
       output: gigafox_rubs.yaml

 # each facility is described separately
 ---
 # Santa Clara - secondary site in USA
 locationId: NA12
 regionId: dd-na

 rub:
   - beachhead: 10.10.10.9
   - beachhead: 10.10.10.10

 basement: beachhead

 blueprints:

   # root resources
   - beachhead:
       domain: &domain
         name: Gigafox
         description: '#us #primary'
         service: advanced
         ipv4: 8
       ethernet: &control
         name: gigafox.control
         description: '#us #ops'
         subnet: 10.0.0.0
         destroy: never
       nodes:
         - beachhead:
             description: '#beachhead #us #ops'
             glue:
               - internet 22
             running: always
             monitoring: essentials
             cloud-config:
               disable_root: false
               ssh_pwauth: True
               packages:
                 - python-pip
                 - python-dev
                 - git
               runcmd:
                 - pip install -e git+https://git-wip-us.apache.org/repos/asf/libcloud.git@trunk#egg=apache-libcloud
                 - pip install -e git+https://github.com/bernard357/plumbery.git#egg=plumbery

   # management, administration, and monitoring
   - control:
       domain: *domain
       ethernet: *control
       nodes:
         - stackstorm:
             description: '#stackstorm #us #ops'
             glue:
               - internet 22 80 443
             running: always
             monitoring: essentials
             cloud-config:
               disable_root: false
               ssh_pwauth: True
               runcmd:
                 - curl -sSL https://raw.githubusercontent.com/DimensionDataCBUSydney/st2_dimensiondata/master/install-eu.sh | sh
         - scom:
             description: 'Microsoft System Center Operation Manager #scom #us #ops'
             appliance: 'Win2012 R2 Std 4 CPU'
             monitoring: essentials

   # workloads dedicated to source code and related
   - source:
       domain: *domain
       ethernet:
         name: gigafox.source
         description: '#us #dev'
         subnet: 10.1.0.0
         destroy: never
       nodes:
         - gitlab:
             description: '#gitlab #us #dev'
             glue:
               - internet 80 443
               - gigafox.control
             monitoring: essentials

   # workloads dealing directly with end-user devices
   - web:
       domain: *domain
       ethernet: &bastion
         name: gigafox.web
         description: '#us'
         subnet: 10.2.0.0
         accept:
           - gigafox.control
         destroy: never
       nodes:
         - web[1..2]_na12:
             description: '#apache #us #primary'
             cpu: 4 2 highperformance
             memory: 8
             disks:
               - 1 500 standard
               - 2 100
             glue:
               - internet 22 80 443
             monitoring: essentials
             cloud-config:
               disable_root: false
               ssh_pwauth: True
               packages:
                 - apache2
                 - mysql-server
                 - libapache2-mod-php5
                 - php5-mysql
               runcmd:
                 - "rm /var/www/index.html"
               write_files:
                 - content: |
                         echo '<?php phpinfo();' >
                   path: /var/www/index.php
       listeners:
         - http:
             port: 80
             protocol: http
             algorithm: round_robin
         - https:
             port: 443
             protocol: http
             algorithm: round_robin

   - wordpress:
       domain: *domain
       ethernet: *bastion
       nodes:
         - wordpress_na12:
             description: '#wordpress #us #primary'
             cpu: 4 2 highperformance
             memory: 8
             glue:
               - internet 22 80 443
             monitoring: essentials
             cloud-config:
               disable_root: false
               ssh_pwauth: True
               packages:
                 - apache2
                 - php5
                 - php5-mysql
                 - mysql-server
               runcmd:
                 - wget http://wordpress.org/latest.tar.gz -P /tmp/
                 - tar -zxf /tmp/latest.tar.gz -C /var/www/
                 - mysql -e "create database wordpress; create user 'wpuser'@'localhost' identified by 'changemetoo'; grant all privileges on wordpress . \* to 'wpuser'@'localhost'; flush privileges;"
                 - mysql -e "drop database test; drop user 'test'@'localhost'; flush privileges;"
                 - mysqladmin -u root password 'changeme'

   # workloads dealing with short-term memory
   - memcache:
       domain: *domain
       ethernet: *bastion
       nodes:
         - redis[1..2]_na12:
             description: '#redis #us #primary'
             memory: 32
             monitoring: essentials

   # docker resources
   - docker:
       domain: *domain
       ethernet: &compute
         name: gigafox.compute
         description: '#us'
         subnet: 10.3.0.0
         accept:
           - gigafox.control
           - gigafox.web
         destroy: never
       nodes:
         - docker[1..5]_na12:
             description: '#docker #us #primary'
             cpu: 32
             memory: 128
             monitoring: essentials
             rub:
               - run rub.update.sh
               - run rub.docker.sh

   # nodejs resources
   - nodejs:
       domain: *domain
       ethernet: *compute
       nodes:
         - nodejs[1..5]_na12:
             description: '#nodejs #us #primary'
             monitoring: essentials
             rub:
               - run rub.update.sh

   # workloads dedicated to data records
   - sql:
       domain: *domain
       ethernet: &data
         name: gigafox.data
         description: '#us'
         subnet: 10.4.0.0
         accept:
           - gigafox.control
           - gigafox.compute
         destroy: never
       nodes:
         - masterSQL_na12:
             description: '#mysql #us #primary'
             appliance: 'RedHat 6 64-bit 4 CPU'
             monitoring: advanced

   # workloads dedicated to big data
   - cassandra:
       domain: *domain
       ethernet: *data
       nodes:
         - cassandra[1..3]_na12:
             description: '#cassandra #us #primary'
             monitoring: essentials

   # workloads dedicated to BLOBs
   - mongodb:
       domain: *domain
       ethernet: *data
       nodes:
         - mongodb[1..7]_na12:
             description: '#mongodb #us #primary'
             monitoring: essentials

   # workloads devoted to object-based storage
   - s3:
       domain: *domain
       ethernet: *data
       nodes:
         - ceph[1..5]_na12:
             description: '#ceph #us #primary'
             monitoring: essentials

 ---
 # Ashburn - primary site in USA
 locationId: NA9
 regionId: dd-na

 blueprints:

   # workloads dealing directly with end-user devices
   - web:
       domain: &domain
         name: Gigafox
         description: '#us #secondary'
         service: advanced
         ipv4: 4
       ethernet: &bastion
         name: gigafox.web
         description: '#us'
         subnet: 10.2.0.0
         accept:
           - NA12::gigafox.control
         destroy: never
       nodes:
         - web[1..2]_na9:
             description: '#apache #us #secondary'
             cpu: 4
             memory: 8
             disks:
               - 1 500 standard
             monitoring: essentials
             rub:
               - put rub.puppet.apache.pp /root/apache.pp
               - run rub.puppet.apache.sh /root/apache.pp
       listeners:
         - http:
             port: 80
             protocol: http
             algorithm: round_robin
         - https:
             port: 443
             protocol: http
             algorithm: round_robin

   # workloads dealing with short-term memory
   - memcache:
       domain: *domain
       ethernet: *bastion
       nodes:
         - redis[1..2]_na9:
             description: '#redis #us #secondary'
             monitoring: essentials

   # docker resources
   - docker:
       domain: *domain
       ethernet: &compute
         name: gigafox.compute
         description: '#us'
         subnet: 10.3.0.0
         accept:
           - NA12::gigafox.control
         destroy: never
       nodes:
         - docker[1..5]_na9:
             description: '#docker #us #secondary'
             monitoring: essentials
             rub:
               - run rub.update.sh
               - run rub.docker.sh

   # nodejs resources
   - nodejs:
       domain: *domain
       ethernet: *compute
       nodes:
         - nodejs[1..5]_na9:
             description: '#nodejs #us #secondary'
             monitoring: essentials
             rub:
               - run rub.update.sh

   # workloads dedicated to data records
   - sql:
       domain: *domain
       ethernet: &data
         name: gigafox.data
         description: '#us'
         subnet: 10.4.0.0
         accept:
           - NA12::gigafox.control
         destroy: never
       nodes:
         - slaveSQL_na9:
             description: '#mysql #us #secondary'
             appliance: 'RedHat 6 64-bit 4 CPU'
             monitoring: advanced

   # workloads dedicated to big data
   - cassandra:
       domain: *domain
       ethernet: *data
       nodes:
         - cassandra[1..3]_na9:
             description: '#cassandra #us #secondary'
             monitoring: essentials

   # workloads dedicated to BLOBs
   - mongodb:
       domain: *domain
       ethernet: *data
       nodes:
         - mongodb[1..7]_na9:
             description: '#mongodb #us #secondary'
             monitoring: essentials

   # workloads devoted to object-based storage
   - s3:
       domain: *domain
       ethernet: *data
       nodes:
         - ceph[1..5]_na9:
             description: '#ceph #us #secondary'
             monitoring: essentials

Deployment commands

$ python -m plumbery fittings.yaml build
$ python -m plumbery fittings.yaml start
$ python -m plumbery fittings.yaml prepare

These commands will build fittings as per the provided plan, and start the server as well. Look at messages displayed by plumbery while it is working, so you can monitor what’s happening.

Destruction commands

Cloud computing has a hard rule. Any resource has a cost, be it used or not. At the end of every session, you are encouraged to destroy everything. Hopefully, plumbery is making this really simple:

$ python -m plumbery fittings.yaml stop
$ python -m plumbery fittings.yaml destroy