You’ll also need a Kubernetes cluster - to develop locally this means installing either minikube or docker (MacOS only at this stage). Potentially you could also do this with native Kubernetes, or with a cloud cluster, but we haven’t tried it.

Trying it out locally

If you just want to try it out locally without actually changing anything, it’s much easier to just install minikube or docker for desktop, then following the instructions at https://github.com/magda-io/magda-config. What follows is instructions on how to build everything, code, databases and all, from scratch into a working application.

Building and running (just) the frontend

If you just want to edit the UI, you don’t actually even need helm - just clone the repo, run yarn install at the root, then cd magda-web-client and run yarn run dev. This will build/run a local version of the client, connecting to the API at https://dev.magda.io/api. If you want to connect to a magda API hosted elsewhere you can modify the config.js file in the client.

Building and running the backend

First clone the magda directory and cd into it.

Then install npm dependencies and set up the links between components by running:

yarn install

Once the above prerequisites are in place, and the npm dependencies are installed, building MAGDA is easy. From the MAGDA root directory, simply run:

lerna run build --include-filtered-dependencies

You can also run the same command in an individual component’s directory (i.e. magda-whatever/) to build just that component.

Set up Helm

Helm is the package manager for Kubernetes - we use it to make it so that you can install all the various services you need for MAGDA at once. To install, follow the instructions at https://github.com/kubernetes/helm/blob/master/docs/install.md.

In a nutshell, once you have helm installed, this is how you initialise helm and Tiller.

Create the necessary secrets with the secret creation script

Install Magda on your minikube/docker-desktop cluster

This can take a while as it does a lot - downloading all the docker images, starting them up and running database migration jobs. You can see what’s happening by opening another tab and running kubectl get pods -w.

Also note that by default there won’t be any minions running, as some of them can be very CPU intensive. You can toggle them on by specifying --set tags.minion-<minionname>=true when you run helm upgrade.

Crawl Data

By default, helm will create a one-time crawl job for data.gov.au to get you started. If you want to crawl other datasets, look at the config under connectors: in deploy/helm/minikube-dev.yml. For sources of data, check out deploy/helm/magda-dev.yml. Once you’ve changed your config, just run the helm upgrade command above again to make it happen.

Kubernetes tricks

Running individual services

If you want to just start up individual pods (e.g. just the combined database) you can do so by setting the all tag to false and the tag for the pod you want to true, e.g.

Once everything starts up, you can access the web front end on http://192.168.99.100:30100. The IP address may be different on your system. Get the real IP address by running:

minikube ip

It’s a good idea to add an entry for minikube.data.gov.au to your hosts file (C:\Windows\System32\drivers\etc\hosts on Windows), mapping it to your Minikube IP address. Some services may assume this is in place. For example:

192.168.99.100 minikube.data.gov.au

Running on both host and minikube

It’s also possible to run what you’re working on your host, and the services your dependent on in minikube. Depending on what you’re doing, this might be simple or complicated.

Using the minikube database

This is super-easy, just run

kubectl port-forward combined-db-0 5432:5432

Now you can connect to the database in minikube as if it were running locally, while still taking advantage of all the automatic schema setup that the docker image does.

Running a microservice locally but still connecting through the gateway

You might find yourself developing an API locally that depends on authentication, which is easiest done by just logging in through the web interface and connecting through the gateway. You can actually make this work by telling the gateway to proxy your service to 192.168.99.1 in deploy/helm/magda/charts/gateway/templates/configmap.yaml. For instance, if I wanted to run the search api locally, I’d change configmap.yaml like so:

data:# When the config map is mounted as a volume, these will be created as files.routes.json:'{"search":{"to":"http://192.168.99.1:6102/v0"},#...etc

Then update helm:

helm upgrade magda -f deploy/helm/minikube-dev.yml deploy/helm/magda

Now when I go to http://${minikube ip}/api/v0/search, it’ll be proxied to my local search rather than the one in minikube.

Be aware that if your local service has to connect to the database or other microservices in minikube you’ll have to use kube-port-forward to proxy from localhost:{port} to the appropriate service in minikube - you can find a list of ports at https://github.com/TerriaJS/magda/blob/master/doc/local-ports.md.

In the likely even you need to figure out what the jwt shared secret is on your minikube, you can cheat by opening up a shell to a container that has that secret and echoing the environment variable:

kubectl exec-it gateway-cf9c575bb-th57x -- /bin/bash
echo$JWT_SECRET

Running local minions

You can use the same pattern for minions - register a webhook with a url host of 192.168.99.1 and it’ll post webhooks to your local machine instead of within the minikube network. Be aware that your minion won’t be able to find the registry until you use kubectl port-forward to make it work… e.g.

kubectl port-forward registry-api-79f7bf7787-5j52x 6101:80

What do I need to run?

Running individual components is easy enough, but how do we get a fully working system? It is rarely necessary to run all of MAGDA locally, but various components depend on other components as follows:

Running on your host machine

You can also avoid minikube and run magda components on your local machine - this is much, much trickier. In any component (except databases/elasticsearch), you can run:

yarn run dev

This will build and launch the component, and automatically stop, build, and restart it whenever source changes are detected. In some cases (e.g. code generation), it is necessary to run yarn run build at least once before yarn run dev will work. Typically it is not necessary to run yarn run build again in the course of development, though, unless you’re changing something other than source code.