.. _smui-dev-setup: =============== Developing SMUI =============== The SMUI sources are hosted on the corresponding `GitHub repository`_. .. _GitHub repository: https://github.com/querqy/smui For development and building, you will need - `docker`_ with `BuildKit`_ capabilities (i.e. version 18.09 or higher) - `docker-compose`_ (optional) - Java Runtime Environment (version 11 is recommended) - NodeJS (version 12 is recommended) - `sbt`_ - `GNU make`_ .. _docker: https://www.docker.com/ .. _docker-compose: https://docs.docker.com/compose/ .. _BuildKit: https://docs.docker.com/develop/develop-images/build_enhancements/ .. _sbt: https://www.scala-sbt.org/download.html .. _GNU make: https://www.gnu.org/software/make/ Note: As SMUI currently build on top of Play 2.7, only Java 8 and 11 are supported (see https://discuss.lightbend.com/t/play-sample-on-windows-java-lang-illegalstateexception-unable-to-load-cache-item/8663). The latest `sbt` might come with a Java 17. One solution is to downgrade `sbt` accordingly, e.g. (Mac OS with `brew`): :: brew info sbt # Might output: "sbt: stable 1.6.1 (bottled)" brew info openjdk # Might output: "openjdk: stable 17.0.1 (bottled) [keg-only]" (WARN: Version too recent) # Uninstall sbt and openjdk brew uninstall sbt brew uninstall openjdk # Unfortunately, openjdk at a downgraded version might not be available (e.g. for Mac M1 chipset) or broke when building, so openjdk@11 should be installed brew install openjdk@11 # Maybe the downgraded sbt & openjdk version got not linked properly: brew unlink sbt brew link sbt@0.13 brew unlink openjdk brew link openjdk@11 # Stick to the advise: `echo 'export PATH="/opt/homebrew/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc` Building SMUI ------------- The code repository provides a Makefile for creating the SMUI docker image. In the project root, run :: make docker-build-only This will build both the backend and frontend components in a dockerized environment and create and tag the following images: ``querqy/smui:latest`` and ``querqy/smui:$VERSION``, where `$VERSION` is the version given in the build.sbt file. If you want to only build the fat jar itself, e.g. to build your own docker image, you may do so by running `sbt`: :: sbt assembly Running SMUI locally -------------------- When working on the SMUI sources, building a docker image on each change would be very time-consuming. Therefore, you may run a local development server by running: :: make serve The SMUI frontend will then be available on http://localhost:4200 , and the backend on http://localhost:9000. ``make serve`` assumes a MySQL/MariaDB instance, as outlined in the :ref:`SMUI quickstart` documentation. Alternatively, you may customize the database configuration. See :ref:`Development configuration` below. The development server is hot-reloading, i.e. will recompile on changes in the source files. In case more control is needed over the SMUI configuration ``sbt`` can also being used directly, e.g.: Here are some frequently used command: * ``compile``: Compiles the Scala/Play backend. * ``run -Dconfig.file=./smui-dev.conf``: Runs SMUI with a dev config (see below). * ``tasks -v``: Returns a complete list over all ``sbt`` commands available. * ``test``: Runs all backend tests. :: sbt "run -Dconfig.file=./smui-dev.conf" .. _smui-dev-config: Running SMUI with SSL locally ----------------------------- For local development that requires accessing SMUI using a HTTPS URL (e.g. to implement external authentication) we recommend using the Caddy reverse proxy. :: caddy reverse-proxy --internal-certs --from https://localhost --to :9000 Development configuration ------------------------- For developing new features and testing the application with different types of configuration, it is recommended to create a local development configuration of the application (instead of the productive one described above). There is the ``smui-dev.conf`` being excluded from version control through the ``.gitignore``, so that you can safely create a local development configuration in the project’s root (naming it ``smui-dev.conf``). Here is an example being used on a local development machine adjusting some features: :: include "application.conf" db.default.url="jdbc:mysql://localhost/smui?autoReconnect=true&useSSL=false" db.default.username="local_dev_db_user" db.default.password="local_dev_db_pass" smui2solr.SRC_TMP_FILE="/PATH/TO/LOCAL_DEV/TMP/FILE.tmp" smui2solr.DST_CP_FILE_TO="PATH/TO/LOCAL_DEV/SOLR/CORE/CONF/rules.txt" smui2solr.SOLR_HOST="localhost:8983" toggle.ui-concept.updown-rules.combined=true toggle.ui-concept.all-rules.with-solr-fields=true toggle.rule-deployment.log-rule-id=true toggle.rule-deployment.split-decompound-rules-txt=true toggle.rule-deployment.split-decompound-rules-txt-DST_CP_FILE_TO="/PATH/TO/LOCAL_DEV/SOLR/CORE/CONF/decompound-rules.txt" toggle.rule-deployment.pre-live.present=true toggle.rule-deployment.custom-script=true toggle.rule-deployment.custom-script-SMUI2SOLR-SH_PATH="/PATH/TO/LOCAL_DEV/smui2solr-dev.sh" toggle.rule-tagging=true toggle.predefined-tags-file="/PATH/TO/LOCAL_DEV/predefined-tags.json" ... play.http.secret.key="" # smui.authAction = controllers.auth.BasicAuthAuthenticatedAction # smui.BasicAuthAuthenticatedAction.user = smui_dev_user # smui.BasicAuthAuthenticatedAction.pass = smui_dev_pass As you can see, for development purposes you are recommended to have a local Solr installation running as well. For running The SMUI application locally on your development machine pass the above config file when starting the application in ``sbt``, e.g.: :: sbt run -Dconfig.file=./smui-dev.conf 9000 Furthermore, above’s configuration points to an alternative development version of the ``smui2solr.sh``-script. The file ``smui2solr-dev.sh`` is as well excluded from the version control. The following example provides a simple custom deployment script approach, that basically just delegates the script call to the main ``smui2solr.sh`` one: :: echo "In smui2solr-dev.sh - DEV wrapper for smui2solr.sh, proving custom scripts work" BASEDIR=$(dirname "$0") $BASEDIR/conf/smui2solr.sh "$@" exit $? It can be used as a basis for extension. .. note:: Remember to make the script executable (`chmod +x`). .. _smui-dev-custom-auth: Testing SMUI ------------ Backend unit tests (specifications) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To execute a specific unit test, you can call the corresponding ``scalatest`` with ``sbt``, e.g.: :: sbt "testOnly models.querqy.QuerqyExactMatchRulesGeneratorSpec" **NOTE:** The ``testOnly`` instruction needs to be in quotations (otherwise all tests will be executed). You can also run through the whole specification using the ``sbt`` ``test`` command. Useful commands: Docker powered MariaDB (with local persistence) ---------------------------------------------------------------- Assuming a folder ``smui_runtime_data`` exists. :: # start MySQL docker run --name smui-mysql -p 3306:3306 -v /LOCAL/PATH/TO/smui_runtime_data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=smui -e MYSQL_USER=smui -e MYSQL_PASSWORD=smui -e MYSQL_DATABASE=smui -d mysql # stop docker stop smui-mysql # list and remove eventually docker container ls -a docker container rm Note: Setting the MySQL root password is only for making potential root access easy (if necessary at some point). Also note: When developing with an Apple Silicon (M1 based) device, there does not seem to exist a suitable arm image for MySQL (as of Jan 2022). Therefore, the x86 architecture needs to be specified explicitly: ``docker run --platform linux/x86_64 --name smui-mysql [...]`` Developing git deployment method -------------------------------- SMUI offers the possibility to deploy rules.txt (files) to a git repository. For doing so in a local development setup, it might therefore be necessary to operate a local git instance. The following section describes how that can be achieved. Bootstrap a local git server (docker) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For the local git server, the dockerhub image `jkarlos/git-server-docker`_ will be used, see (command line): .. _jkarlos/git-server-docker: https://hub.docker.com/r/jkarlos/git-server-docker/ :: # create a private/public (SSH) key # e.g. ssh-keygen -t rsa -C "yourself@YourComputer.local" # create repo folder and provide (public) key mkdir /keys mkdir /repos # TODO better symlink? cp ~/.ssh/id_rsa.pub /keys/ # start the container (and provide public key) docker run -d -p 22:22 -v /keys:/git-server/keys -v /repos:/git-server/repos jkarlos/git-server-docker # NOTE: Your local development user must have permission to access information of your local git user (in case they differ) Init the git repository ~~~~~~~~~~~~~~~~~~~~~~~ You can run the following script (preferred as git test user itself) to init the repo (command line): :: # from within the git server docker container # NOTE: open shell in container: docker exec -it /bin/sh # (docker ps will give you the CONTAINER_ID) cd /repos mkdir smui_rulestxt_repo cd smui_rulestxt_repo git init --shared=true git add . git commit -m "my first commit" cd .. git clone --bare smui_rulestxt_repo smui_rulestxt_repo.git # initial manual checkout (on the host machine) # make sure, there exists an (at least empty) common rules.txt file on the master branch (clone it somewhere and create a master branch) touch rules.txt git add rules.txt git commit -m "empty rules.txt commit" git push To configure and start SMUI using a git deployment see “Deploy rules.txt to a git target“. Developing github actions for SMUI ---------------------------------- Build and deployment of SMUI's official docker image on DockerHub is realised through a github action, which is located under: `.github/workflows `_. Testing SMUI relies on Ryuk test containers within the ``sbt test`` build step. Unfortunately, there seems to be an issue with local build containers used by ``act`` (https://github.com/nektos/act), so that adjustments to the deployment workflow can only be tested while triggering the build on the github infrastructure (master push) and not tested locally with ``act``. This problem is described in the following issue: https://github.com/nektos/act/issues/501. Anyway, the workflow performs well on the github action container infrastructure. Releasing SMUI -------------- Every release of SMUI should come with its own version incrementing at least the build number of the semver. The release number is maintained under: :: smui/build.sbt The ``SmuiVersionSpec`` enforces version number increments in the test automation. Have fun coding SMUI!!