Recent by Daniel Nashed 

Domino Download Bash Script leveraging My HCL Software Portal  

By Daniel Nashed | 11/27/23 12:58 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The My HCL Software Portal is still an early access offering in parallel to Domino 14 early access. It is planned to replace the Flexnet download soon and way easier and much faster to navigate. The website just works and has awesome performance. Domino 14 AutoUpdate leverages a new software download API to automatically download software into autoupdate.nsf. The download just needs a download token, which can be requested if you are log into https://my.hcltechsw.com/. I have been looking for a way to automatically download software for a couple of years. Now with the new portal and this new API it is possible to write a Bash script for full command-line operations including a simple to use menu. There are two different modes. By default the script uses My HCL Software navigation. But alternatively it can also leverage Domino 14 Auto Update software.jwt, which has more granular information and allows a more structured download package browsing experience.

Get your Linux environment ready for Domino V14  

By Daniel Nashed | 10/20/23 3:43 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino V14 is planned to ship end of this year. For Windows the system requirements don't really change, because of the universal run-time. But for Linux a newer compiler brings new OS dependencies. Specially the glibc version, which brings the base run-time support for C and also the C++ standard libs are important. An application build with a newer compiler on a newer Linux version does not run on older versions with lower glibc versions. glibc is the The GNU C Library - https://www.gnu.org/software/libc/ The new version required was released in August 2021 and is part of most current long term release Linux distributions.

Running Domino with SELinux on current REHL/CentOS Stream 9 & Co  

By Daniel Nashed | 10/9/23 2:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino 12.0.2 added support for SELinux in enforced mode, which is enabled by default by newer installations. SELinux is a lower level security feature, which can even limit processes running with root permissions. But the application needs to have a SELinux profile. I ran into this week on my own on a RHEL 9.2 machine and I got the same problem from a partner yesterday. It turns out that systemd can't read from /tmp any more. But the Domino service from my Nash!Com start script writes the domino process id into the /tmp folder. With SELinux enabled you get the following error message when looking into your service status (domino statusd). The start and stop operations of your server will also hang, because systemctl will hang. systemd[1]: domino.service: Can't convert PID files /tmp/domino.pid O_PATH file descriptor to proper file descriptor: Permission denied

Required Notes and Domino anti-virus file exclusions   

By Daniel Nashed | 10/9/23 2:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This discussion came up in an OpenNTF Discord channel. The question was if this might be a good idea to keep OS level anti-virus enabled for Notes/Domino files. There is a clear statement from HCL about exclusions. But the technote doesn't explain why those exclusions are important. The exclusion might be different for each anti-virus production in detail. It also depends on customer IT policies how to exclude data. This can be either by path, extension or process. There are also recommendations from some anti-virus vendors stating the same exclusions for their specific product: Guidelines for excluding Notes and Domino directory and files when running an operating system Antivirus https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0093046 ible Ans

Domino CertMgr GitHub Repository with additional material  

By Daniel Nashed | 9/27/23 1:21 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

Documentation is always a challenge. This is specially true when it comes to complex topics like SSL/TLS certificates. Many admins still use their old cook books to get certificates created. When HCL introduced CertMgr in Domino 12.0 the team asked for feedback in the early code drops. And the team is keeping asking in public and private forums since then. We really need your help to get it right. We need detailed feedback and questions. My new plan is to turn questions into FAQs and Howto documents in this GitHub repository.

Who moved my Domino keyfile.kyr files?  

By Daniel Nashed | 9/25/23 2:00 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino security in 2023 Domino 12.0 introduced a new, standards based and open way to work with web server certificates. Instead of using command-line tools like OpenSSL and the Domino kyrtool you can now manage all web server certificates in a domain wide certstore.nsf. The new functionality based on the well known text based PEM standard for certificates provides simplified flows and automation options for all type of certificates. Domino 12 also introduces the more modern ECDSA (sometimes referred as ECC) keys/certificates which are based on elliptic crypto which has dramatically less overhead. Moving from keyfile.kyr to certstore.nsf The legacy kyr files can be automatically imported into certstore.nsf with a single command-line operation (load certmgr -importkyr all).

Domino V14 backup for notes.ini  

By Daniel Nashed | 6/1/23 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino backup is around since 12.0 and it got improvements in every release. There are not many current AHA ideas for Domino Backup & Restore. One smaller feature you can see in EAP1 is the backup of the notes.ini.

How to use Domino OTS on Kubernetes to import an existing TLS Certificate  

By Daniel Nashed | 5/30/23 12:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino One Touch Setup has been designed with flexibility in mind, with special focus on getting a server up in a secure way. On Docker you can just mount PEM files into the container. On Kubernetes TLS Certificates and Keys are stored in secrets. Personally I am not a big fan of storing PEM files on disk. But you could at least set a password on the PEM file you import. Here is a basic example how to create a secret on K8s and reference it in OTS. Even the simple environment variable setup supports the security settings for CertMgr. Of course the same functionality is also available with the more flexible JSON based configuration.

Importing trusted MicroCA Roots for a Nomad Lab environment  

By Daniel Nashed | 5/29/23 12:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Yesterday I worked on a lab configuration based on Windows Sandbox, Domino and Nomad Web. The biggest challenge is to have a trusted certificate for Nomad Web. Nomad Server running with the Micro CA A Nomad Server can use Domino CertMgr Micro CA Certs. But the root is not trusted in your browser. I took a closer look and came up with a simple solution. which makes the import dramatically easier. No more searching for the right trust store and handling PEM files manually.

Get prepared for Notes/Domino V14 Early Access Code Drop 1  

By Daniel Nashed | 5/26/23 9:32 AM | Business - Events / People | Added by Oliver Busse

You can get hands on experience with Notes/Domino V14 end of this month. Here are some tips to get prepared. All of the software is only intended for non-production use! So you should prepare a VM to get started. But you should really take a look and have a try. Specially for business partners this is a call for action to test their applications with the updated back-end components. As announced earlier Notes/Domino moved to up to date compilers and a newer Java version. Also the client is 64bit only. I hope to see many of you in the EAP forum or at DNUG conference face to face.

Tuning Domino Servers for TLS sessions  

By Daniel Nashed | 5/24/23 2:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

My previous post was mainly about HTTP traffic and I mentioned TLS/SSL don't use the maximum number of connections settings, because they have a SSL/TLS session. Establishing a new TLS session has significant overhead! And you have to make sure in any application, that those sessions are cached and resumed. I revisited a blog post from 2012 where I explained a fix, which went into 8.5.3. And was enabled in 8.5.4 by default (which turned into the 9.0 release when shipped as far I recall). There was an issue with the session cache and a new cache had been implemented in 8.5.3. Today the new cache is the default and SSL_USE_ADDSESSION2=1 does not exist any more.

Debugging program crashes with gdb on Linux  

By Daniel Nashed | 4/10/23 9:37 AM | Development - Notes / Domino | Added by Roberto Boccadoro

This bugged me for a while because I had no idea what was happening. One of my OpenSSL-based tools in C crashed once in a while. I was only able to find out once wrote my own small tool to check the server listener of my other application. The crash happened very intermittently in different places when I opened and closed the connection very quickly. Adding a delay of 1 ms stopped the crash. But if you are running a service on the internet with port scanners around, you better find out in detail. It turned out to be the SIGPIPE event causing my program to terminate. But the interesting part is how I found out about the crash.

TLS/SSL Cipher Troubleshooting  

By Daniel Nashed | 4/10/23 9:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Every Domino release adds more TLS ciphers to the weak list to ensure poper security. We can expect the next versions also to have less ciphers available. Domino ensures for clients and servers, that the list of ciphers provided is safe. In addition the default behavior is that the server decides the order of ciphers to pick. And only allows secure renegotiation to prevent the client to pick a less secure cipher. Usually this doesn't cause a lot of trouble for inbound connections. Modern browsers support modern ciphers. But outgoing connections for LDAP and ICAP could be a challenge. I had to look into an ICAP connection problem this week. To demonstrate how the TLS handshake works, I wrote a small OpenSSL demo program in C. This turned into a quite flexible troubleshooting tool over the weekend.

Picking the right Linux Distribution  

By Daniel Nashed | 4/5/23 10:42 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

There isn't one best or right distribution in general. There are many variations. For example Redhat/CentOS based platforms have flavors like Rocky and Alma Linux. The HCL Domino community project looks into many possible combinations as the base image --> https://opensource.hcltechsw.com/domino-container/concept_environments/ There are basically three different main flavors with different toppings: - Redhat/CentOS based (with yum and dnf in later versions to manage packages) - Ubuntu/Debian (with apt to manage packages) - SUSE Enterprise/Leap etc. (with zypper to manage packages) The right distribution is really depending on your needs.

Nomad Web server connection options  

By Daniel Nashed | 1/25/23 7:30 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

Nomad Web is a modern HCL client offering in form of a Progressive Web Application (PWA) running in your web browser. In addition to Windows or Mac, it also works on Ubuntu and other Linux distributions! So there is finally a client offering for Linux clients again! The Nomad Web application is installed on a server providing the required files for download. Those files can be stored on a SafeLinx or Domino/Nomad Web server.

Windows Sandbox - A feature you should know  

By Daniel Nashed | 1/23/23 2:20 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The sandbox can be a very useful tool for many different situations. I am often using it for Domino server or client install tests. But there are many other use cases including training environments etc. It's a full throw away sandbox environment recreated every time you start it. The only limitation is that you can't reboot the Windows for example after a software update. But even installing the Windows re-distributable run-time package does not require a boot. Most applications like Notes/Domino install it on their own. I needed it to test my own applications. But there is an easy way to download and silent install it:

NGINX TCP Stream with SNI support. More than helpful for lab environments  

By Daniel Nashed | 1/23/23 2:15 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In production you usually want centralized certificate handling and off-loading TLS termination to a load-balancer. I posted scripts to have NGINX realod certs automatically from Domino CertMgr via HTTPS to leverage Domino's Let's Encrypt implementation. But sometimes you really want all your servers directly exposed over TLS. For example in a lab environment with limited resources and only one IP, you might want to still have each of the hosts expose their services on their own.

How to get the error message for a Notes error code  

By Daniel Nashed | 12/13/22 1:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Sometimes scripts or Domino server commands only return an error code and you would like to know the error message. There is an easy way to get the error message back from a server command. "show message [module] In most cases you don't need server tasks specific error messages and just use the decimal error code.

Restic – Command Line Tool supporting Windows VSS  

By Daniel Nashed | 11/24/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that Domino 12.0.2 is has a native VSS Writer, we can look into new interesting integrations. In my session at SUTOL conference this week, I showed a first version of a Restic integration for Domino 12.0.2 via VSS. Restic is a very interesting application (https://restic.net) It's a single binary written in GO And uses a approach like Borg Backup uses. But in contrast to Borg Backup it has full Windows support. This includes VSS Writer + AutoRecovery support! It is Open Source, efficient, flexible & secure. And very simple to setup & use!

Docker cp with permissions and owner change  

By Daniel Nashed | 10/11/22 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

By the default the ownership of a file copied into a running container is always root:root. Depending on how you want to use the copied file, this ownership isn't what you want. You can't pass user/group or permissions to the docker cp command. Changing the owner or mode would need root permissions inside the container. Containers usually run with an unprivileged application user. For Domino this is notes:notes with the IDs 1000:1000.

Domino 12.0.2 One Touch setup with Let’s Encrypt certificates  

By Daniel Nashed | 9/15/22 3:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

One Touch setup (OTS) is quite powerful tool. But sometimes you need to know exactly what happens and combine functionality to make best use of it. With Domino 12.0.2 OTS creates certstore.nsf automatically and you can let it create a MicroCA for you. But what if you want to use a Let's Encrypt certificate instead? There is a quite simple way to just find and update the existing document with a appConfiguration. And if you specify notes.ini CertMgr_ACCEPT_TOU=1 the ACME account license agreement will be automatically accepted (already part of 12.0.0).

Download certificate chain without OpenSSL  

By Daniel Nashed | 9/12/22 3:17 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Usually OpenSSL is the tool of choice for all type of certificate operations. But what if no OpenSSL command line is available? Like in a Domino container where you can't install software? After some research, I came up with the keytool, which is part of the JVM Domino ships.

K3s, Podman and a registry  

By Daniel Nashed | 9/5/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Rancher Desktop is a great all-in-one desktop environment. When running it with the Docker back-end you have all in one environment for development and run-time. For a server, K3s (https://k3s.io) is my platform of choice. It is production ready and easy to deploy. For Kubernetes, you always need a registry to pull images. As soon you need custom images, you will need a registry to upload and download your image. K3s allows you to configure private registries. You could use any registry. I am just running the registry Docker image on Podman in my environment.

Domino Community Image - New Nomad Server install option  

By Daniel Nashed | 7/30/22 10:22 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

The Nomad server is a new offering to directly add Nomad support to your Domino server instead of using a SafeLinx server. Recently I added a SafeLinx container to the Domino community project. Now I am adding the Nomad Server to the Domino image as a new build options.

Domino Container automation testing  

By Daniel Nashed | 7/25/22 12:20 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Containers are not only a good way to run Domino. It is also the perfect environment for automation testing. Domino 12 introduced OneTouch Setup to automate deployments, which also lets you create reproducible Domino test server scenarios. As a starting point I am building an automation test for the Domino image itself. So in future for every commit on the Git repository I can run automation to ensure the image works. The test automation can be used in your own environment as well after an image built in your environment. It can be also be extended for your own application testing.

Customizing Domino Backup mail notifications  

By Daniel Nashed | 7/22/22 6:42 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

Domino Backup offers to send e-mails depending on the status of your backup. By default you are getting an e-mail in case of error or warning. I am rarely getting error messages from my servers. In this case here I updated my server to a new kernel and ZFS drivers failed to build.

Nomad Server 12.0.2 on Linux just works  

By Daniel Nashed | 7/20/22 10:32 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

The Nomad Server is a small component, you install on your Domino server. And it is bundled with the Nomad Web files. So it is a all-in one server add-on solution. The installation sounds more complicated then it is. It's really simple to install. And I am thinking about making it an install option for the Domino community container image.

Why run Domino in a container today  

By Daniel Nashed | 7/20/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As my of you know, I am a big fan of running Domino and other applications in a container. This can be a classical Docker/Podman deployment or K8s. Containers might not be good for everyone. But a lot of software is available in a "Docker image", which can run in multiple environments. Domino's main deployment model will not change to Docker.

Domino 12 Restore point in time  

By Daniel Nashed | 7/15/22 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Backup can restore databases point in time! For other backup applications this functionality is usually only available with archive transaction log. But with circular translog or linear translog mode, the most current backup should have all the translogs available to recover point in time as well. Domino Restore allows you to restore point in time in that case. Even it is not guaranteed that the translogs are still there, this can be still a good configuration if you have servers without dramatic load.

Linux shell scripts: Difference between "set" and "env" -- fixed the Domino start script  

By Daniel Nashed | 7/11/22 4:28 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Domino start script has always been using "set" to list the environment variables, before running the sever. This information can be important to understand the environment passed to your Domino server at start-up. It turns out that there had been a change over time, which causes much more information to be listed, then just the environment variables.