Let’s face it, not everyone necessarily reads all the documentation before performing a Domino server upgrade. Domino server upgrades have always been easy, right? Sometimes the consequences of performing upgrades without reading important documentation, comes back to haunt us. Case in point; If you proceeded with a Domino 14 server upgrade on the Windows platform and you were unable to review the Domino 14 installation documentation to realize that there were very important changes that were documented as it relates to how a Domino Windows service now starts by default, there is a good chance that the Domino 14 server is consistently crashing upon startup. You then proceeded to research the server crash for clues and a remedy on google and google has led you to this document on the Extracomm web site. If you are one of the many Domino administrators who recently upgraded an HCL Domino server - Windows Server platform to Domino 14 and immediately experienced a Domino server crash, upon start up after the upgrade, here are the important details about what you need to know. This document identifies the likely source of the crash, how to resolve the problem, preventing further crashes and how get the Domino 14 server running without having to revert back/downgrade back to e.g. Domino 12.x
What are the symptoms and details of the Domino server crash?
When the Domino server is started for the first time after an upgrade to Domino 14, the following crash may occur, if the Domino server was started as a Windows service:
Crash Details and related error messages:
[0FDC:0002-1DC0] HCL Domino (r) Server (64 Bit), Release 14.0, November 09, 2023
[0FDC:0002-1DC0] (C) Copyright HCL Technologies. 1987, 2023
[0FDC:0002-1DC0] comp = 11, fnc = 81, probeid = 79, errcode = 5010, extsympt = 0065
69200000
Unexpected internal error returned to logger: 0x20692010
[0FDC:0002-1DC0] Thread=[0FDC:0002-1DC0]
[0FDC:0002-1DC0] Stack base=0xDE23D770, Stack size = 12640 bytes
[0FDC:0002-1DC0] PANIC: Unexpected internal error returned to logger: 0x20692010
Attempting to restart the Domino server after the initial crash, results in another similar crash. Interestingly, if the Domino server is started as an application, the server seems to run without issue. So, what is going on here?
Main reason why the Domino 14 server crashes after upgrade:
HCL Software has made some important changes as it relates to how Domino 14 operates on the Windows Server platform. As part of the "ongoing work to continuously improve security for Domino", HCL has updated the Windows installer to configure Domino to run using a non-admin user ID by default. Most notably, starting in Domino 14, the Domino Windows service will now use "NT Authority\Local Service" non-admin user ID instead of using a local System account, which was considered the standard Windows service configuration for Domino prior to Domino 14. Use of the default non-admin user ID "NT Authority\LocalService", is not mandatory and there is the option of changing the Domino Windows service to start using any valid Windows non-admin user ID instead.
Domino 14 – Windows Service Default Configuration:
The Domino 14 installer automatically sets new permissions granting the “Local Service” account, the necessary access rights it needs for the Domino program folder and the Domino data folder. One critical folder permissions assignment that the Domino installer does not automatically adjust is the permissions for the “LocalService” account to access the transaction log folder. As a direct result, the Domino server crashes upon startup, since the Local Service account does not have the permissions, it needs for the Domino Transaction Log folder.
How to change the Transaction Log folder permissions:
Folder/File permissions can be changed using one of two methods. Via the Windows GUI or using command line parameters. Using the command line parameters is the quickest way to assign the new folder permissions. That method is described below:
The following command-lines are just examples. Alter the command line parameters as needed to reflect the path and user id.
Note: S-1-5-19 is the internal representation of the Local Service Account.
At a command prompt, while using elevated privileges (Run as Administrator), type the following three commands in order:
1) The first command below will be used to grant full access rights including inheritance for the LocalService account:
icacls "d:\translog" /grant *S-1-5-19:(OI)(CI)(F)
2) The second command is used to change the owner of the folder/files to be the LocalService account:
icacls "d:\translog" /setowner *S-1-5-19 /t /c /q
3) The third command is used to remove an entry for "Everyone" from the folder/file permissions:
icacls "d:\translog" /remove:g Everyone /t /c /q
If the default LocalService account is not being used and another non-admin ID be used instead to start the Domino service, adjust the command line parameters accordingly. In this example, the non-admin ID being used is named: “Domino”
1) The first command below will be used to grant full access rights including inheritance for the Domino account:
icacls "e:\translog" /grant Domino:(OI)(CI)(F)
2) The second command is used to change the owner of the folder/files to be the LocalService account:
icacls "e:\translog" /setowner Domino /t /c /q
3) The third command is used to remove an entry for "Everyone" from the folder/files permissions:
icacls "e:\translog" /remove:g Everyone /t /c /q
Check 3rd party Domino add-in application folder/permissions requirements:
Affected by these folder/file permission changes that may be needed when running the Domino server using the Local Service account or another non-admin user id, is 3rd party application add-in tasks that are running on the Domino server. Check with the 3rd application vendor to determine how their product can be supported on Domino 14 and determine which folder/file permissions may need to adjusted for those applications to work properly.
Conclusion & Workaround:
If you are experiencing Domino 14 server crashes immediately after an upgrade from a previous Domino version, or you determine that 3rd party application add-ins are no longer functioning normally on Domino 14, consider to run the Domino server as an application or switch back to running the Domino server service using a local System account instead. Using the workarounds are not intended to be long term solution. Using the new preferred method of running the Domino server service using the local service account or another non-admin id should be adopted as soon as possible, since it is the preferred and most secure method for running the Domino 14 server on the Windows platform.
Useful links to information related to this topic:
· Installing SecurTrac on Domino 14 Windows platform that runs using a non-admin user ID
· Enable running Domino as a non-admin user