r/coldfusion • u/jabberwonk • 6d ago
Add-on services not starting after recent hot-fixes (and how we solved it)
TL;DR: Update 11 (and possibly 8 and 10) installer has an issue where it backs up and deletes cfusion/jetty/lib but never actually re-installs the new version into place, silently killing the ColdFusion Add-on Services (PDFg, Solr, Remote Admin).
The symptom:
After running the hotfix update, the **ColdFusion 2025 Add-on Services** Windows service fails to start. Event Viewer shows generic errors like:
- Service error **1067**: *"The process terminated unexpectedly."
We first noticed this starting with HF8, and it was still present in HF11.
Root cause (confirmed via install log):
Pulled the actual Adobe_ColdFusion_2025_Update_11_Install_*.log and traced the sequence of operations on cfusion/jetty/lib:
Installer extracts the *new* jetty payload (jetty 9.4.58, log4j 2.25.4, etc.) into a temp folder:
C:\Users<user>\AppData\Local\Temp<random>.tmp\dist\cfusion\jetty\lib...It then backs up the **live** `jetty\lib` folder:
Delete: C:\ColdFusion2025\cfusion/jetty/lib
Move Directory:
Source: ...\cfusion\jetty\lib
Destination: ...\hf-updates\hf-2025-00011-<id>\backup/jetty/libIt then deletes its own temp extraction folder without ever copying those new files into the live location:
Delete: C:\Users<user>\AppData\Local\Temp<random>.tmp\dist\cfusion\jetty
Every other component in the update (bin, other lib jars, wwwroot, etc.) gets a proper backup-then-copy pair in the log. jetty/lib only gets the backup half. The "install the new version" step is just missing. Result: the folder is empty/gone after the update finishes, and the add-on services - which run out of the embedded Jetty container - have nothing to load.
Why restoring from the hf-updates backup isn't the right fix:
Doing that just gives you back whatever jetty/lib you had before this update - not the actual Update 11 build. If this bug has been present since Update 8 (it has, for us), anyone "fixing" it via backup restore on 8, 9, 10 has been running increasingly stale jetty builds the whole time. That also explains why some servers' jetty/lib differs from others depending on when they last had a working install.
The actual fix:
Since the installer deletes its temp extraction before you can grab it, pull the real files from the hotfix installer package itself:
Find the hotfix installer files, typically under:
C:\ColdFusion2025\bundles\updateinstallers\hotfix-011-<id>\Inside that folder:
Disk1\InstData\Resource1.zipInside that zip:
dist_zg_ia_sf.jar(~200MB ignore any tiny same-named stub file underdist-win, that's just a platform marker)Extract this jar to disk first, then open the extracted copy fresh in 7-Zip (can't drill into a nested archive in one pass - extract, then open the extracted file as its own archive).
Inside, find
cfusion\jetty\lib\...and copy it over the livecfusion\jetty\lib. This worked cleanly on 2 out of 3 affected PCs (both Windows 11).
The gotcha on the 3rd server (Windows Server)
Copying jetty\lib alone wasn't enough service still wouldn't start (1067), even with correct file perms and a valid JVM.
start.jar (which sits in cfusion\jetty\, not lib\) carries a hardcoded jetty.versionstring that Jetty's .mod files use to resolve jar filenames (e.g. jetty-server-${jetty.version}.jar). Our start.jar was still the old 9.4.51.v20230217, while lib\ now had 9.4.58.v20250814 jars. Version mismatch → Jetty couldn't resolve any of its own core jars → nothing to start → the JVM just exits immediately, which Windows reports as a bare 1067.
Diagnosed by running Jetty manually with its own diagnostic flag:
D:\ColdFusion2025\cfusion\jetty>jre\bin\java.exe -jar start.jar --list-config
The fix was to go back to the extracted jetty folder from the downloaded hotfix and copy of the start.jar in that folder to your jetty\lib folder, replacing the one in that folder.
Worth noting that BeyondCompare also showed that the jetty-ipaccess.xml file in the extracted archive had 127.0.0.1 added to allowed address, while (at least my) one in production did not have that IP. If you've previously modified that file and this update changed it, look in the hf-updates\backup folder and restore from there.