Windows update breaks Output service on JBoss

JBossWe saw an interesting issue arise the other day where a Windows Server update caused a Production server instance to fail during a render operation. Both the Development and Test instances were patched successfully and no issues were seen in those environments.

Server was AEM Forms 6.1 SP1 on JBoss

The errors looked like this (extra lines removed for brevity):

16:57:35,336 WARN [com.adobe.formServer.docservice.LifeCycleImpl] (IDPSchedulerService_Worker-1) ALC-OUT-002-701: ServiceUtil already inititalized with service: OutputService. Aborting cache re-initialization.
16:57:35,575 INFO [com.adobe.output.config.OutputConfigImpl] (IDPSchedulerService_Worker-1) FSC008: Using the database to access and persist configuration properties.
16:57:36,057 INFO [com.adobe.service.ProcessResource] (IDPSchedulerService_Worker-1) ALC-BMC-001-505: Service XMLFormService: Starting native process with command line "D:\\Adobe\\Adobe_Experience_Manager_forms\\jboss\\standalone\\svcnative\\XMLFormService\\bin\\XMLForm.exe" -MyPath D:\Adobe\Adobe_Experience_Manager_forms\jboss\standalone\svcnative\XMLFormService -IOR IOR:000000000000002249444C3A636F6D2F61646F62652F736572766963652F4D616E616765723A312E300000000000000100000000000000BC000102000000000F31302E3133312E3131322E31393200000DC8000000000018383136363634303033362F42522D312F0100000000000000000000040000000000000008000000004A4143000000000100000020000000000501000100000001000100010001010900000002050100010001010000000014000000080000001A00000DC90000002100000030000000000000000100000000000000220000000000000000000000000000000000000000000000000000000000000000 -AppServer jboss
16:57:37,767 INFO [com.adobe.service.ProcessResource] (RequestProcessor-5) ALC-BMC-001-508: Service XMLFormService: Native process PID = 4476
16:57:37,915 WARN [com.adobe.service.J2EEConnectionFactoryManagerPeerImpl] (IDPSchedulerService_Worker-1) Service: XMLFormService resource: ProcessResource@6fb137ba(name=XMLForm.exe,pid=4476) could not schedule an interrupt for transaction: TransactionImple < ac, BasicAction: 0:ffff0a8370c0:-2618bc79:57ff2163:30dd status: ActionStatus.RUNNING > because excessive time was spent waiting in queue. Supplied timeout: 0s, Wait adjustment: 1s, Effective timeout: -1s.
16:57:37,956 INFO [com.adobe.document.XMLFormService] (RequestProcessor-5) ALC-XTG-101-000: UseProximity=false
16:57:38,380 ERROR [com.adobe.formServer.PA.XMLFormAgentWrapper] (IDPSchedulerService_Worker-1) ALC-OUT-002-013: XMLFormFactory, PAexecute failure: "java.lang.NullPointerException"
16:57:38,386 ERROR [stderr] (IDPSchedulerService_Worker-1) com.adobe.printSubmitter.PrintException: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.RenderFormException, cause: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.FormServerException
16:57:38,387 ERROR [stderr] (IDPSchedulerService_Worker-1) at com.adobe.printSubmitter.util.FormSubmitter.submit(FormSubmitter.java:211)
...
16:57:38,488 ERROR [stderr] (IDPSchedulerService_Worker-1) ... 300 more
16:57:38,490 ERROR [stderr] (IDPSchedulerService_Worker-1) Caused by: com.adobe.livecycle.formsservice.exception.FormServerException: java.lang.NullPointerException
...
16:57:38,492 ERROR [stderr] (IDPSchedulerService_Worker-1) ... 303 more
16:57:38,494 ERROR [stderr] (IDPSchedulerService_Worker-1) Caused by: java.lang.NullPointerException
16:57:38,494 ERROR [stderr] (IDPSchedulerService_Worker-1) at com.adobe.formServer.PA.XMLFormAgentWrapper.doPAExecute(XMLFormAgentWrapper.java:429)
16:57:38,495 ERROR [stderr] (IDPSchedulerService_Worker-1) ... 308 more
16:57:38,507 ERROR [com.adobe.workflow.AWS] (IDPSchedulerService_Worker-1) An exception was thrown with name com.adobe.livecycle.output.exception.OutputException message:com.adobe.printSubmitter.PrintException: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.RenderFormException, cause: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.FormServerException while invoking service OutputService and operation generatePDFOutput2 and no fault routes were found to be configured.

Solution

Delete the following JBoss temporary directories:

  • [aem_forms_install_dir]\jboss\standalone\svccommon
  • [aem_forms_install_dir]\jboss\standalone\svcnative
  • [aem_forms_install_dir]\jboss\standalone\tmp
  • [aem_forms_TEMP_dir] (C:\Windows\Temp by default on Windows)

Email Batch Size

Simple tip today: Always make sure you change the default email Batch Size when you install your AEM Forms JEE server.

By default the Batch Size on sending emails is set to ‘2’. This means that if you are testing single emails or Forms Workflows notifications from your new server, you wont see an email until there are two of them waiting to be sent.

I would like to say this setting is fine for a production server or some other use-case, but in reality I can’t think of one. If, for some reason, one email for a particular Task notification is all that is processed for the last hour, then the recipient will never see that email until something else happens in the system to generate another email. Definitely not something that should happen.

The fix

  1. Open AdminUI (http://localhost:8080/adminui) and login
  2. Browse to Services > Applications and Services > Service Management
  3.  Enter email into the Name: filter field and click Filter
    email_1.png
  4. Click the Email provider service (1st in the list)
    email_2.png
  5. Change Batch Size: to 1
    email_3.png
  6. Save

Adaptive Forms reserved words

There are a number of reserved words that you should not name your form elements in your Adaptive Forms. It is also suggested you should not use them in your schema for element or attribute names.

As a rule of thumb:

  1. Any property that is defined on the panel should not be used as the name of an element. Because that will restrict access to that property and lead to problems in the code. Examples (case-sensitive)
    1. panel
    2. repeatable
    3. toolbar
    4. instanceManager
    5. title
    6. summary
    7. enabled
    8. visible
    9. id
    10. name
    11. children
    12. items
    13. bindRef
    14. shortDescription/longDescription
    15. index
    16. any other API mentioned in https://helpx.adobe.com/aem-forms/6-1/javascript-api/Panel.html
  1. Some other words that are used in the generated script should not be used in the scripts or element names. e.g.
    1. Calculate
    2. Visibility
    3. Enabled
  2. You should not have field names starting with _, to prevent future name collisions with any of Adobe’s private APIs.

Edit: It wasn’t related to the schema elements per se, but the ‘elements’ in the adaptive form. Since dragging a schema object onto the form automatically names the form element the same name as the schema element or attribute, the form will stop working.

Edit 2: Ok it seems that naming your schema elements and attributes any of the above is aactually not recommended.

How to make JavaScript debugger in Designer work again

LC_bugIf you find that you can not debug your form JavaScript from within Designer, there is a good chance that Acrobat Pro is not configured correctly for debugging.

LiveCycle Designer attempts to preview PDFs by hooking into the current Adobe ActiveX Control add-on that is enabled inside Internet Explorer. This can be either Adobe Reader or Acrobat Pro. Acrobat Pro is required if you want to debug your JavaScript from within Designer. If you never see the Preview PDF option in Designer or it is greyed out, then try enabling the Adobe PDF Reader add-on within IE.

My problem was that I could not debug JavaScript from within Designer natively. Every time I tried to Preview certain PDFs, Designer would hang and FormDesigner.exe was left running when I closed Designer.

After many hours of hair pulling, uninstalling and re-installing Workbench, Designer, Acrobat (argh!) and generally going in circles I found that only forms that contained the FormBridge XFO were failing. Form Bridge is a JavaScript “bridge” library used to communicate externally with Workspace or Flex applications (Guides). It adds a script object to the hierarchy called ContainerFoundation_JS. When Preview PDF was invoked from Designer with this script added to the form, it would lock up.

The fix

It seems that if you set your JavaScript preferences in Acrobat Pro to Break when an exception is thrown – you’re gonna have a bad time…

Only use the Ignore or Trace settings, never Break as this actually breaks. Breaks Designer.

Acrobat Pro Preferences

Clean up after yourself

During this process and having to re-install everything multiple times, I was pointed in the direction of the Acrobat and Reader cleaner. Not to be confused with the Creative Cloud cleaner. The fact that any software requires a cleaner tool and doesn’t automatically clean up after itself both on installation and un-installation is a rant I choose not to go into right now for pure sanity reasons. Also be aware that the cleaner tool does not clean everything. I found remnants of Acrobat in my roaming profile under Windows 7 that even I could not delete and I was the local Administrator. </mini-rant>

Dropbox, Skype and Livecycle

command-prompt-glossyI recently noticed that Livecycle (JBoss) on my development laptop would frequently not start up correctly. It would throw all sorts of errors about ports already in use (1099, etc) and throw a complete wobbly. So I decided to dig around using TcpView and Process Explorer. It seems that Skype and Dropbox (to a lesser extent) would sometimes come in and steal some Livecycle ports when they run. Strangely, it didn’t reproduce every time I went looking for them but I saw enough to see that either one would try to grab the Naming Service port (1099) for some reason.

As both Skype and Dropbox started automatically on my laptop, I inevitably tried to run Livecycle without thinking and it would bomb out. So I decided it was easier to just quit both Dropbox and Skype until after Livecycle had started, then restart those processes when I needed them. They seemed to work fine without the RMI port anyway.

I knocked up a quick batch file to detect if either process was running and kill these apps before running Livecycle so I thought I’d share it. Props go to StackOverflow for the process detection script.

@ECHO OFF
tasklist /FI "IMAGENAME eq dropbox.exe" 2>NUL | find /I /N "dropbox.exe">NUL
if NOT "%ERRORLEVEL%"=="0" goto TEST_SKYPE
echo Dropbox is running...DIE!!
taskkill /f /im "dropbox.exe"

:TEST_SKYPE
tasklist /FI "IMAGENAME eq skype.exe" 2>NUL | find /I /N "skype.exe">NUL
if NOT "%ERRORLEVEL%"=="0" goto START_LC
echo Skype is running...DIE!!
taskkill /f /im "skype.exe"

:START_LC
net start "MySQL for Adobe LiveCycle ES4"
net start JBOSS_FOR_ADOBE_LIVECYCLE_ES4

:EXIT0
pause

CRXDE Lite code formatting

CRXDELite_Logo

One thing I always missed in CRXDE Lite was the lack of any code formatting tools. I always had to copy the code out into Notepad++ or Eclipse to format the code and then paste back into my JSP files. Not very convenient.

Today I accidentally stumbled across what I expect is an undocumented feature in CRXDE Lite – Code Formatting!

Shift-Tab

If you select any or all of the code on a page, and then hit Shift-Tab, it doesn’t tab the code back one tab stop (as you would expect), it actually formats the code.

Before

Before

After

After

Version

This was done with CRXDE Lite in CRX 2.4.23 (CQ 5.6.0.20130125) that comes with Adobe Livecycle ES4.  It seems to work in Chrome, Firefox and Intenet Exploder 10.

Automatically clean JBoss tmp directory

JBoss_by_Red_Hat I have been poking about JBoss lately trying to get messaging working and I had a look in the %JBOSS_HOME%/server/lc_turnkey/tmp directory. As you would imagine, it was chock full of temporary files (no surprise there). What I didn’t expect was how many files! So after a quick email to Adobe Support to confirm I wasn’t going to upset anything Livecycle related, I shut down JBoss and decided to clean the files out. Sometime into the delete, I got bored and lost count at 82,000 files and something like 6GB of space!

JBoss EAP 5.1.0

I had a quick Google around and it took me to a post on the JBoss Community Forum about deleting out files on a JBoss redeploy. Another user mentions that the tmp directory is never purged on restart or starting and stopping JBoss as it was in previous versions. The recommendation was to edit the run.bat file to do this every time the server is started.

The fix

The suggestion was to delete the files before JBoss starts up in the run.bat file. This example is for JBoss on Windows, but its pretty easy to adapt for Linux (just edit the run.sh file instead)

Below is the excerpt from my C:\Adobe\Adobe LiveCycle ES4\jboss\bin\run.bat

pushd %DIRNAME%..
if "x%JBOSS_HOME%" == "x" (
  set "JBOSS_HOME=%CD%"
)
popd

rem Remove tmp folder on startup
set TMP_FOLDER=%JBOSS_HOME%\server\lc_turnkey\tmp
rmdir "%TMP_FOLDER%" /s /q

set DIRNAME=

if "%OS%" == "Windows_NT" (
  set "PROGNAME=%~nx0%"
) else (
  set "PROGNAME=run.bat"
)

Enable JDBC testing

JDBCOne useful feature of Workbench is the ability to test your SQL statements directly from the JDBC activity editors.  However, since ADEP / ES3, this feature was turned off by default in Livecycle. Fortunately, it’s very easy to re-enable this feature.

 

Disabled by default

Disabled Test Feature

Disabled Test Feature


 

Steps to enable

1. Open AdminUI and browse to Home > Services > Applications and Services > Service Management and filter services by typing jdbc into the name field

Service Management

Service Management

2. Click the JdbcService 1:0 link

3. Check the Enable Testing checkbox and click Save

Enable JDBC Testing

Enable JDBC Testing

4. Go back into Workbench and the Test button should now be enabled

Enabled JDBC Testing

Enabled JDBC Testing

Move Workbench application storage location

ScreenShot012

By default, Workbench stores any Applications that it retrieves from the LiveCycle server into your user profile directory. In some circumstances this isn’t ideal. Maybe you have a domain policy that restricts your profile space or you want them stored on a network drive so they are backed up.

As Workbench is just eclipse, you can move the workspace folder wherever you want, but Adobe disabled the menu items to do this in the GUI (because you would never want to change your workspace directory ever – right?)

Just add this line to the workbench.ini file (usually located in the C:\Program Files (x86)\Adobe LiveCycle Workbench ES4\workbench folder – or wherever you installed Workbench)

-Duser.home=[yourNewPath]

e.g.

-Duser.home=D:\Workspaces\

Increase Workbench Timeout

This is a simple one. If you hate being logged out every 2 hours while you are using Workbench, just edit the Session Timeout Limit in AdminUI. Apparently there was a bug with Livecycle ES 2.5 and below that didn’t allow this to be changed, but it seems to work in ES3/ADEP and above.

Increase it to the maximum (1440 minutes) and you wont get logged out during the day.

Session Timeout Limit (Minutes)

Session Timeout Limit (Minutes)

aemblog

Everything AEM aka CQ5 based on my experience listed here.

Adobe AEM The Right Way

Best practices, tips, and tricks for your Adobe AEM project

/home/bkondepudi

A WCM journey with Day/Adobe CQ

Technoracle (a.k.a. "Duane's World")

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

Adobe LiveCycle Blog

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

XFA@Mobile

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

Code Monkey

Ramblings of a Developer