Deserialization error submitting forms to AEM Forms JEE

Found an interesting bug (feature?) in AEM Forms 6.2 where you can’t submit your forms to an AEM Forms JEE Workflows server, even after configuring the DSCConfigService.

The errors you will see are similar to the following:

25.11.2016 09:55:14.210 *ERROR* [14.2.208.49 [1480028113565] POST /content/forms/af/[link-to-your-form]/jcr:content/guideContainer.af.submit.jsp HTTP/1.1] com.adobe.aemds.guide.servlet.GuideSubmitServlet Could not complete Submit Action due to An exception occurred processing JSP page /libs/fd/afaddon/components/actions/lcprocess/post.POST.jsp at line 79
org.apache.sling.api.scripting.ScriptEvaluationException: An exception occurred processing JSP page /libs/fd/afaddon/components/actions/lcprocess/post.POST.jsp at line 79
	at org.apache.sling.scripting.core.impl.DefaultSlingScript.call(DefaultSlingScript.java:416)
...
Caused by: org.apache.sling.api.SlingException: An exception occurred processing JSP page /libs/fd/afaddon/components/actions/lcprocess/post.POST.jsp at line 79
	at org.apache.sling.scripting.jsp.jasper.servlet.JspServletWrapper.handleJspExceptionInternal(JspServletWrapper.java:683)
...
Caused by: com.adobe.aemds.guide.service.GuideException: Deserialization not allowed for class sun.util.calendar.ZoneInfo (on Fri Nov 25 09:55:14 AEDT 2016)
	at com.adobe.aemds.guide.addon.service.impl.GuideLCServiceConnectorImpl.invokeProcess(GuideLCServiceConnectorImpl.java:196)
...

Deserialization not allowed for class sun.util.calendar.ZoneInfo?? – that never happened in 6.1 or below…

Turns out you need to add the sun.util.* packages to the Deserialization Firewall config in OSGi (I didn’t even know that was a thing…). It is mentioned in the Adobe help but is very vague as to why.

The direct link to it on Publish would be similar to:

http://localhost:4503/system/console/configMgr/com.adobe.cq.deserfw.impl.DeserializationFirewallImpl

Just add a new package in the Whitelisted classes or packages prefixes and add the sun.util. package to it. Note the trailing period after “util”.

deserialization_firewall.png

Advertisements

Windows update breaks Output service on JBoss

JBossWe saw an interesting issue arise the other day where a Windows Server update caused a Production server instance to fail during a render operation. Both the Development and Test instances were patched successfully and no issues were seen in those environments.

Server was AEM Forms 6.1 SP1 on JBoss

The errors looked like this (extra lines removed for brevity):

16:57:35,336 WARN [com.adobe.formServer.docservice.LifeCycleImpl] (IDPSchedulerService_Worker-1) ALC-OUT-002-701: ServiceUtil already inititalized with service: OutputService. Aborting cache re-initialization.
16:57:35,575 INFO [com.adobe.output.config.OutputConfigImpl] (IDPSchedulerService_Worker-1) FSC008: Using the database to access and persist configuration properties.
16:57:36,057 INFO [com.adobe.service.ProcessResource] (IDPSchedulerService_Worker-1) ALC-BMC-001-505: Service XMLFormService: Starting native process with command line "D:\\Adobe\\Adobe_Experience_Manager_forms\\jboss\\standalone\\svcnative\\XMLFormService\\bin\\XMLForm.exe" -MyPath D:\Adobe\Adobe_Experience_Manager_forms\jboss\standalone\svcnative\XMLFormService -IOR IOR:000000000000002249444C3A636F6D2F61646F62652F736572766963652F4D616E616765723A312E300000000000000100000000000000BC000102000000000F31302E3133312E3131322E31393200000DC8000000000018383136363634303033362F42522D312F0100000000000000000000040000000000000008000000004A4143000000000100000020000000000501000100000001000100010001010900000002050100010001010000000014000000080000001A00000DC90000002100000030000000000000000100000000000000220000000000000000000000000000000000000000000000000000000000000000 -AppServer jboss
16:57:37,767 INFO [com.adobe.service.ProcessResource] (RequestProcessor-5) ALC-BMC-001-508: Service XMLFormService: Native process PID = 4476
16:57:37,915 WARN [com.adobe.service.J2EEConnectionFactoryManagerPeerImpl] (IDPSchedulerService_Worker-1) Service: XMLFormService resource: ProcessResource@6fb137ba(name=XMLForm.exe,pid=4476) could not schedule an interrupt for transaction: TransactionImple < ac, BasicAction: 0:ffff0a8370c0:-2618bc79:57ff2163:30dd status: ActionStatus.RUNNING > because excessive time was spent waiting in queue. Supplied timeout: 0s, Wait adjustment: 1s, Effective timeout: -1s.
16:57:37,956 INFO [com.adobe.document.XMLFormService] (RequestProcessor-5) ALC-XTG-101-000: UseProximity=false
16:57:38,380 ERROR [com.adobe.formServer.PA.XMLFormAgentWrapper] (IDPSchedulerService_Worker-1) ALC-OUT-002-013: XMLFormFactory, PAexecute failure: "java.lang.NullPointerException"
16:57:38,386 ERROR [stderr] (IDPSchedulerService_Worker-1) com.adobe.printSubmitter.PrintException: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.RenderFormException, cause: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.FormServerException
16:57:38,387 ERROR [stderr] (IDPSchedulerService_Worker-1) at com.adobe.printSubmitter.util.FormSubmitter.submit(FormSubmitter.java:211)
...
16:57:38,488 ERROR [stderr] (IDPSchedulerService_Worker-1) ... 300 more
16:57:38,490 ERROR [stderr] (IDPSchedulerService_Worker-1) Caused by: com.adobe.livecycle.formsservice.exception.FormServerException: java.lang.NullPointerException
...
16:57:38,492 ERROR [stderr] (IDPSchedulerService_Worker-1) ... 303 more
16:57:38,494 ERROR [stderr] (IDPSchedulerService_Worker-1) Caused by: java.lang.NullPointerException
16:57:38,494 ERROR [stderr] (IDPSchedulerService_Worker-1) at com.adobe.formServer.PA.XMLFormAgentWrapper.doPAExecute(XMLFormAgentWrapper.java:429)
16:57:38,495 ERROR [stderr] (IDPSchedulerService_Worker-1) ... 308 more
16:57:38,507 ERROR [com.adobe.workflow.AWS] (IDPSchedulerService_Worker-1) An exception was thrown with name com.adobe.livecycle.output.exception.OutputException message:com.adobe.printSubmitter.PrintException: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.RenderFormException, cause: java.lang.NullPointerException in com.adobe.livecycle.formsservice.exception.FormServerException while invoking service OutputService and operation generatePDFOutput2 and no fault routes were found to be configured.

Solution

Delete the following JBoss temporary directories:

  • [aem_forms_install_dir]\jboss\standalone\svccommon
  • [aem_forms_install_dir]\jboss\standalone\svcnative
  • [aem_forms_install_dir]\jboss\standalone\tmp
  • [aem_forms_TEMP_dir] (C:\Windows\Temp by default on Windows)

PurgeProcess.bat fix

Update

This bug should be fixed in Core | Quick Fix 1048-010

There is a command line tool in AEM Forms JEE that allows you to purge processes from the system. It can be found in <install-dir>/sdk/misc/Foundation/ProcessPurgeTool folder.

Problem is, it doesn’t work. It throws an error similar to:

UNEXPECTED ERROR!!! See the stack trace below for details.
 java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/SecureProtocolSocketFactory
 at com.adobe.idp.workflow.ProcessPurgeTool.getSoapClientFactory(ProcessPurgeTool.java:197)
 at com.adobe.idp.workflow.ProcessPurgeTool.purgeProcess(ProcessPurgeTool.java:291)
 at com.adobe.idp.workflow.ProcessPurgeTool.main(ProcessPurgeTool.java:457)
 Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.protocol.SecureProtocolSocketFactory
 at java.net.URLClassLoader$1.run(Unknown Source)
 at java.net.URLClassLoader$1.run(Unknown Source)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(Unknown Source)
 at java.lang.ClassLoader.loadClass(Unknown Source)
 at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
 at java.lang.ClassLoader.loadClass(Unknown Source)
 ... 3 more

Solution: you have to add %THIRDPARTY_DIR%\commons-httpclient-3.1.jar; to the SOAP_LIBS path in the batch file or shell script.

Windows Batch file

@ECHO OFF
SETLOCAL

IF NOT DEFINED LIVECYCLE_SDK_HOME (
ECHO Please set LIVECYCLE_SDK_HOME environment variable.
ECHO It should point to the LiveCycle SDK root directory.
ECHO For example: set LIVECYCLE_SDK_HOME=&amp;amp;quot;C:\Adobe\Adobe_Experience_Manager_forms\sdk&amp;amp;quot;
GOTO end
)

IF NOT EXIST &amp;amp;quot;%LIVECYCLE_SDK_HOME%\client-libs\common\adobe-livecycle-client.jar&amp;amp;quot; (
ECHO The LIVECYCLE_SDK_HOME environment variable seems to point to a wrong directory.
GOTO end
)

SET SDK_COMMON_DIR=%LIVECYCLE_SDK_HOME%\client-libs\common
SET THIRDPARTY_DIR=%LIVECYCLE_SDK_HOME%\client-libs\thirdparty
SET SOAP_LIBS=%THIRDPARTY_DIR%\activation.jar;%THIRDPARTY_DIR%\axis.jar;%THIRDPARTY_DIR%\commons-codec-1.3.jar;%THIRDPARTY_DIR%\commons-collections-3.1.jar;%THIRDPARTY_DIR%\commons-discovery.jar;%THIRDPARTY_DIR%\commons-logging.jar;%THIRDPARTY_DIR%\dom3-xml-apis-2.5.0.jar;%THIRDPARTY_DIR%\jaxen-1.1-beta-9.jar;%THIRDPARTY_DIR%\jaxrpc.jar;%THIRDPARTY_DIR%\log4j.jar;%THIRDPARTY_DIR%\mail.jar;%THIRDPARTY_DIR%\saaj.jar;%THIRDPARTY_DIR%\serializer.jar;%THIRDPARTY_DIR%\wsdl4j.jar;%THIRDPARTY_DIR%\xalan.jar;%THIRDPARTY_DIR%\xbean.jar;%THIRDPARTY_DIR%\xercesImpl.jar;%THIRDPARTY_DIR%\commons-httpclient-3.1.jar;
SET CLASSPATH=process-purge-tool.jar;%SDK_COMMON_DIR%\adobe-livecycle-client.jar;%SDK_COMMON_DIR%\adobe-workflow-client-sdk.jar;%SDK_COMMON_DIR%\adobe-jobmanager-client-sdk.jar;%SDK_COMMON_DIR%\adobe-usermanager-client.jar;%SOAP_LIBS%

CALL java -cp &amp;amp;quot;%CLASSPATH%&amp;amp;quot; com.adobe.idp.workflow.ProcessPurgeTool %*

:end
ENDLOCAL

Korn Shell script

#!/bin/ksh

if [ &amp;quot;$LIVECYCLE_SDK_HOME&amp;quot; == &amp;quot;&amp;quot; ] ; then
  echo &amp;quot;Please set LIVECYCLE_SDK_HOME environment variable.&amp;quot;
  echo &amp;quot;It should point to the LiveCycle SDK root directory.&amp;quot;
  echo &amp;quot;For example: export LIVECYCLE_SDK_HOME=/home/Adobe_Experience_Manager_forms/sdk&amp;quot;
  exit 1
fi

export SDK_COMMON_DIR=$LIVECYCLE_SDK_HOME/client-libs/common
export THIRDPARTY_DIR=$LIVECYCLE_SDK_HOME/client-libs/thirdparty
export SOAP_LIBS=$THIRDPARTY_DIR/activation.jar:$THIRDPARTY_DIR/axis.jar:$THIRDPARTY_DIR/commons-codec-1.3.jar:$THIRDPARTY_DIR/commons-collections-3.1.jar:$THIRDPARTY_DIR/commons-discovery.jar:$THIRDPARTY_DIR/commons-logging.jar:$THIRDPARTY_DIR/dom3-xml-apis-2.5.0.jar:$THIRDPARTY_DIR/jaxen-1.1-beta-9.jar:$THIRDPARTY_DIR/jaxrpc.jar:$THIRDPARTY_DIR/log4j.jar:$THIRDPARTY_DIR/mail.jar:$THIRDPARTY_DIR/saaj.jar:$THIRDPARTY_DIR/serializer.jar:$THIRDPARTY_DIR/wsdl4j.jar:$THIRDPARTY_DIR/xalan.jar:$THIRDPARTY_DIR/xbean.jar:$THIRDPARTY_DIR/xercesImpl.jar:$THIRDPARTY_DIR/commons-httpclient-3.1.jar
export CLASSPATH=./process-purge-tool.jar:$SDK_COMMON_DIR/adobe-livecycle-client.jar:$SDK_COMMON_DIR/adobe-workflow-client-sdk.jar:$SDK_COMMON_DIR/adobe-jobmanager-client-sdk.jar:$SDK_COMMON_DIR/adobe-usermanager-client.jar:$SOAP_LIBS

set -f

java -cp $CLASSPATH com.adobe.idp.workflow.ProcessPurgeTool &amp;quot;$@&amp;quot;

Invoking AEM Forms 6.2 services via EJB in JBoss

From Jboss AS6 onwards (AEM Forms JEE 6.0+) , the way you invoke Forms services via EJB has changed.

Below is a code snippet from a DSC of how to invoke the TaskManagerService via EJB in AEM Forms 6.2. Note how the scheme ‘jnp://’ has been removed from the URL.

long taskId = 1234;
Document document = null;
ServiceClientFactory factory = null;

Properties ConnectionProps = new Properties();
ConnectionProps.setProperty(ServiceClientFactoryProperties.DSC_DEFAULT_EJB_ENDPOINT, "localhost:4447");
ConnectionProps.setProperty(ServiceClientFactoryProperties.DSC_TRANSPORT_PROTOCOL,
				ServiceClientFactoryProperties.DSC_EJB_PROTOCOL);
ConnectionProps.setProperty(ServiceClientFactoryProperties.DSC_SERVER_TYPE, "JBoss");
ConnectionProps.setProperty(ServiceClientFactoryProperties.DSC_CREDENTIAL_USERNAME, "administrator");
ConnectionProps.setProperty(ServiceClientFactoryProperties.DSC_CREDENTIAL_PASSWORD, "password");

factory = ServiceClientFactory.createInstance(ConnectionProps);
TaskManager tmObj = TaskManagerClientFactory.getTaskManager(factory);
FormInstance formInst = tmObj.getFormInstanceForTask(taskId, 0, true);
document = formInst.getDocument();
System.out.println("Document size = " + document.length());

NOTE: You also need to start using the updated JBoss client library from your JBoss instance (\bin\client\jboss-client.jar)

Inside AEM Forms JEE LDAP Connector

I have been working with a client recently who had some interesting questions about the AEM Forms JEE LDAP connector and how it actually worked. Other than how to configure it, most of the internal workings are a mystery. So I thought I would share some of the (edited) responses on how the LDAP connector actually processes directories internally and how parts of it works.

Shout out goes to Neerav at Adobe Engineering for the great answers.

Filtering LDAP Attributes

LDAP query clients allow for the ability to specify which attributes are searched and also which attributes are returned in the search results. Some complex directories return many attributes by default, thus slowing down performance and creating unnecessary network overhead.

Q: Is there a way to filter out the attributes that LDAP returns when it tries to search for users in AEM Forms JEE?

A: The LDAP sync process fetches only those attributes which are configured in User/Group synchronization screen. There are many attributes which are optional on that screen and those can be set blank. If any attribute is blank on that screen, then it is not fetched from LDAP.

Moving User around Forests

Q: If a user is Active in Forest A and moves to Forest B, they will be made inactive in the first and active in the second. They will be granted the exact same idmGUID [LDAP unique identifier]. Will the LDAP Import Process simply recognise this as the same user and update their details in the Adobe Database with the details from the new forest?

A: As long as its unique fields like canonicalName (i.e. idmGUID) and userid (i.e. samAccountName or uid) remain the same, the user identity would be same.

Delta Synchronization

Q: The Delta Sync process in AEM Forms, is only a Delta in terms of the Adobe product. A Full import against the LDAP backend occurs, before a comparison of modifyTimeStamp fields in the new Full Import is carried out against the current list in Adobe. Hence, would the [network] impact of a Full Sync v.s. a Delta be identical on the LDAP Infrastructure?

A. Delta sync would have less load on LDAP infrastructure. We do not fetch details of groups (and group members) if a particular group is unchanged. modifyTimeStamp is an operational attribute and is not fetched unless it is part of requested attributes. It is present in all standard LDAPs (including Active Directory). This attribute is used to determine if a User or Group record has changed since last synchronization. If delta sync is enabled, users and groups which have not changed since last synchronization are not updated during synchronization.

A note on modifiedTimeStamp

modifiedTimeStamp and delta sync presented an interesting challenge since each server in the IAM farm updated the modifiedTimeStamp when it received the user update from the IAM master. This could possibly be a second or two after the master was updated. Since the LDAP queries were load balanced between the LDAP servers and each server (potentially) had a slightly different modifiedTimeStamp, it made it impossible to reliably do a delta sync with the LDAP directory. This is potentially a large overhead with a large LDAP population.

Email Batch Size

Simple tip today: Always make sure you change the default email Batch Size when you install your AEM Forms JEE server.

By default the Batch Size on sending emails is set to ‘2’. This means that if you are testing single emails or Forms Workflows notifications from your new server, you wont see an email until there are two of them waiting to be sent.

I would like to say this setting is fine for a production server or some other use-case, but in reality I can’t think of one. If, for some reason, one email for a particular Task notification is all that is processed for the last hour, then the recipient will never see that email until something else happens in the system to generate another email. Definitely not something that should happen.

The fix

  1. Open AdminUI (http://localhost:8080/adminui) and login
  2. Browse to Services > Applications and Services > Service Management
  3.  Enter email into the Name: filter field and click Filter
    email_1.png
  4. Click the Email provider service (1st in the list)
    email_2.png
  5. Change Batch Size: to 1
    email_3.png
  6. Save

AEM Forms 6.2 is released

Adobe_Experience_Manager_logo_SCREEN_RGB_128px AEM Forms 6.2 has now been released to the general public. It contains a lot of nice new features like Touch UI forms building, Theme editor and a bigger and better Rule Editor that was introduced in AEM Forms 6.1 FP1. One of the additions I am looking forward to trying is pre-configuring a web service for use in the Rule Editor.

Check out what’s new in 6.2: https://helpx.adobe.com/aem-forms/6-2/whats-new.html

 

A warning about changing the default Super Administrator password

aem_warningOne thing you should always do in your production instance is change the default passwords. Everyone does that right? riiiight

Well when you decide to change the default Super Administrator (administrator) password in AEM Forms Workflows 6.1 or 6.2 there is an unexpected side effect. After changing this password you would notice that it works fine when you test it so you go about your other tasks. Then you will notice that after a period of time you are automatically locked out!

Under the hood, there is an OSGi service that links the OSGi CRX instance to the JEE instance. This is called the Adobe LiveCycle Client SDK Configuration. This service constantly attempts to log into the JEE instance and do stuff. If you change the default administrator password, and don’t change this password, this service will lock your Super Administrator account out.

Great. How do I stop this?

It’s easy. Just open the OSGi system console and change the password in the configuration screen. You can open the config item directly from a browser. e.g.

http://localhost:8080/lc/system/console/configMgr/com.adobe.livecycle.dsc.clientsdk.internal.DSCConfigService

client_sdk_config.png

Ok…I’m already locked out. Now what?

The default unlock time is set to 30 minutes after 20 incorrect attempts, so you can change the OSGi password and wait 30 minutes…or you are super impatient like me and can bust out some SQL-fu.

Remember the password you set on the database when you installed the instance? You’re going to need that now. Don’t remember it? Sorry – you now have to wait. Go browse Reddit for a bit and come back…

First, start by installing a SQL editor for your particular database (I’m going to show MySQL since that is in the turnkey edition)

  1. First log in to your MySQL instance using the root account
    Manage Server Connections_mysql.png

2. Then enter the password by clicking Store in Vault… (this is the password that you used when you installed the instance). Click OK

database_password.png

3. Click Test Connection

 database_success.png

4. You should see a couple of adobe database instances (these will be the database names you specified during setup) I usually call mine ‘adobe’ or ‘adobe_wf’

database_schemas.png

5. Now locate the Super Admin account using the following SQL:

SELECT * FROM adobe.edcprincipalentity where canonicalname = 'SuperAdmin';

6. You should see the column countauthfailure has a number > 20 and islocked is set to 1 (locked)

database_unlock.png

7. Now you can either use the GUI to change the value or run the following SQL:

UPDATE adobe.edcprincipalentity 
SET islocked = 0 
WHERE canonicalname = 'SuperAdmin';

8. You should be able to log in to your Super Administrator account now

Its probably a good idea to create a second Super Administrator account just in case this happens again.

Adaptive Forms reserved words

There are a number of reserved words that you should not name your form elements in your Adaptive Forms. It is also suggested you should not use them in your schema for element or attribute names.

As a rule of thumb:

  1. Any property that is defined on the panel should not be used as the name of an element. Because that will restrict access to that property and lead to problems in the code. Examples (case-sensitive)
    1. panel
    2. repeatable
    3. toolbar
    4. instanceManager
    5. title
    6. summary
    7. enabled
    8. visible
    9. id
    10. name
    11. children
    12. items
    13. bindRef
    14. shortDescription/longDescription
    15. index
    16. any other API mentioned in https://helpx.adobe.com/aem-forms/6-1/javascript-api/Panel.html
  1. Some other words that are used in the generated script should not be used in the scripts or element names. e.g.
    1. Calculate
    2. Visibility
    3. Enabled
  2. You should not have field names starting with _, to prevent future name collisions with any of Adobe’s private APIs.

Edit: It wasn’t related to the schema elements per se, but the ‘elements’ in the adaptive form. Since dragging a schema object onto the form automatically names the form element the same name as the schema element or attribute, the form will stop working.

Edit 2: Ok it seems that naming your schema elements and attributes any of the above is aactually not recommended.

Changing administrator password on AEM Forms JEE

LC_bugOk, so this isn’t really a bug but it was damn annoying to figure out.

So we had a development server that was publicly available, so the first thing you do is change the default administrator password (right?).

Then I start seeing messages from JBoss server.log:

19:14:56,218 WARN  [com.adobe.idp.um.businesslogic.authentication.AuthenticationManagerBean] (Thread-166) Authentication failed for user [admin] (Scheme - Username/Password) Reason: Username or password is incorrect . Refer to debug level logs for category com.adobe.idp.um.businesslogic.authentication for further details

What? Where is that coming from?

Anyway, to cut a long story short, it turns out if you want to change the default administrator password on AEM Forms JEE 6.X, you need to update the OSGi configuration for the Adobe LiveCycle Client SDK Configuration or it will lock out your administrator account!

Adobe_LiveCycleClient_SDK_Config

The URL to get to it directly is: http://localhost:8080/lc/system/console/configMgr/com.adobe.livecycle.dsc.clientsdk.internal.DSCConfigService

Remember that if you have any Author or Publish instances pointing to your Processing server, they will need to be updated too or the same thing will happen.

aemblog

Everything AEM aka CQ5 based on my experience listed here.

Adobe AEM The Right Way

Best practices, tips, and tricks for your Adobe AEM project

/home/bkondepudi

A WCM journey with Day/Adobe CQ

Technoracle (a.k.a. "Duane's World")

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

Adobe LiveCycle Blog

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

XFA@Mobile

A multi-purpose toolkit for the Adobe LiveCycle and AEM Forms developer.

Code Monkey

Ramblings of a Developer