Tuning the Performance of UCF Content Transfer

Tuning the Performance of UCF Content Transfer

ID: esg100749


Use Count: 51


Date Created: 10/29/2008


Date Modified: 02/18/2011



Product(s): WDK, Webtop


Category(ies): Performance


Status: Approved


Related Bugs:  

SOLUTION
Original Author: Chase Harris - Enterprise Engineering Group
Introduction

In Documentum applications, the speed at which users can retrieve or add content to the system plays a big part in the overall end user experience. Network latency and bandwidth restrictions, physical architecture and client configuration all have an impact on the time it takes between initiating a content transfer, performing the upload or download, and returning control back to the user.



This document will discuss various tuning options available to achieve the best possible performance for UCF content transfer in your network environment.


Overview of UCF

UCF (Unified Client Facilities) is the EMC Documentum framework for performing content transfer between the client and content servers over HTTP. It provides many features that extend beyond simple file transfer protocol, such as:

    *
      Content compression to optimize transfers of larger files over the network
    *
      Support for complex documents, such as XML files
    *
      Support for compound document types, such as OLE objects inside a Microsoft Office document
    *
      Awareness of locally downloaded files, and the ability to avoid downloading the same content twice
    *
      Registry support, to allow checkout from one application and check in from another
    *
      Recoverability in the case of a brief network interruption
    *
      Location aware, to allow users to transfer content from the closest ACS or BOCS server

High-Level UCF Operations

When a content transfer is initiated, the UCF client running on the end user machine is initiated if it is not already running. The applet loads the UCF client jars and initiates contact with the UCF server, running on the application server.



The UCF client and server pass information back and forth about the environment and requested action before content is transferred between the two machines. Depending on the architecture and configuration, content may be transferred through the application server, directly to or from an ACS server on the content server, or a BOCS server located near the user.



After the content transfer is complete, there will be some additional communication from server to client to provide instructions on registry entries to be added or modified, directions on how to launch the appropriate application for view or edit operations, or instructions on how to delete the local file on check in.



Optimizing UCF for Best Performance



Tip #1 - Use ACS and BOCS for Content Transfer Whenever Possible



The introduction of ACS and BOCS servers in 5.3 SP1 have allowed content transfer to be performed directly from the content server, rather than requiring that the content always pass through the application server. This not only removes the double-hop that is required (which is very costly for small files) but also reduces the load on the application server.



In the 5.3 timeframe, ACS and BOCS could only be used for download operations such as checkout, export and view. All upload operations still passed through the application server.



From D6 onward, ACS and BOCS servers can also be used for upload operations.



Note that not all applications are network location aware and able to take advantage of remote ACS and BOCS servers.



Tip #2 - Optimize UCF Client Initialization



In 5.3 SP1 through SP5, D6 and D6SP1, the UCF client engine will time out after one minute of inactivity, and the javaw.exe process would be terminated. This means that if a user initiates another content transfer request after the timeout has occurred, the browser must re-initialize the UCF client and restart javaw.exe.



This value can be increased in the ucf.client.config.xml file as follows:



Edit the ucf.client.config.xml file (located at C:\Documents and Settings\<username>\Documentum\ucf\<hostname>\shared\config\ucf.client.config.xml) and add the following option to extend the UCF Client Timeout.

    <option name="client.engine.timeout>

         <value>30</value>

    </option>

Close all browser windows and log back into the application. The next content transfer operation will cause these new settings to take effect.



With these new settings, the UCF client will remain alive for at least 30 minutes of inactivity before terminating itself.



Note that the default value has been increased in 5.3 SP6 and D6.5.



Tip #3 - Configure On-Demand Virus Scanners to exclude unnecessary files



The first time the UCF client is initialized after restart, the on-demand scanner will need to scan all jars and DLLs prior to launching them. In addition, if it is the first time java has been used since reboot, the scanner will scan all of the Java jars and DLLs. This slows initialization down tremendously.



With some small changes to the configuration of the virus scanner, it is possible to see much faster UCF client initialization without sacrificing the security of the system. Please see Appendix A for more detailed instructions on how to implement this recommendation when using McAfee.



Tip #4 - Reduce UCF Client Communications over High Latency



When the UCF client is remote from the UCF server, the time it takes for each request/response is greatly increased. As there can be many such roundtrips, it can add significant time to the end-to-end operation, which is especially noticeable for small files.



In order to reduce the chattiness of the UCF communication, handshake and request optimizations were introduced in D6 SP1 and 5.3 SP6.



Note: These optimizations were also back ported to 5.3 SP5 and can be found at ftp://dev_pre:[e-mail]/UCF/5.3SP5/SP5_HF_GW2. However, EMC strongly recommends that customers upgrade to 5.3 SP6 instead of applying the hotfix as 5.3 SP6 includes a number of other performance and stability enhancements.



Tip #5 - Tune the Buffer or Chunk Size to Improve Download over High Latency



Each application server has a default sized buffer that it uses when transferring data across the network to clients. In most cases, this buffer is sized much too small, resulting in slower transfers over the WAN as more roundtrips are necessary.



Tomcat and Jboss

If you are using Tomcat or Jboss as the application server and you are not using ACS, you can add the socketBuffer parameter to the HTTP Connector as shown below:

    <!-- Define a non-SSL HTTP/1.1 Connector on port 8080 -->

        <Connector port="8080" maxHttpHeaderSize="8192"

                maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

                enableLookups="false" redirectPort="8443" acceptCount="100"

                connectionTimeout="20000" disableUploadTimeout="true"
                socketBuffer="64000"/>

If you are using 5.3 ACS and BOCS the application server is Tomcat 5.0.28. In versions 5.3 SP1 or SP2, you will need to add the socketBuffer parameter. In 5.3 SP3 through SP6, this parameter should already have been added by the installer.



In 6.5 and beyond, Jboss is the container for ACS and BOCS. In 6.5 SP1, enhancements were added to the embedded Jboss server to enable it to recognize and use the socketBuffer parameter.



BEA Weblogic

If you are using BEA Weblogic as your application server and you are not using ACS, be sure to add the -Dweblogic.ChunkSize parameter. By default, the ChunkSize is 4k which is much too small.



It can be set in the config.xml for the domain:

    <server>

    <name>server1</name>

    <ssl>

    <enabled>false</enabled>

    </ssl>

    <machine>localhost</machine>

    <listen-port>17001</listen-port>

    <cluster xsi:nil="true"></cluster>

    <listen-address></listen-address>

    <server-start>

    <arguments>-Dweblogic.Chunksize=64000</arguments>

    <username>weblogic</username>

    <password-encrypted>{3DES}wmzrpHOymvTy3q8TOeF1qQ==</password-encrypted>

    </server-start>

    </server>

It can also be defined in the set_domain_env batch file with the other JAVA_PROPERTIES values:

    set JAVA_PROPERTIES=-Dplatform.home=%WL_HOME% \

    -Dwls.home=%WLS_HOME% -Dwli.home=%WLI_HOME% \

    -Dweblogic.Chunksize=64000

In D6 and D6 SP1, the application server that hosts ACS and BOCS is BEA Weblogic, and the installer has automatically added those options to the script that starts the embedded Weblogic server (startMethodServer.cmd for example).



IBM Websphere

If you are not using ACS and content is being transferred through WAS, then you may see improvements in content download performance for large files by modifying the ResponseChunkSize configuration parameter in the plugin-cfg.xml. 

    <Config ResponseChunkSize="N">

where N equal the chunk size in kilobytes. The default value is 64 (kilobytes).



Tip #6 - Improve Upload Performance in D6 and Above



In D6, a parameter was added to the ucf.client.config.xml file that allowed the users to specify a chunk size to be used when uploading content.



Internal testing showed that a value of 49152 worked best with ACS on BEA when simulating various WAN conditions. However, in environments where customers are not using ACS write, or whose network conditions are different from that which was tested, this value may not be optimal. Increasing or decreasing the optimal.chunk.size parameter based on your specific network conditions can be beneficial.

    <option name="optimal.chunk.size">

                <value>49152</value>

    </option>

Note that this setting is ignored if the client is using Sun Java version 1.6 due to a Java bug (http://bugs.sun.com/bugdatabase/view_bug.do"bug_id=6631048).



Tip #7 - Use Adaptive Parallel Content Transfer to Consume More Available Bandwidth



By default in most versions of UCF, content download operations will be done in a single thread. If sufficient network bandwidth is available, the download operations could perform significantly faster if the file was broken up into smaller pieces and transferred by multiple concurrent threads.



In 5.3 SP6 and D6.5 the UCF client can be configured to use multiple threads when downloading files. This is controlled in the ucf.client.config.xml file using the following parameters:

    <option name="max.parallel.download.streams">

                <value>5</value>

    </option>

    <option name="min.parallel.segment.size">

                <value>131072</value>

    </option>

    <option name="measurement.time.interval">

                <value>300</value>

    </option>

    <option name="single.thread.throughput ">

                <value>131072</value>

    </option>

The "max.parallel.download.streams" parameter limits the number of threads that can be run concurrently when performing a parallel download. In this example, 5 separate streams could potentially be initialized. If one of the streams finishes their assigned task ahead of the others, it is then free to request a new range of bytes to be downloaded.



The "min.parallel.segment.size" parameter specifies that if the remaining portion of a document after a byte range is assigned is smaller than a specified value, it should not be assigned to a new thread. Rather the original thread's byte range should be extended to include those additional bytes. For example, if a thread is supposed to download 500Kb of a 600Kb file, as the remaining number of bytes is less than 128k, no new thread will be started and the original thread will assume ownership of those additional bytes.



The "Adaptive" part of parallel content transfer is controlled by the remaining two parameters. As tests have shown that disk I/O actually becomes a bottleneck and degrades performance if the content is transferred in parallel over LAN conditions, it is important to be able to control when the parallel content transfer is actually turned on. In this case, the UCF client will measure how many bytes were transferred initially by a single thread within a specified timeframe. If the number of bytes is less than what it expected, parallel content transfer will be turned on. If it is more than was expected, then it is assumed that bandwidth and latency are sufficient for a single thread to transfer the content most efficiently.



In the example above, the UCF client will measure the number of bytes that have been downloaded in the first 300ms of transfer (or as close to that time as possible). It will enable parallel content transfer if less than 131072 bytes (128k) have been downloaded, and disable it if more than 128k has been downloaded.



If your remote users have very high latency, it is likely that you will want to increase the "measurement.time.interval" to match the round trip time plus time to download a small portion of the file. For example, if you have 200ms round-trip latency, you might consider increasing the measurement.time.interval to 500ms to compensate for the time it takes for the request to reach the ACS server and start the content transfer.



In order to permanently disable parallel content transfer, all that must be done is changing the "max.parallel.download.streams" to 1, or decreasing the "single.thread.throughput" to a very small number, such as 1. In this case, regardless of the actual throughput, parallel content transfer will not be used. This is especially important in load testing scenarios, where many clients are being simulated from the same client machine.



Tip #8 - Tuning Content Download when Documentum User Directory is on a Network Drive



If the location of the Viewed and Checkout directories are on a network drive, or users regularly export content to a network drive, performance can be improved by setting the value of "ucf.file.buffer.size" to a larger number. The default value is 32768.



To set it for a single user, simply add the following to the ucf.client.config.xml on the client machine:

    <option name="ucf.file.buffer.size">

    <value>61440</value>

    </option>

To set the default for all users, add the same option to the ucf.installer.config.xml file on the application server and force a new version of UCF on the server side using the steps in Appendix B.



Tip #9 - Use UCF Client Logging to Measure performance



UCF Client logs are extremely useful when diagnosing UCF performance.



UCF client logging is enabled on the end user's machine.



Edit the ucf.client.logging.properties file, located at:
C:\Documents and Settings\<username>\Documentum\ucf\<hostname>\shared\config



For best results, set the log level to FINEST, as shown below:

    handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler

    .level=FINEST

    #-------------------

    java.util.logging.FileHandler.pattern=C:/Documentum/logs/ucf.client%u.log

    java.util.logging.FileHandler.limit=10485760

    java.util.logging.FileHandler.count=10

    java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter

    java.util.logging.FileHandler.encoding=UTF-8

    #-------------------

    java.util.logging.ConsoleHandler.level=OFF

    java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter

In addition, to prevent the UCF client log from overwriting itself each time, change the value of  java.util.logging.FileHandler.count to a number greater than 1. As each new client initializes, the old logs will be renamed and preserved for further analysis.



In the UCF client logs you will see entries such as "Logging request: getFile" and "Handled request: getFile" with timestamps. This can be used to better measure the time taken in the UCF pre-processing, content transfer and post-processing phases.



Appendix A:



Reduce the Effect of On-Access Scanning on UCF Initialization and Other Java Applets After Reboot

   1.
      Right-click on the VirusScan icon in the system tray and choose "On Access Scan Properties.
   2.
      Click on All Processes.
   3.
      Click on Advanced.
   4.
      Ensure "Scan inside archives" is NOT selected.
   5.
      Click on the Detection tab.
   6.
      Click on the Exclusions button.
   7.
      Click on Add.
   8.
      Add C:\Documents and Settings\s38737\Documentum\ucf , with "Also exclude subfolders", "On Read" and "On Write".
   9.
      Click on OK to save.
  10.
      Click Add to add another path.
  11.
      Add C:\Program Files\Java with "Also exclude subfolders" and "On Read".
  12.
      Click OK to save.

Appendix B:



Changing UCF Client Settings for All Users



1. Edit the <app>\webtop\wdk\contentXfer\ucf.installer.config.xml file.

2. In this file, you will find the following line,

    <"xml version="1.0" encoding="UTF-8"">
    <"dctm fileVersion="5.3.0.1" compatibilityVersion="5.3.0.1"">
    <ucfInstaller codebase="" loggerLevel="INFO">
    <app id="shared" version="5.3.0.XXX" compatibilityVersion="5.3.0" /> 

3. Change the highlighted text above from 5.3.0.XXX to 5.3.0.XXXa

4. Under the configurations tag, change or add the desired option.

    <option name="name_of_option">

    <value>value</value>

    </option> 

The next time the UCF client engine is initialized on the client side, the new UCF settings will be downloaded and added to the ucf.client.config.xml file.



There may be cases where the "default" values in the ucf.installer.config.xml should not be applied to the client side. In this case, the "persistent" attribute can be added to the option on the client side. This will prevent any changes on the server side from being applied to that particular client.

    <option name="name_of_option" persistent="true">

    <value>value</value>

    </option>

If not specified, then persistent is set to false.



The persistent property can be specified on the ucf.installer.config.xml and in the ucf.client.config.xml. The following chart outlines the expected behavior:



Server           Client             Description                                                                        

false     false     Server value will override client value.

true      true      Server value will override client value.

true      true      Server value will override client value.

false     true      Client value will not be overridden.