Oozie相关函数

 

EL:Expression Language

http://oozie.apache.org/docs/3.3.2/WorkflowFunctionalSpec.html#a4.2.1_Basic_EL_Constants

Oozie相关函数

1.Decision Node

1.1 switch case

default一定要设置。

Example：

<workflow-app name="foo-wf" xmlns="uri:oozie:workflow:0.1">

    ...

    <decision name="mydecision">

        <switch>

            <case to="reconsolidatejob">

              ${fs:fileSize(secondjobOutputDir) gt 10 * GB}

            </case>

            <case to="rexpandjob">

              ${fs:filSize(secondjobOutputDir) lt 100 * MB}

            </case>

            <case to="recomputejob">

              ${ hadoop:counters('secondjob')[RECORDS][REDUCE_OUT] lt 1000000 }

            </case>

            <default to="end"/>

        </switch>

    </decision>

    ...

</workflow-app>

 

1.2 fork join

Fork和join成对出现。join动作的功能是要同步fork动作启动的多个并行执行的线程。如果fork启动的所有执行的线程都能够成功完成，那么join动作就会等待它们全部完成。如果有至少一个线程执行失败，kill节点会“杀掉”剩余运行的线程。

Example：

<workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1">

    ...

    <fork name="forking">

        <path start="firstparalleljob"/>

        <path start="secondparalleljob"/>

    </fork>

    <action name="firstparallejob">

        <map-reduce>

            <job-tracker>foo:8021</job-tracker>

            <name-node>bar:8020</name-node>

            <job-xml>job1.xml</job-xml>

        </map-reduce>

        <ok to="joining"/>

        <error to="kill"/>

    </action>

    <action name="secondparalleljob">

        <map-reduce>

            <job-tracker>foo:8021</job-tracker>

            <name-node>bar:8020</name-node>

            <job-xml>job2.xml</job-xml>

        </map-reduce>

        <ok to="joining"/>

        <error to="kill"/>

    </action>

    <join name="joining" to="nextaction"/>

    ...

</workflow-app>

 

1.Basic EL Functions

Basic EL Constants

• KB: 1024, one kilobyte.
• MB: 1024 * KB, one megabyte.
• GB: 1024 * MB, one gigabyte.
• TB: 1024 * GB, one terabyte.
• PB: 1024 * TG, one petabyte.

All the above constants are of type long .

String firstNotNull(String value1, String value2)

It returns the first not null value, or null if both are null .

Note that if the output of this function is null and it is used as string, the EL library converts it to an empty string. This is the common behavior when using firstNotNull() in node configuration sections.

String concat(String s1, String s2)

It returns the concatenation of 2 strings. A string with nullvalue is considered as an empty string.

String replaceAll(String src, String regex, String replacement)

Replace each occurrence of regular expression match in the first string with the replacement string and return the replaced string. A 'regex' string with null value is considered as no change. A 'replacement' string with null value is consider as an empty string.

String appendAll(String src, String append, String delimeter)

Add the append string into each splitted sub-strings of the first string(=src=). The split is performed into src string using thedelimiter . E.g. appendAll("/a/b/,/c/b/,/c/d/", "ADD", ",")will return /a/b/ADD,/c/b/ADD,/c/d/ADD . A append string withnull value is consider as an empty string. A delimiter string with value null is considered as no append in the string.

String trim(String s)

It returns the trimmed value of the given string. A string withnull value is considered as an empty string.

String urlEncode(String s)

It returns the URL UTF-8 encoded value of the given string. A string with null value is considered as an empty string.

String timestamp()

It returns the UTC current date and time in W3C format down to the second (YYYY-MM-DDThh:mm:ss.sZ). I.e.: 1997-07-16T19:20:30.45Z

String toJsonStr(Map) (since Oozie 3.3)

It returns an XML encoded JSON representation of a Map. This function is useful to encode as a single property the complete action-data of an action, wf:actionData(String actionName) , in order to pass it in full to another action.

String toPropertiesStr(Map) (since Oozie 3.3)

It returns an XML encoded Properties representation of a Map. This function is useful to encode as a single property the complete action-data of an action, wf:actionData(String actionName) , in order to pass it in full to another action.

String toConfigurationStr(Map) (since Oozie 3.3)

It returns an XML encoded Configuration representation of a Map. This function is useful to encode as a single property the complete action-data of an action, wf:actionData(String actionName) , in order to pass it in full to another action.

2.Workflow EL Function

String wf:id()

It returns the workflow job ID for the current workflow job.

String wf:name()

It returns the workflow application name for the current workflow job.

String wf:appPath()

It returns the workflow application path for the current workflow job.

String wf:conf(String name)

It returns the value of the workflow job configuration property for the current workflow job, or an empty string if undefined.

String wf:user()

It returns the user name that started the current workflow job.

String wf:group()

It returns the group/ACL for the current workflow job.

String wf:callback(String stateVar)

It returns the callback URL for the current workflow action node, stateVar can be a valid exit state (=OK= or ERROR ) for the action or a token to be replaced with the exit state by the remote system executing the task.

String wf:transition(String node)

It returns the transition taken by the specified workflow action node, or an empty string if the action has not being executed or it has not completed yet.

String wf:lastErrorNode()

It returns the name of the last workflow action node that exit with an ERROR exit state, or an empty string if no a ction has exited with ERROR state in the current workflow job.

String wf:errorCode(String node)

It returns the error code for the specified action node, or an empty string if the action node has not exited with ERRORstate.

Each type of action node must define its complete error code list.

String wf:errorMessage(String message)

It returns the error message for the specified action node, or an empty string if no action node has not exited with ERRORstate.

The error message can be useful for debugging and notification purposes.

int wf:run()

It returns the run number for the current workflow job, normally 0 unless the workflow job is re-run, in which case indicates the current run.

Map wf:actionData(String node)

This function is only applicable to action nodes that produce output data on completion.

The output data is in a Java Properties format and via this EL function it is available as a Map .

int wf:actionExternalId(String node)

It returns the external Id for an action node, or an empty string if the action has not being executed or it has not completed yet.

int wf:actionTrackerUri(String node)

It returns the tracker URIfor an action node, or an empty string if the action has not being executed or it has not completed yet.

int wf:actionExternalStatus(String node)

It returns the external status for an action node, or an empty string if the action has not being executed or it has not completed yet.

3.Hadoop EL Functions

Hadoop EL Constants：

• RECORDS: Hadoop record counters group name.
• MAP_IN: Hadoop mapper input records counter name.
• MAP_OUT: Hadoop mapper output records counter name.
• REDUCE_IN: Hadoop reducer input records counter name.
• REDUCE_OUT: Hadoop reducer input record counter name.
• GROUPS: 1024 * Hadoop mapper/reducer record groups counter name.

4.Hadoop Jobs EL Function

 wf:actionData()

5.HDFS EL Functions

For all the functions in this section the path must include the FS URI. For example hdfs://foo:8020/user/tucu .

boolean fs:exists(String path)

It returns true or false depending if the specified path URI exists or not.

boolean fs:isDir(String path)

It returns true if the specified path URI exists and it is a directory, otherwise it returns false .

boolean fs:dirSize(String path)

It returns the size in bytes of all the files in the specified path. If the path is not a directory, or if it does not exist it returns -1. It does not work recursively, only computes the size of the files under the specified path.

boolean fs:fileSize(String path)

It returns the size in bytes of specified file. If the path is not a file, or if it does not exist it returns -1.

boolean fs:blockSize(String path)

It returns the block size in bytes of specified file. If the path is not a file, or if it does not exist it returns -1.