storm topology config

http://xumingming.sinaapp.com/185/twitter-storm-%E5%9C%A8%E7%94%9F%E4%BA%A7%E9%9B%86%E7%BE%A4%E4%B8%8A%E8%BF%90%E8%A1%8Ctopology/

public static java.lang.String STORM_ZOOKEEPER_SERVERS

A list of hosts of ZooKeeper servers used to manage the cluster.

 

---

STORM_ZOOKEEPER_PORT

public static java.lang.String STORM_ZOOKEEPER_PORT

The port Storm will use to connect to each of the ZooKeeper servers.

 

---

STORM_LOCAL_DIR

public static java.lang.String STORM_LOCAL_DIR

A directory on the local filesystem used by Storm for any local filesystem usage it needs. The directory must exist and the Storm daemons must have permission to read/write from this location.

 

---

STORM_SCHEDULER

public static java.lang.String STORM_SCHEDULER

A global task scheduler used to assign topologies's tasks to supervisors' wokers. If this is not set, a default system scheduler will be used.

 

---

STORM_CLUSTER_MODE

public static java.lang.String STORM_CLUSTER_MODE

The mode this Storm cluster is running in. Either "distributed" or "local".

 

---

STORM_LOCAL_HOSTNAME

public static java.lang.String STORM_LOCAL_HOSTNAME

The hostname the supervisors/workers should report to nimbus. If unset, Storm will get the hostname to report by calling InetAddress.getLocalHost().getCanonicalHostName(). You should set this config when you dont have a DNS which supervisors/workers can utilize to find each other based on hostname got from calls toInetAddress.getLocalHost().getCanonicalHostName().

 

---

STORM_LOCAL_MODE_ZMQ

public static java.lang.String STORM_LOCAL_MODE_ZMQ

Whether or not to use ZeroMQ for messaging in local mode. If this is set to false, then Storm will use a pure-Java messaging system. The purpose of this flag is to make it easy to run Storm in local mode by eliminating the need for native dependencies, which can be difficult to install. Defaults to false.

 

---

STORM_ZOOKEEPER_ROOT

public static java.lang.String STORM_ZOOKEEPER_ROOT

The root location at which Storm stores data in ZooKeeper.

 

---

STORM_ZOOKEEPER_SESSION_TIMEOUT

public static java.lang.String STORM_ZOOKEEPER_SESSION_TIMEOUT

The session timeout for clients to ZooKeeper.

 

---

STORM_ZOOKEEPER_CONNECTION_TIMEOUT

public static java.lang.String STORM_ZOOKEEPER_CONNECTION_TIMEOUT

The connection timeout for clients to ZooKeeper.

 

---

STORM_ZOOKEEPER_RETRY_TIMES

public static java.lang.String STORM_ZOOKEEPER_RETRY_TIMES

The number of times to retry a Zookeeper operation.

 

---

STORM_ZOOKEEPER_RETRY_INTERVAL

public static java.lang.String STORM_ZOOKEEPER_RETRY_INTERVAL

The interval between retries of a Zookeeper operation.

 

---

STORM_ZOOKEEPER_AUTH_SCHEME

public static java.lang.String STORM_ZOOKEEPER_AUTH_SCHEME

The Zookeeper authentication scheme to use, e.g. "digest". Defaults to no authentication.

 

---

STORM_ZOOKEEPER_AUTH_PAYLOAD

public static java.lang.String STORM_ZOOKEEPER_AUTH_PAYLOAD

A string representing the payload for Zookeeper authentication. It gets serialized using UTF-8 encoding during authentication.

 

---

STORM_ID

public static java.lang.String STORM_ID

The id assigned to a running topology. The id is the storm name with a unique nonce appended.

 

---

NIMBUS_HOST

public static java.lang.String NIMBUS_HOST

The host that the master server is running on.

 

---

NIMBUS_THRIFT_PORT

public static java.lang.String NIMBUS_THRIFT_PORT

Which port the Thrift interface of Nimbus should run on. Clients should connect to this port to upload jars and submit topologies.

 

---

NIMBUS_CHILDOPTS

public static java.lang.String NIMBUS_CHILDOPTS

This parameter is used by the storm-deploy project to configure the jvm options for the nimbus daemon.

 

---

NIMBUS_TASK_TIMEOUT_SECS

public static java.lang.String NIMBUS_TASK_TIMEOUT_SECS

How long without heartbeating a task can go before nimbus will consider the task dead and reassign it to another location.

 

---

NIMBUS_MONITOR_FREQ_SECS

public static java.lang.String NIMBUS_MONITOR_FREQ_SECS

How often nimbus should wake up to check heartbeats and do reassignments. Note that if a machine ever goes down Nimbus will immediately wake up and take action. This parameter is for checking for failures when there's no explicit event like that occuring.

 

---

NIMBUS_CLEANUP_INBOX_FREQ_SECS

public static java.lang.String NIMBUS_CLEANUP_INBOX_FREQ_SECS

How often nimbus should wake the cleanup thread to clean the inbox.

 

See Also:

NIMBUS_INBOX_JAR_EXPIRATION_SECS

---

NIMBUS_INBOX_JAR_EXPIRATION_SECS

public static java.lang.String NIMBUS_INBOX_JAR_EXPIRATION_SECS

The length of time a jar file lives in the inbox before being deleted by the cleanup thread. Probably keep this value greater than or equal to NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS. Note that the time it takes to delete an inbox jar file is going to be somewhat more than NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS (depending on how often NIMBUS_CLEANUP_FREQ_SECS is set to).

 

See Also:

NIMBUS_CLEANUP_FREQ_SECS

---

NIMBUS_SUPERVISOR_TIMEOUT_SECS

public static java.lang.String NIMBUS_SUPERVISOR_TIMEOUT_SECS

How long before a supervisor can go without heartbeating before nimbus considers it dead and stops assigning new work to it.

 

---

NIMBUS_TASK_LAUNCH_SECS

public static java.lang.String NIMBUS_TASK_LAUNCH_SECS

A special timeout used when a task is initially launched. During launch, this is the timeout used until the first heartbeat, overriding nimbus.task.timeout.secs.

A separate timeout exists for launch because there can be quite a bit of overhead to launching new JVM's and configuring them.

 

---

NIMBUS_REASSIGN

public static java.lang.String NIMBUS_REASSIGN

Whether or not nimbus should reassign tasks if it detects that a task goes down. Defaults to true, and it's not recommended to change this value.

 

---

NIMBUS_FILE_COPY_EXPIRATION_SECS

public static java.lang.String NIMBUS_FILE_COPY_EXPIRATION_SECS

During upload/download with the master, how long an upload or download connection is idle before nimbus considers it dead and drops the connection.

 

---

UI_PORT

public static java.lang.String UI_PORT

Storm UI binds to this port.

 

---

UI_CHILDOPTS

public static java.lang.String UI_CHILDOPTS

Childopts for Storm UI Java process.

 

---

DRPC_SERVERS

public static java.lang.String DRPC_SERVERS

List of DRPC servers so that the DRPCSpout knows who to talk to.

 

---

DRPC_PORT

public static java.lang.String DRPC_PORT

This port is used by Storm DRPC for receiving DPRC requests from clients.

 

---

DRPC_INVOCATIONS_PORT

public static java.lang.String DRPC_INVOCATIONS_PORT

This port on Storm DRPC is used by DRPC topologies to receive function invocations and send results back.

 

---

DRPC_REQUEST_TIMEOUT_SECS

public static java.lang.String DRPC_REQUEST_TIMEOUT_SECS

The timeout on DRPC requests within the DRPC server. Defaults to 10 minutes. Note that requests can also timeout based on the socket timeout on the DRPC client, and separately based on the topology message timeout for the topology implementing the DRPC function.

 

---

SUPERVISOR_SCHEDULER_META

public static java.lang.String SUPERVISOR_SCHEDULER_META

the metadata configed on the supervisor

 

---

SUPERVISOR_SLOTS_PORTS

public static java.lang.String SUPERVISOR_SLOTS_PORTS

A list of ports that can run workers on this supervisor. Each worker uses one port, and the supervisor will only run one worker per port. Use this configuration to tune how many workers run on each machine.

 

---

SUPERVISOR_CHILDOPTS

public static java.lang.String SUPERVISOR_CHILDOPTS

This parameter is used by the storm-deploy project to configure the jvm options for the supervisor daemon.

 

---

SUPERVISOR_WORKER_TIMEOUT_SECS

public static java.lang.String SUPERVISOR_WORKER_TIMEOUT_SECS

How long a worker can go without heartbeating before the supervisor tries to restart the worker process.

 

---

SUPERVISOR_WORKER_START_TIMEOUT_SECS

public static java.lang.String SUPERVISOR_WORKER_START_TIMEOUT_SECS

How long a worker can go without heartbeating during the initial launch before the supervisor tries to restart the worker process. This value override supervisor.worker.timeout.secs during launch because there is additional overhead to starting and configuring the JVM on launch.

 

---

SUPERVISOR_ENABLE

public static java.lang.String SUPERVISOR_ENABLE

Whether or not the supervisor should launch workers assigned to it. Defaults to true -- and you should probably never change this value. This configuration is used in the Storm unit tests.

 

---

SUPERVISOR_HEARTBEAT_FREQUENCY_SECS

public static java.lang.String SUPERVISOR_HEARTBEAT_FREQUENCY_SECS

how often the supervisor sends a heartbeat to the master.

 

---

SUPERVISOR_MONITOR_FREQUENCY_SECS

public static java.lang.String SUPERVISOR_MONITOR_FREQUENCY_SECS

How often the supervisor checks the worker heartbeats to see if any of them need to be restarted.

 

---

WORKER_CHILDOPTS

public static java.lang.String WORKER_CHILDOPTS

The jvm opts provided to workers launched by this supervisor. All "%ID%" substrings are replaced with an identifier for this worker.

 

---

WORKER_HEARTBEAT_FREQUENCY_SECS

public static java.lang.String WORKER_HEARTBEAT_FREQUENCY_SECS

How often this worker should heartbeat to the supervisor.

 

---

TASK_HEARTBEAT_FREQUENCY_SECS

public static java.lang.String TASK_HEARTBEAT_FREQUENCY_SECS

How often a task should heartbeat its status to the master.

 

---

TASK_REFRESH_POLL_SECS

public static java.lang.String TASK_REFRESH_POLL_SECS

How often a task should sync its connections with other tasks (if a task is reassigned, the other tasks sending messages to it need to refresh their connections). In general though, when a reassignment happens other tasks will be notified almost immediately. This configuration is here just in case that notification doesn't come through.

 

---

TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS

public static java.lang.String TOPOLOGY_ENABLE_MESSAGE_TIMEOUTS

True if Storm should timeout messages or not. Defaults to true. This is meant to be used in unit tests to prevent tuples from being accidentally timed out during the test.

 

---

TOPOLOGY_DEBUG

public static java.lang.String TOPOLOGY_DEBUG

When set to true, Storm will log every message that's emitted.

 

---

TOPOLOGY_OPTIMIZE

public static java.lang.String TOPOLOGY_OPTIMIZE

Whether or not the master should optimize topologies by running multiple tasks in a single thread where appropriate.

 

---

TOPOLOGY_WORKERS

public static java.lang.String TOPOLOGY_WORKERS

How many processes should be spawned around the cluster to execute this topology. Each process will execute some number of tasks as threads within them. This parameter should be used in conjunction with the parallelism hints on each component in the topology to tune the performance of a topology.

 

---

TOPOLOGY_TASKS

public static java.lang.String TOPOLOGY_TASKS

How many instances to create for a spout/bolt. A task runs on a thread with zero or more other tasks for the same spout/bolt. The number of tasks for a spout/bolt is always the same throughout the lifetime of a topology, but the number of executors (threads) for a spout/bolt can change over time. This allows a topology to scale to more or less resources without redeploying the topology or violating the constraints of Storm (such as a fields grouping guaranteeing that the same value goes to the same task).

 

---

TOPOLOGY_ACKER_EXECUTORS

public static java.lang.String TOPOLOGY_ACKER_EXECUTORS

How many executors to spawn for ackers.

If this is set to 0, then Storm will immediately ack tuples as soon as they come off the spout, effectively disabling reliability.

 

---

TOPOLOGY_MESSAGE_TIMEOUT_SECS

public static java.lang.String TOPOLOGY_MESSAGE_TIMEOUT_SECS

The maximum amount of time given to the topology to fully process a message emitted by a spout. If the message is not acked within this time frame, Storm will fail the message on the spout. Some spouts implementations will then replay the message at a later time.

 

---

TOPOLOGY_KRYO_REGISTER

public static java.lang.String TOPOLOGY_KRYO_REGISTER

A list of serialization registrations for Kryo ( http://code.google.com/p/kryo/ ), the underlying serialization framework for Storm. A serialization can either be the name of a class (in which case Kryo will automatically create a serializer for the class that saves all the object's fields), or an implementation of com.esotericsoftware.kryo.Serializer. See Kryo's documentation for more information about writing custom serializers.

 

---

TOPOLOGY_KRYO_DECORATORS

public static java.lang.String TOPOLOGY_KRYO_DECORATORS

A list of classes that customize storm's kryo instance during start-up. Each listed class name must implement IKryoDecorator. During start-up the listed class is instantiated with 0 arguments, then its 'decorate' method is called with storm's kryo instance as the only argument.

 

---

TOPOLOGY_SKIP_MISSING_KRYO_REGISTRATIONS

public static java.lang.String TOPOLOGY_SKIP_MISSING_KRYO_REGISTRATIONS

Whether or not Storm should skip the loading of kryo registrations for which it does not know the class or have the serializer implementation. Otherwise, the task will fail to load and will throw an error at runtime. The use case of this is if you want to declare your serializations on the storm.yaml files on the cluster rather than every single time you submit a topology. Different applications may use different serializations and so a single application may not have the code for the other serializers used by other apps. By setting this config to true, Storm will ignore that it doesn't have those other serializations rather than throw an error.

 

---

TOPOLOGY_MAX_TASK_PARALLELISM

public static java.lang.String TOPOLOGY_MAX_TASK_PARALLELISM

The maximum parallelism allowed for a component in this topology. This configuration is typically used in testing to limit the number of threads spawned in local mode.

 

---

TOPOLOGY_MAX_SPOUT_PENDING

public static java.lang.String TOPOLOGY_MAX_SPOUT_PENDING

The maximum number of tuples that can be pending on a spout task at any given time. This config applies to individual tasks, not to spouts or topologies as a whole. A pending tuple is one that has been emitted from a spout but has not been acked or failed yet. Note that this config parameter has no effect for unreliable spouts that don't tag their tuples with a message id.

 

---

TOPOLOGY_SPOUT_WAIT_STRATEGY

public static java.lang.String TOPOLOGY_SPOUT_WAIT_STRATEGY

A class that implements a strategy for what to do when a spout needs to wait. Waiting is triggered in one of two conditions: 1. nextTuple emits no tuples 2. The spout has hit maxSpoutPending and can't emit any more tuples

 

---

TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS

public static java.lang.String TOPOLOGY_SLEEP_SPOUT_WAIT_STRATEGY_TIME_MS

The amount of milliseconds the SleepEmptyEmitStrategy should sleep for.

 

---

TOPOLOGY_STATE_SYNCHRONIZATION_TIMEOUT_SECS

public static java.lang.String TOPOLOGY_STATE_SYNCHRONIZATION_TIMEOUT_SECS

The maximum amount of time a component gives a source of state to synchronize before it requests synchronization again.

 

---

TOPOLOGY_STATS_SAMPLE_RATE

public static java.lang.String TOPOLOGY_STATS_SAMPLE_RATE

The percentage of tuples to sample to produce stats for a task.

 

---

TOPOLOGY_FALL_BACK_ON_JAVA_SERIALIZATION

public static java.lang.String TOPOLOGY_FALL_BACK_ON_JAVA_SERIALIZATION

Whether or not to use Java serialization in a topology.

 

---

TOPOLOGY_WORKER_CHILDOPTS

public static java.lang.String TOPOLOGY_WORKER_CHILDOPTS

Topology-specific options for the worker child process. This is used in addition to WORKER_CHILDOPTS.

 

---

TOPOLOGY_TRANSACTIONAL_ID

public static java.lang.String TOPOLOGY_TRANSACTIONAL_ID

This config is available for TransactionalSpouts, and contains the id ( a String) for the transactional topology. This id is used to store the state of the transactional topology in Zookeeper.

 

---

TOPOLOGY_AUTO_TASK_HOOKS

public static java.lang.String TOPOLOGY_AUTO_TASK_HOOKS

A list of task hooks that are automatically added to every spout and bolt in the topology. An example of when you'd do this is to add a hook that integrates with your internal monitoring system. These hooks are instantiated using the zero-arg constructor.

 

---

TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE

public static java.lang.String TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE

The size of the Disruptor receive queue for each executor. Must be a power of 2.

 

---

TOPOLOGY_RECEIVER_BUFFER_SIZE

public static java.lang.String TOPOLOGY_RECEIVER_BUFFER_SIZE

The maximum number of messages to batch from the thread receiving off the network to the executor queues. Must be a power of 2.

 

---

TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE

public static java.lang.String TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE

The size of the Disruptor send queue for each executor. Must be a power of 2.

 

---

TOPOLOGY_TRANSFER_BUFFER_SIZE

public static java.lang.String TOPOLOGY_TRANSFER_BUFFER_SIZE

The size of the Disruptor transfer queue for each worker.

 

---

TOPOLOGY_TICK_TUPLE_FREQ_SECS

public static java.lang.String TOPOLOGY_TICK_TUPLE_FREQ_SECS

How often a tick tuple from the "__system" component and "__tick" stream should be sent to tasks. Meant to be used as a component-specific configuration.

 

---

TOPOLOGY_DISRUPTOR_WAIT_STRATEGY

public static java.lang.String TOPOLOGY_DISRUPTOR_WAIT_STRATEGY

Configure the wait strategy used for internal queuing. Can be used to tradeoff latency vs. throughput

 

---

TOPOLOGY_WORKER_SHARED_THREAD_POOL_SIZE

public static java.lang.String TOPOLOGY_WORKER_SHARED_THREAD_POOL_SIZE

The size of the shared thread pool for worker tasks to make use of. The thread pool can be accessed via the TopologyContext.

 

---

TOPOLOGY_NAME

public static java.lang.String TOPOLOGY_NAME

Name of the topology. This config is automatically set by Storm when the topology is submitted.

 

---

TRANSACTIONAL_ZOOKEEPER_ROOT

public static java.lang.String TRANSACTIONAL_ZOOKEEPER_ROOT

The root directory in ZooKeeper for metadata about TransactionalSpouts.

 

---

TRANSACTIONAL_ZOOKEEPER_SERVERS

public static java.lang.String TRANSACTIONAL_ZOOKEEPER_SERVERS

The list of zookeeper servers in which to keep the transactional state. If null (which is default), will use storm.zookeeper.servers

 

---

TRANSACTIONAL_ZOOKEEPER_PORT

public static java.lang.String TRANSACTIONAL_ZOOKEEPER_PORT

The port to use to connect to the transactional zookeeper servers. If null (which is default), will use storm.zookeeper.port

 

---

ZMQ_THREADS

public static java.lang.String ZMQ_THREADS

The number of threads that should be used by the zeromq context in each worker process.

 

---

ZMQ_LINGER_MILLIS

public static java.lang.String ZMQ_LINGER_MILLIS

How long a connection should retry sending messages to a target host when the connection is closed. This is an advanced configuration and can almost certainly be ignored.

 

---

JAVA_LIBRARY_PATH

public static java.lang.String JAVA_LIBRARY_PATH

This value is passed to spawned JVMs (e.g., Nimbus, Supervisor, and Workers) for the java.library.path value. java.library.path tells the JVM where to look for native libraries. It is necessary to set this config correctly since Storm uses the ZeroMQ and JZMQ native libs.

 

---

DEV_ZOOKEEPER_PATH

public static java.lang.String DEV_ZOOKEEPER_PATH

The path to use as the zookeeper dir when running a zookeeper server via "storm dev-zookeeper". This zookeeper instance is only intended for development; it is not a production grade zookeeper setup.

 

Constructor Detail

Config

public Config()

Method Detail