Package org.apache.hadoop.yarn.api.records

Class LogAggregationContext

java.lang.Object
org.apache.hadoop.yarn.api.records.LogAggregationContext
@Evolving @Public public abstract class LogAggregationContext extends Object
LogAggregationContext represents all of the information needed by the NodeManager to handle the logs for an application.

It includes details such as:

  • includePattern. It uses Java Regex to filter the log files which match the defined include pattern and those log files will be uploaded when the application finishes.
  • excludePattern. It uses Java Regex to filter the log files which match the defined exclude pattern and those log files will not be uploaded when application finishes. If the log file name matches both the include and the exclude pattern, this file will be excluded eventually.
  • rolledLogsIncludePattern. It uses Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion.
  • rolledLogsExcludePattern. It uses Java Regex to filter the log files which match the defined exclude pattern and those log files will not be aggregated in a rolling fashion. If the log file name matches both the include and the exclude pattern, this file will be excluded eventually.
  • policyClassName. The policy class name that implements ContainerLogAggregationPolicy. At runtime, nodemanager will the policy if a given container's log should be aggregated based on the ContainerType and other runtime state such as exit code by calling ContainerLogAggregationPolicy#shouldDoLogAggregation. This is useful when the app only wants to aggregate logs of a subset of containers. Here are the available policies. Please make sure to specify the canonical name by prefixing org.apache.hadoop.yarn.server. nodemanager.containermanager.logaggregation. to the class simple name below. NoneContainerLogAggregationPolicy: skip aggregation for all containers. AllContainerLogAggregationPolicy: aggregate all containers. AMOrFailedContainerLogAggregationPolicy: aggregate application master or failed containers. FailedOrKilledContainerLogAggregationPolicy: aggregate failed or killed containers FailedContainerLogAggregationPolicy: aggregate failed containers AMOnlyLogAggregationPolicy: aggregate application master containers SampleContainerLogAggregationPolicy: sample logs of successful worker containers, in addition to application master and failed/killed containers. LimitSizeContainerLogAggregationPolicy: skip aggregation for killed containers whose log size exceeds the limit of container log size. If it isn't specified, it will use the cluster-wide default policy defined by configuration yarn.nodemanager.log-aggregation.policy.class. The default value of yarn.nodemanager.log-aggregation.policy.class is AllContainerLogAggregationPolicy.
  • policyParameters. The parameters passed to the policy class via ContainerLogAggregationPolicy#parseParameters during the policy object initialization. This is optional. Some policy class might use parameters to adjust its settings. It is up to policy class to define the scheme of parameters. For example, SampleContainerLogAggregationPolicy supports the format of "SR:0.5,MIN:50", which means sample rate of 50% beyond the first 50 successful worker containers.
See Also:

  • Constructor Details

    • LogAggregationContext

      public LogAggregationContext()

  • Method Details

    • newInstance

      @Public @Unstable public static LogAggregationContext newInstance(String includePattern, String excludePattern)

    • newInstance

      @Public @Unstable public static LogAggregationContext newInstance(String includePattern, String excludePattern, String rolledLogsIncludePattern, String rolledLogsExcludePattern)

    • newInstance

      @Public @Unstable public static LogAggregationContext newInstance(String includePattern, String excludePattern, String rolledLogsIncludePattern, String rolledLogsExcludePattern, String policyClassName, String policyParameters)

    • getIncludePattern

      @Public @Unstable public abstract String getIncludePattern()
      Get include pattern. This includePattern only takes affect on logs that exist at the time of application finish.
      Returns:
      include pattern

    • setIncludePattern

      @Public @Unstable public abstract void setIncludePattern(String includePattern)
      Set include pattern. This includePattern only takes affect on logs that exist at the time of application finish.
      Parameters:
      includePattern - include pattern.

    • getExcludePattern

      @Public @Unstable public abstract String getExcludePattern()
      Get exclude pattern. This excludePattern only takes affect on logs that exist at the time of application finish.
      Returns:
      exclude pattern

    • setExcludePattern

      @Public @Unstable public abstract void setExcludePattern(String excludePattern)
      Set exclude pattern. This excludePattern only takes affect on logs that exist at the time of application finish.
      Parameters:
      excludePattern - exclude pattern.

    • getRolledLogsIncludePattern

      @Public @Unstable public abstract String getRolledLogsIncludePattern()
      Get include pattern in a rolling fashion.
      Returns:
      include pattern

    • setRolledLogsIncludePattern

      @Public @Unstable public abstract void setRolledLogsIncludePattern(String rolledLogsIncludePattern)
      Set include pattern in a rolling fashion.
      Parameters:
      rolledLogsIncludePattern - It uses Java Regex to filter the log files which match the defined include pattern and those log files will be aggregated in a rolling fashion.

    • getRolledLogsExcludePattern

      @Public @Unstable public abstract String getRolledLogsExcludePattern()
      Get exclude pattern for aggregation in a rolling fashion.
      Returns:
      exclude pattern

    • setRolledLogsExcludePattern

      @Public @Unstable public abstract void setRolledLogsExcludePattern(String rolledLogsExcludePattern)
      Set exclude pattern for in a rolling fashion.
      Parameters:
      rolledLogsExcludePattern - rolled logs exclude pattern.

    • getLogAggregationPolicyClassName

      @Public @Unstable public abstract String getLogAggregationPolicyClassName()
      Get the log aggregation policy class.
      Returns:
      log aggregation policy class

    • setLogAggregationPolicyClassName

      @Public @Unstable public abstract void setLogAggregationPolicyClassName(String className)
      Set the log aggregation policy class.
      Parameters:
      className - log aggregation policy class name.

    • getLogAggregationPolicyParameters

      @Public @Unstable public abstract String getLogAggregationPolicyParameters()
      Get the log aggregation policy parameters.
      Returns:
      log aggregation policy parameters

    • setLogAggregationPolicyParameters

      @Public @Unstable public abstract void setLogAggregationPolicyParameters(String parameters)
      Set the log aggregation policy parameters. There is no schema defined for the parameters string. It is up to the log aggregation policy class to decide how to parse the parameters string.
      Parameters:
      parameters - log aggregation policy parameters.