Testing the S3A filesystem client and its features

The submitter of any patch is required to run all the integration tests and declare which S3 region/implementation they used.

This is important: patches which do not include this declaration will be ignored

This policy has proven to be the only mechanism to guarantee full regression testing of code changes. Why the declaration of region? Two reasons:

1. It helps us identify regressions which only surface against specific endpoints or third-party implementations of the S3 protocol.
2. It forces the submitters to be more honest about their testing. It’s easy to lie, “yes, I tested this”. To say “yes, I tested this against S3 US-west” is a more specific lie and harder to make. And, if you get caught out: you lose all credibility with the project.

You don’t need to test from a VM within the AWS infrastructure; with the -Dparallel-tests option the non-scale tests complete in under twenty minutes. Because the tests clean up after themselves, they are also designed to be low cost. It’s neither hard nor expensive to run the tests; if you can’t, there’s no guarantee your patch works. The reviewers have enough to do, and don’t have the time to do these tests, especially as every failure will simply make for a slow iterative development.

Please: run the tests. And if you don’t, we are sorry for declining your patch, but we have to.

What if there’s an intermittent failure of a test?

Some of the tests do fail intermittently, especially in parallel runs. If this happens, try to run the test on its own to see if the test succeeds.

If it still fails, include this fact in your declaration. We know some tests are intermittently unreliable.

What if the tests are timing out or failing over my network connection?

The tests and the S3A client are designed to be configurable for different timeouts. If you are seeing problems and this configuration isn’t working, that’s a sign of the configuration mechanism isn’t complete. If it’s happening in the production code, that could be a sign of a problem which may surface over long-haul connections. Please help us identify and fix these problems — especially as you are the one best placed to verify the fixes work.

File core-site.xml

This file pre-exists and sources the configurations created under auth-keys.xml.

For most purposes you will not need to edit this file unless you need to apply a specific, non-default property change during the tests.

File auth-keys.xml

The presence of this file triggers the testing of the S3 classes.

Without this file, none of the integration tests in this module will be executed.

The XML file must contain all the ID/key information needed to connect each of the filesystem clients to the object stores, and a URL for each filesystem for its testing.

1. test.fs.s3a.name : the URL of the bucket for S3a tests
2. fs.contract.test.fs.s3a : the URL of the bucket for S3a filesystem contract tests

The contents of the bucket will be destroyed during the test process: do not use the bucket for any purpose other than testing. Furthermore, for s3a, all in-progress multi-part uploads to the bucket will be aborted at the start of a test (by forcing fs.s3a.multipart.purge=true) to clean up the temporary state of previously failed tests.

Example:

<configuration>

  <property>
    <name>test.fs.s3a.name</name>
    <value>s3a://test-aws-s3a/</value>
  </property>

  <property>
    <name>fs.contract.test.fs.s3a</name>
    <value>${test.fs.s3a.name}</value>
  </property>

  <property>
    <name>fs.s3a.access.key</name>
    <description>AWS access key ID. Omit for IAM role-based authentication.</description>
    <value>DONOTCOMMITTHISKEYTOSCM</value>
  </property>

  <property>
    <name>fs.s3a.secret.key</name>
    <description>AWS secret key. Omit for IAM role-based authentication.</description>
    <value>DONOTEVERSHARETHISSECRETKEY!</value>
  </property>

  <property>
    <name>test.sts.endpoint</name>
    <description>Specific endpoint to use for STS requests.</description>
    <value>sts.amazonaws.com</value>
  </property>

</configuration>

Default Encryption

Buckets can be configured with default encryption on the AWS side. Some S3AFileSystem tests are skipped when default encryption is enabled due to unpredictability in how ETags are generated.

Disabling the encryption tests

If the S3 store/storage class doesn’t support server-side-encryption, these will fail. They can be turned off.

<property>
  <name>test.fs.s3a.encryption.enabled</name>
  <value>false</value>
</property>

Encryption is only used for those specific test suites with Encryption in their classname.

Testing against different regions

S3A can connect to different regions —the tests support this. Simply define the target region in auth-keys.xml.

<property>
  <name>fs.s3a.endpoint.region</name>
  <value>eu-central-1</value>
</property>

CSV Data Tests

The TestS3AInputStreamPerformance tests require read access to a multi-MB text file. The default file for these tests is a public one. s3a://noaa-cors-pds/raw/2023/001/akse/AKSE001a.23_.gz from the NOAA Continuously Operating Reference Stations (CORS) Network (NCN)

Historically it was required to be a csv.gz file to validate S3 Select support. Now that S3 Select support has been removed, other large files may be used instead.

The path to this object is set in the option fs.s3a.scale.test.csvfile,

<property>
  <name>fs.s3a.scale.test.csvfile</name>
  <value>s3a://noaa-cors-pds/raw/2023/001/akse/AKSE001a.23_.gz</value>
</property>

1. If the option is not overridden, the default value is used. This is hosted in Amazon’s US-east datacenter.
2. If fs.s3a.scale.test.csvfile is empty, tests which require it will be skipped.
3. If the data cannot be read for any reason then the test will fail.
4. If the property is set to a different path, then that data must be readable and “sufficiently” large.
5. If a .gz file, expect decompression-related test failures.

(the reason the space or newline is needed is to add “an empty entry”; an empty <value/> would be considered undefined and pick up the default)

If using a test file in a different AWS S3 region then a bucket-specific region must be defined. For the default test dataset, hosted in the noaa-cors-pds bucket, this is:

  <property>
    <name>fs.s3a.bucket.noaa-cors-pds.endpoint.region</name>
    <value>us-east-1</value>
  </property>

Testing Access Point Integration

S3a supports using Access Point ARNs to access data in S3. If you think your changes affect VPC integration, request signing, ARN manipulation, or any code path that deals with the actual sending and retrieving of data to/from S3, make sure you run the entire integration test suite with this feature enabled.

Check out our documentation for steps on how to enable this feature. To create access points for your S3 bucket you can use the AWS Console or CLI.

Enabling the Scale Tests

The tests are enabled if the scale property is set in the maven build this can be done regardless of whether or not the parallel test profile is used

mvn verify -Dscale

mvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8

The most bandwidth intensive tests (those which upload data) always run sequentially; those which are slow due to HTTPS setup costs or server-side actions are included in the set of parallelized tests.

Tuning scale options from Maven

Some of the tests can be tuned from the maven build or from the configuration file used to run the tests.

mvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M

The algorithm is

1. The value is queried from the configuration file, using a default value if it is not set.
2. The value is queried from the JVM System Properties, where it is passed down by maven.
3. If the system property is null, an empty string, or it has the value unset, then the configuration value is used. The unset option is used to work round a quirk in maven property propagation.

Only a few properties can be set this way; more will be added.

The file and partition sizes are numeric values with a k/m/g/t/p suffix depending on the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.

Scale test configuration options

Some scale tests perform multiple operations (such as creating many directories).

The exact number of operations to perform is configurable in the option scale.test.operation.count

<property>
  <name>scale.test.operation.count</name>
  <value>10</value>
</property>

Larger values generate more load, and are recommended when testing locally, or in batch runs.

Smaller values results in faster test runs, especially when the object store is a long way away.

Operations which work on directories have a separate option: this controls the width and depth of tests creating recursive directories. Larger values create exponentially more directories, with consequent performance impact.

<property>
  <name>scale.test.directory.count</name>
  <value>2</value>
</property>

DistCp tests targeting S3A support a configurable file size. The default is 10 MB, but the configuration value is expressed in KB so that it can be tuned smaller to achieve faster test runs.

<property>
  <name>scale.test.distcp.file.size.kb</name>
  <value>10240</value>
</property>

S3A specific scale test properties are

fs.s3a.scale.test.huge.filesize: size in MB for “Huge file tests”.

The Huge File tests validate S3A’s ability to handle large files —the property fs.s3a.scale.test.huge.filesize declares the file size to use.

<property>
  <name>fs.s3a.scale.test.huge.filesize</name>
  <value>200M</value>
</property>

Amazon S3 handles files larger than 5GB differently than smaller ones. Setting the huge filesize to a number greater than that) validates support for huge files.

<property>
  <name>fs.s3a.scale.test.huge.filesize</name>
  <value>6G</value>
</property>

Tests at this scale are slow: they are best executed from hosts running in the cloud infrastructure where the S3 endpoint is based. Otherwise, set a large timeout in fs.s3a.scale.test.timeout

<property>
  <name>fs.s3a.scale.test.timeout</name>
  <value>432000</value>
</property>

The tests are executed in an order to only clean up created files after the end of all the tests. If the tests are interrupted, the test data will remain.

Parallel CI builds.

For CI testing of the module, including the integration tests, it is generally necessary to support testing multiple PRs simultaneously.

To do this 1. A job ID must be supplied in the job.id property, so each job works on an isolated directory tree. This should be a number or unique string, which will be used within a path element, so must only contain characters valid in an S3/hadoop path element. 2. Root directory tests need to be disabled by setting fs.s3a.root.tests.enabled to false, either in the command line to maven or in the XML configurations.

mvn verify -T 1C -Dparallel-tests -DtestsThreadCount=14 -Dscale -Dfs.s3a.root.tests.enabled=false -Djob.id=001

This parallel execution feature is only for isolated builds sharing a single S3 bucket; it does not support parallel builds and tests from the same local source tree.

Without the root tests being executed, set up a scheduled job to purge the test bucket of all data on a regular basis, to keep costs down. The easiest way to do this is to have a bucket lifecycle rule for the bucket to delete all files more than a few days old, alongside one to abort all pending uploads more than 24h old.

Securing CI builds

It’s clearly unsafe to have CI infrastructure testing PRs submitted to apache github account with AWS credentials -which is why it isn’t done by the Yetus-initiated builds.

Anyone doing this privately should: * Review incoming patches before triggering the tests. * Have a dedicated IAM role with restricted access to the test bucket, any KMS keys used, and the external bucket containing the CSV test file. * Have a build process which generates short-lived session credentials for this role. * Run the tests in an EC2 VM/container which collects the restricted IAM credentials from the IAM instance/container credentials provider.

Public datasets used in tests

Some tests rely on the presence of existing public datasets available on Amazon S3. You may find a number of these in org.apache.hadoop.fs.s3a.test.PublicDatasetTestUtils.

When testing against an endpoint which is not part of Amazon S3’s standard commercial partition (aws) such as third-party implementations or AWS’s China regions, you should replace these configurations with an empty space () to disable the tests or an existing path in your object store that supports these tests.

An example of this might be the MarkerTools tests which require a bucket with a large number of objects or the requester pays tests that require requester pays to be enabled for the bucket.

Disabling the storage class tests

When running storage class tests against third party object store that doesn’t support S3 storage class, these tests might fail. They can be disabled.

<property>
  <name>test.fs.s3a.create.storage.class.enabled</name>
  <value>false</value>
</property>

Configuring the CSV file read tests

To test on alternate infrastructures supporting the same APIs, the option fs.s3a.scale.test.csvfile must either be set to " ", or an object of at least 10MB is uploaded to the object store, and the fs.s3a.scale.test.csvfile option set to its path.

<property>
  <name>fs.s3a.scale.test.csvfile</name>
  <value> </value>
</property>

(yes, the space is necessary. The Hadoop Configuration class treats an empty value as “do not override the default”).

Enabling prefetch for all tests

The tests are run with prefetch if the prefetch property is set in the maven build. This can be combined with the scale tests as well.

If ITestS3AContractGetFileStatusV1List fails with any error about unsupported API.

  <property>
    <name>test.fs.s3a.list.v1.enabled</name>
    <value>false</value>
  </property>

Note: there’s no equivalent for turning off v2 listing API, which all stores are now required to support.

Testing Requester Pays

By default, the requester pays tests will look for a bucket that exists on Amazon S3 in us-east-1.

If the endpoint does support requester pays, you can specify an alternative object. The test only requires an object of at least a few bytes in order to check that lists and basic reads work.

<property>
  <name>test.fs.s3a.requester.pays.file</name>
  <value>s3a://my-req-pays-enabled-bucket/on-another-endpoint.json</value>
</property>

If the endpoint does not support requester pays, you can also disable the tests by configuring the test URI as a single space.

<property>
  <name>test.fs.s3a.requester.pays.file</name>
  <value> </value>
</property>

Testing Session Credentials

Some tests requests a session credentials and assumed role credentials from the AWS Secure Token Service, then use them to authenticate with S3 either directly or via delegation tokens.

If an S3 implementation does not support STS, then these functional test cases must be disabled:

<property>
  <name>test.fs.s3a.sts.enabled</name>
  <value>false</value>
</property>

These tests request a temporary set of credentials from the STS service endpoint. An alternate endpoint may be defined in fs.s3a.assumed.role.sts.endpoint. If this is set, a delegation token region must also be defined: in fs.s3a.assumed.role.sts.endpoint.region. This is useful not just for testing alternative infrastructures, but to reduce latency on tests executed away from the central service.

<property>
  <name>fs.s3a.delegation.token.endpoint</name>
  <value>fs.s3a.assumed.role.sts.endpoint</value>
</property>
<property>
  <name>fs.s3a.assumed.role.sts.endpoint.region</name>
  <value>eu-west-2</value>
</property>

The default is ""; meaning “use the amazon default endpoint” (sts.amazonaws.com).

Consult the AWS documentation for the full list of locations.

Disabling Content Encoding tests

Tests in ITestS3AContentEncoding may need disabling

  <property>
    <name>test.fs.s3a.content.encoding.enabled</name>
    <value>false</value>
  </property>

Disabling tests running in performance mode

Some tests running in performance mode turn off the safety checks. They expect operations which break POSIX semantics to succeed. For stores with stricter semantics, these test cases must be disabled.

  <property>
    <name>test.fs.s3a.performance.enabled</name>
    <value>false</value>
  </property>

Changing expectations on multipart upload retries: ITestS3AContractMultipartUploader and ITestUploadRecovery

If the store reports errors when trying to list/abort completed multipart uploads, expect failures in ITestUploadRecovery and ITestS3AContractMultipartUploader. The tests can be reconfigured to expect failure by setting the option fs.s3a.ext.test.multipart.commit.consumes.upload.id to true.

Note how this can be set as a per-bucket option.

  <property>
    <name>fs.s3a.ext.test.multipart.commit.consumes.upload.id</name>
    <value>true</value>
  </property>

Tests which may fail (and which you can ignore)

• ITestS3AMiscOperations.testEmptyFileChecksums: if the FS encrypts data always.

Subclasses Existing Shared Base Classes

Extend AbstractS3ATestBase or AbstractSTestS3AHugeFiles unless justifiable. These set things up for testing against the object stores, provide good threadnames, help generate isolated paths, and for AbstractSTestS3AHugeFiles subclasses, only run if -Dscale is set.

Key features of AbstractS3ATestBase

• getFileSystem() returns the S3A Filesystem bonded to the contract test Filesystem defined in fs.s3a.contract.test
• will automatically skip all tests if that URL is unset.
• Extends AbstractFSContractTestBase
• Uses AssertJ for all assertions, not those of JUnit5.

Having shared base classes may help reduce future maintenance too. Please use them.

We adopted AssertJ assertions long before the move to JUnit5. While there are still many tests with legacy JUnit 1.x assertions, all new test cases should use AssertJ assertions and MUST NOT use JUnit5.

Secure

Don’t ever log credentials. The credential tests go out of their way to not provide meaningful logs or assertion messages precisely to avoid this.

Efficient of Time and Money

This means efficient in test setup/teardown, and, ideally, making use of existing public datasets to save setup time and tester cost.

Strategies of particular note are:

1. ITestS3ADirectoryPerformance: a single test case sets up the directory tree then performs different list operations, measuring the time taken.
2. AbstractSTestS3AHugeFiles: marks the test suite as @FixMethodOrder(MethodSorters.NAME_ASCENDING) then orders the test cases such that each test case expects the previous test to have completed (here: uploaded a file, renamed a file, …). This provides for independent tests in the reports, yet still permits an ordered sequence of operations. Do note the use of Assume.assume() to detect when the preconditions for a single test case are not met, hence, the tests become skipped, rather than fail with a trace which is really a false alarm.

The ordered test case mechanism of AbstractSTestS3AHugeFiles is probably the most elegant way of chaining test setup/teardown.

Regarding reusing existing data, we tend to use the noaa-cors-pds archive of AWS US-East for our testing of input stream operations. This doesn’t work against other regions, or with third party S3 implementations. Thus the URL can be overridden for testing elsewhere.

Works With Other S3 Stores

Don’t assume AWS S3 US-East only, do allow for working with external S3 implementations. Those may be behind the latest S3 API features, not support encryption, session APIs, etc.

They won’t have the same CSV/large test files as some of the input tests rely on. Look at ITestS3AInputStreamPerformance to see how tests can be written to support the declaration of a specific large test file on alternate filesystems.

Works Over Long-haul Links

As well as making file size and operation counts scalable, this includes making test timeouts adequate. The Scale tests make this configurable; it’s hard coded to ten minutes in AbstractS3ATestBase(); subclasses can change this by overriding getTestTimeoutMillis().

Equally importantly: support proxies, as some testers need them.

Provides Diagnostics and timing information

1. Give threads useful names.
2. Create logs, log things. Know that the S3AFileSystem and its input and output streams all provide useful statistics in their {{toString()}} calls; logging them is useful on its own.
3. you can use AbstractS3ATestBase.describe(format-stringm, args) here.; it adds some newlines so as to be easier to spot.
4. Use ContractTestUtils.NanoTimer to measure the duration of operations, and log the output.

Fails Meaningfully

The ContractTestUtils class contains a whole set of assertions for making statements about the expected state of a filesystem, e.g. assertPathExists(FS, path), assertPathDoesNotExists(FS, path), and others. These do their best to provide meaningful diagnostics on failures (e.g. directory listings, file status, …), so help make failures easier to understand.

At the very least, do not use assertTrue() or assertFalse() without including error messages.

Sets up its filesystem and checks for those settings

Tests can overrun createConfiguration() to add new options to the configuration file for the S3A Filesystem instance used in their tests.

However, filesystem caching may mean that a test suite may get a cached instance created with an different configuration. For tests which don’t need specific configurations caching is good: it reduces test setup time.

For those tests which do need unique options (encryption, magic files), things can break, and they will do so in hard-to-replicate ways.

Use S3ATestUtils.disableFilesystemCaching(conf) to disable caching when modifying the config. As an example from AbstractTestS3AEncryption:

@Override
protected Configuration createConfiguration() {
  Configuration conf = super.createConfiguration();
  S3ATestUtils.disableFilesystemCaching(conf);
  removeBaseAndBucketOverrides(conf,
      SERVER_SIDE_ENCRYPTION_ALGORITHM);  
  conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,
          getSSEAlgorithm().getMethod());
  return conf;
}

Then verify in the setup method or test cases that their filesystem actually has the desired feature (fs.getConf().getProperty(...)). This not only catches filesystem reuse problems, it catches the situation where the filesystem configuration in auth-keys.xml has explicit per-bucket settings which override the test suite’s general option settings.

Cleans Up Afterwards

Keeps costs down.

1. Do not only cleanup if a test case completes successfully; test suite teardown must do it.
2. That teardown code must check for the filesystem and other fields being null before the cleanup. Why? If test setup fails, the teardown methods still get called.

Works Reliably

We really appreciate this — you will too.

Runs in parallel unless this is unworkable.

Tests must be designed to run in parallel with other tests, all working with the same shared S3 bucket. This means

• Uses relative and JVM-fork-unique paths provided by the method AbstractFSContractTestBase.path(String filepath).
• Doesn’t manipulate the root directory or make assertions about its contents (for example: delete its contents and assert that it is now empty).
• Doesn’t have a specific requirement of all active clients of the bucket (example: SSE-C tests which require all files, even directory markers, to be encrypted with the same key).
• Doesn’t use so much bandwidth that all other tests will be starved of IO and start timing out (e.g. the scale tests).

Tests such as these can only be run as sequential tests. When adding one, exclude it in the POM file. from the parallel failsafe run and add to the sequential one afterwards. The IO heavy ones must also be subclasses of S3AScaleTestBase and so only run if the system/maven property fs.s3a.scale.test.enabled is true.

Individual test cases can be run in an IDE

This is invaluable for debugging test failures.

How to set test options in your hadoop configuration rather than on the maven command line:

Keeping AWS Costs down

Most of the base S3 tests are designed delete files after test runs, so you don’t have to pay for storage costs. The scale tests do work with more data so will cost more as well as generally take more time to execute.

You are however billed for

1. Data left in S3 after test runs.
2. HTTP operations on files (HEAD, LIST, GET).
3. In-progress multipart uploads from bulk IO or S3A committer tests.
4. Encryption/decryption using AWS KMS keys.

The GET/decrypt costs are incurred on each partial read of a file, so random IO can cost more than sequential IO; the speedup of queries with columnar data usually justifies this.

How to keep costs down

• Don’t run the scale tests with large datasets; keep fs.s3a.scale.test.huge.filesize unset, or a few MB (minimum: 5).
• Remove all files in the filesystem. The root tests usually do this, but it can be manually done:

  *hadoop fs -rm -r -f -skipTrash s3a://test-bucket/

• Abort all outstanding uploads:

  hadoop s3guard uploads -abort -force s3a://test-bucket/

How to keep your credentials really safe

Although the auth-keys.xml file is marked as ignored in git and subversion, it is still in your source tree, and there’s always that risk that it may creep out.

You can avoid this by keeping your keys outside the source tree and using an absolute XInclude reference to it.

<configuration>

  <include xmlns="http://www.w3.org/2001/XInclude"
    href="file:///users/ubuntu/.auth-keys.xml" />

</configuration>