Description

A submitted topology carries two lists of blobstore keys, `dependency_jars` and `dependency_artifacts`, which the client fills in after uploading the corresponding blobs. Nimbus performed no validation of their contents on the submission path, yet acts on them in two places.

During cleanup of a finished topology, Nimbus deletes the keys named in those lists, and the deletion is performed as the Nimbus subject, for which the blobstore short-circuits its ACL check. A submitter who listed a key belonging to another topology, such as its `-stormjar.jar`, could therefore cause that blob to be deleted when their own topology was cleaned up.

Separately, on acquiring leadership a Nimbus compares the dependency keys of all active topologies against the blobstore contents and surrenders leadership if any is missing. A single key that does not exist, on a single active topology, therefore causes every Nimbus to acquire leadership, surrender it and requeue indefinitely, leaving the cluster without a leader and unable to schedule, clean up or accept submissions.

Mitigation

Upgrade to 3.1.0, where a submission is refused unless every entry in both lists is a dependency blob key and exists in the blobstore.

Note that this validates new submissions only; a topology stored by an affected version with an invalid list is unaffected by the upgrade. An operator whose cluster is failing to retain a leader should inspect the Nimbus log for the dependency keys reported as missing…

Description

A submitted topology carries two lists of blobstore keys, `dependency_jars` and `dependency_artifacts`, which the client fills in after uploading the corresponding blobs. Nimbus performed no validation of their contents on the submission path, yet acts on them in two places.

During cleanup of a finished topology, Nimbus deletes the keys named in those lists, and the deletion is performed as the Nimbus subject, for which the blobstore short-circuits its ACL check. A submitter who listed a key belonging to another topology, such as its `-stormjar.jar`, could therefore cause that blob to be deleted when their own topology was cleaned up.

Separately, on acquiring leadership a Nimbus compares the dependency keys of all active topologies against the blobstore contents and surrenders leadership if any is missing. A single key that does not exist, on a single active topology, therefore causes every Nimbus to acquire leadership, surrender it and requeue indefinitely, leaving the cluster without a leader and unable to schedule, clean up or accept submissions.

Mitigation

Upgrade to 3.1.0, where a submission is refused unless every entry in both lists is a dependency blob key and exists in the blobstore.

Note that this validates new submissions only; a topology stored by an affected version with an invalid list is unaffected by the upgrade. An operator whose cluster is failing to retain a leader should inspect the Nimbus log for the dependency keys reported as missing…