Layer 2 — application architecture · 1,820 documents
| ID | Document | Source |
|---|---|---|
| app_invariants__7279cdc3c7786cc2 | INVARIANT: Atomic Offset Rollback on Commit Failure SHADOW: When an offset commit fails or a task pre-commit throws an exception, the system MUST rewind all assigned partitions to their last known… | — |
| app_invariants__124693dbdae62a5e | INVARIANT: Generational Zombie Fencing SHADOW: The system enforces distributed write-integrity by fencing producers from previous task generations using the Admin client. This neutralizes 'zombie'… | — |
| app_invariants__2fcfb64bbee80b0c | INVARIANT: Deterministic Transactional ID Derivation SHADOW: For Exactly-Once (EOS) source tasks, Transactional IDs MUST be deterministically mapped to the (GroupId, ConnectorName, TaskId) tuple.… | — |
| app_invariants__e52c9e018fe564b5 | INVARIANT: Strict Task-Count Quota Enforcement SHADOW: The system prevents resource exhaustion by validating that a connector cannot generate more task configurations than the 'tasks.max' limit.… | — |
| app_invariants__4d84ad22a08c1923 | INVARIANT: Synchronous Lifecycle Barrier SHADOW: Stopping a connector or task MUST invoke a synchronous 'Stop-and-Await' barrier. This ensures that all underlying resources (consumers, producers,… | — |
| app_invariants__870b943886a63661 | INVARIANT: Plugin Classloader Isolation SHADOW: Every plugin (Converter, Transformation, Connector) MUST be wrapped in a specific classloader barrier. This prevents 'Dependency Hell' where different… | — |
| app_invariants__28d44162c127e85a | INVARIANT: Monotonic Configuration Catch-up Enforced via Config Mismatch SHADOW: Workers are strictly forbidden from executing assignments if their local configuration snapshot is behind the… | — |
| app_invariants__4aad8c8d9fecafbd | INVARIANT: Zombie Fencing Precedence SHADOW: Physical and logical zombie fencing (epoch/transactional ID validation) must reach a finalized state before a source task is permitted to instantiate or… | — |
| app_invariants__80762049b75cd10b | INVARIANT: Hysteresis-Based Assignment Retention SHADOW: The system mandates a scheduled rebalance delay during member disconnects. Lost assignments are buffered rather than immediately… | — |
| app_invariants__79636dd8e74a1d5a | INVARIANT: Temporal Priority Queueing for Internal Requests SHADOW: The Internal Herder request queue must sort actions by a temporal delay factor. This ensures that exponential backoff retries… | — |
| app_invariants__e57f8f2d2c997e9e | INVARIANT: Leader-Exclusive Cryptographic Authority SHADOW: Only the elected leader is permitted to rotate session keys and write them to the configuration store. This centralization ensures a… | — |
| app_invariants__5f532728e0189d66 | INVARIANT: Parallel Classloader Deadlock Neutralization SHADOW: Isolation boundaries must utilize parallel-capable classloaders that decouple class-loading locks from object monitor locks. This is… | — |
| app_invariants__ba470f3b6090cac8 | INVARIANT: Only the elected leader worker can perform operations requiring transactional write privileges to the cluster's internal configuration topic. SHADOW: This constraint prevents conflicting… | — |
| app_invariants__d36bb87077201676 | INVARIANT: The internal configuration topic must be configured with exactly one partition. SHADOW: This strict partitioning enforces a total ordering of all configuration changes across the… | — |
| app_invariants__2da721a771477358 | INVARIANT: Internal consumers reading the configuration topic must operate with a 'read_committed' isolation level. SHADOW: This guarantees that workers only observe fully committed configuration… | — |
| app_invariants__16216e9ee01991f9 | INVARIANT: Connector source offsets must be durably written to a persistent Kafka topic, with successful write acknowledgements ensuring subsequent reads reflect the latest committed state. SHADOW:… | — |
| app_invariants__6b13c90b3f35fb2c | INVARIANT: When a composite offset backing store is used, a flush operation must guarantee that writes to the primary offset store succeed for the overall flush to commit. The failure of… | — |
| app_invariants__19a2189e403e5972 | INVARIANT: Status updates for connectors and tasks must be processed by prioritizing messages from the latest known generation, suppressing older or invalid updates. SHADOW: This prevents workers… | — |
| app_invariants__296bad6f6bb23ba9 | INVARIANT: The system must strictly adhere to the configured error tolerance policy, causing processing to halt for any error if tolerance is set to 'NONE', or allowing continued processing with… | — |
| app_invariants__b13a3d8cba14a4de | INVARIANT: Retriable exceptions must trigger an exponential backoff mechanism with a configurable maximum delay. SHADOW: This prevents resource exhaustion and system instability during transient… | — |
| app_invariants__8732d3e133e9759a | INVARIANT: When a Dead Letter Queue is enabled, the reporting of errant records to the DLQ topic must be an atomic operation, ensuring that either the record and its context headers are successfully… | — |
| app_invariants__d8a630926ef53ded | INVARIANT: All REST API endpoints must respond within configured timeouts, accurately reflecting the underlying operational state (e.g., 'Service Unavailable' during startup, 'Internal Server Error'… | — |
| app_invariants__bfb31d9347edf058 | INVARIANT: Snapshots of the cluster's configuration state must be immutable, read-only representations. SHADOW: This architectural rule protects the integrity of the live, internal configuration… | — |
| app_invariants__681b310eaefcf25d | INVARIANT: Plugin-specific classes must be loaded using an isolated and delegated classloader strategy, prioritizing plugin paths over the application's default classpath. SHADOW: This prevents… | — |
| app_invariants__a9c1dc9067f97b27 | INVARIANT: Connectors and tasks must support explicit state transitions (RUNNING, PAUSED, STOPPED) and adhere to graceful shutdown procedures, even when external dependencies are… | — |
| app_invariants__216380c7bb9c36ec | INVARIANT: For exactly-once source connectors, each task's Kafka producer must be configured with a unique transactional ID derived from the cluster's group ID and the task's identifier. SHADOW:… | — |
| app_invariants__13cd620b16b7e065 | INVARIANT: Source tasks must be prevented from producing records to their own dedicated offsets topic. SHADOW: This rule avoids a critical deadlock scenario where a task attempts to read its offsets… | — |
| app_invariants__422677cc2e574dd5 | INVARIANT: Task execution must not cause unbounded resource exhaustion on the Connect cluster or Kafka brokers. SHADOW: Connect workers must maintain stability and responsiveness under high load.… | — |
| app_invariants__f4504ba49e1d38ed | INVARIANT: Task state must be consistently managed and persisted to allow for restarts and failure recovery. SHADOW: Tasks are responsible for tracking their progress (e.g., Kafka offsets) and… | — |
| app_invariants__13314cfa5fbe1a21 | INVARIANT: Record processing and offset commits must be atomic from the perspective of a task's lifecycle or a distributed transaction. SHADOW: Either a set of records is processed and their offsets… | — |
| app_invariants__c704662ad2181e9c | INVARIANT: The system must tolerate retry attempts of record processing or task operations without adverse side effects. SHADOW: If an operation (like producing a record or committing an offset)… | — |
| app_invariants__300e9312f3f8e119 | INVARIANT: Distributed coordination for connector and task assignments must be robust against worker failures and network partitions. SHADOW: In a distributed Connect cluster, workers need to agree… | — |
| app_invariants__e465ef4e861d4607 | INVARIANT: The cached offset synchronization data for a TopicPartition must maintain an exponentially spaced and ordered relationship of upstreamOffset values, enforced by mathematical… | — |
| app_invariants__d10e0e8e8c5100bc | INVARIANT: The number of concurrently executing OffsetSync write operations to the internal offset-syncs topic must not exceed a predefined maximum threshold. SHADOW: This limit prevents a single… | — |
| app_invariants__d3491ddf257ddad1 | INVARIANT: An OffsetSync record for a TopicPartition must be queued for immediate transmission if the last known downstream offset for that partition is deemed sufficiently stale, if an upstream… | — |
| app_invariants__a86c32eae8e68968 | INVARIANT: The CheckpointStore must either successfully load its entire state from the backing Kafka log, or it must clear all partial state and report failure. Partial, inconsistent state from the… | — |
| app_invariants__7ced5b8c8e2e7734 | INVARIANT: Exclusive access to the internal Kafka consumer instance must be enforced, allowing only one thread to interact with it at any given time. SHADOW: This prevents race conditions and… | — |
| app_invariants__330c63a21ab591d5 | INVARIANT: Upon initialization, the consumer for a task must seek to 'offset + 1' for any TopicPartition with a previously committed valid offset. For partitions without a committed offset, no seek… | — |
| app_invariants__7ac94c8310c1b975 | INVARIANT: Following successful replication of a SourceRecord to the target cluster, the corresponding upstream and downstream offsets must be immediately queued for OffsetSync emission. SHADOW:… | — |
| app_invariants__a534d2d60c0a56e6 | INVARIANT: Only topics explicitly permitted by the topicFilter and not identified as internal MirrorMaker topics or replication cycles must be considered for checkpointing. SHADOW: This prevents the… | — |
| app_invariants__52157d22c25fc0f2 | INVARIANT: A new checkpoint for a consumer group and topic partition must only be emitted if its upstream offset is less than the last known upstream offset (indicating a rewind) or if its downstream… | — |
| app_invariants__1d4532073f4b5530 | INVARIANT: Offsets for consumer groups must only be synced to the target cluster if the group is currently 'EMPTY' (idle) or 'DEAD' (non-existent at target), preventing interference with active… | — |
| app_invariants__2a236a126f9c089c | INVARIANT: When syncing consumer group offsets, a translated upstream offset must only update the target consumer group's offset if it is strictly greater than the current downstream offset,… | — |
| app_invariants__37d0f35435422cc6 | INVARIANT: Topics must only be considered for replication if they pass the configured TopicFilter, are not internal MirrorMaker topics, and do not form a replication cycle back to the source… | — |
| app_invariants__cbe411bb70dece26 | INVARIANT: ACL bindings of type 'ALLOW WRITE' must not be replicated to the target cluster. SHADOW: This prevents unintended privilege escalation on the target cluster. Replicating 'ALLOW WRITE'… | — |
| app_invariants__1f9aeccc6f6d81a9 | INVARIANT: If offset synchronization is enabled, the internal offset-syncs topic must be created with a single partition and a compaction cleanup policy. SHADOW: A single partition ensures total… | — |
| app_invariants__1c8443ad5a49e7d8 | INVARIANT: If exactly-once support is 'required', the source consumer must be configured with 'isolation.level=read_committed'. SHADOW: This is a direct enforcement of transactional guarantees.… | — |
| app_invariants__e103b17fbbdcf03b | INVARIANT: User-supplied offsets for 'alterOffsets' must conform to the expected internal SourcePartition and Offset object structure, with specific keys and appropriate data types. SHADOW: This… | — |
| app_invariants__07141d9e9ad650d5 | INVARIANT: Upon becoming the group leader, a MirrorHerder instance must ensure that all required MirrorMaker connectors (MirrorSourceConnector, MirrorHeartbeatConnector, MirrorCheckpointConnector)… | — |
| app_invariants__9c2ef83c7a7c660c | INVARIANT: Schema definition fields are write-once during construction. SHADOW: The SchemaBuilder enforces strict immutability of fields like 'optional', 'version', and 'name' once assigned. This… | — |