Sep 18 16:04:58.782528 ip-10-0-132-41 systemd[1]: Starting Kubernetes Kubelet... Sep 18 16:05:02.863211 ip-10-0-132-41 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:05:02.863211 ip-10-0-132-41 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 16:05:02.863211 ip-10-0-132-41 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:05:02.863211 ip-10-0-132-41 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 16:05:02.863211 ip-10-0-132-41 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 16:05:02.864274 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.863289 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 16:05:02.868385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868360 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:05:02.868385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868380 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:05:02.868385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868384 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:05:02.868385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868388 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:05:02.868385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868392 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:05:02.868385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868396 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868401 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868404 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868407 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868410 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868413 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868417 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868420 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868423 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868426 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868429 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868432 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868435 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868438 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868441 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868447 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868450 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868452 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868455 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868458 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:05:02.868624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868461 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868463 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868466 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868469 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868472 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868474 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868479 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868483 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868486 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868489 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868492 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868495 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868500 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868504 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868508 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868511 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868514 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868517 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868520 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:05:02.869132 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868523 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868526 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868528 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868532 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868535 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868537 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868540 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868543 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868546 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868549 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868552 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868554 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868557 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868560 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868562 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868565 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868568 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868571 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868573 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:05:02.869664 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868578 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868582 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868585 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868588 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868591 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868594 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868598 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868601 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868604 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868607 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868610 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868613 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868615 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868618 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868621 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868624 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868627 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868630 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868633 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868636 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:05:02.870233 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868640 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868643 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868646 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868649 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868653 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868656 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868659 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868662 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868665 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868668 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868671 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868674 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868677 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868680 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868682 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.868685 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869485 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869494 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869497 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:05:02.870751 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869500 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869506 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869509 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869512 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869515 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869520 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869524 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869527 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869531 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869534 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869537 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869540 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869543 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869547 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869550 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869554 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869557 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869559 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869562 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:05:02.871271 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869565 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869568 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869571 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869574 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869577 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869580 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869583 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869587 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869590 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869593 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869596 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869599 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869601 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869604 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869607 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869610 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869613 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869616 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869619 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869622 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:05:02.871822 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869625 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869627 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869630 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869634 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869636 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869639 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869642 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869645 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869648 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869651 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869654 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869657 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869660 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869664 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869666 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869669 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869672 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869675 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869678 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869681 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:05:02.872375 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869684 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869686 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869690 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869692 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869697 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869700 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869705 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869708 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869711 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869713 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869716 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869719 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869722 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869724 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869727 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869730 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869733 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869736 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869739 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:05:02.872898 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869742 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869745 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869752 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869755 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869758 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869761 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869764 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869766 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869769 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869772 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869775 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869778 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869780 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869783 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869786 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869789 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869791 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.869794 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870209 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:05:02.873420 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870215 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870218 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870221 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870224 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870227 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870230 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870234 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870237 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870239 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870244 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870249 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870252 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870255 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870258 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870261 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870265 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870268 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870271 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870274 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:05:02.873926 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870277 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870280 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870284 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870286 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870290 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870293 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870295 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870298 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870301 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870304 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870307 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870310 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870313 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870322 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870325 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870327 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870330 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870333 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870336 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870338 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:05:02.874441 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870341 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870344 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870346 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870349 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870352 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870355 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870357 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870360 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870363 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870366 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870369 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870372 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870374 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870377 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870380 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870383 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870388 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870391 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870393 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870396 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870399 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:05:02.874959 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870402 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870405 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870408 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870411 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870413 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870416 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870419 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870422 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870425 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870428 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870431 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870434 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870437 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870441 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870445 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870448 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870451 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870454 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870459 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:05:02.875539 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870463 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870466 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870469 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870472 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870475 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870479 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870482 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870484 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870488 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870491 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870494 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870497 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870499 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870502 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870505 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870508 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870511 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870514 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.870516 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870595 2563 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 16:05:02.876046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870633 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870641 2563 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870646 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870650 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870654 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870658 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870663 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870666 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870669 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870673 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870677 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870681 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870684 2563 flags.go:64] FLAG: --cgroup-root="" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870687 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870691 2563 flags.go:64] FLAG: --client-ca-file="" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870694 2563 flags.go:64] FLAG: --cloud-provider="external" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870698 2563 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870702 2563 flags.go:64] FLAG: --cluster-domain="" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870705 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870709 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870714 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870718 2563 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870722 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 16:05:02.876611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870725 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870728 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870732 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870735 2563 flags.go:64] FLAG: --contention-profiling="false" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870738 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870742 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870745 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870748 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870753 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870757 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870760 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870763 2563 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870767 2563 flags.go:64] FLAG: --enable-server="true" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870770 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870775 2563 flags.go:64] FLAG: --event-burst="100" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870778 2563 flags.go:64] FLAG: --event-qps="50" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870781 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870784 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870787 2563 flags.go:64] FLAG: --eviction-hard="" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870792 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870795 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870798 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870801 2563 flags.go:64] FLAG: --eviction-soft="" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870804 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870807 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 16:05:02.877281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870811 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870814 2563 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870817 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870820 2563 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870826 2563 flags.go:64] FLAG: --feature-gates="" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870830 2563 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870833 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870836 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870840 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870845 2563 flags.go:64] FLAG: --healthz-port="10248" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870848 2563 flags.go:64] FLAG: --help="false" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870851 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-132-41.ec2.internal" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870854 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870858 2563 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870861 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870865 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870869 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870872 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870875 2563 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870878 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870882 2563 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870885 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870888 2563 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870891 2563 flags.go:64] FLAG: --kube-reserved="" Sep 18 16:05:02.877948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870894 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870897 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870900 2563 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870904 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870907 2563 flags.go:64] FLAG: --lock-file="" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870910 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870913 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870916 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870921 2563 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870925 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870928 2563 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870931 2563 flags.go:64] FLAG: --logging-format="text" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870936 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870939 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870942 2563 flags.go:64] FLAG: --manifest-url="" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870946 2563 flags.go:64] FLAG: --manifest-url-header="" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870950 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870955 2563 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870959 2563 flags.go:64] FLAG: --max-pods="110" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870963 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870966 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870969 2563 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870973 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870976 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870979 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 16:05:02.878645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870982 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870991 2563 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870994 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.870997 2563 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871001 2563 flags.go:64] FLAG: --pod-cidr="" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871004 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871011 2563 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871014 2563 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871017 2563 flags.go:64] FLAG: --pods-per-core="0" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871021 2563 flags.go:64] FLAG: --port="10250" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871024 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871027 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0691f825d2d3cd47b" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871039 2563 flags.go:64] FLAG: --qos-reserved="" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871043 2563 flags.go:64] FLAG: --read-only-port="10255" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871046 2563 flags.go:64] FLAG: --register-node="true" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871050 2563 flags.go:64] FLAG: --register-with-taints="" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871053 2563 flags.go:64] FLAG: --registry-burst="10" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871058 2563 flags.go:64] FLAG: --registry-qps="5" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871061 2563 flags.go:64] FLAG: --reserved-cpus="" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871066 2563 flags.go:64] FLAG: --reserved-memory="" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871070 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871073 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871077 2563 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871080 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871083 2563 flags.go:64] FLAG: --runonce="false" Sep 18 16:05:02.879315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871088 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871091 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871095 2563 flags.go:64] FLAG: --seccomp-default="false" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871098 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871101 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871104 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871107 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871111 2563 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871114 2563 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871117 2563 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871120 2563 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871123 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871126 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871130 2563 flags.go:64] FLAG: --system-cgroups="" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871133 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871138 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871141 2563 flags.go:64] FLAG: --tls-cert-file="" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871145 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871149 2563 flags.go:64] FLAG: --tls-min-version="" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871152 2563 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871155 2563 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871170 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871174 2563 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871177 2563 flags.go:64] FLAG: --v="2" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871181 2563 flags.go:64] FLAG: --version="false" Sep 18 16:05:02.879979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871187 2563 flags.go:64] FLAG: --vmodule="" Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871192 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.871196 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871314 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871319 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871322 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871325 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871330 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871333 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871336 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871339 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871342 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871345 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871350 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871354 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871357 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871360 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871363 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871366 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871369 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:05:02.880696 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871372 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871375 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871378 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871381 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871384 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871387 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871389 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871392 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871395 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871398 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871400 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871403 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871406 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871410 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871413 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871416 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871419 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871421 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871424 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871428 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:05:02.881277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871431 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871434 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871437 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871440 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871442 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871445 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871448 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871451 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871453 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871457 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871460 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871463 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871465 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871468 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871471 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871473 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871476 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871479 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871482 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871485 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:05:02.881819 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871487 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871490 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871493 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871497 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871501 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871505 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871509 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871512 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871516 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871519 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871523 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871528 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871531 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871534 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871537 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871549 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871552 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871555 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871558 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:05:02.882373 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871561 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871563 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871566 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871569 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871572 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871575 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871578 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871581 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871584 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871587 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871589 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871592 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871595 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871597 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871600 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871603 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871606 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871609 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871614 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871616 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:05:02.882872 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871622 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871625 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.871628 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.872373 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.880352 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.880369 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.880392 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.880404 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880471 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880476 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880480 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880484 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880487 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880490 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880493 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:05:02.883413 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880496 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880499 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880503 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880505 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880508 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880511 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880514 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880517 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880520 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880524 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880527 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880529 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880532 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880535 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880538 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880541 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880544 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880547 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880550 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880553 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:05:02.883816 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880556 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880559 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880569 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880572 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880576 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880581 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880584 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880587 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880590 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880592 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880595 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880598 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880601 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880604 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880607 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880610 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880615 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880619 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880622 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:05:02.884433 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880626 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880629 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880633 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880636 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880640 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880643 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880646 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880649 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880652 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880656 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880659 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880661 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880664 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880667 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880670 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880673 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880682 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880686 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880688 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:05:02.884965 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880691 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880694 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880697 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880700 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880703 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880706 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880709 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880712 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880715 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880717 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880720 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880723 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880726 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880729 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880732 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880734 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880737 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880740 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880744 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880746 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:05:02.885560 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880749 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880752 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880755 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880758 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880762 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880764 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880767 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880770 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880773 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880776 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880785 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880788 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880791 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880794 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.880799 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:05:02.886103 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880953 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880958 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880962 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880965 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880968 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880971 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880974 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880977 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880979 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880982 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880985 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880988 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880992 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880995 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.880998 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881001 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881003 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881006 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881009 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881012 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 16:05:02.886534 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881015 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881017 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881020 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881023 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881025 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881029 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881032 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881035 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881044 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881048 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881053 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881056 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881059 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881062 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881066 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881069 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881072 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881075 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881078 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881081 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 16:05:02.887071 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881084 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881087 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881091 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881093 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881096 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881099 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881102 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881105 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881108 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881111 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881114 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881116 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881119 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881123 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881126 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881130 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881133 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881135 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881138 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881141 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 16:05:02.887624 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881144 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881153 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881155 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881174 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881179 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881182 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881185 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881188 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881190 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881193 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881196 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881199 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881202 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881205 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881207 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881210 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881213 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881216 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881219 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881222 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 16:05:02.888221 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881225 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881228 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881232 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881236 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881239 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881242 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881245 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881248 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881251 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881254 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881257 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881260 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881263 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881266 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881276 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881279 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881282 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881285 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 16:05:02.888937 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:02.881288 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.881293 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.881431 2563 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.883564 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.884748 2563 server.go:1019] "Starting client certificate rotation" Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.884868 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.884905 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 16:05:02.889689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.887977 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:05:02.891878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.891851 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 16:05:02.900203 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.900187 2563 log.go:25] "Validated CRI v1 runtime API" Sep 18 16:05:02.901781 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.901763 2563 log.go:25] "Validated CRI v1 image API" Sep 18 16:05:02.904778 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.904760 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 16:05:02.907873 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.907851 2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9e7b2819-a794-4a9a-935b-7d3ad189561d:/dev/nvme0n1p4 b7aceee4-6682-4ca0-a3ba-74264142b889:/dev/nvme0n1p3] Sep 18 16:05:02.907944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.907871 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 16:05:02.913891 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.913701 2563 manager.go:216] Machine: {Timestamp:2026-09-18 16:05:02.912783163 +0000 UTC m=+2.473627826 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091190 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dbf0c364c6ad69c9d70130423827a SystemUUID:ec2dbf0c-364c-6ad6-9c9d-70130423827a BootID:f390d919-82ea-4f76-b88f-592158f352c3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:34:c2:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:34:c2:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:e9:f8:4a:5d:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 16:05:02.913891 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.913888 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 16:05:02.914055 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.914014 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 16:05:02.916596 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.916565 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 16:05:02.916751 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.916598 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 16:05:02.917849 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.917837 2563 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 16:05:02.917884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.917851 2563 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 16:05:02.917884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.917859 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:05:02.917884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.917871 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 16:05:02.917972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.917945 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 16:05:02.922193 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922180 2563 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:05:02.922331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922321 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 16:05:02.922365 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922356 2563 kubelet.go:486] "Attempting to sync node with API server" Sep 18 16:05:02.922394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922373 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 16:05:02.922394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922390 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 16:05:02.922452 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922425 2563 kubelet.go:398] "Adding apiserver pod source" Sep 18 16:05:02.922452 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.922448 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 16:05:02.923039 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.923008 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:05:02.923039 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.923042 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 16:05:02.929699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.929672 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 16:05:02.929880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.929859 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 16:05:02.933458 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.933439 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 16:05:02.933540 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.933473 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 16:05:02.934562 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934549 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934566 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934574 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934580 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934589 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934596 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934610 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 16:05:02.934621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934615 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 16:05:02.934833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934628 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 16:05:02.934833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934634 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 16:05:02.934833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934643 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 16:05:02.934833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934649 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 16:05:02.934833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934674 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 16:05:02.934833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.934679 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 16:05:02.938844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.938827 2563 server.go:1268] "Started kubelet" Sep 18 16:05:02.939015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.938967 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 16:05:02.939087 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.939071 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 16:05:02.939087 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.939080 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 16:05:02.939197 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.939145 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 16:05:02.939266 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.939208 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 16:05:02.939266 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.939242 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 16:05:02.939804 ip-10-0-132-41 systemd[1]: Started Kubernetes Kubelet. Sep 18 16:05:02.940498 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.940465 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 16:05:02.940561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.940524 2563 server.go:310] "Adding debug handlers to kubelet server" Sep 18 16:05:02.943763 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.943734 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 16:05:02.946564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.946545 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 16:05:02.946653 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.946588 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 16:05:02.947072 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.946870 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:02.947241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947222 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 16:05:02.947241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947233 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 16:05:02.947368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947257 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 16:05:02.947417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947382 2563 reconstruct.go:97] "Volume reconstruction finished" Sep 18 16:05:02.947417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947392 2563 reconciler.go:29] "Reconciler: start to sync state" Sep 18 16:05:02.947605 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947592 2563 factory.go:55] Registering systemd factory Sep 18 16:05:02.947672 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947607 2563 factory.go:223] Registration of the systemd container factory successfully Sep 18 16:05:02.947835 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947821 2563 factory.go:153] Registering CRI-O factory Sep 18 16:05:02.947917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947840 2563 factory.go:223] Registration of the crio container factory successfully Sep 18 16:05:02.947917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947893 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 16:05:02.947917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947916 2563 factory.go:103] Registering Raw factory Sep 18 16:05:02.948049 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.947935 2563 manager.go:1196] Started watching for new ooms in manager Sep 18 16:05:02.948964 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.948936 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 16:05:02.949206 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.949190 2563 manager.go:349] Starting recovery of all containers Sep 18 16:05:02.949354 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.949331 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 16:05:02.954728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.954696 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 16:05:02.982356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.982326 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t58qz" Sep 18 16:05:02.984333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.984312 2563 manager.go:354] Recovery completed Sep 18 16:05:02.988680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.988667 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:02.990869 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.990845 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t58qz" Sep 18 16:05:02.991001 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.990979 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:02.991053 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.991015 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:02.991053 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.991031 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:02.991580 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.991566 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 16:05:02.991580 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.991580 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 16:05:02.991688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.991596 2563 state_mem.go:36] "Initialized new in-memory state store" Sep 18 16:05:02.991688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.991628 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:02.992378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.992350 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:02.992474 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.992392 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:02.992474 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.992408 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:02.994665 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.994468 2563 policy_none.go:49] "None policy: Start" Sep 18 16:05:02.994742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.994671 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 16:05:02.994742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.994688 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 16:05:02.997012 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:02.996998 2563 policy_none.go:47] "Start" Sep 18 16:05:02.998312 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:02.997252 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-41.ec2.internal.18d675df164f4e22 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-41.ec2.internal,UID:ip-10-0-132-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:05:02.938803746 +0000 UTC m=+2.499648407,LastTimestamp:2026-09-18 16:05:02.938803746 +0000 UTC m=+2.499648407,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:05:03.047949 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.047281 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.053795 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.053780 2563 manager.go:337] "Starting Device Plugin manager" Sep 18 16:05:03.053882 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.053825 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 16:05:03.053882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.053836 2563 server.go:87] "Starting device plugin registration server" Sep 18 16:05:03.054161 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.054147 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 16:05:03.054208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.054176 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 16:05:03.054314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.054287 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 16:05:03.054417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.054386 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 16:05:03.054417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.054397 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 16:05:03.054963 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.054940 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 16:05:03.055030 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.055016 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.073793 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.073767 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 16:05:03.073909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.073818 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 16:05:03.073909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.073840 2563 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 16:05:03.073909 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.073878 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 16:05:03.076223 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.076203 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:05:03.155021 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.154942 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:03.156675 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.156658 2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-41.ec2.internal\" not found" node="ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.157028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.157013 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:03.157121 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.157051 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:03.157121 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.157064 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:03.157121 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.157089 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.167407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.167392 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.167449 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.167414 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.174436 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.174407 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal"] Sep 18 16:05:03.174491 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.174482 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:03.175394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.175379 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:03.175468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.175407 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:03.175468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.175423 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:03.177271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.177258 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:03.177417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.177403 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.177453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.177434 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:03.178197 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.178175 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:03.178301 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.178204 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:03.178301 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.178229 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:03.178301 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.178239 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:03.178301 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.178206 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:03.178463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.178305 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:03.179870 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.179855 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.179943 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.179882 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:03.180652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.180636 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:03.180733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.180668 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:03.180733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.180680 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:03.197653 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.197629 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.203872 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.203855 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-41.ec2.internal\" not found" node="ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.208497 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.208483 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-41.ec2.internal\" not found" node="ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.249396 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.249365 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8d19fe14a09d468a2dde486d300cd1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal\" (UID: \"d8d19fe14a09d468a2dde486d300cd1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.249396 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.249398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d19fe14a09d468a2dde486d300cd1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal\" (UID: \"d8d19fe14a09d468a2dde486d300cd1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.249396 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.249414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ae31834f4280439711acddc59629bae-config\") pod \"kube-apiserver-proxy-ip-10-0-132-41.ec2.internal\" (UID: \"9ae31834f4280439711acddc59629bae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.298418 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.298384 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.350434 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.350398 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8d19fe14a09d468a2dde486d300cd1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal\" (UID: \"d8d19fe14a09d468a2dde486d300cd1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.350434 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.350435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d19fe14a09d468a2dde486d300cd1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal\" (UID: \"d8d19fe14a09d468a2dde486d300cd1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.350635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.350451 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ae31834f4280439711acddc59629bae-config\") pod \"kube-apiserver-proxy-ip-10-0-132-41.ec2.internal\" (UID: \"9ae31834f4280439711acddc59629bae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.350635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.350501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8d19fe14a09d468a2dde486d300cd1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal\" (UID: \"d8d19fe14a09d468a2dde486d300cd1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.350635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.350511 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8d19fe14a09d468a2dde486d300cd1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal\" (UID: \"d8d19fe14a09d468a2dde486d300cd1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.350635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.350546 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ae31834f4280439711acddc59629bae-config\") pod \"kube-apiserver-proxy-ip-10-0-132-41.ec2.internal\" (UID: \"9ae31834f4280439711acddc59629bae\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.399224 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.399179 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.503325 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.503231 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.507674 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.507655 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.513325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.512762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" Sep 18 16:05:03.603862 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.603826 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.707200 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.707156 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.807665 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.807581 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.885007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.884976 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 16:05:03.885472 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.885141 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 16:05:03.908239 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:03.908219 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:03.970666 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.970637 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:05:03.996649 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.996598 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:00:03 +0000 UTC" deadline="2028-07-01 21:31:01.907238898 +0000 UTC" Sep 18 16:05:03.996649 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:03.996642 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15653h25m57.910599636s" Sep 18 16:05:04.008739 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:04.008713 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:04.089422 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.089156 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:05:04.089617 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.089499 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 16:05:04.090539 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.090518 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientMemory" Sep 18 16:05:04.090634 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.090552 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 16:05:04.090634 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.090564 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeHasSufficientPID" Sep 18 16:05:04.109396 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:04.109367 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:04.209800 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:04.209778 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-41.ec2.internal\" not found" Sep 18 16:05:04.223134 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:04.223107 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae31834f4280439711acddc59629bae.slice/crio-f15658f0e2d9f8d08462404be147ff8e4eee2be41daa271b1eff7fb9963bc47d WatchSource:0}: Error finding container f15658f0e2d9f8d08462404be147ff8e4eee2be41daa271b1eff7fb9963bc47d: Status 404 returned error can't find the container with id f15658f0e2d9f8d08462404be147ff8e4eee2be41daa271b1eff7fb9963bc47d Sep 18 16:05:04.230948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.230266 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:05:04.277001 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.276978 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:05:04.328667 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:04.328639 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d19fe14a09d468a2dde486d300cd1c.slice/crio-830133fcaf8f4dfc1478b701e3f13a0b22ee08f0367b2c72d82e446fb5dada74 WatchSource:0}: Error finding container 830133fcaf8f4dfc1478b701e3f13a0b22ee08f0367b2c72d82e446fb5dada74: Status 404 returned error can't find the container with id 830133fcaf8f4dfc1478b701e3f13a0b22ee08f0367b2c72d82e446fb5dada74 Sep 18 16:05:04.347364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.347346 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" Sep 18 16:05:04.363438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.363423 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 16:05:04.364572 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.364560 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" Sep 18 16:05:04.377426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.377409 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 16:05:04.888710 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.888680 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:05:04.924632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.924397 2563 apiserver.go:52] "Watching apiserver" Sep 18 16:05:04.932507 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.932338 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 16:05:04.932938 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.932905 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-wsw6t","kube-system/konnectivity-agent-9z7rv","kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal","openshift-dns/node-resolver-bbmtn","openshift-image-registry/node-ca-qcxrf","openshift-multus/multus-zj9qd","openshift-multus/network-metrics-daemon-4j77c","openshift-network-operator/iptables-alerter-8dxgl","openshift-ovn-kubernetes/ovnkube-node-r7hqn","kube-system/global-pull-secret-syncer-r2z6d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4","openshift-cluster-node-tuning-operator/tuned-s5l54","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal","openshift-multus/multus-additional-cni-plugins-phmtm"] Sep 18 16:05:04.935905 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.935502 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:04.935905 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:04.935584 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:04.938154 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.937687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.941533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.939915 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.941533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.941201 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.941533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.941270 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 16:05:04.941533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.941394 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvq9z\"" Sep 18 16:05:04.942405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.942380 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:04.945333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.943295 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 16:05:04.945333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.943503 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 16:05:04.945333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.943520 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.945333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.943705 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vx59n\"" Sep 18 16:05:04.945333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.943718 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 16:05:04.945333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.943872 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.945686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.945613 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:04.946563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.945980 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.946563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.946281 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.946563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.946381 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcclj\"" Sep 18 16:05:04.948476 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.946816 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.948476 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.947048 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 16:05:04.948476 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.947461 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.948476 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.947918 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.948476 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.948442 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 16:05:04.948818 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.948511 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjzzn\"" Sep 18 16:05:04.948818 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.948716 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.949235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.949005 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.949973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.949942 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 16:05:04.950067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.949978 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdh8x\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.951061 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.951462 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.951829 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.952014 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.952516 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.953408 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.953433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4v9d\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.953694 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.954077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.953911 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:04.954931 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.954912 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.955395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.955376 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 16:05:04.955632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.955612 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8cs6v\"" Sep 18 16:05:04.955853 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.955837 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.956117 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.956052 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:04.959004 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:04.957438 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:04.959004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.958283 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6qnc4\"" Sep 18 16:05:04.959004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.958537 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 16:05:04.959004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.958795 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.961921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cnibin\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.961985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvphp\" (UniqueName: \"kubernetes.io/projected/77789e19-a00a-403e-9556-65748730930e-kube-api-access-jvphp\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962035 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b2534ae-985e-4a89-ba4b-99ca53d9a179-cni-binary-copy\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962068 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-daemon-config\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962146 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-env-overrides\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962191 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47p52\" (UniqueName: \"kubernetes.io/projected/2aea08e0-a545-4ddb-bafa-d1a277968ade-kube-api-access-47p52\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-system-cni-dir\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962292 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-socket-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-kubelet\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962383 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-cni-bin\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cni-binary-copy\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kl6\" (UniqueName: \"kubernetes.io/projected/acc1a685-72d0-471f-808d-22cfc451dee0-kube-api-access-84kl6\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77789e19-a00a-403e-9556-65748730930e-hosts-file\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:04.963231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962607 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvgr\" (UniqueName: \"kubernetes.io/projected/4b2534ae-985e-4a89-ba4b-99ca53d9a179-kube-api-access-tgvgr\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.962666 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-kubelet\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-cni-bin\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq9fq\" (UniqueName: \"kubernetes.io/projected/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-kube-api-access-fq9fq\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963301 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-multus-certs\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-conf-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963403 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-systemd-units\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963445 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-slash\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963532 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovnkube-script-lib\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963561 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-etc-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwntz\" (UniqueName: \"kubernetes.io/projected/d47181ff-68b6-410e-827d-2f4de7db7d7b-kube-api-access-lwntz\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963647 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-cni-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-etc-kubernetes\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-node-log\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrjm\" (UniqueName: \"kubernetes.io/projected/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-kube-api-access-zzrjm\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963772 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-registration-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-hostroot\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963856 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-run-netns\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-device-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.963966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-system-cni-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964068 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-cnibin\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964108 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acc1a685-72d0-471f-808d-22cfc451dee0-serviceca\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964456 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-cni-multus\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-var-lib-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovnkube-config\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964552 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-sys-fs\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77789e19-a00a-403e-9556-65748730930e-tmp-dir\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964618 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-socket-dir-parent\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.964965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovn-node-metrics-cert\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-k8s-cni-cncf-io\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964743 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964806 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acc1a685-72d0-471f-808d-22cfc451dee0-host\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-ovn\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964874 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-log-socket\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964905 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-netns\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.964968 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-systemd\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.965764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.965143 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-cni-netd\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:04.966234 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.965855 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:04.966234 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:04.965947 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:04.969146 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.968954 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:04.971730 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.971710 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:05:04.971958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.971935 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 16:05:04.972052 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.972028 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bm6xr\"" Sep 18 16:05:04.972946 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.972924 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 16:05:04.985125 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.985097 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rm6k5" Sep 18 16:05:04.992378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:04.992350 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rm6k5" Sep 18 16:05:05.048395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.048371 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 16:05:05.065707 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.065858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-lib-modules\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.065858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-netns\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.065858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065766 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-systemd\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.065858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-cni-netd\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.065858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-systemd\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.065858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f9d4fd6-0082-4200-8fbb-099af79e209e-konnectivity-ca\") pod \"konnectivity-agent-9z7rv\" (UID: \"2f9d4fd6-0082-4200-8fbb-099af79e209e\") " pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-systemd\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065908 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-netns\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065913 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cnibin\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065916 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-cni-netd\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065946 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cnibin\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvphp\" (UniqueName: \"kubernetes.io/projected/77789e19-a00a-403e-9556-65748730930e-kube-api-access-jvphp\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.065986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b2534ae-985e-4a89-ba4b-99ca53d9a179-cni-binary-copy\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-daemon-config\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-env-overrides\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066037 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47p52\" (UniqueName: \"kubernetes.io/projected/2aea08e0-a545-4ddb-bafa-d1a277968ade-kube-api-access-47p52\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-system-cni-dir\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-socket-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066154 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-kubelet\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066175 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-system-cni-dir\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-cni-bin\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cni-binary-copy\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066300 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-socket-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066327 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066347 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-kubelet\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-cni-bin\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kl6\" (UniqueName: \"kubernetes.io/projected/acc1a685-72d0-471f-808d-22cfc451dee0-kube-api-access-84kl6\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066392 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b914e1-c1d7-4f98-afdf-ad2a58a52309-host-slash\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066419 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77789e19-a00a-403e-9556-65748730930e-hosts-file\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvgr\" (UniqueName: \"kubernetes.io/projected/4b2534ae-985e-4a89-ba4b-99ca53d9a179-kube-api-access-tgvgr\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-kubelet\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066534 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62f743f6-9ca7-41f0-85cf-4266d18a1f78-kubelet-config\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.066641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-kubernetes\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-cni-bin\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066623 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq9fq\" (UniqueName: \"kubernetes.io/projected/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-kube-api-access-fq9fq\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62f743f6-9ca7-41f0-85cf-4266d18a1f78-dbus\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-tuned\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-multus-certs\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066770 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77789e19-a00a-403e-9556-65748730930e-hosts-file\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-env-overrides\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-conf-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066741 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-kubelet\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066818 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4b2534ae-985e-4a89-ba4b-99ca53d9a179-cni-binary-copy\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066824 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cni-binary-copy\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-systemd-units\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066853 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-multus-certs\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.067496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066863 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066775 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-daemon-config\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066868 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-conf-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-slash\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066901 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-cni-bin\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066928 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-systemd-units\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066946 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-slash\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066977 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.066985 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovnkube-script-lib\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-etc-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwntz\" (UniqueName: \"kubernetes.io/projected/d47181ff-68b6-410e-827d-2f4de7db7d7b-kube-api-access-lwntz\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysctl-conf\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067112 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-etc-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-cni-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067156 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-etc-kubernetes\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-node-log\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.068325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f9d4fd6-0082-4200-8fbb-099af79e209e-agent-certs\") pod \"konnectivity-agent-9z7rv\" (UID: \"2f9d4fd6-0082-4200-8fbb-099af79e209e\") " pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067239 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99b914e1-c1d7-4f98-afdf-ad2a58a52309-iptables-alerter-script\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067248 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067269 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-cni-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067270 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrjm\" (UniqueName: \"kubernetes.io/projected/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-kube-api-access-zzrjm\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067324 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-etc-kubernetes\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067325 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-run\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067331 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-node-log\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mtf\" (UniqueName: \"kubernetes.io/projected/99b914e1-c1d7-4f98-afdf-ad2a58a52309-kube-api-access-s6mtf\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067396 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-registration-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-hostroot\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-run-netns\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067493 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-hostroot\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067515 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-registration-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067499 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysconfig\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067530 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovnkube-script-lib\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.069189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067547 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-run-netns\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-sys\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067579 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-device-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067552 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067617 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-system-cni-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067651 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-device-dir\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-cnibin\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067734 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-system-cni-dir\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acc1a685-72d0-471f-808d-22cfc451dee0-serviceca\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-var-lib-kubelet\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067838 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-host\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067875 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-cni-multus\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067882 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-cnibin\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-var-lib-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovnkube-config\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067937 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-var-lib-cni-multus\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-modprobe-d\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-sys-fs\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.070367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.067999 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-var-lib-openvswitch\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068006 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77789e19-a00a-403e-9556-65748730930e-tmp-dir\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068031 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-socket-dir-parent\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovn-node-metrics-cert\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-k8s-cni-cncf-io\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acc1a685-72d0-471f-808d-22cfc451dee0-host\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068186 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/acc1a685-72d0-471f-808d-22cfc451dee0-serviceca\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnx5f\" (UniqueName: \"kubernetes.io/projected/28e3f91b-3eac-4bdc-b329-42da5929b082-kube-api-access-hnx5f\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-multus-socket-dir-parent\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-ovn\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-log-socket\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysctl-d\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77789e19-a00a-403e-9556-65748730930e-tmp-dir\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.071726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068372 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28e3f91b-3eac-4bdc-b329-42da5929b082-tmp\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068057 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-sys-fs\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-log-socket\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068452 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovnkube-config\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068456 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-host-run-ovn-kubernetes\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068479 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2aea08e0-a545-4ddb-bafa-d1a277968ade-run-ovn\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acc1a685-72d0-471f-808d-22cfc451dee0-host\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068543 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.068604 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.068671 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:05:05.568651417 +0000 UTC m=+5.129496067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068672 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4b2534ae-985e-4a89-ba4b-99ca53d9a179-host-run-k8s-cni-cncf-io\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.068914 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.072652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.072160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2aea08e0-a545-4ddb-bafa-d1a277968ade-ovn-node-metrics-cert\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.079793 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.079412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvgr\" (UniqueName: \"kubernetes.io/projected/4b2534ae-985e-4a89-ba4b-99ca53d9a179-kube-api-access-tgvgr\") pod \"multus-zj9qd\" (UID: \"4b2534ae-985e-4a89-ba4b-99ca53d9a179\") " pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.080577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.080473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" event={"ID":"d8d19fe14a09d468a2dde486d300cd1c","Type":"ContainerStarted","Data":"830133fcaf8f4dfc1478b701e3f13a0b22ee08f0367b2c72d82e446fb5dada74"} Sep 18 16:05:05.080684 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.080611 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvphp\" (UniqueName: \"kubernetes.io/projected/77789e19-a00a-403e-9556-65748730930e-kube-api-access-jvphp\") pod \"node-resolver-bbmtn\" (UID: \"77789e19-a00a-403e-9556-65748730930e\") " pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.081810 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.081768 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" event={"ID":"9ae31834f4280439711acddc59629bae","Type":"ContainerStarted","Data":"f15658f0e2d9f8d08462404be147ff8e4eee2be41daa271b1eff7fb9963bc47d"} Sep 18 16:05:05.082244 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.082129 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kl6\" (UniqueName: \"kubernetes.io/projected/acc1a685-72d0-471f-808d-22cfc451dee0-kube-api-access-84kl6\") pod \"node-ca-qcxrf\" (UID: \"acc1a685-72d0-471f-808d-22cfc451dee0\") " pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.082748 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.082717 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrjm\" (UniqueName: \"kubernetes.io/projected/28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d-kube-api-access-zzrjm\") pod \"aws-ebs-csi-driver-node-89mj4\" (UID: \"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.084200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.084176 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47p52\" (UniqueName: \"kubernetes.io/projected/2aea08e0-a545-4ddb-bafa-d1a277968ade-kube-api-access-47p52\") pod \"ovnkube-node-r7hqn\" (UID: \"2aea08e0-a545-4ddb-bafa-d1a277968ade\") " pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.084425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.084402 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq9fq\" (UniqueName: \"kubernetes.io/projected/91e4adb2-6caf-4bcb-9b16-bb53b68e1162-kube-api-access-fq9fq\") pod \"multus-additional-cni-plugins-phmtm\" (UID: \"91e4adb2-6caf-4bcb-9b16-bb53b68e1162\") " pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.084498 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.084433 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwntz\" (UniqueName: \"kubernetes.io/projected/d47181ff-68b6-410e-827d-2f4de7db7d7b-kube-api-access-lwntz\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:05.168962 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.168876 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-modprobe-d\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.168962 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.168954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnx5f\" (UniqueName: \"kubernetes.io/projected/28e3f91b-3eac-4bdc-b329-42da5929b082-kube-api-access-hnx5f\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.168981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysctl-d\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28e3f91b-3eac-4bdc-b329-42da5929b082-tmp\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169058 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-modprobe-d\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-lib-modules\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.169167 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f9d4fd6-0082-4200-8fbb-099af79e209e-konnectivity-ca\") pod \"konnectivity-agent-9z7rv\" (UID: \"2f9d4fd6-0082-4200-8fbb-099af79e209e\") " pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.169209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-systemd\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.169222 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:05.669205395 +0000 UTC m=+5.230050044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169248 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-systemd\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b914e1-c1d7-4f98-afdf-ad2a58a52309-host-slash\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62f743f6-9ca7-41f0-85cf-4266d18a1f78-kubelet-config\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-kubernetes\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62f743f6-9ca7-41f0-85cf-4266d18a1f78-dbus\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-lib-modules\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-tuned\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169404 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysctl-conf\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f9d4fd6-0082-4200-8fbb-099af79e209e-agent-certs\") pod \"konnectivity-agent-9z7rv\" (UID: \"2f9d4fd6-0082-4200-8fbb-099af79e209e\") " pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169461 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99b914e1-c1d7-4f98-afdf-ad2a58a52309-iptables-alerter-script\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-run\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mtf\" (UniqueName: \"kubernetes.io/projected/99b914e1-c1d7-4f98-afdf-ad2a58a52309-kube-api-access-s6mtf\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62f743f6-9ca7-41f0-85cf-4266d18a1f78-kubelet-config\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysconfig\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.169648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169640 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-sys\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-var-lib-kubelet\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-host\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169762 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-host\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169853 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-kubernetes\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169967 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99b914e1-c1d7-4f98-afdf-ad2a58a52309-host-slash\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.169976 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2f9d4fd6-0082-4200-8fbb-099af79e209e-konnectivity-ca\") pod \"konnectivity-agent-9z7rv\" (UID: \"2f9d4fd6-0082-4200-8fbb-099af79e209e\") " pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170025 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-sys\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170030 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysconfig\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170106 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62f743f6-9ca7-41f0-85cf-4266d18a1f78-dbus\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170182 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysctl-conf\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-run\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.170468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-var-lib-kubelet\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.171346 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.170514 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-sysctl-d\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.172973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.172944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99b914e1-c1d7-4f98-afdf-ad2a58a52309-iptables-alerter-script\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.173101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.172956 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28e3f91b-3eac-4bdc-b329-42da5929b082-tmp\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.173101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.172970 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28e3f91b-3eac-4bdc-b329-42da5929b082-etc-tuned\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.177413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.177389 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2f9d4fd6-0082-4200-8fbb-099af79e209e-agent-certs\") pod \"konnectivity-agent-9z7rv\" (UID: \"2f9d4fd6-0082-4200-8fbb-099af79e209e\") " pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.178727 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.178703 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:05.178811 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.178733 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:05.178811 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.178748 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:05.178932 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.178815 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:05.67879743 +0000 UTC m=+5.239642100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:05.179257 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.179211 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mtf\" (UniqueName: \"kubernetes.io/projected/99b914e1-c1d7-4f98-afdf-ad2a58a52309-kube-api-access-s6mtf\") pod \"iptables-alerter-8dxgl\" (UID: \"99b914e1-c1d7-4f98-afdf-ad2a58a52309\") " pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.179444 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.179424 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnx5f\" (UniqueName: \"kubernetes.io/projected/28e3f91b-3eac-4bdc-b329-42da5929b082-kube-api-access-hnx5f\") pod \"tuned-s5l54\" (UID: \"28e3f91b-3eac-4bdc-b329-42da5929b082\") " pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.253044 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.253009 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" Sep 18 16:05:05.262818 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.262789 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phmtm" Sep 18 16:05:05.266042 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.266005 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28caa9fb_e1a1_4ef4_98be_a47e55a2fc8d.slice/crio-0fda738ebbae5065b9dc47aaab02fcb18e58b73bfcc9134219c8300dbaa2def6 WatchSource:0}: Error finding container 0fda738ebbae5065b9dc47aaab02fcb18e58b73bfcc9134219c8300dbaa2def6: Status 404 returned error can't find the container with id 0fda738ebbae5065b9dc47aaab02fcb18e58b73bfcc9134219c8300dbaa2def6 Sep 18 16:05:05.275591 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.275568 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e4adb2_6caf_4bcb_9b16_bb53b68e1162.slice/crio-7a08506815fcbc4e5ddfda00ef0343264e343b0188f76884a2b9cbe2fb4e5010 WatchSource:0}: Error finding container 7a08506815fcbc4e5ddfda00ef0343264e343b0188f76884a2b9cbe2fb4e5010: Status 404 returned error can't find the container with id 7a08506815fcbc4e5ddfda00ef0343264e343b0188f76884a2b9cbe2fb4e5010 Sep 18 16:05:05.277284 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.277259 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bbmtn" Sep 18 16:05:05.285305 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.285166 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qcxrf" Sep 18 16:05:05.288019 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.287996 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77789e19_a00a_403e_9556_65748730930e.slice/crio-42b1a1eb386a5f6457204967b531c8079302e32714c1dda5cdb8a69df5f6b033 WatchSource:0}: Error finding container 42b1a1eb386a5f6457204967b531c8079302e32714c1dda5cdb8a69df5f6b033: Status 404 returned error can't find the container with id 42b1a1eb386a5f6457204967b531c8079302e32714c1dda5cdb8a69df5f6b033 Sep 18 16:05:05.294334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.294312 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zj9qd" Sep 18 16:05:05.294616 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.294597 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc1a685_72d0_471f_808d_22cfc451dee0.slice/crio-eeefa95365d3f72f25d366ba3157878311b585aec34c602cfa5b48b5c7166969 WatchSource:0}: Error finding container eeefa95365d3f72f25d366ba3157878311b585aec34c602cfa5b48b5c7166969: Status 404 returned error can't find the container with id eeefa95365d3f72f25d366ba3157878311b585aec34c602cfa5b48b5c7166969 Sep 18 16:05:05.301766 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.301620 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:05.304705 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.304668 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2534ae_985e_4a89_ba4b_99ca53d9a179.slice/crio-a10d18a186a8c90f28f7af556d7952377937ab5d2d8b918ce14922c2884c1e8a WatchSource:0}: Error finding container a10d18a186a8c90f28f7af556d7952377937ab5d2d8b918ce14922c2884c1e8a: Status 404 returned error can't find the container with id a10d18a186a8c90f28f7af556d7952377937ab5d2d8b918ce14922c2884c1e8a Sep 18 16:05:05.310401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.310379 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8dxgl" Sep 18 16:05:05.311532 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.311509 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aea08e0_a545_4ddb_bafa_d1a277968ade.slice/crio-7adbccdbc2ce4087015ce05eeed83064cfd26327e0c662c254e0e91ca60efecc WatchSource:0}: Error finding container 7adbccdbc2ce4087015ce05eeed83064cfd26327e0c662c254e0e91ca60efecc: Status 404 returned error can't find the container with id 7adbccdbc2ce4087015ce05eeed83064cfd26327e0c662c254e0e91ca60efecc Sep 18 16:05:05.317988 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.317906 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:05.320435 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:05.320416 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b914e1_c1d7_4f98_afdf_ad2a58a52309.slice/crio-a410e7a1bab33f250e4dc9edc1b516a2fb20f888105bd16c3ad0b46eb651ce77 WatchSource:0}: Error finding container a410e7a1bab33f250e4dc9edc1b516a2fb20f888105bd16c3ad0b46eb651ce77: Status 404 returned error can't find the container with id a410e7a1bab33f250e4dc9edc1b516a2fb20f888105bd16c3ad0b46eb651ce77 Sep 18 16:05:05.331370 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.331345 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s5l54" Sep 18 16:05:05.572574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.572496 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:05.572721 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.572660 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:05.572771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.572727 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:05:06.57271285 +0000 UTC m=+6.133557500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:05.673507 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.673472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:05.673667 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.673589 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:05.673667 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.673641 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:06.673629211 +0000 UTC m=+6.234473856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:05.774846 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.774812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:05.775019 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.774987 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:05.775019 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.775015 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:05.775113 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.775025 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:05.775170 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:05.775113 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:06.775080769 +0000 UTC m=+6.335925434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:05.995644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.995600 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:00:04 +0000 UTC" deadline="2028-06-13 21:35:46.524638778 +0000 UTC" Sep 18 16:05:05.995644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:05.995640 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15221h30m40.529002672s" Sep 18 16:05:06.084696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.084660 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"7adbccdbc2ce4087015ce05eeed83064cfd26327e0c662c254e0e91ca60efecc"} Sep 18 16:05:06.085918 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.085890 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qcxrf" event={"ID":"acc1a685-72d0-471f-808d-22cfc451dee0","Type":"ContainerStarted","Data":"eeefa95365d3f72f25d366ba3157878311b585aec34c602cfa5b48b5c7166969"} Sep 18 16:05:06.087070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.087051 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerStarted","Data":"7a08506815fcbc4e5ddfda00ef0343264e343b0188f76884a2b9cbe2fb4e5010"} Sep 18 16:05:06.088247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.088215 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8dxgl" event={"ID":"99b914e1-c1d7-4f98-afdf-ad2a58a52309","Type":"ContainerStarted","Data":"a410e7a1bab33f250e4dc9edc1b516a2fb20f888105bd16c3ad0b46eb651ce77"} Sep 18 16:05:06.089440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.089406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zj9qd" event={"ID":"4b2534ae-985e-4a89-ba4b-99ca53d9a179","Type":"ContainerStarted","Data":"a10d18a186a8c90f28f7af556d7952377937ab5d2d8b918ce14922c2884c1e8a"} Sep 18 16:05:06.090612 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.090588 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bbmtn" event={"ID":"77789e19-a00a-403e-9556-65748730930e","Type":"ContainerStarted","Data":"42b1a1eb386a5f6457204967b531c8079302e32714c1dda5cdb8a69df5f6b033"} Sep 18 16:05:06.091741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.091718 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" event={"ID":"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d","Type":"ContainerStarted","Data":"0fda738ebbae5065b9dc47aaab02fcb18e58b73bfcc9134219c8300dbaa2def6"} Sep 18 16:05:06.416016 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:06.415933 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9d4fd6_0082_4200_8fbb_099af79e209e.slice/crio-c77a6a567fc35536e6f7372266d145baa44676367177c1aa496db1ea4ee64ca0 WatchSource:0}: Error finding container c77a6a567fc35536e6f7372266d145baa44676367177c1aa496db1ea4ee64ca0: Status 404 returned error can't find the container with id c77a6a567fc35536e6f7372266d145baa44676367177c1aa496db1ea4ee64ca0 Sep 18 16:05:06.582509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.582475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:06.582703 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.582645 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:06.583188 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.582854 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:05:08.582830774 +0000 UTC m=+8.143675432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:06.684877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.684018 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:06.684877 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.684397 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:06.684877 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.684522 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:08.684502069 +0000 UTC m=+8.245346729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:06.785231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.785193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:06.785392 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.785381 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:06.785469 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.785402 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:06.785469 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.785414 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:06.785567 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:06.785475 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:08.785453634 +0000 UTC m=+8.346298282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:06.995924 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.995885 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:00:04 +0000 UTC" deadline="2028-03-13 22:32:10.593824387 +0000 UTC" Sep 18 16:05:06.995924 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:06.995923 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13014h27m3.597905454s" Sep 18 16:05:07.086198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.084397 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:07.086198 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:07.084555 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:07.086198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.085026 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:07.086198 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:07.085158 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:07.086198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.085688 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:07.086198 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:07.085814 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:07.109821 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.109788 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" event={"ID":"9ae31834f4280439711acddc59629bae","Type":"ContainerStarted","Data":"4aa376866b4b6358184e5e9c5712094705d8f89c3e287e72ddaec91afd06d9d5"} Sep 18 16:05:07.115193 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.115163 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s5l54" event={"ID":"28e3f91b-3eac-4bdc-b329-42da5929b082","Type":"ContainerStarted","Data":"ea4db749f8dd60d67d43ad78936f8797f28091ba518f1bfbcb1813796c0b1c92"} Sep 18 16:05:07.120773 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.120743 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" event={"ID":"d8d19fe14a09d468a2dde486d300cd1c","Type":"ContainerStarted","Data":"25aaaaff06711ab7e01c99603c369ad1816dbc18dc10a7fc3a054c2240d4bb6b"} Sep 18 16:05:07.123480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.123404 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9z7rv" event={"ID":"2f9d4fd6-0082-4200-8fbb-099af79e209e","Type":"ContainerStarted","Data":"c77a6a567fc35536e6f7372266d145baa44676367177c1aa496db1ea4ee64ca0"} Sep 18 16:05:07.153449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:07.153006 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-41.ec2.internal" podStartSLOduration=3.1529880869999998 podStartE2EDuration="3.152988087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:07.136354743 +0000 UTC m=+6.697199411" watchObservedRunningTime="2026-09-18 16:05:07.152988087 +0000 UTC m=+6.713832755" Sep 18 16:05:08.134099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:08.132847 2563 generic.go:356] "Generic (PLEG): container finished" podID="d8d19fe14a09d468a2dde486d300cd1c" containerID="25aaaaff06711ab7e01c99603c369ad1816dbc18dc10a7fc3a054c2240d4bb6b" exitCode=0 Sep 18 16:05:08.134099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:08.133864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" event={"ID":"d8d19fe14a09d468a2dde486d300cd1c","Type":"ContainerDied","Data":"25aaaaff06711ab7e01c99603c369ad1816dbc18dc10a7fc3a054c2240d4bb6b"} Sep 18 16:05:08.602336 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:08.601629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:08.602336 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.601834 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:08.602336 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.601902 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:05:12.601884867 +0000 UTC m=+12.162729515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:08.702593 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:08.702555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:08.702772 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.702751 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:08.702848 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.702818 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:12.702801088 +0000 UTC m=+12.263645744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:08.803332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:08.803290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:08.803519 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.803488 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:08.803519 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.803510 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:08.803634 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.803522 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:08.803634 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:08.803581 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:12.803563516 +0000 UTC m=+12.364408165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:09.074450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:09.074365 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:09.074450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:09.074419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:09.074680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:09.074365 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:09.074680 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:09.074525 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:09.074680 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:09.074604 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:09.074829 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:09.074685 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:11.074932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:11.074894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:11.075823 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:11.075036 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:11.075823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:11.074895 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:11.075823 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:11.075185 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:11.075823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:11.075228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:11.075823 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:11.075293 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:12.638290 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:12.638230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:12.638727 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.638401 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:12.638727 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.638480 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:05:20.638459362 +0000 UTC m=+20.199304027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:12.739432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:12.739386 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:12.739615 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.739554 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:12.739678 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.739618 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:20.739599327 +0000 UTC m=+20.300443986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:12.840639 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:12.840589 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:12.840824 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.840757 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:12.840824 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.840777 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:12.840824 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.840790 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:12.840987 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:12.840851 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:20.840833749 +0000 UTC m=+20.401678420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:13.076574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:13.075590 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:13.076574 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:13.075709 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:13.076574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:13.076200 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:13.076574 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:13.076353 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:13.076574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:13.076416 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:13.076574 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:13.076489 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:15.074549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:15.074509 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:15.074549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:15.074544 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:15.075204 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:15.074513 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:15.075204 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:15.074640 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:15.075204 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:15.074723 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:15.075204 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:15.074819 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:17.075199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:17.075157 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:17.075630 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:17.075156 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:17.075630 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:17.075294 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:17.075630 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:17.075414 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:17.075630 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:17.075156 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:17.075630 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:17.075513 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:19.074990 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:19.074956 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:19.075505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:19.074998 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:19.075505 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:19.075110 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:19.075505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:19.075204 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:19.075505 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:19.075273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:19.075505 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:19.075195 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:20.698200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:20.698169 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:20.698624 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.698318 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:20.698624 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.698392 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:05:36.698372922 +0000 UTC m=+36.259217583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:20.799445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:20.799408 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:20.799604 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.799565 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:20.799646 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.799624 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:36.799610441 +0000 UTC m=+36.360455085 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:20.899986 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:20.899953 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:20.900175 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.900154 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:20.900231 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.900184 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:20.900231 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.900201 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:20.900330 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:20.900266 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:36.900246739 +0000 UTC m=+36.461091390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:21.074586 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:21.074511 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:21.074586 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:21.074526 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:21.074779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:21.074513 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:21.074779 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:21.074636 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:21.074779 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:21.074723 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:21.074894 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:21.074820 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:23.074953 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:23.074930 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:23.075498 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:23.075471 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:23.075592 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:23.075021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:23.075753 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:23.075735 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:23.075861 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:23.074986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:23.076031 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:23.076014 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:23.165281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:23.165253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" event={"ID":"d8d19fe14a09d468a2dde486d300cd1c","Type":"ContainerStarted","Data":"83582117b302b5188e640b85108c7ea90e51a3ba139f505cc26b40b2f97daaab"} Sep 18 16:05:23.183651 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:23.183594 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-41.ec2.internal" podStartSLOduration=19.183551892 podStartE2EDuration="19.183551892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:23.183434345 +0000 UTC m=+22.744279014" watchObservedRunningTime="2026-09-18 16:05:23.183551892 +0000 UTC m=+22.744396557" Sep 18 16:05:24.173402 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.172903 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s5l54" event={"ID":"28e3f91b-3eac-4bdc-b329-42da5929b082","Type":"ContainerStarted","Data":"b14b9e675d2fd3aa7354ae178cfa8dd73ecc3f222eea351afdd8d5a02d770b52"} Sep 18 16:05:24.174688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.174655 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zj9qd" event={"ID":"4b2534ae-985e-4a89-ba4b-99ca53d9a179","Type":"ContainerStarted","Data":"ed3de001db1d0a0060f96282d669c77dd49b3d3f1db4f13907e594e5f8228429"} Sep 18 16:05:24.179329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.177642 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bbmtn" event={"ID":"77789e19-a00a-403e-9556-65748730930e","Type":"ContainerStarted","Data":"5384e08f448d7695109ac99e9f2705b3b365fcbd96fa5d1cf0ae132ce98a9494"} Sep 18 16:05:24.179800 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.179774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" event={"ID":"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d","Type":"ContainerStarted","Data":"ff3f814b629f047ce998d03c266e0a85d953d88fa00e59360fc6267216f105c0"} Sep 18 16:05:24.181336 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.181314 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9z7rv" event={"ID":"2f9d4fd6-0082-4200-8fbb-099af79e209e","Type":"ContainerStarted","Data":"0477f8f10d0009c94813a352b296fb5195a4434f61317c7241397eaa66ccf8cd"} Sep 18 16:05:24.184717 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.184690 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"086cafab0b5e588e26839116b3069af67a8a17fc96ffe6c2b29f5b526fece5b7"} Sep 18 16:05:24.184796 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.184746 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"c64c194dfcd6ec94d6be7d48ee9436982496e202edfe63a97309b3547cd25ad7"} Sep 18 16:05:24.184796 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.184787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"f8c1584495772bcb5730da3bfff00a196af3bd2ae9a4c53ee1e481a884203aff"} Sep 18 16:05:24.184905 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.184837 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"216d6dc45e99786f65cdf58cdd36e4216d6e159153c6772efcc3394b55d5f2a6"} Sep 18 16:05:24.184905 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.184884 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"36f523a2efe9aed2741768b04a85d7cb9cf3760790a2c855d53886e341e8856a"} Sep 18 16:05:24.184991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.184912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"c73016f1cdff3181b722e8f938d12fe7a85e23696caddbe614f56012d0246046"} Sep 18 16:05:24.186326 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.186304 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qcxrf" event={"ID":"acc1a685-72d0-471f-808d-22cfc451dee0","Type":"ContainerStarted","Data":"67e2f04c685447ad2949f74a263491c1171f99d3596d1819568db79e20b64589"} Sep 18 16:05:24.188047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.188022 2563 generic.go:356] "Generic (PLEG): container finished" podID="91e4adb2-6caf-4bcb-9b16-bb53b68e1162" containerID="f47692259e36e32cfd35da22e4d0678977da084c2480b234c9058a847f22d670" exitCode=0 Sep 18 16:05:24.188163 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.188134 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerDied","Data":"f47692259e36e32cfd35da22e4d0678977da084c2480b234c9058a847f22d670"} Sep 18 16:05:24.190569 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.190532 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s5l54" podStartSLOduration=4.661198461 podStartE2EDuration="21.190516891s" totalImagesPullingTime="16.52931843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:24.190471639 +0000 UTC m=+23.751316310" watchObservedRunningTime="2026-09-18 16:05:24.190516891 +0000 UTC m=+23.751361555" Sep 18 16:05:24.205771 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.205731 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qcxrf" podStartSLOduration=3.775603267 podStartE2EDuration="21.205716884s" totalImagesPullingTime="17.430113617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:24.205555182 +0000 UTC m=+23.766399860" watchObservedRunningTime="2026-09-18 16:05:24.205716884 +0000 UTC m=+23.766561551" Sep 18 16:05:24.225713 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.225665 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zj9qd" podStartSLOduration=3.458219655 podStartE2EDuration="21.22564973s" totalImagesPullingTime="17.767430075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:24.225583461 +0000 UTC m=+23.786428129" watchObservedRunningTime="2026-09-18 16:05:24.22564973 +0000 UTC m=+23.786494398" Sep 18 16:05:24.275532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.275481 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9z7rv" podStartSLOduration=4.730928169 podStartE2EDuration="21.275464908s" totalImagesPullingTime="16.544536739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:24.25356774 +0000 UTC m=+23.814412408" watchObservedRunningTime="2026-09-18 16:05:24.275464908 +0000 UTC m=+23.836309575" Sep 18 16:05:24.289541 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.289487 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bbmtn" podStartSLOduration=3.853857025 podStartE2EDuration="21.289467138s" totalImagesPullingTime="17.435610113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:24.288716223 +0000 UTC m=+23.849560891" watchObservedRunningTime="2026-09-18 16:05:24.289467138 +0000 UTC m=+23.850311806" Sep 18 16:05:24.388274 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:24.388250 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 16:05:25.067449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.067342 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:05:24.388268444Z","UUID":"9d746a2d-86c6-4517-b540-088fbf558614","Handler":null,"Name":"","Endpoint":""} Sep 18 16:05:25.068926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.068901 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 16:05:25.069038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.068935 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 16:05:25.074394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.074370 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:25.074514 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:25.074487 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:25.074579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.074535 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:25.074634 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.074617 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:25.074634 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:25.074624 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:25.074732 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:25.074716 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:25.191823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.191779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8dxgl" event={"ID":"99b914e1-c1d7-4f98-afdf-ad2a58a52309","Type":"ContainerStarted","Data":"4b6e44eca63a2cdbcd7e3b1bac456766887bd6786f081d6c3f7e26cd29a40e0d"} Sep 18 16:05:25.193934 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.193900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" event={"ID":"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d","Type":"ContainerStarted","Data":"8982368c29b9e9251fed45cc2a69fc481227b66de1660784459f5f73699528e1"} Sep 18 16:05:25.252256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.252220 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:25.252995 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.252954 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:25.268345 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:25.268294 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8dxgl" podStartSLOduration=4.865073035 podStartE2EDuration="22.268278954s" totalImagesPullingTime="17.403205919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.20581903 +0000 UTC m=+24.766663692" watchObservedRunningTime="2026-09-18 16:05:25.268278954 +0000 UTC m=+24.829123618" Sep 18 16:05:26.199017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:26.198986 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"2c97c3011d4161f0bde02b60a3868c269f686130ba56315885a18b5db53c71be"} Sep 18 16:05:26.200841 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:26.200810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" event={"ID":"28caa9fb-e1a1-4ef4-98be-a47e55a2fc8d","Type":"ContainerStarted","Data":"274119f515c4309be2265606bdb9a6a69e8b27b9da2876f36a75258792098475"} Sep 18 16:05:26.201110 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:26.201076 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:26.201555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:26.201540 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9z7rv" Sep 18 16:05:26.220027 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:26.219986 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-89mj4" podStartSLOduration=3.288828033 podStartE2EDuration="23.219975183s" totalImagesPullingTime="19.93114715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:26.21950455 +0000 UTC m=+25.780349217" watchObservedRunningTime="2026-09-18 16:05:26.219975183 +0000 UTC m=+25.780819851" Sep 18 16:05:27.074102 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:27.074064 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:27.074288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:27.074065 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:27.074288 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:27.074181 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:27.074406 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:27.074285 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:27.074406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:27.074069 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:27.074406 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:27.074377 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:29.074231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.074050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:29.074671 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.074050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:29.074671 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:29.074347 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:29.074671 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.074122 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:29.074854 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:29.074817 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:29.074980 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:29.074958 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:29.211494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.211461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" event={"ID":"2aea08e0-a545-4ddb-bafa-d1a277968ade","Type":"ContainerStarted","Data":"b895354a3bf4f63a10a55c27dc9a4dc9a4993e1172e3bb51c7f5174b0bc2b05d"} Sep 18 16:05:29.211750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.211719 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:29.211874 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.211787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:29.211874 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.211829 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:29.226827 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.226802 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:29.226967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.226923 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:05:29.241756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:29.241698 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" podStartSLOduration=8.518691363 podStartE2EDuration="26.241680797s" totalImagesPullingTime="17.722989434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:29.241377317 +0000 UTC m=+28.802221984" watchObservedRunningTime="2026-09-18 16:05:29.241680797 +0000 UTC m=+28.802525465" Sep 18 16:05:30.215471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:30.215433 2563 generic.go:356] "Generic (PLEG): container finished" podID="91e4adb2-6caf-4bcb-9b16-bb53b68e1162" containerID="5dfead5527c65626e6f1c450171d9871f9423b55903434bc89183a486178c884" exitCode=0 Sep 18 16:05:30.216166 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:30.215522 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerDied","Data":"5dfead5527c65626e6f1c450171d9871f9423b55903434bc89183a486178c884"} Sep 18 16:05:31.035339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:31.035131 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wsw6t"] Sep 18 16:05:31.035595 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:31.035443 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:31.035595 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:31.035551 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:31.037750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:31.037717 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4j77c"] Sep 18 16:05:31.037876 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:31.037832 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:31.037960 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:31.037939 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:31.048370 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:31.048348 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r2z6d"] Sep 18 16:05:31.048483 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:31.048439 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:31.048536 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:31.048516 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:32.221907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:32.221877 2563 generic.go:356] "Generic (PLEG): container finished" podID="91e4adb2-6caf-4bcb-9b16-bb53b68e1162" containerID="35e4d3c6f1cafeef104b4cef790ebeefe428647133b6326c384e43837d26ae67" exitCode=0 Sep 18 16:05:32.222359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:32.221939 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerDied","Data":"35e4d3c6f1cafeef104b4cef790ebeefe428647133b6326c384e43837d26ae67"} Sep 18 16:05:33.075686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:33.075652 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:33.075847 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:33.075741 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:33.075847 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:33.075813 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:33.075925 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:33.075908 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:33.075970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:33.075959 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:33.076043 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:33.076028 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:34.227975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:34.227943 2563 generic.go:356] "Generic (PLEG): container finished" podID="91e4adb2-6caf-4bcb-9b16-bb53b68e1162" containerID="a688248057a35028d5414d2ec3a86bdde95cafed03188e0caa717b06063c3dbd" exitCode=0 Sep 18 16:05:34.228460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:34.227997 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerDied","Data":"a688248057a35028d5414d2ec3a86bdde95cafed03188e0caa717b06063c3dbd"} Sep 18 16:05:35.074311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:35.074282 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:35.074492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:35.074285 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:35.074492 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:35.074404 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r2z6d" podUID="62f743f6-9ca7-41f0-85cf-4266d18a1f78" Sep 18 16:05:35.074492 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:35.074485 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4j77c" podUID="d47181ff-68b6-410e-827d-2f4de7db7d7b" Sep 18 16:05:35.074667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:35.074292 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:35.074667 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:35.074564 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wsw6t" podUID="57ed03d9-d2bd-4fa3-8f77-cb3848b92923" Sep 18 16:05:36.222609 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.222578 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-41.ec2.internal" event="NodeReady" Sep 18 16:05:36.222990 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.222692 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 16:05:36.258219 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.258182 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c757cc57c-4tsdg"] Sep 18 16:05:36.278104 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.278056 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zkpmd"] Sep 18 16:05:36.278260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.278182 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.281266 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.280975 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 16:05:36.281420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.281269 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bxqxp\"" Sep 18 16:05:36.281420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.281308 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 16:05:36.282774 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.282480 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 16:05:36.291419 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.291252 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 16:05:36.300075 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.300051 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c757cc57c-4tsdg"] Sep 18 16:05:36.300203 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.300096 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-krt6v"] Sep 18 16:05:36.300203 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.300190 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.302689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.302634 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 16:05:36.302999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.302950 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 16:05:36.302999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.302975 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\"" Sep 18 16:05:36.325629 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.325592 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zkpmd"] Sep 18 16:05:36.325629 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.325637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-krt6v"] Sep 18 16:05:36.325841 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.325716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.328271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.328250 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69r9k\"" Sep 18 16:05:36.328401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.328319 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 16:05:36.328401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.328319 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 16:05:36.328499 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.328250 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 16:05:36.408588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408537 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c18c74-c0ae-4cd9-a6af-9733861063b3-tmp-dir\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.408588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6xk\" (UniqueName: \"kubernetes.io/projected/a2730fdd-0e9f-4a38-8eeb-24c40fc83f02-kube-api-access-5m6xk\") pod \"ingress-canary-krt6v\" (UID: \"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02\") " pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.408804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408614 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c18c74-c0ae-4cd9-a6af-9733861063b3-config-volume\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.408804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/804f9e66-c2ea-4cd5-b1ae-92311ce42617-ca-trust-extracted\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.408804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-installation-pull-secrets\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.408804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhsz\" (UniqueName: \"kubernetes.io/projected/f5c18c74-c0ae-4cd9-a6af-9733861063b3-kube-api-access-jqhsz\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.408804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-certificates\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408832 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-trusted-ca\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtlb\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-kube-api-access-4gtlb\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-bound-sa-token\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-image-registry-private-configuration\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.409015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.408950 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2730fdd-0e9f-4a38-8eeb-24c40fc83f02-cert\") pod \"ingress-canary-krt6v\" (UID: \"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02\") " pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.509336 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-image-registry-private-configuration\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2730fdd-0e9f-4a38-8eeb-24c40fc83f02-cert\") pod \"ingress-canary-krt6v\" (UID: \"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02\") " pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.509513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c18c74-c0ae-4cd9-a6af-9733861063b3-tmp-dir\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.509513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6xk\" (UniqueName: \"kubernetes.io/projected/a2730fdd-0e9f-4a38-8eeb-24c40fc83f02-kube-api-access-5m6xk\") pod \"ingress-canary-krt6v\" (UID: \"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02\") " pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.509513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c18c74-c0ae-4cd9-a6af-9733861063b3-config-volume\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.509513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509446 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/804f9e66-c2ea-4cd5-b1ae-92311ce42617-ca-trust-extracted\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509497 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-installation-pull-secrets\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhsz\" (UniqueName: \"kubernetes.io/projected/f5c18c74-c0ae-4cd9-a6af-9733861063b3-kube-api-access-jqhsz\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509581 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509640 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-certificates\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509663 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-trusted-ca\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtlb\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-kube-api-access-4gtlb\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-bound-sa-token\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.509786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.509752 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5c18c74-c0ae-4cd9-a6af-9733861063b3-tmp-dir\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.510171 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.510130 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/804f9e66-c2ea-4cd5-b1ae-92311ce42617-ca-trust-extracted\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.510662 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.510466 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c18c74-c0ae-4cd9-a6af-9733861063b3-config-volume\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.510662 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.510477 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:05:36.510662 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.510549 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:05:36.510662 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.510565 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:05:36.510662 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.510613 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:37.010591719 +0000 UTC m=+36.571436368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:05:36.510662 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.510633 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:37.010624023 +0000 UTC m=+36.571468667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:05:36.511313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.511289 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-certificates\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.511535 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.511511 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-trusted-ca\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.514451 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.514430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-installation-pull-secrets\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.514554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.514449 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2730fdd-0e9f-4a38-8eeb-24c40fc83f02-cert\") pod \"ingress-canary-krt6v\" (UID: \"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02\") " pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.514554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.514451 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-image-registry-private-configuration\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.522077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.522050 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtlb\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-kube-api-access-4gtlb\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.522281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.522242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6xk\" (UniqueName: \"kubernetes.io/projected/a2730fdd-0e9f-4a38-8eeb-24c40fc83f02-kube-api-access-5m6xk\") pod \"ingress-canary-krt6v\" (UID: \"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02\") " pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.523018 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.522994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhsz\" (UniqueName: \"kubernetes.io/projected/f5c18c74-c0ae-4cd9-a6af-9733861063b3-kube-api-access-jqhsz\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:36.523313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.523295 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-bound-sa-token\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:36.635306 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.635269 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-krt6v" Sep 18 16:05:36.711778 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.711733 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:36.711953 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.711895 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:36.712024 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.711983 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:08.711964462 +0000 UTC m=+68.272809122 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 16:05:36.812757 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.812715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:36.812946 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.812840 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:36.812946 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.812914 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret podName:62f743f6-9ca7-41f0-85cf-4266d18a1f78 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:08.812898633 +0000 UTC m=+68.373743278 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret") pod "global-pull-secret-syncer-r2z6d" (UID: "62f743f6-9ca7-41f0-85cf-4266d18a1f78") : object "kube-system"/"original-pull-secret" not registered Sep 18 16:05:36.913938 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.913906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:36.914214 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.914183 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 16:05:36.914214 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.914214 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 16:05:36.914315 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.914228 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d6fbp for pod openshift-network-diagnostics/network-check-target-wsw6t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:36.914315 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:36.914283 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp podName:57ed03d9-d2bd-4fa3-8f77-cb3848b92923 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:08.914270133 +0000 UTC m=+68.475114778 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d6fbp" (UniqueName: "kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp") pod "network-check-target-wsw6t" (UID: "57ed03d9-d2bd-4fa3-8f77-cb3848b92923") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 16:05:36.987613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:36.987577 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-krt6v"] Sep 18 16:05:36.990862 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:05:36.990835 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2730fdd_0e9f_4a38_8eeb_24c40fc83f02.slice/crio-ff4a5215c36808c2c017dfff224d21e9593f0fe19a8a23f691ce1ad1daa4af65 WatchSource:0}: Error finding container ff4a5215c36808c2c017dfff224d21e9593f0fe19a8a23f691ce1ad1daa4af65: Status 404 returned error can't find the container with id ff4a5215c36808c2c017dfff224d21e9593f0fe19a8a23f691ce1ad1daa4af65 Sep 18 16:05:37.015068 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.014979 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:37.015068 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.015054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:37.015289 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:37.015105 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:05:37.015289 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:37.015149 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:05:37.015289 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:37.015160 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:05:37.015289 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:37.015182 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:38.015151434 +0000 UTC m=+37.575996092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:05:37.015289 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:37.015206 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:38.015190705 +0000 UTC m=+37.576035354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:05:37.074230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.074193 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:05:37.074390 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.074193 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:05:37.074471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.074204 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:05:37.077293 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.077267 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:05:37.077447 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.077319 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\"" Sep 18 16:05:37.077447 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.077350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 16:05:37.077447 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.077267 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:05:37.077626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.077610 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\"" Sep 18 16:05:37.077626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.077621 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 16:05:37.238327 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:37.238284 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-krt6v" event={"ID":"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02","Type":"ContainerStarted","Data":"ff4a5215c36808c2c017dfff224d21e9593f0fe19a8a23f691ce1ad1daa4af65"} Sep 18 16:05:38.024331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:38.024285 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:38.024538 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:38.024371 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:38.024538 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:38.024472 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:05:38.024538 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:38.024502 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:05:38.024538 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:38.024517 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:05:38.024720 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:38.024552 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:40.024531419 +0000 UTC m=+39.585376081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:05:38.024720 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:38.024575 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:40.024563205 +0000 UTC m=+39.585407859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:05:40.041316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:40.041272 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:40.041776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:40.041385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:40.041776 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:40.041435 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:05:40.041776 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:40.041454 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:05:40.041776 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:40.041481 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:05:40.041776 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:40.041520 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:44.041500916 +0000 UTC m=+43.602345575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:05:40.041776 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:40.041533 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:44.041527504 +0000 UTC m=+43.602372149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:05:40.249377 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:40.249195 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerStarted","Data":"9a7388e2d5fc25048a342eec4301cacf120a3699c4c6f6f7f3b643d15ff2c288"} Sep 18 16:05:41.254482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.254446 2563 generic.go:356] "Generic (PLEG): container finished" podID="91e4adb2-6caf-4bcb-9b16-bb53b68e1162" containerID="9a7388e2d5fc25048a342eec4301cacf120a3699c4c6f6f7f3b643d15ff2c288" exitCode=0 Sep 18 16:05:41.254895 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.254529 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerDied","Data":"9a7388e2d5fc25048a342eec4301cacf120a3699c4c6f6f7f3b643d15ff2c288"} Sep 18 16:05:41.256362 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.256323 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-krt6v" event={"ID":"a2730fdd-0e9f-4a38-8eeb-24c40fc83f02","Type":"ContainerStarted","Data":"5ad768a93b9481bb3f543744a203fb4a7a4a898998635980b1820b3041a17845"} Sep 18 16:05:41.291080 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.291023 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-krt6v" podStartSLOduration=1.98705544 podStartE2EDuration="5.291004444s" totalImagesPullingTime="3.303949004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:41.290667204 +0000 UTC m=+40.851511873" watchObservedRunningTime="2026-09-18 16:05:41.291004444 +0000 UTC m=+40.851849111" Sep 18 16:05:41.423304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.423251 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:41.529465 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.529391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:41.933276 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:41.933202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:42.033364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:42.033333 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:42.264805 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:42.264772 2563 generic.go:356] "Generic (PLEG): container finished" podID="91e4adb2-6caf-4bcb-9b16-bb53b68e1162" containerID="26d15b66a8f9ae45d265b0d4c10c0054ab35f06b1f1663102a21a5b8c8514783" exitCode=0 Sep 18 16:05:42.265165 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:42.264854 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerDied","Data":"26d15b66a8f9ae45d265b0d4c10c0054ab35f06b1f1663102a21a5b8c8514783"} Sep 18 16:05:42.923379 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:42.923333 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:42.978584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:42.978553 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:43.270908 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:43.270876 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phmtm" event={"ID":"91e4adb2-6caf-4bcb-9b16-bb53b68e1162","Type":"ContainerStarted","Data":"0f7aa85fa80bd13a84f44ac637a5db3789adc8c77701dde64acb67b3a5998834"} Sep 18 16:05:43.294996 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:43.294940 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phmtm" podStartSLOduration=8.746674627 podStartE2EDuration="40.294921142s" totalImagesPullingTime="31.548246515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:43.293180158 +0000 UTC m=+42.854024825" watchObservedRunningTime="2026-09-18 16:05:43.294921142 +0000 UTC m=+42.855765809" Sep 18 16:05:43.419549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:43.419490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:43.489136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:43.489102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:44.076434 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:44.076397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:44.076603 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:44.076450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:44.076603 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:44.076548 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:05:44.076686 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:44.076615 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:52.076599025 +0000 UTC m=+51.637443683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:05:44.076686 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:44.076549 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:05:44.076686 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:44.076652 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:05:44.076867 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:44.076700 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:52.076687069 +0000 UTC m=+51.637531714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:05:44.660637 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:44.660605 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:44.719427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:44.719397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:45.605299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:45.605261 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:45.719030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:45.718995 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:46.186768 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:46.186734 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:05:52.133812 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:52.133764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:05:52.133812 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:05:52.133822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:05:52.134278 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:52.133920 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:05:52.134278 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:52.133922 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:05:52.134278 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:52.133988 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:08.133972333 +0000 UTC m=+67.694816978 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:05:52.134278 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:52.133930 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:05:52.134278 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:05:52.134058 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:08.134043701 +0000 UTC m=+67.694888346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:06:01.237239 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:01.237213 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r7hqn" Sep 18 16:06:01.392924 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:01.392892 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:01.452021 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:01.451990 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:02.931313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:02.931281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:02.992843 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:02.992814 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:06.653408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:06.653369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:06.716379 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:06.716352 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:08.156327 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.156288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:06:08.156771 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.156353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:06:08.156771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.156431 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:06:08.156771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.156495 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:40.15647932 +0000 UTC m=+99.717323964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:06:08.156771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.156495 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:06:08.156771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.156515 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:06:08.156771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.156569 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:40.156552812 +0000 UTC m=+99.717397458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:06:08.762063 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.762013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:06:08.764974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.764953 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:06:08.773059 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.773039 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 16:06:08.773127 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:08.773118 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs podName:d47181ff-68b6-410e-827d-2f4de7db7d7b nodeName:}" failed. No retries permitted until 2026-09-18 16:07:12.773097327 +0000 UTC m=+132.333941985 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs") pod "network-metrics-daemon-4j77c" (UID: "d47181ff-68b6-410e-827d-2f4de7db7d7b") : secret "metrics-daemon-secret" not found Sep 18 16:06:08.863225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.863178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:06:08.865861 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.865844 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:06:08.876741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.876721 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62f743f6-9ca7-41f0-85cf-4266d18a1f78-original-pull-secret\") pod \"global-pull-secret-syncer-r2z6d\" (UID: \"62f743f6-9ca7-41f0-85cf-4266d18a1f78\") " pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:06:08.894188 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.894153 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r2z6d" Sep 18 16:06:08.964576 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.964543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:06:08.967458 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.967441 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 16:06:08.976975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.976956 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 16:06:08.987432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:08.987402 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fbp\" (UniqueName: \"kubernetes.io/projected/57ed03d9-d2bd-4fa3-8f77-cb3848b92923-kube-api-access-d6fbp\") pod \"network-check-target-wsw6t\" (UID: \"57ed03d9-d2bd-4fa3-8f77-cb3848b92923\") " pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:06:09.161867 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:09.161787 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r2z6d"] Sep 18 16:06:09.166625 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:06:09.166602 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f743f6_9ca7_41f0_85cf_4266d18a1f78.slice/crio-46ce98a69285ee460f25c0ab12ae0400983c58264ac085a975e3db8994ebdb1c WatchSource:0}: Error finding container 46ce98a69285ee460f25c0ab12ae0400983c58264ac085a975e3db8994ebdb1c: Status 404 returned error can't find the container with id 46ce98a69285ee460f25c0ab12ae0400983c58264ac085a975e3db8994ebdb1c Sep 18 16:06:09.203926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:09.203890 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\"" Sep 18 16:06:09.212240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:09.212216 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:06:09.334800 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:09.334753 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r2z6d" event={"ID":"62f743f6-9ca7-41f0-85cf-4266d18a1f78","Type":"ContainerStarted","Data":"46ce98a69285ee460f25c0ab12ae0400983c58264ac085a975e3db8994ebdb1c"} Sep 18 16:06:09.477687 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:09.477605 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wsw6t"] Sep 18 16:06:09.480484 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:06:09.480460 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ed03d9_d2bd_4fa3_8f77_cb3848b92923.slice/crio-9a118957fce6ed7f4c99359e7204112e2741f84796e9a6de861fd5deddf300fe WatchSource:0}: Error finding container 9a118957fce6ed7f4c99359e7204112e2741f84796e9a6de861fd5deddf300fe: Status 404 returned error can't find the container with id 9a118957fce6ed7f4c99359e7204112e2741f84796e9a6de861fd5deddf300fe Sep 18 16:06:10.339173 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:10.339136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wsw6t" event={"ID":"57ed03d9-d2bd-4fa3-8f77-cb3848b92923","Type":"ContainerStarted","Data":"9a118957fce6ed7f4c99359e7204112e2741f84796e9a6de861fd5deddf300fe"} Sep 18 16:06:15.354158 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:15.354117 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r2z6d" event={"ID":"62f743f6-9ca7-41f0-85cf-4266d18a1f78","Type":"ContainerStarted","Data":"811e7dd3d1f1b60e7c9f7524ac84eb6db639d1a7f88d7285828b76440dc53d64"} Sep 18 16:06:15.355564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:15.355541 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wsw6t" event={"ID":"57ed03d9-d2bd-4fa3-8f77-cb3848b92923","Type":"ContainerStarted","Data":"7ad16cb50db91f5cd46f7947ee4372671ab09a79f3a4f3fc0b7e5e40215db2c7"} Sep 18 16:06:15.355681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:15.355664 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:06:15.370438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:15.370380 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r2z6d" podStartSLOduration=66.720424332 podStartE2EDuration="1m12.370360636s" totalImagesPullingTime="5.649936304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:15.369962418 +0000 UTC m=+74.930807084" watchObservedRunningTime="2026-09-18 16:06:15.370360636 +0000 UTC m=+74.931205304" Sep 18 16:06:15.387074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:15.387027 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wsw6t" podStartSLOduration=67.061505464 podStartE2EDuration="1m12.387014302s" totalImagesPullingTime="5.325508838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:15.386909159 +0000 UTC m=+74.947753825" watchObservedRunningTime="2026-09-18 16:06:15.387014302 +0000 UTC m=+74.947858968" Sep 18 16:06:16.434904 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:16.434868 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:17.649933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:17.649887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:40.179788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:40.179727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:06:40.180212 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:40.179813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:06:40.180212 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:40.179893 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 16:06:40.180212 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:40.179915 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 16:06:40.180212 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:40.179926 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c757cc57c-4tsdg: secret "image-registry-tls" not found Sep 18 16:06:40.180212 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:40.179968 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls podName:f5c18c74-c0ae-4cd9-a6af-9733861063b3 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:44.179948455 +0000 UTC m=+163.740793123 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls") pod "dns-default-zkpmd" (UID: "f5c18c74-c0ae-4cd9-a6af-9733861063b3") : secret "dns-default-metrics-tls" not found Sep 18 16:06:40.180212 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:40.179984 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls podName:804f9e66-c2ea-4cd5-b1ae-92311ce42617 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:44.179977868 +0000 UTC m=+163.740822513 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls") pod "image-registry-c757cc57c-4tsdg" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617") : secret "image-registry-tls" not found Sep 18 16:06:45.660537 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.660506 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-z2x82"] Sep 18 16:06:45.662155 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.662139 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.678258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.678239 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 18 16:06:45.683998 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.683980 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nkc54\"" Sep 18 16:06:45.688366 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.688350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 18 16:06:45.688465 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.688450 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 16:06:45.688558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.688544 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 16:06:45.698998 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.698977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-z2x82"] Sep 18 16:06:45.716180 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.716158 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 18 16:06:45.759640 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.759616 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:45.762435 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.762414 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jd8k8"] Sep 18 16:06:45.764020 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.764003 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:45.766636 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.766612 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 18 16:06:45.766745 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.766651 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 18 16:06:45.766745 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.766662 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 18 16:06:45.766859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.766760 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:06:45.767007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.766992 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kthwg\"" Sep 18 16:06:45.772598 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.772574 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 18 16:06:45.777750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.777716 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jd8k8"] Sep 18 16:06:45.819300 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.819278 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85d4feac-2eae-4628-a1c5-ba2ae60eb119-serving-cert\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.819384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.819327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85d4feac-2eae-4628-a1c5-ba2ae60eb119-snapshots\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.819427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.819401 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d4feac-2eae-4628-a1c5-ba2ae60eb119-tmp\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.819460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.819429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85d4feac-2eae-4628-a1c5-ba2ae60eb119-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.819460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.819451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85d4feac-2eae-4628-a1c5-ba2ae60eb119-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.819520 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.819465 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxr7\" (UniqueName: \"kubernetes.io/projected/85d4feac-2eae-4628-a1c5-ba2ae60eb119-kube-api-access-hvxr7\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.826241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.826216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:45.919862 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.919810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85d4feac-2eae-4628-a1c5-ba2ae60eb119-serving-cert\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.919862 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.919836 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13e5f050-0e55-4386-8e80-8927af0ec4d3-serving-cert\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:45.919862 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.919856 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cd9b\" (UniqueName: \"kubernetes.io/projected/13e5f050-0e55-4386-8e80-8927af0ec4d3-kube-api-access-9cd9b\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:45.919999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.919902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85d4feac-2eae-4628-a1c5-ba2ae60eb119-snapshots\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.919999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.919926 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13e5f050-0e55-4386-8e80-8927af0ec4d3-trusted-ca\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:45.919999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.919981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d4feac-2eae-4628-a1c5-ba2ae60eb119-tmp\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85d4feac-2eae-4628-a1c5-ba2ae60eb119-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13e5f050-0e55-4386-8e80-8927af0ec4d3-config\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:45.920136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920047 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85d4feac-2eae-4628-a1c5-ba2ae60eb119-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920071 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxr7\" (UniqueName: \"kubernetes.io/projected/85d4feac-2eae-4628-a1c5-ba2ae60eb119-kube-api-access-hvxr7\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920612 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920587 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85d4feac-2eae-4628-a1c5-ba2ae60eb119-tmp\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920624 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85d4feac-2eae-4628-a1c5-ba2ae60eb119-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920591 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85d4feac-2eae-4628-a1c5-ba2ae60eb119-snapshots\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.920966 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.920949 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85d4feac-2eae-4628-a1c5-ba2ae60eb119-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.922180 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.922164 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85d4feac-2eae-4628-a1c5-ba2ae60eb119-serving-cert\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.927842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.927822 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxr7\" (UniqueName: \"kubernetes.io/projected/85d4feac-2eae-4628-a1c5-ba2ae60eb119-kube-api-access-hvxr7\") pod \"insights-operator-6f9c644dd9-z2x82\" (UID: \"85d4feac-2eae-4628-a1c5-ba2ae60eb119\") " pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:45.969801 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:45.969780 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" Sep 18 16:06:46.020715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.020686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13e5f050-0e55-4386-8e80-8927af0ec4d3-serving-cert\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.020833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.020731 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cd9b\" (UniqueName: \"kubernetes.io/projected/13e5f050-0e55-4386-8e80-8927af0ec4d3-kube-api-access-9cd9b\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.020883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.020866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13e5f050-0e55-4386-8e80-8927af0ec4d3-trusted-ca\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.020958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.020943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13e5f050-0e55-4386-8e80-8927af0ec4d3-config\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.021957 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.021750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13e5f050-0e55-4386-8e80-8927af0ec4d3-config\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.021957 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.021909 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13e5f050-0e55-4386-8e80-8927af0ec4d3-trusted-ca\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.023442 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.023422 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13e5f050-0e55-4386-8e80-8927af0ec4d3-serving-cert\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.034050 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.032941 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cd9b\" (UniqueName: \"kubernetes.io/projected/13e5f050-0e55-4386-8e80-8927af0ec4d3-kube-api-access-9cd9b\") pod \"console-operator-fd4bb46c-jd8k8\" (UID: \"13e5f050-0e55-4386-8e80-8927af0ec4d3\") " pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.073473 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.073446 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:46.235026 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.235001 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-z2x82"] Sep 18 16:06:46.237628 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:06:46.237600 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d4feac_2eae_4628_a1c5_ba2ae60eb119.slice/crio-a02783b799c2ac50b807b7ef245e7ca5865616b3e2ec5473872bad0f81f54e53 WatchSource:0}: Error finding container a02783b799c2ac50b807b7ef245e7ca5865616b3e2ec5473872bad0f81f54e53: Status 404 returned error can't find the container with id a02783b799c2ac50b807b7ef245e7ca5865616b3e2ec5473872bad0f81f54e53 Sep 18 16:06:46.340687 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.340659 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-jd8k8"] Sep 18 16:06:46.343184 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:06:46.343164 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e5f050_0e55_4386_8e80_8927af0ec4d3.slice/crio-cae95c71874d72d1d927f7e9932a40565ae22dea082b43add8a14d81581729a3 WatchSource:0}: Error finding container cae95c71874d72d1d927f7e9932a40565ae22dea082b43add8a14d81581729a3: Status 404 returned error can't find the container with id cae95c71874d72d1d927f7e9932a40565ae22dea082b43add8a14d81581729a3 Sep 18 16:06:46.361141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.361121 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wsw6t" Sep 18 16:06:46.432774 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.432741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" event={"ID":"85d4feac-2eae-4628-a1c5-ba2ae60eb119","Type":"ContainerStarted","Data":"a02783b799c2ac50b807b7ef245e7ca5865616b3e2ec5473872bad0f81f54e53"} Sep 18 16:06:46.433885 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:46.433852 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"cae95c71874d72d1d927f7e9932a40565ae22dea082b43add8a14d81581729a3"} Sep 18 16:06:47.832963 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:47.832911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:49.747532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:49.747504 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:49.806196 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:49.806174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:50.176729 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.176699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:50.228099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.228060 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:50.333723 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.333693 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:50.410270 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.410250 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:50.445122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.445043 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" event={"ID":"85d4feac-2eae-4628-a1c5-ba2ae60eb119","Type":"ContainerStarted","Data":"0a94ef3145e2455212f160686e0e7ad8bba21b31e8f45bf3befb58d576cace81"} Sep 18 16:06:50.446680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.446659 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/0.log" Sep 18 16:06:50.446784 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.446698 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="93f24cdadafdb227770ee11df23a43887fa5c3ae77e012e04c5c6e685074dc78" exitCode=255 Sep 18 16:06:50.446784 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.446749 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"93f24cdadafdb227770ee11df23a43887fa5c3ae77e012e04c5c6e685074dc78"} Sep 18 16:06:50.446957 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.446943 2563 scope.go:117] "RemoveContainer" containerID="93f24cdadafdb227770ee11df23a43887fa5c3ae77e012e04c5c6e685074dc78" Sep 18 16:06:50.463434 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.463290 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" podStartSLOduration=2.016181722 podStartE2EDuration="5.463273898s" totalImagesPullingTime="3.447092176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:50.463183296 +0000 UTC m=+110.024027965" watchObservedRunningTime="2026-09-18 16:06:50.463273898 +0000 UTC m=+110.024118566" Sep 18 16:06:50.483907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:50.483886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:51.452923 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.452894 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:06:51.453472 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.453452 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/0.log" Sep 18 16:06:51.453564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.453491 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d" exitCode=255 Sep 18 16:06:51.453615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.453557 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d"} Sep 18 16:06:51.453659 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.453632 2563 scope.go:117] "RemoveContainer" containerID="93f24cdadafdb227770ee11df23a43887fa5c3ae77e012e04c5c6e685074dc78" Sep 18 16:06:51.454253 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.454013 2563 scope.go:117] "RemoveContainer" containerID="a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d" Sep 18 16:06:51.454380 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:51.454247 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:06:51.484748 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.484720 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qhb9c"] Sep 18 16:06:51.487229 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.487214 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.489993 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.489977 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 16:06:51.490111 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.489980 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 16:06:51.490179 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.490129 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\"" Sep 18 16:06:51.501942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.501922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhb9c"] Sep 18 16:06:51.505858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.505840 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:51.564188 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.564162 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0325a6c-18db-43f3-b375-7e95e3555ecf-data-volume\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.564295 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.564242 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.564410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.564389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0325a6c-18db-43f3-b375-7e95e3555ecf-crio-socket\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.564502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.564485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbtw\" (UniqueName: \"kubernetes.io/projected/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-api-access-nxbtw\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.564566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.564534 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.582174 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.582153 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:51.664928 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.664898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.665049 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.664934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0325a6c-18db-43f3-b375-7e95e3555ecf-crio-socket\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.665049 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.664996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbtw\" (UniqueName: \"kubernetes.io/projected/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-api-access-nxbtw\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.665049 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:51.665023 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:06:51.665049 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.665036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.665277 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:51.665074 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:52.165058814 +0000 UTC m=+111.725903460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : secret "insights-runtime-extractor-tls" not found Sep 18 16:06:51.665277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.665080 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0325a6c-18db-43f3-b375-7e95e3555ecf-crio-socket\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.665277 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:51.665115 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:06:51.665277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.665115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0325a6c-18db-43f3-b375-7e95e3555ecf-data-volume\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.665277 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:51.665202 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:52.165186184 +0000 UTC m=+111.726030857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : configmap "kube-rbac-proxy" not found Sep 18 16:06:51.665820 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.665804 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0325a6c-18db-43f3-b375-7e95e3555ecf-data-volume\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:51.674212 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:51.674190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbtw\" (UniqueName: \"kubernetes.io/projected/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-api-access-nxbtw\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:52.084588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:52.084547 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:52.153768 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:52.153736 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:52.169777 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:52.169750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:52.169885 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:52.169866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:52.169959 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:52.169898 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:06:52.170013 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:52.169960 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.169942593 +0000 UTC m=+112.730787251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : secret "insights-runtime-extractor-tls" not found Sep 18 16:06:52.170013 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:52.169979 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:06:52.170135 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:52.170050 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.170033182 +0000 UTC m=+112.730877843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : configmap "kube-rbac-proxy" not found Sep 18 16:06:52.457292 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:52.457225 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:06:52.457613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:52.457549 2563 scope.go:117] "RemoveContainer" containerID="a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d" Sep 18 16:06:52.457708 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:52.457689 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:06:53.178218 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:53.178169 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:53.178411 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:53.178253 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:53.178411 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:53.178289 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:06:53.178411 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:53.178359 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:55.178341994 +0000 UTC m=+114.739186638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : configmap "kube-rbac-proxy" not found Sep 18 16:06:53.178411 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:53.178362 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:06:53.178411 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:53.178411 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:55.178396903 +0000 UTC m=+114.739241549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : secret "insights-runtime-extractor-tls" not found Sep 18 16:06:53.800690 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:53.800656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:53.880170 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:53.880142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:53.981016 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:53.980973 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:54.113669 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.113594 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:54.210655 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.210619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:54.245656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.245631 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bbmtn_77789e19-a00a-403e-9556-65748730930e/dns-node-resolver/0.log" Sep 18 16:06:54.274266 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.274237 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:54.339859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.339818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:54.467484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.467409 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:54.569693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:54.569660 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:55.195475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:55.195443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:55.195854 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:55.195507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:55.195854 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:55.195584 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:06:55.195854 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:55.195608 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:06:55.195854 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:55.195657 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:59.195639154 +0000 UTC m=+118.756483813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : configmap "kube-rbac-proxy" not found Sep 18 16:06:55.195854 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:55.195672 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:06:59.195666049 +0000 UTC m=+118.756510693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : secret "insights-runtime-extractor-tls" not found Sep 18 16:06:55.199917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:55.199894 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:55.446315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:55.446243 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qcxrf_acc1a685-72d0-471f-808d-22cfc451dee0/node-ca/0.log" Sep 18 16:06:55.609841 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:55.609811 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:56.020940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.020911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:56.045205 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.045183 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-krt6v_a2730fdd-0e9f-4a38-8eeb-24c40fc83f02/serve-healthcheck-canary/0.log" Sep 18 16:06:56.073611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.073591 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:56.073688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.073627 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:06:56.073937 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.073913 2563 scope.go:117] "RemoveContainer" containerID="a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d" Sep 18 16:06:56.074115 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:56.074076 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:06:56.074470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.074443 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:56.409195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.409112 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:56.609322 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:56.609294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:57.009454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:57.009420 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:57.416511 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:57.416430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:57.482982 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:57.482908 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:57.801233 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:57.801201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:58.009249 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:58.009219 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:58.608209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:58.608177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:58.661261 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:58.661232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:58.802882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:58.802848 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:59.209312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:59.209282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:59.228099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:59.228058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:59.228263 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:59.228165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:06:59.228263 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:59.228179 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 16:06:59.228263 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:59.228239 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:07:07.228222674 +0000 UTC m=+126.789067339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : secret "insights-runtime-extractor-tls" not found Sep 18 16:06:59.228263 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:59.228262 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 16:06:59.228448 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:06:59.228332 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm podName:a0325a6c-18db-43f3-b375-7e95e3555ecf nodeName:}" failed. No retries permitted until 2026-09-18 16:07:07.228315809 +0000 UTC m=+126.789160454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhb9c" (UID: "a0325a6c-18db-43f3-b375-7e95e3555ecf") : configmap "kube-rbac-proxy" not found Sep 18 16:06:59.407677 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:59.407644 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:06:59.804811 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:06:59.804780 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:00.409572 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:00.409542 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:00.804139 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:00.804079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:00.974675 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:00.974643 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:01.039611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:01.039583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:01.401893 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:01.401860 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:02.034058 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:02.034023 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:02.428676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:02.428594 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:02.805197 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:02.805164 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:03.205014 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:03.204934 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:03.407041 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:03.407007 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:03.505096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:03.505065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:03.804775 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:03.804695 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:04.091080 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:04.090996 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:04.187887 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:04.187846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:04.287788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:04.287755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:04.600678 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:04.600647 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:04.805283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:04.805253 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:05.214525 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:05.214493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:05.739121 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:05.739077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:07.294712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.294672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:07:07.295112 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.294728 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:07:07.295242 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.295222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0325a6c-18db-43f3-b375-7e95e3555ecf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:07:07.296973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.296955 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0325a6c-18db-43f3-b375-7e95e3555ecf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhb9c\" (UID: \"a0325a6c-18db-43f3-b375-7e95e3555ecf\") " pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:07:07.399453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.399423 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\"" Sep 18 16:07:07.406140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.406124 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhb9c" Sep 18 16:07:07.672198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:07.672170 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhb9c"] Sep 18 16:07:07.674658 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:07.674636 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0325a6c_18db_43f3_b375_7e95e3555ecf.slice/crio-7a282686c1b5d7dafcbeb53f64dbc16ee34424bb9dae70e56b745654e95ccbf5 WatchSource:0}: Error finding container 7a282686c1b5d7dafcbeb53f64dbc16ee34424bb9dae70e56b745654e95ccbf5: Status 404 returned error can't find the container with id 7a282686c1b5d7dafcbeb53f64dbc16ee34424bb9dae70e56b745654e95ccbf5 Sep 18 16:07:08.501844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:08.501808 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhb9c" event={"ID":"a0325a6c-18db-43f3-b375-7e95e3555ecf","Type":"ContainerStarted","Data":"263f7887915ae99473af12b6023a90eea637d53c6d34ec37c44b0abd5dc8d648"} Sep 18 16:07:08.502230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:08.501883 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhb9c" event={"ID":"a0325a6c-18db-43f3-b375-7e95e3555ecf","Type":"ContainerStarted","Data":"d569ab7f5feb85252edaaa91f3721b73ac718ffe09598ad1eade23830eced85c"} Sep 18 16:07:08.502230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:08.501902 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhb9c" event={"ID":"a0325a6c-18db-43f3-b375-7e95e3555ecf","Type":"ContainerStarted","Data":"7a282686c1b5d7dafcbeb53f64dbc16ee34424bb9dae70e56b745654e95ccbf5"} Sep 18 16:07:09.075303 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:09.075271 2563 scope.go:117] "RemoveContainer" containerID="a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d" Sep 18 16:07:09.506933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:09.506905 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:07:09.507453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:09.506969 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"456cd87ce21ac2ee19105a5790b3e6fd7365712ae68cced02849f8c345e4e76e"} Sep 18 16:07:09.507453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:09.507268 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:07:09.524395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:09.524339 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podStartSLOduration=21.178217269 podStartE2EDuration="24.524320806s" totalImagesPullingTime="3.346103537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:09.52377183 +0000 UTC m=+129.084616497" watchObservedRunningTime="2026-09-18 16:07:09.524320806 +0000 UTC m=+129.085165472" Sep 18 16:07:10.421170 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.421142 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:07:10.511593 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.511552 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhb9c" event={"ID":"a0325a6c-18db-43f3-b375-7e95e3555ecf","Type":"ContainerStarted","Data":"980cfc8023cadcc1064b8ab0b96bd64d25dda565ee647ce74264c0b9da6bcbde"} Sep 18 16:07:10.541425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.541358 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qhb9c" podStartSLOduration=17.588826 podStartE2EDuration="19.541336228s" totalImagesPullingTime="1.952510228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:06:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:10.53953057 +0000 UTC m=+130.100375244" watchObservedRunningTime="2026-09-18 16:07:10.541336228 +0000 UTC m=+130.102180899" Sep 18 16:07:10.601638 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.601599 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:10.699473 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.699387 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:10.781265 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.781236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:10.862183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.862149 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8"] Sep 18 16:07:10.864734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.864714 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:10.868051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.868032 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-m7msx\"" Sep 18 16:07:10.868278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.868265 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 18 16:07:10.883705 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.883677 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8"] Sep 18 16:07:10.924894 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.924860 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-44tzz"] Sep 18 16:07:10.925209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.925187 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40b404d1-9ae4-4b27-82e4-b08a0a3597cb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-4mxn8\" (UID: \"40b404d1-9ae4-4b27-82e4-b08a0a3597cb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:10.927801 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.927784 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:10.930411 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.930392 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zwprx\"" Sep 18 16:07:10.930951 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.930930 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 16:07:10.931036 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.930985 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 16:07:10.938001 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.937977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-44tzz"] Sep 18 16:07:10.961215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:10.961134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:11.025654 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.025618 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40b404d1-9ae4-4b27-82e4-b08a0a3597cb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-4mxn8\" (UID: \"40b404d1-9ae4-4b27-82e4-b08a0a3597cb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:11.025836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.025675 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4lmv\" (UniqueName: \"kubernetes.io/projected/ad5ffd53-9297-4327-901f-01a54926300c-kube-api-access-s4lmv\") pod \"downloads-6594f4c75-44tzz\" (UID: \"ad5ffd53-9297-4327-901f-01a54926300c\") " pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:11.027998 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.027979 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40b404d1-9ae4-4b27-82e4-b08a0a3597cb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-4mxn8\" (UID: \"40b404d1-9ae4-4b27-82e4-b08a0a3597cb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:11.034212 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.034187 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:11.126890 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.126850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4lmv\" (UniqueName: \"kubernetes.io/projected/ad5ffd53-9297-4327-901f-01a54926300c-kube-api-access-s4lmv\") pod \"downloads-6594f4c75-44tzz\" (UID: \"ad5ffd53-9297-4327-901f-01a54926300c\") " pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:11.137436 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.137409 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4lmv\" (UniqueName: \"kubernetes.io/projected/ad5ffd53-9297-4327-901f-01a54926300c-kube-api-access-s4lmv\") pod \"downloads-6594f4c75-44tzz\" (UID: \"ad5ffd53-9297-4327-901f-01a54926300c\") " pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:11.173506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.173477 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:11.237060 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.236864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:11.282708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.282670 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:11.383040 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.382998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:11.439758 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.439681 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8"] Sep 18 16:07:11.441662 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:11.441638 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b404d1_9ae4_4b27_82e4_b08a0a3597cb.slice/crio-e88df05ca75deee6212b21e0e2b796f18647fbad3db41f17b6b7438d3fecf3b7 WatchSource:0}: Error finding container e88df05ca75deee6212b21e0e2b796f18647fbad3db41f17b6b7438d3fecf3b7: Status 404 returned error can't find the container with id e88df05ca75deee6212b21e0e2b796f18647fbad3db41f17b6b7438d3fecf3b7 Sep 18 16:07:11.506228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.506199 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-44tzz"] Sep 18 16:07:11.509003 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:11.508977 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5ffd53_9297_4327_901f_01a54926300c.slice/crio-a61838e75304bfb3ccff4841d7170ab57b935b822aa647562df956354799b1d5 WatchSource:0}: Error finding container a61838e75304bfb3ccff4841d7170ab57b935b822aa647562df956354799b1d5: Status 404 returned error can't find the container with id a61838e75304bfb3ccff4841d7170ab57b935b822aa647562df956354799b1d5 Sep 18 16:07:11.515605 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.515582 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-44tzz" event={"ID":"ad5ffd53-9297-4327-901f-01a54926300c","Type":"ContainerStarted","Data":"a61838e75304bfb3ccff4841d7170ab57b935b822aa647562df956354799b1d5"} Sep 18 16:07:11.516745 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.516720 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" event={"ID":"40b404d1-9ae4-4b27-82e4-b08a0a3597cb","Type":"ContainerStarted","Data":"e88df05ca75deee6212b21e0e2b796f18647fbad3db41f17b6b7438d3fecf3b7"} Sep 18 16:07:11.700624 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.700590 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:11.799517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:11.799438 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:12.219096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:12.218983 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:12.321951 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:12.321915 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:12.498544 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:12.498509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:12.600583 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:12.600550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:12.839992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:12.839962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:07:12.842571 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:12.842543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d47181ff-68b6-410e-827d-2f4de7db7d7b-metrics-certs\") pod \"network-metrics-daemon-4j77c\" (UID: \"d47181ff-68b6-410e-827d-2f4de7db7d7b\") " pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:07:13.090242 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.090157 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\"" Sep 18 16:07:13.097525 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.097494 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4j77c" Sep 18 16:07:13.201848 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.201814 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:13.268166 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.268136 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:13.331591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.331558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:13.372772 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.372750 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4j77c"] Sep 18 16:07:13.375154 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:13.375124 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47181ff_68b6_410e_827d_2f4de7db7d7b.slice/crio-28b2b4c2e67bc7aa6b62c580fbb116032c476c203e19fed0782b10779b513bb9 WatchSource:0}: Error finding container 28b2b4c2e67bc7aa6b62c580fbb116032c476c203e19fed0782b10779b513bb9: Status 404 returned error can't find the container with id 28b2b4c2e67bc7aa6b62c580fbb116032c476c203e19fed0782b10779b513bb9 Sep 18 16:07:13.525364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.525329 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4j77c" event={"ID":"d47181ff-68b6-410e-827d-2f4de7db7d7b","Type":"ContainerStarted","Data":"28b2b4c2e67bc7aa6b62c580fbb116032c476c203e19fed0782b10779b513bb9"} Sep 18 16:07:13.526991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.526959 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" event={"ID":"40b404d1-9ae4-4b27-82e4-b08a0a3597cb","Type":"ContainerStarted","Data":"7107fffc6ada2db76a32d0bd871fd1c50be8f83cffa6208414752813beec5c32"} Sep 18 16:07:13.527238 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.527191 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:13.532520 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.532499 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" Sep 18 16:07:13.545627 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:13.545571 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-4mxn8" podStartSLOduration=2.253123296 podStartE2EDuration="3.545550444s" totalImagesPullingTime="1.292427148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:13.544413924 +0000 UTC m=+133.105258603" watchObservedRunningTime="2026-09-18 16:07:13.545550444 +0000 UTC m=+133.106395122" Sep 18 16:07:14.289851 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.289797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:14.318232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.318191 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t"] Sep 18 16:07:14.321467 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.321436 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.325640 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.325617 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 16:07:14.326339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.325902 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 18 16:07:14.326339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.325912 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 18 16:07:14.326339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.326151 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 16:07:14.326339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.326180 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-57xkq\"" Sep 18 16:07:14.326339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.326333 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 16:07:14.333055 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.332955 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t"] Sep 18 16:07:14.351003 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.350963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95254775-b58f-4f42-bb0b-bd48d2cd5141-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.351200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.351122 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95254775-b58f-4f42-bb0b-bd48d2cd5141-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.351200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.351171 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95254775-b58f-4f42-bb0b-bd48d2cd5141-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.351321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.351262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9jj\" (UniqueName: \"kubernetes.io/projected/95254775-b58f-4f42-bb0b-bd48d2cd5141-kube-api-access-2r9jj\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.356310 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.356245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:14.422663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.422632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:14.452181 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.452142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9jj\" (UniqueName: \"kubernetes.io/projected/95254775-b58f-4f42-bb0b-bd48d2cd5141-kube-api-access-2r9jj\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.452374 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.452186 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95254775-b58f-4f42-bb0b-bd48d2cd5141-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.452374 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.452270 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95254775-b58f-4f42-bb0b-bd48d2cd5141-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.452374 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.452309 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95254775-b58f-4f42-bb0b-bd48d2cd5141-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.453171 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.453140 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95254775-b58f-4f42-bb0b-bd48d2cd5141-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.455600 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.455576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95254775-b58f-4f42-bb0b-bd48d2cd5141-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.455774 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.455747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/95254775-b58f-4f42-bb0b-bd48d2cd5141-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.461448 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.461416 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9jj\" (UniqueName: \"kubernetes.io/projected/95254775-b58f-4f42-bb0b-bd48d2cd5141-kube-api-access-2r9jj\") pod \"prometheus-operator-7bd5d56d6-4ck5t\" (UID: \"95254775-b58f-4f42-bb0b-bd48d2cd5141\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.634000 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.633926 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" Sep 18 16:07:14.913950 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:14.913913 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t"] Sep 18 16:07:14.916826 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:14.916795 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95254775_b58f_4f42_bb0b_bd48d2cd5141.slice/crio-461ad8a14d637a8b93c8589c226568184394b618317250af162199fc9d83b0ff WatchSource:0}: Error finding container 461ad8a14d637a8b93c8589c226568184394b618317250af162199fc9d83b0ff: Status 404 returned error can't find the container with id 461ad8a14d637a8b93c8589c226568184394b618317250af162199fc9d83b0ff Sep 18 16:07:15.535567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.535535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:15.538218 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.538188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4j77c" event={"ID":"d47181ff-68b6-410e-827d-2f4de7db7d7b","Type":"ContainerStarted","Data":"5b63cdc1e9edd04429295d482951150cd52698858d5efb97bae6f85db23a5305"} Sep 18 16:07:15.538374 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.538247 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4j77c" event={"ID":"d47181ff-68b6-410e-827d-2f4de7db7d7b","Type":"ContainerStarted","Data":"73873dbb6ddd913e5f3e25587f5ee3c43cfbfa1498a4db4309c182bdee9d2b6d"} Sep 18 16:07:15.539632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.539602 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" event={"ID":"95254775-b58f-4f42-bb0b-bd48d2cd5141","Type":"ContainerStarted","Data":"461ad8a14d637a8b93c8589c226568184394b618317250af162199fc9d83b0ff"} Sep 18 16:07:15.555068 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.555013 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4j77c" podStartSLOduration=131.244138311 podStartE2EDuration="2m12.554993535s" totalImagesPullingTime="1.310855224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:15.554469166 +0000 UTC m=+135.115313857" watchObservedRunningTime="2026-09-18 16:07:15.554993535 +0000 UTC m=+135.115838201" Sep 18 16:07:15.628102 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.628055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:15.678563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.678528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:15.733266 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:15.733229 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:16.197260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.197222 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:16.259198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.259160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:16.315203 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.315171 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56d8747ff-2zlzf"] Sep 18 16:07:16.318463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.318436 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.321289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.321149 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 16:07:16.321289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.321173 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4tf4\"" Sep 18 16:07:16.322364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.322335 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 16:07:16.322505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.322343 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 16:07:16.322622 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.322606 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 16:07:16.322728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.322708 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 16:07:16.328513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.328489 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d8747ff-2zlzf"] Sep 18 16:07:16.368891 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.368861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-config\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.369062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.368931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dq5\" (UniqueName: \"kubernetes.io/projected/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-kube-api-access-78dq5\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.369062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.369010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-oauth-serving-cert\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.369062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.369042 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-oauth-config\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.369262 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.369073 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-serving-cert\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.369262 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.369144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-service-ca\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.469833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.469742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-service-ca\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.469833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.469825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-config\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.469891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dq5\" (UniqueName: \"kubernetes.io/projected/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-kube-api-access-78dq5\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.469927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-oauth-serving-cert\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.469962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-oauth-config\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.470004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-serving-cert\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470543 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.470516 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-service-ca\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.470579 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-oauth-serving-cert\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.470681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.470576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-config\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.473216 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.473177 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-oauth-config\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.473966 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.473925 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-serving-cert\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.479810 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.479777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dq5\" (UniqueName: \"kubernetes.io/projected/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-kube-api-access-78dq5\") pod \"console-56d8747ff-2zlzf\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:16.631972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:16.631936 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:17.004259 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:17.004129 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56d8747ff-2zlzf"] Sep 18 16:07:17.007861 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:17.007829 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1ec144_415e_4a6c_ab4d_fa29ce171cf3.slice/crio-ec56c63cd6fc8414898b1d6fb73ebba9018128ff119c949fdff31d1d916f47ad WatchSource:0}: Error finding container ec56c63cd6fc8414898b1d6fb73ebba9018128ff119c949fdff31d1d916f47ad: Status 404 returned error can't find the container with id ec56c63cd6fc8414898b1d6fb73ebba9018128ff119c949fdff31d1d916f47ad Sep 18 16:07:17.057369 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:17.057335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:17.550994 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:17.550589 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" event={"ID":"95254775-b58f-4f42-bb0b-bd48d2cd5141","Type":"ContainerStarted","Data":"2a6e6565fb99fefca4b6a1188806750d6ed8df827cb0b6c7992925b602bbb5bb"} Sep 18 16:07:17.550994 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:17.550662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" event={"ID":"95254775-b58f-4f42-bb0b-bd48d2cd5141","Type":"ContainerStarted","Data":"93518bcf36203f53a592cbecf7c761dfa9c51a4ff9143669eaa0728c8e7fe7eb"} Sep 18 16:07:17.552252 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:17.552191 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d8747ff-2zlzf" event={"ID":"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3","Type":"ContainerStarted","Data":"ec56c63cd6fc8414898b1d6fb73ebba9018128ff119c949fdff31d1d916f47ad"} Sep 18 16:07:17.568344 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:17.568280 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4ck5t" podStartSLOduration=1.735086186 podStartE2EDuration="3.568259342s" totalImagesPullingTime="1.833173156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:17.567515583 +0000 UTC m=+137.128360292" watchObservedRunningTime="2026-09-18 16:07:17.568259342 +0000 UTC m=+137.129104011" Sep 18 16:07:18.941473 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:18.941440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:18.995361 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:18.995282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:19.435123 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.435073 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:19.668228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.668184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:19.704886 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.704795 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-px7qf"] Sep 18 16:07:19.710430 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.709657 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.712737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.712713 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5n88n\"" Sep 18 16:07:19.713165 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.713144 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 18 16:07:19.714255 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.714232 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 18 16:07:19.714345 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.714244 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 18 16:07:19.720881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.720858 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xfn6b"] Sep 18 16:07:19.727485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.727444 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-px7qf"] Sep 18 16:07:19.727592 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.727578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.730745 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.730719 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 16:07:19.730862 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.730837 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 16:07:19.730972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.730954 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgw4g\"" Sep 18 16:07:19.731028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.730730 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 16:07:19.758525 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.758490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:19.801412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801371 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.801591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhtg\" (UniqueName: \"kubernetes.io/projected/4ba373af-2334-4925-82df-e15fed6a8fcc-kube-api-access-dxhtg\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.801591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.801591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801515 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt97z\" (UniqueName: \"kubernetes.io/projected/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-api-access-rt97z\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.801591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801549 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-textfile\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.801591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-wtmp\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.801803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.801803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.801803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.801803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba373af-2334-4925-82df-e15fed6a8fcc-metrics-client-ca\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.801803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-tls\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.801803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801795 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.802107 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.802107 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801911 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-root\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.802107 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.801940 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-sys\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.858417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.858382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:19.903043 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903006 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-tls\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903055 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903122 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-root\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903240 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:07:19.903166 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 18 16:07:19.903240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-sys\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903227 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903272 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhtg\" (UniqueName: \"kubernetes.io/projected/4ba373af-2334-4925-82df-e15fed6a8fcc-kube-api-access-dxhtg\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-root\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:07:19.903340 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-tls podName:4ba373af-2334-4925-82df-e15fed6a8fcc nodeName:}" failed. No retries permitted until 2026-09-18 16:07:20.403308791 +0000 UTC m=+139.964153450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-tls") pod "node-exporter-xfn6b" (UID: "4ba373af-2334-4925-82df-e15fed6a8fcc") : secret "node-exporter-tls" not found Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-sys\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903411 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt97z\" (UniqueName: \"kubernetes.io/projected/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-api-access-rt97z\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-textfile\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-wtmp\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903636 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903714 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba373af-2334-4925-82df-e15fed6a8fcc-metrics-client-ca\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-wtmp\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903786 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-textfile\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.903956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.903890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-accelerators-collector-config\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.904314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.904064 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.904314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.904154 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.904314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.904288 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ba373af-2334-4925-82df-e15fed6a8fcc-metrics-client-ca\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.904463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.904435 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.907001 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.906961 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.907582 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.907557 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.908351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.908306 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.915280 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.915222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhtg\" (UniqueName: \"kubernetes.io/projected/4ba373af-2334-4925-82df-e15fed6a8fcc-kube-api-access-dxhtg\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:19.916817 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.916719 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt97z\" (UniqueName: \"kubernetes.io/projected/a17009e2-0cb4-4f46-9bd9-1e0d8d20de27-kube-api-access-rt97z\") pod \"kube-state-metrics-84c885658c-px7qf\" (UID: \"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:19.960493 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:19.960366 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.023550 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.023507 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" Sep 18 16:07:20.060914 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.060880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.167589 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.167553 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.297276 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.297239 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.407637 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.407605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-tls\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:20.410634 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.410606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ba373af-2334-4925-82df-e15fed6a8fcc-node-exporter-tls\") pod \"node-exporter-xfn6b\" (UID: \"4ba373af-2334-4925-82df-e15fed6a8fcc\") " pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:20.629866 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.629750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.639666 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.639627 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xfn6b" Sep 18 16:07:20.759199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.759130 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.827103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.827057 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:20.833420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.833392 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:07:20.843306 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.843272 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.846368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.846345 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 18 16:07:20.846565 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.846347 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 18 16:07:20.846750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.846671 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 18 16:07:20.846750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.846718 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 18 16:07:20.846952 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.846798 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 18 16:07:20.847030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.846972 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 18 16:07:20.847116 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.847103 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 18 16:07:20.847195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.847125 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 18 16:07:20.847622 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.847564 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 18 16:07:20.847763 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.847657 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lhc9c\"" Sep 18 16:07:20.853925 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.853903 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:07:20.912618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912511 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912582 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912622 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7clv\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-kube-api-access-f7clv\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-web-config\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912766 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912811 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912854 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912879 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-config-out\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.912907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.913275 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912934 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.913275 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.912964 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.913275 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.913115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:20.913275 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:20.913161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014510 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014524 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-config-out\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014817 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7clv\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-kube-api-access-f7clv\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.014960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.014972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-web-config\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.015460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.015010 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.015460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.015038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.015671 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.015646 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.016340 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:07:21.016315 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle podName:e943ec65-12d7-467c-957e-5e416a6023d0 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:21.516292718 +0000 UTC m=+141.077137378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0") : configmap references non-existent config key: ca-bundle.crt Sep 18 16:07:21.017115 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.017076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.018520 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.018477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.019708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.019212 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.019708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.019658 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.020502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.020447 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-config-out\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.020502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.020484 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-web-config\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.022343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.022284 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.022445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.022351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.022782 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.022673 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.022916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.022896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.025627 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.025603 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7clv\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-kube-api-access-f7clv\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.078774 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.078722 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:21.521047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.521010 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.522024 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.521997 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.611351 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:21.611318 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba373af_2334_4925_82df_e15fed6a8fcc.slice/crio-ac1e5d4c627dfdf7dc7ffae73c5bb45b938f79ec85bad21ae677fc0362636b01 WatchSource:0}: Error finding container ac1e5d4c627dfdf7dc7ffae73c5bb45b938f79ec85bad21ae677fc0362636b01: Status 404 returned error can't find the container with id ac1e5d4c627dfdf7dc7ffae73c5bb45b938f79ec85bad21ae677fc0362636b01 Sep 18 16:07:21.756860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.756827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:07:21.888510 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:21.888484 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-px7qf"] Sep 18 16:07:21.891165 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:21.891138 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17009e2_0cb4_4f46_9bd9_1e0d8d20de27.slice/crio-5d8ab9c9852b71d9c317d70b1fe766d2ecb256eea77a3f6cc06b30019b014939 WatchSource:0}: Error finding container 5d8ab9c9852b71d9c317d70b1fe766d2ecb256eea77a3f6cc06b30019b014939: Status 404 returned error can't find the container with id 5d8ab9c9852b71d9c317d70b1fe766d2ecb256eea77a3f6cc06b30019b014939 Sep 18 16:07:22.058998 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.058966 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:07:22.061750 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:22.061716 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode943ec65_12d7_467c_957e_5e416a6023d0.slice/crio-5bf296d5f16646df55001d8b372f766f00fe821546f1fb245c589b60214fa7c5 WatchSource:0}: Error finding container 5bf296d5f16646df55001d8b372f766f00fe821546f1fb245c589b60214fa7c5: Status 404 returned error can't find the container with id 5bf296d5f16646df55001d8b372f766f00fe821546f1fb245c589b60214fa7c5 Sep 18 16:07:22.575285 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.575217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d8747ff-2zlzf" event={"ID":"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3","Type":"ContainerStarted","Data":"b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1"} Sep 18 16:07:22.576771 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.576741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"5bf296d5f16646df55001d8b372f766f00fe821546f1fb245c589b60214fa7c5"} Sep 18 16:07:22.578256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.578199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" event={"ID":"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27","Type":"ContainerStarted","Data":"5d8ab9c9852b71d9c317d70b1fe766d2ecb256eea77a3f6cc06b30019b014939"} Sep 18 16:07:22.579683 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.579617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfn6b" event={"ID":"4ba373af-2334-4925-82df-e15fed6a8fcc","Type":"ContainerStarted","Data":"ac1e5d4c627dfdf7dc7ffae73c5bb45b938f79ec85bad21ae677fc0362636b01"} Sep 18 16:07:22.597967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.597906 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56d8747ff-2zlzf" podStartSLOduration=1.934257488 podStartE2EDuration="6.597882152s" totalImagesPullingTime="4.663624664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:22.594919651 +0000 UTC m=+142.155764320" watchObservedRunningTime="2026-09-18 16:07:22.597882152 +0000 UTC m=+142.158726816" Sep 18 16:07:22.646357 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.646317 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:22.703846 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:22.703811 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:23.079285 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:23.079248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:23.180031 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:23.180005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:24.442744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:24.442712 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:24.505371 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:24.505334 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:24.681169 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:24.681133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:25.911435 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:25.911389 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:26.632422 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:26.632386 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:26.632614 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:26.632471 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:07:26.633728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:26.633698 2563 patch_prober.go:28] interesting pod/console-56d8747ff-2zlzf container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Sep 18 16:07:26.633860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:26.633751 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-56d8747ff-2zlzf" podUID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" containerName="console" probeResult="failure" output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" Sep 18 16:07:26.904607 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:26.904525 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:26.976189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:26.976153 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:30.546389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.546280 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:30.615524 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.614622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-44tzz" event={"ID":"ad5ffd53-9297-4327-901f-01a54926300c","Type":"ContainerStarted","Data":"1ccf2f09ad18ebdca2f72e12ed808737f059cc0d720a271c9cc5149312e57ce4"} Sep 18 16:07:30.615524 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.615428 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:30.616973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.616596 2563 patch_prober.go:28] interesting pod/downloads-6594f4c75-44tzz container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused" start-of-body= Sep 18 16:07:30.616973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.616645 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-44tzz" podUID="ad5ffd53-9297-4327-901f-01a54926300c" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.13:8080/\": dial tcp 10.132.0.13:8080: connect: connection refused" Sep 18 16:07:30.618785 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.618701 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3" exitCode=0 Sep 18 16:07:30.618882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.618785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3"} Sep 18 16:07:30.621907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.621813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" event={"ID":"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27","Type":"ContainerStarted","Data":"8bb277038ec2cd20bf26fb8526a5ddd7f1890c28a32988315f5e761b78b1bf72"} Sep 18 16:07:30.621907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.621864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" event={"ID":"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27","Type":"ContainerStarted","Data":"e0105f6ea6ba85d79d213652422d3b86ccb0a015bc4c22bbfd00482fcec0baca"} Sep 18 16:07:30.624560 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.624534 2563 generic.go:356] "Generic (PLEG): container finished" podID="4ba373af-2334-4925-82df-e15fed6a8fcc" containerID="f61d762ee01311063d18fa981a361f546b0546e1d9a546ecfc43b36b35061226" exitCode=0 Sep 18 16:07:30.624642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.624587 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfn6b" event={"ID":"4ba373af-2334-4925-82df-e15fed6a8fcc","Type":"ContainerDied","Data":"f61d762ee01311063d18fa981a361f546b0546e1d9a546ecfc43b36b35061226"} Sep 18 16:07:30.634740 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.634670 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-44tzz" podStartSLOduration=1.789602574 podStartE2EDuration="20.634630742s" totalImagesPullingTime="18.845028168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:30.632046144 +0000 UTC m=+150.192890825" watchObservedRunningTime="2026-09-18 16:07:30.634630742 +0000 UTC m=+150.195475420" Sep 18 16:07:30.638910 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.638887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:30.916233 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.916147 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:30.965909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:30.965869 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:31.632745 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:31.632652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" event={"ID":"a17009e2-0cb4-4f46-9bd9-1e0d8d20de27","Type":"ContainerStarted","Data":"52a5d822599a10787f1f07cf956e15d407907fc93249a214dd4ee75e53d9fd25"} Sep 18 16:07:31.638153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:31.638124 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfn6b" event={"ID":"4ba373af-2334-4925-82df-e15fed6a8fcc","Type":"ContainerStarted","Data":"c781619d944f862c76056ee41f4b84848891f903e4e1a8a10da68b31aad5cf09"} Sep 18 16:07:31.638281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:31.638197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xfn6b" event={"ID":"4ba373af-2334-4925-82df-e15fed6a8fcc","Type":"ContainerStarted","Data":"d979c67e83fd06feea00313da93f189ac76da14ce9d77effaa7b07c2c00f5543"} Sep 18 16:07:31.650741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:31.650688 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-44tzz" Sep 18 16:07:31.654153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:31.654107 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-px7qf" podStartSLOduration=4.186758013 podStartE2EDuration="12.654073692s" totalImagesPullingTime="8.467315679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:31.653671532 +0000 UTC m=+151.214516197" watchObservedRunningTime="2026-09-18 16:07:31.654073692 +0000 UTC m=+151.214918359" Sep 18 16:07:31.677731 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:31.677676 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xfn6b" podStartSLOduration=3.857555807 podStartE2EDuration="12.677666915s" totalImagesPullingTime="8.742168306s" totalInitContainerRuntime="77.942802ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:31.676244021 +0000 UTC m=+151.237088711" watchObservedRunningTime="2026-09-18 16:07:31.677666915 +0000 UTC m=+151.238511583" Sep 18 16:07:33.127778 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.127750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:33.232592 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.232556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:33.319133 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.319106 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d8747ff-2zlzf"] Sep 18 16:07:33.650567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.650527 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8"} Sep 18 16:07:33.650694 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.650630 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253"} Sep 18 16:07:33.650694 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.650679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b"} Sep 18 16:07:33.650794 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:33.650718 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f"} Sep 18 16:07:34.659172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:34.659140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248"} Sep 18 16:07:35.668261 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:35.668226 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerStarted","Data":"e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2"} Sep 18 16:07:35.701242 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:35.701180 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.714832733 podStartE2EDuration="15.701159291s" totalImagesPullingTime="11.96753106s" totalInitContainerRuntime="18.795498ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:35.698924873 +0000 UTC m=+155.259769576" watchObservedRunningTime="2026-09-18 16:07:35.701159291 +0000 UTC m=+155.262003959" Sep 18 16:07:36.716129 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:36.716078 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:38.120680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:38.120642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:38.980175 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:38.980131 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:39.037658 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:39.037610 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:39.291429 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:07:39.291376 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" Sep 18 16:07:39.311054 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:07:39.311017 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zkpmd" podUID="f5c18c74-c0ae-4cd9-a6af-9733861063b3" Sep 18 16:07:39.685430 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:39.685356 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:07:39.685430 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:39.685395 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkpmd" Sep 18 16:07:44.227054 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.226981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:07:44.227054 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.227019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:07:44.229343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.229313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5c18c74-c0ae-4cd9-a6af-9733861063b3-metrics-tls\") pod \"dns-default-zkpmd\" (UID: \"f5c18c74-c0ae-4cd9-a6af-9733861063b3\") " pod="openshift-dns/dns-default-zkpmd" Sep 18 16:07:44.229548 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.229526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"image-registry-c757cc57c-4tsdg\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:07:44.490384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.490312 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bxqxp\"" Sep 18 16:07:44.490384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.490317 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\"" Sep 18 16:07:44.497410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.497388 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:07:44.497460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.497417 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkpmd" Sep 18 16:07:44.784609 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.784531 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c757cc57c-4tsdg"] Sep 18 16:07:44.786715 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:44.786691 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804f9e66_c2ea_4cd5_b1ae_92311ce42617.slice/crio-f5d2ff27e395a5fcbc48f1c9583339e68c6850efa3ce30ebdd8a366f86f9ad42 WatchSource:0}: Error finding container f5d2ff27e395a5fcbc48f1c9583339e68c6850efa3ce30ebdd8a366f86f9ad42: Status 404 returned error can't find the container with id f5d2ff27e395a5fcbc48f1c9583339e68c6850efa3ce30ebdd8a366f86f9ad42 Sep 18 16:07:44.802877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:44.802856 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zkpmd"] Sep 18 16:07:44.804860 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:07:44.804838 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c18c74_c0ae_4cd9_a6af_9733861063b3.slice/crio-8a4e6a41a33789c2cd8cf441eb9a2c8205c42e01b8cd204cbfb37f162cc2df41 WatchSource:0}: Error finding container 8a4e6a41a33789c2cd8cf441eb9a2c8205c42e01b8cd204cbfb37f162cc2df41: Status 404 returned error can't find the container with id 8a4e6a41a33789c2cd8cf441eb9a2c8205c42e01b8cd204cbfb37f162cc2df41 Sep 18 16:07:45.710034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:45.709991 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zkpmd" event={"ID":"f5c18c74-c0ae-4cd9-a6af-9733861063b3","Type":"ContainerStarted","Data":"8a4e6a41a33789c2cd8cf441eb9a2c8205c42e01b8cd204cbfb37f162cc2df41"} Sep 18 16:07:45.711909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:45.711866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" event={"ID":"804f9e66-c2ea-4cd5-b1ae-92311ce42617","Type":"ContainerStarted","Data":"299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6"} Sep 18 16:07:45.712052 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:45.711919 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" event={"ID":"804f9e66-c2ea-4cd5-b1ae-92311ce42617","Type":"ContainerStarted","Data":"f5d2ff27e395a5fcbc48f1c9583339e68c6850efa3ce30ebdd8a366f86f9ad42"} Sep 18 16:07:45.712052 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:45.711967 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:07:47.721024 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:47.720989 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zkpmd" event={"ID":"f5c18c74-c0ae-4cd9-a6af-9733861063b3","Type":"ContainerStarted","Data":"443a7ac8e3e42319e41d526d2080a85825960b5c51037bc282fa965ef66af67c"} Sep 18 16:07:47.721412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:47.721042 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zkpmd" event={"ID":"f5c18c74-c0ae-4cd9-a6af-9733861063b3","Type":"ContainerStarted","Data":"d69b55daf0d4b3df209929e86ff4b095a5233a14e65593cab9110e253100a402"} Sep 18 16:07:47.721412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:47.721116 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zkpmd" Sep 18 16:07:47.737881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:47.737833 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" podStartSLOduration=164.737819628 podStartE2EDuration="2m44.737819628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:45.733356451 +0000 UTC m=+165.294201123" watchObservedRunningTime="2026-09-18 16:07:47.737819628 +0000 UTC m=+167.298664295" Sep 18 16:07:47.738131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:47.738111 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zkpmd" podStartSLOduration=129.297336353 podStartE2EDuration="2m11.738104679s" totalImagesPullingTime="2.440768326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:47.73701573 +0000 UTC m=+167.297860497" watchObservedRunningTime="2026-09-18 16:07:47.738104679 +0000 UTC m=+167.298949340" Sep 18 16:07:48.505326 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:48.505295 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:57.727682 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:57.727650 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zkpmd" Sep 18 16:07:58.063949 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:58.063912 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:58.164381 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:58.164355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:07:58.342957 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:07:58.342866 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56d8747ff-2zlzf" podUID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" containerName="console" containerID="cri-o://b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1" gracePeriod=15 Sep 18 16:08:00.528489 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:00.528456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:00.590772 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:00.590738 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:02.117007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.116981 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:08:02.160896 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.160829 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-config\") pod \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " Sep 18 16:08:02.160896 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.160862 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78dq5\" (UniqueName: \"kubernetes.io/projected/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-kube-api-access-78dq5\") pod \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " Sep 18 16:08:02.160896 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.160879 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-oauth-serving-cert\") pod \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " Sep 18 16:08:02.161159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.160912 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-serving-cert\") pod \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " Sep 18 16:08:02.161159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.160957 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-oauth-config\") pod \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " Sep 18 16:08:02.161159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.160979 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-service-ca\") pod \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\" (UID: \"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3\") " Sep 18 16:08:02.161316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.161241 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-config" (OuterVolumeSpecName: "console-config") pod "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" (UID: "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:02.161316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.161298 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" (UID: "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:02.161420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.161388 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-service-ca" (OuterVolumeSpecName: "service-ca") pod "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" (UID: "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:02.163170 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.163135 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" (UID: "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:02.163277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.163196 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-kube-api-access-78dq5" (OuterVolumeSpecName: "kube-api-access-78dq5") pod "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" (UID: "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3"). InnerVolumeSpecName "kube-api-access-78dq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:08:02.163277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.163220 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" (UID: "1c1ec144-415e-4a6c-ab4d-fa29ce171cf3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:02.261850 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.261815 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-config\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:02.261850 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.261843 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78dq5\" (UniqueName: \"kubernetes.io/projected/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-kube-api-access-78dq5\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:02.261850 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.261853 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-oauth-serving-cert\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:02.262059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.261861 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-serving-cert\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:02.262059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.261870 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-console-oauth-config\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:02.262059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.261878 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:02.775781 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.775742 2563 generic.go:356] "Generic (PLEG): container finished" podID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" containerID="b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1" exitCode=0 Sep 18 16:08:02.775973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.775791 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56d8747ff-2zlzf" Sep 18 16:08:02.775973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.775828 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d8747ff-2zlzf" event={"ID":"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3","Type":"ContainerDied","Data":"b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1"} Sep 18 16:08:02.775973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.775870 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56d8747ff-2zlzf" event={"ID":"1c1ec144-415e-4a6c-ab4d-fa29ce171cf3","Type":"ContainerDied","Data":"ec56c63cd6fc8414898b1d6fb73ebba9018128ff119c949fdff31d1d916f47ad"} Sep 18 16:08:02.775973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.775889 2563 scope.go:117] "RemoveContainer" containerID="b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1" Sep 18 16:08:02.786653 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.786632 2563 scope.go:117] "RemoveContainer" containerID="b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1" Sep 18 16:08:02.787171 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:02.787149 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1\": container with ID starting with b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1 not found: ID does not exist" containerID="b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1" Sep 18 16:08:02.787237 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.787179 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1"} err="failed to get container status \"b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1\": rpc error: code = NotFound desc = could not find container \"b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1\": container with ID starting with b622a4bf2b79278f4f51bce7ef9644aa5cdade3b953f97a8a1a6c84d6e2a0bc1 not found: ID does not exist" Sep 18 16:08:02.799183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.799161 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56d8747ff-2zlzf"] Sep 18 16:08:02.803364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:02.803342 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56d8747ff-2zlzf"] Sep 18 16:08:03.079215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:03.079138 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" path="/var/lib/kubelet/pods/1c1ec144-415e-4a6c-ab4d-fa29ce171cf3/volumes" Sep 18 16:08:03.836311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:03.836282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:03.938720 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:03.938684 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:03.992125 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:03.992101 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:04.441203 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:04.441174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:04.501725 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:04.501694 2563 patch_prober.go:28] interesting pod/image-registry-c757cc57c-4tsdg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 18 16:08:04.501871 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:04.501741 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 16:08:04.506202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:04.506178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:05.788642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:05.788612 2563 generic.go:356] "Generic (PLEG): container finished" podID="85d4feac-2eae-4628-a1c5-ba2ae60eb119" containerID="0a94ef3145e2455212f160686e0e7ad8bba21b31e8f45bf3befb58d576cace81" exitCode=0 Sep 18 16:08:05.788953 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:05.788684 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" event={"ID":"85d4feac-2eae-4628-a1c5-ba2ae60eb119","Type":"ContainerDied","Data":"0a94ef3145e2455212f160686e0e7ad8bba21b31e8f45bf3befb58d576cace81"} Sep 18 16:08:05.789022 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:05.789008 2563 scope.go:117] "RemoveContainer" containerID="0a94ef3145e2455212f160686e0e7ad8bba21b31e8f45bf3befb58d576cace81" Sep 18 16:08:06.720396 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:06.720366 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:08:06.794456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:06.794421 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-z2x82" event={"ID":"85d4feac-2eae-4628-a1c5-ba2ae60eb119","Type":"ContainerStarted","Data":"08b7485fcd85ccae9d7841e6796500261c4a429979df8766b259aa3557d8cfc9"} Sep 18 16:08:06.908469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:06.908437 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:07.008735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:07.008702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:07.070598 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:07.070570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:08.005700 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:08.005671 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zkpmd_f5c18c74-c0ae-4cd9-a6af-9733861063b3/dns/0.log" Sep 18 16:08:08.205896 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:08.205869 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zkpmd_f5c18c74-c0ae-4cd9-a6af-9733861063b3/kube-rbac-proxy/0.log" Sep 18 16:08:08.605034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:08.605006 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bbmtn_77789e19-a00a-403e-9556-65748730930e/dns-node-resolver/0.log" Sep 18 16:08:09.205399 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:09.205368 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c757cc57c-4tsdg_804f9e66-c2ea-4cd5-b1ae-92311ce42617/registry/0.log" Sep 18 16:08:09.804689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:09.804656 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qcxrf_acc1a685-72d0-471f-808d-22cfc451dee0/node-ca/0.log" Sep 18 16:08:10.405179 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:10.405155 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-krt6v_a2730fdd-0e9f-4a38-8eeb-24c40fc83f02/serve-healthcheck-canary/0.log" Sep 18 16:08:11.318798 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:11.318766 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c757cc57c-4tsdg"] Sep 18 16:08:11.406807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:11.406775 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:11.512913 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:11.512865 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:15.543937 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:15.543900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:15.604113 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:15.604070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:15.870074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:15.869999 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:15.931728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:15.931697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:16.050892 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:16.050867 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:16.152208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:16.152135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:18.396981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:18.396945 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:18.587867 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:18.587827 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:18.649445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:18.649370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:30.539968 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:30.539936 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:30.591214 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:30.591184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:35.778264 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:35.778230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:35.843820 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:35.843793 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:36.342890 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:36.342824 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerName="registry" containerID="cri-o://299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6" gracePeriod=30 Sep 18 16:08:36.717282 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:36.717201 2563 patch_prober.go:28] interesting pod/image-registry-c757cc57c-4tsdg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" start-of-body= Sep 18 16:08:36.717282 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:36.717269 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.6:5000/healthz\": dial tcp 10.132.0.6:5000: connect: connection refused" Sep 18 16:08:37.029442 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.029416 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:37.130304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.130257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:37.231034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.231008 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:37.289405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.289334 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:37.590329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.590303 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:08:37.729469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729438 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/804f9e66-c2ea-4cd5-b1ae-92311ce42617-ca-trust-extracted\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729473 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-installation-pull-secrets\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729518 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-bound-sa-token\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729545 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gtlb\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-kube-api-access-4gtlb\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729597 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-trusted-ca\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729663 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-image-registry-private-configuration\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729719 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-certificates\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.729919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.729740 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") pod \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\" (UID: \"804f9e66-c2ea-4cd5-b1ae-92311ce42617\") " Sep 18 16:08:37.730225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.730154 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:37.730408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.730229 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:37.732141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.732099 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:37.732141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.732111 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:08:37.732299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.732196 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:37.732404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.732374 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-kube-api-access-4gtlb" (OuterVolumeSpecName: "kube-api-access-4gtlb") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "kube-api-access-4gtlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:08:37.732486 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.732429 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:08:37.741871 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.741848 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804f9e66-c2ea-4cd5-b1ae-92311ce42617-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "804f9e66-c2ea-4cd5-b1ae-92311ce42617" (UID: "804f9e66-c2ea-4cd5-b1ae-92311ce42617"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:08:37.830779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830727 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-trusted-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830747 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-image-registry-private-configuration\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830757 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-certificates\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830768 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-registry-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830776 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/804f9e66-c2ea-4cd5-b1ae-92311ce42617-ca-trust-extracted\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830784 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/804f9e66-c2ea-4cd5-b1ae-92311ce42617-installation-pull-secrets\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830792 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-bound-sa-token\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.830977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.830800 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtlb\" (UniqueName: \"kubernetes.io/projected/804f9e66-c2ea-4cd5-b1ae-92311ce42617-kube-api-access-4gtlb\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:37.911318 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.911289 2563 generic.go:356] "Generic (PLEG): container finished" podID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerID="299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6" exitCode=0 Sep 18 16:08:37.911432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.911325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" event={"ID":"804f9e66-c2ea-4cd5-b1ae-92311ce42617","Type":"ContainerDied","Data":"299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6"} Sep 18 16:08:37.911432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.911334 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" Sep 18 16:08:37.911432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.911393 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c757cc57c-4tsdg" event={"ID":"804f9e66-c2ea-4cd5-b1ae-92311ce42617","Type":"ContainerDied","Data":"f5d2ff27e395a5fcbc48f1c9583339e68c6850efa3ce30ebdd8a366f86f9ad42"} Sep 18 16:08:37.911432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.911414 2563 scope.go:117] "RemoveContainer" containerID="299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6" Sep 18 16:08:37.919920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.919902 2563 scope.go:117] "RemoveContainer" containerID="299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6" Sep 18 16:08:37.920193 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:37.920172 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6\": container with ID starting with 299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6 not found: ID does not exist" containerID="299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6" Sep 18 16:08:37.920278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.920198 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6"} err="failed to get container status \"299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6\": rpc error: code = NotFound desc = could not find container \"299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6\": container with ID starting with 299b8bb4750686ba0dc305e1f59bc63a6f951c228bf61d5d7a097bd0b3971cc6 not found: ID does not exist" Sep 18 16:08:37.933415 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.933390 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c757cc57c-4tsdg"] Sep 18 16:08:37.937033 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:37.937014 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c757cc57c-4tsdg"] Sep 18 16:08:38.604591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604563 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84565469dc-m9kgf"] Sep 18 16:08:38.604958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604836 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" containerName="console" Sep 18 16:08:38.604958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604847 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" containerName="console" Sep 18 16:08:38.604958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604858 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerName="registry" Sep 18 16:08:38.604958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604864 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerName="registry" Sep 18 16:08:38.604958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604910 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1ec144-415e-4a6c-ab4d-fa29ce171cf3" containerName="console" Sep 18 16:08:38.604958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.604921 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" containerName="registry" Sep 18 16:08:38.608491 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.608468 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.611216 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.611186 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 16:08:38.611333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.611286 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4tf4\"" Sep 18 16:08:38.611333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.611286 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 16:08:38.611448 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.611334 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 16:08:38.611448 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.611335 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 16:08:38.611544 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.611448 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 16:08:38.616131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.616109 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 18 16:08:38.617516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.617498 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84565469dc-m9kgf"] Sep 18 16:08:38.686013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.685994 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:38.737277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.737251 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:38.738298 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-service-ca\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.738344 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-serving-cert\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.738375 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-config\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.738410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738388 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-oauth-serving-cert\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.738440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738407 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-trusted-ca-bundle\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.738440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738432 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-oauth-config\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.738503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.738455 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjswp\" (UniqueName: \"kubernetes.io/projected/ddc8ce26-a8f9-4350-97e7-778985818d5f-kube-api-access-jjswp\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839568 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839527 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-serving-cert\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839568 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-config\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-oauth-serving-cert\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839625 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-trusted-ca-bundle\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839640 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-oauth-config\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjswp\" (UniqueName: \"kubernetes.io/projected/ddc8ce26-a8f9-4350-97e7-778985818d5f-kube-api-access-jjswp\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.839741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.839714 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-service-ca\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.840421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.840397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-oauth-serving-cert\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.840519 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.840397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-config\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.840519 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.840407 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-service-ca\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.840662 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.840637 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-trusted-ca-bundle\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.842034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.842018 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-serving-cert\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.842176 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.842160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-oauth-config\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.847641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.847616 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjswp\" (UniqueName: \"kubernetes.io/projected/ddc8ce26-a8f9-4350-97e7-778985818d5f-kube-api-access-jjswp\") pod \"console-84565469dc-m9kgf\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:38.918794 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:38.918719 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:39.079045 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.079006 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="804f9e66-c2ea-4cd5-b1ae-92311ce42617" path="/var/lib/kubelet/pods/804f9e66-c2ea-4cd5-b1ae-92311ce42617/volumes" Sep 18 16:08:39.187038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.186965 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84565469dc-m9kgf"] Sep 18 16:08:39.190529 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:08:39.190506 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc8ce26_a8f9_4350_97e7_778985818d5f.slice/crio-46b816333417d01baa6036028388ee2eed9b4ba3ddcb3e3fc93ec5ae95a24f33 WatchSource:0}: Error finding container 46b816333417d01baa6036028388ee2eed9b4ba3ddcb3e3fc93ec5ae95a24f33: Status 404 returned error can't find the container with id 46b816333417d01baa6036028388ee2eed9b4ba3ddcb3e3fc93ec5ae95a24f33 Sep 18 16:08:39.577517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.577486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:39.678619 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.678588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:39.920611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.920533 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84565469dc-m9kgf" event={"ID":"ddc8ce26-a8f9-4350-97e7-778985818d5f","Type":"ContainerStarted","Data":"ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99"} Sep 18 16:08:39.920611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.920578 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84565469dc-m9kgf" event={"ID":"ddc8ce26-a8f9-4350-97e7-778985818d5f","Type":"ContainerStarted","Data":"46b816333417d01baa6036028388ee2eed9b4ba3ddcb3e3fc93ec5ae95a24f33"} Sep 18 16:08:39.951765 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:39.951704 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84565469dc-m9kgf" podStartSLOduration=1.951688305 podStartE2EDuration="1.951688305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:08:39.949705587 +0000 UTC m=+219.510550256" watchObservedRunningTime="2026-09-18 16:08:39.951688305 +0000 UTC m=+219.512532972" Sep 18 16:08:40.108122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108073 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:08:40.108459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108404 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="alertmanager" containerID="cri-o://82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f" gracePeriod=120 Sep 18 16:08:40.108459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108432 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-metric" containerID="cri-o://375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248" gracePeriod=120 Sep 18 16:08:40.108663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108483 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy" containerID="cri-o://9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8" gracePeriod=120 Sep 18 16:08:40.108663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108477 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-web" containerID="cri-o://299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253" gracePeriod=120 Sep 18 16:08:40.108663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108565 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="prom-label-proxy" containerID="cri-o://e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2" gracePeriod=120 Sep 18 16:08:40.108663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.108492 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="config-reloader" containerID="cri-o://69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b" gracePeriod=120 Sep 18 16:08:40.927161 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927072 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2" exitCode=0 Sep 18 16:08:40.927161 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927114 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248" exitCode=0 Sep 18 16:08:40.927161 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927110 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2"} Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927171 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248"} Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927122 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8" exitCode=0 Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927195 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b" exitCode=0 Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927200 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8"} Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927225 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b"} Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927248 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f"} Sep 18 16:08:40.927553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:40.927202 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f" exitCode=0 Sep 18 16:08:41.351013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.350982 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:41.459317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459283 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-config-out\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459340 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459360 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7clv\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-kube-api-access-f7clv\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459384 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-tls-assets\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459409 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-main-tls\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459425 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459447 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-web-config\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.459944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459920 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.460029 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.459979 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.460029 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.460009 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-metrics-client-ca\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.460151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.460065 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-config-volume\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.460151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.460137 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-main-db\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.460257 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.460169 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-cluster-tls-config\") pod \"e943ec65-12d7-467c-957e-5e416a6023d0\" (UID: \"e943ec65-12d7-467c-957e-5e416a6023d0\") " Sep 18 16:08:41.462215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.462155 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:08:41.462215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.462164 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:41.462619 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.462590 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:08:41.462696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.462650 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.462780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.462757 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:08:41.462946 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.462917 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-kube-api-access-f7clv" (OuterVolumeSpecName: "kube-api-access-f7clv") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "kube-api-access-f7clv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:08:41.463078 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.463050 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.463200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.463185 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.463467 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.463437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-config-out" (OuterVolumeSpecName: "config-out") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:08:41.464791 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.464764 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-config-volume" (OuterVolumeSpecName: "config-volume") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.464884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.464860 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.466666 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.466550 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.472232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.472209 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-web-config" (OuterVolumeSpecName: "web-config") pod "e943ec65-12d7-467c-957e-5e416a6023d0" (UID: "e943ec65-12d7-467c-957e-5e416a6023d0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:08:41.561764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561719 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561757 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7clv\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-kube-api-access-f7clv\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561768 2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e943ec65-12d7-467c-957e-5e416a6023d0-tls-assets\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561777 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-main-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561786 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561795 2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-web-config\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561803 2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561811 2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561819 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e943ec65-12d7-467c-957e-5e416a6023d0-metrics-client-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561826 2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561834 2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-alertmanager-main-db\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561841 2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e943ec65-12d7-467c-957e-5e416a6023d0-cluster-tls-config\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.561983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.561848 2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e943ec65-12d7-467c-957e-5e416a6023d0-config-out\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:08:41.934741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.934711 2563 generic.go:356] "Generic (PLEG): container finished" podID="e943ec65-12d7-467c-957e-5e416a6023d0" containerID="299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253" exitCode=0 Sep 18 16:08:41.935120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.934753 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253"} Sep 18 16:08:41.935120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.934801 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:41.935120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.934831 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e943ec65-12d7-467c-957e-5e416a6023d0","Type":"ContainerDied","Data":"5bf296d5f16646df55001d8b372f766f00fe821546f1fb245c589b60214fa7c5"} Sep 18 16:08:41.935120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.934866 2563 scope.go:117] "RemoveContainer" containerID="e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2" Sep 18 16:08:41.943731 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.943709 2563 scope.go:117] "RemoveContainer" containerID="375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248" Sep 18 16:08:41.951445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.951429 2563 scope.go:117] "RemoveContainer" containerID="9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8" Sep 18 16:08:41.957762 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.957744 2563 scope.go:117] "RemoveContainer" containerID="299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253" Sep 18 16:08:41.960457 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.960436 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:08:41.965195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.965175 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:08:41.965879 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.965863 2563 scope.go:117] "RemoveContainer" containerID="69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b" Sep 18 16:08:41.972302 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.972285 2563 scope.go:117] "RemoveContainer" containerID="82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f" Sep 18 16:08:41.978930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.978913 2563 scope.go:117] "RemoveContainer" containerID="afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3" Sep 18 16:08:41.985424 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.985408 2563 scope.go:117] "RemoveContainer" containerID="e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2" Sep 18 16:08:41.985688 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.985670 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2\": container with ID starting with e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2 not found: ID does not exist" containerID="e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2" Sep 18 16:08:41.985753 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.985693 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2"} err="failed to get container status \"e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2\": rpc error: code = NotFound desc = could not find container \"e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2\": container with ID starting with e57fe3b29c216792336ebcdaa42e2795fd541f233d898dcafdd53c1e7b5dd1a2 not found: ID does not exist" Sep 18 16:08:41.985753 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.985709 2563 scope.go:117] "RemoveContainer" containerID="375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248" Sep 18 16:08:41.985948 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.985931 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248\": container with ID starting with 375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248 not found: ID does not exist" containerID="375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248" Sep 18 16:08:41.985990 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.985955 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248"} err="failed to get container status \"375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248\": rpc error: code = NotFound desc = could not find container \"375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248\": container with ID starting with 375acae6a354b466484a0a0fa35a8f618648fdfc6ce34464f75daad001843248 not found: ID does not exist" Sep 18 16:08:41.985990 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.985970 2563 scope.go:117] "RemoveContainer" containerID="9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8" Sep 18 16:08:41.986224 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.986209 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8\": container with ID starting with 9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8 not found: ID does not exist" containerID="9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8" Sep 18 16:08:41.986286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986227 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8"} err="failed to get container status \"9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8\": rpc error: code = NotFound desc = could not find container \"9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8\": container with ID starting with 9676528623cb911fc2ce7a9f02ffefb98978397e402d3b4213363ec831af15d8 not found: ID does not exist" Sep 18 16:08:41.986286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986241 2563 scope.go:117] "RemoveContainer" containerID="299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253" Sep 18 16:08:41.986467 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.986453 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253\": container with ID starting with 299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253 not found: ID does not exist" containerID="299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253" Sep 18 16:08:41.986505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986469 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253"} err="failed to get container status \"299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253\": rpc error: code = NotFound desc = could not find container \"299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253\": container with ID starting with 299605bf00bf1c8d611f43ef3f8e7e90e5492c57c00344d052e482dc29154253 not found: ID does not exist" Sep 18 16:08:41.986505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986481 2563 scope.go:117] "RemoveContainer" containerID="69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b" Sep 18 16:08:41.986670 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.986655 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b\": container with ID starting with 69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b not found: ID does not exist" containerID="69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b" Sep 18 16:08:41.986711 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986671 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b"} err="failed to get container status \"69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b\": rpc error: code = NotFound desc = could not find container \"69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b\": container with ID starting with 69eba11170b2d997771aa06baaef0dd6dfbf666317f602c1a554d11de0a6074b not found: ID does not exist" Sep 18 16:08:41.986711 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986682 2563 scope.go:117] "RemoveContainer" containerID="82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f" Sep 18 16:08:41.986891 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.986873 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f\": container with ID starting with 82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f not found: ID does not exist" containerID="82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f" Sep 18 16:08:41.986930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986897 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f"} err="failed to get container status \"82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f\": rpc error: code = NotFound desc = could not find container \"82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f\": container with ID starting with 82a907065e405b50e819d9a9cc58b50b23381af10a5a0976e7702111e8544b8f not found: ID does not exist" Sep 18 16:08:41.986930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.986912 2563 scope.go:117] "RemoveContainer" containerID="afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3" Sep 18 16:08:41.987165 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:08:41.987151 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3\": container with ID starting with afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3 not found: ID does not exist" containerID="afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3" Sep 18 16:08:41.987227 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.987167 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3"} err="failed to get container status \"afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3\": rpc error: code = NotFound desc = could not find container \"afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3\": container with ID starting with afb830cfd54f41d4ace9347e812993a5a24313d2cf9e55abf002c108615474b3 not found: ID does not exist" Sep 18 16:08:41.990030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990011 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:08:41.990269 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990257 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="init-config-reloader" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990271 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="init-config-reloader" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990280 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="prom-label-proxy" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990286 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="prom-label-proxy" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990292 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="config-reloader" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990297 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="config-reloader" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990306 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-web" Sep 18 16:08:41.990321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990312 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-web" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990326 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-metric" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990332 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-metric" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990340 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990345 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990352 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="alertmanager" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990357 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="alertmanager" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990408 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-web" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990415 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="alertmanager" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990420 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990425 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="kube-rbac-proxy-metric" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990453 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="prom-label-proxy" Sep 18 16:08:41.990515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.990460 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" containerName="config-reloader" Sep 18 16:08:41.993495 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.993480 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:41.995966 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.995938 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lhc9c\"" Sep 18 16:08:41.995966 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.995955 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 18 16:08:41.996169 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996151 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 18 16:08:41.996330 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996303 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 18 16:08:41.996389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996312 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 18 16:08:41.996580 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996564 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 18 16:08:41.996626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 18 16:08:41.996626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996614 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 18 16:08:41.996710 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:41.996628 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 18 16:08:42.001380 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.001361 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 18 16:08:42.009703 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.009682 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:08:42.065886 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.065858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.065895 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-config-volume\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.065914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.065932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b87742c0-2af7-462c-afcb-a38343acfe63-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.065991 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b87742c0-2af7-462c-afcb-a38343acfe63-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066059 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066120 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87742c0-2af7-462c-afcb-a38343acfe63-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b87742c0-2af7-462c-afcb-a38343acfe63-config-out\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-web-config\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5x2\" (UniqueName: \"kubernetes.io/projected/b87742c0-2af7-462c-afcb-a38343acfe63-kube-api-access-5h5x2\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066239 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.066421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.066263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b87742c0-2af7-462c-afcb-a38343acfe63-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.166934 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.166911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b87742c0-2af7-462c-afcb-a38343acfe63-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.166939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.166957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87742c0-2af7-462c-afcb-a38343acfe63-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.166973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b87742c0-2af7-462c-afcb-a38343acfe63-config-out\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.166991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-web-config\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167245 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5x2\" (UniqueName: \"kubernetes.io/projected/b87742c0-2af7-462c-afcb-a38343acfe63-kube-api-access-5h5x2\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167245 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167186 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167245 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b87742c0-2af7-462c-afcb-a38343acfe63-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-config-volume\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167326 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b87742c0-2af7-462c-afcb-a38343acfe63-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.167463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.168732 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.167993 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b87742c0-2af7-462c-afcb-a38343acfe63-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.168732 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.168644 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b87742c0-2af7-462c-afcb-a38343acfe63-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.168732 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.168678 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b87742c0-2af7-462c-afcb-a38343acfe63-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170127 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170055 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170534 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170483 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b87742c0-2af7-462c-afcb-a38343acfe63-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170736 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170507 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170590 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-config-volume\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170709 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b87742c0-2af7-462c-afcb-a38343acfe63-config-out\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.170897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.170875 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-web-config\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.171469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.171450 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.171819 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.171802 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b87742c0-2af7-462c-afcb-a38343acfe63-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.175542 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.175520 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5x2\" (UniqueName: \"kubernetes.io/projected/b87742c0-2af7-462c-afcb-a38343acfe63-kube-api-access-5h5x2\") pod \"alertmanager-main-0\" (UID: \"b87742c0-2af7-462c-afcb-a38343acfe63\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.303228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.303205 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 16:08:42.579258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.579233 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 16:08:42.581734 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:08:42.581692 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87742c0_2af7_462c_afcb_a38343acfe63.slice/crio-405f2f8cedae17bc985251dfac5083fb35dd34401c9b28ff6d4cf1a3d30ab6ce WatchSource:0}: Error finding container 405f2f8cedae17bc985251dfac5083fb35dd34401c9b28ff6d4cf1a3d30ab6ce: Status 404 returned error can't find the container with id 405f2f8cedae17bc985251dfac5083fb35dd34401c9b28ff6d4cf1a3d30ab6ce Sep 18 16:08:42.940906 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.940836 2563 generic.go:356] "Generic (PLEG): container finished" podID="b87742c0-2af7-462c-afcb-a38343acfe63" containerID="1d18c37824485c174853592c4e7d02afad68d3d8030469055cf962a214fe64df" exitCode=0 Sep 18 16:08:42.940906 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.940885 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerDied","Data":"1d18c37824485c174853592c4e7d02afad68d3d8030469055cf962a214fe64df"} Sep 18 16:08:42.941289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:42.940929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"405f2f8cedae17bc985251dfac5083fb35dd34401c9b28ff6d4cf1a3d30ab6ce"} Sep 18 16:08:43.083983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.083961 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e943ec65-12d7-467c-957e-5e416a6023d0" path="/var/lib/kubelet/pods/e943ec65-12d7-467c-957e-5e416a6023d0/volumes" Sep 18 16:08:43.948204 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.948168 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"1e65e240b1e3517ec2923b821d0be93f0a105b80a24ce241c85d0f08ddd7355e"} Sep 18 16:08:43.948561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.948234 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"19d0200e5bcae9d75a89d82804e59ebe68ebf0bc05016477ca3b1b3b39c198a9"} Sep 18 16:08:43.948561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.948259 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"7b2b0d566f5fc9d7f2d711aafbf21aea28e4678a315b24dc3a6af389ffaa7314"} Sep 18 16:08:43.948561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.948283 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"2a76645d6767d64d0feb1c4fb209faff4c1739cf9311b09b6cbf25a2c1f277fe"} Sep 18 16:08:43.948561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.948306 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"86322260a9c3e7fabe6d2c4ea1b52f843c6e66dd4c12803d1fd51177b49ce870"} Sep 18 16:08:43.948561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.948331 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b87742c0-2af7-462c-afcb-a38343acfe63","Type":"ContainerStarted","Data":"389057740c87f3d38eced832981c80a6f5d8b1832e04922b1541e52d43bb702d"} Sep 18 16:08:43.977723 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:43.977672 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.958890342 podStartE2EDuration="2.977659691s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.769349ms" isStatefulPod=false podCreationTimestamp="2026-09-18 16:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:08:43.976108855 +0000 UTC m=+223.536953522" watchObservedRunningTime="2026-09-18 16:08:43.977659691 +0000 UTC m=+223.538504359" Sep 18 16:08:45.542349 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:45.542301 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:45.599082 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:45.599029 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:48.919254 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:48.919206 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:48.919254 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:48.919265 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:48.923864 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:48.923841 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:48.971564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:48.971536 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:08:49.122173 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:49.122142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:49.224445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:49.224361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:49.286552 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:49.286529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:08:51.515752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:08:51.515712 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:09:00.549904 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:09:00.549871 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:09:00.642817 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:09:00.642788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:09:07.476446 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:09:07.476410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:09:22.621155 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:09:22.621066 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:09:36.799335 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:09:36.799303 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:02.957775 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:02.957746 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:10:02.961233 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:02.961201 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:10:02.968288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:02.968264 2563 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 16:10:12.667365 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:12.667328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:12.723117 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:12.723065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:18.151551 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:18.151517 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:18.259388 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:18.259357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:18.916102 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:18.916061 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd"] Sep 18 16:10:18.918267 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:18.918250 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:18.921926 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:10:18.921893 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:18.922055 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:10:18.921968 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 18 16:10:18.922055 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:10:18.921971 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-z4rdv\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4rdv\"" type="*v1.Secret" Sep 18 16:10:18.922160 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:10:18.922050 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 18 16:10:18.937019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:18.936993 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd"] Sep 18 16:10:19.095341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.095311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g476k\" (UniqueName: \"kubernetes.io/projected/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-kube-api-access-g476k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.095341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.095344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.095542 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.095373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.196415 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.196351 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.196723 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.196410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g476k\" (UniqueName: \"kubernetes.io/projected/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-kube-api-access-g476k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.196723 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.196441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.196723 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.196696 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.196865 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.196723 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:19.732901 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:19.732865 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 16:10:20.410073 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:20.410046 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4rdv\"" Sep 18 16:10:20.455621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:20.455598 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 16:10:20.466914 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:20.466890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g476k\" (UniqueName: \"kubernetes.io/projected/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-kube-api-access-g476k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:20.727674 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:20.727580 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:20.997521 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:20.997496 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd"] Sep 18 16:10:20.999629 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:10:20.999600 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91b6288_1c0b_4961_9b6a_c729e39e6a3e.slice/crio-95f45aa6bfd4d893ba3aacdc3aaf377ea2b6ac3f20a33cd5a6e5e6fc776a9bac WatchSource:0}: Error finding container 95f45aa6bfd4d893ba3aacdc3aaf377ea2b6ac3f20a33cd5a6e5e6fc776a9bac: Status 404 returned error can't find the container with id 95f45aa6bfd4d893ba3aacdc3aaf377ea2b6ac3f20a33cd5a6e5e6fc776a9bac Sep 18 16:10:21.001533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:21.001516 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:10:21.300839 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:21.300809 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" event={"ID":"e91b6288-1c0b-4961-9b6a-c729e39e6a3e","Type":"ContainerStarted","Data":"95f45aa6bfd4d893ba3aacdc3aaf377ea2b6ac3f20a33cd5a6e5e6fc776a9bac"} Sep 18 16:10:27.327519 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:27.327488 2563 generic.go:356] "Generic (PLEG): container finished" podID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerID="857bda19f8a2aba2772ba278f39d2785876125ab206e052f0ef69249a9d5320b" exitCode=0 Sep 18 16:10:27.327879 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:27.327536 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" event={"ID":"e91b6288-1c0b-4961-9b6a-c729e39e6a3e","Type":"ContainerDied","Data":"857bda19f8a2aba2772ba278f39d2785876125ab206e052f0ef69249a9d5320b"} Sep 18 16:10:30.341283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:30.341251 2563 generic.go:356] "Generic (PLEG): container finished" podID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerID="5977b2f325313f67d856bd312022f54374300f356e19ca72aef7950ba289c26b" exitCode=0 Sep 18 16:10:30.341642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:30.341296 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" event={"ID":"e91b6288-1c0b-4961-9b6a-c729e39e6a3e","Type":"ContainerDied","Data":"5977b2f325313f67d856bd312022f54374300f356e19ca72aef7950ba289c26b"} Sep 18 16:10:38.377139 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:38.377100 2563 generic.go:356] "Generic (PLEG): container finished" podID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerID="fb89fef3fdd7d42283f75e20be825c17c0a56bc1e8d6f8b05dbbe886006f431a" exitCode=0 Sep 18 16:10:38.377139 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:38.377139 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" event={"ID":"e91b6288-1c0b-4961-9b6a-c729e39e6a3e","Type":"ContainerDied","Data":"fb89fef3fdd7d42283f75e20be825c17c0a56bc1e8d6f8b05dbbe886006f431a"} Sep 18 16:10:39.502676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.502649 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:39.551177 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.551147 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-util\") pod \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " Sep 18 16:10:39.551317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.551207 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g476k\" (UniqueName: \"kubernetes.io/projected/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-kube-api-access-g476k\") pod \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " Sep 18 16:10:39.551317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.551239 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-bundle\") pod \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\" (UID: \"e91b6288-1c0b-4961-9b6a-c729e39e6a3e\") " Sep 18 16:10:39.551845 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.551825 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-bundle" (OuterVolumeSpecName: "bundle") pod "e91b6288-1c0b-4961-9b6a-c729e39e6a3e" (UID: "e91b6288-1c0b-4961-9b6a-c729e39e6a3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:10:39.553278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.553257 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-kube-api-access-g476k" (OuterVolumeSpecName: "kube-api-access-g476k") pod "e91b6288-1c0b-4961-9b6a-c729e39e6a3e" (UID: "e91b6288-1c0b-4961-9b6a-c729e39e6a3e"). InnerVolumeSpecName "kube-api-access-g476k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:10:39.555177 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.555148 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-util" (OuterVolumeSpecName: "util") pod "e91b6288-1c0b-4961-9b6a-c729e39e6a3e" (UID: "e91b6288-1c0b-4961-9b6a-c729e39e6a3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:10:39.652202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.652142 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g476k\" (UniqueName: \"kubernetes.io/projected/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-kube-api-access-g476k\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:10:39.652202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.652164 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:10:39.652202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:39.652173 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e91b6288-1c0b-4961-9b6a-c729e39e6a3e-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:10:40.387311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:40.387284 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" Sep 18 16:10:40.387311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:40.387298 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402p2vd" event={"ID":"e91b6288-1c0b-4961-9b6a-c729e39e6a3e","Type":"ContainerDied","Data":"95f45aa6bfd4d893ba3aacdc3aaf377ea2b6ac3f20a33cd5a6e5e6fc776a9bac"} Sep 18 16:10:40.387515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:40.387337 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95f45aa6bfd4d893ba3aacdc3aaf377ea2b6ac3f20a33cd5a6e5e6fc776a9bac" Sep 18 16:10:45.153823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:45.153741 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:46.505017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:46.504981 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:47.124624 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124594 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn"] Sep 18 16:10:47.124885 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124872 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="util" Sep 18 16:10:47.124929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124886 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="util" Sep 18 16:10:47.124929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124896 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="pull" Sep 18 16:10:47.124929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124902 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="pull" Sep 18 16:10:47.124929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124913 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="extract" Sep 18 16:10:47.124929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124918 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="extract" Sep 18 16:10:47.125075 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.124975 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e91b6288-1c0b-4961-9b6a-c729e39e6a3e" containerName="extract" Sep 18 16:10:47.145307 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.145282 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.145954 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.145934 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn"] Sep 18 16:10:47.148289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.148267 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:10:47.148405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.148297 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 18 16:10:47.148405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.148360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r6bjb\"" Sep 18 16:10:47.206677 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.206652 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5ls\" (UniqueName: \"kubernetes.io/projected/1a61c3d2-0189-4126-838d-884662540d50-kube-api-access-cs5ls\") pod \"cert-manager-operator-controller-manager-55c444bf86-2r5cn\" (UID: \"1a61c3d2-0189-4126-838d-884662540d50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.206800 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.206692 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a61c3d2-0189-4126-838d-884662540d50-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2r5cn\" (UID: \"1a61c3d2-0189-4126-838d-884662540d50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.308032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.307988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a61c3d2-0189-4126-838d-884662540d50-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2r5cn\" (UID: \"1a61c3d2-0189-4126-838d-884662540d50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.308213 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.308115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5ls\" (UniqueName: \"kubernetes.io/projected/1a61c3d2-0189-4126-838d-884662540d50-kube-api-access-cs5ls\") pod \"cert-manager-operator-controller-manager-55c444bf86-2r5cn\" (UID: \"1a61c3d2-0189-4126-838d-884662540d50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.308421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.308396 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a61c3d2-0189-4126-838d-884662540d50-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2r5cn\" (UID: \"1a61c3d2-0189-4126-838d-884662540d50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.320788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.320760 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5ls\" (UniqueName: \"kubernetes.io/projected/1a61c3d2-0189-4126-838d-884662540d50-kube-api-access-cs5ls\") pod \"cert-manager-operator-controller-manager-55c444bf86-2r5cn\" (UID: \"1a61c3d2-0189-4126-838d-884662540d50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.454877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.454789 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" Sep 18 16:10:47.747172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:47.747148 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn"] Sep 18 16:10:47.748619 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:10:47.748591 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a61c3d2_0189_4126_838d_884662540d50.slice/crio-fb5db98e19eda6b59e7d7033967ed1596b2f648b0866b09a7497c8692b7c135f WatchSource:0}: Error finding container fb5db98e19eda6b59e7d7033967ed1596b2f648b0866b09a7497c8692b7c135f: Status 404 returned error can't find the container with id fb5db98e19eda6b59e7d7033967ed1596b2f648b0866b09a7497c8692b7c135f Sep 18 16:10:48.419590 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:48.419549 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" event={"ID":"1a61c3d2-0189-4126-838d-884662540d50","Type":"ContainerStarted","Data":"fb5db98e19eda6b59e7d7033967ed1596b2f648b0866b09a7497c8692b7c135f"} Sep 18 16:10:48.773999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:48.773954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:48.873032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:48.873000 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:50.188791 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.188761 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:50.288663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.288634 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:50.429750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.429720 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" event={"ID":"1a61c3d2-0189-4126-838d-884662540d50","Type":"ContainerStarted","Data":"035520c86ab20c0b77472b2b8b3d2e3c9046123ea5f36dec2fdcb3e567e79a8e"} Sep 18 16:10:50.453353 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.453244 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" podStartSLOduration=1.778221483 podStartE2EDuration="3.453227267s" totalImagesPullingTime="1.675005784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:10:50.450541793 +0000 UTC m=+350.011386503" watchObservedRunningTime="2026-09-18 16:10:50.453227267 +0000 UTC m=+350.014071934" Sep 18 16:10:50.518566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.518533 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kfmrt"] Sep 18 16:10:50.520618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.520603 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.524146 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.524118 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 18 16:10:50.524348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.524335 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gscf4\"" Sep 18 16:10:50.524761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.524745 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 18 16:10:50.540113 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.540072 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kfmrt"] Sep 18 16:10:50.635646 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.635600 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72crc\" (UniqueName: \"kubernetes.io/projected/d20d22e7-f1d8-45b6-84d0-b4b2de628a2f-kube-api-access-72crc\") pod \"cert-manager-webhook-6486ffd997-kfmrt\" (UID: \"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.635646 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.635640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d20d22e7-f1d8-45b6-84d0-b4b2de628a2f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kfmrt\" (UID: \"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.737044 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.736952 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72crc\" (UniqueName: \"kubernetes.io/projected/d20d22e7-f1d8-45b6-84d0-b4b2de628a2f-kube-api-access-72crc\") pod \"cert-manager-webhook-6486ffd997-kfmrt\" (UID: \"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.737044 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.736992 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d20d22e7-f1d8-45b6-84d0-b4b2de628a2f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kfmrt\" (UID: \"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.747431 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.747406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d20d22e7-f1d8-45b6-84d0-b4b2de628a2f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kfmrt\" (UID: \"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.747708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.747681 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72crc\" (UniqueName: \"kubernetes.io/projected/d20d22e7-f1d8-45b6-84d0-b4b2de628a2f-kube-api-access-72crc\") pod \"cert-manager-webhook-6486ffd997-kfmrt\" (UID: \"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.844371 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.844335 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:50.868626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.868598 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:50.989297 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:50.989206 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:51.126195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.126170 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kfmrt"] Sep 18 16:10:51.128385 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:10:51.128353 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd20d22e7_f1d8_45b6_84d0_b4b2de628a2f.slice/crio-c2aadabbbff63eac4b3b99fe2eb9784a72e6342176055e45c7897186d57ba7a4 WatchSource:0}: Error finding container c2aadabbbff63eac4b3b99fe2eb9784a72e6342176055e45c7897186d57ba7a4: Status 404 returned error can't find the container with id c2aadabbbff63eac4b3b99fe2eb9784a72e6342176055e45c7897186d57ba7a4 Sep 18 16:10:51.436930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.436899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" event={"ID":"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f","Type":"ContainerStarted","Data":"c2aadabbbff63eac4b3b99fe2eb9784a72e6342176055e45c7897186d57ba7a4"} Sep 18 16:10:51.633469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.633435 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx"] Sep 18 16:10:51.636264 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.636246 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.639208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.639184 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 16:10:51.640235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.640220 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 16:10:51.640287 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.640226 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4rdv\"" Sep 18 16:10:51.659788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.659765 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx"] Sep 18 16:10:51.745426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.745355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.745426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.745391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.745426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.745419 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplzh\" (UniqueName: \"kubernetes.io/projected/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-kube-api-access-kplzh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.846675 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.846636 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.846797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.846686 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.846797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.846716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplzh\" (UniqueName: \"kubernetes.io/projected/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-kube-api-access-kplzh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.847024 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.847007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.847060 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.847037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.855214 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.855187 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplzh\" (UniqueName: \"kubernetes.io/projected/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-kube-api-access-kplzh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:51.945337 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:51.945311 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:52.214537 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:52.214486 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx"] Sep 18 16:10:52.216609 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:10:52.216580 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0157fd42_40c5_49e4_b0fc_ad45282ea6d8.slice/crio-64e7a1461a029f1a07cca1d80014a24dbbedbf5ce1b3b4267ee5f6c35fbcfae4 WatchSource:0}: Error finding container 64e7a1461a029f1a07cca1d80014a24dbbedbf5ce1b3b4267ee5f6c35fbcfae4: Status 404 returned error can't find the container with id 64e7a1461a029f1a07cca1d80014a24dbbedbf5ce1b3b4267ee5f6c35fbcfae4 Sep 18 16:10:52.364958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:52.364907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:52.443072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:52.443040 2563 generic.go:356] "Generic (PLEG): container finished" podID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerID="49b8635362b0f316fd2c3c36455569a0071d58963859ea10d90f738bd809af88" exitCode=0 Sep 18 16:10:52.443479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:52.443152 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" event={"ID":"0157fd42-40c5-49e4-b0fc-ad45282ea6d8","Type":"ContainerDied","Data":"49b8635362b0f316fd2c3c36455569a0071d58963859ea10d90f738bd809af88"} Sep 18 16:10:52.443479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:52.443197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" event={"ID":"0157fd42-40c5-49e4-b0fc-ad45282ea6d8","Type":"ContainerStarted","Data":"64e7a1461a029f1a07cca1d80014a24dbbedbf5ce1b3b4267ee5f6c35fbcfae4"} Sep 18 16:10:52.549253 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:52.549208 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:55.459793 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.459700 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" event={"ID":"d20d22e7-f1d8-45b6-84d0-b4b2de628a2f","Type":"ContainerStarted","Data":"8db46872b8ca3d9d2e2b8770860f59988cc51659d848fc9c89b295c8c16c1bb9"} Sep 18 16:10:55.459793 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.459784 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:10:55.461755 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.461726 2563 generic.go:356] "Generic (PLEG): container finished" podID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerID="dbcb23e798c324298263043395255515093a12aefe51f43ad3ff27ffa03580a4" exitCode=0 Sep 18 16:10:55.461854 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.461755 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" event={"ID":"0157fd42-40c5-49e4-b0fc-ad45282ea6d8","Type":"ContainerDied","Data":"dbcb23e798c324298263043395255515093a12aefe51f43ad3ff27ffa03580a4"} Sep 18 16:10:55.477569 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.477528 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" podStartSLOduration=1.521878499 podStartE2EDuration="5.477515685s" totalImagesPullingTime="3.955637186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:10:55.476556403 +0000 UTC m=+355.037401084" watchObservedRunningTime="2026-09-18 16:10:55.477515685 +0000 UTC m=+355.038360352" Sep 18 16:10:55.760291 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.760257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:55.822099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:55.822055 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:10:56.468564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:56.468530 2563 generic.go:356] "Generic (PLEG): container finished" podID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerID="30a029aa11775e3dc70bf7d264a2f59bfe57d505a9350efcbaab6aa3eda705a2" exitCode=0 Sep 18 16:10:56.469045 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:56.468621 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" event={"ID":"0157fd42-40c5-49e4-b0fc-ad45282ea6d8","Type":"ContainerDied","Data":"30a029aa11775e3dc70bf7d264a2f59bfe57d505a9350efcbaab6aa3eda705a2"} Sep 18 16:10:57.596594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.596569 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:57.698397 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.698368 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-bundle\") pod \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " Sep 18 16:10:57.698579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.698409 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kplzh\" (UniqueName: \"kubernetes.io/projected/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-kube-api-access-kplzh\") pod \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " Sep 18 16:10:57.698579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.698467 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-util\") pod \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\" (UID: \"0157fd42-40c5-49e4-b0fc-ad45282ea6d8\") " Sep 18 16:10:57.698812 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.698789 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-bundle" (OuterVolumeSpecName: "bundle") pod "0157fd42-40c5-49e4-b0fc-ad45282ea6d8" (UID: "0157fd42-40c5-49e4-b0fc-ad45282ea6d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:10:57.700502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.700480 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-kube-api-access-kplzh" (OuterVolumeSpecName: "kube-api-access-kplzh") pod "0157fd42-40c5-49e4-b0fc-ad45282ea6d8" (UID: "0157fd42-40c5-49e4-b0fc-ad45282ea6d8"). InnerVolumeSpecName "kube-api-access-kplzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:10:57.705427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.705402 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-util" (OuterVolumeSpecName: "util") pod "0157fd42-40c5-49e4-b0fc-ad45282ea6d8" (UID: "0157fd42-40c5-49e4-b0fc-ad45282ea6d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:10:57.799199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.799163 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:10:57.799199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.799198 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplzh\" (UniqueName: \"kubernetes.io/projected/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-kube-api-access-kplzh\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:10:57.799199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:57.799207 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0157fd42-40c5-49e4-b0fc-ad45282ea6d8-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:10:58.479529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:58.479499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" event={"ID":"0157fd42-40c5-49e4-b0fc-ad45282ea6d8","Type":"ContainerDied","Data":"64e7a1461a029f1a07cca1d80014a24dbbedbf5ce1b3b4267ee5f6c35fbcfae4"} Sep 18 16:10:58.479529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:58.479532 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvxcvx" Sep 18 16:10:58.479726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:10:58.479547 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64e7a1461a029f1a07cca1d80014a24dbbedbf5ce1b3b4267ee5f6c35fbcfae4" Sep 18 16:11:00.779190 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:00.779159 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:01.470828 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:01.470797 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kfmrt" Sep 18 16:11:02.188015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:02.187989 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:02.285929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:02.285904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:03.025941 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:03.025911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:03.086189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:03.086164 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:04.217685 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.217652 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm"] Sep 18 16:11:04.218046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.217954 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="extract" Sep 18 16:11:04.218046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.217966 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="extract" Sep 18 16:11:04.218046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.217980 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="util" Sep 18 16:11:04.218046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.217986 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="util" Sep 18 16:11:04.218046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.218001 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="pull" Sep 18 16:11:04.218046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.218009 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="pull" Sep 18 16:11:04.218252 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.218061 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0157fd42-40c5-49e4-b0fc-ad45282ea6d8" containerName="extract" Sep 18 16:11:04.219991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.219976 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.223154 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.223131 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 18 16:11:04.224251 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.224234 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5zw7d\"" Sep 18 16:11:04.224251 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.224242 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 18 16:11:04.230229 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.230208 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm"] Sep 18 16:11:04.346364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.346327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n822\" (UniqueName: \"kubernetes.io/projected/f25112d0-8747-4ead-8f96-9b968a57d914-kube-api-access-8n822\") pod \"openshift-lws-operator-fd8ccff4c-wvvkm\" (UID: \"f25112d0-8747-4ead-8f96-9b968a57d914\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.346510 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.346380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f25112d0-8747-4ead-8f96-9b968a57d914-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wvvkm\" (UID: \"f25112d0-8747-4ead-8f96-9b968a57d914\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.446699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.446658 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n822\" (UniqueName: \"kubernetes.io/projected/f25112d0-8747-4ead-8f96-9b968a57d914-kube-api-access-8n822\") pod \"openshift-lws-operator-fd8ccff4c-wvvkm\" (UID: \"f25112d0-8747-4ead-8f96-9b968a57d914\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.446699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.446706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f25112d0-8747-4ead-8f96-9b968a57d914-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wvvkm\" (UID: \"f25112d0-8747-4ead-8f96-9b968a57d914\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.447038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.447024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f25112d0-8747-4ead-8f96-9b968a57d914-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wvvkm\" (UID: \"f25112d0-8747-4ead-8f96-9b968a57d914\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.455719 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.455690 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n822\" (UniqueName: \"kubernetes.io/projected/f25112d0-8747-4ead-8f96-9b968a57d914-kube-api-access-8n822\") pod \"openshift-lws-operator-fd8ccff4c-wvvkm\" (UID: \"f25112d0-8747-4ead-8f96-9b968a57d914\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.529003 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.528977 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" Sep 18 16:11:04.798341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:04.798270 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm"] Sep 18 16:11:04.800342 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:11:04.800310 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf25112d0_8747_4ead_8f96_9b968a57d914.slice/crio-cf3fbac8e62a0d81faedfe9827929b97d659d4e4d385d9c8760672133b85a56e WatchSource:0}: Error finding container cf3fbac8e62a0d81faedfe9827929b97d659d4e4d385d9c8760672133b85a56e: Status 404 returned error can't find the container with id cf3fbac8e62a0d81faedfe9827929b97d659d4e4d385d9c8760672133b85a56e Sep 18 16:11:05.512584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:05.512550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerStarted","Data":"cf3fbac8e62a0d81faedfe9827929b97d659d4e4d385d9c8760672133b85a56e"} Sep 18 16:11:08.527186 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:08.527147 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerStarted","Data":"359b3bdb0ecfe3fc6f67c0fe750e99c6030fe01efc93af3e2a0dc6e5230a81df"} Sep 18 16:11:08.544316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:08.544267 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" podStartSLOduration=1.8663865309999998 podStartE2EDuration="4.544251838s" totalImagesPullingTime="2.677865307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:08.543117271 +0000 UTC m=+368.103961932" watchObservedRunningTime="2026-09-18 16:11:08.544251838 +0000 UTC m=+368.105096505" Sep 18 16:11:09.183201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:09.183171 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:09.283692 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:09.283655 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:10.687231 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.687195 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xxvn7"] Sep 18 16:11:10.689481 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.689463 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.695116 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.695077 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5qj9k\"" Sep 18 16:11:10.708964 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.708935 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xxvn7"] Sep 18 16:11:10.797985 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.797955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8vk\" (UniqueName: \"kubernetes.io/projected/2e14a582-6f0c-454e-87c9-73fbb5852911-kube-api-access-bp8vk\") pod \"cert-manager-55b4d954d5-xxvn7\" (UID: \"2e14a582-6f0c-454e-87c9-73fbb5852911\") " pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.798135 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.798016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14a582-6f0c-454e-87c9-73fbb5852911-bound-sa-token\") pod \"cert-manager-55b4d954d5-xxvn7\" (UID: \"2e14a582-6f0c-454e-87c9-73fbb5852911\") " pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.815417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.815388 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm"] Sep 18 16:11:10.817967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.817953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:10.820681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.820652 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4rdv\"" Sep 18 16:11:10.820681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.820670 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 16:11:10.820816 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.820673 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 16:11:10.826855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.826836 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm"] Sep 18 16:11:10.898397 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.898371 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8vk\" (UniqueName: \"kubernetes.io/projected/2e14a582-6f0c-454e-87c9-73fbb5852911-kube-api-access-bp8vk\") pod \"cert-manager-55b4d954d5-xxvn7\" (UID: \"2e14a582-6f0c-454e-87c9-73fbb5852911\") " pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.898522 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.898418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:10.898522 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.898455 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqm7\" (UniqueName: \"kubernetes.io/projected/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-kube-api-access-8vqm7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:10.898522 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.898487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14a582-6f0c-454e-87c9-73fbb5852911-bound-sa-token\") pod \"cert-manager-55b4d954d5-xxvn7\" (UID: \"2e14a582-6f0c-454e-87c9-73fbb5852911\") " pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.898646 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.898547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:10.907108 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.907063 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14a582-6f0c-454e-87c9-73fbb5852911-bound-sa-token\") pod \"cert-manager-55b4d954d5-xxvn7\" (UID: \"2e14a582-6f0c-454e-87c9-73fbb5852911\") " pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.907204 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.907152 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8vk\" (UniqueName: \"kubernetes.io/projected/2e14a582-6f0c-454e-87c9-73fbb5852911-kube-api-access-bp8vk\") pod \"cert-manager-55b4d954d5-xxvn7\" (UID: \"2e14a582-6f0c-454e-87c9-73fbb5852911\") " pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.998328 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.998309 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:11:10.999607 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.999587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:10.999682 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.999628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqm7\" (UniqueName: \"kubernetes.io/projected/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-kube-api-access-8vqm7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:10.999682 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:10.999674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:11.000031 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.000004 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:11.000148 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.000011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:11.007493 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.007474 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqm7\" (UniqueName: \"kubernetes.io/projected/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-kube-api-access-8vqm7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:11.073804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.073767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:11.126770 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.126739 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:11.267566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.267538 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xxvn7"] Sep 18 16:11:11.268942 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:11:11.268914 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e14a582_6f0c_454e_87c9_73fbb5852911.slice/crio-92c4d971801a857a1d193972d9df29cd7801026983bbfeaaebf86e712ad80593 WatchSource:0}: Error finding container 92c4d971801a857a1d193972d9df29cd7801026983bbfeaaebf86e712ad80593: Status 404 returned error can't find the container with id 92c4d971801a857a1d193972d9df29cd7801026983bbfeaaebf86e712ad80593 Sep 18 16:11:11.395599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.395557 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm"] Sep 18 16:11:11.398485 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:11:11.398461 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8777e559_3c6e_4d4a_8f1e_34dfa02920eb.slice/crio-8533434fca1986ff14d6d0aaeca5d667a09c6e7cda4af4849a0de448811323ef WatchSource:0}: Error finding container 8533434fca1986ff14d6d0aaeca5d667a09c6e7cda4af4849a0de448811323ef: Status 404 returned error can't find the container with id 8533434fca1986ff14d6d0aaeca5d667a09c6e7cda4af4849a0de448811323ef Sep 18 16:11:11.541287 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.541210 2563 generic.go:356] "Generic (PLEG): container finished" podID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerID="f2b15cccfa3d235b5766f20f3257352c246a1b924967d28ec4607c975d09e49d" exitCode=0 Sep 18 16:11:11.541426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.541282 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" event={"ID":"8777e559-3c6e-4d4a-8f1e-34dfa02920eb","Type":"ContainerDied","Data":"f2b15cccfa3d235b5766f20f3257352c246a1b924967d28ec4607c975d09e49d"} Sep 18 16:11:11.541426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.541337 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" event={"ID":"8777e559-3c6e-4d4a-8f1e-34dfa02920eb","Type":"ContainerStarted","Data":"8533434fca1986ff14d6d0aaeca5d667a09c6e7cda4af4849a0de448811323ef"} Sep 18 16:11:11.542881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.542859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerStarted","Data":"3603836f51dc4046b8275cbc276d9c1d03e5cdd239f6ab31ee61fc4625a73961"} Sep 18 16:11:11.542992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.542891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerStarted","Data":"92c4d971801a857a1d193972d9df29cd7801026983bbfeaaebf86e712ad80593"} Sep 18 16:11:11.577274 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:11.577236 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" podStartSLOduration=1.57722348 podStartE2EDuration="1.57722348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:11.576184632 +0000 UTC m=+371.137029298" watchObservedRunningTime="2026-09-18 16:11:11.57722348 +0000 UTC m=+371.138068146" Sep 18 16:11:12.549038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:12.549004 2563 generic.go:356] "Generic (PLEG): container finished" podID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerID="ee77dcc7aeb2c3f29c38626a60987b183f7f83487f7e313861403dba4d51ef2b" exitCode=0 Sep 18 16:11:12.549426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:12.549115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" event={"ID":"8777e559-3c6e-4d4a-8f1e-34dfa02920eb","Type":"ContainerDied","Data":"ee77dcc7aeb2c3f29c38626a60987b183f7f83487f7e313861403dba4d51ef2b"} Sep 18 16:11:13.555535 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:13.555500 2563 generic.go:356] "Generic (PLEG): container finished" podID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerID="7ae38a7339bcfea2c2671b91b26e18f5b7f2216af0d4ad5946841851b98c48b8" exitCode=0 Sep 18 16:11:13.555909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:13.555585 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" event={"ID":"8777e559-3c6e-4d4a-8f1e-34dfa02920eb","Type":"ContainerDied","Data":"7ae38a7339bcfea2c2671b91b26e18f5b7f2216af0d4ad5946841851b98c48b8"} Sep 18 16:11:14.683613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.683593 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:14.829979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.829904 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-bundle\") pod \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " Sep 18 16:11:14.829979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.829934 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-util\") pod \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " Sep 18 16:11:14.830175 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.829986 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vqm7\" (UniqueName: \"kubernetes.io/projected/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-kube-api-access-8vqm7\") pod \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\" (UID: \"8777e559-3c6e-4d4a-8f1e-34dfa02920eb\") " Sep 18 16:11:14.830649 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.830619 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-bundle" (OuterVolumeSpecName: "bundle") pod "8777e559-3c6e-4d4a-8f1e-34dfa02920eb" (UID: "8777e559-3c6e-4d4a-8f1e-34dfa02920eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:11:14.832062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.832035 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-kube-api-access-8vqm7" (OuterVolumeSpecName: "kube-api-access-8vqm7") pod "8777e559-3c6e-4d4a-8f1e-34dfa02920eb" (UID: "8777e559-3c6e-4d4a-8f1e-34dfa02920eb"). InnerVolumeSpecName "kube-api-access-8vqm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:11:14.838337 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.838313 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-util" (OuterVolumeSpecName: "util") pod "8777e559-3c6e-4d4a-8f1e-34dfa02920eb" (UID: "8777e559-3c6e-4d4a-8f1e-34dfa02920eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:11:14.931230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.931188 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:11:14.931230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.931225 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:11:14.931230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:14.931239 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqm7\" (UniqueName: \"kubernetes.io/projected/8777e559-3c6e-4d4a-8f1e-34dfa02920eb-kube-api-access-8vqm7\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:11:15.566366 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:15.566332 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" event={"ID":"8777e559-3c6e-4d4a-8f1e-34dfa02920eb","Type":"ContainerDied","Data":"8533434fca1986ff14d6d0aaeca5d667a09c6e7cda4af4849a0de448811323ef"} Sep 18 16:11:15.566543 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:15.566408 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8533434fca1986ff14d6d0aaeca5d667a09c6e7cda4af4849a0de448811323ef" Sep 18 16:11:15.566543 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:15.566340 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqx4hm" Sep 18 16:11:21.424893 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.424855 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p"] Sep 18 16:11:21.425391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425278 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="pull" Sep 18 16:11:21.425391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425296 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="pull" Sep 18 16:11:21.425391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425325 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="util" Sep 18 16:11:21.425391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425333 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="util" Sep 18 16:11:21.425391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425343 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="extract" Sep 18 16:11:21.425391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425351 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="extract" Sep 18 16:11:21.425705 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.425430 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8777e559-3c6e-4d4a-8f1e-34dfa02920eb" containerName="extract" Sep 18 16:11:21.428054 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.428033 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.431198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.431179 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4rdv\"" Sep 18 16:11:21.431308 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.431218 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 16:11:21.431308 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.431230 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 16:11:21.437330 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.437310 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p"] Sep 18 16:11:21.582981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.582945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.583190 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.583026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.583190 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.583144 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/032139ce-fc2a-470b-a432-a74687697b4d-kube-api-access-x85rv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.683989 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.683898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.683989 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.683959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/032139ce-fc2a-470b-a432-a74687697b4d-kube-api-access-x85rv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.684244 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.684024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.684322 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.684298 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.684378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.684334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.692931 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.692907 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/032139ce-fc2a-470b-a432-a74687697b4d-kube-api-access-x85rv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:21.736612 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:21.736561 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:22.034254 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:22.034225 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p"] Sep 18 16:11:22.036175 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:11:22.036143 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032139ce_fc2a_470b_a432_a74687697b4d.slice/crio-d8166fdb8d77bc668222827865fecba4fd82341ae3fa9e6b237b724cd6ac1f0f WatchSource:0}: Error finding container d8166fdb8d77bc668222827865fecba4fd82341ae3fa9e6b237b724cd6ac1f0f: Status 404 returned error can't find the container with id d8166fdb8d77bc668222827865fecba4fd82341ae3fa9e6b237b724cd6ac1f0f Sep 18 16:11:22.482780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:22.482708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:22.603352 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:22.603318 2563 generic.go:356] "Generic (PLEG): container finished" podID="032139ce-fc2a-470b-a432-a74687697b4d" containerID="3f89e9312aee6370331a7946e49bd0afba3887b863b9f0fc5040054ae778ccc6" exitCode=0 Sep 18 16:11:22.603516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:22.603399 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" event={"ID":"032139ce-fc2a-470b-a432-a74687697b4d","Type":"ContainerDied","Data":"3f89e9312aee6370331a7946e49bd0afba3887b863b9f0fc5040054ae778ccc6"} Sep 18 16:11:22.603516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:22.603459 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" event={"ID":"032139ce-fc2a-470b-a432-a74687697b4d","Type":"ContainerStarted","Data":"d8166fdb8d77bc668222827865fecba4fd82341ae3fa9e6b237b724cd6ac1f0f"} Sep 18 16:11:23.605384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.605285 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj"] Sep 18 16:11:23.607730 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.607710 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.611372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.611342 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 16:11:23.611492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.611453 2563 generic.go:356] "Generic (PLEG): container finished" podID="032139ce-fc2a-470b-a432-a74687697b4d" containerID="cc30188db93b08995197e8cb1cfc81f5673dec0a781edc1e031a02648d4b8b61" exitCode=0 Sep 18 16:11:23.611492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.611486 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 16:11:23.611580 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.611541 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" event={"ID":"032139ce-fc2a-470b-a432-a74687697b4d","Type":"ContainerDied","Data":"cc30188db93b08995197e8cb1cfc81f5673dec0a781edc1e031a02648d4b8b61"} Sep 18 16:11:23.611929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.611914 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mvfv2\"" Sep 18 16:11:23.612255 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.612228 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 16:11:23.624628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.624600 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 16:11:23.683535 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.683507 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj"] Sep 18 16:11:23.701499 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.701466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3082c931-af4a-4f03-b525-968fb7f112dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.701499 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.701505 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3082c931-af4a-4f03-b525-968fb7f112dc-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.701712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.701531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7842l\" (UniqueName: \"kubernetes.io/projected/3082c931-af4a-4f03-b525-968fb7f112dc-kube-api-access-7842l\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.701712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.701602 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3082c931-af4a-4f03-b525-968fb7f112dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.701851 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.701710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3082c931-af4a-4f03-b525-968fb7f112dc-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.802581 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.802544 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7842l\" (UniqueName: \"kubernetes.io/projected/3082c931-af4a-4f03-b525-968fb7f112dc-kube-api-access-7842l\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.802742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.802588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3082c931-af4a-4f03-b525-968fb7f112dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.802742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.802630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3082c931-af4a-4f03-b525-968fb7f112dc-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.802742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.802655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3082c931-af4a-4f03-b525-968fb7f112dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.802742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.802675 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3082c931-af4a-4f03-b525-968fb7f112dc-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.803129 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.803104 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3082c931-af4a-4f03-b525-968fb7f112dc-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.804836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.804811 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3082c931-af4a-4f03-b525-968fb7f112dc-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.805104 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.805061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3082c931-af4a-4f03-b525-968fb7f112dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.805175 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.805069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3082c931-af4a-4f03-b525-968fb7f112dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.817121 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.817073 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7842l\" (UniqueName: \"kubernetes.io/projected/3082c931-af4a-4f03-b525-968fb7f112dc-kube-api-access-7842l\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj\" (UID: \"3082c931-af4a-4f03-b525-968fb7f112dc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:23.919258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:23.919166 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:24.084604 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.084572 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:24.167363 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.167336 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:24.211408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.211385 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj"] Sep 18 16:11:24.213507 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:11:24.213484 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3082c931_af4a_4f03_b525_968fb7f112dc.slice/crio-e03b635b6630caeff7817fba5a18aa2a12cbafd0a2f9ad90b6fb7cf065f986cb WatchSource:0}: Error finding container e03b635b6630caeff7817fba5a18aa2a12cbafd0a2f9ad90b6fb7cf065f986cb: Status 404 returned error can't find the container with id e03b635b6630caeff7817fba5a18aa2a12cbafd0a2f9ad90b6fb7cf065f986cb Sep 18 16:11:24.246986 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.246952 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:24.347532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.347502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:24.452295 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.452267 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:24.619578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.619543 2563 generic.go:356] "Generic (PLEG): container finished" podID="032139ce-fc2a-470b-a432-a74687697b4d" containerID="9641d772deb7c1cc2055b2e59515fa0798f770d43b3ecd5b16ad0bd93b2c67a2" exitCode=0 Sep 18 16:11:24.620017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.619623 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" event={"ID":"032139ce-fc2a-470b-a432-a74687697b4d","Type":"ContainerDied","Data":"9641d772deb7c1cc2055b2e59515fa0798f770d43b3ecd5b16ad0bd93b2c67a2"} Sep 18 16:11:24.621047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:24.621024 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"e03b635b6630caeff7817fba5a18aa2a12cbafd0a2f9ad90b6fb7cf065f986cb"} Sep 18 16:11:25.789799 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.789771 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:25.920932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.920851 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-util\") pod \"032139ce-fc2a-470b-a432-a74687697b4d\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " Sep 18 16:11:25.920932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.920897 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-bundle\") pod \"032139ce-fc2a-470b-a432-a74687697b4d\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " Sep 18 16:11:25.921185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.920944 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/032139ce-fc2a-470b-a432-a74687697b4d-kube-api-access-x85rv\") pod \"032139ce-fc2a-470b-a432-a74687697b4d\" (UID: \"032139ce-fc2a-470b-a432-a74687697b4d\") " Sep 18 16:11:25.921923 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.921890 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-bundle" (OuterVolumeSpecName: "bundle") pod "032139ce-fc2a-470b-a432-a74687697b4d" (UID: "032139ce-fc2a-470b-a432-a74687697b4d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:11:25.923728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.923694 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032139ce-fc2a-470b-a432-a74687697b4d-kube-api-access-x85rv" (OuterVolumeSpecName: "kube-api-access-x85rv") pod "032139ce-fc2a-470b-a432-a74687697b4d" (UID: "032139ce-fc2a-470b-a432-a74687697b4d"). InnerVolumeSpecName "kube-api-access-x85rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:11:25.928412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:25.928387 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-util" (OuterVolumeSpecName: "util") pod "032139ce-fc2a-470b-a432-a74687697b4d" (UID: "032139ce-fc2a-470b-a432-a74687697b4d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:11:26.021882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:26.021841 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:11:26.021882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:26.021878 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/032139ce-fc2a-470b-a432-a74687697b4d-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:11:26.021882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:26.021889 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/032139ce-fc2a-470b-a432-a74687697b4d-kube-api-access-x85rv\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:11:26.633797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:26.633758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" event={"ID":"032139ce-fc2a-470b-a432-a74687697b4d","Type":"ContainerDied","Data":"d8166fdb8d77bc668222827865fecba4fd82341ae3fa9e6b237b724cd6ac1f0f"} Sep 18 16:11:26.633971 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:26.633804 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b09ts8p" Sep 18 16:11:26.633971 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:26.633815 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8166fdb8d77bc668222827865fecba4fd82341ae3fa9e6b237b724cd6ac1f0f" Sep 18 16:11:27.640272 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:27.640237 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"9689047a2c3444ea4440b6ef85a3cffe8ae0e306191e3c2f56b70a9f267abc34"} Sep 18 16:11:28.647356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:28.647322 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="9689047a2c3444ea4440b6ef85a3cffe8ae0e306191e3c2f56b70a9f267abc34" exitCode=0 Sep 18 16:11:28.647692 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:28.647405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"9689047a2c3444ea4440b6ef85a3cffe8ae0e306191e3c2f56b70a9f267abc34"} Sep 18 16:11:29.654256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:29.654219 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"cab643eea50a8a73216b23002672ba88fe7fb5ffb56fcaa1c856fd1bbd9f1fcb"} Sep 18 16:11:29.654815 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:29.654367 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:29.678099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:29.678027 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podStartSLOduration=2.066445352 podStartE2EDuration="6.678008879s" totalImagesPullingTime="3.563837596s" totalInitContainerRuntime="1.047725931s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:29.67629303 +0000 UTC m=+389.237137699" watchObservedRunningTime="2026-09-18 16:11:29.678008879 +0000 UTC m=+389.238853547" Sep 18 16:11:39.422686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:39.422650 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:39.542308 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:39.542234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:39.907754 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:39.907719 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:39.989633 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:39.989603 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:40.661402 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:40.661375 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:11:41.391597 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391561 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh"] Sep 18 16:11:41.391900 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391888 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="pull" Sep 18 16:11:41.391900 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391901 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="pull" Sep 18 16:11:41.391983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391911 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="util" Sep 18 16:11:41.391983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391916 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="util" Sep 18 16:11:41.391983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391924 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="extract" Sep 18 16:11:41.391983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391929 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="extract" Sep 18 16:11:41.392145 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.391989 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="032139ce-fc2a-470b-a432-a74687697b4d" containerName="extract" Sep 18 16:11:41.396565 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.396545 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.399101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.399064 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 18 16:11:41.400328 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.400311 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 18 16:11:41.400404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.400380 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 18 16:11:41.400404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.400318 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ln5jh\"" Sep 18 16:11:41.407889 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.407866 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh"] Sep 18 16:11:41.443961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.443932 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/132cdab9-9fc4-4348-b41a-bb96fd4116c7-metrics-cert\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.443961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.443961 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/132cdab9-9fc4-4348-b41a-bb96fd4116c7-cert\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.444192 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.444051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/132cdab9-9fc4-4348-b41a-bb96fd4116c7-manager-config\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.444192 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.444128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdjq\" (UniqueName: \"kubernetes.io/projected/132cdab9-9fc4-4348-b41a-bb96fd4116c7-kube-api-access-frdjq\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.545283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.545248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/132cdab9-9fc4-4348-b41a-bb96fd4116c7-manager-config\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.545283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.545283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdjq\" (UniqueName: \"kubernetes.io/projected/132cdab9-9fc4-4348-b41a-bb96fd4116c7-kube-api-access-frdjq\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.545517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.545311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/132cdab9-9fc4-4348-b41a-bb96fd4116c7-metrics-cert\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.545517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.545432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/132cdab9-9fc4-4348-b41a-bb96fd4116c7-cert\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.546004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.545976 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/132cdab9-9fc4-4348-b41a-bb96fd4116c7-manager-config\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.547703 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.547686 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/132cdab9-9fc4-4348-b41a-bb96fd4116c7-metrics-cert\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.547799 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.547781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/132cdab9-9fc4-4348-b41a-bb96fd4116c7-cert\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.553299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.553277 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdjq\" (UniqueName: \"kubernetes.io/projected/132cdab9-9fc4-4348-b41a-bb96fd4116c7-kube-api-access-frdjq\") pod \"lws-controller-manager-6847d847f7-fgmrh\" (UID: \"132cdab9-9fc4-4348-b41a-bb96fd4116c7\") " pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.705498 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.705410 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:41.975252 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:41.975227 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh"] Sep 18 16:11:41.977137 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:11:41.977108 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132cdab9_9fc4_4348_b41a_bb96fd4116c7.slice/crio-97c3b2a551a25085cace22847c9c738fa87f01b0597ca42f289217ba4582657a WatchSource:0}: Error finding container 97c3b2a551a25085cace22847c9c738fa87f01b0597ca42f289217ba4582657a: Status 404 returned error can't find the container with id 97c3b2a551a25085cace22847c9c738fa87f01b0597ca42f289217ba4582657a Sep 18 16:11:42.721434 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:42.721380 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" event={"ID":"132cdab9-9fc4-4348-b41a-bb96fd4116c7","Type":"ContainerStarted","Data":"97c3b2a551a25085cace22847c9c738fa87f01b0597ca42f289217ba4582657a"} Sep 18 16:11:42.762820 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:42.762790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:42.886501 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:42.886474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:43.728286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:43.728172 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" event={"ID":"132cdab9-9fc4-4348-b41a-bb96fd4116c7","Type":"ContainerStarted","Data":"6e1cd407b1fdfec4d1f3be7ac8aa7eee0f44031043cac479fc351d073894e66b"} Sep 18 16:11:43.728286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:43.728240 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:43.757311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:43.757261 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" podStartSLOduration=1.444493308 podStartE2EDuration="2.757245802s" totalImagesPullingTime="1.312752494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:43.756617406 +0000 UTC m=+403.317462108" watchObservedRunningTime="2026-09-18 16:11:43.757245802 +0000 UTC m=+403.318090467" Sep 18 16:11:53.748566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:53.748533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:53.916773 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:53.916731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:54.019464 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:54.019387 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:54.115517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:54.115487 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:54.527784 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:54.527752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:54.718257 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:54.718222 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:54.735842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:54.735815 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:11:55.051202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:55.051160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:55.248480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:55.248426 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:55.340803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:55.340686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:11:56.425220 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:11:56.425157 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:00.930158 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.930114 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw"] Sep 18 16:12:00.940253 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.940229 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:00.943560 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943533 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 18 16:12:00.943560 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943547 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 18 16:12:00.943758 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943580 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 18 16:12:00.943758 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943642 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 18 16:12:00.943858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943848 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 16:12:00.943921 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943906 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tt4rz\"" Sep 18 16:12:00.943981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.943934 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 16:12:00.963979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:00.963951 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw"] Sep 18 16:12:01.005309 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005281 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0962fb23-150f-4073-9710-a999268241c7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.005309 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.005518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005353 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0962fb23-150f-4073-9710-a999268241c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.005518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.005518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.005518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005444 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0962fb23-150f-4073-9710-a999268241c7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.005518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.005463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8v7\" (UniqueName: \"kubernetes.io/projected/0962fb23-150f-4073-9710-a999268241c7-kube-api-access-cd8v7\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.106838 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.106801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0962fb23-150f-4073-9710-a999268241c7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.106850 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.106884 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0962fb23-150f-4073-9710-a999268241c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107154 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.107063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107192 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.107153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.107191 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0962fb23-150f-4073-9710-a999268241c7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107263 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.107220 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8v7\" (UniqueName: \"kubernetes.io/projected/0962fb23-150f-4073-9710-a999268241c7-kube-api-access-cd8v7\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.107552 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.107518 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0962fb23-150f-4073-9710-a999268241c7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.109330 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.109309 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.109504 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.109483 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.109734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.109716 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0962fb23-150f-4073-9710-a999268241c7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.109841 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.109827 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0962fb23-150f-4073-9710-a999268241c7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.124642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.124611 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0962fb23-150f-4073-9710-a999268241c7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.125279 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.125253 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8v7\" (UniqueName: \"kubernetes.io/projected/0962fb23-150f-4073-9710-a999268241c7-kube-api-access-cd8v7\") pod \"istiod-openshift-gateway-5677ccb49b-2qsjw\" (UID: \"0962fb23-150f-4073-9710-a999268241c7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.249591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.249562 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:01.303144 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.303102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:01.462581 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.462491 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:01.552291 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.552205 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw"] Sep 18 16:12:01.555210 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:12:01.555179 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0962fb23_150f_4073_9710_a999268241c7.slice/crio-1b66e1c0e3836b6434ede219d6c08da6e00ba70cddef2a9be8e57a7aac9bdeec WatchSource:0}: Error finding container 1b66e1c0e3836b6434ede219d6c08da6e00ba70cddef2a9be8e57a7aac9bdeec: Status 404 returned error can't find the container with id 1b66e1c0e3836b6434ede219d6c08da6e00ba70cddef2a9be8e57a7aac9bdeec Sep 18 16:12:01.732912 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.732863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:01.815993 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:01.815897 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" event={"ID":"0962fb23-150f-4073-9710-a999268241c7","Type":"ContainerStarted","Data":"1b66e1c0e3836b6434ede219d6c08da6e00ba70cddef2a9be8e57a7aac9bdeec"} Sep 18 16:12:02.043439 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:02.043403 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:02.488806 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:02.488768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:03.706841 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:03.706797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:03.804645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:03.804608 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:04.160369 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:04.160335 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:04.254513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:04.254469 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 16:12:04.254632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:04.254539 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 16:12:04.509455 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:04.509425 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:04.836366 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:04.836221 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" event={"ID":"0962fb23-150f-4073-9710-a999268241c7","Type":"ContainerStarted","Data":"574c72c5d62b7310dd98f475f809f184f260fbad6a6462f26159803a7da5ee27"} Sep 18 16:12:04.836809 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:04.836415 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:05.117016 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:05.116936 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:05.242032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:05.241996 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:05.842791 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:05.842760 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" Sep 18 16:12:05.869470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:05.869426 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-2qsjw" podStartSLOduration=3.172510522 podStartE2EDuration="5.869412152s" totalImagesPullingTime="2.69690163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:04.86640832 +0000 UTC m=+424.427252989" watchObservedRunningTime="2026-09-18 16:12:05.869412152 +0000 UTC m=+425.430256819" Sep 18 16:12:27.057178 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:27.057102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:27.118916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:27.118866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:27.190863 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:27.190829 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:27.292100 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:27.292068 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:27.791940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:27.791910 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:37.413902 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:37.413842 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84565469dc-m9kgf"] Sep 18 16:12:37.534383 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:37.534351 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:37.636411 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:37.636378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:37.737108 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:37.737009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:51.046791 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:51.046757 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:51.131315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:51.131284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:52.960785 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:52.960753 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:53.085901 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:53.085872 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:53.172185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:53.172144 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:54.933045 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:54.933014 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:55.034292 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:55.034255 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:12:59.509638 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.509605 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57"] Sep 18 16:12:59.513054 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.513033 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.515962 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.515935 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 18 16:12:59.516116 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.515966 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 18 16:12:59.516920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.516903 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4rdv\"" Sep 18 16:12:59.524440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.524414 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57"] Sep 18 16:12:59.605040 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.605008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.605255 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.605118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.605255 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.605146 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967kk\" (UniqueName: \"kubernetes.io/projected/4965b023-3df4-4081-b2f3-d7bc7d59f466-kube-api-access-967kk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.606567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.606547 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9"] Sep 18 16:12:59.610080 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.610061 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.622574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.622552 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9"] Sep 18 16:12:59.706350 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xqv\" (UniqueName: \"kubernetes.io/projected/ccf35a9c-cad8-4520-9b15-375f75787d16-kube-api-access-29xqv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.706558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.706558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.706558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967kk\" (UniqueName: \"kubernetes.io/projected/4965b023-3df4-4081-b2f3-d7bc7d59f466-kube-api-access-967kk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.706558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.706558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706480 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.706856 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706830 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.706916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.706881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.713517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.713490 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6"] Sep 18 16:12:59.717219 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.717202 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.731546 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.731485 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-967kk\" (UniqueName: \"kubernetes.io/projected/4965b023-3df4-4081-b2f3-d7bc7d59f466-kube-api-access-967kk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.734930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.734908 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6"] Sep 18 16:12:59.803189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.803101 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt"] Sep 18 16:12:59.807167 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807138 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:12:59.807513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807490 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbx8\" (UniqueName: \"kubernetes.io/projected/b4144a3d-dbff-4899-ad28-8002885139a9-kube-api-access-wtbx8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.807603 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807548 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.807656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807604 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.807656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xqv\" (UniqueName: \"kubernetes.io/projected/ccf35a9c-cad8-4520-9b15-375f75787d16-kube-api-access-29xqv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.807763 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.807763 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.807744 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.808134 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.808113 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.808207 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.808192 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.820818 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.820787 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt"] Sep 18 16:12:59.821896 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.821876 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:12:59.822285 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.822267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xqv\" (UniqueName: \"kubernetes.io/projected/ccf35a9c-cad8-4520-9b15-375f75787d16-kube-api-access-29xqv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:12:59.908383 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbx8\" (UniqueName: \"kubernetes.io/projected/b4144a3d-dbff-4899-ad28-8002885139a9-kube-api-access-wtbx8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.908567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908391 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.908567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.908567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx785\" (UniqueName: \"kubernetes.io/projected/b550f1c3-06e6-44b5-acb8-6e723202fbe5-kube-api-access-lx785\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:12:59.908567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:12:59.908567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:12:59.908836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908811 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.908881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.908841 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.917530 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.917501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbx8\" (UniqueName: \"kubernetes.io/projected/b4144a3d-dbff-4899-ad28-8002885139a9-kube-api-access-wtbx8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:12:59.918205 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:12:59.918185 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:13:00.009734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.009697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.009904 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.009743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.009904 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.009797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx785\" (UniqueName: \"kubernetes.io/projected/b550f1c3-06e6-44b5-acb8-6e723202fbe5-kube-api-access-lx785\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.010205 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.010181 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.010260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.010190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.021373 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.021347 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx785\" (UniqueName: \"kubernetes.io/projected/b550f1c3-06e6-44b5-acb8-6e723202fbe5-kube-api-access-lx785\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.026159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.026141 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:13:00.104646 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.104605 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57"] Sep 18 16:13:00.106360 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:13:00.106333 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4965b023_3df4_4081_b2f3_d7bc7d59f466.slice/crio-94e9bcfc2b4462c04eab0fb79808093786e66f2b98bb098472942024b2911866 WatchSource:0}: Error finding container 94e9bcfc2b4462c04eab0fb79808093786e66f2b98bb098472942024b2911866: Status 404 returned error can't find the container with id 94e9bcfc2b4462c04eab0fb79808093786e66f2b98bb098472942024b2911866 Sep 18 16:13:00.122207 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.122185 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:00.202120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.202074 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9"] Sep 18 16:13:00.310351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.310325 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6"] Sep 18 16:13:00.311729 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:13:00.311699 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4144a3d_dbff_4899_ad28_8002885139a9.slice/crio-68ffc6a5c5a3aca870e101da0f42bde3b12fd6e2358506b876cec869610b10a1 WatchSource:0}: Error finding container 68ffc6a5c5a3aca870e101da0f42bde3b12fd6e2358506b876cec869610b10a1: Status 404 returned error can't find the container with id 68ffc6a5c5a3aca870e101da0f42bde3b12fd6e2358506b876cec869610b10a1 Sep 18 16:13:00.410061 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:00.410039 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt"] Sep 18 16:13:00.437987 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:13:00.437952 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb550f1c3_06e6_44b5_acb8_6e723202fbe5.slice/crio-d1db1885fa3ec38955942292c62dd05d99d7e34d71dbbfeec6d514112f6f5fe9 WatchSource:0}: Error finding container d1db1885fa3ec38955942292c62dd05d99d7e34d71dbbfeec6d514112f6f5fe9: Status 404 returned error can't find the container with id d1db1885fa3ec38955942292c62dd05d99d7e34d71dbbfeec6d514112f6f5fe9 Sep 18 16:13:01.112261 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.112228 2563 generic.go:356] "Generic (PLEG): container finished" podID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerID="f03e0e709b1d17c275aa8d5f0c2acea7e7511bc1af6d58c82fad225f24f19c4f" exitCode=0 Sep 18 16:13:01.112701 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.112327 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" event={"ID":"4965b023-3df4-4081-b2f3-d7bc7d59f466","Type":"ContainerDied","Data":"f03e0e709b1d17c275aa8d5f0c2acea7e7511bc1af6d58c82fad225f24f19c4f"} Sep 18 16:13:01.112701 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.112390 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" event={"ID":"4965b023-3df4-4081-b2f3-d7bc7d59f466","Type":"ContainerStarted","Data":"94e9bcfc2b4462c04eab0fb79808093786e66f2b98bb098472942024b2911866"} Sep 18 16:13:01.113926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.113890 2563 generic.go:356] "Generic (PLEG): container finished" podID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerID="d26267c608c4a7bfa8c48d649d56f20b39d9e3868cd01f49bb7c37126d238f40" exitCode=0 Sep 18 16:13:01.114042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.113919 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" event={"ID":"b550f1c3-06e6-44b5-acb8-6e723202fbe5","Type":"ContainerDied","Data":"d26267c608c4a7bfa8c48d649d56f20b39d9e3868cd01f49bb7c37126d238f40"} Sep 18 16:13:01.114042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.113970 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" event={"ID":"b550f1c3-06e6-44b5-acb8-6e723202fbe5","Type":"ContainerStarted","Data":"d1db1885fa3ec38955942292c62dd05d99d7e34d71dbbfeec6d514112f6f5fe9"} Sep 18 16:13:01.115572 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.115549 2563 generic.go:356] "Generic (PLEG): container finished" podID="b4144a3d-dbff-4899-ad28-8002885139a9" containerID="b5adb6eaf7926eaf89c3a64d6013969afade5b654e2768261ff2e79469c8d0b4" exitCode=0 Sep 18 16:13:01.115667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.115613 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" event={"ID":"b4144a3d-dbff-4899-ad28-8002885139a9","Type":"ContainerDied","Data":"b5adb6eaf7926eaf89c3a64d6013969afade5b654e2768261ff2e79469c8d0b4"} Sep 18 16:13:01.115667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.115650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" event={"ID":"b4144a3d-dbff-4899-ad28-8002885139a9","Type":"ContainerStarted","Data":"68ffc6a5c5a3aca870e101da0f42bde3b12fd6e2358506b876cec869610b10a1"} Sep 18 16:13:01.117293 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.117269 2563 generic.go:356] "Generic (PLEG): container finished" podID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerID="815d6d8a36f751eaf9dc41cc6a8ae7ee338421296ec0edcc7dca16f743e70666" exitCode=0 Sep 18 16:13:01.117396 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.117340 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" event={"ID":"ccf35a9c-cad8-4520-9b15-375f75787d16","Type":"ContainerDied","Data":"815d6d8a36f751eaf9dc41cc6a8ae7ee338421296ec0edcc7dca16f743e70666"} Sep 18 16:13:01.117396 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:01.117371 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" event={"ID":"ccf35a9c-cad8-4520-9b15-375f75787d16","Type":"ContainerStarted","Data":"bf48d5b5c0dafea16c4c3b9cc27431c19553c490730d986c1ba209fdd939dd14"} Sep 18 16:13:02.124911 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.124868 2563 generic.go:356] "Generic (PLEG): container finished" podID="b4144a3d-dbff-4899-ad28-8002885139a9" containerID="286ca2d8f9d18737b9e027d69b18a07855c3f9bfcc2912a3bb8a2ec3cd7e47be" exitCode=0 Sep 18 16:13:02.125338 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.124927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" event={"ID":"b4144a3d-dbff-4899-ad28-8002885139a9","Type":"ContainerDied","Data":"286ca2d8f9d18737b9e027d69b18a07855c3f9bfcc2912a3bb8a2ec3cd7e47be"} Sep 18 16:13:02.127136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.127115 2563 generic.go:356] "Generic (PLEG): container finished" podID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerID="30efb462debb0c4d6be404847dcfb1cfcf4fc366ab4050c3ae1cd017bf71e3c3" exitCode=0 Sep 18 16:13:02.127265 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.127159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" event={"ID":"4965b023-3df4-4081-b2f3-d7bc7d59f466","Type":"ContainerDied","Data":"30efb462debb0c4d6be404847dcfb1cfcf4fc366ab4050c3ae1cd017bf71e3c3"} Sep 18 16:13:02.434390 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.434362 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84565469dc-m9kgf" podUID="ddc8ce26-a8f9-4350-97e7-778985818d5f" containerName="console" containerID="cri-o://ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99" gracePeriod=15 Sep 18 16:13:02.674518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.674493 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:13:02.732701 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732667 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-config\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.732878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732773 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-service-ca\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.732878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732795 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-oauth-config\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.732878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732822 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-serving-cert\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.732878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732845 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjswp\" (UniqueName: \"kubernetes.io/projected/ddc8ce26-a8f9-4350-97e7-778985818d5f-kube-api-access-jjswp\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.733115 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732883 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-oauth-serving-cert\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.733115 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.732899 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-trusted-ca-bundle\") pod \"ddc8ce26-a8f9-4350-97e7-778985818d5f\" (UID: \"ddc8ce26-a8f9-4350-97e7-778985818d5f\") " Sep 18 16:13:02.733115 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.733059 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-config" (OuterVolumeSpecName: "console-config") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:13:02.733333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.733309 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-service-ca" (OuterVolumeSpecName: "service-ca") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:13:02.733469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.733435 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:13:02.733469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.733447 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:13:02.735106 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.735072 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc8ce26-a8f9-4350-97e7-778985818d5f-kube-api-access-jjswp" (OuterVolumeSpecName: "kube-api-access-jjswp") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "kube-api-access-jjswp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:13:02.735106 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.735078 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:13:02.735222 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.735114 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ddc8ce26-a8f9-4350-97e7-778985818d5f" (UID: "ddc8ce26-a8f9-4350-97e7-778985818d5f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:13:02.834393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834301 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:02.834393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834336 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-oauth-config\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:02.834393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834371 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-serving-cert\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:02.834393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834380 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjswp\" (UniqueName: \"kubernetes.io/projected/ddc8ce26-a8f9-4350-97e7-778985818d5f-kube-api-access-jjswp\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:02.834393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834390 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-oauth-serving-cert\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:02.834393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834398 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-trusted-ca-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:02.834688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:02.834406 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc8ce26-a8f9-4350-97e7-778985818d5f-console-config\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:03.134702 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.134662 2563 generic.go:356] "Generic (PLEG): container finished" podID="b4144a3d-dbff-4899-ad28-8002885139a9" containerID="12969e46b9528f041cf137fa229bdd17513e55675d19aea10467dcf3707c5d49" exitCode=0 Sep 18 16:13:03.135149 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.134748 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" event={"ID":"b4144a3d-dbff-4899-ad28-8002885139a9","Type":"ContainerDied","Data":"12969e46b9528f041cf137fa229bdd17513e55675d19aea10467dcf3707c5d49"} Sep 18 16:13:03.136644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.136619 2563 generic.go:356] "Generic (PLEG): container finished" podID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerID="171d2245def8679c8165d14cd8b240ea4dac34d9c6532557a1f5b7482cd57aa8" exitCode=0 Sep 18 16:13:03.136751 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.136709 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" event={"ID":"ccf35a9c-cad8-4520-9b15-375f75787d16","Type":"ContainerDied","Data":"171d2245def8679c8165d14cd8b240ea4dac34d9c6532557a1f5b7482cd57aa8"} Sep 18 16:13:03.137985 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.137966 2563 generic.go:356] "Generic (PLEG): container finished" podID="ddc8ce26-a8f9-4350-97e7-778985818d5f" containerID="ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99" exitCode=0 Sep 18 16:13:03.138055 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.138013 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84565469dc-m9kgf" Sep 18 16:13:03.138125 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.138049 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84565469dc-m9kgf" event={"ID":"ddc8ce26-a8f9-4350-97e7-778985818d5f","Type":"ContainerDied","Data":"ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99"} Sep 18 16:13:03.138125 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.138118 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84565469dc-m9kgf" event={"ID":"ddc8ce26-a8f9-4350-97e7-778985818d5f","Type":"ContainerDied","Data":"46b816333417d01baa6036028388ee2eed9b4ba3ddcb3e3fc93ec5ae95a24f33"} Sep 18 16:13:03.138238 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.138139 2563 scope.go:117] "RemoveContainer" containerID="ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99" Sep 18 16:13:03.140677 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.140652 2563 generic.go:356] "Generic (PLEG): container finished" podID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerID="d0e74dc490b16c6deb88f95d27ae3750e4901e154b2d95f981b4815eb060fe42" exitCode=0 Sep 18 16:13:03.140862 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.140705 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" event={"ID":"4965b023-3df4-4081-b2f3-d7bc7d59f466","Type":"ContainerDied","Data":"d0e74dc490b16c6deb88f95d27ae3750e4901e154b2d95f981b4815eb060fe42"} Sep 18 16:13:03.142740 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.142718 2563 generic.go:356] "Generic (PLEG): container finished" podID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerID="baf875704daa82b3e4d38018ebd2be4cc46d1c4d49799f082236349bca080fd0" exitCode=0 Sep 18 16:13:03.142820 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.142780 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" event={"ID":"b550f1c3-06e6-44b5-acb8-6e723202fbe5","Type":"ContainerDied","Data":"baf875704daa82b3e4d38018ebd2be4cc46d1c4d49799f082236349bca080fd0"} Sep 18 16:13:03.147321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.147299 2563 scope.go:117] "RemoveContainer" containerID="ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99" Sep 18 16:13:03.147617 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:13:03.147593 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99\": container with ID starting with ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99 not found: ID does not exist" containerID="ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99" Sep 18 16:13:03.147681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.147631 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99"} err="failed to get container status \"ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99\": rpc error: code = NotFound desc = could not find container \"ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99\": container with ID starting with ffab773e8b5dffd151e1cacd380bc5b5cb65aff92d55fa192b2d9f950aee5d99 not found: ID does not exist" Sep 18 16:13:03.194324 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.194250 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84565469dc-m9kgf"] Sep 18 16:13:03.199418 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:03.199389 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84565469dc-m9kgf"] Sep 18 16:13:04.150325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.150294 2563 generic.go:356] "Generic (PLEG): container finished" podID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerID="dbd0f9e0b816e89325a01aaecc8bec44626d68fc2e9080c31054eb6458b95b97" exitCode=0 Sep 18 16:13:04.150777 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.150371 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" event={"ID":"ccf35a9c-cad8-4520-9b15-375f75787d16","Type":"ContainerDied","Data":"dbd0f9e0b816e89325a01aaecc8bec44626d68fc2e9080c31054eb6458b95b97"} Sep 18 16:13:04.153372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.153348 2563 generic.go:356] "Generic (PLEG): container finished" podID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerID="8776e86ce515c9f9c83f5bbbe7f8d301f39ec7071653681538e69f0f4bca3f7e" exitCode=0 Sep 18 16:13:04.153518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.153428 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" event={"ID":"b550f1c3-06e6-44b5-acb8-6e723202fbe5","Type":"ContainerDied","Data":"8776e86ce515c9f9c83f5bbbe7f8d301f39ec7071653681538e69f0f4bca3f7e"} Sep 18 16:13:04.285626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.285595 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:13:04.323055 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.323024 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:13:04.347007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.346930 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtbx8\" (UniqueName: \"kubernetes.io/projected/b4144a3d-dbff-4899-ad28-8002885139a9-kube-api-access-wtbx8\") pod \"b4144a3d-dbff-4899-ad28-8002885139a9\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " Sep 18 16:13:04.347220 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.347200 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-bundle\") pod \"b4144a3d-dbff-4899-ad28-8002885139a9\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " Sep 18 16:13:04.347302 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.347286 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-util\") pod \"b4144a3d-dbff-4899-ad28-8002885139a9\" (UID: \"b4144a3d-dbff-4899-ad28-8002885139a9\") " Sep 18 16:13:04.349367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.349297 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-bundle" (OuterVolumeSpecName: "bundle") pod "b4144a3d-dbff-4899-ad28-8002885139a9" (UID: "b4144a3d-dbff-4899-ad28-8002885139a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:04.351011 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.350981 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4144a3d-dbff-4899-ad28-8002885139a9-kube-api-access-wtbx8" (OuterVolumeSpecName: "kube-api-access-wtbx8") pod "b4144a3d-dbff-4899-ad28-8002885139a9" (UID: "b4144a3d-dbff-4899-ad28-8002885139a9"). InnerVolumeSpecName "kube-api-access-wtbx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:13:04.354561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.354504 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-util" (OuterVolumeSpecName: "util") pod "b4144a3d-dbff-4899-ad28-8002885139a9" (UID: "b4144a3d-dbff-4899-ad28-8002885139a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:04.448750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.448652 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-bundle\") pod \"4965b023-3df4-4081-b2f3-d7bc7d59f466\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " Sep 18 16:13:04.448750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.448705 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-util\") pod \"4965b023-3df4-4081-b2f3-d7bc7d59f466\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " Sep 18 16:13:04.448978 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.448774 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-967kk\" (UniqueName: \"kubernetes.io/projected/4965b023-3df4-4081-b2f3-d7bc7d59f466-kube-api-access-967kk\") pod \"4965b023-3df4-4081-b2f3-d7bc7d59f466\" (UID: \"4965b023-3df4-4081-b2f3-d7bc7d59f466\") " Sep 18 16:13:04.449033 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.448988 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbx8\" (UniqueName: \"kubernetes.io/projected/b4144a3d-dbff-4899-ad28-8002885139a9-kube-api-access-wtbx8\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:04.449033 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.449006 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:04.449033 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.449020 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4144a3d-dbff-4899-ad28-8002885139a9-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:04.449195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.449167 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-bundle" (OuterVolumeSpecName: "bundle") pod "4965b023-3df4-4081-b2f3-d7bc7d59f466" (UID: "4965b023-3df4-4081-b2f3-d7bc7d59f466"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:04.450848 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.450824 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4965b023-3df4-4081-b2f3-d7bc7d59f466-kube-api-access-967kk" (OuterVolumeSpecName: "kube-api-access-967kk") pod "4965b023-3df4-4081-b2f3-d7bc7d59f466" (UID: "4965b023-3df4-4081-b2f3-d7bc7d59f466"). InnerVolumeSpecName "kube-api-access-967kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:13:04.454071 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.454047 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-util" (OuterVolumeSpecName: "util") pod "4965b023-3df4-4081-b2f3-d7bc7d59f466" (UID: "4965b023-3df4-4081-b2f3-d7bc7d59f466"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:04.549615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.549579 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-967kk\" (UniqueName: \"kubernetes.io/projected/4965b023-3df4-4081-b2f3-d7bc7d59f466-kube-api-access-967kk\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:04.549615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.549610 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:04.549615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:04.549621 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4965b023-3df4-4081-b2f3-d7bc7d59f466-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:05.081313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.081280 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc8ce26-a8f9-4350-97e7-778985818d5f" path="/var/lib/kubelet/pods/ddc8ce26-a8f9-4350-97e7-778985818d5f/volumes" Sep 18 16:13:05.161046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.161015 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" Sep 18 16:13:05.161644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.161012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c5f7l6" event={"ID":"b4144a3d-dbff-4899-ad28-8002885139a9","Type":"ContainerDied","Data":"68ffc6a5c5a3aca870e101da0f42bde3b12fd6e2358506b876cec869610b10a1"} Sep 18 16:13:05.161644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.161132 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ffc6a5c5a3aca870e101da0f42bde3b12fd6e2358506b876cec869610b10a1" Sep 18 16:13:05.163281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.163228 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" Sep 18 16:13:05.163281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.163262 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xnr57" event={"ID":"4965b023-3df4-4081-b2f3-d7bc7d59f466","Type":"ContainerDied","Data":"94e9bcfc2b4462c04eab0fb79808093786e66f2b98bb098472942024b2911866"} Sep 18 16:13:05.163438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.163302 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e9bcfc2b4462c04eab0fb79808093786e66f2b98bb098472942024b2911866" Sep 18 16:13:05.311840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.311819 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:05.313799 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.313783 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:13:05.459210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459122 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-util\") pod \"ccf35a9c-cad8-4520-9b15-375f75787d16\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " Sep 18 16:13:05.459210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459159 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-bundle\") pod \"ccf35a9c-cad8-4520-9b15-375f75787d16\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " Sep 18 16:13:05.459210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459184 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-bundle\") pod \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " Sep 18 16:13:05.459503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459239 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29xqv\" (UniqueName: \"kubernetes.io/projected/ccf35a9c-cad8-4520-9b15-375f75787d16-kube-api-access-29xqv\") pod \"ccf35a9c-cad8-4520-9b15-375f75787d16\" (UID: \"ccf35a9c-cad8-4520-9b15-375f75787d16\") " Sep 18 16:13:05.459503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459285 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx785\" (UniqueName: \"kubernetes.io/projected/b550f1c3-06e6-44b5-acb8-6e723202fbe5-kube-api-access-lx785\") pod \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " Sep 18 16:13:05.459503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459303 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-util\") pod \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\" (UID: \"b550f1c3-06e6-44b5-acb8-6e723202fbe5\") " Sep 18 16:13:05.459833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.459787 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-bundle" (OuterVolumeSpecName: "bundle") pod "b550f1c3-06e6-44b5-acb8-6e723202fbe5" (UID: "b550f1c3-06e6-44b5-acb8-6e723202fbe5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:05.460038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.460013 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-bundle" (OuterVolumeSpecName: "bundle") pod "ccf35a9c-cad8-4520-9b15-375f75787d16" (UID: "ccf35a9c-cad8-4520-9b15-375f75787d16"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:05.461518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.461496 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b550f1c3-06e6-44b5-acb8-6e723202fbe5-kube-api-access-lx785" (OuterVolumeSpecName: "kube-api-access-lx785") pod "b550f1c3-06e6-44b5-acb8-6e723202fbe5" (UID: "b550f1c3-06e6-44b5-acb8-6e723202fbe5"). InnerVolumeSpecName "kube-api-access-lx785". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:13:05.461606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.461502 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf35a9c-cad8-4520-9b15-375f75787d16-kube-api-access-29xqv" (OuterVolumeSpecName: "kube-api-access-29xqv") pod "ccf35a9c-cad8-4520-9b15-375f75787d16" (UID: "ccf35a9c-cad8-4520-9b15-375f75787d16"). InnerVolumeSpecName "kube-api-access-29xqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:13:05.466233 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.466203 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-util" (OuterVolumeSpecName: "util") pod "b550f1c3-06e6-44b5-acb8-6e723202fbe5" (UID: "b550f1c3-06e6-44b5-acb8-6e723202fbe5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:05.514162 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.514126 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-util" (OuterVolumeSpecName: "util") pod "ccf35a9c-cad8-4520-9b15-375f75787d16" (UID: "ccf35a9c-cad8-4520-9b15-375f75787d16"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:13:05.560240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.560201 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx785\" (UniqueName: \"kubernetes.io/projected/b550f1c3-06e6-44b5-acb8-6e723202fbe5-kube-api-access-lx785\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:05.560240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.560235 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:05.560391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.560248 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-util\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:05.560391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.560259 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ccf35a9c-cad8-4520-9b15-375f75787d16-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:05.560391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.560272 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b550f1c3-06e6-44b5-acb8-6e723202fbe5-bundle\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:05.560391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:05.560283 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xqv\" (UniqueName: \"kubernetes.io/projected/ccf35a9c-cad8-4520-9b15-375f75787d16-kube-api-access-29xqv\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:13:06.170945 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:06.170910 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" Sep 18 16:13:06.170945 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:06.170933 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an6jl9" event={"ID":"ccf35a9c-cad8-4520-9b15-375f75787d16","Type":"ContainerDied","Data":"bf48d5b5c0dafea16c4c3b9cc27431c19553c490730d986c1ba209fdd939dd14"} Sep 18 16:13:06.171462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:06.170988 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf48d5b5c0dafea16c4c3b9cc27431c19553c490730d986c1ba209fdd939dd14" Sep 18 16:13:06.173051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:06.173020 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" event={"ID":"b550f1c3-06e6-44b5-acb8-6e723202fbe5","Type":"ContainerDied","Data":"d1db1885fa3ec38955942292c62dd05d99d7e34d71dbbfeec6d514112f6f5fe9"} Sep 18 16:13:06.173051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:06.173051 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c648bdt" Sep 18 16:13:06.173221 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:06.173070 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1db1885fa3ec38955942292c62dd05d99d7e34d71dbbfeec6d514112f6f5fe9" Sep 18 16:13:10.530282 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:10.530250 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:11.362833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:11.362799 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:14.379626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:14.379595 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:14.491650 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:14.491614 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:14.606172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:14.606135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:16.612667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:16.612640 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:16.742099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:16.742057 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:18.724320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:18.724285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:18.993372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:18.993287 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:19.138369 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:19.138337 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:19.740930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:19.740896 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:20.756391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:20.756359 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:20.942408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:20.942376 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:26.036253 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:26.036223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:26.175230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:26.175200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:26.564696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:26.564668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:26.663334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:26.663294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:30.345975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:30.345940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:30.447336 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:30.447303 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:31.414831 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:31.414790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:31.543510 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:31.543482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:32.348763 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:32.348731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:32.457693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:32.457668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:38.152974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:38.152939 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:38.273276 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:38.273244 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:38.364070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:38.364020 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:46.222691 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:46.222613 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:46.329576 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:46.329547 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:46.811228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:46.811191 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:46.912382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:46.912355 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:50.206364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:50.206332 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:50.306002 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:50.305975 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:52.412959 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:52.412928 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:52.494847 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:52.494818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:58.355290 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:58.355257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:58.455298 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:58.455268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:58.592227 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:58.592197 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:58.753940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:58.753911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:13:58.922689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:13:58.922654 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:10.653258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:10.653224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:10.749337 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:10.749304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:12.996702 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:12.996664 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:13.200578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:13.200542 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:15.200362 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.200330 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:15.201074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201050 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:15.201404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201388 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="extract" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201407 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="extract" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201426 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="pull" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201435 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="pull" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201450 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="pull" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201459 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="pull" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201487 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="pull" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201497 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="pull" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201514 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="extract" Sep 18 16:14:15.201532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201522 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201536 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201547 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201555 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201565 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201576 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="pull" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201584 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="pull" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201595 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201603 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201613 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc8ce26-a8f9-4350-97e7-778985818d5f" containerName="console" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201621 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc8ce26-a8f9-4350-97e7-778985818d5f" containerName="console" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201631 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201639 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201648 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201656 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201668 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201676 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="util" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201761 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b550f1c3-06e6-44b5-acb8-6e723202fbe5" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201772 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc8ce26-a8f9-4350-97e7-778985818d5f" containerName="console" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201782 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4144a3d-dbff-4899-ad28-8002885139a9" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201790 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccf35a9c-cad8-4520-9b15-375f75787d16" containerName="extract" Sep 18 16:14:15.201970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.201800 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4965b023-3df4-4081-b2f3-d7bc7d59f466" containerName="extract" Sep 18 16:14:15.204872 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.204853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:15.222471 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:15.222446 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 18 16:14:15.223789 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:15.223756 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 18 16:14:15.225493 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:15.225460 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-limitador-8588749977-shqqd\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" podUID="32028d1e-c01d-4c1e-9ee6-59096a093194" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:15.231199 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:15.231172 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-dc6ds\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc6ds\"" type="*v1.Secret" Sep 18 16:14:15.231431 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:15.231411 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"limitador-limits-config-limitador\" is forbidden: User \"system:node:ip-10-0-132-41.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-41.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" Sep 18 16:14:15.248880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.248857 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:15.307312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.307278 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:15.328545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.328518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:15.328645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.328580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvp8\" (UniqueName: \"kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:15.344127 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.344103 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:15.429134 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.429078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:15.429292 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.429178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvp8\" (UniqueName: \"kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:15.545290 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:15.545259 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:16.068652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.068622 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 18 16:14:16.289919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.289890 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:16.401983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.401909 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:16.429650 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:16.429620 2563 configmap.go:193] Couldn't get configMap kuadrant-system/limitador-limits-config-limitador: failed to sync configmap cache: timed out waiting for the condition Sep 18 16:14:16.429805 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:16.429702 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file podName:32028d1e-c01d-4c1e-9ee6-59096a093194 nodeName:}" failed. No retries permitted until 2026-09-18 16:14:16.929682501 +0000 UTC m=+556.490527162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-file" (UniqueName: "kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file") pod "limitador-limitador-8588749977-shqqd" (UID: "32028d1e-c01d-4c1e-9ee6-59096a093194") : failed to sync configmap cache: timed out waiting for the condition Sep 18 16:14:16.442127 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:16.442104 2563 projected.go:291] Couldn't get configMap kuadrant-system/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 18 16:14:16.442245 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:16.442138 2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvvp8 for pod kuadrant-system/limitador-limitador-8588749977-shqqd: failed to sync configmap cache: timed out waiting for the condition Sep 18 16:14:16.442245 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:16.442199 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8 podName:32028d1e-c01d-4c1e-9ee6-59096a093194 nodeName:}" failed. No retries permitted until 2026-09-18 16:14:16.942181892 +0000 UTC m=+556.503026537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gvvp8" (UniqueName: "kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8") pod "limitador-limitador-8588749977-shqqd" (UID: "32028d1e-c01d-4c1e-9ee6-59096a093194") : failed to sync configmap cache: timed out waiting for the condition Sep 18 16:14:16.488876 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.488851 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 18 16:14:16.784183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.784155 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc6ds\"" Sep 18 16:14:16.823578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.823549 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 18 16:14:16.943667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.943637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvp8\" (UniqueName: \"kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:16.943835 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.943691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:16.944428 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.944396 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:16.946325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:16.946302 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvp8\" (UniqueName: \"kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8\") pod \"limitador-limitador-8588749977-shqqd\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:17.014812 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:17.014776 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:17.299854 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:17.299829 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:17.301877 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:14:17.301852 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32028d1e_c01d_4c1e_9ee6_59096a093194.slice/crio-49f81fd51c0f02b4acd7867dc7dd4caa3716e70f6b7f6897db521ba9e8d82984 WatchSource:0}: Error finding container 49f81fd51c0f02b4acd7867dc7dd4caa3716e70f6b7f6897db521ba9e8d82984: Status 404 returned error can't find the container with id 49f81fd51c0f02b4acd7867dc7dd4caa3716e70f6b7f6897db521ba9e8d82984 Sep 18 16:14:17.539465 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:17.539431 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" event={"ID":"32028d1e-c01d-4c1e-9ee6-59096a093194","Type":"ContainerStarted","Data":"49f81fd51c0f02b4acd7867dc7dd4caa3716e70f6b7f6897db521ba9e8d82984"} Sep 18 16:14:19.551961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:19.551923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" event={"ID":"32028d1e-c01d-4c1e-9ee6-59096a093194","Type":"ContainerStarted","Data":"729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211"} Sep 18 16:14:19.552347 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:19.552058 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:19.577795 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:19.577750 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" podStartSLOduration=2.469046778 podStartE2EDuration="4.577735892s" totalImagesPullingTime="2.108689114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:19.575523337 +0000 UTC m=+559.136368015" watchObservedRunningTime="2026-09-18 16:14:19.577735892 +0000 UTC m=+559.138580559" Sep 18 16:14:29.898191 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:29.898157 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:29.898753 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:29.898348 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" podUID="32028d1e-c01d-4c1e-9ee6-59096a093194" containerName="limitador" containerID="cri-o://729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211" gracePeriod=30 Sep 18 16:14:29.899059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:29.899021 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:30.444631 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.444607 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:30.453395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.453356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvvp8\" (UniqueName: \"kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8\") pod \"32028d1e-c01d-4c1e-9ee6-59096a093194\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " Sep 18 16:14:30.453553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.453432 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file\") pod \"32028d1e-c01d-4c1e-9ee6-59096a093194\" (UID: \"32028d1e-c01d-4c1e-9ee6-59096a093194\") " Sep 18 16:14:30.453764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.453741 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file" (OuterVolumeSpecName: "config-file") pod "32028d1e-c01d-4c1e-9ee6-59096a093194" (UID: "32028d1e-c01d-4c1e-9ee6-59096a093194"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:14:30.455586 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.455564 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8" (OuterVolumeSpecName: "kube-api-access-gvvp8") pod "32028d1e-c01d-4c1e-9ee6-59096a093194" (UID: "32028d1e-c01d-4c1e-9ee6-59096a093194"). InnerVolumeSpecName "kube-api-access-gvvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:14:30.554074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.554036 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvvp8\" (UniqueName: \"kubernetes.io/projected/32028d1e-c01d-4c1e-9ee6-59096a093194-kube-api-access-gvvp8\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:14:30.554074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.554067 2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/32028d1e-c01d-4c1e-9ee6-59096a093194-config-file\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:14:30.614190 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.614155 2563 generic.go:356] "Generic (PLEG): container finished" podID="32028d1e-c01d-4c1e-9ee6-59096a093194" containerID="729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211" exitCode=0 Sep 18 16:14:30.614331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.614203 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" Sep 18 16:14:30.614331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.614230 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" event={"ID":"32028d1e-c01d-4c1e-9ee6-59096a093194","Type":"ContainerDied","Data":"729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211"} Sep 18 16:14:30.614331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.614285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-shqqd" event={"ID":"32028d1e-c01d-4c1e-9ee6-59096a093194","Type":"ContainerDied","Data":"49f81fd51c0f02b4acd7867dc7dd4caa3716e70f6b7f6897db521ba9e8d82984"} Sep 18 16:14:30.614331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.614309 2563 scope.go:117] "RemoveContainer" containerID="729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211" Sep 18 16:14:30.622990 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.622969 2563 scope.go:117] "RemoveContainer" containerID="729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211" Sep 18 16:14:30.623315 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:30.623296 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211\": container with ID starting with 729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211 not found: ID does not exist" containerID="729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211" Sep 18 16:14:30.623378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.623328 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211"} err="failed to get container status \"729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211\": rpc error: code = NotFound desc = could not find container \"729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211\": container with ID starting with 729a50299542aea0e77229da3fb93b4bf7b3197d1fa5f3a6c58378765e2e0211 not found: ID does not exist" Sep 18 16:14:30.645513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.645437 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:30.647358 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:30.647327 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-shqqd"] Sep 18 16:14:31.080137 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:31.080076 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32028d1e-c01d-4c1e-9ee6-59096a093194" path="/var/lib/kubelet/pods/32028d1e-c01d-4c1e-9ee6-59096a093194/volumes" Sep 18 16:14:31.601708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:31.601677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:31.705395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:31.705363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:32.081696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.081657 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8"] Sep 18 16:14:32.082067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.081993 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32028d1e-c01d-4c1e-9ee6-59096a093194" containerName="limitador" Sep 18 16:14:32.082067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.082005 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32028d1e-c01d-4c1e-9ee6-59096a093194" containerName="limitador" Sep 18 16:14:32.082164 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.082099 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32028d1e-c01d-4c1e-9ee6-59096a093194" containerName="limitador" Sep 18 16:14:32.086634 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.086616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.094720 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.094687 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-c7qbt\"" Sep 18 16:14:32.094720 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.094700 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 18 16:14:32.095453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.095434 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 16:14:32.095453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.095451 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 16:14:32.096825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.096801 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8"] Sep 18 16:14:32.125895 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.125863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:32.264226 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.264190 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttjth\" (UniqueName: \"kubernetes.io/projected/2652bcc2-afa2-4ae2-88e6-1b016cffdbf5-kube-api-access-ttjth\") pod \"postgres-69fb4594cf-8h5h8\" (UID: \"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.264410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.264290 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2652bcc2-afa2-4ae2-88e6-1b016cffdbf5-data\") pod \"postgres-69fb4594cf-8h5h8\" (UID: \"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.365680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.365591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttjth\" (UniqueName: \"kubernetes.io/projected/2652bcc2-afa2-4ae2-88e6-1b016cffdbf5-kube-api-access-ttjth\") pod \"postgres-69fb4594cf-8h5h8\" (UID: \"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.365680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.365657 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2652bcc2-afa2-4ae2-88e6-1b016cffdbf5-data\") pod \"postgres-69fb4594cf-8h5h8\" (UID: \"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.366061 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.366045 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2652bcc2-afa2-4ae2-88e6-1b016cffdbf5-data\") pod \"postgres-69fb4594cf-8h5h8\" (UID: \"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.380286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.380262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttjth\" (UniqueName: \"kubernetes.io/projected/2652bcc2-afa2-4ae2-88e6-1b016cffdbf5-kube-api-access-ttjth\") pod \"postgres-69fb4594cf-8h5h8\" (UID: \"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.397206 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.397180 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:32.678311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:32.678284 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8"] Sep 18 16:14:32.679397 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:14:32.679372 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2652bcc2_afa2_4ae2_88e6_1b016cffdbf5.slice/crio-4f4a5b8d686484ab85be6d3650b9c5b744a10b78fd6b76440afa764193df9141 WatchSource:0}: Error finding container 4f4a5b8d686484ab85be6d3650b9c5b744a10b78fd6b76440afa764193df9141: Status 404 returned error can't find the container with id 4f4a5b8d686484ab85be6d3650b9c5b744a10b78fd6b76440afa764193df9141 Sep 18 16:14:33.429737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:33.429669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:33.634073 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:33.633974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" event={"ID":"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5","Type":"ContainerStarted","Data":"4f4a5b8d686484ab85be6d3650b9c5b744a10b78fd6b76440afa764193df9141"} Sep 18 16:14:34.224558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:34.224520 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:37.659735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:37.659697 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" event={"ID":"2652bcc2-afa2-4ae2-88e6-1b016cffdbf5","Type":"ContainerStarted","Data":"993d1ceca59c4a0a1fcee0c02456e18bcf72e0c0bc1159912c6dae0cd119aafe"} Sep 18 16:14:37.660105 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:37.659761 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:43.693399 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:43.693369 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" Sep 18 16:14:43.732807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:43.732757 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8h5h8" podStartSLOduration=6.879274523 podStartE2EDuration="11.732742482s" totalImagesPullingTime="4.853467959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:37.694897268 +0000 UTC m=+577.255741935" watchObservedRunningTime="2026-09-18 16:14:43.732742482 +0000 UTC m=+583.293587149" Sep 18 16:14:44.121859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:44.121826 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:44.747555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:44.747517 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:52.057772 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.057743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:52.067378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.067349 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5864496c67-sk7p7"] Sep 18 16:14:52.084697 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.084676 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.093459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.093436 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 16:14:52.093590 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.093465 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 16:14:52.093590 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.093472 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8gffl\"" Sep 18 16:14:52.093590 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.093447 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 16:14:52.119027 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.119002 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5864496c67-sk7p7"] Sep 18 16:14:52.133776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.133753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.133888 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.133868 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-metrics-certs\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.133953 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.133908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jff\" (UniqueName: \"kubernetes.io/projected/02527316-c449-4a82-a331-4cd102b76fb8-kube-api-access-56jff\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.220412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.220380 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:52.234484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.234455 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jff\" (UniqueName: \"kubernetes.io/projected/02527316-c449-4a82-a331-4cd102b76fb8-kube-api-access-56jff\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.234611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.234502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.234611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.234550 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-metrics-certs\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.234711 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:52.234682 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 18 16:14:52.234768 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:14:52.234755 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert podName:02527316-c449-4a82-a331-4cd102b76fb8 nodeName:}" failed. No retries permitted until 2026-09-18 16:14:52.734735828 +0000 UTC m=+592.295580478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert") pod "maas-controller-5864496c67-sk7p7" (UID: "02527316-c449-4a82-a331-4cd102b76fb8") : secret "maas-controller-webhook-cert" not found Sep 18 16:14:52.237188 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.237164 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-metrics-certs\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.248944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.248923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jff\" (UniqueName: \"kubernetes.io/projected/02527316-c449-4a82-a331-4cd102b76fb8-kube-api-access-56jff\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.358270 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.358204 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-jkzmr"] Sep 18 16:14:52.361414 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.361398 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.376111 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.376067 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:52.399918 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.399891 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-jkzmr"] Sep 18 16:14:52.435855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.435821 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-webhook-cert\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.435991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.435881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-metrics-certs\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.435991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.435967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf2t2\" (UniqueName: \"kubernetes.io/projected/5f3c3a54-6101-430a-aa20-0d33a75784a4-kube-api-access-tf2t2\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.536613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.536587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-webhook-cert\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.536752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.536631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-metrics-certs\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.536752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.536667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf2t2\" (UniqueName: \"kubernetes.io/projected/5f3c3a54-6101-430a-aa20-0d33a75784a4-kube-api-access-tf2t2\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.539006 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.538983 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-webhook-cert\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.539125 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.539040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-metrics-certs\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.548095 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.548061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf2t2\" (UniqueName: \"kubernetes.io/projected/5f3c3a54-6101-430a-aa20-0d33a75784a4-kube-api-access-tf2t2\") pod \"maas-controller-7b7898d6ff-jkzmr\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.672466 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.672388 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:52.738944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.738914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.741375 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.741352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert\") pod \"maas-controller-5864496c67-sk7p7\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:52.954202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.954146 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-jkzmr"] Sep 18 16:14:52.994858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:52.994834 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:53.278686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:53.278664 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5864496c67-sk7p7"] Sep 18 16:14:53.280288 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:14:53.280260 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02527316_c449_4a82_a331_4cd102b76fb8.slice/crio-a69e97b65306287d5ca3a1056f51d5eeca6238b610e441cabf1e00a141f4a38e WatchSource:0}: Error finding container a69e97b65306287d5ca3a1056f51d5eeca6238b610e441cabf1e00a141f4a38e: Status 404 returned error can't find the container with id a69e97b65306287d5ca3a1056f51d5eeca6238b610e441cabf1e00a141f4a38e Sep 18 16:14:53.756691 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:53.756645 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"62100d10d76a85b56eb4c639b4d525da69bee243e4f71806c3404f2173e48305"} Sep 18 16:14:53.758229 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:53.758197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5864496c67-sk7p7" event={"ID":"02527316-c449-4a82-a331-4cd102b76fb8","Type":"ContainerStarted","Data":"a69e97b65306287d5ca3a1056f51d5eeca6238b610e441cabf1e00a141f4a38e"} Sep 18 16:14:56.035899 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.035862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:56.231451 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.231416 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:14:56.777716 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.777669 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5864496c67-sk7p7" event={"ID":"02527316-c449-4a82-a331-4cd102b76fb8","Type":"ContainerStarted","Data":"c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444"} Sep 18 16:14:56.777960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.777740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:14:56.779663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.779637 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"526a10f0e65c1323900408c857f46963e15d6afb8cbd5666118fb13052cff717"} Sep 18 16:14:56.779803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.779763 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:14:56.816652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.816593 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5864496c67-sk7p7" podStartSLOduration=2.348199884 podStartE2EDuration="4.816576192s" totalImagesPullingTime="2.468376308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:56.815031032 +0000 UTC m=+596.375875699" watchObservedRunningTime="2026-09-18 16:14:56.816576192 +0000 UTC m=+596.377420859" Sep 18 16:14:56.850757 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:14:56.850703 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podStartSLOduration=2.065620052 podStartE2EDuration="4.850685555s" totalImagesPullingTime="2.785065503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:56.850579822 +0000 UTC m=+596.411424492" watchObservedRunningTime="2026-09-18 16:14:56.850685555 +0000 UTC m=+596.411530222" Sep 18 16:15:02.999154 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:02.999126 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:15:03.000191 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:03.000169 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:15:07.786719 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:07.786686 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:15:07.787163 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:07.786787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:15:07.983807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:07.983768 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5864496c67-sk7p7"] Sep 18 16:15:07.983966 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:07.983947 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5864496c67-sk7p7" podUID="02527316-c449-4a82-a331-4cd102b76fb8" containerName="manager" containerID="cri-o://c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444" gracePeriod=30 Sep 18 16:15:08.225916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.225895 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:15:08.379649 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.379570 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert\") pod \"02527316-c449-4a82-a331-4cd102b76fb8\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " Sep 18 16:15:08.379649 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.379647 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56jff\" (UniqueName: \"kubernetes.io/projected/02527316-c449-4a82-a331-4cd102b76fb8-kube-api-access-56jff\") pod \"02527316-c449-4a82-a331-4cd102b76fb8\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " Sep 18 16:15:08.379829 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.379687 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-metrics-certs\") pod \"02527316-c449-4a82-a331-4cd102b76fb8\" (UID: \"02527316-c449-4a82-a331-4cd102b76fb8\") " Sep 18 16:15:08.381829 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.381795 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "02527316-c449-4a82-a331-4cd102b76fb8" (UID: "02527316-c449-4a82-a331-4cd102b76fb8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:15:08.381943 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.381845 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "02527316-c449-4a82-a331-4cd102b76fb8" (UID: "02527316-c449-4a82-a331-4cd102b76fb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:15:08.381943 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.381886 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02527316-c449-4a82-a331-4cd102b76fb8-kube-api-access-56jff" (OuterVolumeSpecName: "kube-api-access-56jff") pod "02527316-c449-4a82-a331-4cd102b76fb8" (UID: "02527316-c449-4a82-a331-4cd102b76fb8"). InnerVolumeSpecName "kube-api-access-56jff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:15:08.480479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.480449 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:15:08.480479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.480478 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02527316-c449-4a82-a331-4cd102b76fb8-webhook-cert\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:15:08.480654 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.480493 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56jff\" (UniqueName: \"kubernetes.io/projected/02527316-c449-4a82-a331-4cd102b76fb8-kube-api-access-56jff\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:15:08.841953 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.841917 2563 generic.go:356] "Generic (PLEG): container finished" podID="02527316-c449-4a82-a331-4cd102b76fb8" containerID="c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444" exitCode=0 Sep 18 16:15:08.842382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.841958 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5864496c67-sk7p7" Sep 18 16:15:08.842382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.841995 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5864496c67-sk7p7" event={"ID":"02527316-c449-4a82-a331-4cd102b76fb8","Type":"ContainerDied","Data":"c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444"} Sep 18 16:15:08.842382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.842040 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5864496c67-sk7p7" event={"ID":"02527316-c449-4a82-a331-4cd102b76fb8","Type":"ContainerDied","Data":"a69e97b65306287d5ca3a1056f51d5eeca6238b610e441cabf1e00a141f4a38e"} Sep 18 16:15:08.842382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.842065 2563 scope.go:117] "RemoveContainer" containerID="c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444" Sep 18 16:15:08.851496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.851480 2563 scope.go:117] "RemoveContainer" containerID="c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444" Sep 18 16:15:08.851761 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:15:08.851743 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444\": container with ID starting with c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444 not found: ID does not exist" containerID="c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444" Sep 18 16:15:08.851816 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.851769 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444"} err="failed to get container status \"c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444\": rpc error: code = NotFound desc = could not find container \"c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444\": container with ID starting with c0fa1f99c55bd21cb3bd90ad8569d175ea5914cc297981b21469537140064444 not found: ID does not exist" Sep 18 16:15:08.876574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.876548 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5864496c67-sk7p7"] Sep 18 16:15:08.883214 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:08.883192 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5864496c67-sk7p7"] Sep 18 16:15:09.086631 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:09.086599 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02527316-c449-4a82-a331-4cd102b76fb8" path="/var/lib/kubelet/pods/02527316-c449-4a82-a331-4cd102b76fb8/volumes" Sep 18 16:15:26.676639 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:26.676557 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:26.903810 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:26.903773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:28.386389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:28.386360 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:28.765992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:28.765966 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:29.149615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.149537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:29.242337 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.242309 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx"] Sep 18 16:15:29.242637 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.242625 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02527316-c449-4a82-a331-4cd102b76fb8" containerName="manager" Sep 18 16:15:29.242683 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.242639 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02527316-c449-4a82-a331-4cd102b76fb8" containerName="manager" Sep 18 16:15:29.242717 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.242688 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02527316-c449-4a82-a331-4cd102b76fb8" containerName="manager" Sep 18 16:15:29.291264 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.291231 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx"] Sep 18 16:15:29.291409 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.291338 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.297395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.297372 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 16:15:29.302635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.302614 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 18 16:15:29.306776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.306750 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n9smt\"" Sep 18 16:15:29.345729 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.345702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a8691-29f9-4faf-b67a-83960a43c348-openshift-service-ca\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.345830 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.345747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-maas-api-tls\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.345830 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.345813 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-metrics-certs\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.345925 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.345852 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c6g\" (UniqueName: \"kubernetes.io/projected/4c7a8691-29f9-4faf-b67a-83960a43c348-kube-api-access-48c6g\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.447013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.446940 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a8691-29f9-4faf-b67a-83960a43c348-openshift-service-ca\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.447013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.446988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-maas-api-tls\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.447471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.447038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-metrics-certs\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.447471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.447079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c6g\" (UniqueName: \"kubernetes.io/projected/4c7a8691-29f9-4faf-b67a-83960a43c348-kube-api-access-48c6g\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.447733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.447705 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a8691-29f9-4faf-b67a-83960a43c348-openshift-service-ca\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.449567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.449550 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-metrics-certs\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.449633 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.449594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-maas-api-tls\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.464176 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.464148 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c6g\" (UniqueName: \"kubernetes.io/projected/4c7a8691-29f9-4faf-b67a-83960a43c348-kube-api-access-48c6g\") pod \"maas-api-6669c6f9b7-rn9hx\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.600957 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.600928 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:29.927021 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.926985 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx"] Sep 18 16:15:29.929277 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:15:29.929250 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7a8691_29f9_4faf_b67a_83960a43c348.slice/crio-521e43ef7ce5c8ef3786136f7cdac7a1d0504899f9c588180fc1b60df88107a4 WatchSource:0}: Error finding container 521e43ef7ce5c8ef3786136f7cdac7a1d0504899f9c588180fc1b60df88107a4: Status 404 returned error can't find the container with id 521e43ef7ce5c8ef3786136f7cdac7a1d0504899f9c588180fc1b60df88107a4 Sep 18 16:15:29.930566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.930547 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:15:29.948564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:29.948537 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" event={"ID":"4c7a8691-29f9-4faf-b67a-83960a43c348","Type":"ContainerStarted","Data":"521e43ef7ce5c8ef3786136f7cdac7a1d0504899f9c588180fc1b60df88107a4"} Sep 18 16:15:30.696499 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:30.696469 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:30.932564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:30.932532 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:31.116530 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.116495 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98"] Sep 18 16:15:31.119897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.119876 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.146199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.146175 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-d8pjj\"" Sep 18 16:15:31.158676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.158652 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 18 16:15:31.158834 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.158813 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvk8b\" (UniqueName: \"kubernetes.io/projected/8fccffb1-1210-4544-9243-105721024d50-kube-api-access-dvk8b\") pod \"payload-pre-processing-59fbc47bf8-zrd98\" (UID: \"8fccffb1-1210-4544-9243-105721024d50\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.158903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.158866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fccffb1-1210-4544-9243-105721024d50-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-zrd98\" (UID: \"8fccffb1-1210-4544-9243-105721024d50\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.159280 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.159254 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98"] Sep 18 16:15:31.260013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.259980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvk8b\" (UniqueName: \"kubernetes.io/projected/8fccffb1-1210-4544-9243-105721024d50-kube-api-access-dvk8b\") pod \"payload-pre-processing-59fbc47bf8-zrd98\" (UID: \"8fccffb1-1210-4544-9243-105721024d50\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.260221 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.260030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fccffb1-1210-4544-9243-105721024d50-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-zrd98\" (UID: \"8fccffb1-1210-4544-9243-105721024d50\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.260842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.260821 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fccffb1-1210-4544-9243-105721024d50-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-zrd98\" (UID: \"8fccffb1-1210-4544-9243-105721024d50\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.276794 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.276744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvk8b\" (UniqueName: \"kubernetes.io/projected/8fccffb1-1210-4544-9243-105721024d50-kube-api-access-dvk8b\") pod \"payload-pre-processing-59fbc47bf8-zrd98\" (UID: \"8fccffb1-1210-4544-9243-105721024d50\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.325318 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.325284 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bsk8t"] Sep 18 16:15:31.328961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.328941 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.361160 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.361138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a014f89f-e0fb-4319-b6ae-6cba48d802bd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bsk8t\" (UID: \"a014f89f-e0fb-4319-b6ae-6cba48d802bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.361283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.361169 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkpl\" (UniqueName: \"kubernetes.io/projected/a014f89f-e0fb-4319-b6ae-6cba48d802bd-kube-api-access-gdkpl\") pod \"payload-processing-5b54b7fbb7-bsk8t\" (UID: \"a014f89f-e0fb-4319-b6ae-6cba48d802bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.387170 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.387104 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bsk8t"] Sep 18 16:15:31.432219 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.432173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:31.462450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.462416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a014f89f-e0fb-4319-b6ae-6cba48d802bd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bsk8t\" (UID: \"a014f89f-e0fb-4319-b6ae-6cba48d802bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.462620 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.462459 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkpl\" (UniqueName: \"kubernetes.io/projected/a014f89f-e0fb-4319-b6ae-6cba48d802bd-kube-api-access-gdkpl\") pod \"payload-processing-5b54b7fbb7-bsk8t\" (UID: \"a014f89f-e0fb-4319-b6ae-6cba48d802bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.463201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.463178 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a014f89f-e0fb-4319-b6ae-6cba48d802bd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bsk8t\" (UID: \"a014f89f-e0fb-4319-b6ae-6cba48d802bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.476941 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.476912 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkpl\" (UniqueName: \"kubernetes.io/projected/a014f89f-e0fb-4319-b6ae-6cba48d802bd-kube-api-access-gdkpl\") pod \"payload-processing-5b54b7fbb7-bsk8t\" (UID: \"a014f89f-e0fb-4319-b6ae-6cba48d802bd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.642587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.642531 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:31.880199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.880170 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98"] Sep 18 16:15:31.882079 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:15:31.882055 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fccffb1_1210_4544_9243_105721024d50.slice/crio-d420c448727e4f6d4b5b2bcf33177d961d28cb7e12e55aba8043188332d307d1 WatchSource:0}: Error finding container d420c448727e4f6d4b5b2bcf33177d961d28cb7e12e55aba8043188332d307d1: Status 404 returned error can't find the container with id d420c448727e4f6d4b5b2bcf33177d961d28cb7e12e55aba8043188332d307d1 Sep 18 16:15:31.960427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.960228 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bsk8t"] Sep 18 16:15:31.961842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.961812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" event={"ID":"8fccffb1-1210-4544-9243-105721024d50","Type":"ContainerStarted","Data":"d420c448727e4f6d4b5b2bcf33177d961d28cb7e12e55aba8043188332d307d1"} Sep 18 16:15:31.962794 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:15:31.962767 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda014f89f_e0fb_4319_b6ae_6cba48d802bd.slice/crio-a57cdea8160f98ea8560606041dfa40c9722c863eee7ff5a44d0e8c908a702fe WatchSource:0}: Error finding container a57cdea8160f98ea8560606041dfa40c9722c863eee7ff5a44d0e8c908a702fe: Status 404 returned error can't find the container with id a57cdea8160f98ea8560606041dfa40c9722c863eee7ff5a44d0e8c908a702fe Sep 18 16:15:31.963584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.963561 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" event={"ID":"4c7a8691-29f9-4faf-b67a-83960a43c348","Type":"ContainerStarted","Data":"1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de"} Sep 18 16:15:31.963705 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:31.963663 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:32.026834 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:32.026741 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" podStartSLOduration=1.3287034979999999 podStartE2EDuration="3.026725039s" totalImagesPullingTime="1.698021541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:32.026029876 +0000 UTC m=+631.586874544" watchObservedRunningTime="2026-09-18 16:15:32.026725039 +0000 UTC m=+631.587569706" Sep 18 16:15:32.974550 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:32.974502 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" event={"ID":"a014f89f-e0fb-4319-b6ae-6cba48d802bd","Type":"ContainerStarted","Data":"a57cdea8160f98ea8560606041dfa40c9722c863eee7ff5a44d0e8c908a702fe"} Sep 18 16:15:33.980929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:33.980826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" event={"ID":"a014f89f-e0fb-4319-b6ae-6cba48d802bd","Type":"ContainerStarted","Data":"803c0db225d219b79f9ac3466b329e98de5c73918abe40d14096c1c4cf963ed9"} Sep 18 16:15:33.981413 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:33.981030 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:33.982667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:33.982636 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" event={"ID":"8fccffb1-1210-4544-9243-105721024d50","Type":"ContainerStarted","Data":"e19c9c79f3a06d8e9c915cbb85a844ee92b9b06b0d4bcb8f7dc989297d8ea932"} Sep 18 16:15:33.982780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:33.982732 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:34.058687 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:34.058625 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" podStartSLOduration=1.37749416 podStartE2EDuration="3.058609236s" totalImagesPullingTime="1.681115076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:34.056235402 +0000 UTC m=+633.617080073" watchObservedRunningTime="2026-09-18 16:15:34.058609236 +0000 UTC m=+633.619453903" Sep 18 16:15:34.059699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:34.059667 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" podStartSLOduration=1.461603727 podStartE2EDuration="3.059656319s" totalImagesPullingTime="1.598052592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:15:34.010394732 +0000 UTC m=+633.571239413" watchObservedRunningTime="2026-09-18 16:15:34.059656319 +0000 UTC m=+633.620501024" Sep 18 16:15:37.979721 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:37.979691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:15:39.247582 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:39.247552 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:39.609884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:39.609853 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:15:44.990052 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:44.990021 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsk8t" Sep 18 16:15:44.990496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:44.990118 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-zrd98" Sep 18 16:15:56.119218 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:15:56.119185 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:04.687097 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.687049 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw"] Sep 18 16:16:04.692223 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.692205 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.696375 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.696355 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sr7qq\"" Sep 18 16:16:04.696497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.696467 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 18 16:16:04.697284 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.697267 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 16:16:04.697391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.697371 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 16:16:04.738450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.738424 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw"] Sep 18 16:16:04.762454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.762429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:04.841227 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.841191 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.841407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.841251 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.841407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.841300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.841407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.841320 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxvb\" (UniqueName: \"kubernetes.io/projected/ea823774-144e-4a34-8aee-8a6d63ff69aa-kube-api-access-6nxvb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.841407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.841350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea823774-144e-4a34-8aee-8a6d63ff69aa-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.841596 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.841425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.942886 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.942787 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.942886 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.942830 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxvb\" (UniqueName: \"kubernetes.io/projected/ea823774-144e-4a34-8aee-8a6d63ff69aa-kube-api-access-6nxvb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.942886 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.942863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea823774-144e-4a34-8aee-8a6d63ff69aa-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.943235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.942929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.943235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.942969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.943235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.943013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.943400 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.943328 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.943400 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.943373 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.943482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.943394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.945287 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.945262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea823774-144e-4a34-8aee-8a6d63ff69aa-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.945548 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.945526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea823774-144e-4a34-8aee-8a6d63ff69aa-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:04.956997 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:04.956969 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxvb\" (UniqueName: \"kubernetes.io/projected/ea823774-144e-4a34-8aee-8a6d63ff69aa-kube-api-access-6nxvb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wm8cw\" (UID: \"ea823774-144e-4a34-8aee-8a6d63ff69aa\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:05.002794 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:05.002751 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:05.306599 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:16:05.306573 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea823774_144e_4a34_8aee_8a6d63ff69aa.slice/crio-b3ef847f09bf826bfd1066f00edda33e111268bea3d7e4a2b0f1c6652c9752fe WatchSource:0}: Error finding container b3ef847f09bf826bfd1066f00edda33e111268bea3d7e4a2b0f1c6652c9752fe: Status 404 returned error can't find the container with id b3ef847f09bf826bfd1066f00edda33e111268bea3d7e4a2b0f1c6652c9752fe Sep 18 16:16:05.306728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:05.306703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw"] Sep 18 16:16:06.151273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:06.151231 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" event={"ID":"ea823774-144e-4a34-8aee-8a6d63ff69aa","Type":"ContainerStarted","Data":"b3ef847f09bf826bfd1066f00edda33e111268bea3d7e4a2b0f1c6652c9752fe"} Sep 18 16:16:07.157555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:07.157517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" event={"ID":"ea823774-144e-4a34-8aee-8a6d63ff69aa","Type":"ContainerStarted","Data":"77c1a885f0fa762244bf437a93c9dc3c83467bd00edec9a550e6d6067ee88f93"} Sep 18 16:16:07.194803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:07.194756 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" podStartSLOduration=2.225770131 podStartE2EDuration="3.194740877s" totalImagesPullingTime="968.970746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:07.194673961 +0000 UTC m=+666.755518629" watchObservedRunningTime="2026-09-18 16:16:07.194740877 +0000 UTC m=+666.755585543" Sep 18 16:16:15.003435 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:15.003400 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:15.003921 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:15.003487 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:15.015953 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:15.015931 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:15.214863 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:15.214837 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wm8cw" Sep 18 16:16:27.188551 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:27.188518 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:34.951905 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:34.951872 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l"] Sep 18 16:16:34.955486 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:34.955463 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:34.959693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:34.959635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 18 16:16:34.979234 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:34.979208 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l"] Sep 18 16:16:35.001993 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.001961 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.002151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.002006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.002151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.002037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.002288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.002155 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.002288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.002186 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.002288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.002258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwz74\" (UniqueName: \"kubernetes.io/projected/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-kube-api-access-dwz74\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.041724 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.041676 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:35.103688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.103662 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwz74\" (UniqueName: \"kubernetes.io/projected/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-kube-api-access-dwz74\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.103844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.103709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.103844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.103824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.103967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.103864 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.103967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.103924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.103967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.103948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.104300 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.104265 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.104388 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.104354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.104433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.104386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.106179 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.106152 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.106281 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.106266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.131376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.131339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwz74\" (UniqueName: \"kubernetes.io/projected/6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0-kube-api-access-dwz74\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6tt7l\" (UID: \"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.269824 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.269790 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:35.559404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:35.559378 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l"] Sep 18 16:16:35.560011 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:16:35.559987 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6791a1f8_61ce_42a8_95c8_7a3fc65f8bb0.slice/crio-ac44ba0fb74146cdfef69088593bb8411cd337b1eafaf22b1c5b3154b4926307 WatchSource:0}: Error finding container ac44ba0fb74146cdfef69088593bb8411cd337b1eafaf22b1c5b3154b4926307: Status 404 returned error can't find the container with id ac44ba0fb74146cdfef69088593bb8411cd337b1eafaf22b1c5b3154b4926307 Sep 18 16:16:36.318017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:36.317976 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" event={"ID":"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0","Type":"ContainerStarted","Data":"c556c9b6e916c6baa56d6e68b3071b67a4978137fae067ad807c83b5bd9a9072"} Sep 18 16:16:36.318471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:36.318048 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" event={"ID":"6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0","Type":"ContainerStarted","Data":"ac44ba0fb74146cdfef69088593bb8411cd337b1eafaf22b1c5b3154b4926307"} Sep 18 16:16:36.352027 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:36.351975 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" podStartSLOduration=2.16475756 podStartE2EDuration="2.351958974s" totalImagesPullingTime="187.201414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:36.351550705 +0000 UTC m=+695.912395373" watchObservedRunningTime="2026-09-18 16:16:36.351958974 +0000 UTC m=+695.912803643" Sep 18 16:16:45.270067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:45.269971 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:45.270067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:45.270053 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:45.282343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:45.282312 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:45.375497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:45.375468 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6tt7l" Sep 18 16:16:51.255313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:51.255271 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:51.851994 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:51.851963 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:51.965343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:51.965312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:16:59.025313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:16:59.025278 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:17:06.447162 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:17:06.447132 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:17:11.990627 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:17:11.990593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:17:15.510530 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:17:15.510496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:17:39.639022 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:17:39.638982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:18:07.696571 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:18:07.696535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:18:37.420981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:18:37.420949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:18:37.501583 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:18:37.501548 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:03.922908 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:03.922872 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:04.074706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:04.074678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:05.201729 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:05.201698 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:05.453804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:05.453725 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:23.495151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:23.495117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:23.601575 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:23.601535 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:23.719099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:23.719064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:23.897306 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:23.897275 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:24.120295 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:24.120264 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:24.324365 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:24.324331 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:24.469502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:24.469466 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:24.582626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:24.582544 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:24.997672 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:24.997579 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:25.103012 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:25.097881 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:25.331278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:25.331248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:27.342433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:27.342401 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:27.484763 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:27.484733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:27.588765 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:27.588735 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:28.649636 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.649602 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j"] Sep 18 16:19:28.652954 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.652926 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.660052 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.660030 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4f2d8a-serving-cert\"" Sep 18 16:19:28.744550 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.744517 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j"] Sep 18 16:19:28.750941 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.750915 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:28.791155 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.791114 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-metrics-certs\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.791332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.791171 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d61092-8da2-4fa0-a0ab-e189d8742447-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.791332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.791237 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.791332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.791288 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnknd\" (UniqueName: \"kubernetes.io/projected/e3d61092-8da2-4fa0-a0ab-e189d8742447-kube-api-access-bnknd\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.892503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.892465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnknd\" (UniqueName: \"kubernetes.io/projected/e3d61092-8da2-4fa0-a0ab-e189d8742447-kube-api-access-bnknd\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.892661 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.892582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-metrics-certs\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.892661 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.892606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d61092-8da2-4fa0-a0ab-e189d8742447-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.892661 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.892624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.892844 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:19:28.892733 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-serving-cert: secret "maas-api-e2e-worker-w4-4f2d8a-serving-cert" not found Sep 18 16:19:28.892844 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:19:28.892794 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls podName:e3d61092-8da2-4fa0-a0ab-e189d8742447 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:29.39277868 +0000 UTC m=+868.953623329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls") pod "maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" (UID: "e3d61092-8da2-4fa0-a0ab-e189d8742447") : secret "maas-api-e2e-worker-w4-4f2d8a-serving-cert" not found Sep 18 16:19:28.893319 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.893292 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d61092-8da2-4fa0-a0ab-e189d8742447-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.895062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.895040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-metrics-certs\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.906555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.906497 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnknd\" (UniqueName: \"kubernetes.io/projected/e3d61092-8da2-4fa0-a0ab-e189d8742447-kube-api-access-bnknd\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:28.947062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:28.947041 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:29.171148 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.170998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:29.396512 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.396476 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:29.398807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.398784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:29.565580 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.565548 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:29.656981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.656944 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:29.766509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.766478 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:29.904066 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.904038 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j"] Sep 18 16:19:29.904484 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:29.904461 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d61092_8da2_4fa0_a0ab_e189d8742447.slice/crio-6d1ed9af88c8771b18317e7e08ea1087a9efea59055a46b41c43fb1ea0033350 WatchSource:0}: Error finding container 6d1ed9af88c8771b18317e7e08ea1087a9efea59055a46b41c43fb1ea0033350: Status 404 returned error can't find the container with id 6d1ed9af88c8771b18317e7e08ea1087a9efea59055a46b41c43fb1ea0033350 Sep 18 16:19:29.906564 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.906539 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp"] Sep 18 16:19:29.911061 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.911044 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:29.915484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.915459 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4f2d8a-dockercfg-jcg4j\"" Sep 18 16:19:29.915792 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.915770 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4f2d8a\"" Sep 18 16:19:29.927512 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.927484 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp"] Sep 18 16:19:29.954492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.954463 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4"] Sep 18 16:19:29.957908 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.957882 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:29.983628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:29.983602 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4"] Sep 18 16:19:30.003612 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.003583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d162f08-db46-4065-ad8e-e61620b5c225-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.003733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.003661 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdjs\" (UniqueName: \"kubernetes.io/projected/2d162f08-db46-4065-ad8e-e61620b5c225-kube-api-access-qbdjs\") pod \"payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.104368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.104334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d162f08-db46-4065-ad8e-e61620b5c225-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.104514 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.104411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5tx\" (UniqueName: \"kubernetes.io/projected/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-kube-api-access-zk5tx\") pod \"payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.104514 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.104481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdjs\" (UniqueName: \"kubernetes.io/projected/2d162f08-db46-4065-ad8e-e61620b5c225-kube-api-access-qbdjs\") pod \"payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.104589 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.104516 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.105105 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.105073 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d162f08-db46-4065-ad8e-e61620b5c225-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.118663 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.118634 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdjs\" (UniqueName: \"kubernetes.io/projected/2d162f08-db46-4065-ad8e-e61620b5c225-kube-api-access-qbdjs\") pod \"payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.205376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.205290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5tx\" (UniqueName: \"kubernetes.io/projected/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-kube-api-access-zk5tx\") pod \"payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.205526 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.205375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.206293 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.206267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.216740 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.216715 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5tx\" (UniqueName: \"kubernetes.io/projected/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-kube-api-access-zk5tx\") pod \"payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.225468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.223707 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:30.226985 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.226946 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" event={"ID":"e3d61092-8da2-4fa0-a0ab-e189d8742447","Type":"ContainerStarted","Data":"6d1ed9af88c8771b18317e7e08ea1087a9efea59055a46b41c43fb1ea0033350"} Sep 18 16:19:30.269048 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.269004 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:30.296133 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.296075 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:30.534806 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.534781 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp"] Sep 18 16:19:30.536470 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:30.536444 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d162f08_db46_4065_ad8e_e61620b5c225.slice/crio-2e76fd6c2119eefec95d87922c74d65519cc446c2d02a42975c471f91ae3373f WatchSource:0}: Error finding container 2e76fd6c2119eefec95d87922c74d65519cc446c2d02a42975c471f91ae3373f: Status 404 returned error can't find the container with id 2e76fd6c2119eefec95d87922c74d65519cc446c2d02a42975c471f91ae3373f Sep 18 16:19:30.551208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.551178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:30.585462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.584962 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4"] Sep 18 16:19:30.589561 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:30.589533 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fef0cc0_a218_4e16_b4bd_b06dad7d3e06.slice/crio-89a893a01532f55946aa0153b637fa2ebc3769f93f99df018b663daead170d38 WatchSource:0}: Error finding container 89a893a01532f55946aa0153b637fa2ebc3769f93f99df018b663daead170d38: Status 404 returned error can't find the container with id 89a893a01532f55946aa0153b637fa2ebc3769f93f99df018b663daead170d38 Sep 18 16:19:30.727381 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.727328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:30.886416 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:30.886328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:31.173037 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.172954 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:31.237029 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.236990 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" event={"ID":"2d162f08-db46-4065-ad8e-e61620b5c225","Type":"ContainerStarted","Data":"ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642"} Sep 18 16:19:31.237228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.237063 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:31.237228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.237103 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" event={"ID":"2d162f08-db46-4065-ad8e-e61620b5c225","Type":"ContainerStarted","Data":"2e76fd6c2119eefec95d87922c74d65519cc446c2d02a42975c471f91ae3373f"} Sep 18 16:19:31.238900 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.238872 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" event={"ID":"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06","Type":"ContainerStarted","Data":"ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2"} Sep 18 16:19:31.239043 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.238918 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" event={"ID":"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06","Type":"ContainerStarted","Data":"89a893a01532f55946aa0153b637fa2ebc3769f93f99df018b663daead170d38"} Sep 18 16:19:31.239043 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.238945 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:31.240668 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.240641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" event={"ID":"e3d61092-8da2-4fa0-a0ab-e189d8742447","Type":"ContainerStarted","Data":"fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10"} Sep 18 16:19:31.240797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.240758 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:31.266280 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.266236 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" podStartSLOduration=2.266219644 podStartE2EDuration="2.266219644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:31.262634481 +0000 UTC m=+870.823479148" watchObservedRunningTime="2026-09-18 16:19:31.266219644 +0000 UTC m=+870.827064311" Sep 18 16:19:31.274107 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.274069 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:31.317742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.317698 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" podStartSLOduration=2.317683083 podStartE2EDuration="2.317683083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:31.315777819 +0000 UTC m=+870.876622487" watchObservedRunningTime="2026-09-18 16:19:31.317683083 +0000 UTC m=+870.878527750" Sep 18 16:19:31.365334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.365209 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" podStartSLOduration=2.8661592860000003 podStartE2EDuration="3.365187798s" totalImagesPullingTime="499.028512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:31.359477567 +0000 UTC m=+870.920322235" watchObservedRunningTime="2026-09-18 16:19:31.365187798 +0000 UTC m=+870.926032467" Sep 18 16:19:31.420950 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.420907 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn"] Sep 18 16:19:31.425451 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.425386 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.433186 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.433162 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fbc529\"" Sep 18 16:19:31.433445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.433428 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fbc529-dockercfg-zqtrs\"" Sep 18 16:19:31.451839 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.451365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn"] Sep 18 16:19:31.483248 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.483214 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:31.503172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.503138 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t"] Sep 18 16:19:31.506628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.506608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.518422 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.518402 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4pz\" (UniqueName: \"kubernetes.io/projected/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-kube-api-access-wl4pz\") pod \"payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.518524 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.518468 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.544169 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.544141 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t"] Sep 18 16:19:31.619332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.619298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.619470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.619346 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d64134-383e-433b-87ef-40e77bc80fb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.619470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.619401 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27lx\" (UniqueName: \"kubernetes.io/projected/01d64134-383e-433b-87ef-40e77bc80fb4-kube-api-access-g27lx\") pod \"payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.619470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.619425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4pz\" (UniqueName: \"kubernetes.io/projected/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-kube-api-access-wl4pz\") pod \"payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.619994 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.619976 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.636949 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.636919 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4pz\" (UniqueName: \"kubernetes.io/projected/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-kube-api-access-wl4pz\") pod \"payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.720160 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.720074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d64134-383e-433b-87ef-40e77bc80fb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.720322 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.720172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27lx\" (UniqueName: \"kubernetes.io/projected/01d64134-383e-433b-87ef-40e77bc80fb4-kube-api-access-g27lx\") pod \"payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.720920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.720893 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d64134-383e-433b-87ef-40e77bc80fb4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.732070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.732049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27lx\" (UniqueName: \"kubernetes.io/projected/01d64134-383e-433b-87ef-40e77bc80fb4-kube-api-access-g27lx\") pod \"payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:31.738359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.738340 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:31.816375 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:31.816333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:32.023440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.023414 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn"] Sep 18 16:19:32.025399 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:32.025370 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b2cf67_9e46_4ffa_898f_f4735b2cebb2.slice/crio-f9fb92c24399242f986e1589ab9d1f43221a4bebff1b722d8cfd8f128d187d1f WatchSource:0}: Error finding container f9fb92c24399242f986e1589ab9d1f43221a4bebff1b722d8cfd8f128d187d1f: Status 404 returned error can't find the container with id f9fb92c24399242f986e1589ab9d1f43221a4bebff1b722d8cfd8f128d187d1f Sep 18 16:19:32.111272 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.111247 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t"] Sep 18 16:19:32.112124 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:32.112095 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d64134_383e_433b_87ef_40e77bc80fb4.slice/crio-3dca93b252530e892f0cbb84db84ca171e01fdc34d08ba1217227d07fb48601e WatchSource:0}: Error finding container 3dca93b252530e892f0cbb84db84ca171e01fdc34d08ba1217227d07fb48601e: Status 404 returned error can't find the container with id 3dca93b252530e892f0cbb84db84ca171e01fdc34d08ba1217227d07fb48601e Sep 18 16:19:32.199047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.199024 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:32.249823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.249791 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" event={"ID":"01d64134-383e-433b-87ef-40e77bc80fb4","Type":"ContainerStarted","Data":"0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73"} Sep 18 16:19:32.249984 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.249850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" event={"ID":"01d64134-383e-433b-87ef-40e77bc80fb4","Type":"ContainerStarted","Data":"3dca93b252530e892f0cbb84db84ca171e01fdc34d08ba1217227d07fb48601e"} Sep 18 16:19:32.249984 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.249957 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:32.252002 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.251966 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" event={"ID":"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2","Type":"ContainerStarted","Data":"cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c"} Sep 18 16:19:32.252139 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.252024 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" event={"ID":"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2","Type":"ContainerStarted","Data":"f9fb92c24399242f986e1589ab9d1f43221a4bebff1b722d8cfd8f128d187d1f"} Sep 18 16:19:32.252235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.252218 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:32.273767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.273658 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" podStartSLOduration=1.273639303 podStartE2EDuration="1.273639303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:32.269678031 +0000 UTC m=+871.830522712" watchObservedRunningTime="2026-09-18 16:19:32.273639303 +0000 UTC m=+871.834483971" Sep 18 16:19:32.299259 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.299204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:32.299843 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.299782 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" podStartSLOduration=1.299767767 podStartE2EDuration="1.299767767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:32.295926034 +0000 UTC m=+871.856770704" watchObservedRunningTime="2026-09-18 16:19:32.299767767 +0000 UTC m=+871.860612437" Sep 18 16:19:32.652015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.651845 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:32.752975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.752943 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:32.837374 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.837338 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq"] Sep 18 16:19:32.841531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.841507 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:32.844503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.844483 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1eec1d\"" Sep 18 16:19:32.848031 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.848006 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp"] Sep 18 16:19:32.851799 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.851778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:32.856166 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.856140 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq"] Sep 18 16:19:32.857401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.857380 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1eec1d-dockercfg-rfw66\"" Sep 18 16:19:32.866768 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.866747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp"] Sep 18 16:19:32.932320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.932229 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbznn\" (UniqueName: \"kubernetes.io/projected/c166202f-1ca2-4ada-8ff9-b096dca559d8-kube-api-access-vbznn\") pod \"payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:32.932320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.932291 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491163c7-d81c-4112-bfd0-99e5847a9637-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:32.932532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.932333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c166202f-1ca2-4ada-8ff9-b096dca559d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:32.932532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.932428 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2qn\" (UniqueName: \"kubernetes.io/projected/491163c7-d81c-4112-bfd0-99e5847a9637-kube-api-access-pf2qn\") pod \"payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:32.948880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:32.948849 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:33.033959 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.033926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbznn\" (UniqueName: \"kubernetes.io/projected/c166202f-1ca2-4ada-8ff9-b096dca559d8-kube-api-access-vbznn\") pod \"payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:33.034131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.033978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491163c7-d81c-4112-bfd0-99e5847a9637-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:33.034131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.034007 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c166202f-1ca2-4ada-8ff9-b096dca559d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:33.034131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.034037 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2qn\" (UniqueName: \"kubernetes.io/projected/491163c7-d81c-4112-bfd0-99e5847a9637-kube-api-access-pf2qn\") pod \"payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:33.034725 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.034705 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491163c7-d81c-4112-bfd0-99e5847a9637-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:33.034778 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.034755 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c166202f-1ca2-4ada-8ff9-b096dca559d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:33.047654 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.047609 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2qn\" (UniqueName: \"kubernetes.io/projected/491163c7-d81c-4112-bfd0-99e5847a9637-kube-api-access-pf2qn\") pod \"payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:33.048310 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.048292 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbznn\" (UniqueName: \"kubernetes.io/projected/c166202f-1ca2-4ada-8ff9-b096dca559d8-kube-api-access-vbznn\") pod \"payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:33.153643 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.153612 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:33.165551 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.165529 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:33.217782 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.217695 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:33.437801 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.437772 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq"] Sep 18 16:19:33.438561 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:33.438534 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc166202f_1ca2_4ada_8ff9_b096dca559d8.slice/crio-4a269df3b1da00aeed8bfd145a81449aa1b378e1fb9d240e775bf8e702a71bc2 WatchSource:0}: Error finding container 4a269df3b1da00aeed8bfd145a81449aa1b378e1fb9d240e775bf8e702a71bc2: Status 404 returned error can't find the container with id 4a269df3b1da00aeed8bfd145a81449aa1b378e1fb9d240e775bf8e702a71bc2 Sep 18 16:19:33.445776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:33.445747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp"] Sep 18 16:19:33.447053 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:33.447032 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491163c7_d81c_4112_bfd0_99e5847a9637.slice/crio-31eaf3e8eb893c673bf91a507611128044b0da821cf8f5e5d510cb51dc39ec3e WatchSource:0}: Error finding container 31eaf3e8eb893c673bf91a507611128044b0da821cf8f5e5d510cb51dc39ec3e: Status 404 returned error can't find the container with id 31eaf3e8eb893c673bf91a507611128044b0da821cf8f5e5d510cb51dc39ec3e Sep 18 16:19:34.184348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.184313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:34.273790 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.273750 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" event={"ID":"491163c7-d81c-4112-bfd0-99e5847a9637","Type":"ContainerStarted","Data":"b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e"} Sep 18 16:19:34.273973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.273814 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" event={"ID":"491163c7-d81c-4112-bfd0-99e5847a9637","Type":"ContainerStarted","Data":"31eaf3e8eb893c673bf91a507611128044b0da821cf8f5e5d510cb51dc39ec3e"} Sep 18 16:19:34.273973 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.273943 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:34.275577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.275550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" event={"ID":"c166202f-1ca2-4ada-8ff9-b096dca559d8","Type":"ContainerStarted","Data":"37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7"} Sep 18 16:19:34.275708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.275596 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" event={"ID":"c166202f-1ca2-4ada-8ff9-b096dca559d8","Type":"ContainerStarted","Data":"4a269df3b1da00aeed8bfd145a81449aa1b378e1fb9d240e775bf8e702a71bc2"} Sep 18 16:19:34.275708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.275676 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:34.284652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.284629 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:34.295706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.295659 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" podStartSLOduration=2.2956405970000002 podStartE2EDuration="2.295640597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:34.293940909 +0000 UTC m=+873.854785580" watchObservedRunningTime="2026-09-18 16:19:34.295640597 +0000 UTC m=+873.856485266" Sep 18 16:19:34.321637 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.321594 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" podStartSLOduration=2.321581099 podStartE2EDuration="2.321581099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:34.320636309 +0000 UTC m=+873.881480979" watchObservedRunningTime="2026-09-18 16:19:34.321581099 +0000 UTC m=+873.882425766" Sep 18 16:19:34.398960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.398932 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:34.530310 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.530275 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:34.826457 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.826375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:34.950236 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:34.950206 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:35.049789 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:35.049746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:35.918768 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:35.918734 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:37.257992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:37.257964 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:19:37.433699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:37.433669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:37.538497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:37.538413 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:38.009509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.009482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:38.131636 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.131607 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:38.257302 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.257276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:38.369496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.369411 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48"] Sep 18 16:19:38.373258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.373238 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.375886 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.375865 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-eef18d\"" Sep 18 16:19:38.376036 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.376016 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-eef18d-dockercfg-lk7rp\"" Sep 18 16:19:38.388181 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.388157 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48"] Sep 18 16:19:38.486584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.486546 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.486727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.486602 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5db67\" (UniqueName: \"kubernetes.io/projected/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-kube-api-access-5db67\") pod \"payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.587659 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.587623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.587857 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.587670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5db67\" (UniqueName: \"kubernetes.io/projected/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-kube-api-access-5db67\") pod \"payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.588429 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.588406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.597305 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.597280 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5db67\" (UniqueName: \"kubernetes.io/projected/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-kube-api-access-5db67\") pod \"payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.715780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.715706 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:38.860569 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:38.860539 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:39.003512 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:39.003483 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acb4e39_21a7_4cbc_ad37_fc7c5e5d8436.slice/crio-83e515933bba56e7fc9b970eb977240007cfe6ec9f61a5b3a5520a1aa9ee0b61 WatchSource:0}: Error finding container 83e515933bba56e7fc9b970eb977240007cfe6ec9f61a5b3a5520a1aa9ee0b61: Status 404 returned error can't find the container with id 83e515933bba56e7fc9b970eb977240007cfe6ec9f61a5b3a5520a1aa9ee0b61 Sep 18 16:19:39.003647 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:39.003577 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48"] Sep 18 16:19:39.306479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:39.306389 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" event={"ID":"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436","Type":"ContainerStarted","Data":"c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5"} Sep 18 16:19:39.306479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:39.306442 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" event={"ID":"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436","Type":"ContainerStarted","Data":"83e515933bba56e7fc9b970eb977240007cfe6ec9f61a5b3a5520a1aa9ee0b61"} Sep 18 16:19:39.306671 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:39.306497 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:39.335160 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:39.335102 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" podStartSLOduration=1.335061008 podStartE2EDuration="1.335061008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:39.330052941 +0000 UTC m=+878.890897608" watchObservedRunningTime="2026-09-18 16:19:39.335061008 +0000 UTC m=+878.895905677" Sep 18 16:19:39.609722 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:39.609645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:40.256345 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:40.256314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:41.331360 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:41.331329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:42.052277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.052243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:42.254236 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.254200 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:19:42.254427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.254346 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:19:42.434131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.434031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:42.572201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.572165 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:42.691482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.691395 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:42.972765 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:42.972683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:43.203705 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.203631 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:43.267470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.267445 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:19:43.267644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.267512 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:19:43.414810 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.414777 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:43.474589 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.474554 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f"] Sep 18 16:19:43.478779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.478759 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.485208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.485185 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-75688a34-dockercfg-6jkwf\"" Sep 18 16:19:43.485525 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.485507 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-75688a34\"" Sep 18 16:19:43.500244 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.500222 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f"] Sep 18 16:19:43.561141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.561051 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth"] Sep 18 16:19:43.564606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.564587 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.584195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.584168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth"] Sep 18 16:19:43.634561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.634536 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29dz\" (UniqueName: \"kubernetes.io/projected/5b5423ba-ffca-4d60-904f-d6f36141e7d1-kube-api-access-v29dz\") pod \"payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.634696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.634598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5423ba-ffca-4d60-904f-d6f36141e7d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.735544 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.735513 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-plugins-config-volume\") pod \"payload-processing-e2e-ait-75688a34-5659bd945f-xbnth\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.735727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.735551 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29dz\" (UniqueName: \"kubernetes.io/projected/5b5423ba-ffca-4d60-904f-d6f36141e7d1-kube-api-access-v29dz\") pod \"payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.735727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.735572 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vps69\" (UniqueName: \"kubernetes.io/projected/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-kube-api-access-vps69\") pod \"payload-processing-e2e-ait-75688a34-5659bd945f-xbnth\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.735727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.735710 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5423ba-ffca-4d60-904f-d6f36141e7d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.736403 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.736387 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5423ba-ffca-4d60-904f-d6f36141e7d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.750831 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.750806 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29dz\" (UniqueName: \"kubernetes.io/projected/5b5423ba-ffca-4d60-904f-d6f36141e7d1-kube-api-access-v29dz\") pod \"payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.788884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.788854 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:43.837056 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.836929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-plugins-config-volume\") pod \"payload-processing-e2e-ait-75688a34-5659bd945f-xbnth\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.837056 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.836982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vps69\" (UniqueName: \"kubernetes.io/projected/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-kube-api-access-vps69\") pod \"payload-processing-e2e-ait-75688a34-5659bd945f-xbnth\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.837888 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.837850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-plugins-config-volume\") pod \"payload-processing-e2e-ait-75688a34-5659bd945f-xbnth\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.850346 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.849904 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vps69\" (UniqueName: \"kubernetes.io/projected/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-kube-api-access-vps69\") pod \"payload-processing-e2e-ait-75688a34-5659bd945f-xbnth\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.874938 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.874906 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:43.978042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:43.978010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:44.080964 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.080898 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f"] Sep 18 16:19:44.083263 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:44.083239 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5423ba_ffca_4d60_904f_d6f36141e7d1.slice/crio-0902af13c39d82bceac24b858e7c3468c7a0311a9f5fd9995f2d2764132b6e42 WatchSource:0}: Error finding container 0902af13c39d82bceac24b858e7c3468c7a0311a9f5fd9995f2d2764132b6e42: Status 404 returned error can't find the container with id 0902af13c39d82bceac24b858e7c3468c7a0311a9f5fd9995f2d2764132b6e42 Sep 18 16:19:44.155172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.155145 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth"] Sep 18 16:19:44.156320 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:44.156298 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9875c8_98bc_48b0_a4d9_ef5c6148f449.slice/crio-fc3f8390cd30b3251d4d1d32ba2459c926a3f431d85b0752cdc7e14407040ba2 WatchSource:0}: Error finding container fc3f8390cd30b3251d4d1d32ba2459c926a3f431d85b0752cdc7e14407040ba2: Status 404 returned error can't find the container with id fc3f8390cd30b3251d4d1d32ba2459c926a3f431d85b0752cdc7e14407040ba2 Sep 18 16:19:44.340315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.340283 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" event={"ID":"7c9875c8-98bc-48b0-a4d9-ef5c6148f449","Type":"ContainerStarted","Data":"7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0"} Sep 18 16:19:44.340504 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.340335 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" event={"ID":"7c9875c8-98bc-48b0-a4d9-ef5c6148f449","Type":"ContainerStarted","Data":"fc3f8390cd30b3251d4d1d32ba2459c926a3f431d85b0752cdc7e14407040ba2"} Sep 18 16:19:44.340504 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.340363 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:44.341977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.341907 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" event={"ID":"5b5423ba-ffca-4d60-904f-d6f36141e7d1","Type":"ContainerStarted","Data":"671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5"} Sep 18 16:19:44.341977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.341952 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" event={"ID":"5b5423ba-ffca-4d60-904f-d6f36141e7d1","Type":"ContainerStarted","Data":"0902af13c39d82bceac24b858e7c3468c7a0311a9f5fd9995f2d2764132b6e42"} Sep 18 16:19:44.342170 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.342020 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:44.367739 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.367676 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" podStartSLOduration=1.367654607 podStartE2EDuration="1.367654607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:44.364454808 +0000 UTC m=+883.925299476" watchObservedRunningTime="2026-09-18 16:19:44.367654607 +0000 UTC m=+883.928499276" Sep 18 16:19:44.397296 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.397241 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" podStartSLOduration=1.397226219 podStartE2EDuration="1.397226219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:44.397019454 +0000 UTC m=+883.957864122" watchObservedRunningTime="2026-09-18 16:19:44.397226219 +0000 UTC m=+883.958070930" Sep 18 16:19:44.445587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.445561 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:44.688120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.687973 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:44.819229 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.819197 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:44.935947 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:44.935918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:45.284495 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:45.284465 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:19:45.284722 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:45.284702 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:19:45.498485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:45.498422 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:46.800263 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:46.800218 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:47.165780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:47.165673 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:47.269246 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:47.269215 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:47.536065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:47.536033 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:47.675464 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:47.675409 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:47.796191 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:47.796072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:47.989219 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:47.989194 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:48.299565 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:48.299533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:48.627138 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:48.627049 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:49.577399 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:49.577364 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:50.315420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:50.315390 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:19:50.622399 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:50.622314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:51.691637 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:51.691602 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:51.793374 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:51.793339 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:51.901421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:51.901385 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:52.857320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:52.857282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:53.130484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:53.130401 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:54.054690 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:54.054651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:54.590488 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:54.590455 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:54.968420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:54.968328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:55.080123 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.080081 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:55.355694 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.355658 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:19:55.355876 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.355742 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:19:55.666528 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.666450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:55.786464 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.786433 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:55.856138 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.856101 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q"] Sep 18 16:19:55.859849 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.859826 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:55.862849 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.862827 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f22789\"" Sep 18 16:19:55.862948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.862855 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f22789-dockercfg-t5snx\"" Sep 18 16:19:55.873932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.873909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q"] Sep 18 16:19:55.923479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.923399 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj"] Sep 18 16:19:55.927169 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.927154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:55.937630 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.937585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4e7878-b08a-4004-b618-fd634fbbfdde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:55.937842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.937694 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:55.937842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.937732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t6rm\" (UniqueName: \"kubernetes.io/projected/ca4e7878-b08a-4004-b618-fd634fbbfdde-kube-api-access-8t6rm\") pod \"payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:55.937842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.937783 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84rd\" (UniqueName: \"kubernetes.io/projected/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-kube-api-access-v84rd\") pod \"payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:55.941799 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:55.941637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj"] Sep 18 16:19:56.038691 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.038649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4e7878-b08a-4004-b618-fd634fbbfdde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:56.038847 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.038822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:56.038889 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.038845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t6rm\" (UniqueName: \"kubernetes.io/projected/ca4e7878-b08a-4004-b618-fd634fbbfdde-kube-api-access-8t6rm\") pod \"payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:56.038922 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.038888 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84rd\" (UniqueName: \"kubernetes.io/projected/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-kube-api-access-v84rd\") pod \"payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:56.039496 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.039474 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4e7878-b08a-4004-b618-fd634fbbfdde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:56.039578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.039536 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:56.049035 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.049002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84rd\" (UniqueName: \"kubernetes.io/projected/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-kube-api-access-v84rd\") pod \"payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:56.049457 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.049436 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t6rm\" (UniqueName: \"kubernetes.io/projected/ca4e7878-b08a-4004-b618-fd634fbbfdde-kube-api-access-8t6rm\") pod \"payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:56.171986 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.171954 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:56.237431 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.237378 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:56.237944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.237918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:56.452097 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.452053 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q"] Sep 18 16:19:56.453061 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:56.453035 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6b5a0b_a1ef_4000_83b6_43c2686813fd.slice/crio-821e912cb938b1c49338978b1218e8a99ce6f4743d66a8ab9a4099ce491cabf5 WatchSource:0}: Error finding container 821e912cb938b1c49338978b1218e8a99ce6f4743d66a8ab9a4099ce491cabf5: Status 404 returned error can't find the container with id 821e912cb938b1c49338978b1218e8a99ce6f4743d66a8ab9a4099ce491cabf5 Sep 18 16:19:56.522265 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.521293 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj"] Sep 18 16:19:56.523408 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:19:56.523383 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4e7878_b08a_4004_b618_fd634fbbfdde.slice/crio-f1538540157f4cf998f10bc79e8cccd17f133b13cdc6e555d81f80944d8d11ac WatchSource:0}: Error finding container f1538540157f4cf998f10bc79e8cccd17f133b13cdc6e555d81f80944d8d11ac: Status 404 returned error can't find the container with id f1538540157f4cf998f10bc79e8cccd17f133b13cdc6e555d81f80944d8d11ac Sep 18 16:19:56.572093 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:56.572047 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:57.026474 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.026441 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:57.426299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.426188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" event={"ID":"ca4e7878-b08a-4004-b618-fd634fbbfdde","Type":"ContainerStarted","Data":"6a4299dc0ee781ccb8e28da2a7b02fb5e043b4508111ae349378a52bdca8f3e1"} Sep 18 16:19:57.426299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.426261 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" event={"ID":"ca4e7878-b08a-4004-b618-fd634fbbfdde","Type":"ContainerStarted","Data":"f1538540157f4cf998f10bc79e8cccd17f133b13cdc6e555d81f80944d8d11ac"} Sep 18 16:19:57.426299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.426300 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:19:57.428295 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.428269 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" event={"ID":"fa6b5a0b-a1ef-4000-83b6-43c2686813fd","Type":"ContainerStarted","Data":"5124d71d8b0c54ee186d2ccc837de505f34cc310e4eceb2869542738cfe4cdf3"} Sep 18 16:19:57.428427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.428308 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" event={"ID":"fa6b5a0b-a1ef-4000-83b6-43c2686813fd","Type":"ContainerStarted","Data":"821e912cb938b1c49338978b1218e8a99ce6f4743d66a8ab9a4099ce491cabf5"} Sep 18 16:19:57.428427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.428360 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:19:57.448979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.448923 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" podStartSLOduration=2.448904692 podStartE2EDuration="2.448904692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:57.446723227 +0000 UTC m=+897.007567896" watchObservedRunningTime="2026-09-18 16:19:57.448904692 +0000 UTC m=+897.009749359" Sep 18 16:19:57.466827 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.466773 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" podStartSLOduration=2.466756325 podStartE2EDuration="2.466756325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:57.465357038 +0000 UTC m=+897.026201705" watchObservedRunningTime="2026-09-18 16:19:57.466756325 +0000 UTC m=+897.027601024" Sep 18 16:19:57.809840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:57.809809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:58.667940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:58.667892 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:19:59.783007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:19:59.782978 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:00.530122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:00.530074 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:01.700372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:01.700340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:02.531159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:02.531117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:02.627285 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:02.627253 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:03.043468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:03.043441 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:20:03.043932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:03.043459 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:20:03.239154 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:03.239122 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:04.130506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.130474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:04.169108 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.169057 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth"] Sep 18 16:20:04.169376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.169351 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" podUID="7c9875c8-98bc-48b0-a4d9-ef5c6148f449" containerName="payload-processing" containerID="cri-o://7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0" gracePeriod=30 Sep 18 16:20:04.184526 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.184496 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f"] Sep 18 16:20:04.184926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.184746 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" podUID="5b5423ba-ffca-4d60-904f-d6f36141e7d1" containerName="payload-pre-processing" containerID="cri-o://671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5" gracePeriod=30 Sep 18 16:20:04.232404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.232378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:04.362173 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.362141 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:04.447729 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.447645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:04.465885 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.465857 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:20:04.469242 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.469220 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:20:04.471400 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.471378 2563 generic.go:356] "Generic (PLEG): container finished" podID="7c9875c8-98bc-48b0-a4d9-ef5c6148f449" containerID="7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0" exitCode=0 Sep 18 16:20:04.471500 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.471417 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" Sep 18 16:20:04.471500 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.471449 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" event={"ID":"7c9875c8-98bc-48b0-a4d9-ef5c6148f449","Type":"ContainerDied","Data":"7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0"} Sep 18 16:20:04.471614 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.471501 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth" event={"ID":"7c9875c8-98bc-48b0-a4d9-ef5c6148f449","Type":"ContainerDied","Data":"fc3f8390cd30b3251d4d1d32ba2459c926a3f431d85b0752cdc7e14407040ba2"} Sep 18 16:20:04.471614 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.471525 2563 scope.go:117] "RemoveContainer" containerID="7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0" Sep 18 16:20:04.473020 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.472995 2563 generic.go:356] "Generic (PLEG): container finished" podID="5b5423ba-ffca-4d60-904f-d6f36141e7d1" containerID="671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5" exitCode=0 Sep 18 16:20:04.473144 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.473033 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" Sep 18 16:20:04.473144 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.473031 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" event={"ID":"5b5423ba-ffca-4d60-904f-d6f36141e7d1","Type":"ContainerDied","Data":"671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5"} Sep 18 16:20:04.473144 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.473132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f" event={"ID":"5b5423ba-ffca-4d60-904f-d6f36141e7d1","Type":"ContainerDied","Data":"0902af13c39d82bceac24b858e7c3468c7a0311a9f5fd9995f2d2764132b6e42"} Sep 18 16:20:04.480618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.480599 2563 scope.go:117] "RemoveContainer" containerID="7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0" Sep 18 16:20:04.480884 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:20:04.480865 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0\": container with ID starting with 7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0 not found: ID does not exist" containerID="7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0" Sep 18 16:20:04.480933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.480893 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0"} err="failed to get container status \"7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0\": rpc error: code = NotFound desc = could not find container \"7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0\": container with ID starting with 7eb56a2a20616af6045a6fd2af765a1cb83943c3337113b1747e2796095039b0 not found: ID does not exist" Sep 18 16:20:04.480933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.480911 2563 scope.go:117] "RemoveContainer" containerID="671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5" Sep 18 16:20:04.489853 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.489830 2563 scope.go:117] "RemoveContainer" containerID="671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5" Sep 18 16:20:04.490247 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:20:04.490216 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5\": container with ID starting with 671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5 not found: ID does not exist" containerID="671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5" Sep 18 16:20:04.490346 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.490266 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5"} err="failed to get container status \"671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5\": rpc error: code = NotFound desc = could not find container \"671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5\": container with ID starting with 671af75096a7449426c43885417b4e157f3b70be852f43bb1f667f58c04b81a5 not found: ID does not exist" Sep 18 16:20:04.519176 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.519141 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vps69\" (UniqueName: \"kubernetes.io/projected/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-kube-api-access-vps69\") pod \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " Sep 18 16:20:04.519372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.519212 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v29dz\" (UniqueName: \"kubernetes.io/projected/5b5423ba-ffca-4d60-904f-d6f36141e7d1-kube-api-access-v29dz\") pod \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " Sep 18 16:20:04.519372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.519248 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5423ba-ffca-4d60-904f-d6f36141e7d1-plugins-config-volume\") pod \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\" (UID: \"5b5423ba-ffca-4d60-904f-d6f36141e7d1\") " Sep 18 16:20:04.519372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.519268 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-plugins-config-volume\") pod \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\" (UID: \"7c9875c8-98bc-48b0-a4d9-ef5c6148f449\") " Sep 18 16:20:04.519686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.519659 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5423ba-ffca-4d60-904f-d6f36141e7d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b5423ba-ffca-4d60-904f-d6f36141e7d1" (UID: "5b5423ba-ffca-4d60-904f-d6f36141e7d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:20:04.519789 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.519686 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c9875c8-98bc-48b0-a4d9-ef5c6148f449" (UID: "7c9875c8-98bc-48b0-a4d9-ef5c6148f449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:20:04.522247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.522217 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5423ba-ffca-4d60-904f-d6f36141e7d1-kube-api-access-v29dz" (OuterVolumeSpecName: "kube-api-access-v29dz") pod "5b5423ba-ffca-4d60-904f-d6f36141e7d1" (UID: "5b5423ba-ffca-4d60-904f-d6f36141e7d1"). InnerVolumeSpecName "kube-api-access-v29dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:20:04.522701 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.522654 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-kube-api-access-vps69" (OuterVolumeSpecName: "kube-api-access-vps69") pod "7c9875c8-98bc-48b0-a4d9-ef5c6148f449" (UID: "7c9875c8-98bc-48b0-a4d9-ef5c6148f449"). InnerVolumeSpecName "kube-api-access-vps69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:20:04.620122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.620073 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vps69\" (UniqueName: \"kubernetes.io/projected/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-kube-api-access-vps69\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:04.620122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.620123 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29dz\" (UniqueName: \"kubernetes.io/projected/5b5423ba-ffca-4d60-904f-d6f36141e7d1-kube-api-access-v29dz\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:04.620358 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.620137 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5423ba-ffca-4d60-904f-d6f36141e7d1-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:04.620358 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.620150 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9875c8-98bc-48b0-a4d9-ef5c6148f449-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:04.620524 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.620467 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:04.801160 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.801103 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth"] Sep 18 16:20:04.804068 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.804042 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-75688a34-5659bd945f-xbnth"] Sep 18 16:20:04.822929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.822894 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f"] Sep 18 16:20:04.825423 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.825404 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-75688a34-768467cf6b-l2q5f"] Sep 18 16:20:04.990384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:04.990354 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:05.084823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:05.084736 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5423ba-ffca-4d60-904f-d6f36141e7d1" path="/var/lib/kubelet/pods/5b5423ba-ffca-4d60-904f-d6f36141e7d1/volumes" Sep 18 16:20:05.085316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:05.085294 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9875c8-98bc-48b0-a4d9-ef5c6148f449" path="/var/lib/kubelet/pods/7c9875c8-98bc-48b0-a4d9-ef5c6148f449/volumes" Sep 18 16:20:05.191460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:05.191429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:05.367459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:05.367369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:05.507751 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:05.507683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:05.735979 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:05.735891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:06.243220 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:06.243189 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:07.593350 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:07.593314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:08.436279 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:08.436227 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:20:08.436470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:08.436388 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:20:08.604599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:08.604569 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:08.691826 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:08.691743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:08.794570 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:08.794537 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:08.943203 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:08.943118 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:10.016336 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:10.016302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:10.156103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:10.156063 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:10.303332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:10.303249 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:11.379958 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:11.379911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:12.758822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:12.758787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:12.908337 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:12.908302 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:13.073408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:13.073321 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:13.779527 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:13.779493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:14.035775 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.035685 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:14.071361 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071323 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz"] Sep 18 16:20:14.071784 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071771 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5423ba-ffca-4d60-904f-d6f36141e7d1" containerName="payload-pre-processing" Sep 18 16:20:14.071839 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071786 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5423ba-ffca-4d60-904f-d6f36141e7d1" containerName="payload-pre-processing" Sep 18 16:20:14.071839 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071812 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9875c8-98bc-48b0-a4d9-ef5c6148f449" containerName="payload-processing" Sep 18 16:20:14.071839 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071820 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9875c8-98bc-48b0-a4d9-ef5c6148f449" containerName="payload-processing" Sep 18 16:20:14.071940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071903 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5423ba-ffca-4d60-904f-d6f36141e7d1" containerName="payload-pre-processing" Sep 18 16:20:14.071940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.071915 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9875c8-98bc-48b0-a4d9-ef5c6148f449" containerName="payload-processing" Sep 18 16:20:14.076579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.076561 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.080942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.080909 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-50cecd8f-dockercfg-82vs4\"" Sep 18 16:20:14.081051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.080951 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-50cecd8f\"" Sep 18 16:20:14.160718 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.160683 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll"] Sep 18 16:20:14.166610 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.166582 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.169320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.169289 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz"] Sep 18 16:20:14.187874 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.187847 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll"] Sep 18 16:20:14.216764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.216736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkrf\" (UniqueName: \"kubernetes.io/projected/49e94b4b-76b3-489e-b40f-e57e693c72dc-kube-api-access-mwkrf\") pod \"payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.216902 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.216794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7xd\" (UniqueName: \"kubernetes.io/projected/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-kube-api-access-5w7xd\") pod \"payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.216902 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.216864 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.216981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.216945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e94b4b-76b3-489e-b40f-e57e693c72dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.318160 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.318056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7xd\" (UniqueName: \"kubernetes.io/projected/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-kube-api-access-5w7xd\") pod \"payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.318160 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.318141 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.318341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.318209 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e94b4b-76b3-489e-b40f-e57e693c72dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.318341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.318271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkrf\" (UniqueName: \"kubernetes.io/projected/49e94b4b-76b3-489e-b40f-e57e693c72dc-kube-api-access-mwkrf\") pod \"payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.318868 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.318846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.318905 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.318863 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e94b4b-76b3-489e-b40f-e57e693c72dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.336568 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.336543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkrf\" (UniqueName: \"kubernetes.io/projected/49e94b4b-76b3-489e-b40f-e57e693c72dc-kube-api-access-mwkrf\") pod \"payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.353822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.353795 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7xd\" (UniqueName: \"kubernetes.io/projected/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-kube-api-access-5w7xd\") pod \"payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.387688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.387655 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:14.478645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.478615 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:14.624537 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.624447 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:14.690880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.690855 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz"] Sep 18 16:20:14.692692 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:20:14.692667 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d6deb2_5d63_4114_ba67_3fe0ab1747f0.slice/crio-d5b589a9e513354750b9e0d0447a4873ffd23af681a63f25af7fb5533d4a712c WatchSource:0}: Error finding container d5b589a9e513354750b9e0d0447a4873ffd23af681a63f25af7fb5533d4a712c: Status 404 returned error can't find the container with id d5b589a9e513354750b9e0d0447a4873ffd23af681a63f25af7fb5533d4a712c Sep 18 16:20:14.769796 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:14.769764 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll"] Sep 18 16:20:14.769907 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:20:14.769820 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e94b4b_76b3_489e_b40f_e57e693c72dc.slice/crio-c99f2d51a48241fff61c2f23efa5a3acaacde1e2905e9b4a8137863bc8ec1c45 WatchSource:0}: Error finding container c99f2d51a48241fff61c2f23efa5a3acaacde1e2905e9b4a8137863bc8ec1c45: Status 404 returned error can't find the container with id c99f2d51a48241fff61c2f23efa5a3acaacde1e2905e9b4a8137863bc8ec1c45 Sep 18 16:20:15.542798 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.542745 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" event={"ID":"49e94b4b-76b3-489e-b40f-e57e693c72dc","Type":"ContainerStarted","Data":"7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9"} Sep 18 16:20:15.543283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.542823 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:15.543283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.542848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" event={"ID":"49e94b4b-76b3-489e-b40f-e57e693c72dc","Type":"ContainerStarted","Data":"c99f2d51a48241fff61c2f23efa5a3acaacde1e2905e9b4a8137863bc8ec1c45"} Sep 18 16:20:15.544611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.544580 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" event={"ID":"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0","Type":"ContainerStarted","Data":"8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746"} Sep 18 16:20:15.544719 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.544637 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" event={"ID":"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0","Type":"ContainerStarted","Data":"d5b589a9e513354750b9e0d0447a4873ffd23af681a63f25af7fb5533d4a712c"} Sep 18 16:20:15.544719 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.544668 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:15.583308 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.583264 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" podStartSLOduration=1.583246284 podStartE2EDuration="1.583246284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:15.580206478 +0000 UTC m=+915.141051145" watchObservedRunningTime="2026-09-18 16:20:15.583246284 +0000 UTC m=+915.144090997" Sep 18 16:20:15.637530 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:15.637479 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" podStartSLOduration=1.63746371 podStartE2EDuration="1.63746371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:15.636851496 +0000 UTC m=+915.197696164" watchObservedRunningTime="2026-09-18 16:20:15.63746371 +0000 UTC m=+915.198308376" Sep 18 16:20:16.643643 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:16.643609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:16.772897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:16.772861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:17.888648 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:17.888613 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:19.273045 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:19.273013 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:19.562720 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:19.562644 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:19.801283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:19.801251 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:19.973544 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:19.973456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:20.214315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:20.214282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:20.683298 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:20.683262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:22.122867 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:22.122835 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:23.607562 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:23.607529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:25.370272 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:25.370245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:26.248561 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:26.248511 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:26.554306 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:26.554221 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:26.554306 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:26.554294 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:26.831311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:26.831238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:26.957304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:26.957271 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:27.197318 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:27.197233 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:28.438340 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:28.438308 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:29.604301 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:29.604270 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:31.073314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:31.073264 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:32.081890 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:32.081855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:33.059204 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:33.059175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:34.096073 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:34.096035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:35.470525 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:35.470491 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:36.702324 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:36.702288 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:37.862400 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:37.862369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:38.972330 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:38.972301 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:39.047693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.047663 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz"] Sep 18 16:20:39.047919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.047886 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" podUID="e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" containerName="payload-processing" containerID="cri-o://8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746" gracePeriod=30 Sep 18 16:20:39.095915 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.095877 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll"] Sep 18 16:20:39.096273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.096219 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" podUID="49e94b4b-76b3-489e-b40f-e57e693c72dc" containerName="payload-pre-processing" containerID="cri-o://7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9" gracePeriod=30 Sep 18 16:20:39.140412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.140382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:39.356139 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.356107 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:39.371999 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.371968 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:39.391116 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.391060 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:39.450215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.450174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e94b4b-76b3-489e-b40f-e57e693c72dc-plugins-config-volume\") pod \"49e94b4b-76b3-489e-b40f-e57e693c72dc\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " Sep 18 16:20:39.450438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.450229 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w7xd\" (UniqueName: \"kubernetes.io/projected/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-kube-api-access-5w7xd\") pod \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " Sep 18 16:20:39.450438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.450255 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-plugins-config-volume\") pod \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\" (UID: \"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0\") " Sep 18 16:20:39.450438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.450275 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwkrf\" (UniqueName: \"kubernetes.io/projected/49e94b4b-76b3-489e-b40f-e57e693c72dc-kube-api-access-mwkrf\") pod \"49e94b4b-76b3-489e-b40f-e57e693c72dc\" (UID: \"49e94b4b-76b3-489e-b40f-e57e693c72dc\") " Sep 18 16:20:39.450761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.450703 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e94b4b-76b3-489e-b40f-e57e693c72dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49e94b4b-76b3-489e-b40f-e57e693c72dc" (UID: "49e94b4b-76b3-489e-b40f-e57e693c72dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:20:39.451108 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.451056 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" (UID: "e7d6deb2-5d63-4114-ba67-3fe0ab1747f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:20:39.452947 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.452917 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-kube-api-access-5w7xd" (OuterVolumeSpecName: "kube-api-access-5w7xd") pod "e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" (UID: "e7d6deb2-5d63-4114-ba67-3fe0ab1747f0"). InnerVolumeSpecName "kube-api-access-5w7xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:20:39.453143 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.453120 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e94b4b-76b3-489e-b40f-e57e693c72dc-kube-api-access-mwkrf" (OuterVolumeSpecName: "kube-api-access-mwkrf") pod "49e94b4b-76b3-489e-b40f-e57e693c72dc" (UID: "49e94b4b-76b3-489e-b40f-e57e693c72dc"). InnerVolumeSpecName "kube-api-access-mwkrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:20:39.543739 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.543664 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:39.551026 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.550985 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49e94b4b-76b3-489e-b40f-e57e693c72dc-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:39.551026 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.551024 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7xd\" (UniqueName: \"kubernetes.io/projected/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-kube-api-access-5w7xd\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:39.551026 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.551034 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:39.551258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.551044 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwkrf\" (UniqueName: \"kubernetes.io/projected/49e94b4b-76b3-489e-b40f-e57e693c72dc-kube-api-access-mwkrf\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:20:39.696651 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.696620 2563 generic.go:356] "Generic (PLEG): container finished" podID="e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" containerID="8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746" exitCode=0 Sep 18 16:20:39.696842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.696665 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" Sep 18 16:20:39.696842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.696701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" event={"ID":"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0","Type":"ContainerDied","Data":"8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746"} Sep 18 16:20:39.696842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.696774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz" event={"ID":"e7d6deb2-5d63-4114-ba67-3fe0ab1747f0","Type":"ContainerDied","Data":"d5b589a9e513354750b9e0d0447a4873ffd23af681a63f25af7fb5533d4a712c"} Sep 18 16:20:39.696842 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.696805 2563 scope.go:117] "RemoveContainer" containerID="8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746" Sep 18 16:20:39.698489 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.698466 2563 generic.go:356] "Generic (PLEG): container finished" podID="49e94b4b-76b3-489e-b40f-e57e693c72dc" containerID="7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9" exitCode=0 Sep 18 16:20:39.698616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.698503 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" event={"ID":"49e94b4b-76b3-489e-b40f-e57e693c72dc","Type":"ContainerDied","Data":"7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9"} Sep 18 16:20:39.698616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.698514 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" Sep 18 16:20:39.698616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.698544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll" event={"ID":"49e94b4b-76b3-489e-b40f-e57e693c72dc","Type":"ContainerDied","Data":"c99f2d51a48241fff61c2f23efa5a3acaacde1e2905e9b4a8137863bc8ec1c45"} Sep 18 16:20:39.712007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.710652 2563 scope.go:117] "RemoveContainer" containerID="8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746" Sep 18 16:20:39.713028 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:20:39.713006 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746\": container with ID starting with 8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746 not found: ID does not exist" containerID="8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746" Sep 18 16:20:39.713153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.713036 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746"} err="failed to get container status \"8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746\": rpc error: code = NotFound desc = could not find container \"8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746\": container with ID starting with 8d7880d8d9fde5f21643d916540c3578f8a9aa7beacf881ab3b0867414834746 not found: ID does not exist" Sep 18 16:20:39.713153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.713055 2563 scope.go:117] "RemoveContainer" containerID="7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9" Sep 18 16:20:39.722497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.722479 2563 scope.go:117] "RemoveContainer" containerID="7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9" Sep 18 16:20:39.722940 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:20:39.722883 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9\": container with ID starting with 7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9 not found: ID does not exist" containerID="7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9" Sep 18 16:20:39.723148 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.722944 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9"} err="failed to get container status \"7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9\": rpc error: code = NotFound desc = could not find container \"7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9\": container with ID starting with 7bfc8e4779d9d36921a1a100a147cb2396cadc5c2fdd4f196987e7c08cffd9c9 not found: ID does not exist" Sep 18 16:20:39.736257 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.736212 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:39.746035 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.746009 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz"] Sep 18 16:20:39.752096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.752061 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50cecd8f-777b899db6-fx8nz"] Sep 18 16:20:39.778353 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.778326 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll"] Sep 18 16:20:39.783452 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.783431 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50cecd8f-7fc88669c8-wxgll"] Sep 18 16:20:39.859930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:39.859845 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:40.287208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:40.287176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:40.532271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:40.532243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:40.837236 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:40.837201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:41.081233 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:41.081204 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49e94b4b-76b3-489e-b40f-e57e693c72dc" path="/var/lib/kubelet/pods/49e94b4b-76b3-489e-b40f-e57e693c72dc/volumes" Sep 18 16:20:41.081571 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:41.081557 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" path="/var/lib/kubelet/pods/e7d6deb2-5d63-4114-ba67-3fe0ab1747f0/volumes" Sep 18 16:20:41.100072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:41.100044 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:42.132258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:42.132228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:43.687056 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:43.687020 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:43.856921 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:43.856891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:45.956351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:45.956320 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:47.331945 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:47.331913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:48.889449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:48.889417 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:50.305403 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:50.305347 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:50.698250 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:50.698161 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:50.905894 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:50.905863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:51.232951 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:51.232922 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:51.850910 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:51.850882 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:52.149861 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:52.149782 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:52.545291 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:52.545262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:54.062228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:54.062191 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:55.334541 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:55.334505 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:56.903536 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:56.903504 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:20:59.942288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:20:59.942257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:12.500820 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:12.500782 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:14.633379 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:14.633292 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:15.779664 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:15.779632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:16.832412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:16.832372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:18.143720 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:18.143681 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:19.430120 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:19.430065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:20.844202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:20.844163 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:22.396255 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:22.396223 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:23.694680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:23.694652 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:25.051572 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:25.051542 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:26.398944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:26.398911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:27.608351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:27.608316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:33.999656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:33.999624 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:36.228686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:36.228651 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:36.510924 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:36.510895 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:36.681916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:36.681886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:36.911579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:36.911502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:37.296358 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:37.296329 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:37.450894 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:37.450861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:38.012667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:38.012635 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:39.439914 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:39.439879 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:39.978261 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:39.978227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:40.207809 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:40.207778 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:41.276135 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:41.276069 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:42.734780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:42.734748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:44.033777 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:44.033743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:45.311371 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:45.311341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:45.416585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:45.416552 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:45.541342 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:45.541312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:46.153069 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:46.153039 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:46.275498 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:46.275453 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:46.716352 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:46.716325 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:48.065317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:48.065284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:48.960024 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:48.959989 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:49.553286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:49.553257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:50.612774 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:50.612738 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:50.730822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:50.730789 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48"] Sep 18 16:21:50.731019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:50.730993 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" podUID="1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" containerName="payload-pre-processing" containerID="cri-o://c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5" gracePeriod=30 Sep 18 16:21:50.783313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:50.783281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:50.916320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:50.916290 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:50.983346 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:50.983322 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:21:51.047209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.047176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:51.087454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.087423 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-plugins-config-volume\") pod \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " Sep 18 16:21:51.087617 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.087466 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5db67\" (UniqueName: \"kubernetes.io/projected/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-kube-api-access-5db67\") pod \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\" (UID: \"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436\") " Sep 18 16:21:51.087795 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.087772 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" (UID: "1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:21:51.089749 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.089724 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-kube-api-access-5db67" (OuterVolumeSpecName: "kube-api-access-5db67") pod "1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" (UID: "1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436"). InnerVolumeSpecName "kube-api-access-5db67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:21:51.114376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.114344 2563 generic.go:356] "Generic (PLEG): container finished" podID="1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" containerID="c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5" exitCode=0 Sep 18 16:21:51.114531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.114389 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" Sep 18 16:21:51.114531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.114424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" event={"ID":"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436","Type":"ContainerDied","Data":"c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5"} Sep 18 16:21:51.114628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.114546 2563 scope.go:117] "RemoveContainer" containerID="c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5" Sep 18 16:21:51.116291 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.114477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48" event={"ID":"1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436","Type":"ContainerDied","Data":"83e515933bba56e7fc9b970eb977240007cfe6ec9f61a5b3a5520a1aa9ee0b61"} Sep 18 16:21:51.131981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.131957 2563 scope.go:117] "RemoveContainer" containerID="c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5" Sep 18 16:21:51.132306 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:21:51.132284 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5\": container with ID starting with c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5 not found: ID does not exist" containerID="c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5" Sep 18 16:21:51.132389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.132314 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5"} err="failed to get container status \"c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5\": rpc error: code = NotFound desc = could not find container \"c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5\": container with ID starting with c0639651ddeecd33c5e8cd9e160bf54fddc51fa03757ae80a57c44ccd1b4a7e5 not found: ID does not exist" Sep 18 16:21:51.154497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.154417 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48"] Sep 18 16:21:51.157235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.157208 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eef18d-86c9598f4-sww48"] Sep 18 16:21:51.186626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.186585 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:51.188803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.188775 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:21:51.188803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.188802 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5db67\" (UniqueName: \"kubernetes.io/projected/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436-kube-api-access-5db67\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:21:51.542582 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:51.542548 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:52.930435 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:52.930405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:53.080918 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:53.080881 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" path="/var/lib/kubelet/pods/1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436/volumes" Sep 18 16:21:54.136545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:54.136512 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:55.170096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:55.170062 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:56.272614 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:56.272581 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:56.560764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:56.560682 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:57.413031 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:57.412996 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:21:58.590358 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:21:58.590321 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:00.611968 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:00.611936 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:02.083378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:02.083340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:03.228754 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:03.228717 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:04.393282 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:04.393252 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:05.823177 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:05.823145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:06.880451 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:06.880418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:08.314977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:08.314943 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:09.598742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:09.598710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:10.900018 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:10.899985 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:12.200449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:12.200417 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:13.293346 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:13.293314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:13.436341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:13.436310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:13.558487 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:13.558394 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:13.692693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:13.692657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:13.904454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:13.904378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:14.377359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:14.377326 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:14.803380 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:14.803352 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:14.947573 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:14.947546 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:16.192422 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:16.192391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:16.364078 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:16.364044 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:16.518319 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:16.518293 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:17.777734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:17.777706 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:18.382833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:18.382797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:19.110932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:19.110875 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:20.475742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:20.475708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:21.010288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:21.010259 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:23.807076 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:23.807045 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:24.037356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:24.037321 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:24.227658 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:24.227567 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:24.684600 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:24.684570 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:27.522464 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:27.522423 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:27.632977 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:27.632944 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:27.829426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:27.829348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:28.326323 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:28.326291 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:28.725356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:28.725281 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:29.024598 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:29.024567 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:29.739286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:29.739253 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:34.538443 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:34.538406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:35.146282 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:35.146248 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:35.363992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:35.363962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:35.955642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:35.955609 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:37.489449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:37.489414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:38.863367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:38.863327 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:40.294386 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:40.294357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:41.473737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:41.473704 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:41.588300 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:41.588274 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:42.296487 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:42.296456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:42.491553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:42.491523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:50.535341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:50.535263 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:57.914888 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:57.914852 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:58.104690 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:58.104659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:22:58.277948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:22:58.277919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:04.466689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:04.466654 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:05.123405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:05.123369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:05.376180 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:05.376099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:06.285652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:06.285612 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:06.857304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:06.857277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:07.293267 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:07.293234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:08.615237 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:08.615207 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:09.736844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:09.736796 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:14.066404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:14.066365 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:20.378986 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:20.378952 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:20.506424 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:20.506391 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:20.621941 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:20.621909 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:25.965258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:25.965225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:31.127606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:31.127571 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:31.229093 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:31.229051 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:31.948443 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:31.948406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:38.485051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:38.485019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:38.598484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:38.598448 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:38.702312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:38.702285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:38.801914 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:38.801889 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:39.519126 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:39.519077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:39.635668 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:39.635636 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:41.190688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:41.190649 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:48.445713 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:48.445668 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:48.558146 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:48.558074 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:49.122234 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:49.122200 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:51.639576 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:51.639549 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:56.080765 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:56.080726 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:56.215817 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:56.215777 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:56.322825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:56.322788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:56.736376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:56.736346 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:23:56.841573 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:23:56.841542 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:04.493902 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:04.493856 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:05.161050 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:05.161017 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:05.266943 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:05.266913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:05.583248 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:05.583211 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:06.533667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:06.533633 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:07.498610 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:07.498576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:08.582351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:08.582319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:09.577453 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:09.577421 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:10.578902 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.578873 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk"] Sep 18 16:24:10.579406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579352 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" containerName="payload-processing" Sep 18 16:24:10.579406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579367 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" containerName="payload-processing" Sep 18 16:24:10.579406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579374 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49e94b4b-76b3-489e-b40f-e57e693c72dc" containerName="payload-pre-processing" Sep 18 16:24:10.579406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579379 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49e94b4b-76b3-489e-b40f-e57e693c72dc" containerName="payload-pre-processing" Sep 18 16:24:10.579406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579393 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" containerName="payload-pre-processing" Sep 18 16:24:10.579406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579403 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" containerName="payload-pre-processing" Sep 18 16:24:10.579617 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579469 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49e94b4b-76b3-489e-b40f-e57e693c72dc" containerName="payload-pre-processing" Sep 18 16:24:10.579617 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579483 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acb4e39-21a7-4cbc-ad37-fc7c5e5d8436" containerName="payload-pre-processing" Sep 18 16:24:10.579617 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.579493 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d6deb2-5d63-4114-ba67-3fe0ab1747f0" containerName="payload-processing" Sep 18 16:24:10.582744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.582725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.586004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.585985 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-eab152b6-serving-cert\"" Sep 18 16:24:10.600834 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.600806 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk"] Sep 18 16:24:10.639143 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.639110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f771c7de-ffac-4f39-8bc5-81ecd769310d-openshift-service-ca\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.639273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.639154 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmxq\" (UniqueName: \"kubernetes.io/projected/f771c7de-ffac-4f39-8bc5-81ecd769310d-kube-api-access-cqmxq\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.639273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.639256 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-metrics-certs\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.639348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.639285 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.677563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.677536 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:10.740227 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.740194 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f771c7de-ffac-4f39-8bc5-81ecd769310d-openshift-service-ca\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.740389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.740239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmxq\" (UniqueName: \"kubernetes.io/projected/f771c7de-ffac-4f39-8bc5-81ecd769310d-kube-api-access-cqmxq\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.740389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.740289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-metrics-certs\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.740389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.740309 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.740500 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:24:10.740423 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-serving-cert: secret "maas-api-e2e-mt-eab152b6-serving-cert" not found Sep 18 16:24:10.740500 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:24:10.740480 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls podName:f771c7de-ffac-4f39-8bc5-81ecd769310d nodeName:}" failed. No retries permitted until 2026-09-18 16:24:11.24046114 +0000 UTC m=+1150.801305808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls") pod "maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" (UID: "f771c7de-ffac-4f39-8bc5-81ecd769310d") : secret "maas-api-e2e-mt-eab152b6-serving-cert" not found Sep 18 16:24:10.740921 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.740901 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f771c7de-ffac-4f39-8bc5-81ecd769310d-openshift-service-ca\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.742688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.742670 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-metrics-certs\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.758004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.757980 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmxq\" (UniqueName: \"kubernetes.io/projected/f771c7de-ffac-4f39-8bc5-81ecd769310d-kube-api-access-cqmxq\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:10.776435 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.776410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:10.881509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:10.881429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:11.244926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.244849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:11.247574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.247552 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls\") pod \"maas-api-e2e-mt-eab152b6-679994dbc5-wqldk\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:11.259239 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.259201 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:11.363463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.363429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:11.388935 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.388899 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4"] Sep 18 16:24:11.392675 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.392647 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.398055 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.398019 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eab152b6\"" Sep 18 16:24:11.417306 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.416365 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4"] Sep 18 16:24:11.446940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.446909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz4rp\" (UniqueName: \"kubernetes.io/projected/0d34147e-56f1-4e35-9a50-7c329b2c495f-kube-api-access-dz4rp\") pod \"payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.447105 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.446994 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34147e-56f1-4e35-9a50-7c329b2c495f-plugins-config-volume\") pod \"payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.456825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.456797 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq"] Sep 18 16:24:11.460389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.460371 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.463599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.463582 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eab152b6-dockercfg-h7dl5\"" Sep 18 16:24:11.474831 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.474806 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq"] Sep 18 16:24:11.494923 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.494858 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:11.548063 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.548033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2289f90d-735f-4b5b-826f-64a0eeba573f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.548251 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.548132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34147e-56f1-4e35-9a50-7c329b2c495f-plugins-config-volume\") pod \"payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.548301 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.548261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdcpk\" (UniqueName: \"kubernetes.io/projected/2289f90d-735f-4b5b-826f-64a0eeba573f-kube-api-access-qdcpk\") pod \"payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.548342 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.548317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz4rp\" (UniqueName: \"kubernetes.io/projected/0d34147e-56f1-4e35-9a50-7c329b2c495f-kube-api-access-dz4rp\") pod \"payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.548840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.548814 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34147e-56f1-4e35-9a50-7c329b2c495f-plugins-config-volume\") pod \"payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.559732 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.559710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz4rp\" (UniqueName: \"kubernetes.io/projected/0d34147e-56f1-4e35-9a50-7c329b2c495f-kube-api-access-dz4rp\") pod \"payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.649225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.649192 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdcpk\" (UniqueName: \"kubernetes.io/projected/2289f90d-735f-4b5b-826f-64a0eeba573f-kube-api-access-qdcpk\") pod \"payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.649654 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.649344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2289f90d-735f-4b5b-826f-64a0eeba573f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.650025 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.650004 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2289f90d-735f-4b5b-826f-64a0eeba573f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.661437 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.661412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdcpk\" (UniqueName: \"kubernetes.io/projected/2289f90d-735f-4b5b-826f-64a0eeba573f-kube-api-access-qdcpk\") pod \"payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.703388 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.703351 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:11.770780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.770748 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:11.796811 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:24:11.796780 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf771c7de_ffac_4f39_8bc5_81ecd769310d.slice/crio-ba006b1b6ccc3f8b42ba8b94e29f47527b856b3f5bde5a88c8a5e44102d3f768 WatchSource:0}: Error finding container ba006b1b6ccc3f8b42ba8b94e29f47527b856b3f5bde5a88c8a5e44102d3f768: Status 404 returned error can't find the container with id ba006b1b6ccc3f8b42ba8b94e29f47527b856b3f5bde5a88c8a5e44102d3f768 Sep 18 16:24:11.796948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.796919 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk"] Sep 18 16:24:11.798546 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.798528 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:24:11.923470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.923435 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" event={"ID":"f771c7de-ffac-4f39-8bc5-81ecd769310d","Type":"ContainerStarted","Data":"ba006b1b6ccc3f8b42ba8b94e29f47527b856b3f5bde5a88c8a5e44102d3f768"} Sep 18 16:24:11.988183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:11.988158 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4"] Sep 18 16:24:11.989440 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:24:11.989413 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d34147e_56f1_4e35_9a50_7c329b2c495f.slice/crio-c05ccb59c983230d1632d6f05b705e57da99dfe7fb379cd58fc86a56ffc092e3 WatchSource:0}: Error finding container c05ccb59c983230d1632d6f05b705e57da99dfe7fb379cd58fc86a56ffc092e3: Status 404 returned error can't find the container with id c05ccb59c983230d1632d6f05b705e57da99dfe7fb379cd58fc86a56ffc092e3 Sep 18 16:24:12.054074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.053692 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq"] Sep 18 16:24:12.056306 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:24:12.056276 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2289f90d_735f_4b5b_826f_64a0eeba573f.slice/crio-48aa89177dc02ff62ad0ec822cbaba03628e891a10780f47c41dbf2472759e8a WatchSource:0}: Error finding container 48aa89177dc02ff62ad0ec822cbaba03628e891a10780f47c41dbf2472759e8a: Status 404 returned error can't find the container with id 48aa89177dc02ff62ad0ec822cbaba03628e891a10780f47c41dbf2472759e8a Sep 18 16:24:12.931755 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.931716 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" event={"ID":"0d34147e-56f1-4e35-9a50-7c329b2c495f","Type":"ContainerStarted","Data":"c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a"} Sep 18 16:24:12.932287 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.931793 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" event={"ID":"0d34147e-56f1-4e35-9a50-7c329b2c495f","Type":"ContainerStarted","Data":"c05ccb59c983230d1632d6f05b705e57da99dfe7fb379cd58fc86a56ffc092e3"} Sep 18 16:24:12.932287 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.931941 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:12.933493 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.933465 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" event={"ID":"f771c7de-ffac-4f39-8bc5-81ecd769310d","Type":"ContainerStarted","Data":"62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521"} Sep 18 16:24:12.933597 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.933514 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:12.935012 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.934993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" event={"ID":"2289f90d-735f-4b5b-826f-64a0eeba573f","Type":"ContainerStarted","Data":"1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8"} Sep 18 16:24:12.935097 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.935026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" event={"ID":"2289f90d-735f-4b5b-826f-64a0eeba573f","Type":"ContainerStarted","Data":"48aa89177dc02ff62ad0ec822cbaba03628e891a10780f47c41dbf2472759e8a"} Sep 18 16:24:12.935148 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.935109 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:12.951498 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.951433 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" podStartSLOduration=1.951417476 podStartE2EDuration="1.951417476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:12.950976158 +0000 UTC m=+1152.511820817" watchObservedRunningTime="2026-09-18 16:24:12.951417476 +0000 UTC m=+1152.512262144" Sep 18 16:24:12.975184 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.975144 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" podStartSLOduration=2.41368693 podStartE2EDuration="2.975127712s" totalImagesPullingTime="561.440782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:12.972372756 +0000 UTC m=+1152.533217423" watchObservedRunningTime="2026-09-18 16:24:12.975127712 +0000 UTC m=+1152.535972380" Sep 18 16:24:12.995104 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:12.995048 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" podStartSLOduration=1.99503692 podStartE2EDuration="1.99503692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:12.992817696 +0000 UTC m=+1152.553662387" watchObservedRunningTime="2026-09-18 16:24:12.99503692 +0000 UTC m=+1152.555881586" Sep 18 16:24:18.945978 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:18.945947 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:23.943204 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:23.943168 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:23.943628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:23.943256 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:26.310410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:26.310377 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:26.460403 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:26.460375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:27.404377 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:27.404341 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:28.338980 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:28.338948 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:29.392942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:29.392904 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:30.408788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:30.408759 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:31.507267 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:31.507227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:32.642659 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:32.642621 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:33.432141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:33.432105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:33.746308 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:33.746216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:34.965423 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:34.965392 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:35.949797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:35.949763 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:36.107406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:36.107377 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:36.566304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:36.566272 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:36.673576 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:36.673541 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:37.010034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:37.010002 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:37.259803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:37.259759 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:38.447398 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:38.447362 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:39.695848 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:39.695809 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:40.566271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:40.566235 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:41.467875 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:41.467846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:41.570334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:41.570306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:41.693351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:41.693314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:41.818578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:41.818543 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:42.124337 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:42.124253 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:43.099807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:43.099769 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:44.043515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:44.043480 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:44.919950 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:44.919913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:45.025141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:45.025110 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:45.132837 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:45.132807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:46.194054 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:46.194024 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:47.168239 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:47.168211 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:47.254667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:47.254634 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:47.590213 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:47.590178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:48.136667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:48.136640 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:49.072229 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:49.072197 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:49.945313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:49.945283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:50.902778 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:50.902718 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:53.828291 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:53.828257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:54.103692 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:54.103612 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:54.545234 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:54.545204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:54.852343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:54.852243 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:54.950682 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:54.950636 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:55.134443 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.134357 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:55.686296 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.686264 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk"] Sep 18 16:24:55.686505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.686479 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" podUID="f771c7de-ffac-4f39-8bc5-81ecd769310d" containerName="maas-api" containerID="cri-o://62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521" gracePeriod=30 Sep 18 16:24:55.736386 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.736358 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:55.772998 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.772966 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4"] Sep 18 16:24:55.773222 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.773188 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" podUID="0d34147e-56f1-4e35-9a50-7c329b2c495f" containerName="payload-processing" containerID="cri-o://c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a" gracePeriod=30 Sep 18 16:24:55.783588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.783559 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq"] Sep 18 16:24:55.783903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.783859 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" podUID="2289f90d-735f-4b5b-826f-64a0eeba573f" containerName="payload-pre-processing" containerID="cri-o://1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8" gracePeriod=30 Sep 18 16:24:55.864270 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.864238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:55.971162 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:55.971069 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:56.042278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.042247 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:56.066012 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.065982 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-metrics-certs\") pod \"f771c7de-ffac-4f39-8bc5-81ecd769310d\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " Sep 18 16:24:56.066200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.066039 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f771c7de-ffac-4f39-8bc5-81ecd769310d-openshift-service-ca\") pod \"f771c7de-ffac-4f39-8bc5-81ecd769310d\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " Sep 18 16:24:56.066200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.066115 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls\") pod \"f771c7de-ffac-4f39-8bc5-81ecd769310d\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " Sep 18 16:24:56.066327 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.066216 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqmxq\" (UniqueName: \"kubernetes.io/projected/f771c7de-ffac-4f39-8bc5-81ecd769310d-kube-api-access-cqmxq\") pod \"f771c7de-ffac-4f39-8bc5-81ecd769310d\" (UID: \"f771c7de-ffac-4f39-8bc5-81ecd769310d\") " Sep 18 16:24:56.066485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.066437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f771c7de-ffac-4f39-8bc5-81ecd769310d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f771c7de-ffac-4f39-8bc5-81ecd769310d" (UID: "f771c7de-ffac-4f39-8bc5-81ecd769310d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:24:56.067738 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.067710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:56.068559 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.068532 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f771c7de-ffac-4f39-8bc5-81ecd769310d" (UID: "f771c7de-ffac-4f39-8bc5-81ecd769310d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:24:56.068655 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.068637 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f771c7de-ffac-4f39-8bc5-81ecd769310d" (UID: "f771c7de-ffac-4f39-8bc5-81ecd769310d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:24:56.068727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.068693 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f771c7de-ffac-4f39-8bc5-81ecd769310d-kube-api-access-cqmxq" (OuterVolumeSpecName: "kube-api-access-cqmxq") pod "f771c7de-ffac-4f39-8bc5-81ecd769310d" (UID: "f771c7de-ffac-4f39-8bc5-81ecd769310d"). InnerVolumeSpecName "kube-api-access-cqmxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:24:56.078994 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.078970 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:56.086432 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.086412 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:56.166878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.166845 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz4rp\" (UniqueName: \"kubernetes.io/projected/0d34147e-56f1-4e35-9a50-7c329b2c495f-kube-api-access-dz4rp\") pod \"0d34147e-56f1-4e35-9a50-7c329b2c495f\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " Sep 18 16:24:56.167070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.166918 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2289f90d-735f-4b5b-826f-64a0eeba573f-plugins-config-volume\") pod \"2289f90d-735f-4b5b-826f-64a0eeba573f\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " Sep 18 16:24:56.167070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.166984 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34147e-56f1-4e35-9a50-7c329b2c495f-plugins-config-volume\") pod \"0d34147e-56f1-4e35-9a50-7c329b2c495f\" (UID: \"0d34147e-56f1-4e35-9a50-7c329b2c495f\") " Sep 18 16:24:56.167070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167028 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdcpk\" (UniqueName: \"kubernetes.io/projected/2289f90d-735f-4b5b-826f-64a0eeba573f-kube-api-access-qdcpk\") pod \"2289f90d-735f-4b5b-826f-64a0eeba573f\" (UID: \"2289f90d-735f-4b5b-826f-64a0eeba573f\") " Sep 18 16:24:56.167446 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167402 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-maas-api-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.167446 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167428 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmxq\" (UniqueName: \"kubernetes.io/projected/f771c7de-ffac-4f39-8bc5-81ecd769310d-kube-api-access-cqmxq\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.167446 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167441 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f771c7de-ffac-4f39-8bc5-81ecd769310d-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.167635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167454 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f771c7de-ffac-4f39-8bc5-81ecd769310d-openshift-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.167635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167398 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2289f90d-735f-4b5b-826f-64a0eeba573f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2289f90d-735f-4b5b-826f-64a0eeba573f" (UID: "2289f90d-735f-4b5b-826f-64a0eeba573f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:24:56.167635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.167428 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d34147e-56f1-4e35-9a50-7c329b2c495f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d34147e-56f1-4e35-9a50-7c329b2c495f" (UID: "0d34147e-56f1-4e35-9a50-7c329b2c495f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:24:56.169669 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.169644 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d34147e-56f1-4e35-9a50-7c329b2c495f-kube-api-access-dz4rp" (OuterVolumeSpecName: "kube-api-access-dz4rp") pod "0d34147e-56f1-4e35-9a50-7c329b2c495f" (UID: "0d34147e-56f1-4e35-9a50-7c329b2c495f"). InnerVolumeSpecName "kube-api-access-dz4rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:24:56.169780 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.169758 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2289f90d-735f-4b5b-826f-64a0eeba573f-kube-api-access-qdcpk" (OuterVolumeSpecName: "kube-api-access-qdcpk") pod "2289f90d-735f-4b5b-826f-64a0eeba573f" (UID: "2289f90d-735f-4b5b-826f-64a0eeba573f"). InnerVolumeSpecName "kube-api-access-qdcpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:24:56.210118 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.210010 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" Sep 18 16:24:56.210118 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.210038 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" event={"ID":"0d34147e-56f1-4e35-9a50-7c329b2c495f","Type":"ContainerDied","Data":"c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a"} Sep 18 16:24:56.210118 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.209960 2563 generic.go:356] "Generic (PLEG): container finished" podID="0d34147e-56f1-4e35-9a50-7c329b2c495f" containerID="c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a" exitCode=0 Sep 18 16:24:56.210423 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.210122 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4" event={"ID":"0d34147e-56f1-4e35-9a50-7c329b2c495f","Type":"ContainerDied","Data":"c05ccb59c983230d1632d6f05b705e57da99dfe7fb379cd58fc86a56ffc092e3"} Sep 18 16:24:56.210423 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.210152 2563 scope.go:117] "RemoveContainer" containerID="c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a" Sep 18 16:24:56.212431 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.212392 2563 generic.go:356] "Generic (PLEG): container finished" podID="f771c7de-ffac-4f39-8bc5-81ecd769310d" containerID="62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521" exitCode=0 Sep 18 16:24:56.212566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.212546 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" event={"ID":"f771c7de-ffac-4f39-8bc5-81ecd769310d","Type":"ContainerDied","Data":"62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521"} Sep 18 16:24:56.212642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.212631 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" event={"ID":"f771c7de-ffac-4f39-8bc5-81ecd769310d","Type":"ContainerDied","Data":"ba006b1b6ccc3f8b42ba8b94e29f47527b856b3f5bde5a88c8a5e44102d3f768"} Sep 18 16:24:56.214225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.214193 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk" Sep 18 16:24:56.226152 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.226023 2563 generic.go:356] "Generic (PLEG): container finished" podID="2289f90d-735f-4b5b-826f-64a0eeba573f" containerID="1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8" exitCode=0 Sep 18 16:24:56.226567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.226329 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" event={"ID":"2289f90d-735f-4b5b-826f-64a0eeba573f","Type":"ContainerDied","Data":"1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8"} Sep 18 16:24:56.226688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.226612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" event={"ID":"2289f90d-735f-4b5b-826f-64a0eeba573f","Type":"ContainerDied","Data":"48aa89177dc02ff62ad0ec822cbaba03628e891a10780f47c41dbf2472759e8a"} Sep 18 16:24:56.226688 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.226418 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq" Sep 18 16:24:56.229679 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.229659 2563 scope.go:117] "RemoveContainer" containerID="c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a" Sep 18 16:24:56.229985 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:24:56.229962 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a\": container with ID starting with c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a not found: ID does not exist" containerID="c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a" Sep 18 16:24:56.230058 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.229993 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a"} err="failed to get container status \"c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a\": rpc error: code = NotFound desc = could not find container \"c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a\": container with ID starting with c7b59bd723b056817879368fa141432c9351019fbe019e5ea7f81c09fff7fa6a not found: ID does not exist" Sep 18 16:24:56.230058 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.230009 2563 scope.go:117] "RemoveContainer" containerID="62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521" Sep 18 16:24:56.240814 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.240792 2563 scope.go:117] "RemoveContainer" containerID="62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521" Sep 18 16:24:56.241167 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:24:56.241138 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521\": container with ID starting with 62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521 not found: ID does not exist" containerID="62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521" Sep 18 16:24:56.241247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.241170 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521"} err="failed to get container status \"62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521\": rpc error: code = NotFound desc = could not find container \"62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521\": container with ID starting with 62947e3e859e3fd6043f953c670930a78268ba5c6acd0e82e21f2c5f45da5521 not found: ID does not exist" Sep 18 16:24:56.241247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.241197 2563 scope.go:117] "RemoveContainer" containerID="1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8" Sep 18 16:24:56.242595 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.242574 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4"] Sep 18 16:24:56.247983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.247956 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-eab152b6-d866898cc-fxsf4"] Sep 18 16:24:56.251713 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.251683 2563 scope.go:117] "RemoveContainer" containerID="1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8" Sep 18 16:24:56.252031 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:24:56.252009 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8\": container with ID starting with 1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8 not found: ID does not exist" containerID="1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8" Sep 18 16:24:56.252127 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.252037 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8"} err="failed to get container status \"1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8\": rpc error: code = NotFound desc = could not find container \"1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8\": container with ID starting with 1051d5860d7bf0344d4aaaa35381a94a34e6803d0fb25ab9dbf90f92265474a8 not found: ID does not exist" Sep 18 16:24:56.259883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.259858 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk"] Sep 18 16:24:56.266995 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.266972 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eab152b6-679994dbc5-wqldk"] Sep 18 16:24:56.268325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.268307 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d34147e-56f1-4e35-9a50-7c329b2c495f-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.268397 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.268329 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdcpk\" (UniqueName: \"kubernetes.io/projected/2289f90d-735f-4b5b-826f-64a0eeba573f-kube-api-access-qdcpk\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.268397 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.268339 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz4rp\" (UniqueName: \"kubernetes.io/projected/0d34147e-56f1-4e35-9a50-7c329b2c495f-kube-api-access-dz4rp\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.268397 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.268347 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2289f90d-735f-4b5b-826f-64a0eeba573f-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:24:56.278877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.278850 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq"] Sep 18 16:24:56.286919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.286893 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eab152b6-6d85b7b7db-jnlgq"] Sep 18 16:24:56.371760 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.371716 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:56.463827 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.463734 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:56.562506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.562458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:56.696227 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:56.696194 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:57.083238 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:57.083201 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d34147e-56f1-4e35-9a50-7c329b2c495f" path="/var/lib/kubelet/pods/0d34147e-56f1-4e35-9a50-7c329b2c495f/volumes" Sep 18 16:24:57.083673 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:57.083581 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2289f90d-735f-4b5b-826f-64a0eeba573f" path="/var/lib/kubelet/pods/2289f90d-735f-4b5b-826f-64a0eeba573f/volumes" Sep 18 16:24:57.083933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:57.083919 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f771c7de-ffac-4f39-8bc5-81ecd769310d" path="/var/lib/kubelet/pods/f771c7de-ffac-4f39-8bc5-81ecd769310d/volumes" Sep 18 16:24:57.924740 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:57.924706 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:58.852516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:58.852483 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:24:59.839594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:24:59.839559 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:00.784493 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:00.784462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:03.085060 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.085030 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:25:03.086767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.086745 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:25:03.192970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.192944 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:03.308406 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.308373 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:03.497404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.497331 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:03.805021 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.804971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:03.963201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:03.963175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:04.245450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:04.245361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:11.424018 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:11.423981 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:11.544494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:11.544466 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:11.673001 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:11.672971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:11.849201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:11.849170 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:15.044362 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:15.044318 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:18.501676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:18.501643 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:18.602716 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:18.602686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:19.388665 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:19.388635 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:19.816241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:19.816210 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:20.058361 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:20.058331 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:22.773314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:22.773277 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:22.990334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:22.990299 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:23.124483 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:23.124408 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:23.826269 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:23.826232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:24.873689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:24.873652 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:25.874253 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:25.874216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:26.958178 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:26.958061 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:28.060506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.060477 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:28.192856 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.192823 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:28.617211 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.617176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:28.720024 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.719985 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:28.734746 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.734717 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t"] Sep 18 16:25:28.735158 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735144 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2289f90d-735f-4b5b-826f-64a0eeba573f" containerName="payload-pre-processing" Sep 18 16:25:28.735210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735159 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2289f90d-735f-4b5b-826f-64a0eeba573f" containerName="payload-pre-processing" Sep 18 16:25:28.735210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735175 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f771c7de-ffac-4f39-8bc5-81ecd769310d" containerName="maas-api" Sep 18 16:25:28.735210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735181 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f771c7de-ffac-4f39-8bc5-81ecd769310d" containerName="maas-api" Sep 18 16:25:28.735210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735197 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d34147e-56f1-4e35-9a50-7c329b2c495f" containerName="payload-processing" Sep 18 16:25:28.735210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735202 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d34147e-56f1-4e35-9a50-7c329b2c495f" containerName="payload-processing" Sep 18 16:25:28.735372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735259 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f771c7de-ffac-4f39-8bc5-81ecd769310d" containerName="maas-api" Sep 18 16:25:28.735372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735267 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2289f90d-735f-4b5b-826f-64a0eeba573f" containerName="payload-pre-processing" Sep 18 16:25:28.735372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.735277 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d34147e-56f1-4e35-9a50-7c329b2c495f" containerName="payload-processing" Sep 18 16:25:28.739604 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.739589 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:28.743603 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.743579 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5b98dc-dockercfg-vlk6x\"" Sep 18 16:25:28.743741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.743627 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5b98dc\"" Sep 18 16:25:28.753332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.753307 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t"] Sep 18 16:25:28.865582 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.865546 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrts2\" (UniqueName: \"kubernetes.io/projected/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-kube-api-access-qrts2\") pod \"payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:28.865755 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.865599 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:28.967165 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.967034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrts2\" (UniqueName: \"kubernetes.io/projected/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-kube-api-access-qrts2\") pod \"payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:28.967323 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.967178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:28.967877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.967857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:28.976684 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:28.976654 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrts2\" (UniqueName: \"kubernetes.io/projected/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-kube-api-access-qrts2\") pod \"payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:29.049423 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.049388 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:29.150589 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.150509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:29.334881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.334851 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t"] Sep 18 16:25:29.335504 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:25:29.335475 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e7cc2c_1ffe_4aac_a985_c8f314199e40.slice/crio-0ff7e009acf195d8dabf33d3da30d6d357315b9051e3c31bcbcf5cff8dd9adde WatchSource:0}: Error finding container 0ff7e009acf195d8dabf33d3da30d6d357315b9051e3c31bcbcf5cff8dd9adde: Status 404 returned error can't find the container with id 0ff7e009acf195d8dabf33d3da30d6d357315b9051e3c31bcbcf5cff8dd9adde Sep 18 16:25:29.431570 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.431530 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" event={"ID":"d7e7cc2c-1ffe-4aac-a985-c8f314199e40","Type":"ContainerStarted","Data":"2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c"} Sep 18 16:25:29.431735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.431631 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" event={"ID":"d7e7cc2c-1ffe-4aac-a985-c8f314199e40","Type":"ContainerStarted","Data":"0ff7e009acf195d8dabf33d3da30d6d357315b9051e3c31bcbcf5cff8dd9adde"} Sep 18 16:25:29.431735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.431664 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:29.452381 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:29.452306 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" podStartSLOduration=1.452290627 podStartE2EDuration="1.452290627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:29.451910105 +0000 UTC m=+1229.012754772" watchObservedRunningTime="2026-09-18 16:25:29.452290627 +0000 UTC m=+1229.013135293" Sep 18 16:25:40.439850 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:40.439818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:25:50.574455 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:50.574377 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:51.610689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:51.610653 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:52.481822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:52.481789 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:52.564323 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:52.564290 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:52.654420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:52.654389 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:52.864664 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:52.864626 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:52.965518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:52.965488 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:53.207387 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:53.207307 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:53.515528 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:53.515502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:53.619135 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:53.619077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:53.825749 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:53.825667 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:54.094868 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:54.094791 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:54.578313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:54.578284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:55.423529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:55.423494 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:56.516620 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:56.516585 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:57.242339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:57.242306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:57.406311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:57.406280 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:57.816883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:57.816844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:58.268408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:58.268378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:25:59.165974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:25:59.165941 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:00.192507 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:00.192478 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:01.031712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:01.031680 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:02.067440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.067406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:02.932028 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.931997 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76"] Sep 18 16:26:02.935733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.935710 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:02.938492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.938465 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5865ea-serving-cert\"" Sep 18 16:26:02.948818 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.948781 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76"] Sep 18 16:26:02.968658 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.968631 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73381fc-907b-4cb0-93fd-7a35a7b3652b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:02.968808 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.968663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5r2s\" (UniqueName: \"kubernetes.io/projected/d73381fc-907b-4cb0-93fd-7a35a7b3652b-kube-api-access-c5r2s\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:02.968808 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.968697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:02.968808 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.968797 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:02.981185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:02.981161 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:03.069424 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.069392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.069797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.069471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73381fc-907b-4cb0-93fd-7a35a7b3652b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.069797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.069492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5r2s\" (UniqueName: \"kubernetes.io/projected/d73381fc-907b-4cb0-93fd-7a35a7b3652b-kube-api-access-c5r2s\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.069797 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.069522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.070270 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.070246 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73381fc-907b-4cb0-93fd-7a35a7b3652b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.072621 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.072594 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.073474 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.073452 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.075145 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.075120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:03.079742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.079721 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5r2s\" (UniqueName: \"kubernetes.io/projected/d73381fc-907b-4cb0-93fd-7a35a7b3652b-kube-api-access-c5r2s\") pod \"maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.247247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.247161 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:03.530879 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.530853 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76"] Sep 18 16:26:03.533122 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:26:03.533078 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73381fc_907b_4cb0_93fd_7a35a7b3652b.slice/crio-2b6dbcb0b5a452b35f23ac6334c6566418454b235bb2ba5fffbe684f3e5ae4de WatchSource:0}: Error finding container 2b6dbcb0b5a452b35f23ac6334c6566418454b235bb2ba5fffbe684f3e5ae4de: Status 404 returned error can't find the container with id 2b6dbcb0b5a452b35f23ac6334c6566418454b235bb2ba5fffbe684f3e5ae4de Sep 18 16:26:03.545728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.545702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:03.635198 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.635165 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" event={"ID":"d73381fc-907b-4cb0-93fd-7a35a7b3652b","Type":"ContainerStarted","Data":"2b6dbcb0b5a452b35f23ac6334c6566418454b235bb2ba5fffbe684f3e5ae4de"} Sep 18 16:26:03.641274 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.641245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:03.671904 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.671875 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7"] Sep 18 16:26:03.676616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.676591 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.681026 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.681001 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5865ea\"" Sep 18 16:26:03.681727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.681385 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5865ea-dockercfg-wfbjn\"" Sep 18 16:26:03.689709 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.689686 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7"] Sep 18 16:26:03.775690 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.775657 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.775877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.775722 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddzn\" (UniqueName: \"kubernetes.io/projected/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-kube-api-access-gddzn\") pod \"payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.876724 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.876622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.876724 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.876691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddzn\" (UniqueName: \"kubernetes.io/projected/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-kube-api-access-gddzn\") pod \"payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.877389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.877366 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.888425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.888397 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddzn\" (UniqueName: \"kubernetes.io/projected/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-kube-api-access-gddzn\") pod \"payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:03.992592 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:03.992570 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:04.132368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.132284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:04.281701 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.281672 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7"] Sep 18 16:26:04.282934 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:26:04.282910 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9c5448_a8e7_49a4_a8cd_25015dab6a75.slice/crio-94bfb3e0ab3e51756bc4760154cb7418f8ba064a5b257452c4df4e916a7c7c4f WatchSource:0}: Error finding container 94bfb3e0ab3e51756bc4760154cb7418f8ba064a5b257452c4df4e916a7c7c4f: Status 404 returned error can't find the container with id 94bfb3e0ab3e51756bc4760154cb7418f8ba064a5b257452c4df4e916a7c7c4f Sep 18 16:26:04.642856 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.642810 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" event={"ID":"7c9c5448-a8e7-49a4-a8cd-25015dab6a75","Type":"ContainerStarted","Data":"ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525"} Sep 18 16:26:04.643050 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.642906 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" event={"ID":"7c9c5448-a8e7-49a4-a8cd-25015dab6a75","Type":"ContainerStarted","Data":"94bfb3e0ab3e51756bc4760154cb7418f8ba064a5b257452c4df4e916a7c7c4f"} Sep 18 16:26:04.643050 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.642934 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:04.644543 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.644520 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" event={"ID":"d73381fc-907b-4cb0-93fd-7a35a7b3652b","Type":"ContainerStarted","Data":"d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c"} Sep 18 16:26:04.644624 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.644592 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:04.666288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.666235 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" podStartSLOduration=1.666221305 podStartE2EDuration="1.666221305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:04.66444689 +0000 UTC m=+1264.225291556" watchObservedRunningTime="2026-09-18 16:26:04.666221305 +0000 UTC m=+1264.227066018" Sep 18 16:26:04.689899 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.689856 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" podStartSLOduration=2.250657858 podStartE2EDuration="2.689842231s" totalImagesPullingTime="439.184373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:04.687579728 +0000 UTC m=+1264.248424395" watchObservedRunningTime="2026-09-18 16:26:04.689842231 +0000 UTC m=+1264.250686898" Sep 18 16:26:04.977006 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:04.976919 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:10.655754 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:10.655723 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:26:15.653265 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:15.653230 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:26:16.048750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:16.048713 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:25.355894 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:25.355862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:26.210954 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:26.210918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:27.212370 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:27.212333 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:27.303791 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:27.303738 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:28.566852 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:28.566819 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:29.698887 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:29.698851 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:30.791176 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:30.791145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:31.854272 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:31.854228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:32.567700 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:32.567666 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:35.141975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:35.141943 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:35.911944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:35.911917 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:36.544495 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:36.544460 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:38.093804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:38.093759 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:40.074327 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:40.074294 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:40.319573 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:40.319541 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:40.731647 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:40.731619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:40.887132 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:40.887031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:41.842029 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:41.841998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:43.214606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:43.214575 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:44.294311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:44.294272 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:45.558361 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:45.558324 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:46.652726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:46.652697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:47.765778 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:47.765746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:49.002267 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.002237 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:49.150072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.150043 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:49.284607 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.284521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:49.680691 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.680617 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:49.782177 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.782149 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:49.820795 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.820760 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86"] Sep 18 16:26:49.824381 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.824358 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:49.827247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.827222 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-157a333a\"" Sep 18 16:26:49.827418 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.827399 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-157a333a-dockercfg-vq24t\"" Sep 18 16:26:49.838333 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.838304 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86"] Sep 18 16:26:49.893827 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.893787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wllsp\" (UniqueName: \"kubernetes.io/projected/fc8174c5-b188-4376-8788-fb954b573a31-kube-api-access-wllsp\") pod \"payload-processing-e2e-mt-157a333a-769d548657-7qn86\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:49.894007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.893840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8174c5-b188-4376-8788-fb954b573a31-plugins-config-volume\") pod \"payload-processing-e2e-mt-157a333a-769d548657-7qn86\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:49.995456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.995364 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wllsp\" (UniqueName: \"kubernetes.io/projected/fc8174c5-b188-4376-8788-fb954b573a31-kube-api-access-wllsp\") pod \"payload-processing-e2e-mt-157a333a-769d548657-7qn86\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:49.995456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.995421 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8174c5-b188-4376-8788-fb954b573a31-plugins-config-volume\") pod \"payload-processing-e2e-mt-157a333a-769d548657-7qn86\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:49.996143 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:49.996123 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8174c5-b188-4376-8788-fb954b573a31-plugins-config-volume\") pod \"payload-processing-e2e-mt-157a333a-769d548657-7qn86\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:50.009709 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.009678 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wllsp\" (UniqueName: \"kubernetes.io/projected/fc8174c5-b188-4376-8788-fb954b573a31-kube-api-access-wllsp\") pod \"payload-processing-e2e-mt-157a333a-769d548657-7qn86\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:50.135236 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.135192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:50.199225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.198369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:50.420616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.420583 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86"] Sep 18 16:26:50.422246 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:26:50.422219 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8174c5_b188_4376_8788_fb954b573a31.slice/crio-b2786c078337194587c1b3d93e1a3af60b722512176ed539fc4886244258bfd1 WatchSource:0}: Error finding container b2786c078337194587c1b3d93e1a3af60b722512176ed539fc4886244258bfd1: Status 404 returned error can't find the container with id b2786c078337194587c1b3d93e1a3af60b722512176ed539fc4886244258bfd1 Sep 18 16:26:50.911433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.911394 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" event={"ID":"fc8174c5-b188-4376-8788-fb954b573a31","Type":"ContainerStarted","Data":"5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc"} Sep 18 16:26:50.911652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.911476 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" event={"ID":"fc8174c5-b188-4376-8788-fb954b573a31","Type":"ContainerStarted","Data":"b2786c078337194587c1b3d93e1a3af60b722512176ed539fc4886244258bfd1"} Sep 18 16:26:50.911652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.911595 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:26:50.936210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:50.936148 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" podStartSLOduration=1.93613272 podStartE2EDuration="1.93613272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:50.933219543 +0000 UTC m=+1310.494064212" watchObservedRunningTime="2026-09-18 16:26:50.93613272 +0000 UTC m=+1310.496977387" Sep 18 16:26:51.156859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:51.156818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:52.418911 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:52.418871 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:53.205747 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:53.205709 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:54.067986 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:54.067953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:54.876991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:54.876953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:56.260567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:56.260530 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.062355 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.062312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.113733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.113694 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t"] Sep 18 16:26:57.113992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.113958 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" podUID="01d64134-383e-433b-87ef-40e77bc80fb4" containerName="payload-processing" containerID="cri-o://0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73" gracePeriod=30 Sep 18 16:26:57.127141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.127101 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn"] Sep 18 16:26:57.127353 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.127331 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" podUID="a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" containerName="payload-pre-processing" containerID="cri-o://cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c" gracePeriod=30 Sep 18 16:26:57.165349 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.165316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.266570 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.266520 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.366256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.366221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.398367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.398346 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:26:57.422911 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.422875 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:26:57.456053 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.456025 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.465789 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.465762 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d64134-383e-433b-87ef-40e77bc80fb4-plugins-config-volume\") pod \"01d64134-383e-433b-87ef-40e77bc80fb4\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " Sep 18 16:26:57.465937 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.465806 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl4pz\" (UniqueName: \"kubernetes.io/projected/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-kube-api-access-wl4pz\") pod \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " Sep 18 16:26:57.465937 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.465822 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-plugins-config-volume\") pod \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\" (UID: \"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2\") " Sep 18 16:26:57.465937 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.465928 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g27lx\" (UniqueName: \"kubernetes.io/projected/01d64134-383e-433b-87ef-40e77bc80fb4-kube-api-access-g27lx\") pod \"01d64134-383e-433b-87ef-40e77bc80fb4\" (UID: \"01d64134-383e-433b-87ef-40e77bc80fb4\") " Sep 18 16:26:57.466240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.466214 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" (UID: "a3b2cf67-9e46-4ffa-898f-f4735b2cebb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:26:57.466240 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.466214 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d64134-383e-433b-87ef-40e77bc80fb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01d64134-383e-433b-87ef-40e77bc80fb4" (UID: "01d64134-383e-433b-87ef-40e77bc80fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:26:57.466449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.466421 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d64134-383e-433b-87ef-40e77bc80fb4-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:26:57.466584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.466453 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:26:57.468143 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.468118 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d64134-383e-433b-87ef-40e77bc80fb4-kube-api-access-g27lx" (OuterVolumeSpecName: "kube-api-access-g27lx") pod "01d64134-383e-433b-87ef-40e77bc80fb4" (UID: "01d64134-383e-433b-87ef-40e77bc80fb4"). InnerVolumeSpecName "kube-api-access-g27lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:26:57.468558 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.468539 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-kube-api-access-wl4pz" (OuterVolumeSpecName: "kube-api-access-wl4pz") pod "a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" (UID: "a3b2cf67-9e46-4ffa-898f-f4735b2cebb2"). InnerVolumeSpecName "kube-api-access-wl4pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:26:57.567898 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.567859 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl4pz\" (UniqueName: \"kubernetes.io/projected/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2-kube-api-access-wl4pz\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:26:57.567898 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.567883 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g27lx\" (UniqueName: \"kubernetes.io/projected/01d64134-383e-433b-87ef-40e77bc80fb4-kube-api-access-g27lx\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:26:57.743291 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.743207 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.858320 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.858283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:57.967425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.967391 2563 generic.go:356] "Generic (PLEG): container finished" podID="01d64134-383e-433b-87ef-40e77bc80fb4" containerID="0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73" exitCode=0 Sep 18 16:26:57.967599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.967435 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" Sep 18 16:26:57.967599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.967478 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" event={"ID":"01d64134-383e-433b-87ef-40e77bc80fb4","Type":"ContainerDied","Data":"0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73"} Sep 18 16:26:57.967599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.967532 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t" event={"ID":"01d64134-383e-433b-87ef-40e77bc80fb4","Type":"ContainerDied","Data":"3dca93b252530e892f0cbb84db84ca171e01fdc34d08ba1217227d07fb48601e"} Sep 18 16:26:57.967599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.967561 2563 scope.go:117] "RemoveContainer" containerID="0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73" Sep 18 16:26:57.969046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.969019 2563 generic.go:356] "Generic (PLEG): container finished" podID="a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" containerID="cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c" exitCode=0 Sep 18 16:26:57.969209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.969156 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" Sep 18 16:26:57.969209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.969192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" event={"ID":"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2","Type":"ContainerDied","Data":"cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c"} Sep 18 16:26:57.969300 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.969235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn" event={"ID":"a3b2cf67-9e46-4ffa-898f-f4735b2cebb2","Type":"ContainerDied","Data":"f9fb92c24399242f986e1589ab9d1f43221a4bebff1b722d8cfd8f128d187d1f"} Sep 18 16:26:57.977476 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.977460 2563 scope.go:117] "RemoveContainer" containerID="0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73" Sep 18 16:26:57.977731 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:26:57.977713 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73\": container with ID starting with 0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73 not found: ID does not exist" containerID="0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73" Sep 18 16:26:57.977810 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.977738 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73"} err="failed to get container status \"0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73\": rpc error: code = NotFound desc = could not find container \"0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73\": container with ID starting with 0e2cc2f983f2cfe001789f6955e504f6f33976483cebe3bc6eae2f8758829f73 not found: ID does not exist" Sep 18 16:26:57.977810 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.977758 2563 scope.go:117] "RemoveContainer" containerID="cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c" Sep 18 16:26:57.986257 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.986240 2563 scope.go:117] "RemoveContainer" containerID="cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c" Sep 18 16:26:57.986502 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:26:57.986482 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c\": container with ID starting with cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c not found: ID does not exist" containerID="cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c" Sep 18 16:26:57.986555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.986509 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c"} err="failed to get container status \"cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c\": rpc error: code = NotFound desc = could not find container \"cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c\": container with ID starting with cccedd938d2fdc63be61b5a4e3acadfac57f8228ad9017119fb62c8b9c9aa39c not found: ID does not exist" Sep 18 16:26:57.998959 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:57.998939 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t"] Sep 18 16:26:58.006394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:58.006372 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fbc529-74c64f9c9-47s7t"] Sep 18 16:26:58.024456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:58.024435 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn"] Sep 18 16:26:58.030516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:58.030496 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fbc529-5cf5446bbb-4wnnn"] Sep 18 16:26:58.790696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:58.790664 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:26:59.081798 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:59.081715 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d64134-383e-433b-87ef-40e77bc80fb4" path="/var/lib/kubelet/pods/01d64134-383e-433b-87ef-40e77bc80fb4/volumes" Sep 18 16:26:59.082059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:26:59.082047 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" path="/var/lib/kubelet/pods/a3b2cf67-9e46-4ffa-898f-f4735b2cebb2/volumes" Sep 18 16:27:00.085733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:00.085699 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:01.293786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:01.293752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:01.920954 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:01.920920 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:27:02.171362 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:02.171279 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:02.998026 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:02.997979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:03.989260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:03.989225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:04.372475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:04.372431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:04.484881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:04.484846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:04.916371 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:04.916336 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:05.009808 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:05.009765 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:05.232539 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:05.232454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:05.526542 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:05.526505 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:06.008344 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:06.008311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:06.881651 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:06.881617 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:07.990335 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:07.990301 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:08.889606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:08.889573 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:09.873096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:09.873063 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:10.927728 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:10.927697 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:11.800401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:11.800363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:12.170616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:12.170528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:12.903382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:12.903320 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:14.983556 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:14.983524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:15.720831 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:15.720781 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:16.814477 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:16.814448 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:16.904390 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:16.904359 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj"] Sep 18 16:27:16.904588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:16.904564 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" podUID="ca4e7878-b08a-4004-b618-fd634fbbfdde" containerName="payload-processing" containerID="cri-o://6a4299dc0ee781ccb8e28da2a7b02fb5e043b4508111ae349378a52bdca8f3e1" gracePeriod=30 Sep 18 16:27:16.919438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:16.916350 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q"] Sep 18 16:27:16.919438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:16.916650 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" podUID="fa6b5a0b-a1ef-4000-83b6-43c2686813fd" containerName="payload-pre-processing" containerID="cri-o://5124d71d8b0c54ee186d2ccc837de505f34cc310e4eceb2869542738cfe4cdf3" gracePeriod=30 Sep 18 16:27:16.952102 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:16.952058 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:17.087719 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.087310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:17.095881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.095558 2563 generic.go:356] "Generic (PLEG): container finished" podID="fa6b5a0b-a1ef-4000-83b6-43c2686813fd" containerID="5124d71d8b0c54ee186d2ccc837de505f34cc310e4eceb2869542738cfe4cdf3" exitCode=0 Sep 18 16:27:17.095881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.095744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" event={"ID":"fa6b5a0b-a1ef-4000-83b6-43c2686813fd","Type":"ContainerDied","Data":"5124d71d8b0c54ee186d2ccc837de505f34cc310e4eceb2869542738cfe4cdf3"} Sep 18 16:27:17.097716 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.097693 2563 generic.go:356] "Generic (PLEG): container finished" podID="ca4e7878-b08a-4004-b618-fd634fbbfdde" containerID="6a4299dc0ee781ccb8e28da2a7b02fb5e043b4508111ae349378a52bdca8f3e1" exitCode=0 Sep 18 16:27:17.097836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.097742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" event={"ID":"ca4e7878-b08a-4004-b618-fd634fbbfdde","Type":"ContainerDied","Data":"6a4299dc0ee781ccb8e28da2a7b02fb5e043b4508111ae349378a52bdca8f3e1"} Sep 18 16:27:17.217133 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.217104 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:27:17.219155 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.219134 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:27:17.250128 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.250080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:17.359066 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.358976 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v84rd\" (UniqueName: \"kubernetes.io/projected/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-kube-api-access-v84rd\") pod \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " Sep 18 16:27:17.359066 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.359013 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t6rm\" (UniqueName: \"kubernetes.io/projected/ca4e7878-b08a-4004-b618-fd634fbbfdde-kube-api-access-8t6rm\") pod \"ca4e7878-b08a-4004-b618-fd634fbbfdde\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " Sep 18 16:27:17.359338 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.359073 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-plugins-config-volume\") pod \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\" (UID: \"fa6b5a0b-a1ef-4000-83b6-43c2686813fd\") " Sep 18 16:27:17.359338 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.359119 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4e7878-b08a-4004-b618-fd634fbbfdde-plugins-config-volume\") pod \"ca4e7878-b08a-4004-b618-fd634fbbfdde\" (UID: \"ca4e7878-b08a-4004-b618-fd634fbbfdde\") " Sep 18 16:27:17.359579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.359551 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa6b5a0b-a1ef-4000-83b6-43c2686813fd" (UID: "fa6b5a0b-a1ef-4000-83b6-43c2686813fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:27:17.359662 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.359577 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4e7878-b08a-4004-b618-fd634fbbfdde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca4e7878-b08a-4004-b618-fd634fbbfdde" (UID: "ca4e7878-b08a-4004-b618-fd634fbbfdde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:27:17.361482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.361455 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4e7878-b08a-4004-b618-fd634fbbfdde-kube-api-access-8t6rm" (OuterVolumeSpecName: "kube-api-access-8t6rm") pod "ca4e7878-b08a-4004-b618-fd634fbbfdde" (UID: "ca4e7878-b08a-4004-b618-fd634fbbfdde"). InnerVolumeSpecName "kube-api-access-8t6rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:27:17.361590 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.361487 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-kube-api-access-v84rd" (OuterVolumeSpecName: "kube-api-access-v84rd") pod "fa6b5a0b-a1ef-4000-83b6-43c2686813fd" (UID: "fa6b5a0b-a1ef-4000-83b6-43c2686813fd"). InnerVolumeSpecName "kube-api-access-v84rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:27:17.428452 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.428418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:17.460704 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.460673 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v84rd\" (UniqueName: \"kubernetes.io/projected/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-kube-api-access-v84rd\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:27:17.460704 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.460702 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t6rm\" (UniqueName: \"kubernetes.io/projected/ca4e7878-b08a-4004-b618-fd634fbbfdde-kube-api-access-8t6rm\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:27:17.460884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.460715 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6b5a0b-a1ef-4000-83b6-43c2686813fd-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:27:17.460884 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.460724 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4e7878-b08a-4004-b618-fd634fbbfdde-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:27:17.755711 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.755677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:17.911335 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:17.911306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:18.104982 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.104894 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" Sep 18 16:27:18.104982 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.104929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q" event={"ID":"fa6b5a0b-a1ef-4000-83b6-43c2686813fd","Type":"ContainerDied","Data":"821e912cb938b1c49338978b1218e8a99ce6f4743d66a8ab9a4099ce491cabf5"} Sep 18 16:27:18.104982 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.104981 2563 scope.go:117] "RemoveContainer" containerID="5124d71d8b0c54ee186d2ccc837de505f34cc310e4eceb2869542738cfe4cdf3" Sep 18 16:27:18.106507 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.106491 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" Sep 18 16:27:18.106609 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.106489 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj" event={"ID":"ca4e7878-b08a-4004-b618-fd634fbbfdde","Type":"ContainerDied","Data":"f1538540157f4cf998f10bc79e8cccd17f133b13cdc6e555d81f80944d8d11ac"} Sep 18 16:27:18.116043 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.116026 2563 scope.go:117] "RemoveContainer" containerID="6a4299dc0ee781ccb8e28da2a7b02fb5e043b4508111ae349378a52bdca8f3e1" Sep 18 16:27:18.140403 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.140375 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q"] Sep 18 16:27:18.155243 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.155217 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f22789-559b7694db-nzw6q"] Sep 18 16:27:18.173513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.173480 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj"] Sep 18 16:27:18.186017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:18.185988 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f22789-68d465879f-2wnrj"] Sep 18 16:27:19.081543 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:19.081513 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4e7878-b08a-4004-b618-fd634fbbfdde" path="/var/lib/kubelet/pods/ca4e7878-b08a-4004-b618-fd634fbbfdde/volumes" Sep 18 16:27:19.081971 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:19.081915 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6b5a0b-a1ef-4000-83b6-43c2686813fd" path="/var/lib/kubelet/pods/fa6b5a0b-a1ef-4000-83b6-43c2686813fd/volumes" Sep 18 16:27:19.086470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:19.086450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:20.968619 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:20.968578 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:22.209641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:22.209606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:22.933907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:22.933874 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:23.485492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:23.485463 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:24.436136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:24.436107 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:27:43.189619 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:27:43.189580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:01.040445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:01.040413 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:01.314511 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:01.314433 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:01.808390 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:01.808352 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:02.178739 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:02.178656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:02.499428 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:02.499400 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:04.178719 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:04.178684 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:04.864989 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:04.864955 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:04.994787 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:04.994759 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:05.372009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:05.371978 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:06.777264 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:06.777218 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:07.852550 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:07.852508 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:09.084586 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:09.084555 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:10.380620 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:10.380587 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:11.441282 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441246 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr"] Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441667 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" containerName="payload-pre-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441679 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" containerName="payload-pre-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441693 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d64134-383e-433b-87ef-40e77bc80fb4" containerName="payload-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441699 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d64134-383e-433b-87ef-40e77bc80fb4" containerName="payload-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441714 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6b5a0b-a1ef-4000-83b6-43c2686813fd" containerName="payload-pre-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441720 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6b5a0b-a1ef-4000-83b6-43c2686813fd" containerName="payload-pre-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441728 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4e7878-b08a-4004-b618-fd634fbbfdde" containerName="payload-processing" Sep 18 16:28:11.441767 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441733 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4e7878-b08a-4004-b618-fd634fbbfdde" containerName="payload-processing" Sep 18 16:28:11.442221 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441794 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6b5a0b-a1ef-4000-83b6-43c2686813fd" containerName="payload-pre-processing" Sep 18 16:28:11.442221 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441802 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4e7878-b08a-4004-b618-fd634fbbfdde" containerName="payload-processing" Sep 18 16:28:11.442221 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441811 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b2cf67-9e46-4ffa-898f-f4735b2cebb2" containerName="payload-pre-processing" Sep 18 16:28:11.442221 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.441816 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d64134-383e-433b-87ef-40e77bc80fb4" containerName="payload-processing" Sep 18 16:28:11.446341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.446319 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.451236 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.451218 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a75117-serving-cert\"" Sep 18 16:28:11.466316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.466293 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr"] Sep 18 16:28:11.514494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.514466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkfd\" (UniqueName: \"kubernetes.io/projected/55d7e433-39cf-4b18-9efe-6aa26d02eb09-kube-api-access-mfkfd\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.514661 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.514523 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.514661 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.514583 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d7e433-39cf-4b18-9efe-6aa26d02eb09-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.514752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.514675 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.516289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.516265 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:11.615595 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.615553 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.615777 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.615607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d7e433-39cf-4b18-9efe-6aa26d02eb09-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.615777 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.615649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.615777 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.615746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkfd\" (UniqueName: \"kubernetes.io/projected/55d7e433-39cf-4b18-9efe-6aa26d02eb09-kube-api-access-mfkfd\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.615959 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:28:11.615799 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-serving-cert: secret "maas-api-e2e-shared-a-w3-a75117-serving-cert" not found Sep 18 16:28:11.615959 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:28:11.615865 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-maas-api-tls podName:55d7e433-39cf-4b18-9efe-6aa26d02eb09 nodeName:}" failed. No retries permitted until 2026-09-18 16:28:12.115844792 +0000 UTC m=+1391.676689459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" (UID: "55d7e433-39cf-4b18-9efe-6aa26d02eb09") : secret "maas-api-e2e-shared-a-w3-a75117-serving-cert" not found Sep 18 16:28:11.616334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.616316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d7e433-39cf-4b18-9efe-6aa26d02eb09-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.617940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.617923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.623800 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.623772 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:11.639680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.639639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkfd\" (UniqueName: \"kubernetes.io/projected/55d7e433-39cf-4b18-9efe-6aa26d02eb09-kube-api-access-mfkfd\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:11.736584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:11.736503 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:12.121718 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.121677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:12.124046 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.124018 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d7e433-39cf-4b18-9efe-6aa26d02eb09-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr\" (UID: \"55d7e433-39cf-4b18-9efe-6aa26d02eb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:12.170163 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.170131 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:12.272049 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.272016 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:12.357782 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.357748 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:12.365984 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.365960 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq"] Sep 18 16:28:12.369629 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.369612 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.373038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.372984 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a75117\"" Sep 18 16:28:12.373210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.373195 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a75117-dockercfg-lqq85\"" Sep 18 16:28:12.393513 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.393484 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq"] Sep 18 16:28:12.524792 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.524757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdfb\" (UniqueName: \"kubernetes.io/projected/d6b62a3f-96a5-4e56-aca8-3e13af815341-kube-api-access-nxdfb\") pod \"payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq\" (UID: \"d6b62a3f-96a5-4e56-aca8-3e13af815341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.525225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.524912 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b62a3f-96a5-4e56-aca8-3e13af815341-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq\" (UID: \"d6b62a3f-96a5-4e56-aca8-3e13af815341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.626135 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.626028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b62a3f-96a5-4e56-aca8-3e13af815341-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq\" (UID: \"d6b62a3f-96a5-4e56-aca8-3e13af815341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.626299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.626140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdfb\" (UniqueName: \"kubernetes.io/projected/d6b62a3f-96a5-4e56-aca8-3e13af815341-kube-api-access-nxdfb\") pod \"payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq\" (UID: \"d6b62a3f-96a5-4e56-aca8-3e13af815341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.626765 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.626745 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b62a3f-96a5-4e56-aca8-3e13af815341-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq\" (UID: \"d6b62a3f-96a5-4e56-aca8-3e13af815341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.637059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.637034 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdfb\" (UniqueName: \"kubernetes.io/projected/d6b62a3f-96a5-4e56-aca8-3e13af815341-kube-api-access-nxdfb\") pod \"payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq\" (UID: \"d6b62a3f-96a5-4e56-aca8-3e13af815341\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.660750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.660730 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr"] Sep 18 16:28:12.662705 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:28:12.662680 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d7e433_39cf_4b18_9efe_6aa26d02eb09.slice/crio-73a705992e1c9687b41f141d6fb515c537b1d154b031978f3b26157413c06582 WatchSource:0}: Error finding container 73a705992e1c9687b41f141d6fb515c537b1d154b031978f3b26157413c06582: Status 404 returned error can't find the container with id 73a705992e1c9687b41f141d6fb515c537b1d154b031978f3b26157413c06582 Sep 18 16:28:12.707759 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.707729 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:12.984787 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:12.984749 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq"] Sep 18 16:28:12.986177 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:28:12.986151 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b62a3f_96a5_4e56_aca8_3e13af815341.slice/crio-cc8ec99e997f900909732587007aa94d7fab57aefd6d5c1d62e7690ca9697bae WatchSource:0}: Error finding container cc8ec99e997f900909732587007aa94d7fab57aefd6d5c1d62e7690ca9697bae: Status 404 returned error can't find the container with id cc8ec99e997f900909732587007aa94d7fab57aefd6d5c1d62e7690ca9697bae Sep 18 16:28:13.444882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.444844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" event={"ID":"d6b62a3f-96a5-4e56-aca8-3e13af815341","Type":"ContainerStarted","Data":"486604eea64791539115c6b05803693348c78a17277ce927cffa2e67960df46d"} Sep 18 16:28:13.445072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.444903 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" event={"ID":"d6b62a3f-96a5-4e56-aca8-3e13af815341","Type":"ContainerStarted","Data":"cc8ec99e997f900909732587007aa94d7fab57aefd6d5c1d62e7690ca9697bae"} Sep 18 16:28:13.445072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.444928 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:13.446685 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.446661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" event={"ID":"55d7e433-39cf-4b18-9efe-6aa26d02eb09","Type":"ContainerStarted","Data":"fe139c4052edf14acfc928839b8db06059ea87a87636ebdd99813a2061d2e15b"} Sep 18 16:28:13.446822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.446704 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" event={"ID":"55d7e433-39cf-4b18-9efe-6aa26d02eb09","Type":"ContainerStarted","Data":"73a705992e1c9687b41f141d6fb515c537b1d154b031978f3b26157413c06582"} Sep 18 16:28:13.446822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.446736 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:13.469186 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.469133 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" podStartSLOduration=1.469068953 podStartE2EDuration="1.469068953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:13.468760554 +0000 UTC m=+1393.029605224" watchObservedRunningTime="2026-09-18 16:28:13.469068953 +0000 UTC m=+1393.029913621" Sep 18 16:28:13.496967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:13.496866 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" podStartSLOduration=1.935240276 podStartE2EDuration="2.496848075s" totalImagesPullingTime="561.607799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:13.494131878 +0000 UTC m=+1393.054976546" watchObservedRunningTime="2026-09-18 16:28:13.496848075 +0000 UTC m=+1393.057692747" Sep 18 16:28:19.458833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:19.458804 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr" Sep 18 16:28:22.072930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:22.072900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:22.159366 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:22.159338 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:24.456017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:24.455983 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq" Sep 18 16:28:30.987559 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:30.987524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:34.195760 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:34.195726 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:34.921356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:34.921323 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:35.291515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:35.291488 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:35.503031 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:35.502994 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:35.664171 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:35.664099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:36.108159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:36.108126 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:36.514348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:36.514314 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:37.497414 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:37.497379 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:38.568829 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:38.568791 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:39.218405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:39.218378 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:39.361331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:39.361299 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:39.633065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:39.633027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:40.769763 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:40.769735 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:41.755970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:41.755938 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:41.849968 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:41.849937 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:41.961312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:41.961282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:42.317215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.317183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:42.490190 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.490156 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:42.523348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.523321 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9"] Sep 18 16:28:42.526878 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.526860 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.530010 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.529991 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-11725e-dockercfg-djmph\"" Sep 18 16:28:42.530358 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.530338 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-11725e\"" Sep 18 16:28:42.539631 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.539611 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9"] Sep 18 16:28:42.597975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.597886 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541268bf-2fa4-4a20-8403-3eb28672a296-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9\" (UID: \"541268bf-2fa4-4a20-8403-3eb28672a296\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.597975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.597947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddjz\" (UniqueName: \"kubernetes.io/projected/541268bf-2fa4-4a20-8403-3eb28672a296-kube-api-access-jddjz\") pod \"payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9\" (UID: \"541268bf-2fa4-4a20-8403-3eb28672a296\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.605017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.604987 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76"] Sep 18 16:28:42.609325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.609304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.623766 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.623736 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76"] Sep 18 16:28:42.699421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.699390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gfjw\" (UniqueName: \"kubernetes.io/projected/c5002529-ed7b-4d73-9154-26a20dfbff00-kube-api-access-2gfjw\") pod \"payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76\" (UID: \"c5002529-ed7b-4d73-9154-26a20dfbff00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.699594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.699499 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541268bf-2fa4-4a20-8403-3eb28672a296-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9\" (UID: \"541268bf-2fa4-4a20-8403-3eb28672a296\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.699594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.699533 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5002529-ed7b-4d73-9154-26a20dfbff00-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76\" (UID: \"c5002529-ed7b-4d73-9154-26a20dfbff00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.699594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.699560 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddjz\" (UniqueName: \"kubernetes.io/projected/541268bf-2fa4-4a20-8403-3eb28672a296-kube-api-access-jddjz\") pod \"payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9\" (UID: \"541268bf-2fa4-4a20-8403-3eb28672a296\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.700312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.700293 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541268bf-2fa4-4a20-8403-3eb28672a296-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9\" (UID: \"541268bf-2fa4-4a20-8403-3eb28672a296\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.711264 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.711242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddjz\" (UniqueName: \"kubernetes.io/projected/541268bf-2fa4-4a20-8403-3eb28672a296-kube-api-access-jddjz\") pod \"payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9\" (UID: \"541268bf-2fa4-4a20-8403-3eb28672a296\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.800529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.800496 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gfjw\" (UniqueName: \"kubernetes.io/projected/c5002529-ed7b-4d73-9154-26a20dfbff00-kube-api-access-2gfjw\") pod \"payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76\" (UID: \"c5002529-ed7b-4d73-9154-26a20dfbff00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.800702 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.800608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5002529-ed7b-4d73-9154-26a20dfbff00-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76\" (UID: \"c5002529-ed7b-4d73-9154-26a20dfbff00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.801405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.801386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5002529-ed7b-4d73-9154-26a20dfbff00-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76\" (UID: \"c5002529-ed7b-4d73-9154-26a20dfbff00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.814243 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.814215 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gfjw\" (UniqueName: \"kubernetes.io/projected/c5002529-ed7b-4d73-9154-26a20dfbff00-kube-api-access-2gfjw\") pod \"payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76\" (UID: \"c5002529-ed7b-4d73-9154-26a20dfbff00\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:42.838717 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.838685 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:42.863133 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.863080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:42.925161 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:42.925045 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:43.117197 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.117170 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9"] Sep 18 16:28:43.118246 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:28:43.118223 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541268bf_2fa4_4a20_8403_3eb28672a296.slice/crio-61b2c10d0780db9d328a6a8264f1abbd35f152cad2ca70b9b18ae85dee602442 WatchSource:0}: Error finding container 61b2c10d0780db9d328a6a8264f1abbd35f152cad2ca70b9b18ae85dee602442: Status 404 returned error can't find the container with id 61b2c10d0780db9d328a6a8264f1abbd35f152cad2ca70b9b18ae85dee602442 Sep 18 16:28:43.214692 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.214664 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76"] Sep 18 16:28:43.215171 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:28:43.215148 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5002529_ed7b_4d73_9154_26a20dfbff00.slice/crio-723da75dfd2f16eaa5fa4582229d0db68da0826d9536cb8cafa17a54527470bc WatchSource:0}: Error finding container 723da75dfd2f16eaa5fa4582229d0db68da0826d9536cb8cafa17a54527470bc: Status 404 returned error can't find the container with id 723da75dfd2f16eaa5fa4582229d0db68da0826d9536cb8cafa17a54527470bc Sep 18 16:28:43.633655 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.633615 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" event={"ID":"c5002529-ed7b-4d73-9154-26a20dfbff00","Type":"ContainerStarted","Data":"11f1003eb120f2752fde8528f55bdb4acd32a15ae62cbfd5873334894ddf65af"} Sep 18 16:28:43.633840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.633682 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:28:43.633840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.633710 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" event={"ID":"c5002529-ed7b-4d73-9154-26a20dfbff00","Type":"ContainerStarted","Data":"723da75dfd2f16eaa5fa4582229d0db68da0826d9536cb8cafa17a54527470bc"} Sep 18 16:28:43.635462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.635432 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" event={"ID":"541268bf-2fa4-4a20-8403-3eb28672a296","Type":"ContainerStarted","Data":"d2e8be5cf39196962f3c6134497bca8fb27dcba1e48a15daf28e2a9c7f891b0a"} Sep 18 16:28:43.635585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.635490 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:43.635585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.635515 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" event={"ID":"541268bf-2fa4-4a20-8403-3eb28672a296","Type":"ContainerStarted","Data":"61b2c10d0780db9d328a6a8264f1abbd35f152cad2ca70b9b18ae85dee602442"} Sep 18 16:28:43.668315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.668249 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" podStartSLOduration=1.668231126 podStartE2EDuration="1.668231126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:43.666982568 +0000 UTC m=+1423.227827236" watchObservedRunningTime="2026-09-18 16:28:43.668231126 +0000 UTC m=+1423.229075798" Sep 18 16:28:43.696636 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.696576 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" podStartSLOduration=1.696556786 podStartE2EDuration="1.696556786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:43.691853628 +0000 UTC m=+1423.252698298" watchObservedRunningTime="2026-09-18 16:28:43.696556786 +0000 UTC m=+1423.257401460" Sep 18 16:28:43.991352 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:43.991266 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:44.322000 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:44.321965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:44.452963 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:44.452934 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:45.057518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:45.057483 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:46.283811 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:46.283776 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:47.326841 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:47.326805 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:47.494747 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:47.494714 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:28:54.642710 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:54.642675 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9" Sep 18 16:28:54.643116 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:28:54.642750 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76" Sep 18 16:29:04.088538 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:04.088506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:04.530724 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:04.530692 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:05.873524 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:05.873467 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:06.971260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:06.971229 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:08.198509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:08.198474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:09.177716 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:09.177681 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:10.328255 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:10.328224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:11.486077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:11.486046 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:11.772172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:11.772142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:12.684444 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:12.684412 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:13.917315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:13.917282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:15.797743 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:15.797711 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:17.127916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:17.127888 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:18.425986 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:18.425806 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:29.276119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:29.276060 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:31.876833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:31.876793 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:36.852570 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:36.852535 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86"] Sep 18 16:29:36.853024 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:36.852801 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" podUID="fc8174c5-b188-4376-8788-fb954b573a31" containerName="payload-processing" containerID="cri-o://5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc" gracePeriod=30 Sep 18 16:29:36.988463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:36.988415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:37.115276 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.115213 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:29:37.203336 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.203300 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wllsp\" (UniqueName: \"kubernetes.io/projected/fc8174c5-b188-4376-8788-fb954b573a31-kube-api-access-wllsp\") pod \"fc8174c5-b188-4376-8788-fb954b573a31\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " Sep 18 16:29:37.203506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.203392 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8174c5-b188-4376-8788-fb954b573a31-plugins-config-volume\") pod \"fc8174c5-b188-4376-8788-fb954b573a31\" (UID: \"fc8174c5-b188-4376-8788-fb954b573a31\") " Sep 18 16:29:37.203754 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.203730 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8174c5-b188-4376-8788-fb954b573a31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc8174c5-b188-4376-8788-fb954b573a31" (UID: "fc8174c5-b188-4376-8788-fb954b573a31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:29:37.205597 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.205576 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8174c5-b188-4376-8788-fb954b573a31-kube-api-access-wllsp" (OuterVolumeSpecName: "kube-api-access-wllsp") pod "fc8174c5-b188-4376-8788-fb954b573a31" (UID: "fc8174c5-b188-4376-8788-fb954b573a31"). InnerVolumeSpecName "kube-api-access-wllsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:29:37.242349 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.242320 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:37.304545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.304516 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wllsp\" (UniqueName: \"kubernetes.io/projected/fc8174c5-b188-4376-8788-fb954b573a31-kube-api-access-wllsp\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:29:37.304545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.304542 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8174c5-b188-4376-8788-fb954b573a31-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:29:37.467462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.467386 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:37.961668 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.961635 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc8174c5-b188-4376-8788-fb954b573a31" containerID="5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc" exitCode=0 Sep 18 16:29:37.962140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.961682 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" event={"ID":"fc8174c5-b188-4376-8788-fb954b573a31","Type":"ContainerDied","Data":"5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc"} Sep 18 16:29:37.962140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.961685 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" Sep 18 16:29:37.962140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.961721 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86" event={"ID":"fc8174c5-b188-4376-8788-fb954b573a31","Type":"ContainerDied","Data":"b2786c078337194587c1b3d93e1a3af60b722512176ed539fc4886244258bfd1"} Sep 18 16:29:37.962140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.961743 2563 scope.go:117] "RemoveContainer" containerID="5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc" Sep 18 16:29:37.972309 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.972288 2563 scope.go:117] "RemoveContainer" containerID="5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc" Sep 18 16:29:37.972606 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:29:37.972580 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc\": container with ID starting with 5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc not found: ID does not exist" containerID="5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc" Sep 18 16:29:37.972679 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:37.972615 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc"} err="failed to get container status \"5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc\": rpc error: code = NotFound desc = could not find container \"5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc\": container with ID starting with 5af7f9ec724c64b292487b0d21a0bd24039c06e1c85b49b5fd57e26fadcc58fc not found: ID does not exist" Sep 18 16:29:38.012201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:38.012176 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86"] Sep 18 16:29:38.015936 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:38.015914 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-157a333a-769d548657-7qn86"] Sep 18 16:29:38.061356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:38.061332 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:39.081825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:39.081793 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8174c5-b188-4376-8788-fb954b573a31" path="/var/lib/kubelet/pods/fc8174c5-b188-4376-8788-fb954b573a31/volumes" Sep 18 16:29:39.399578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:39.399500 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:40.625279 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:40.625252 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:40.791503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:40.791474 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:41.951523 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:41.951496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:43.238206 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:43.238169 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:44.537920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:44.537885 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:45.702499 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:45.702461 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:45.841494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:45.841462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:55.726840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:55.726804 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:55.811273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:55.811239 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:29:58.924911 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:29:58.924878 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:00.241817 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.241785 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm"] Sep 18 16:30:00.242304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.242287 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8174c5-b188-4376-8788-fb954b573a31" containerName="payload-processing" Sep 18 16:30:00.242304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.242304 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8174c5-b188-4376-8788-fb954b573a31" containerName="payload-processing" Sep 18 16:30:00.242440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.242374 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8174c5-b188-4376-8788-fb954b573a31" containerName="payload-processing" Sep 18 16:30:00.245615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.245590 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" Sep 18 16:30:00.249741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.249711 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\"" Sep 18 16:30:00.256896 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.256873 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" Sep 18 16:30:00.265903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.265878 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm"] Sep 18 16:30:00.539641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.539609 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm"] Sep 18 16:30:00.540794 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:30:00.540767 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc3b1e5_ce5c_44da_b0aa_55b2591d7a90.slice/crio-581d6a0c67daaf9bba10d96c72604c44b70e950e8057ee3d005f669d66a9a250 WatchSource:0}: Error finding container 581d6a0c67daaf9bba10d96c72604c44b70e950e8057ee3d005f669d66a9a250: Status 404 returned error can't find the container with id 581d6a0c67daaf9bba10d96c72604c44b70e950e8057ee3d005f669d66a9a250 Sep 18 16:30:00.542965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.542943 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:30:00.817042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:00.816953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:01.109712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:01.109635 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" event={"ID":"9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90","Type":"ContainerStarted","Data":"581d6a0c67daaf9bba10d96c72604c44b70e950e8057ee3d005f669d66a9a250"} Sep 18 16:30:03.126880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:03.126850 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:30:03.128356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:03.128339 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:30:04.129932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:04.129832 2563 generic.go:356] "Generic (PLEG): container finished" podID="9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90" containerID="83dce14125b8ad3a4a806d336acfaf875027f32d2be97abf30cd062146a1d2a6" exitCode=0 Sep 18 16:30:04.129932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:04.129874 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" event={"ID":"9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90","Type":"ContainerDied","Data":"83dce14125b8ad3a4a806d336acfaf875027f32d2be97abf30cd062146a1d2a6"} Sep 18 16:30:05.280245 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:05.280219 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" Sep 18 16:30:06.143616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:06.143579 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" event={"ID":"9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90","Type":"ContainerDied","Data":"581d6a0c67daaf9bba10d96c72604c44b70e950e8057ee3d005f669d66a9a250"} Sep 18 16:30:06.143804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:06.143618 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm" Sep 18 16:30:06.143804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:06.143625 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581d6a0c67daaf9bba10d96c72604c44b70e950e8057ee3d005f669d66a9a250" Sep 18 16:30:19.590855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:19.590813 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:22.820189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:22.820155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:24.021189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:24.021150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:28.340407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:28.340371 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:30.628744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:30.628715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:30.734176 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:30.734147 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:30.921494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:30.921422 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:32.307266 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:32.307234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:32.500630 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:32.500599 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:33.032373 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:33.032340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:33.133334 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:33.133295 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:33.240072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:33.240038 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:34.553641 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:34.553608 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:41.506833 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:41.506800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:41.636257 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:41.636222 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:42.252350 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:42.252320 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:49.537857 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:49.537825 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:49.652881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:49.652842 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:49.783299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:49.783271 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:49.884271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:49.884192 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:30:56.502147 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:30:56.502111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:00.329319 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:00.329275 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:00.548304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:00.548258 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:05.777440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:05.777406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:15.111260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:15.111225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:21.532633 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:21.532600 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:24.762011 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:24.761979 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:28.543094 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:28.543058 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:28.664346 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:28.664311 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:28.993907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:28.993831 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:29.412214 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:29.412184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:29.607006 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:29.606976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:32.252918 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:32.252880 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:32.526679 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:32.526601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:32.960608 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:32.960492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:34.446933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:34.446900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:35.660482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:35.660411 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:36.878974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:36.878935 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:38.574426 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:38.574395 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:40.314603 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:40.314571 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:41.161059 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.161025 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv"] Sep 18 16:31:41.161761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.161737 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90" containerName="cleanup" Sep 18 16:31:41.161761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.161760 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90" containerName="cleanup" Sep 18 16:31:41.161947 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.161877 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90" containerName="cleanup" Sep 18 16:31:41.164981 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.164959 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.168477 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.168460 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ac5968-serving-cert\"" Sep 18 16:31:41.189675 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.189647 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv"] Sep 18 16:31:41.237258 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.237224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:41.257554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.257522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.257554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.257556 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-metrics-certs\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.257770 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.257647 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmz96\" (UniqueName: \"kubernetes.io/projected/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-kube-api-access-fmz96\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.257770 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.257760 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.351931 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.351878 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:41.358898 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.358877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmz96\" (UniqueName: \"kubernetes.io/projected/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-kube-api-access-fmz96\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.358980 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.358932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.358980 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.358972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.359053 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.358991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-metrics-certs\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.359142 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:31:41.359126 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-serving-cert: secret "maas-api-e2e-api-a-ac5968-serving-cert" not found Sep 18 16:31:41.359200 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:31:41.359193 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls podName:ea2a6f72-1aa7-470d-a921-47b0aab19a0f nodeName:}" failed. No retries permitted until 2026-09-18 16:31:41.859170804 +0000 UTC m=+1601.420015462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls") pod "maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" (UID: "ea2a6f72-1aa7-470d-a921-47b0aab19a0f") : secret "maas-api-e2e-api-a-ac5968-serving-cert" not found Sep 18 16:31:41.359735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.359713 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-openshift-service-ca\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.361536 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.361514 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-metrics-certs\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.390619 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.390587 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmz96\" (UniqueName: \"kubernetes.io/projected/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-kube-api-access-fmz96\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.472802 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.472710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:41.863577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.863532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.876479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.876446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls\") pod \"maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:41.906345 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:41.906313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:42.075187 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:42.075149 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:42.083870 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:42.083843 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:31:42.361698 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:42.361666 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv"] Sep 18 16:31:42.363183 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:31:42.363149 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2a6f72_1aa7_470d_a921_47b0aab19a0f.slice/crio-6eb29473fcd75ec511ea79c15e0f16c6665c58e7b6a70cf42249065dfcdb706b WatchSource:0}: Error finding container 6eb29473fcd75ec511ea79c15e0f16c6665c58e7b6a70cf42249065dfcdb706b: Status 404 returned error can't find the container with id 6eb29473fcd75ec511ea79c15e0f16c6665c58e7b6a70cf42249065dfcdb706b Sep 18 16:31:42.723980 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:42.723943 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" event={"ID":"ea2a6f72-1aa7-470d-a921-47b0aab19a0f","Type":"ContainerStarted","Data":"6eb29473fcd75ec511ea79c15e0f16c6665c58e7b6a70cf42249065dfcdb706b"} Sep 18 16:31:43.732773 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:43.732681 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" event={"ID":"ea2a6f72-1aa7-470d-a921-47b0aab19a0f","Type":"ContainerStarted","Data":"f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f"} Sep 18 16:31:43.733167 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:43.732799 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:31:43.759009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:43.758958 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" podStartSLOduration=2.361615143 podStartE2EDuration="2.758940506s" totalImagesPullingTime="397.325363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:43.757998654 +0000 UTC m=+1603.318843334" watchObservedRunningTime="2026-09-18 16:31:43.758940506 +0000 UTC m=+1603.319785173" Sep 18 16:31:49.743939 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:31:49.743905 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:32:04.208009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:04.207965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:05.417949 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:05.417913 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:06.394292 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:06.394254 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:07.428490 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:07.428452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:07.622961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:07.622926 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:07.869265 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:07.869228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:08.437032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:08.436998 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:08.544877 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:08.544837 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:09.030806 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:09.030767 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:10.425288 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:10.425252 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:11.575259 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:11.575197 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:11.764131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:11.764077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:12.837436 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:12.837405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:13.987505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:13.987467 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc"] Sep 18 16:32:13.991585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:13.991563 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:13.996699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:13.996678 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9eae22-serving-cert\"" Sep 18 16:32:14.062755 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.062723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:14.074003 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.073962 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxthv\" (UniqueName: \"kubernetes.io/projected/30ede228-961f-43d0-8817-4324879b36dc-kube-api-access-xxthv\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.074181 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.074039 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-maas-api-tls\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.074181 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.074106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ede228-961f-43d0-8817-4324879b36dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.074289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.074273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-metrics-certs\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.094463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.094432 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc"] Sep 18 16:32:14.175285 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.175246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-maas-api-tls\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.175483 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.175299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ede228-961f-43d0-8817-4324879b36dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.175483 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.175436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-metrics-certs\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.175483 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.175474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxthv\" (UniqueName: \"kubernetes.io/projected/30ede228-961f-43d0-8817-4324879b36dc-kube-api-access-xxthv\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.176074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.176038 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ede228-961f-43d0-8817-4324879b36dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.177860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.177838 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-maas-api-tls\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.177938 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.177868 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-metrics-certs\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.190140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.190115 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxthv\" (UniqueName: \"kubernetes.io/projected/30ede228-961f-43d0-8817-4324879b36dc-kube-api-access-xxthv\") pod \"maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.210499 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.210472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:14.303389 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.303348 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:14.597932 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.597902 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc"] Sep 18 16:32:14.600014 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:32:14.599991 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ede228_961f_43d0_8817_4324879b36dc.slice/crio-52c7b4140870b118a22bac65c23bb61298d0ea986b2fa31fb3a8423994ce5818 WatchSource:0}: Error finding container 52c7b4140870b118a22bac65c23bb61298d0ea986b2fa31fb3a8423994ce5818: Status 404 returned error can't find the container with id 52c7b4140870b118a22bac65c23bb61298d0ea986b2fa31fb3a8423994ce5818 Sep 18 16:32:14.837852 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.837821 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:14.921215 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.921120 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" event={"ID":"30ede228-961f-43d0-8817-4324879b36dc","Type":"ContainerStarted","Data":"52c7b4140870b118a22bac65c23bb61298d0ea986b2fa31fb3a8423994ce5818"} Sep 18 16:32:14.994232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:14.994202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:15.404123 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.404078 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl"] Sep 18 16:32:15.407693 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.407674 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.410573 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.410552 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9eae22\"" Sep 18 16:32:15.411129 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.411074 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9eae22-dockercfg-vmkc2\"" Sep 18 16:32:15.438680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.438642 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl"] Sep 18 16:32:15.489015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.488979 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl8xg\" (UniqueName: \"kubernetes.io/projected/dab537d1-94c1-4a00-aee7-70680e2aff9d-kube-api-access-kl8xg\") pod \"payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.489015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.489020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab537d1-94c1-4a00-aee7-70680e2aff9d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.590516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.590481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl8xg\" (UniqueName: \"kubernetes.io/projected/dab537d1-94c1-4a00-aee7-70680e2aff9d-kube-api-access-kl8xg\") pod \"payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.590730 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.590531 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab537d1-94c1-4a00-aee7-70680e2aff9d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.591292 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.591268 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab537d1-94c1-4a00-aee7-70680e2aff9d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.602596 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.602567 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl8xg\" (UniqueName: \"kubernetes.io/projected/dab537d1-94c1-4a00-aee7-70680e2aff9d-kube-api-access-kl8xg\") pod \"payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.713837 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.713751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:15.720741 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.720713 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:15.928394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.928349 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" event={"ID":"30ede228-961f-43d0-8817-4324879b36dc","Type":"ContainerStarted","Data":"4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33"} Sep 18 16:32:15.928565 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.928415 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:15.958267 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:15.958217 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" podStartSLOduration=2.54169133 podStartE2EDuration="2.958199365s" totalImagesPullingTime="416.508035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:15.955910574 +0000 UTC m=+1635.516755240" watchObservedRunningTime="2026-09-18 16:32:15.958199365 +0000 UTC m=+1635.519044032" Sep 18 16:32:16.012321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:16.012293 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl"] Sep 18 16:32:16.013331 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:32:16.013305 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab537d1_94c1_4a00_aee7_70680e2aff9d.slice/crio-8532ebbaeeb068d8817fc3de69bd5f28ab1cc71627e960ddeebadccc50eaadb8 WatchSource:0}: Error finding container 8532ebbaeeb068d8817fc3de69bd5f28ab1cc71627e960ddeebadccc50eaadb8: Status 404 returned error can't find the container with id 8532ebbaeeb068d8817fc3de69bd5f28ab1cc71627e960ddeebadccc50eaadb8 Sep 18 16:32:16.761960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:16.761924 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:16.937611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:16.937573 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" event={"ID":"dab537d1-94c1-4a00-aee7-70680e2aff9d","Type":"ContainerStarted","Data":"0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba"} Sep 18 16:32:16.937840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:16.937668 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" event={"ID":"dab537d1-94c1-4a00-aee7-70680e2aff9d","Type":"ContainerStarted","Data":"8532ebbaeeb068d8817fc3de69bd5f28ab1cc71627e960ddeebadccc50eaadb8"} Sep 18 16:32:16.937840 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:16.937707 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:16.958364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:16.958311 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" podStartSLOduration=1.958296831 podStartE2EDuration="1.958296831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:16.956545007 +0000 UTC m=+1636.517389675" watchObservedRunningTime="2026-09-18 16:32:16.958296831 +0000 UTC m=+1636.519141498" Sep 18 16:32:17.945319 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:17.945291 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:19.121351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:19.121319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:20.287382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:20.287350 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:21.489261 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:21.489226 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:21.943095 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:21.943050 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:32:22.653748 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:22.653710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:24.786755 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:24.786722 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:25.771550 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:25.771516 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:26.311792 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:26.311751 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:26.874961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:26.874927 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:27.895379 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:27.895348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:27.945225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:27.945197 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:32:29.237110 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:29.233300 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:30.460846 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:30.460795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:31.428455 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:31.428415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:32.564761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:32.564706 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:33.429421 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:33.429393 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:34.440189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:34.440154 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:35.401676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:35.401644 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:35.743360 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:35.743285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:36.591756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:36.591725 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:37.550259 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:37.550225 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:38.486348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:38.486319 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:39.579733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:39.579702 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:41.906236 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:41.906203 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:45.386352 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.386291 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv"] Sep 18 16:32:45.386828 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.386611 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" podUID="ea2a6f72-1aa7-470d-a921-47b0aab19a0f" containerName="maas-api" containerID="cri-o://f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f" gracePeriod=30 Sep 18 16:32:45.457681 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.457653 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:45.647359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.647325 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:45.657578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.657554 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:32:45.790580 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.790549 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmz96\" (UniqueName: \"kubernetes.io/projected/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-kube-api-access-fmz96\") pod \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " Sep 18 16:32:45.790770 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.790620 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-openshift-service-ca\") pod \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " Sep 18 16:32:45.790770 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.790749 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls\") pod \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " Sep 18 16:32:45.790894 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.790776 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-metrics-certs\") pod \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\" (UID: \"ea2a6f72-1aa7-470d-a921-47b0aab19a0f\") " Sep 18 16:32:45.791036 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.790999 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea2a6f72-1aa7-470d-a921-47b0aab19a0f" (UID: "ea2a6f72-1aa7-470d-a921-47b0aab19a0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:32:45.791155 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.791138 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-openshift-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:32:45.792855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.792828 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea2a6f72-1aa7-470d-a921-47b0aab19a0f" (UID: "ea2a6f72-1aa7-470d-a921-47b0aab19a0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:32:45.792855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.792837 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-kube-api-access-fmz96" (OuterVolumeSpecName: "kube-api-access-fmz96") pod "ea2a6f72-1aa7-470d-a921-47b0aab19a0f" (UID: "ea2a6f72-1aa7-470d-a921-47b0aab19a0f"). InnerVolumeSpecName "kube-api-access-fmz96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:32:45.793013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.792895 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea2a6f72-1aa7-470d-a921-47b0aab19a0f" (UID: "ea2a6f72-1aa7-470d-a921-47b0aab19a0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:32:45.892030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.891940 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-maas-api-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:32:45.892030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.891965 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:32:45.892030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.891974 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmz96\" (UniqueName: \"kubernetes.io/projected/ea2a6f72-1aa7-470d-a921-47b0aab19a0f-kube-api-access-fmz96\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:32:45.897040 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:45.897010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:46.065644 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.065606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:46.114585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.114546 2563 generic.go:356] "Generic (PLEG): container finished" podID="ea2a6f72-1aa7-470d-a921-47b0aab19a0f" containerID="f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f" exitCode=0 Sep 18 16:32:46.114585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.114587 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" Sep 18 16:32:46.114782 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.114606 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" event={"ID":"ea2a6f72-1aa7-470d-a921-47b0aab19a0f","Type":"ContainerDied","Data":"f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f"} Sep 18 16:32:46.114782 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.114660 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv" event={"ID":"ea2a6f72-1aa7-470d-a921-47b0aab19a0f","Type":"ContainerDied","Data":"6eb29473fcd75ec511ea79c15e0f16c6665c58e7b6a70cf42249065dfcdb706b"} Sep 18 16:32:46.114782 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.114683 2563 scope.go:117] "RemoveContainer" containerID="f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f" Sep 18 16:32:46.127803 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.127786 2563 scope.go:117] "RemoveContainer" containerID="f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f" Sep 18 16:32:46.128077 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:32:46.128057 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f\": container with ID starting with f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f not found: ID does not exist" containerID="f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f" Sep 18 16:32:46.128196 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.128099 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f"} err="failed to get container status \"f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f\": rpc error: code = NotFound desc = could not find container \"f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f\": container with ID starting with f290787c84989e8bb7efec3841f20ca3867cdb44023cf1e85fe1546d45abae2f not found: ID does not exist" Sep 18 16:32:46.156455 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.156399 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv"] Sep 18 16:32:46.163815 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.163790 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ac5968-66f7bbf786-hwkfv"] Sep 18 16:32:46.337433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.337403 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:46.611677 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:46.611645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:47.081666 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:47.081633 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2a6f72-1aa7-470d-a921-47b0aab19a0f" path="/var/lib/kubelet/pods/ea2a6f72-1aa7-470d-a921-47b0aab19a0f/volumes" Sep 18 16:32:47.901072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:47.901038 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:49.360449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:49.360409 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:50.463706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:50.463678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:51.774835 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:51.774802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:52.960906 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:52.960866 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:53.968788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:53.968750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:54.474153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:54.474075 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:54.592014 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:54.591984 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:54.790153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:54.790121 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:54.967066 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:54.967034 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:56.199480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:56.199448 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:57.475534 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:57.475498 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:32:59.077289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:32:59.077256 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:00.232658 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:00.232628 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:01.710468 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:01.710404 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:02.707831 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:02.707797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:03.797650 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:03.797617 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:04.810742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:04.810711 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:10.916454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:10.916414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:10.917122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:10.916672 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc"] Sep 18 16:33:10.917122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:10.916918 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" podUID="30ede228-961f-43d0-8817-4324879b36dc" containerName="maas-api" containerID="cri-o://4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33" gracePeriod=30 Sep 18 16:33:10.973506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:10.973475 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl"] Sep 18 16:33:10.973698 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:10.973668 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" podUID="dab537d1-94c1-4a00-aee7-70680e2aff9d" containerName="payload-processing" containerID="cri-o://0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba" gracePeriod=30 Sep 18 16:33:11.108751 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.108715 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:11.222367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.222341 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:33:11.257597 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.257571 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:33:11.271925 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.271895 2563 generic.go:356] "Generic (PLEG): container finished" podID="30ede228-961f-43d0-8817-4324879b36dc" containerID="4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33" exitCode=0 Sep 18 16:33:11.272114 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.271945 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" Sep 18 16:33:11.272114 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.271972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" event={"ID":"30ede228-961f-43d0-8817-4324879b36dc","Type":"ContainerDied","Data":"4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33"} Sep 18 16:33:11.272114 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.272055 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc" event={"ID":"30ede228-961f-43d0-8817-4324879b36dc","Type":"ContainerDied","Data":"52c7b4140870b118a22bac65c23bb61298d0ea986b2fa31fb3a8423994ce5818"} Sep 18 16:33:11.272114 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.272071 2563 scope.go:117] "RemoveContainer" containerID="4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33" Sep 18 16:33:11.273629 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.273607 2563 generic.go:356] "Generic (PLEG): container finished" podID="dab537d1-94c1-4a00-aee7-70680e2aff9d" containerID="0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba" exitCode=0 Sep 18 16:33:11.273738 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.273645 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" Sep 18 16:33:11.273738 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.273688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" event={"ID":"dab537d1-94c1-4a00-aee7-70680e2aff9d","Type":"ContainerDied","Data":"0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba"} Sep 18 16:33:11.273838 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.273752 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl" event={"ID":"dab537d1-94c1-4a00-aee7-70680e2aff9d","Type":"ContainerDied","Data":"8532ebbaeeb068d8817fc3de69bd5f28ab1cc71627e960ddeebadccc50eaadb8"} Sep 18 16:33:11.282879 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.282861 2563 scope.go:117] "RemoveContainer" containerID="4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33" Sep 18 16:33:11.283215 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:11.283195 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33\": container with ID starting with 4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33 not found: ID does not exist" containerID="4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33" Sep 18 16:33:11.283280 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.283223 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33"} err="failed to get container status \"4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33\": rpc error: code = NotFound desc = could not find container \"4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33\": container with ID starting with 4d028cf924cb106b6c00acde638b70d0172db14c2f5481e8ee69effc3e861d33 not found: ID does not exist" Sep 18 16:33:11.283280 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.283240 2563 scope.go:117] "RemoveContainer" containerID="0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba" Sep 18 16:33:11.297982 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.297957 2563 scope.go:117] "RemoveContainer" containerID="0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba" Sep 18 16:33:11.298365 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:11.298345 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba\": container with ID starting with 0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba not found: ID does not exist" containerID="0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba" Sep 18 16:33:11.298448 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.298394 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba"} err="failed to get container status \"0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba\": rpc error: code = NotFound desc = could not find container \"0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba\": container with ID starting with 0d6cddc9dbf1d12fda7a1848879f0dd74b54d92728bd33c4ab0e55e05a6691ba not found: ID does not exist" Sep 18 16:33:11.316273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.316204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:11.333796 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.333769 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxthv\" (UniqueName: \"kubernetes.io/projected/30ede228-961f-43d0-8817-4324879b36dc-kube-api-access-xxthv\") pod \"30ede228-961f-43d0-8817-4324879b36dc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " Sep 18 16:33:11.333915 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.333806 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab537d1-94c1-4a00-aee7-70680e2aff9d-plugins-config-volume\") pod \"dab537d1-94c1-4a00-aee7-70680e2aff9d\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " Sep 18 16:33:11.333915 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.333866 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-maas-api-tls\") pod \"30ede228-961f-43d0-8817-4324879b36dc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " Sep 18 16:33:11.333915 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.333908 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-metrics-certs\") pod \"30ede228-961f-43d0-8817-4324879b36dc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " Sep 18 16:33:11.334119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.333940 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl8xg\" (UniqueName: \"kubernetes.io/projected/dab537d1-94c1-4a00-aee7-70680e2aff9d-kube-api-access-kl8xg\") pod \"dab537d1-94c1-4a00-aee7-70680e2aff9d\" (UID: \"dab537d1-94c1-4a00-aee7-70680e2aff9d\") " Sep 18 16:33:11.334119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.333976 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ede228-961f-43d0-8817-4324879b36dc-openshift-service-ca\") pod \"30ede228-961f-43d0-8817-4324879b36dc\" (UID: \"30ede228-961f-43d0-8817-4324879b36dc\") " Sep 18 16:33:11.334410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.334386 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab537d1-94c1-4a00-aee7-70680e2aff9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dab537d1-94c1-4a00-aee7-70680e2aff9d" (UID: "dab537d1-94c1-4a00-aee7-70680e2aff9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:11.334502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.334424 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ede228-961f-43d0-8817-4324879b36dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30ede228-961f-43d0-8817-4324879b36dc" (UID: "30ede228-961f-43d0-8817-4324879b36dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:11.335951 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.335927 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ede228-961f-43d0-8817-4324879b36dc-kube-api-access-xxthv" (OuterVolumeSpecName: "kube-api-access-xxthv") pod "30ede228-961f-43d0-8817-4324879b36dc" (UID: "30ede228-961f-43d0-8817-4324879b36dc"). InnerVolumeSpecName "kube-api-access-xxthv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:11.336079 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.336059 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab537d1-94c1-4a00-aee7-70680e2aff9d-kube-api-access-kl8xg" (OuterVolumeSpecName: "kube-api-access-kl8xg") pod "dab537d1-94c1-4a00-aee7-70680e2aff9d" (UID: "dab537d1-94c1-4a00-aee7-70680e2aff9d"). InnerVolumeSpecName "kube-api-access-kl8xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:11.336445 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.336426 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30ede228-961f-43d0-8817-4324879b36dc" (UID: "30ede228-961f-43d0-8817-4324879b36dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:33:11.336497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.336443 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "30ede228-961f-43d0-8817-4324879b36dc" (UID: "30ede228-961f-43d0-8817-4324879b36dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:33:11.434942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.434871 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-maas-api-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:11.434942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.434898 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ede228-961f-43d0-8817-4324879b36dc-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:11.434942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.434911 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl8xg\" (UniqueName: \"kubernetes.io/projected/dab537d1-94c1-4a00-aee7-70680e2aff9d-kube-api-access-kl8xg\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:11.434942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.434925 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ede228-961f-43d0-8817-4324879b36dc-openshift-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:11.434942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.434938 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxthv\" (UniqueName: \"kubernetes.io/projected/30ede228-961f-43d0-8817-4324879b36dc-kube-api-access-xxthv\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:11.435228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.434950 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab537d1-94c1-4a00-aee7-70680e2aff9d-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:11.510511 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.510476 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:11.613920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.613895 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc"] Sep 18 16:33:11.616450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.616420 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9eae22-8dc955dc5-48kjc"] Sep 18 16:33:11.627122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.627102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:11.666816 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.666771 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl"] Sep 18 16:33:11.673471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.673443 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9eae22-67c578b589-gwmpl"] Sep 18 16:33:11.939530 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:11.939507 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:12.210410 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:12.210328 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:13.081025 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:13.080990 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ede228-961f-43d0-8817-4324879b36dc" path="/var/lib/kubelet/pods/30ede228-961f-43d0-8817-4324879b36dc/volumes" Sep 18 16:33:13.081419 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:13.081383 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab537d1-94c1-4a00-aee7-70680e2aff9d" path="/var/lib/kubelet/pods/dab537d1-94c1-4a00-aee7-70680e2aff9d/volumes" Sep 18 16:33:13.815864 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:13.815815 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:15.456493 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:15.456455 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:17.063786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:17.063749 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:18.362836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:18.362802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:19.877630 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:19.877599 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:21.163290 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:21.163257 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:21.475135 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:21.475040 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:21.624828 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:21.624797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:21.934138 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:21.934107 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:23.202798 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:23.202754 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:24.870293 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:24.870261 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:26.067613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:26.067582 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:27.104130 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:27.104079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:28.607608 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:28.607577 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:30.054122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:30.054080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:31.444643 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:31.444604 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:32.543614 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.543580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:32.552882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.552844 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm"] Sep 18 16:33:32.561743 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.561708 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5b98dc-29829150-p9ghm"] Sep 18 16:33:32.633862 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.633830 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t"] Sep 18 16:33:32.634096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.634064 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" podUID="d7e7cc2c-1ffe-4aac-a985-c8f314199e40" containerName="payload-processing" containerID="cri-o://2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c" gracePeriod=30 Sep 18 16:33:32.717685 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.717648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:32.834544 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.834508 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:32.890485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.890463 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:33:32.958554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:32.958515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:33.031153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.031123 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-plugins-config-volume\") pod \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " Sep 18 16:33:33.031329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.031238 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrts2\" (UniqueName: \"kubernetes.io/projected/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-kube-api-access-qrts2\") pod \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\" (UID: \"d7e7cc2c-1ffe-4aac-a985-c8f314199e40\") " Sep 18 16:33:33.031551 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.031527 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7e7cc2c-1ffe-4aac-a985-c8f314199e40" (UID: "d7e7cc2c-1ffe-4aac-a985-c8f314199e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:33.033276 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.033251 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-kube-api-access-qrts2" (OuterVolumeSpecName: "kube-api-access-qrts2") pod "d7e7cc2c-1ffe-4aac-a985-c8f314199e40" (UID: "d7e7cc2c-1ffe-4aac-a985-c8f314199e40"). InnerVolumeSpecName "kube-api-access-qrts2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:33.083774 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.083677 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90" path="/var/lib/kubelet/pods/9dc3b1e5-ce5c-44da-b0aa-55b2591d7a90/volumes" Sep 18 16:33:33.088167 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.088113 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:33.132065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.132036 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:33.132065 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.132064 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrts2\" (UniqueName: \"kubernetes.io/projected/d7e7cc2c-1ffe-4aac-a985-c8f314199e40-kube-api-access-qrts2\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:33.366029 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.365895 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:33.406412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.406383 2563 generic.go:356] "Generic (PLEG): container finished" podID="d7e7cc2c-1ffe-4aac-a985-c8f314199e40" containerID="2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c" exitCode=0 Sep 18 16:33:33.406577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.406432 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" Sep 18 16:33:33.406577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.406460 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" event={"ID":"d7e7cc2c-1ffe-4aac-a985-c8f314199e40","Type":"ContainerDied","Data":"2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c"} Sep 18 16:33:33.406577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.406515 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t" event={"ID":"d7e7cc2c-1ffe-4aac-a985-c8f314199e40","Type":"ContainerDied","Data":"0ff7e009acf195d8dabf33d3da30d6d357315b9051e3c31bcbcf5cff8dd9adde"} Sep 18 16:33:33.406577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.406540 2563 scope.go:117] "RemoveContainer" containerID="2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c" Sep 18 16:33:33.415605 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.415591 2563 scope.go:117] "RemoveContainer" containerID="2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c" Sep 18 16:33:33.415886 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:33.415868 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c\": container with ID starting with 2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c not found: ID does not exist" containerID="2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c" Sep 18 16:33:33.415956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.415891 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c"} err="failed to get container status \"2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c\": rpc error: code = NotFound desc = could not find container \"2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c\": container with ID starting with 2e219a711de3d6913647331d24f8a29aee9ab9754b244fadbbe966ab059d6d9c not found: ID does not exist" Sep 18 16:33:33.434246 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.434219 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t"] Sep 18 16:33:33.449119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.444394 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5b98dc-684dfb7f5f-5s55t"] Sep 18 16:33:33.602960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:33.602924 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:34.740097 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:34.740047 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:35.081715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:35.081682 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7e7cc2c-1ffe-4aac-a985-c8f314199e40" path="/var/lib/kubelet/pods/d7e7cc2c-1ffe-4aac-a985-c8f314199e40/volumes" Sep 18 16:33:35.766553 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:35.766519 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:36.771314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:36.771284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:37.941401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:37.941369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:39.067881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:39.067843 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:40.312658 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:40.312618 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:41.588242 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:41.588211 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:41.717885 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:41.717858 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:41.966633 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:41.966556 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:42.385577 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:42.385543 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:42.485945 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:42.485916 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:43.414613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:43.414582 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:44.284729 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:44.284692 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:45.051746 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:45.051716 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:46.065712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:46.065679 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:46.924485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:46.924456 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:47.534318 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:47.534283 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:48.047839 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:48.047807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:48.702172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:48.702144 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:48.803271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:48.803245 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:53.080913 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.080881 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76"] Sep 18 16:33:53.081662 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.081629 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" podUID="d73381fc-907b-4cb0-93fd-7a35a7b3652b" containerName="maas-api" containerID="cri-o://d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c" gracePeriod=30 Sep 18 16:33:53.117518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.117490 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:53.160844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.160812 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7"] Sep 18 16:33:53.161023 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.160997 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" podUID="7c9c5448-a8e7-49a4-a8cd-25015dab6a75" containerName="payload-processing" containerID="cri-o://ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525" gracePeriod=30 Sep 18 16:33:53.298455 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.298367 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:53.464974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.464950 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:33:53.466815 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.466796 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:33:53.508159 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508132 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-maas-api-tls\") pod \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " Sep 18 16:33:53.508329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508185 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-metrics-certs\") pod \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " Sep 18 16:33:53.508329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508206 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-plugins-config-volume\") pod \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " Sep 18 16:33:53.508329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508269 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5r2s\" (UniqueName: \"kubernetes.io/projected/d73381fc-907b-4cb0-93fd-7a35a7b3652b-kube-api-access-c5r2s\") pod \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " Sep 18 16:33:53.508509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508336 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gddzn\" (UniqueName: \"kubernetes.io/projected/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-kube-api-access-gddzn\") pod \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\" (UID: \"7c9c5448-a8e7-49a4-a8cd-25015dab6a75\") " Sep 18 16:33:53.508509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508370 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73381fc-907b-4cb0-93fd-7a35a7b3652b-openshift-service-ca\") pod \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\" (UID: \"d73381fc-907b-4cb0-93fd-7a35a7b3652b\") " Sep 18 16:33:53.508626 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508588 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c9c5448-a8e7-49a4-a8cd-25015dab6a75" (UID: "7c9c5448-a8e7-49a4-a8cd-25015dab6a75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:53.508721 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508703 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:53.508915 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.508874 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73381fc-907b-4cb0-93fd-7a35a7b3652b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d73381fc-907b-4cb0-93fd-7a35a7b3652b" (UID: "d73381fc-907b-4cb0-93fd-7a35a7b3652b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:53.510360 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.510329 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d73381fc-907b-4cb0-93fd-7a35a7b3652b" (UID: "d73381fc-907b-4cb0-93fd-7a35a7b3652b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:33:53.510952 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.510926 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d73381fc-907b-4cb0-93fd-7a35a7b3652b" (UID: "d73381fc-907b-4cb0-93fd-7a35a7b3652b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:33:53.510952 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.510934 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73381fc-907b-4cb0-93fd-7a35a7b3652b-kube-api-access-c5r2s" (OuterVolumeSpecName: "kube-api-access-c5r2s") pod "d73381fc-907b-4cb0-93fd-7a35a7b3652b" (UID: "d73381fc-907b-4cb0-93fd-7a35a7b3652b"). InnerVolumeSpecName "kube-api-access-c5r2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:53.511076 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.510964 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-kube-api-access-gddzn" (OuterVolumeSpecName: "kube-api-access-gddzn") pod "7c9c5448-a8e7-49a4-a8cd-25015dab6a75" (UID: "7c9c5448-a8e7-49a4-a8cd-25015dab6a75"). InnerVolumeSpecName "kube-api-access-gddzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:53.528071 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.528047 2563 generic.go:356] "Generic (PLEG): container finished" podID="7c9c5448-a8e7-49a4-a8cd-25015dab6a75" containerID="ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525" exitCode=0 Sep 18 16:33:53.528171 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.528107 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" Sep 18 16:33:53.528171 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.528119 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" event={"ID":"7c9c5448-a8e7-49a4-a8cd-25015dab6a75","Type":"ContainerDied","Data":"ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525"} Sep 18 16:33:53.528263 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.528170 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7" event={"ID":"7c9c5448-a8e7-49a4-a8cd-25015dab6a75","Type":"ContainerDied","Data":"94bfb3e0ab3e51756bc4760154cb7418f8ba064a5b257452c4df4e916a7c7c4f"} Sep 18 16:33:53.528263 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.528200 2563 scope.go:117] "RemoveContainer" containerID="ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525" Sep 18 16:33:53.529790 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.529766 2563 generic.go:356] "Generic (PLEG): container finished" podID="d73381fc-907b-4cb0-93fd-7a35a7b3652b" containerID="d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c" exitCode=0 Sep 18 16:33:53.529892 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.529814 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" Sep 18 16:33:53.529892 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.529850 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" event={"ID":"d73381fc-907b-4cb0-93fd-7a35a7b3652b","Type":"ContainerDied","Data":"d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c"} Sep 18 16:33:53.529892 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.529886 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76" event={"ID":"d73381fc-907b-4cb0-93fd-7a35a7b3652b","Type":"ContainerDied","Data":"2b6dbcb0b5a452b35f23ac6334c6566418454b235bb2ba5fffbe684f3e5ae4de"} Sep 18 16:33:53.542730 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.542647 2563 scope.go:117] "RemoveContainer" containerID="ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525" Sep 18 16:33:53.543034 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:53.542999 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525\": container with ID starting with ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525 not found: ID does not exist" containerID="ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525" Sep 18 16:33:53.543224 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.543032 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525"} err="failed to get container status \"ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525\": rpc error: code = NotFound desc = could not find container \"ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525\": container with ID starting with ab3573aada7f099a337f8fcfe610e38e783e61283e7e1a72c0a8ef42915e4525 not found: ID does not exist" Sep 18 16:33:53.543224 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.543057 2563 scope.go:117] "RemoveContainer" containerID="d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c" Sep 18 16:33:53.552144 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.552127 2563 scope.go:117] "RemoveContainer" containerID="d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c" Sep 18 16:33:53.552422 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:53.552402 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c\": container with ID starting with d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c not found: ID does not exist" containerID="d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c" Sep 18 16:33:53.552503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.552425 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c"} err="failed to get container status \"d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c\": rpc error: code = NotFound desc = could not find container \"d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c\": container with ID starting with d540a7cfff2192ece682ed667402fe3af7af16f173679d5bcd47c41fb3fe973c not found: ID does not exist" Sep 18 16:33:53.570230 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.570203 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76"] Sep 18 16:33:53.591259 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.591237 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5865ea-56cc9d54fd-r8z76"] Sep 18 16:33:53.609659 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.609636 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:53.609659 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.609656 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5r2s\" (UniqueName: \"kubernetes.io/projected/d73381fc-907b-4cb0-93fd-7a35a7b3652b-kube-api-access-c5r2s\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:53.609779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.609666 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gddzn\" (UniqueName: \"kubernetes.io/projected/7c9c5448-a8e7-49a4-a8cd-25015dab6a75-kube-api-access-gddzn\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:53.609779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.609675 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73381fc-907b-4cb0-93fd-7a35a7b3652b-openshift-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:53.609779 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.609683 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d73381fc-907b-4cb0-93fd-7a35a7b3652b-maas-api-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:53.623150 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.623064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:53.656433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.656392 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7"] Sep 18 16:33:53.664309 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.664270 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5865ea-65d85487f9-spjz7"] Sep 18 16:33:53.741256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:53.741227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:54.007906 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:54.007877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:54.294246 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:54.294166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:55.081008 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.080964 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9c5448-a8e7-49a4-a8cd-25015dab6a75" path="/var/lib/kubelet/pods/7c9c5448-a8e7-49a4-a8cd-25015dab6a75/volumes" Sep 18 16:33:55.081403 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.081389 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73381fc-907b-4cb0-93fd-7a35a7b3652b" path="/var/lib/kubelet/pods/d73381fc-907b-4cb0-93fd-7a35a7b3652b/volumes" Sep 18 16:33:55.560425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.560393 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:55.624034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.623999 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp"] Sep 18 16:33:55.624278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.624233 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" podUID="491163c7-d81c-4112-bfd0-99e5847a9637" containerName="payload-processing" containerID="cri-o://b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e" gracePeriod=30 Sep 18 16:33:55.657606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.657527 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq"] Sep 18 16:33:55.658251 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.658029 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" podUID="c166202f-1ca2-4ada-8ff9-b096dca559d8" containerName="payload-pre-processing" containerID="cri-o://37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7" gracePeriod=30 Sep 18 16:33:55.693260 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.693227 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:55.789557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.789515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:55.938761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.936776 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:33:55.941275 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.941239 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:55.941275 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:55.941261 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:33:56.031613 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.031584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf2qn\" (UniqueName: \"kubernetes.io/projected/491163c7-d81c-4112-bfd0-99e5847a9637-kube-api-access-pf2qn\") pod \"491163c7-d81c-4112-bfd0-99e5847a9637\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " Sep 18 16:33:56.031786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.031644 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbznn\" (UniqueName: \"kubernetes.io/projected/c166202f-1ca2-4ada-8ff9-b096dca559d8-kube-api-access-vbznn\") pod \"c166202f-1ca2-4ada-8ff9-b096dca559d8\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " Sep 18 16:33:56.031786 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.031772 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c166202f-1ca2-4ada-8ff9-b096dca559d8-plugins-config-volume\") pod \"c166202f-1ca2-4ada-8ff9-b096dca559d8\" (UID: \"c166202f-1ca2-4ada-8ff9-b096dca559d8\") " Sep 18 16:33:56.031859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.031820 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491163c7-d81c-4112-bfd0-99e5847a9637-plugins-config-volume\") pod \"491163c7-d81c-4112-bfd0-99e5847a9637\" (UID: \"491163c7-d81c-4112-bfd0-99e5847a9637\") " Sep 18 16:33:56.032200 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.032171 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c166202f-1ca2-4ada-8ff9-b096dca559d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c166202f-1ca2-4ada-8ff9-b096dca559d8" (UID: "c166202f-1ca2-4ada-8ff9-b096dca559d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:56.032348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.032206 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/491163c7-d81c-4112-bfd0-99e5847a9637-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "491163c7-d81c-4112-bfd0-99e5847a9637" (UID: "491163c7-d81c-4112-bfd0-99e5847a9637"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:33:56.032348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.032308 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c166202f-1ca2-4ada-8ff9-b096dca559d8-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:56.032348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.032327 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/491163c7-d81c-4112-bfd0-99e5847a9637-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:56.033781 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.033757 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c166202f-1ca2-4ada-8ff9-b096dca559d8-kube-api-access-vbznn" (OuterVolumeSpecName: "kube-api-access-vbznn") pod "c166202f-1ca2-4ada-8ff9-b096dca559d8" (UID: "c166202f-1ca2-4ada-8ff9-b096dca559d8"). InnerVolumeSpecName "kube-api-access-vbznn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:56.033879 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.033779 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/491163c7-d81c-4112-bfd0-99e5847a9637-kube-api-access-pf2qn" (OuterVolumeSpecName: "kube-api-access-pf2qn") pod "491163c7-d81c-4112-bfd0-99e5847a9637" (UID: "491163c7-d81c-4112-bfd0-99e5847a9637"). InnerVolumeSpecName "kube-api-access-pf2qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:33:56.040579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.040558 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:56.133459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.133379 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2qn\" (UniqueName: \"kubernetes.io/projected/491163c7-d81c-4112-bfd0-99e5847a9637-kube-api-access-pf2qn\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:56.133459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.133404 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbznn\" (UniqueName: \"kubernetes.io/projected/c166202f-1ca2-4ada-8ff9-b096dca559d8-kube-api-access-vbznn\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:33:56.348438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.348407 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:56.550684 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.550653 2563 generic.go:356] "Generic (PLEG): container finished" podID="c166202f-1ca2-4ada-8ff9-b096dca559d8" containerID="37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7" exitCode=0 Sep 18 16:33:56.550903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.550705 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" Sep 18 16:33:56.550903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.550740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" event={"ID":"c166202f-1ca2-4ada-8ff9-b096dca559d8","Type":"ContainerDied","Data":"37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7"} Sep 18 16:33:56.550903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.550814 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq" event={"ID":"c166202f-1ca2-4ada-8ff9-b096dca559d8","Type":"ContainerDied","Data":"4a269df3b1da00aeed8bfd145a81449aa1b378e1fb9d240e775bf8e702a71bc2"} Sep 18 16:33:56.550903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.550850 2563 scope.go:117] "RemoveContainer" containerID="37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7" Sep 18 16:33:56.552325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.552301 2563 generic.go:356] "Generic (PLEG): container finished" podID="491163c7-d81c-4112-bfd0-99e5847a9637" containerID="b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e" exitCode=0 Sep 18 16:33:56.552443 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.552394 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" Sep 18 16:33:56.552443 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.552403 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" event={"ID":"491163c7-d81c-4112-bfd0-99e5847a9637","Type":"ContainerDied","Data":"b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e"} Sep 18 16:33:56.552567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.552463 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp" event={"ID":"491163c7-d81c-4112-bfd0-99e5847a9637","Type":"ContainerDied","Data":"31eaf3e8eb893c673bf91a507611128044b0da821cf8f5e5d510cb51dc39ec3e"} Sep 18 16:33:56.560815 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.560795 2563 scope.go:117] "RemoveContainer" containerID="37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7" Sep 18 16:33:56.561226 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:56.561123 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7\": container with ID starting with 37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7 not found: ID does not exist" containerID="37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7" Sep 18 16:33:56.561226 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.561156 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7"} err="failed to get container status \"37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7\": rpc error: code = NotFound desc = could not find container \"37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7\": container with ID starting with 37ee7e4f190e3666e1000bdcc418cc737d7543a7cdc664132faffb4cc5e654e7 not found: ID does not exist" Sep 18 16:33:56.561226 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.561174 2563 scope.go:117] "RemoveContainer" containerID="b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e" Sep 18 16:33:56.569955 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.569940 2563 scope.go:117] "RemoveContainer" containerID="b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e" Sep 18 16:33:56.570284 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:33:56.570263 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e\": container with ID starting with b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e not found: ID does not exist" containerID="b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e" Sep 18 16:33:56.570343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.570292 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e"} err="failed to get container status \"b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e\": rpc error: code = NotFound desc = could not find container \"b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e\": container with ID starting with b3e46041ccd25c89916dc6aa59001c021828476d37919890fe5754492d48f66e not found: ID does not exist" Sep 18 16:33:56.578734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.578713 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp"] Sep 18 16:33:56.585425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.585403 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1eec1d-7799cc4d5c-hkwfp"] Sep 18 16:33:56.598600 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.598580 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq"] Sep 18 16:33:56.602172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:56.602150 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1eec1d-7b7c957878-zpmtq"] Sep 18 16:33:57.072859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:57.072830 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:57.081143 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:57.081116 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="491163c7-d81c-4112-bfd0-99e5847a9637" path="/var/lib/kubelet/pods/491163c7-d81c-4112-bfd0-99e5847a9637/volumes" Sep 18 16:33:57.081587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:57.081567 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c166202f-1ca2-4ada-8ff9-b096dca559d8" path="/var/lib/kubelet/pods/c166202f-1ca2-4ada-8ff9-b096dca559d8/volumes" Sep 18 16:33:58.101443 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:58.101412 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:33:59.118067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:33:59.118030 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:00.039776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:00.039738 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:01.173429 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:01.173393 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:01.274295 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:01.274265 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:01.601244 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:01.601205 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:03.613457 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:03.613356 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:03.712756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:03.712723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:03.943944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:03.943827 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:20.534286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:20.534246 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:21.032042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:21.032009 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:37.627164 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:37.627125 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:41.593892 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:41.593863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:42.654378 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:42.654336 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:42.879923 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:42.879887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:43.079618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:43.079588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:43.822638 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:43.822600 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:43.922313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:43.922264 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:44.177620 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:44.177492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:44.982363 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:44.982307 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:50.215129 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.215080 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j"] Sep 18 16:34:50.215512 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.215291 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" podUID="e3d61092-8da2-4fa0-a0ab-e189d8742447" containerName="maas-api" containerID="cri-o://fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10" gracePeriod=30 Sep 18 16:34:50.264805 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.264772 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:50.376493 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.376461 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4"] Sep 18 16:34:50.376750 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.376708 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" podUID="6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" containerName="payload-processing" containerID="cri-o://ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2" gracePeriod=30 Sep 18 16:34:50.410294 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.410267 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp"] Sep 18 16:34:50.410516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.410492 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" podUID="2d162f08-db46-4065-ad8e-e61620b5c225" containerName="payload-pre-processing" containerID="cri-o://ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642" gracePeriod=30 Sep 18 16:34:50.485749 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.485700 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:50.534604 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.534572 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:34:50.618971 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.618907 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnknd\" (UniqueName: \"kubernetes.io/projected/e3d61092-8da2-4fa0-a0ab-e189d8742447-kube-api-access-bnknd\") pod \"e3d61092-8da2-4fa0-a0ab-e189d8742447\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " Sep 18 16:34:50.619193 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.619120 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls\") pod \"e3d61092-8da2-4fa0-a0ab-e189d8742447\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " Sep 18 16:34:50.619193 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.619187 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d61092-8da2-4fa0-a0ab-e189d8742447-openshift-service-ca\") pod \"e3d61092-8da2-4fa0-a0ab-e189d8742447\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " Sep 18 16:34:50.619321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.619298 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-metrics-certs\") pod \"e3d61092-8da2-4fa0-a0ab-e189d8742447\" (UID: \"e3d61092-8da2-4fa0-a0ab-e189d8742447\") " Sep 18 16:34:50.619529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.619506 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d61092-8da2-4fa0-a0ab-e189d8742447-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3d61092-8da2-4fa0-a0ab-e189d8742447" (UID: "e3d61092-8da2-4fa0-a0ab-e189d8742447"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:34:50.619792 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.619768 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3d61092-8da2-4fa0-a0ab-e189d8742447-openshift-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.621836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.621814 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d61092-8da2-4fa0-a0ab-e189d8742447-kube-api-access-bnknd" (OuterVolumeSpecName: "kube-api-access-bnknd") pod "e3d61092-8da2-4fa0-a0ab-e189d8742447" (UID: "e3d61092-8da2-4fa0-a0ab-e189d8742447"). InnerVolumeSpecName "kube-api-access-bnknd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:34:50.623259 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.623220 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3d61092-8da2-4fa0-a0ab-e189d8742447" (UID: "e3d61092-8da2-4fa0-a0ab-e189d8742447"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:34:50.623883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.623854 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3d61092-8da2-4fa0-a0ab-e189d8742447" (UID: "e3d61092-8da2-4fa0-a0ab-e189d8742447"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:34:50.647067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.647044 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:34:50.682342 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.682314 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:34:50.721151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.721081 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk5tx\" (UniqueName: \"kubernetes.io/projected/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-kube-api-access-zk5tx\") pod \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " Sep 18 16:34:50.721254 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.721188 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-plugins-config-volume\") pod \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\" (UID: \"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06\") " Sep 18 16:34:50.721390 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.721374 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnknd\" (UniqueName: \"kubernetes.io/projected/e3d61092-8da2-4fa0-a0ab-e189d8742447-kube-api-access-bnknd\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.721462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.721393 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-maas-api-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.721462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.721405 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3d61092-8da2-4fa0-a0ab-e189d8742447-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.721574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.721536 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" (UID: "6fef0cc0-a218-4e16-b4bd-b06dad7d3e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:34:50.723017 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.722995 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:50.723229 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.723208 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-kube-api-access-zk5tx" (OuterVolumeSpecName: "kube-api-access-zk5tx") pod "6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" (UID: "6fef0cc0-a218-4e16-b4bd-b06dad7d3e06"). InnerVolumeSpecName "kube-api-access-zk5tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:34:50.822011 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.821981 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d162f08-db46-4065-ad8e-e61620b5c225-plugins-config-volume\") pod \"2d162f08-db46-4065-ad8e-e61620b5c225\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " Sep 18 16:34:50.822228 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.822073 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbdjs\" (UniqueName: \"kubernetes.io/projected/2d162f08-db46-4065-ad8e-e61620b5c225-kube-api-access-qbdjs\") pod \"2d162f08-db46-4065-ad8e-e61620b5c225\" (UID: \"2d162f08-db46-4065-ad8e-e61620b5c225\") " Sep 18 16:34:50.822304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.822246 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.822304 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.822257 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5tx\" (UniqueName: \"kubernetes.io/projected/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06-kube-api-access-zk5tx\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.822461 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.822437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d162f08-db46-4065-ad8e-e61620b5c225-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d162f08-db46-4065-ad8e-e61620b5c225" (UID: "2d162f08-db46-4065-ad8e-e61620b5c225"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:34:50.824135 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.824114 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d162f08-db46-4065-ad8e-e61620b5c225-kube-api-access-qbdjs" (OuterVolumeSpecName: "kube-api-access-qbdjs") pod "2d162f08-db46-4065-ad8e-e61620b5c225" (UID: "2d162f08-db46-4065-ad8e-e61620b5c225"). InnerVolumeSpecName "kube-api-access-qbdjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:34:50.856735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.856707 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:50.866076 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.866053 2563 generic.go:356] "Generic (PLEG): container finished" podID="6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" containerID="ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2" exitCode=0 Sep 18 16:34:50.866205 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.866114 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" Sep 18 16:34:50.866205 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.866148 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" event={"ID":"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06","Type":"ContainerDied","Data":"ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2"} Sep 18 16:34:50.866323 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.866205 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4" event={"ID":"6fef0cc0-a218-4e16-b4bd-b06dad7d3e06","Type":"ContainerDied","Data":"89a893a01532f55946aa0153b637fa2ebc3769f93f99df018b663daead170d38"} Sep 18 16:34:50.866323 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.866269 2563 scope.go:117] "RemoveContainer" containerID="ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2" Sep 18 16:34:50.867747 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.867726 2563 generic.go:356] "Generic (PLEG): container finished" podID="e3d61092-8da2-4fa0-a0ab-e189d8742447" containerID="fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10" exitCode=0 Sep 18 16:34:50.867861 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.867775 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" Sep 18 16:34:50.867861 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.867802 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" event={"ID":"e3d61092-8da2-4fa0-a0ab-e189d8742447","Type":"ContainerDied","Data":"fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10"} Sep 18 16:34:50.867861 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.867844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j" event={"ID":"e3d61092-8da2-4fa0-a0ab-e189d8742447","Type":"ContainerDied","Data":"6d1ed9af88c8771b18317e7e08ea1087a9efea59055a46b41c43fb1ea0033350"} Sep 18 16:34:50.869446 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.869425 2563 generic.go:356] "Generic (PLEG): container finished" podID="2d162f08-db46-4065-ad8e-e61620b5c225" containerID="ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642" exitCode=0 Sep 18 16:34:50.869549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.869471 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" Sep 18 16:34:50.869549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.869527 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" event={"ID":"2d162f08-db46-4065-ad8e-e61620b5c225","Type":"ContainerDied","Data":"ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642"} Sep 18 16:34:50.869643 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.869591 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp" event={"ID":"2d162f08-db46-4065-ad8e-e61620b5c225","Type":"ContainerDied","Data":"2e76fd6c2119eefec95d87922c74d65519cc446c2d02a42975c471f91ae3373f"} Sep 18 16:34:50.878237 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.878196 2563 scope.go:117] "RemoveContainer" containerID="ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2" Sep 18 16:34:50.878503 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:34:50.878474 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2\": container with ID starting with ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2 not found: ID does not exist" containerID="ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2" Sep 18 16:34:50.878584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.878515 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2"} err="failed to get container status \"ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2\": rpc error: code = NotFound desc = could not find container \"ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2\": container with ID starting with ee6d087d0aca12b64aa3bc354e42bb9a15a3a5be670944498eca25272cdfd8e2 not found: ID does not exist" Sep 18 16:34:50.878584 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.878538 2563 scope.go:117] "RemoveContainer" containerID="fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10" Sep 18 16:34:50.887582 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.887566 2563 scope.go:117] "RemoveContainer" containerID="fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10" Sep 18 16:34:50.887878 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:34:50.887860 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10\": container with ID starting with fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10 not found: ID does not exist" containerID="fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10" Sep 18 16:34:50.887972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.887886 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10"} err="failed to get container status \"fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10\": rpc error: code = NotFound desc = could not find container \"fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10\": container with ID starting with fa7251e259aa1f432a12da93ab5fe15fff9ec185ed8df5a2840d98e87742ec10 not found: ID does not exist" Sep 18 16:34:50.887972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.887901 2563 scope.go:117] "RemoveContainer" containerID="ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642" Sep 18 16:34:50.895940 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.895902 2563 scope.go:117] "RemoveContainer" containerID="ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642" Sep 18 16:34:50.896236 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:34:50.896215 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642\": container with ID starting with ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642 not found: ID does not exist" containerID="ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642" Sep 18 16:34:50.896326 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.896239 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642"} err="failed to get container status \"ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642\": rpc error: code = NotFound desc = could not find container \"ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642\": container with ID starting with ad78849927abbe1eef01aedc91ea2024b77312e1e286c9728716d9281be05642 not found: ID does not exist" Sep 18 16:34:50.923124 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.923073 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbdjs\" (UniqueName: \"kubernetes.io/projected/2d162f08-db46-4065-ad8e-e61620b5c225-kube-api-access-qbdjs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.923124 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.923126 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d162f08-db46-4065-ad8e-e61620b5c225-plugins-config-volume\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:34:50.927566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.927541 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j"] Sep 18 16:34:50.941676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.941646 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4f2d8a-7b49765847-ftw5j"] Sep 18 16:34:50.977177 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.977115 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4"] Sep 18 16:34:50.978835 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.978816 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:50.990671 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:50.990643 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4f2d8a-6f66d947b9-kq5k4"] Sep 18 16:34:51.028554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.028522 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp"] Sep 18 16:34:51.055876 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.055852 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4f2d8a-5fff85c574-6w7zp"] Sep 18 16:34:51.081339 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.081313 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d162f08-db46-4065-ad8e-e61620b5c225" path="/var/lib/kubelet/pods/2d162f08-db46-4065-ad8e-e61620b5c225/volumes" Sep 18 16:34:51.081683 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.081670 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" path="/var/lib/kubelet/pods/6fef0cc0-a218-4e16-b4bd-b06dad7d3e06/volumes" Sep 18 16:34:51.082012 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.082000 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d61092-8da2-4fa0-a0ab-e189d8742447" path="/var/lib/kubelet/pods/e3d61092-8da2-4fa0-a0ab-e189d8742447/volumes" Sep 18 16:34:51.270578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.270550 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:51.703412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:51.703382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:53.420501 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:53.420475 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:55.004462 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:55.004425 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:55.134955 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:55.134923 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:59.299819 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:59.299787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:59.475549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:59.475516 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:34:59.782326 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:34:59.782295 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:03.166185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:03.166150 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:35:03.168731 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:03.168712 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:35:14.650372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:14.650332 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:14.752518 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:14.752482 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:15.349361 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:15.349330 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:33.269419 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:33.269385 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:41.279321 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:41.279285 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:42.371502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:42.371472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:35:50.238507 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:35:50.238473 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:36:03.318712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:03.318677 2563 scope.go:117] "RemoveContainer" containerID="83dce14125b8ad3a4a806d336acfaf875027f32d2be97abf30cd062146a1d2a6" Sep 18 16:36:03.984687 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:03.984649 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:36:06.386718 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:06.386686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:36:14.273756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:14.273682 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:36:30.548312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:30.548273 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:36:49.030559 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:49.030523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:36:50.357408 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:36:50.357370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:37:09.193232 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:37:09.193194 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:37:10.288595 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:37:10.288548 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:37:13.165293 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:37:13.165262 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:37:44.390888 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:37:44.390802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:38:00.917956 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:00.917920 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="526a10f0e65c1323900408c857f46963e15d6afb8cbd5666118fb13052cff717" exitCode=1 Sep 18 16:38:00.918353 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:00.917997 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"526a10f0e65c1323900408c857f46963e15d6afb8cbd5666118fb13052cff717"} Sep 18 16:38:00.918404 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:00.918392 2563 scope.go:117] "RemoveContainer" containerID="526a10f0e65c1323900408c857f46963e15d6afb8cbd5666118fb13052cff717" Sep 18 16:38:00.918706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:00.918692 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:38:01.470762 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:01.470739 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 16:38:01.925744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:01.925713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"cd880fe8db8ad720b8c719bd20cb7860deb3c5e884a37cff24b6996b5d056e25"} Sep 18 16:38:01.926174 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:01.925911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:38:02.933516 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:02.933485 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="cab643eea50a8a73216b23002672ba88fe7fb5ffb56fcaa1c856fd1bbd9f1fcb" exitCode=1 Sep 18 16:38:02.933889 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:02.933570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"cab643eea50a8a73216b23002672ba88fe7fb5ffb56fcaa1c856fd1bbd9f1fcb"} Sep 18 16:38:02.934016 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:02.933998 2563 scope.go:117] "RemoveContainer" containerID="cab643eea50a8a73216b23002672ba88fe7fb5ffb56fcaa1c856fd1bbd9f1fcb" Sep 18 16:38:02.935194 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:02.935137 2563 generic.go:356] "Generic (PLEG): container finished" podID="132cdab9-9fc4-4348-b41a-bb96fd4116c7" containerID="6e1cd407b1fdfec4d1f3be7ac8aa7eee0f44031043cac479fc351d073894e66b" exitCode=1 Sep 18 16:38:02.935272 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:02.935212 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" event={"ID":"132cdab9-9fc4-4348-b41a-bb96fd4116c7","Type":"ContainerDied","Data":"6e1cd407b1fdfec4d1f3be7ac8aa7eee0f44031043cac479fc351d073894e66b"} Sep 18 16:38:02.935671 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:02.935654 2563 scope.go:117] "RemoveContainer" containerID="6e1cd407b1fdfec4d1f3be7ac8aa7eee0f44031043cac479fc351d073894e66b" Sep 18 16:38:03.919668 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:03.919627 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:38:03.943168 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:03.943134 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"26ab7c97ef1672a33131c5078d053af90a540b00e1c099721dfa2b80de766840"} Sep 18 16:38:03.943588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:03.943333 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:38:03.944974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:03.944953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" event={"ID":"132cdab9-9fc4-4348-b41a-bb96fd4116c7","Type":"ContainerStarted","Data":"66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17"} Sep 18 16:38:03.945143 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:03.945129 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:38:12.937703 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:12.937661 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:14.952568 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:14.952528 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:14.953047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:14.953029 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:38:19.631434 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:19.631389 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:09Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:09Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:09Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:09Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-132-41.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal/status?timeout=10s\": context deadline exceeded" Sep 18 16:38:19.749208 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:19.749168 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:38:22.936172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:22.936136 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:23.919586 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:23.919543 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:24.951505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:24.951460 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:32.673385 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:32.673340 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:32.936181 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:32.936062 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:34.922871 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:34.922781 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d676687603636d opendatahub 20912 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/maas-controller@sha256:661d832135c8b074e952db3713524d6455658541eeda4bd0dd5b51592933f34e\",Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:14:52 +0000 UTC,LastTimestamp:2026-09-18 16:38:00.918850752 +0000 UTC m=+1980.479695398,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:38:34.951380 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:34.951337 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:36.044441 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:36.044393 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:26Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:26Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:26Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:26Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-132-41.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal/status?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 16:38:42.935922 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:42.935875 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:43.919587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:43.919546 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:44.951766 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:44.951707 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:46.045179 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:46.045144 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:38:46.084500 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:46.084463 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:38:52.672887 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:52.672838 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:52.936051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:52.935960 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:38:54.951124 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:38:54.951062 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:38:56.046395 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:56.046347 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:38:56.084911 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:38:56.084874 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:39:00.920629 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:00.920588 2563 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods maas-controller-7b7898d6ff-jkzmr)" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:39:02.268102 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:02.268051 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="cd880fe8db8ad720b8c719bd20cb7860deb3c5e884a37cff24b6996b5d056e25" exitCode=1 Sep 18 16:39:02.268480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:02.268125 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"cd880fe8db8ad720b8c719bd20cb7860deb3c5e884a37cff24b6996b5d056e25"} Sep 18 16:39:02.268480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:02.268217 2563 scope.go:117] "RemoveContainer" containerID="526a10f0e65c1323900408c857f46963e15d6afb8cbd5666118fb13052cff717" Sep 18 16:39:02.268593 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:02.268571 2563 scope.go:117] "RemoveContainer" containerID="cd880fe8db8ad720b8c719bd20cb7860deb3c5e884a37cff24b6996b5d056e25" Sep 18 16:39:02.268781 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:02.268761 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:39:03.920325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:03.920281 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:03.920794 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:03.920372 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:39:03.920943 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:03.920921 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"26ab7c97ef1672a33131c5078d053af90a540b00e1c099721dfa2b80de766840"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:39:03.921025 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:03.920968 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" containerID="cri-o://26ab7c97ef1672a33131c5078d053af90a540b00e1c099721dfa2b80de766840" gracePeriod=10 Sep 18 16:39:03.921025 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:03.920977 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:04.282743 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:04.282714 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="26ab7c97ef1672a33131c5078d053af90a540b00e1c099721dfa2b80de766840" exitCode=1 Sep 18 16:39:04.282906 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:04.282787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"26ab7c97ef1672a33131c5078d053af90a540b00e1c099721dfa2b80de766840"} Sep 18 16:39:04.282906 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:04.282846 2563 scope.go:117] "RemoveContainer" containerID="cab643eea50a8a73216b23002672ba88fe7fb5ffb56fcaa1c856fd1bbd9f1fcb" Sep 18 16:39:05.290880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:05.290837 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"e1076d14f2f7eb59caa6a41349de30a1f80f9c12860c01cc65a849354f7d1b66"} Sep 18 16:39:05.291290 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:05.290911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:39:06.047027 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:06.046995 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:39:06.047027 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:06.047022 2563 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count" Sep 18 16:39:06.085534 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:06.085501 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:39:08.926770 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:08.926676 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d677abbfc6bd3c opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Successfully pulled image \"quay.io/opendatahub/maas-controller@sha256:661d832135c8b074e952db3713524d6455658541eeda4bd0dd5b51592933f34e\" in 548ms (548ms including waiting). Image size: 198052350 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:01.466936636 +0000 UTC m=+1981.027781297,LastTimestamp:2026-09-18 16:38:01.466936636 +0000 UTC m=+1981.027781297,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:39:12.673529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:12.673479 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:39:12.673929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:12.673833 2563 scope.go:117] "RemoveContainer" containerID="cd880fe8db8ad720b8c719bd20cb7860deb3c5e884a37cff24b6996b5d056e25" Sep 18 16:39:13.334276 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:13.334237 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"8ffd6a707a6c00b53934aa0c9f27309fef7ff1f12f342169a3757f23efe716b0"} Sep 18 16:39:13.334470 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:13.334330 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:39:16.086247 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:16.086209 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:39:16.297864 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:16.297821 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:23.920101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:23.920037 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:24.341475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:24.341433 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:39:26.053933 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:26.053884 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:16Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:16Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:16Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:16Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-132-41.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal/status?timeout=10s\": context deadline exceeded" Sep 18 16:39:26.087379 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:26.087344 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:39:26.087379 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:26.087379 2563 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Sep 18 16:39:26.297294 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:26.297253 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:32.672918 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:32.672876 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:39:32.673322 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:32.672959 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:39:32.673386 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:32.673369 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"8ffd6a707a6c00b53934aa0c9f27309fef7ff1f12f342169a3757f23efe716b0"} pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:39:32.673425 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:32.673406 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" containerID="cri-o://8ffd6a707a6c00b53934aa0c9f27309fef7ff1f12f342169a3757f23efe716b0" gracePeriod=30 Sep 18 16:39:32.673503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:32.673474 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:39:33.449072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:33.448990 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="8ffd6a707a6c00b53934aa0c9f27309fef7ff1f12f342169a3757f23efe716b0" exitCode=2 Sep 18 16:39:33.449072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:33.449041 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"8ffd6a707a6c00b53934aa0c9f27309fef7ff1f12f342169a3757f23efe716b0"} Sep 18 16:39:33.449325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:33.449106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"b381712becd8fb4dbb2264e88795c330e8af4170ec359d4626306ec40ab03b8e"} Sep 18 16:39:33.449325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:33.449129 2563 scope.go:117] "RemoveContainer" containerID="cd880fe8db8ad720b8c719bd20cb7860deb3c5e884a37cff24b6996b5d056e25" Sep 18 16:39:33.449325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:33.449275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:39:36.054425 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:36.054387 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:39:36.088182 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:36.088144 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="200ms" Sep 18 16:39:36.296597 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:36.296556 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:42.930802 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:42.930714 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d676691fb87f09 opendatahub 20944 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:14:55 +0000 UTC,LastTimestamp:2026-09-18 16:38:01.528206424 +0000 UTC m=+1981.089051091,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:39:43.920140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:43.920071 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:44.457809 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:44.457770 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:39:46.055270 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:46.055233 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:39:46.289697 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:46.289658 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="400ms" Sep 18 16:39:46.297091 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:46.297058 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:52.673482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:52.673439 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:39:54.457503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:54.457464 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:39:56.056147 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:56.056102 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:39:56.297246 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:39:56.297201 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:39:56.690855 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:39:56.690811 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="800ms" Sep 18 16:40:00.923257 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:00.923208 2563 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods maas-controller-7b7898d6ff-jkzmr)" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:40:01.603667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:01.603634 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e14a582-6f0c-454e-87c9-73fbb5852911" containerID="3603836f51dc4046b8275cbc276d9c1d03e5cdd239f6ab31ee61fc4625a73961" exitCode=1 Sep 18 16:40:01.603847 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:01.603685 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerDied","Data":"3603836f51dc4046b8275cbc276d9c1d03e5cdd239f6ab31ee61fc4625a73961"} Sep 18 16:40:01.603989 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:01.603976 2563 scope.go:117] "RemoveContainer" containerID="3603836f51dc4046b8275cbc276d9c1d03e5cdd239f6ab31ee61fc4625a73961" Sep 18 16:40:02.611009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:02.610974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerStarted","Data":"3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd"} Sep 18 16:40:03.204449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.204421 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:40:03.208907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.208874 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:40:03.920414 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.920375 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:03.920796 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.920451 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:40:03.920891 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.920872 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"e1076d14f2f7eb59caa6a41349de30a1f80f9c12860c01cc65a849354f7d1b66"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:40:03.920952 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.920908 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" containerID="cri-o://e1076d14f2f7eb59caa6a41349de30a1f80f9c12860c01cc65a849354f7d1b66" gracePeriod=10 Sep 18 16:40:03.921008 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:03.920970 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:04.457725 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:04.457683 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:04.631581 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:04.631549 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="e1076d14f2f7eb59caa6a41349de30a1f80f9c12860c01cc65a849354f7d1b66" exitCode=1 Sep 18 16:40:04.631769 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:04.631604 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"e1076d14f2f7eb59caa6a41349de30a1f80f9c12860c01cc65a849354f7d1b66"} Sep 18 16:40:04.631769 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:04.631669 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"0ee7e920496e320a491bacf93bfde47395e2daa66d10c3ca631a3a7c71b8c30a"} Sep 18 16:40:04.631769 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:04.631691 2563 scope.go:117] "RemoveContainer" containerID="26ab7c97ef1672a33131c5078d053af90a540b00e1c099721dfa2b80de766840" Sep 18 16:40:04.631946 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:04.631907 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:40:06.057160 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:06.057121 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:40:06.057160 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:06.057150 2563 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count" Sep 18 16:40:07.492394 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:07.492353 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="1.6s" Sep 18 16:40:12.672920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:12.672873 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:14.457734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:14.457683 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:15.641548 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:15.641511 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:16.934961 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:16.934877 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d67669202c6568 opendatahub 20947 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:14:55 +0000 UTC,LastTimestamp:2026-09-18 16:38:01.535627247 +0000 UTC m=+1981.096471915,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:40:19.093213 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:19.093170 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="3.2s" Sep 18 16:40:23.740238 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:23.740205 2563 generic.go:356] "Generic (PLEG): container finished" podID="f25112d0-8747-4ead-8f96-9b968a57d914" containerID="359b3bdb0ecfe3fc6f67c0fe750e99c6030fe01efc93af3e2a0dc6e5230a81df" exitCode=0 Sep 18 16:40:23.740603 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:23.740278 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerDied","Data":"359b3bdb0ecfe3fc6f67c0fe750e99c6030fe01efc93af3e2a0dc6e5230a81df"} Sep 18 16:40:23.740603 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:23.740582 2563 scope.go:117] "RemoveContainer" containerID="359b3bdb0ecfe3fc6f67c0fe750e99c6030fe01efc93af3e2a0dc6e5230a81df" Sep 18 16:40:23.920401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:23.920361 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:24.457401 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:24.457350 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:24.749131 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:24.749072 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerStarted","Data":"cd877fe97c993fc832e8d4a23a1cf5804523f6f29f37c9dc1b3f569cfe2cdabe"} Sep 18 16:40:25.640918 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:25.640869 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:32.293812 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:32.293767 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="6.4s" Sep 18 16:40:32.673185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.673078 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:32.673185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.673178 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:40:32.673587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.673570 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"b381712becd8fb4dbb2264e88795c330e8af4170ec359d4626306ec40ab03b8e"} pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:40:32.673631 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.673603 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" containerID="cri-o://b381712becd8fb4dbb2264e88795c330e8af4170ec359d4626306ec40ab03b8e" gracePeriod=30 Sep 18 16:40:32.673670 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.673567 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:32.794008 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.793980 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="b381712becd8fb4dbb2264e88795c330e8af4170ec359d4626306ec40ab03b8e" exitCode=2 Sep 18 16:40:32.794110 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.794056 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"b381712becd8fb4dbb2264e88795c330e8af4170ec359d4626306ec40ab03b8e"} Sep 18 16:40:32.794176 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:32.794153 2563 scope.go:117] "RemoveContainer" containerID="8ffd6a707a6c00b53934aa0c9f27309fef7ff1f12f342169a3757f23efe716b0" Sep 18 16:40:33.801099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:33.801055 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430"} Sep 18 16:40:33.801505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:33.801154 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:40:35.641207 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:35.641155 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:43.919911 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:43.919825 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:44.808043 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:44.807999 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:45.640676 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:45.640634 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:40:48.694784 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:48.694743 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:40:50.938687 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:40:50.938599 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d67638e45cb2ab opendatahub 13348 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2\",Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:11:28 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.934376873 +0000 UTC m=+1982.495221519,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:40:52.673360 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:52.673314 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:54.806950 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:54.806903 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:40:55.641054 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:40:55.641005 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:03.920417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:03.920369 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:03.921005 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:03.920467 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:41:03.921103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:03.921043 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"0ee7e920496e320a491bacf93bfde47395e2daa66d10c3ca631a3a7c71b8c30a"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:41:03.921103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:03.921056 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:03.921217 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:03.921108 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" containerID="cri-o://0ee7e920496e320a491bacf93bfde47395e2daa66d10c3ca631a3a7c71b8c30a" gracePeriod=10 Sep 18 16:41:04.807866 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:04.807824 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:41:04.980441 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:04.980408 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="0ee7e920496e320a491bacf93bfde47395e2daa66d10c3ca631a3a7c71b8c30a" exitCode=1 Sep 18 16:41:04.980897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:04.980482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"0ee7e920496e320a491bacf93bfde47395e2daa66d10c3ca631a3a7c71b8c30a"} Sep 18 16:41:04.980897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:04.980557 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e"} Sep 18 16:41:04.980897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:04.980596 2563 scope.go:117] "RemoveContainer" containerID="e1076d14f2f7eb59caa6a41349de30a1f80f9c12860c01cc65a849354f7d1b66" Sep 18 16:41:04.980897 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:04.980806 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:41:05.695938 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:05.695891 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:41:12.673859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:12.673807 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:41:14.807037 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:14.806989 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:41:15.990383 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:15.990343 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:16.690573 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:16.690513 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"132cdab9-9fc4-4348-b41a-bb96fd4116c7\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:38:14Z\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:38:14Z\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"1\\\",\\\"memory\\\":\\\"1Gi\\\"},\\\"containerID\\\":\\\"cri-o://66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17\\\",\\\"image\\\":\\\"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f\\\",\\\"imageID\\\":\\\"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:70099611ea7781bb7b113af98e8a85ddf0ab01d5fe804cf4fec44d66a569940f\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://6e1cd407b1fdfec4d1f3be7ac8aa7eee0f44031043cac479fc351d073894e66b\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-09-18T16:38:02Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-09-18T16:11:43Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"1\\\",\\\"memory\\\":\\\"1Gi\\\"}},\\\"restartCount\\\":1,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-18T16:38:02Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000630000],\\\"uid\\\":1000630000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/controller_manager_config.yaml\\\",\\\"name\\\":\\\"manager-config\\\"},{\\\"mountPath\\\":\\\"/tmp/k8s-webhook-server/serving-certs\\\",\\\"name\\\":\\\"cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/tmp/k8s-metrics-server/serving-certs\\\",\\\"name\\\":\\\"metrics-cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-frdjq\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-lws-operator\"/\"lws-controller-manager-6847d847f7-fgmrh\": Timeout: request did not complete within requested timeout - context deadline exceeded" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:41:17.052356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:17.052328 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/1.log" Sep 18 16:41:17.052744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:17.052366 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="456cd87ce21ac2ee19105a5790b3e6fd7365712ae68cced02849f8c345e4e76e" exitCode=0 Sep 18 16:41:17.052744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:17.052451 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"456cd87ce21ac2ee19105a5790b3e6fd7365712ae68cced02849f8c345e4e76e"} Sep 18 16:41:17.052744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:17.052517 2563 scope.go:117] "RemoveContainer" containerID="a6fe731ad6e640233584d6e457d699dfa5ad7bb312dde147345d7a92cf0e756d" Sep 18 16:41:17.052869 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:17.052788 2563 scope.go:117] "RemoveContainer" containerID="456cd87ce21ac2ee19105a5790b3e6fd7365712ae68cced02849f8c345e4e76e" Sep 18 16:41:18.059332 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:18.059299 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb"} Sep 18 16:41:18.059724 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:18.059567 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:41:18.059808 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:18.059787 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:18.059860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:18.059842 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:19.065699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:19.065667 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:19.066122 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:19.065716 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:22.697292 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:22.697244 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:41:23.919559 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:23.919513 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:24.807451 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:24.807407 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:41:24.942165 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:24.942058 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{lws-controller-manager-6847d847f7-fgmrh.18d677ac175d6596 openshift-lws-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-lws-operator,Name:lws-controller-manager-6847d847f7-fgmrh,UID:132cdab9-9fc4-4348-b41a-bb96fd4116c7,APIVersion:v1,ResourceVersion:13564,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Container image \"registry.redhat.io/leader-worker-set/lws-rhel9@sha256:affb303b1173c273231bb50fef07310b0e220d2f08bfc0aa5912d0825e3e0d4f\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:02.936427926 +0000 UTC m=+1982.497272571,LastTimestamp:2026-09-18 16:38:02.936427926 +0000 UTC m=+1982.497272571,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:41:25.989205 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:25.989161 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:26.074554 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:26.074519 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:26.074696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:26.074571 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:29.066524 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:29.066487 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:29.066894 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:29.066553 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:29.423645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:29.423598 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:41:30.126485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:30.126453 2563 generic.go:356] "Generic (PLEG): container finished" podID="1a61c3d2-0189-4126-838d-884662540d50" containerID="035520c86ab20c0b77472b2b8b3d2e3c9046123ea5f36dec2fdcb3e567e79a8e" exitCode=0 Sep 18 16:41:30.126854 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:30.126522 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" event={"ID":"1a61c3d2-0189-4126-838d-884662540d50","Type":"ContainerDied","Data":"035520c86ab20c0b77472b2b8b3d2e3c9046123ea5f36dec2fdcb3e567e79a8e"} Sep 18 16:41:30.126854 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:30.126815 2563 scope.go:117] "RemoveContainer" containerID="035520c86ab20c0b77472b2b8b3d2e3c9046123ea5f36dec2fdcb3e567e79a8e" Sep 18 16:41:31.133253 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:31.133217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" event={"ID":"1a61c3d2-0189-4126-838d-884662540d50","Type":"ContainerStarted","Data":"7cc601ca3095428e8bb0e749aca3ba6f384ef77cda4ee2703b936e78449f45a5"} Sep 18 16:41:32.673706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:32.673669 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:41:32.674072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:32.673738 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:41:32.674185 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:32.674167 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430"} pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:41:32.674225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:32.674202 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" containerID="cri-o://686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" gracePeriod=30 Sep 18 16:41:32.674286 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:32.674250 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:41:32.790958 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:32.790928 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:41:33.146391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:33.146350 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" exitCode=2 Sep 18 16:41:33.146594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:33.146403 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430"} Sep 18 16:41:33.146594 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:33.146463 2563 scope.go:117] "RemoveContainer" containerID="b381712becd8fb4dbb2264e88795c330e8af4170ec359d4626306ec40ab03b8e" Sep 18 16:41:33.146792 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:33.146771 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:41:33.147022 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:33.147001 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:41:35.989394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:35.989354 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:36.074183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:36.074145 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:36.074357 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:36.074198 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:39.065824 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:39.065782 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:39.066271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:39.065838 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:39.698962 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:39.698915 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:41:42.158282 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:42.158242 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-132-41.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:41:43.919972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:43.919934 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:44.075133 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:44.075107 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:41:44.075326 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:44.075300 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:41:45.990127 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:45.990062 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:46.074761 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.074730 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:46.074919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.074777 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:46.074919 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.074816 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:41:46.075263 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.075240 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb"} pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" containerMessage="Container console-operator failed liveness probe, will be restarted" Sep 18 16:41:46.075311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.075268 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:46.075311 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.075282 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" containerID="cri-o://53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb" gracePeriod=30 Sep 18 16:41:46.075387 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:46.075307 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:47.225701 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.225673 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/3.log" Sep 18 16:41:47.226105 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.225992 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb" exitCode=255 Sep 18 16:41:47.226105 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.226069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb"} Sep 18 16:41:47.226188 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.226145 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e"} Sep 18 16:41:47.226188 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.226167 2563 scope.go:117] "RemoveContainer" containerID="456cd87ce21ac2ee19105a5790b3e6fd7365712ae68cced02849f8c345e4e76e" Sep 18 16:41:47.226381 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.226357 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:41:47.226588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.226570 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:47.226679 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:47.226611 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:48.233305 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:48.233281 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/3.log" Sep 18 16:41:48.233772 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:48.233749 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:48.233830 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:48.233802 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:52.158470 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:52.158430 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:41:55.075140 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:55.075107 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:41:55.075612 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:55.075280 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:41:55.989807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:55.989770 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:41:56.073865 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:56.073834 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:56.073996 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:56.073879 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:56.485037 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:56.484988 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2e14a582-6f0c-454e-87c9-73fbb5852911\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:40:02Z\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:40:02Z\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"containerID\\\":\\\"cri-o://3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd\\\",\\\"image\\\":\\\"registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:f3683d29aa703829fb0d0ba93711188fb717cc1ac156877c9d0ab5fb5083513f\\\",\\\"imageID\\\":\\\"registry.redhat.io/cert-manager/jetstack-cert-manager-rhel9@sha256:681a5f3e02131df294a2b83933584403fe4035d43d98530a13f83a77f71171fa\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://3603836f51dc4046b8275cbc276d9c1d03e5cdd239f6ab31ee61fc4625a73961\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-09-18T16:40:01Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-09-18T16:11:11Z\\\"}},\\\"name\\\":\\\"cert-manager-controller\\\",\\\"ready\\\":true,\\\"resources\\\":{},\\\"restartCount\\\":1,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-18T16:40:01Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000620000],\\\"uid\\\":1000620000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/secrets/openshift/serviceaccount\\\",\\\"name\\\":\\\"bound-sa-token\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-bp8vk\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"cert-manager\"/\"cert-manager-55b4d954d5-xxvn7\": etcdserver: request timed out" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:41:56.700583 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:56.700547 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:41:57.311195 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:57.311157 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:41:58.234506 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:58.234471 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:41:58.234881 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:41:58.234530 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:41:58.946785 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:41:58.946701 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{lws-controller-manager-6847d847f7-fgmrh.18d6763c509706e4 openshift-lws-operator 13924 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-lws-operator,Name:lws-controller-manager-6847d847f7-fgmrh,UID:132cdab9-9fc4-4348-b41a-bb96fd4116c7,APIVersion:v1,ResourceVersion:13564,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:11:43 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.986744315 +0000 UTC m=+1982.547588983,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:42:03.920303 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:03.920260 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:42:03.920825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:03.920343 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:42:03.920825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:03.920758 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:42:03.920825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:03.920792 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" containerID="cri-o://2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" gracePeriod=10 Sep 18 16:42:03.920984 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:03.920891 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:42:04.037509 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:04.037482 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:42:04.320588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:04.320557 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" exitCode=1 Sep 18 16:42:04.320738 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:04.320624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e"} Sep 18 16:42:04.320738 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:04.320683 2563 scope.go:117] "RemoveContainer" containerID="0ee7e920496e320a491bacf93bfde47395e2daa66d10c3ca631a3a7c71b8c30a" Sep 18 16:42:04.321022 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:04.321009 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:42:04.321222 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:04.321198 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:42:06.074225 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:06.074191 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:06.074611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:06.074246 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:08.234099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:08.234046 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:08.234481 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:08.234122 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:10.074629 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:10.074599 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:42:10.075078 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:10.074784 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:42:13.701827 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:13.701745 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:42:15.075214 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:15.075179 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:42:15.075603 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:15.075436 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:42:16.074331 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074291 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:16.074510 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074350 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:16.074510 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074387 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:42:16.074801 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074783 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e"} pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" containerMessage="Container console-operator failed liveness probe, will be restarted" Sep 18 16:42:16.074851 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074817 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" containerID="cri-o://47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e" gracePeriod=30 Sep 18 16:42:16.074907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074883 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:16.074966 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:16.074925 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:17.398409 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.398381 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/4.log" Sep 18 16:42:17.398856 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.398814 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/3.log" Sep 18 16:42:17.398925 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.398856 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e" exitCode=255 Sep 18 16:42:17.398970 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.398930 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e"} Sep 18 16:42:17.399019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.398978 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa"} Sep 18 16:42:17.399019 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.399001 2563 scope.go:117] "RemoveContainer" containerID="53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb" Sep 18 16:42:17.399316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.399302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:42:17.399530 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.399508 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:17.399598 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:17.399557 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:18.407202 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:18.407171 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/4.log" Sep 18 16:42:18.407769 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:18.407746 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:18.407860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:18.407786 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:24.074689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:24.074657 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:42:24.075126 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:24.074841 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:42:24.614185 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:24.614130 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"13e5f050-0e55-4386-8e80-8927af0ec4d3\\\"},\\\"status\\\":{\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"},\\\"containerID\\\":\\\"cri-o://47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e\\\",\\\"image\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff12d08d05379ba6cf62fe8dd9e33da283be9c68b68f66ada7dad62f9fadd6b5\\\",\\\"imageID\\\":\\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8c4ba923ce15361076a7adeeff01173eb3e7318c30532ffac962f1dd6e41166d\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://53e803177c3fc54a4637a68a1e687393da41c6f6afe9c3499455c0d939c21fcb\\\",\\\"exitCode\\\":255,\\\"finishedAt\\\":\\\"2026-09-18T16:41:46Z\\\",\\\"message\\\":\\\"I0918 16:41:17.164043 1 cmd.go:253] Using service-serving-cert provided certificates\\\\nI0918 16:41:17.164163 1 leaderelection.go:121] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}.\\\\nI0918 16:41:17.164485 1 observer_polling.go:159] Starting file observer\\\\nI0918 16:41:46.077800 1 cmd.go:141] Received SIGTERM or SIGINT signal, shutting down controller.\\\\nW0918 16:41:46.077920 1 builder.go:272] unable to get owner reference (falling back to namespace): Get \\\\\\\"https://172.31.0.1:443/apis/apps/v1/namespaces/openshift-console-operator/replicasets/console-operator-fd4bb46c\\\\\\\": context canceled\\\\nI0918 16:41:46.078055 1 builder.go:304] console-operator version -\\\\nF0918 16:41:46.517040 1 cmd.go:182] failed checking apiserver connectivity: client rate limiter Wait returned an error: context canceled\\\\n\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-09-18T16:41:17Z\\\"}},\\\"name\\\":\\\"console-operator\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"100Mi\\\"}},\\\"restartCount\\\":4,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-18T16:41:46Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000220000],\\\"uid\\\":1000220000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/var/run/configmaps/config\\\",\\\"name\\\":\\\"config\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/serving-cert\\\",\\\"name\\\":\\\"serving-cert\\\"},{\\\"mountPath\\\":\\\"/etc/pki/ca-trust/extracted/pem\\\",\\\"name\\\":\\\"trusted-ca\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-9cd9b\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"openshift-console-operator\"/\"console-operator-fd4bb46c-jd8k8\": etcdserver: request timed out" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:42:26.074377 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:26.074332 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:26.074744 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:26.074392 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:28.408101 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:28.408049 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:28.408495 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:28.408150 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:29.075038 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:29.075005 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:42:29.075239 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:29.075207 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:42:30.702893 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:30.702857 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:42:32.953881 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:32.953786 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{lws-controller-manager-6847d847f7-fgmrh.18d6763c5109730b openshift-lws-operator 13925 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-lws-operator,Name:lws-controller-manager-6847d847f7-fgmrh,UID:132cdab9-9fc4-4348-b41a-bb96fd4116c7,APIVersion:v1,ResourceVersion:13564,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:11:43 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.994496568 +0000 UTC m=+1982.555341232,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:42:36.074353 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:36.074312 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:36.074743 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:36.074366 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:38.074367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:38.074333 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:42:38.074760 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:38.074515 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:42:38.408495 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:38.408415 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:38.408495 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:38.408468 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:42.075256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:42.075218 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:42:42.075709 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:42.075422 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:42:46.074809 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.074756 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:46.075243 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.074840 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:46.075243 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.074879 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:42:46.075395 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.075371 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:42:46.075440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.075406 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa"} pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" containerMessage="Container console-operator failed liveness probe, will be restarted" Sep 18 16:42:46.075440 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.075426 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:42:46.075528 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.075441 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" containerID="cri-o://0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" gracePeriod=30 Sep 18 16:42:46.392096 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:46.392059 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:42:46.559849 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.559818 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/5.log" Sep 18 16:42:46.560289 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.560273 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/4.log" Sep 18 16:42:46.560384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.560305 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" exitCode=255 Sep 18 16:42:46.560384 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.560361 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa"} Sep 18 16:42:46.560455 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.560405 2563 scope.go:117] "RemoveContainer" containerID="47f5078c065813d6c1fbb93f37e53361528bdbf076b8c2d8455f2fc69d641b8e" Sep 18 16:42:46.560736 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:46.560710 2563 scope.go:117] "RemoveContainer" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" Sep 18 16:42:46.560916 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:46.560897 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:42:47.567268 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:47.567237 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/5.log" Sep 18 16:42:47.704075 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:47.704027 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:42:53.075222 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:53.075184 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:42:54.606629 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:54.606594 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e"} Sep 18 16:42:54.607030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:54.606808 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:42:56.074776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:56.074741 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:42:56.075198 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:56.074915 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:42:58.267846 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:58.267798 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:48Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:48Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:48Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:42:48Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-132-41.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:42:59.075132 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:42:59.075075 2563 scope.go:117] "RemoveContainer" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" Sep 18 16:42:59.075312 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:42:59.075293 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:43:04.706379 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:04.706330 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:43:05.614010 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:05.613971 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:06.958221 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:06.958127 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d677ac31b04b6d opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Successfully pulled image \"quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2\" in 443ms (443ms including waiting). Image size: 293036549 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:03.378068333 +0000 UTC m=+1982.938912999,LastTimestamp:2026-09-18 16:38:03.378068333 +0000 UTC m=+1982.938912999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:43:08.268766 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:08.268727 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:43:11.075475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:11.075439 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:43:11.075923 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:11.075605 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:43:12.672837 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:12.672790 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:14.074879 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:14.074846 2563 scope.go:117] "RemoveContainer" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" Sep 18 16:43:14.075369 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:14.075014 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:43:15.612660 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:15.612608 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:18.269921 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:18.269878 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:43:21.706907 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:21.706870 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:43:24.620228 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:24.620176 2563 status_manager.go:1018] "Failed to get status for pod" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/cert-manager/pods/cert-manager-55b4d954d5-xxvn7\": stream error: stream ID 4645; INTERNAL_ERROR; received from peer" podUID="2e14a582-6f0c-454e-87c9-73fbb5852911" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:43:25.074454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:25.074420 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:43:25.074632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:25.074544 2563 scope.go:117] "RemoveContainer" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" Sep 18 16:43:25.074703 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:25.074683 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:43:25.074748 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:25.074717 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:43:25.613174 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:25.613134 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:25.779448 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:25.779409 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe"} Sep 18 16:43:25.779922 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:25.779767 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:43:27.793341 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:27.793306 2563 generic.go:356] "Generic (PLEG): container finished" podID="1a61c3d2-0189-4126-838d-884662540d50" containerID="7cc601ca3095428e8bb0e749aca3ba6f384ef77cda4ee2703b936e78449f45a5" exitCode=255 Sep 18 16:43:27.793756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:27.793353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" event={"ID":"1a61c3d2-0189-4126-838d-884662540d50","Type":"ContainerDied","Data":"7cc601ca3095428e8bb0e749aca3ba6f384ef77cda4ee2703b936e78449f45a5"} Sep 18 16:43:27.793756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:27.793397 2563 scope.go:117] "RemoveContainer" containerID="035520c86ab20c0b77472b2b8b3d2e3c9046123ea5f36dec2fdcb3e567e79a8e" Sep 18 16:43:27.793756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:27.793689 2563 scope.go:117] "RemoveContainer" containerID="7cc601ca3095428e8bb0e749aca3ba6f384ef77cda4ee2703b936e78449f45a5" Sep 18 16:43:27.793899 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:27.793875 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cert-manager-operator pod=cert-manager-operator-controller-manager-55c444bf86-2r5cn_cert-manager-operator(1a61c3d2-0189-4126-838d-884662540d50)\"" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" podUID="1a61c3d2-0189-4126-838d-884662540d50" Sep 18 16:43:28.270738 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:28.270710 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:43:32.673682 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:32.673582 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:32.826456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:32.826423 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e14a582-6f0c-454e-87c9-73fbb5852911" containerID="3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd" exitCode=1 Sep 18 16:43:32.826656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:32.826505 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerDied","Data":"3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd"} Sep 18 16:43:32.826656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:32.826570 2563 scope.go:117] "RemoveContainer" containerID="3603836f51dc4046b8275cbc276d9c1d03e5cdd239f6ab31ee61fc4625a73961" Sep 18 16:43:32.826931 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:32.826910 2563 scope.go:117] "RemoveContainer" containerID="3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd" Sep 18 16:43:32.827166 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:32.827145 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-controller\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cert-manager-controller pod=cert-manager-55b4d954d5-xxvn7_cert-manager(2e14a582-6f0c-454e-87c9-73fbb5852911)\"" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" podUID="2e14a582-6f0c-454e-87c9-73fbb5852911" Sep 18 16:43:35.612486 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:35.612443 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:36.786873 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:36.786831 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:43:38.075018 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:38.074984 2563 scope.go:117] "RemoveContainer" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" Sep 18 16:43:38.271159 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:38.271119 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:43:38.271159 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:38.271150 2563 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count" Sep 18 16:43:38.708068 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:38.708029 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:43:38.862233 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:38.862204 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/5.log" Sep 18 16:43:38.862376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:38.862327 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"c5fb90761e2be3c6de41556334cb65108c7baaea25eb9c842221953b28250e9f"} Sep 18 16:43:38.862630 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:38.862601 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:43:38.863712 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:38.863683 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:43:38.863827 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:38.863734 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:43:39.869073 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:39.869034 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:43:39.869475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:39.869113 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:43:40.966495 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:40.966408 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d67638fcbea76b opendatahub 13360 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:11:29 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.441186746 +0000 UTC m=+1983.002031414,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:43:40.998694 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:40.998657 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:43:40.999044 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:40.999028 2563 scope.go:117] "RemoveContainer" containerID="3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd" Sep 18 16:43:40.999275 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:40.999254 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-controller\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cert-manager-controller pod=cert-manager-55b4d954d5-xxvn7_cert-manager(2e14a582-6f0c-454e-87c9-73fbb5852911)\"" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" podUID="2e14a582-6f0c-454e-87c9-73fbb5852911" Sep 18 16:43:41.074607 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:41.074575 2563 scope.go:117] "RemoveContainer" containerID="7cc601ca3095428e8bb0e749aca3ba6f384ef77cda4ee2703b936e78449f45a5" Sep 18 16:43:41.882062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:41.882027 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2r5cn" event={"ID":"1a61c3d2-0189-4126-838d-884662540d50","Type":"ContainerStarted","Data":"c8c35ecc646e0dc3c6f62993b88f517196979dd0a997688828d30f9d954c052d"} Sep 18 16:43:43.919812 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:43.919727 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:43:45.612480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:45.612433 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:46.074756 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:46.074721 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:43:46.074927 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:46.074776 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:43:46.786072 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:46.786027 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:43:49.869471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:49.869438 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:43:49.869865 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:49.869494 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:43:50.842858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:50.842827 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:43:51.101013 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:51.100930 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:43:52.673050 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.673004 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:52.673545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.673113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:43:52.673662 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.673639 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e"} pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:43:52.673726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.673685 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" containerID="cri-o://560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" gracePeriod=30 Sep 18 16:43:52.673726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.673697 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:43:52.790525 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:52.790496 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:43:52.945517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.945434 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" exitCode=2 Sep 18 16:43:52.945517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.945511 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e"} Sep 18 16:43:52.945714 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.945566 2563 scope.go:117] "RemoveContainer" containerID="686994ebc7a886956c35d5dd0b8e4c17a6d44525dfdaf4480413cdb15edcc430" Sep 18 16:43:52.945920 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:52.945896 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:43:52.946143 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:43:52.946117 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:43:56.074351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:56.074311 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:43:56.074351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:56.074345 2563 scope.go:117] "RemoveContainer" containerID="3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd" Sep 18 16:43:56.074800 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:56.074371 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:43:56.785527 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:56.785489 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:43:56.969863 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:56.969820 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerStarted","Data":"84995ab23782e1ae38c130f7877ac62cb59a778d1e8763ec87cd668274ca2c0c"} Sep 18 16:43:59.870128 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:59.870079 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:43:59.870494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:43:59.870145 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:03.919899 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:03.919857 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:44:06.074880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.074836 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:06.075316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.074903 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:06.075316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.074937 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:44:06.075316 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.074943 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:44:06.075316 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:06.075138 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:44:06.075477 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.075410 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"c5fb90761e2be3c6de41556334cb65108c7baaea25eb9c842221953b28250e9f"} pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" containerMessage="Container console-operator failed liveness probe, will be restarted" Sep 18 16:44:06.075477 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.075441 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" containerID="cri-o://c5fb90761e2be3c6de41556334cb65108c7baaea25eb9c842221953b28250e9f" gracePeriod=30 Sep 18 16:44:06.075544 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.075442 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:06.075587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.075537 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:06.785250 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:06.785215 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:44:07.030023 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.029993 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/6.log" Sep 18 16:44:07.030447 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.030432 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/5.log" Sep 18 16:44:07.030497 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.030466 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="c5fb90761e2be3c6de41556334cb65108c7baaea25eb9c842221953b28250e9f" exitCode=255 Sep 18 16:44:07.030569 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.030545 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"c5fb90761e2be3c6de41556334cb65108c7baaea25eb9c842221953b28250e9f"} Sep 18 16:44:07.030611 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.030601 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326"} Sep 18 16:44:07.030666 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.030621 2563 scope.go:117] "RemoveContainer" containerID="0e8d889aa5f2d1cd1ac0ed4df29a46fe8069750e5e8a2ab84382388e111e83aa" Sep 18 16:44:07.030930 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.030913 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:44:07.031098 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.031062 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:07.031153 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:07.031120 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:08.038916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:08.038885 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/6.log" Sep 18 16:44:08.039472 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:08.039447 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:08.039545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:08.039492 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:14.951238 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:14.951206 2563 patch_prober.go:28] interesting pod/lws-controller-manager-6847d847f7-fgmrh container/manager namespace/openshift-lws-operator: Readiness probe status=failure output="Get \"http://10.132.0.29:8081/readyz\": dial tcp 10.132.0.29:8081: connect: connection refused" start-of-body= Sep 18 16:44:14.951597 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:14.951261 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" podUID="132cdab9-9fc4-4348-b41a-bb96fd4116c7" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.29:8081/readyz\": dial tcp 10.132.0.29:8081: connect: connection refused" Sep 18 16:44:14.970316 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:14.970237 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d67638fd2240c0 opendatahub 13361 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:11:29 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.449208856 +0000 UTC m=+1983.010053522,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:44:15.077620 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:15.077533 2563 generic.go:356] "Generic (PLEG): container finished" podID="132cdab9-9fc4-4348-b41a-bb96fd4116c7" containerID="66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17" exitCode=1 Sep 18 16:44:15.079907 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:15.079881 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" event={"ID":"132cdab9-9fc4-4348-b41a-bb96fd4116c7","Type":"ContainerDied","Data":"66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17"} Sep 18 16:44:15.080043 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:15.079938 2563 scope.go:117] "RemoveContainer" containerID="6e1cd407b1fdfec4d1f3be7ac8aa7eee0f44031043cac479fc351d073894e66b" Sep 18 16:44:15.080299 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:15.080281 2563 scope.go:117] "RemoveContainer" containerID="66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17" Sep 18 16:44:15.080514 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:15.080492 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=lws-controller-manager-6847d847f7-fgmrh_openshift-lws-operator(132cdab9-9fc4-4348-b41a-bb96fd4116c7)\"" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" podUID="132cdab9-9fc4-4348-b41a-bb96fd4116c7" Sep 18 16:44:16.074785 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:16.074743 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:16.075197 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:16.074819 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:16.786103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:16.786043 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:44:17.081359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:17.081276 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:44:17.081701 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:17.081461 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:44:18.039634 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:18.039598 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:18.039817 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:18.039656 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:20.881802 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:20.881753 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:44:21.705872 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:21.705837 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:44:21.706241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:21.706225 2563 scope.go:117] "RemoveContainer" containerID="66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17" Sep 18 16:44:21.706412 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:21.706392 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=lws-controller-manager-6847d847f7-fgmrh_openshift-lws-operator(132cdab9-9fc4-4348-b41a-bb96fd4116c7)\"" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" podUID="132cdab9-9fc4-4348-b41a-bb96fd4116c7" Sep 18 16:44:23.920161 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:23.920117 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:44:23.920642 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:23.920196 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:44:23.920819 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:23.920795 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:44:23.920893 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:23.920844 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" containerID="cri-o://c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" gracePeriod=10 Sep 18 16:44:23.920893 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:23.920864 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:44:24.038157 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:24.038125 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:44:24.133705 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:24.133670 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" exitCode=1 Sep 18 16:44:24.133860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:24.133744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe"} Sep 18 16:44:24.133860 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:24.133804 2563 scope.go:117] "RemoveContainer" containerID="2bf93a82532112497dd4a261b704db8aebd94334063b2a58e055bd465b3bd17e" Sep 18 16:44:24.134148 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:24.134132 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:44:24.134335 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:24.134313 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:44:24.950992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:24.950957 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:44:24.951386 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:24.951340 2563 scope.go:117] "RemoveContainer" containerID="66a5d17c17108ab0fc05124abfb6c8b7d29afac9485bdf5bca235eb4d7f05b17" Sep 18 16:44:25.142502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:25.142466 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" event={"ID":"132cdab9-9fc4-4348-b41a-bb96fd4116c7","Type":"ContainerStarted","Data":"1a1e01cb1b83e5ce9cf852d87ebd039b9ee28c776e5999126224d168085998d1"} Sep 18 16:44:25.142690 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:25.142666 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:44:26.074039 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:26.074000 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:26.074465 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:26.074058 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:28.039888 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:28.039844 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:28.040326 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:28.039899 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:29.074816 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:29.074785 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:44:29.075212 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:29.074975 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:44:30.882310 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:30.882263 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:44:33.019162 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:33.019120 2563 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-132-41.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again" Sep 18 16:44:36.074449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.074408 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:36.074855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.074468 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:36.074855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.074507 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:44:36.074928 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.074916 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326"} pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" containerMessage="Container console-operator failed liveness probe, will be restarted" Sep 18 16:44:36.074965 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.074949 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" containerID="cri-o://3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" gracePeriod=30 Sep 18 16:44:36.075027 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.075006 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:44:36.075075 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.075051 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:44:36.150349 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:36.150320 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6847d847f7-fgmrh" Sep 18 16:44:36.492099 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:36.492043 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:44:37.209207 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:37.209174 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:44:37.209664 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:37.209633 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/6.log" Sep 18 16:44:37.209735 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:37.209675 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" exitCode=255 Sep 18 16:44:37.209787 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:37.209745 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326"} Sep 18 16:44:37.209847 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:37.209802 2563 scope.go:117] "RemoveContainer" containerID="c5fb90761e2be3c6de41556334cb65108c7baaea25eb9c842221953b28250e9f" Sep 18 16:44:37.210181 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:37.210164 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:44:37.210413 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:37.210393 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:44:38.217010 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:38.216980 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:44:39.075475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:39.075440 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:44:39.075661 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:39.075636 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:44:40.075465 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:40.075425 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:44:40.075866 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:40.075658 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:44:48.978423 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:48.978326 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d677ae6b7cab82 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:12.937698178 +0000 UTC m=+1992.498542836,LastTimestamp:2026-09-18 16:38:12.937698178 +0000 UTC m=+1992.498542836,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:44:51.074766 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:51.074729 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:44:51.075271 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:51.074925 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:44:54.074678 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:54.074641 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:44:54.075134 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:44:54.074698 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:44:54.075134 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:54.074855 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:44:54.075134 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:44:54.074861 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:45:03.080423 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:03.080394 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:45:03.080782 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:03.080552 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:45:03.248051 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:03.248023 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:45:03.252149 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:03.252130 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:45:07.075429 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:07.075395 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:45:07.075812 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:07.075608 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:45:07.075812 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:07.075707 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:45:07.075941 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:07.075918 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:45:13.660381 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:13.660299 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:45:14.075211 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:14.075183 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:45:14.075435 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:14.075407 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:45:18.075271 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:18.075240 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:45:18.075691 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:18.075445 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:45:22.074989 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:22.074959 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:45:22.075481 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:22.075162 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:45:22.982219 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:22.982137 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d677aee3950e86 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:14.95256231 +0000 UTC m=+1994.513406968,LastTimestamp:2026-09-18 16:38:14.95256231 +0000 UTC m=+1994.513406968,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:45:23.661469 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:23.661418 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:45:29.074600 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:29.074570 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:45:29.075210 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:29.074730 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:45:32.075256 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:32.075225 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:45:32.075684 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:32.075450 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:45:33.662417 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:33.662378 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:45:34.074368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:34.074343 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:45:34.074536 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:34.074516 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:45:40.074733 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:40.074695 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:45:40.075297 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:40.074877 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:45:42.818657 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:42.818613 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:32Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:32Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:32Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:32Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-132-41.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal/status?timeout=10s\": context deadline exceeded" Sep 18 16:45:43.663157 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:43.663121 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" Sep 18 16:45:45.074785 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:45.074751 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:45:45.075292 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:45.074931 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:45:46.074368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:46.074335 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:45:46.074568 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:46.074499 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:45:52.074679 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:52.074648 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:45:52.075050 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:52.074824 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:45:52.819338 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:52.819298 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:45:53.664278 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:53.664236 2563 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:45:53.664278 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:53.664275 2563 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Sep 18 16:45:56.986557 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:56.986462 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d677ae6b7cab82 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:12.937698178 +0000 UTC m=+1992.498542836,LastTimestamp:2026-09-18 16:38:22.936170487 +0000 UTC m=+2002.497015144,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:45:57.075419 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:57.075388 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:45:57.075623 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:57.075583 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:45:58.074467 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:45:58.074436 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:45:58.074843 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:45:58.074642 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:46:02.819911 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:02.819872 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:46:03.664723 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:03.664683 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="200ms" Sep 18 16:46:07.074824 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:07.074786 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:46:07.075231 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:07.074974 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:46:08.074692 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:08.074659 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:46:08.074867 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:08.074848 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:46:12.075207 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:12.075172 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:46:12.075678 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:12.075364 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:46:12.820955 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:12.820911 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:46:13.865783 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:13.865735 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="400ms" Sep 18 16:46:20.074983 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:20.074949 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:46:20.075398 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:20.075174 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:46:21.074758 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:21.074719 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:46:21.074933 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:21.074886 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:46:22.821771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:22.821731 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-41.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 18 16:46:22.821771 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:22.821765 2563 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count" Sep 18 16:46:23.078144 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:23.078058 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:46:23.078306 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:23.078285 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:46:23.788905 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:23.788872 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e14a582-6f0c-454e-87c9-73fbb5852911" containerID="84995ab23782e1ae38c130f7877ac62cb59a778d1e8763ec87cd668274ca2c0c" exitCode=1 Sep 18 16:46:23.789118 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:23.788945 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerDied","Data":"84995ab23782e1ae38c130f7877ac62cb59a778d1e8763ec87cd668274ca2c0c"} Sep 18 16:46:23.789118 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:23.788997 2563 scope.go:117] "RemoveContainer" containerID="3ccbae9b28b3023da46f5f5282e3e0ce68fd16bb9fce5cb10d5244ab06d334bd" Sep 18 16:46:23.789373 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:23.789354 2563 scope.go:117] "RemoveContainer" containerID="84995ab23782e1ae38c130f7877ac62cb59a778d1e8763ec87cd668274ca2c0c" Sep 18 16:46:23.789576 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:23.789552 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-controller\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cert-manager-controller pod=cert-manager-55b4d954d5-xxvn7_cert-manager(2e14a582-6f0c-454e-87c9-73fbb5852911)\"" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" podUID="2e14a582-6f0c-454e-87c9-73fbb5852911" Sep 18 16:46:24.267615 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:24.267583 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="800ms" Sep 18 16:46:30.990518 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:30.990419 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d677b0fa0ee822 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Liveness probe failed: Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:23.919581218 +0000 UTC m=+2003.480425879,LastTimestamp:2026-09-18 16:38:23.919581218 +0000 UTC m=+2003.480425879,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:46:30.998533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:30.998503 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:46:30.998871 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:30.998855 2563 scope.go:117] "RemoveContainer" containerID="84995ab23782e1ae38c130f7877ac62cb59a778d1e8763ec87cd668274ca2c0c" Sep 18 16:46:30.999030 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:30.999012 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-controller\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cert-manager-controller pod=cert-manager-55b4d954d5-xxvn7_cert-manager(2e14a582-6f0c-454e-87c9-73fbb5852911)\"" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" podUID="2e14a582-6f0c-454e-87c9-73fbb5852911" Sep 18 16:46:32.074657 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:32.074624 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:46:32.075039 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:32.074792 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:46:33.075273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:33.075232 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:46:33.075705 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:33.075450 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:46:35.068449 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:35.068404 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="1.6s" Sep 18 16:46:36.074825 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:36.074792 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:46:36.862192 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:36.862075 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerStarted","Data":"f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42"} Sep 18 16:46:36.862340 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:36.862311 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:46:43.074764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:43.074729 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:46:43.075241 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:43.074953 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:46:45.074954 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:45.074922 2563 scope.go:117] "RemoveContainer" containerID="84995ab23782e1ae38c130f7877ac62cb59a778d1e8763ec87cd668274ca2c0c" Sep 18 16:46:45.075387 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:45.075075 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:46:45.075387 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:45.075247 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:46:45.911348 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:45.911310 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" event={"ID":"2e14a582-6f0c-454e-87c9-73fbb5852911","Type":"ContainerStarted","Data":"29cee0277bc1be82203b3fe044d3f2a39f558aa1343a72fcb37a47c8dd3a7525"} Sep 18 16:46:46.669662 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:46.669620 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="3.2s" Sep 18 16:46:47.869011 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:47.868963 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:46:52.673147 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:52.673106 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:46:57.075110 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:57.075056 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:46:57.075525 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:57.075256 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:46:57.075525 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:57.075322 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:46:57.075525 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:57.075413 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:46:57.868099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:46:57.868052 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:46:59.870869 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:46:59.870824 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="6.4s" Sep 18 16:47:04.994407 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:04.994315 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d677aee3950e86 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:14.95256231 +0000 UTC m=+1994.513406968,LastTimestamp:2026-09-18 16:38:24.951505659 +0000 UTC m=+2004.512350324,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:47:07.867851 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:07.867795 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:47:11.074903 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:11.074870 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:47:12.058060 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:12.058023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c"} Sep 18 16:47:12.058356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:12.058337 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:47:12.074822 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:12.074801 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:47:12.074968 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:12.074948 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:47:12.673433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:12.673384 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:47:16.272678 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:16.272623 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:47:17.868130 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:17.868064 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:47:20.103583 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:20.103554 2563 generic.go:356] "Generic (PLEG): container finished" podID="f25112d0-8747-4ead-8f96-9b968a57d914" containerID="cd877fe97c993fc832e8d4a23a1cf5804523f6f29f37c9dc1b3f569cfe2cdabe" exitCode=0 Sep 18 16:47:20.104007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:20.103627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerDied","Data":"cd877fe97c993fc832e8d4a23a1cf5804523f6f29f37c9dc1b3f569cfe2cdabe"} Sep 18 16:47:20.104007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:20.103695 2563 scope.go:117] "RemoveContainer" containerID="359b3bdb0ecfe3fc6f67c0fe750e99c6030fe01efc93af3e2a0dc6e5230a81df" Sep 18 16:47:20.104007 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:20.103998 2563 scope.go:117] "RemoveContainer" containerID="cd877fe97c993fc832e8d4a23a1cf5804523f6f29f37c9dc1b3f569cfe2cdabe" Sep 18 16:47:20.104219 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:20.104198 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"openshift-lws-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=openshift-lws-operator pod=openshift-lws-operator-fd8ccff4c-wvvkm_openshift-lws-operator(f25112d0-8747-4ead-8f96-9b968a57d914)\"" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" podUID="f25112d0-8747-4ead-8f96-9b968a57d914" Sep 18 16:47:23.066483 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:23.066437 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:47:23.792004 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:23.791957 2563 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods cert-manager-55b4d954d5-xxvn7)" podUID="2e14a582-6f0c-454e-87c9-73fbb5852911" pod="cert-manager/cert-manager-55b4d954d5-xxvn7" Sep 18 16:47:27.074783 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:27.074745 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:47:27.146145 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:27.146121 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:47:27.868342 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:27.868297 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:47:28.159574 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:28.159491 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:47:28.159991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:28.159606 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889"} Sep 18 16:47:28.159991 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:28.159911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:47:28.161042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:28.161017 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:28.161175 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:28.161057 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:29.166040 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:29.166002 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:29.166433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:29.166051 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:32.672916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:32.672871 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:47:32.673328 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:32.672949 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:47:32.673429 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:32.673411 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42"} pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" containerMessage="Container manager failed liveness probe, will be restarted" Sep 18 16:47:32.673473 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:32.673456 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" containerID="cri-o://f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" gracePeriod=30 Sep 18 16:47:32.673508 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:32.673490 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused" Sep 18 16:47:32.791984 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:32.791955 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:47:33.065070 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:33.065031 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:47:33.190823 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:33.190789 2563 generic.go:356] "Generic (PLEG): container finished" podID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" exitCode=2 Sep 18 16:47:33.191004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:33.190848 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42"} Sep 18 16:47:33.191004 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:33.190899 2563 scope.go:117] "RemoveContainer" containerID="560dfab3ec1f05bc6fb9eb8fb721dcedfc8537b91b243e136bbdfb461560c35e" Sep 18 16:47:33.191294 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:33.191278 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:47:33.191514 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:33.191476 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:47:33.273951 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:33.273908 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s" Sep 18 16:47:34.074503 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:34.074471 2563 scope.go:117] "RemoveContainer" containerID="cd877fe97c993fc832e8d4a23a1cf5804523f6f29f37c9dc1b3f569cfe2cdabe" Sep 18 16:47:35.206235 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:35.206195 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerStarted","Data":"77dad2aaa069caf08d91a16519fe7696c3bac774ddf5fcfb4b634f4d5ba6b987"} Sep 18 16:47:36.074380 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:36.074337 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:36.074565 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:36.074408 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:38.998596 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:38.998500 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d677b303d3521e opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Liveness probe failed: Get \"http://10.132.0.38:8081/healthz\": dial tcp 10.132.0.38:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:32.673382942 +0000 UTC m=+2012.234227602,LastTimestamp:2026-09-18 16:38:32.673382942 +0000 UTC m=+2012.234227602,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:47:39.166972 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:39.166936 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:39.167183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:39.167008 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:43.065587 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:43.065539 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:47:43.920098 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:43.920045 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:47:44.074876 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:44.074833 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:47:44.075296 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:44.075054 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:47:46.074343 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:46.074306 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:46.074706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:46.074356 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:49.166319 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:49.166283 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:49.166785 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:49.166345 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:50.274858 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:50.274817 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:47:53.065632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:53.065589 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:47:55.074588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:55.074548 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:47:55.075080 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:55.074729 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:47:56.074172 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074133 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:56.074355 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074187 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/healthz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:56.074355 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074227 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:47:56.074655 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074637 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889"} pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" containerMessage="Container console-operator failed liveness probe, will be restarted" Sep 18 16:47:56.074936 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074653 2563 patch_prober.go:28] interesting pod/console-operator-fd4bb46c-jd8k8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body= Sep 18 16:47:56.074936 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074670 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" containerID="cri-o://9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" gracePeriod=30 Sep 18 16:47:56.074936 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:56.074688 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" Sep 18 16:47:56.591870 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:56.591839 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:47:57.326925 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:57.326895 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:47:57.327359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:57.327295 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/7.log" Sep 18 16:47:57.327359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:57.327330 2563 generic.go:356] "Generic (PLEG): container finished" podID="13e5f050-0e55-4386-8e80-8927af0ec4d3" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" exitCode=255 Sep 18 16:47:57.327436 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:57.327392 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerDied","Data":"9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889"} Sep 18 16:47:57.327475 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:57.327443 2563 scope.go:117] "RemoveContainer" containerID="3085b7a54da0f615cae5e515f9f2c0d47a20e59eedf9c8e0960e7273d9ebc326" Sep 18 16:47:57.327776 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:57.327760 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:47:57.327971 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:47:57.327952 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:47:58.334557 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:47:58.334520 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:48:03.065362 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:03.065314 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:48:03.919895 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:03.919846 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.28:8081/healthz\": dial tcp 10.132.0.28:8081: connect: connection refused" Sep 18 16:48:06.074662 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:06.074627 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:48:06.075059 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:06.074794 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:48:07.276024 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:07.275976 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-132-41.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s" Sep 18 16:48:12.419394 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:12.419357 2563 generic.go:356] "Generic (PLEG): container finished" podID="3082c931-af4a-4f03-b525-968fb7f112dc" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" exitCode=1 Sep 18 16:48:12.419788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:12.419427 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerDied","Data":"ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c"} Sep 18 16:48:12.419788 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:12.419520 2563 scope.go:117] "RemoveContainer" containerID="c0950d7e8ff42023a3daa6e8bccbd44530dfd890b21d91cf2ca72ac364db23fe" Sep 18 16:48:12.419948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:12.419932 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:48:12.420251 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:12.420224 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:48:13.003320 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:13.003224 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d677ae6b7cab82 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:12.937698178 +0000 UTC m=+1992.498542836,LastTimestamp:2026-09-18 16:38:32.936123549 +0000 UTC m=+2012.496968206,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:48:13.081317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:13.081291 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:48:13.081490 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:13.081469 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:48:18.074770 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:18.074738 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:48:18.075160 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:18.074904 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:48:23.797188 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:23.797146 2563 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods maas-controller-7b7898d6ff-jkzmr)" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:48:23.919588 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:23.919549 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:48:23.920042 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:23.920023 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:48:23.920292 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:23.920270 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:48:28.074352 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:28.074313 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:48:28.074746 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:28.074489 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:48:31.074945 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:31.074907 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:48:31.075466 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:31.075145 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:48:38.075208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:38.075175 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:48:38.075590 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:38.075374 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:48:41.074667 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:41.074627 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:48:41.075174 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:41.074802 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:48:46.075375 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:46.075343 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:48:46.075808 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:46.075546 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:48:47.007594 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:47.007507 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj.18d677aee3950e86 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj,UID:3082c931-af4a-4f03-b525-968fb7f112dc,APIVersion:v1,ResourceVersion:13097,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.28:8081/readyz\": dial tcp 10.132.0.28:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:14.95256231 +0000 UTC m=+1994.513406968,LastTimestamp:2026-09-18 16:38:34.951382816 +0000 UTC m=+2014.512227484,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:48:51.075187 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:51.075151 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:48:51.075595 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:51.075335 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:48:53.075193 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:48:53.075157 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:48:53.075584 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:48:53.075324 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:49:00.074467 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:00.074436 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:49:00.074920 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:00.074599 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:49:03.593727 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:03.593694 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:49:04.074901 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:04.074873 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:49:04.075072 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:04.075054 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:49:06.074660 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:06.074627 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:49:06.075039 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:06.074799 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:49:14.074655 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:14.074625 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:49:14.075029 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:14.074808 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:49:16.074459 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:16.074426 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:49:16.074840 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:16.074601 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:49:19.074982 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:19.074947 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:49:19.075418 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:19.075161 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:49:21.011702 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:21.011616 2563 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{maas-controller-7b7898d6ff-jkzmr.18d677ae6b7cab82 opendatahub 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:maas-controller-7b7898d6ff-jkzmr,UID:5f3c3a54-6101-430a-aa20-0d33a75784a4,APIVersion:v1,ResourceVersion:20895,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Readiness probe failed: Get \"http://10.132.0.38:8081/readyz\": dial tcp 10.132.0.38:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-132-41.ec2.internal,},FirstTimestamp:2026-09-18 16:38:12.937698178 +0000 UTC m=+1992.498542836,LastTimestamp:2026-09-18 16:38:42.935910462 +0000 UTC m=+2022.496755129,Count:4,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-41.ec2.internal,}" Sep 18 16:49:23.800965 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:23.800925 2563 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj)" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:49:28.548015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:28.547980 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 16:49:29.008686 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.008655 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 18 16:49:29.025569 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.025549 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 18 16:49:29.074771 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.074748 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:49:29.074921 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:29.074902 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:49:29.091194 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.091172 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 18 16:49:29.099960 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.099942 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\"" Sep 18 16:49:29.112874 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.112850 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 16:49:29.132501 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.132482 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vx59n\"" Sep 18 16:49:29.162717 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.162694 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6qnc4\"" Sep 18 16:49:29.166796 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.166778 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 18 16:49:29.174975 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.174953 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 18 16:49:29.191867 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.191846 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5qj9k\"" Sep 18 16:49:29.196030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.196009 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 18 16:49:29.220793 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.220773 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 18 16:49:29.222296 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.222279 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 18 16:49:29.223485 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.223464 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nkc54\"" Sep 18 16:49:29.223699 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.223681 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 18 16:49:29.231691 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.231674 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 18 16:49:29.237194 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.237179 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 16:49:29.261247 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.261199 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 16:49:29.265062 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.265047 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-c7qbt\"" Sep 18 16:49:29.268599 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.268581 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lhc9c\"" Sep 18 16:49:29.276367 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.276352 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 16:49:29.290223 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.290203 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zwprx\"" Sep 18 16:49:29.297729 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.297711 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r6bjb\"" Sep 18 16:49:29.303029 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.303016 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcclj\"" Sep 18 16:49:29.324124 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.324105 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 18 16:49:29.324317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.324302 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gscf4\"" Sep 18 16:49:29.330843 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.330826 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 18 16:49:29.336411 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.336389 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 16:49:29.342145 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.342126 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 18 16:49:29.367479 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.367460 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 18 16:49:29.386721 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.386702 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 16:49:29.389211 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.389193 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\"" Sep 18 16:49:29.389313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.389236 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 18 16:49:29.389438 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.389424 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvq9z\"" Sep 18 16:49:29.389529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.389514 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kthwg\"" Sep 18 16:49:29.405858 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.405840 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-11725e-dockercfg-djmph\"" Sep 18 16:49:29.408488 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.408472 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8cs6v\"" Sep 18 16:49:29.408900 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.408887 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 16:49:29.414764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.414750 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\"" Sep 18 16:49:29.423934 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.423915 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ln5jh\"" Sep 18 16:49:29.425078 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.425057 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdh8x\"" Sep 18 16:49:29.425184 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.425167 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 16:49:29.455764 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.455743 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 18 16:49:29.466740 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.466722 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sr7qq\"" Sep 18 16:49:29.473961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.473944 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 16:49:29.478936 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.478912 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bm6xr\"" Sep 18 16:49:29.484855 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.484838 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 18 16:49:29.492771 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.492752 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tt4rz\"" Sep 18 16:49:29.498864 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.498849 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-m7msx\"" Sep 18 16:49:29.512305 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.512253 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 16:49:29.555548 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.555522 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgw4g\"" Sep 18 16:49:29.570529 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.570504 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5n88n\"" Sep 18 16:49:29.575480 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.575463 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 18 16:49:29.579961 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.579941 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 18 16:49:29.586376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.586356 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\"" Sep 18 16:49:29.615364 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.615340 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 18 16:49:29.621818 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.621796 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-57xkq\"" Sep 18 16:49:29.629581 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.629563 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-d8pjj\"" Sep 18 16:49:29.634315 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.634299 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5zw7d\"" Sep 18 16:49:29.636651 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.636633 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a75117-dockercfg-lqq85\"" Sep 18 16:49:29.640926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.640912 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n9smt\"" Sep 18 16:49:29.659576 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.659559 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 16:49:29.673635 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.673619 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 16:49:29.676129 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.675913 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 18 16:49:29.711734 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.711706 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 18 16:49:29.735217 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.735191 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8gffl\"" Sep 18 16:49:29.735298 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.735191 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 18 16:49:29.750250 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.750227 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mvfv2\"" Sep 18 16:49:29.754572 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.754554 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4v9d\"" Sep 18 16:49:29.757589 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.757574 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69r9k\"" Sep 18 16:49:29.762989 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.762948 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 16:49:29.799208 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.799188 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjzzn\"" Sep 18 16:49:29.802300 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:29.802286 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a75117-serving-cert\"" Sep 18 16:49:30.075405 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:30.075326 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:49:30.075573 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:30.075438 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:49:30.075573 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:30.075509 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:49:30.075708 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:30.075604 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:49:35.875996 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:35.875963 2563 generic.go:356] "Generic (PLEG): container finished" podID="f25112d0-8747-4ead-8f96-9b968a57d914" containerID="77dad2aaa069caf08d91a16519fe7696c3bac774ddf5fcfb4b634f4d5ba6b987" exitCode=255 Sep 18 16:49:35.876494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:35.876028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerDied","Data":"77dad2aaa069caf08d91a16519fe7696c3bac774ddf5fcfb4b634f4d5ba6b987"} Sep 18 16:49:35.876494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:35.876075 2563 scope.go:117] "RemoveContainer" containerID="cd877fe97c993fc832e8d4a23a1cf5804523f6f29f37c9dc1b3f569cfe2cdabe" Sep 18 16:49:35.876494 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:35.876314 2563 scope.go:117] "RemoveContainer" containerID="77dad2aaa069caf08d91a16519fe7696c3bac774ddf5fcfb4b634f4d5ba6b987" Sep 18 16:49:35.876494 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:35.876480 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"openshift-lws-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=openshift-lws-operator pod=openshift-lws-operator-fd8ccff4c-wvvkm_openshift-lws-operator(f25112d0-8747-4ead-8f96-9b968a57d914)\"" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" podUID="f25112d0-8747-4ead-8f96-9b968a57d914" Sep 18 16:49:39.912806 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:39.912775 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:41.092368 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:41.092336 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:41.207210 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:41.207178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:42.075267 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:42.075240 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:49:42.075452 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:42.075412 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:49:43.077926 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:43.077896 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:49:43.078343 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:43.078176 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:49:44.074929 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:44.074894 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:49:44.075125 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:44.075048 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:49:45.864477 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:45.864445 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:45.955357 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:45.955325 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:47.074515 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:47.074483 2563 scope.go:117] "RemoveContainer" containerID="77dad2aaa069caf08d91a16519fe7696c3bac774ddf5fcfb4b634f4d5ba6b987" Sep 18 16:49:47.074907 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:47.074674 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"openshift-lws-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=openshift-lws-operator pod=openshift-lws-operator-fd8ccff4c-wvvkm_openshift-lws-operator(f25112d0-8747-4ead-8f96-9b968a57d914)\"" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" podUID="f25112d0-8747-4ead-8f96-9b968a57d914" Sep 18 16:49:52.396450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:52.396384 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:52.491351 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:52.491306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:54.075047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:54.075016 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:49:54.075443 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:54.075233 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:49:55.074708 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:55.074672 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:49:55.074920 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:55.074852 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:49:56.273844 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:56.273802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:57.075201 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:57.075171 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:49:57.075376 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:49:57.075356 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:49:58.075136 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:58.075068 2563 scope.go:117] "RemoveContainer" containerID="77dad2aaa069caf08d91a16519fe7696c3bac774ddf5fcfb4b634f4d5ba6b987" Sep 18 16:49:58.075579 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:58.075397 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:49:59.004949 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:59.004908 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wvvkm" event={"ID":"f25112d0-8747-4ead-8f96-9b968a57d914","Type":"ContainerStarted","Data":"a00807d6cbc5b5ed3316869368ca434189286e3efb25d32f4ca414168456d521"} Sep 18 16:49:59.621523 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:59.621489 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:49:59.690931 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:49:59.690897 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:03.287132 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:03.287097 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:50:03.292187 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:03.292158 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:50:08.074817 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:08.074783 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:50:08.075326 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:08.074964 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:50:09.075078 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:09.075046 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:50:09.075574 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:09.075299 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:50:11.075109 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:11.075055 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:50:11.075506 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:11.075310 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:50:13.538517 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:13.538486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:20.075158 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:20.075121 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:50:20.075545 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:20.075297 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:50:21.075555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:21.075520 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:50:21.075943 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:21.075704 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:50:25.075163 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:25.075127 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:50:25.075653 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:25.075336 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:50:33.077283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:33.077248 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:50:33.077726 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:33.077336 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:50:33.077726 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:33.077461 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:50:33.077726 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:33.077577 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:50:40.074545 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:40.074501 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:50:40.076875 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:40.074763 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:50:44.075063 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:44.075022 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:50:44.075484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:44.075135 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:50:44.075484 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:44.075217 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:50:44.075484 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:44.075267 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:50:46.347382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:46.347350 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:46.445067 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:46.445035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:50.799220 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:50.799179 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:50.887925 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:50.887892 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:51.912279 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:51.912244 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:51.982359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:51.982316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:50:54.075371 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:54.075333 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:50:54.075813 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:54.075519 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=maas-controller-7b7898d6ff-jkzmr_opendatahub(5f3c3a54-6101-430a-aa20-0d33a75784a4)\"" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" Sep 18 16:50:55.075060 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:55.075024 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:50:55.075060 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:50:55.075065 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:50:55.075361 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:55.075266 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:50:55.075361 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:50:55.075279 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:51:02.577118 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.577048 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:51:02.617199 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.617161 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-jkzmr"] Sep 18 16:51:02.754942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.754914 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:51:02.865186 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.865067 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-metrics-certs\") pod \"5f3c3a54-6101-430a-aa20-0d33a75784a4\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " Sep 18 16:51:02.865186 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.865150 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-webhook-cert\") pod \"5f3c3a54-6101-430a-aa20-0d33a75784a4\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " Sep 18 16:51:02.865381 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.865205 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf2t2\" (UniqueName: \"kubernetes.io/projected/5f3c3a54-6101-430a-aa20-0d33a75784a4-kube-api-access-tf2t2\") pod \"5f3c3a54-6101-430a-aa20-0d33a75784a4\" (UID: \"5f3c3a54-6101-430a-aa20-0d33a75784a4\") " Sep 18 16:51:02.867566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.867526 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3c3a54-6101-430a-aa20-0d33a75784a4-kube-api-access-tf2t2" (OuterVolumeSpecName: "kube-api-access-tf2t2") pod "5f3c3a54-6101-430a-aa20-0d33a75784a4" (UID: "5f3c3a54-6101-430a-aa20-0d33a75784a4"). InnerVolumeSpecName "kube-api-access-tf2t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:51:02.867566 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.867532 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5f3c3a54-6101-430a-aa20-0d33a75784a4" (UID: "5f3c3a54-6101-430a-aa20-0d33a75784a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:51:02.867748 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.867629 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5f3c3a54-6101-430a-aa20-0d33a75784a4" (UID: "5f3c3a54-6101-430a-aa20-0d33a75784a4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:51:02.966016 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.965974 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf2t2\" (UniqueName: \"kubernetes.io/projected/5f3c3a54-6101-430a-aa20-0d33a75784a4-kube-api-access-tf2t2\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:51:02.966016 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.966011 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:51:02.966016 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:02.966020 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f3c3a54-6101-430a-aa20-0d33a75784a4-webhook-cert\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:51:03.367631 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:03.367604 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" Sep 18 16:51:03.367631 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:03.367619 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-jkzmr" event={"ID":"5f3c3a54-6101-430a-aa20-0d33a75784a4","Type":"ContainerDied","Data":"62100d10d76a85b56eb4c639b4d525da69bee243e4f71806c3404f2173e48305"} Sep 18 16:51:03.367888 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:03.367678 2563 scope.go:117] "RemoveContainer" containerID="f52f62c1b7f903fa9dc5de1a2a5825523e1a8c9c950788f4d53e0eccc33b9f42" Sep 18 16:51:03.393934 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:03.393896 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-jkzmr"] Sep 18 16:51:03.396382 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:03.396357 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-jkzmr"] Sep 18 16:51:04.200636 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.200597 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-2zwrd"] Sep 18 16:51:04.201040 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201025 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2a6f72-1aa7-470d-a921-47b0aab19a0f" containerName="maas-api" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201041 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2a6f72-1aa7-470d-a921-47b0aab19a0f" containerName="maas-api" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201052 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201059 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201066 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201071 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201079 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c166202f-1ca2-4ada-8ff9-b096dca559d8" containerName="payload-pre-processing" Sep 18 16:51:04.201099 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201098 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c166202f-1ca2-4ada-8ff9-b096dca559d8" containerName="payload-pre-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201109 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="491163c7-d81c-4112-bfd0-99e5847a9637" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201117 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="491163c7-d81c-4112-bfd0-99e5847a9637" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201137 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201142 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201148 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201154 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201163 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73381fc-907b-4cb0-93fd-7a35a7b3652b" containerName="maas-api" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201169 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73381fc-907b-4cb0-93fd-7a35a7b3652b" containerName="maas-api" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201176 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d162f08-db46-4065-ad8e-e61620b5c225" containerName="payload-pre-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201181 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d162f08-db46-4065-ad8e-e61620b5c225" containerName="payload-pre-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201203 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201208 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201215 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e7cc2c-1ffe-4aac-a985-c8f314199e40" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201220 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e7cc2c-1ffe-4aac-a985-c8f314199e40" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201228 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9c5448-a8e7-49a4-a8cd-25015dab6a75" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201233 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9c5448-a8e7-49a4-a8cd-25015dab6a75" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201241 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d61092-8da2-4fa0-a0ab-e189d8742447" containerName="maas-api" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201246 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d61092-8da2-4fa0-a0ab-e189d8742447" containerName="maas-api" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201253 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ede228-961f-43d0-8817-4324879b36dc" containerName="maas-api" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201258 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ede228-961f-43d0-8817-4324879b36dc" containerName="maas-api" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201268 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201273 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201281 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab537d1-94c1-4a00-aee7-70680e2aff9d" containerName="payload-processing" Sep 18 16:51:04.201329 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201286 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab537d1-94c1-4a00-aee7-70680e2aff9d" containerName="payload-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201351 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201358 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d162f08-db46-4065-ad8e-e61620b5c225" containerName="payload-pre-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201364 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab537d1-94c1-4a00-aee7-70680e2aff9d" containerName="payload-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201370 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d61092-8da2-4fa0-a0ab-e189d8742447" containerName="maas-api" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201376 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201382 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="491163c7-d81c-4112-bfd0-99e5847a9637" containerName="payload-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201387 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fef0cc0-a218-4e16-b4bd-b06dad7d3e06" containerName="payload-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201394 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2a6f72-1aa7-470d-a921-47b0aab19a0f" containerName="maas-api" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201400 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ede228-961f-43d0-8817-4324879b36dc" containerName="maas-api" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201405 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9c5448-a8e7-49a4-a8cd-25015dab6a75" containerName="payload-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201411 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201417 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c166202f-1ca2-4ada-8ff9-b096dca559d8" containerName="payload-pre-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201422 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e7cc2c-1ffe-4aac-a985-c8f314199e40" containerName="payload-processing" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201427 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73381fc-907b-4cb0-93fd-7a35a7b3652b" containerName="maas-api" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201432 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.202032 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.201437 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:51:04.204955 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.204927 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:51:04.205753 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.205736 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.209399 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.209378 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8gffl\"" Sep 18 16:51:04.210452 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.210434 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 18 16:51:04.210533 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.210454 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 18 16:51:04.211112 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.210934 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 18 16:51:04.221555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.221523 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-2zwrd"] Sep 18 16:51:04.283976 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.283933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8jc\" (UniqueName: \"kubernetes.io/projected/d83c9487-feb6-408a-bc8f-1a7add83687d-kube-api-access-5m8jc\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.283976 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.283974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d83c9487-feb6-408a-bc8f-1a7add83687d-webhook-cert\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.284244 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.284109 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d83c9487-feb6-408a-bc8f-1a7add83687d-metrics-certs\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.384645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.384609 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8jc\" (UniqueName: \"kubernetes.io/projected/d83c9487-feb6-408a-bc8f-1a7add83687d-kube-api-access-5m8jc\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.384856 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.384654 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d83c9487-feb6-408a-bc8f-1a7add83687d-webhook-cert\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.384856 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.384757 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d83c9487-feb6-408a-bc8f-1a7add83687d-metrics-certs\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.387241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.387207 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d83c9487-feb6-408a-bc8f-1a7add83687d-webhook-cert\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.387241 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.387224 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d83c9487-feb6-408a-bc8f-1a7add83687d-metrics-certs\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.396073 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.396044 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8jc\" (UniqueName: \"kubernetes.io/projected/d83c9487-feb6-408a-bc8f-1a7add83687d-kube-api-access-5m8jc\") pod \"maas-controller-7b7898d6ff-2zwrd\" (UID: \"d83c9487-feb6-408a-bc8f-1a7add83687d\") " pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.518724 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.518678 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:04.800375 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:04.800345 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b7898d6ff-2zwrd"] Sep 18 16:51:04.801237 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:51:04.801212 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83c9487_feb6_408a_bc8f_1a7add83687d.slice/crio-d912e80636a141c31b396fea4d7e1397ecad8d0be7028bd6c3f93910127948fa WatchSource:0}: Error finding container d912e80636a141c31b396fea4d7e1397ecad8d0be7028bd6c3f93910127948fa: Status 404 returned error can't find the container with id d912e80636a141c31b396fea4d7e1397ecad8d0be7028bd6c3f93910127948fa Sep 18 16:51:05.081403 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:05.081318 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" path="/var/lib/kubelet/pods/5f3c3a54-6101-430a-aa20-0d33a75784a4/volumes" Sep 18 16:51:05.380827 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:05.380799 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" event={"ID":"d83c9487-feb6-408a-bc8f-1a7add83687d","Type":"ContainerStarted","Data":"d912e80636a141c31b396fea4d7e1397ecad8d0be7028bd6c3f93910127948fa"} Sep 18 16:51:06.388465 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:06.388424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" event={"ID":"d83c9487-feb6-408a-bc8f-1a7add83687d","Type":"ContainerStarted","Data":"e129bfe478fa81c92ff1a36ff19dcb459263fabdedbc55a361c11c278c585530"} Sep 18 16:51:06.388831 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:06.388511 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:06.410927 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:06.410875 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" podStartSLOduration=1.850161092 podStartE2EDuration="2.410858428s" totalImagesPullingTime="560.697336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:51:06.409038367 +0000 UTC m=+2765.969883035" watchObservedRunningTime="2026-09-18 16:51:06.410858428 +0000 UTC m=+2765.971703099" Sep 18 16:51:07.074944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:07.074913 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:51:07.075156 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:07.075119 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:51:10.074583 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:10.074543 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:51:10.075060 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:10.074796 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:51:16.478836 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:16.478756 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:51:17.396505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:17.396470 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b7898d6ff-2zwrd" Sep 18 16:51:22.074995 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:22.074962 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:51:22.075420 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:22.075178 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:51:23.078021 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:23.077993 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:51:23.078471 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:23.078194 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:51:26.071420 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:26.071381 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:51:27.251680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:27.251648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:51:34.075234 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:34.075199 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:51:34.075621 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:34.075393 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:51:35.074482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:35.074442 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:51:35.074692 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:35.074654 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:51:45.074946 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:45.074906 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:51:45.075366 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:45.075141 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:51:46.074815 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:46.074786 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:51:46.075000 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:46.074978 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:51:59.075168 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:51:59.075133 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:51:59.075646 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:51:59.075329 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:52:00.075593 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:00.075562 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:52:00.076043 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:00.075745 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:52:11.075393 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:11.075358 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:52:11.075777 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:11.075535 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:52:13.077696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:13.077662 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:52:13.078067 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:13.077863 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:52:25.075450 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:25.075418 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:52:25.075842 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:25.075611 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:52:26.075020 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:26.074987 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:52:26.075244 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:26.075221 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:52:36.698467 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.698436 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-5vkn6"] Sep 18 16:52:36.698942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.698800 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:52:36.698942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.698810 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:52:36.698942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.698827 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:52:36.698942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.698833 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:52:36.698942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.698901 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:52:36.701838 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.701822 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.706326 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.706308 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\"" Sep 18 16:52:36.825292 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.825264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.825414 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.825310 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m4b\" (UniqueName: \"kubernetes.io/projected/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-kube-api-access-m5m4b\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.825414 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.825342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-ready\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.825414 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.825368 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.844268 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.844240 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:36.925857 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.925824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.926009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.925877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m4b\" (UniqueName: \"kubernetes.io/projected/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-kube-api-access-m5m4b\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.926009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.925906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-ready\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.926009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.925932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.926009 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.925954 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.926356 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.926334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-ready\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.926471 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.926452 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:36.938222 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:36.938202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m4b\" (UniqueName: \"kubernetes.io/projected/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-kube-api-access-m5m4b\") pod \"cni-sysctl-allowlist-ds-5vkn6\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:37.011290 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:37.011255 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:37.024109 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:52:37.024064 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4c184f_c7fe_4f61_832c_5e7652a81cd3.slice/crio-39666272eb57c97bc0a79b54338a0b841266d377e52986b2d93fa8f86e5c493b WatchSource:0}: Error finding container 39666272eb57c97bc0a79b54338a0b841266d377e52986b2d93fa8f86e5c493b: Status 404 returned error can't find the container with id 39666272eb57c97bc0a79b54338a0b841266d377e52986b2d93fa8f86e5c493b Sep 18 16:52:37.475537 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:37.475440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:37.907417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:37.907379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" event={"ID":"5a4c184f-c7fe-4f61-832c-5e7652a81cd3","Type":"ContainerStarted","Data":"2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e"} Sep 18 16:52:37.907868 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:37.907437 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" event={"ID":"5a4c184f-c7fe-4f61-832c-5e7652a81cd3","Type":"ContainerStarted","Data":"39666272eb57c97bc0a79b54338a0b841266d377e52986b2d93fa8f86e5c493b"} Sep 18 16:52:37.907868 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:37.907584 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:37.931133 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:37.931070 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" podStartSLOduration=1.931055567 podStartE2EDuration="1.931055567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:37.928616609 +0000 UTC m=+2857.489461277" watchObservedRunningTime="2026-09-18 16:52:37.931055567 +0000 UTC m=+2857.491900234" Sep 18 16:52:38.075312 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:38.075279 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:52:38.075481 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:38.075457 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:52:38.924454 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:38.924423 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:52:39.678898 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:39.678861 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-5vkn6"] Sep 18 16:52:40.074615 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:40.074575 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:52:40.075117 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:40.074830 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:52:40.924807 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:40.924760 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" gracePeriod=30 Sep 18 16:52:47.409532 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:47.409451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:47.509077 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:47.509042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:47.700313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:47.700222 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:48.101771 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:48.101739 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:48.184447 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:48.184419 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:48.915986 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:48.915938 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:52:48.916954 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:48.916923 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:52:48.917857 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:48.917822 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:52:48.917973 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:48.917869 2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 18 16:52:50.074758 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:50.074719 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:52:50.075206 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:50.074926 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=console-operator pod=console-operator-fd4bb46c-jd8k8_openshift-console-operator(13e5f050-0e55-4386-8e80-8927af0ec4d3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" podUID="13e5f050-0e55-4386-8e80-8927af0ec4d3" Sep 18 16:52:52.074679 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:52.074643 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:52:52.075155 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:52.074874 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:52:53.370463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:53.370417 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:53.471816 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:53.471771 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:53.571217 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:52:53.571184 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:52:58.915930 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:58.915887 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:52:58.916843 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:58.916814 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:52:58.917711 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:58.917675 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:52:58.917812 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:52:58.917727 2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 18 16:53:04.074482 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:04.074446 2563 scope.go:117] "RemoveContainer" containerID="9ae673a6f08a740063b8ffff57397f5acc86ce403c5a8fb2269c921658a4e889" Sep 18 16:53:05.061309 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:05.061277 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:53:05.061500 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:05.061398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" event={"ID":"13e5f050-0e55-4386-8e80-8927af0ec4d3","Type":"ContainerStarted","Data":"466da6e53d0306a3e5394d93f77e22a49f3b3fe84284e55b26157a6f0bb42271"} Sep 18 16:53:05.061666 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:05.061645 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:53:05.066355 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:05.066332 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-jd8k8" Sep 18 16:53:07.074372 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:07.074338 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:53:07.074843 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:53:07.074586 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_opendatahub(3082c931-af4a-4f03-b525-968fb7f112dc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" podUID="3082c931-af4a-4f03-b525-968fb7f112dc" Sep 18 16:53:08.916373 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:53:08.916331 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:53:08.917375 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:53:08.917345 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:53:08.918215 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:53:08.918182 2563 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 18 16:53:08.918279 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:53:08.918228 2563 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 18 16:53:11.011505 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.011485 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-5vkn6_5a4c184f-c7fe-4f61-832c-5e7652a81cd3/kube-multus-additional-cni-plugins/0.log" Sep 18 16:53:11.013616 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.013600 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:53:11.096318 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.096293 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-5vkn6_5a4c184f-c7fe-4f61-832c-5e7652a81cd3/kube-multus-additional-cni-plugins/0.log" Sep 18 16:53:11.096456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.096332 2563 generic.go:356] "Generic (PLEG): container finished" podID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" exitCode=137 Sep 18 16:53:11.096456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.096376 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" Sep 18 16:53:11.096456 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.096406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" event={"ID":"5a4c184f-c7fe-4f61-832c-5e7652a81cd3","Type":"ContainerDied","Data":"2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e"} Sep 18 16:53:11.096618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.096455 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-5vkn6" event={"ID":"5a4c184f-c7fe-4f61-832c-5e7652a81cd3","Type":"ContainerDied","Data":"39666272eb57c97bc0a79b54338a0b841266d377e52986b2d93fa8f86e5c493b"} Sep 18 16:53:11.096618 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.096471 2563 scope.go:117] "RemoveContainer" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" Sep 18 16:53:11.105502 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.105485 2563 scope.go:117] "RemoveContainer" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" Sep 18 16:53:11.105763 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:53:11.105743 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e\": container with ID starting with 2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e not found: ID does not exist" containerID="2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e" Sep 18 16:53:11.105804 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.105771 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e"} err="failed to get container status \"2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e\": rpc error: code = NotFound desc = could not find container \"2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e\": container with ID starting with 2eff3469e96d0d8908433ae2ece71e570306685adbf1a422c26b63b59690003e not found: ID does not exist" Sep 18 16:53:11.144589 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144566 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-cni-sysctl-allowlist\") pod \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " Sep 18 16:53:11.144680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144594 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-tuning-conf-dir\") pod \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " Sep 18 16:53:11.144680 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144633 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-ready\") pod \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " Sep 18 16:53:11.144809 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144686 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5m4b\" (UniqueName: \"kubernetes.io/projected/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-kube-api-access-m5m4b\") pod \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\" (UID: \"5a4c184f-c7fe-4f61-832c-5e7652a81cd3\") " Sep 18 16:53:11.144809 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144682 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "5a4c184f-c7fe-4f61-832c-5e7652a81cd3" (UID: "5a4c184f-c7fe-4f61-832c-5e7652a81cd3"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 18 16:53:11.144947 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144922 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "5a4c184f-c7fe-4f61-832c-5e7652a81cd3" (UID: "5a4c184f-c7fe-4f61-832c-5e7652a81cd3"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:53:11.145063 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.144958 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-ready" (OuterVolumeSpecName: "ready") pod "5a4c184f-c7fe-4f61-832c-5e7652a81cd3" (UID: "5a4c184f-c7fe-4f61-832c-5e7652a81cd3"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 16:53:11.145469 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.145452 2563 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-cni-sysctl-allowlist\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:53:11.145563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.145473 2563 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-tuning-conf-dir\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:53:11.145563 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.145488 2563 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-ready\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:53:11.146923 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.146902 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-kube-api-access-m5m4b" (OuterVolumeSpecName: "kube-api-access-m5m4b") pod "5a4c184f-c7fe-4f61-832c-5e7652a81cd3" (UID: "5a4c184f-c7fe-4f61-832c-5e7652a81cd3"). InnerVolumeSpecName "kube-api-access-m5m4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:53:11.245910 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.245852 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5m4b\" (UniqueName: \"kubernetes.io/projected/5a4c184f-c7fe-4f61-832c-5e7652a81cd3-kube-api-access-m5m4b\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:53:11.419325 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.419290 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-5vkn6"] Sep 18 16:53:11.423859 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:11.423834 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-5vkn6"] Sep 18 16:53:13.080944 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:13.080901 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" path="/var/lib/kubelet/pods/5a4c184f-c7fe-4f61-832c-5e7652a81cd3/volumes" Sep 18 16:53:20.074990 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:20.074949 2563 scope.go:117] "RemoveContainer" containerID="ca52a1bbf0fa2939bd2a0331a4802d2461896c3ce67ef8f00de3143edd28c22c" Sep 18 16:53:21.154633 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:21.154601 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" event={"ID":"3082c931-af4a-4f03-b525-968fb7f112dc","Type":"ContainerStarted","Data":"563f68d3f7d3fee70d69d4a7d6887d769a7c091fb6cda1fa85ab150dc5ac964c"} Sep 18 16:53:21.155034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:21.154812 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:53:32.162182 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:53:32.162150 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj" Sep 18 16:54:30.242535 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:54:30.242498 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:54:47.628900 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:54:47.628863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:03.327472 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:03.327437 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:55:03.334209 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:03.334188 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:55:11.538783 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:11.538750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:12.587015 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:12.586977 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:36.940047 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.940012 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5"] Sep 18 16:55:36.940514 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.940459 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" Sep 18 16:55:36.940514 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.940472 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" Sep 18 16:55:36.940591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.940529 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4c184f-c7fe-4f61-832c-5e7652a81cd3" containerName="kube-multus-additional-cni-plugins" Sep 18 16:55:36.940591 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.940541 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3c3a54-6101-430a-aa20-0d33a75784a4" containerName="manager" Sep 18 16:55:36.943487 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.943467 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:36.958165 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.958139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5"] Sep 18 16:55:36.990180 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:36.990131 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:37.060428 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.060382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-metrics-certs\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.060628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.060447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-openshift-service-ca\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.060628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.060492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-maas-api-tls\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.060628 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.060556 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m49m\" (UniqueName: \"kubernetes.io/projected/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-kube-api-access-4m49m\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.161685 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.161646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-metrics-certs\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.161883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.161709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-openshift-service-ca\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.161883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.161736 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-maas-api-tls\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.161883 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.161788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m49m\" (UniqueName: \"kubernetes.io/projected/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-kube-api-access-4m49m\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.162463 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.162433 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-openshift-service-ca\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.164314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.164289 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-maas-api-tls\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.164314 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.164294 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-metrics-certs\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.170531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.170505 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m49m\" (UniqueName: \"kubernetes.io/projected/d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2-kube-api-access-4m49m\") pod \"maas-api-5b84f89b99-qvcw5\" (UID: \"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2\") " pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.256093 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.256062 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:37.532189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.532161 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5"] Sep 18 16:55:37.533567 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:55:37.533537 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d94142_ed1e_4cc1_bdb8_9c2754dae2c2.slice/crio-0a9a7dcf1556a4dcbd6877789485773b815944fb896e29d5cf7d4c3ed8ee2ef7 WatchSource:0}: Error finding container 0a9a7dcf1556a4dcbd6877789485773b815944fb896e29d5cf7d4c3ed8ee2ef7: Status 404 returned error can't find the container with id 0a9a7dcf1556a4dcbd6877789485773b815944fb896e29d5cf7d4c3ed8ee2ef7 Sep 18 16:55:37.534909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.534893 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 16:55:37.916606 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:37.916515 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" event={"ID":"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2","Type":"ContainerStarted","Data":"0a9a7dcf1556a4dcbd6877789485773b815944fb896e29d5cf7d4c3ed8ee2ef7"} Sep 18 16:55:38.050909 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:38.050878 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:38.874308 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:38.874276 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:38.925096 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:38.925049 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" event={"ID":"d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2","Type":"ContainerStarted","Data":"2ac521c4c014801d44360b70ffe6dba2b8aaa26a2e4b4548cc5c91d443893877"} Sep 18 16:55:38.925277 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:38.925135 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:38.957955 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:38.957902 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" podStartSLOduration=2.388305687 podStartE2EDuration="2.9578844s" totalImagesPullingTime="569.578713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:55:38.956210712 +0000 UTC m=+3038.517055385" watchObservedRunningTime="2026-09-18 16:55:38.9578844 +0000 UTC m=+3038.518729064" Sep 18 16:55:39.764887 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:39.764845 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:40.726365 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:40.726334 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:42.484933 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:42.484893 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:44.936376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:44.936291 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b84f89b99-qvcw5" Sep 18 16:55:45.035567 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:45.035538 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx"] Sep 18 16:55:45.035742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:45.035722 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" podUID="4c7a8691-29f9-4faf-b67a-83960a43c348" containerName="maas-api" containerID="cri-o://1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de" gracePeriod=30 Sep 18 16:55:46.284002 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.283978 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:55:46.341294 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.341259 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a8691-29f9-4faf-b67a-83960a43c348-openshift-service-ca\") pod \"4c7a8691-29f9-4faf-b67a-83960a43c348\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " Sep 18 16:55:46.341431 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.341337 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48c6g\" (UniqueName: \"kubernetes.io/projected/4c7a8691-29f9-4faf-b67a-83960a43c348-kube-api-access-48c6g\") pod \"4c7a8691-29f9-4faf-b67a-83960a43c348\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " Sep 18 16:55:46.341431 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.341402 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-metrics-certs\") pod \"4c7a8691-29f9-4faf-b67a-83960a43c348\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " Sep 18 16:55:46.341511 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.341431 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-maas-api-tls\") pod \"4c7a8691-29f9-4faf-b67a-83960a43c348\" (UID: \"4c7a8691-29f9-4faf-b67a-83960a43c348\") " Sep 18 16:55:46.341610 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.341591 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7a8691-29f9-4faf-b67a-83960a43c348-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4c7a8691-29f9-4faf-b67a-83960a43c348" (UID: "4c7a8691-29f9-4faf-b67a-83960a43c348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 16:55:46.341715 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.341700 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c7a8691-29f9-4faf-b67a-83960a43c348-openshift-service-ca\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:55:46.343328 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.343299 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4c7a8691-29f9-4faf-b67a-83960a43c348" (UID: "4c7a8691-29f9-4faf-b67a-83960a43c348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:55:46.343449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.343383 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7a8691-29f9-4faf-b67a-83960a43c348-kube-api-access-48c6g" (OuterVolumeSpecName: "kube-api-access-48c6g") pod "4c7a8691-29f9-4faf-b67a-83960a43c348" (UID: "4c7a8691-29f9-4faf-b67a-83960a43c348"). InnerVolumeSpecName "kube-api-access-48c6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 16:55:46.343449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.343425 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4c7a8691-29f9-4faf-b67a-83960a43c348" (UID: "4c7a8691-29f9-4faf-b67a-83960a43c348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 16:55:46.442974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.442905 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48c6g\" (UniqueName: \"kubernetes.io/projected/4c7a8691-29f9-4faf-b67a-83960a43c348-kube-api-access-48c6g\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:55:46.442974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.442932 2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-metrics-certs\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:55:46.442974 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.442942 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c7a8691-29f9-4faf-b67a-83960a43c348-maas-api-tls\") on node \"ip-10-0-132-41.ec2.internal\" DevicePath \"\"" Sep 18 16:55:46.971585 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.971554 2563 generic.go:356] "Generic (PLEG): container finished" podID="4c7a8691-29f9-4faf-b67a-83960a43c348" containerID="1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de" exitCode=0 Sep 18 16:55:46.971752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.971596 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" Sep 18 16:55:46.971752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.971614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" event={"ID":"4c7a8691-29f9-4faf-b67a-83960a43c348","Type":"ContainerDied","Data":"1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de"} Sep 18 16:55:46.971752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.971659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx" event={"ID":"4c7a8691-29f9-4faf-b67a-83960a43c348","Type":"ContainerDied","Data":"521e43ef7ce5c8ef3786136f7cdac7a1d0504899f9c588180fc1b60df88107a4"} Sep 18 16:55:46.971752 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.971681 2563 scope.go:117] "RemoveContainer" containerID="1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de" Sep 18 16:55:46.981407 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.981390 2563 scope.go:117] "RemoveContainer" containerID="1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de" Sep 18 16:55:46.981667 ip-10-0-132-41 kubenswrapper[2563]: E0918 16:55:46.981646 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de\": container with ID starting with 1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de not found: ID does not exist" containerID="1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de" Sep 18 16:55:46.981717 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.981677 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de"} err="failed to get container status \"1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de\": rpc error: code = NotFound desc = could not find container \"1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de\": container with ID starting with 1b8dfdbe4f70de0dc45bb25716dae47163f3de7a3fc878439a9971e974a587de not found: ID does not exist" Sep 18 16:55:46.994252 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:46.994233 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx"] Sep 18 16:55:47.002652 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:47.002633 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6669c6f9b7-rn9hx"] Sep 18 16:55:47.080057 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:47.080032 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c7a8691-29f9-4faf-b67a-83960a43c348" path="/var/lib/kubelet/pods/4c7a8691-29f9-4faf-b67a-83960a43c348/volumes" Sep 18 16:55:57.731124 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:57.731065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:57.831102 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:57.831065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:58.438186 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:58.438155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:58.576696 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:58.576659 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:59.232992 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:59.232964 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:59.338828 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:59.338802 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:55:59.768578 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:55:59.768543 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:00.421127 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:00.421077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:00.519119 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:00.519072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:00.706742 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:00.706662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:01.630521 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:01.630493 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:10.053913 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:10.053873 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:10.154211 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:10.154181 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:23.792934 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:23.792901 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b7898d6ff-2zwrd_d83c9487-feb6-408a-bc8f-1a7add83687d/manager/0.log" Sep 18 16:56:24.142189 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:24.142115 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_3082c931-af4a-4f03-b525-968fb7f112dc/copy-manifests/0.log" Sep 18 16:56:24.150317 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:24.150294 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_3082c931-af4a-4f03-b525-968fb7f112dc/manager/7.log" Sep 18 16:56:24.765376 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:24.765346 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b84f89b99-qvcw5_d9d94142-ed1e-4cc1-bdb8-9c2754dae2c2/maas-api/0.log" Sep 18 16:56:25.000569 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:25.000541 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w3-a75117-77744c48fd-w96tr_55d7e433-39cf-4b18-9efe-6aa26d02eb09/maas-api/0.log" Sep 18 16:56:25.927175 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:25.927141 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8h5h8_2652bcc2-afa2-4ae2-88e6-1b016cffdbf5/postgres/0.log" Sep 18 16:56:27.161484 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:27.161449 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:28.182996 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:28.182966 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2qsjw_0962fb23-150f-4073-9710-a999268241c7/discovery/0.log" Sep 18 16:56:28.495359 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:28.495282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:28.673942 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:28.673913 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zrd98_8fccffb1-1210-4544-9243-105721024d50/payload-pre-processing/0.log" Sep 18 16:56:28.793183 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:28.793148 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq_d6b62a3f-96a5-4e56-aca8-3e13af815341/payload-pre-processing/0.log" Sep 18 16:56:28.904967 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:28.904925 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9_541268bf-2fa4-4a20-8403-3eb28672a296/payload-pre-processing/0.log" Sep 18 16:56:29.039948 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:29.039910 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsk8t_a014f89f-e0fb-4319-b6ae-6cba48d802bd/payload-processing/0.log" Sep 18 16:56:29.259412 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:29.259380 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76_c5002529-ed7b-4d73-9154-26a20dfbff00/payload-processing/0.log" Sep 18 16:56:29.835508 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:29.835478 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wm8cw_ea823774-144e-4a34-8aee-8a6d63ff69aa/main/0.log" Sep 18 16:56:30.066034 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:30.066007 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6tt7l_6791a1f8-61ce-42a8-95c8-7a3fc65f8bb0/main/0.log" Sep 18 16:56:35.543880 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:35.543843 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:35.637509 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:35.637466 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" limit=1024 Sep 18 16:56:38.713417 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:38.713385 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r2z6d_62f743f6-9ca7-41f0-85cf-4266d18a1f78/global-pull-secret-syncer/0.log" Sep 18 16:56:38.770362 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:38.770333 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9z7rv_2f9d4fd6-0082-4200-8fbb-099af79e209e/konnectivity-agent/0.log" Sep 18 16:56:38.905738 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:38.905711 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-41.ec2.internal_9ae31834f4280439711acddc59629bae/haproxy/0.log" Sep 18 16:56:45.247508 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.247474 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/alertmanager/0.log" Sep 18 16:56:45.293508 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.293479 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/config-reloader/0.log" Sep 18 16:56:45.381273 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.381242 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/kube-rbac-proxy-web/0.log" Sep 18 16:56:45.434261 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.434238 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/kube-rbac-proxy/0.log" Sep 18 16:56:45.508957 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.508930 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/kube-rbac-proxy-metric/0.log" Sep 18 16:56:45.551612 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.551586 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/prom-label-proxy/0.log" Sep 18 16:56:45.582665 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.582639 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b87742c0-2af7-462c-afcb-a38343acfe63/init-config-reloader/0.log" Sep 18 16:56:45.668893 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.668864 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-px7qf_a17009e2-0cb4-4f46-9bd9-1e0d8d20de27/kube-state-metrics/0.log" Sep 18 16:56:45.693549 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.693523 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-px7qf_a17009e2-0cb4-4f46-9bd9-1e0d8d20de27/kube-rbac-proxy-main/0.log" Sep 18 16:56:45.720875 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:45.720851 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-px7qf_a17009e2-0cb4-4f46-9bd9-1e0d8d20de27/kube-rbac-proxy-self/0.log" Sep 18 16:56:46.106689 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:46.106663 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xfn6b_4ba373af-2334-4925-82df-e15fed6a8fcc/node-exporter/0.log" Sep 18 16:56:46.137551 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:46.137523 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xfn6b_4ba373af-2334-4925-82df-e15fed6a8fcc/kube-rbac-proxy/0.log" Sep 18 16:56:46.179854 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:46.179837 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xfn6b_4ba373af-2334-4925-82df-e15fed6a8fcc/init-textfile/0.log" Sep 18 16:56:46.639721 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:46.639697 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4ck5t_95254775-b58f-4f42-bb0b-bd48d2cd5141/prometheus-operator/0.log" Sep 18 16:56:46.673624 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:46.673593 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4ck5t_95254775-b58f-4f42-bb0b-bd48d2cd5141/kube-rbac-proxy/0.log" Sep 18 16:56:46.712460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:46.712432 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-4mxn8_40b404d1-9ae4-4b27-82e4-b08a0a3597cb/prometheus-operator-admission-webhook/0.log" Sep 18 16:56:48.276647 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.276599 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2"] Sep 18 16:56:48.277041 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.277028 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7a8691-29f9-4faf-b67a-83960a43c348" containerName="maas-api" Sep 18 16:56:48.277103 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.277043 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7a8691-29f9-4faf-b67a-83960a43c348" containerName="maas-api" Sep 18 16:56:48.277138 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.277120 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7a8691-29f9-4faf-b67a-83960a43c348" containerName="maas-api" Sep 18 16:56:48.280444 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.280419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.285916 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.285893 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7rjqm\"/\"kube-root-ca.crt\"" Sep 18 16:56:48.286141 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.286122 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7rjqm\"/\"openshift-service-ca.crt\"" Sep 18 16:56:48.286391 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.286375 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7rjqm\"/\"default-dockercfg-8knlp\"" Sep 18 16:56:48.293012 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.292984 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2"] Sep 18 16:56:48.392555 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.392518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-podres\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.392737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.392562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62hqf\" (UniqueName: \"kubernetes.io/projected/d312d55e-f987-40e2-ad44-3bfddbc07d69-kube-api-access-62hqf\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.392737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.392623 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-sys\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.392737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.392647 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-lib-modules\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.392737 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.392674 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-proc\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493108 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-podres\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62hqf\" (UniqueName: \"kubernetes.io/projected/d312d55e-f987-40e2-ad44-3bfddbc07d69-kube-api-access-62hqf\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493194 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-sys\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-lib-modules\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493242 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-podres\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493313 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-proc\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493334 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-sys\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493383 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-proc\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.493531 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.493395 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d312d55e-f987-40e2-ad44-3bfddbc07d69-lib-modules\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.507941 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.507904 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62hqf\" (UniqueName: \"kubernetes.io/projected/d312d55e-f987-40e2-ad44-3bfddbc07d69-kube-api-access-62hqf\") pod \"perf-node-gather-daemonset-r26z2\" (UID: \"d312d55e-f987-40e2-ad44-3bfddbc07d69\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.590507 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.590430 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:48.870731 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:48.870702 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2"] Sep 18 16:56:48.872459 ip-10-0-132-41 kubenswrapper[2563]: W0918 16:56:48.872433 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd312d55e_f987_40e2_ad44_3bfddbc07d69.slice/crio-2fb36f341c1b183188d9814f15e49aef4a1a74a43977bef198b7dd2bc69a1214 WatchSource:0}: Error finding container 2fb36f341c1b183188d9814f15e49aef4a1a74a43977bef198b7dd2bc69a1214: Status 404 returned error can't find the container with id 2fb36f341c1b183188d9814f15e49aef4a1a74a43977bef198b7dd2bc69a1214 Sep 18 16:56:49.004772 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.004748 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/8.log" Sep 18 16:56:49.006449 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.006431 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-jd8k8_13e5f050-0e55-4386-8e80-8927af0ec4d3/console-operator/9.log" Sep 18 16:56:49.323527 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.323492 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" event={"ID":"d312d55e-f987-40e2-ad44-3bfddbc07d69","Type":"ContainerStarted","Data":"564b35309df402a144019c7161694ea6db7a70b0bd4f8d31e138cfaf05cc98a9"} Sep 18 16:56:49.323917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.323548 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" event={"ID":"d312d55e-f987-40e2-ad44-3bfddbc07d69","Type":"ContainerStarted","Data":"2fb36f341c1b183188d9814f15e49aef4a1a74a43977bef198b7dd2bc69a1214"} Sep 18 16:56:49.323917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.323579 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:49.344656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.344609 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" podStartSLOduration=1.344595355 podStartE2EDuration="1.344595355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:49.342265236 +0000 UTC m=+3108.903109904" watchObservedRunningTime="2026-09-18 16:56:49.344595355 +0000 UTC m=+3108.905440055" Sep 18 16:56:49.684704 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:49.684632 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-44tzz_ad5ffd53-9297-4327-901f-01a54926300c/download-server/0.log" Sep 18 16:56:51.641446 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:51.641397 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zkpmd_f5c18c74-c0ae-4cd9-a6af-9733861063b3/dns/0.log" Sep 18 16:56:51.678460 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:51.678431 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zkpmd_f5c18c74-c0ae-4cd9-a6af-9733861063b3/kube-rbac-proxy/0.log" Sep 18 16:56:51.754074 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:51.754051 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bbmtn_77789e19-a00a-403e-9556-65748730930e/dns-node-resolver/0.log" Sep 18 16:56:52.570073 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:52.570045 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qcxrf_acc1a685-72d0-471f-808d-22cfc451dee0/node-ca/0.log" Sep 18 16:56:54.009871 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:54.009835 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2qsjw_0962fb23-150f-4073-9710-a999268241c7/discovery/0.log" Sep 18 16:56:54.165998 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:54.165974 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zrd98_8fccffb1-1210-4544-9243-105721024d50/payload-pre-processing/0.log" Sep 18 16:56:54.191552 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:54.191529 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq_d6b62a3f-96a5-4e56-aca8-3e13af815341/payload-pre-processing/0.log" Sep 18 16:56:54.220632 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:54.220608 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9_541268bf-2fa4-4a20-8403-3eb28672a296/payload-pre-processing/0.log" Sep 18 16:56:54.265030 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:54.264968 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsk8t_a014f89f-e0fb-4319-b6ae-6cba48d802bd/payload-processing/0.log" Sep 18 16:56:54.316287 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:54.316253 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76_c5002529-ed7b-4d73-9154-26a20dfbff00/payload-processing/0.log" Sep 18 16:56:55.339285 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:55.339256 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-r26z2" Sep 18 16:56:55.352538 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:55.352513 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-krt6v_a2730fdd-0e9f-4a38-8eeb-24c40fc83f02/serve-healthcheck-canary/0.log" Sep 18 16:56:55.905638 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:55.905608 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-z2x82_85d4feac-2eae-4628-a1c5-ba2ae60eb119/insights-operator/0.log" Sep 18 16:56:55.906706 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:55.906686 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-z2x82_85d4feac-2eae-4628-a1c5-ba2ae60eb119/insights-operator/1.log" Sep 18 16:56:56.033988 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:56.033961 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhb9c_a0325a6c-18db-43f3-b375-7e95e3555ecf/kube-rbac-proxy/0.log" Sep 18 16:56:56.061832 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:56.061807 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhb9c_a0325a6c-18db-43f3-b375-7e95e3555ecf/exporter/0.log" Sep 18 16:56:56.092151 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:56.092103 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhb9c_a0325a6c-18db-43f3-b375-7e95e3555ecf/extractor/0.log" Sep 18 16:56:58.779283 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.779247 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2qsjw_0962fb23-150f-4073-9710-a999268241c7/discovery/0.log" Sep 18 16:56:58.860081 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.860052 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zrd98_8fccffb1-1210-4544-9243-105721024d50/payload-pre-processing/0.log" Sep 18 16:56:58.884246 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.884218 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq_d6b62a3f-96a5-4e56-aca8-3e13af815341/payload-pre-processing/0.log" Sep 18 16:56:58.903917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.903891 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9_541268bf-2fa4-4a20-8403-3eb28672a296/payload-pre-processing/0.log" Sep 18 16:56:58.936656 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.936622 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsk8t_a014f89f-e0fb-4319-b6ae-6cba48d802bd/payload-processing/0.log" Sep 18 16:56:58.950917 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.950891 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b7898d6ff-2zwrd_d83c9487-feb6-408a-bc8f-1a7add83687d/manager/0.log" Sep 18 16:56:58.989179 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:58.989155 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76_c5002529-ed7b-4d73-9154-26a20dfbff00/payload-processing/0.log" Sep 18 16:56:59.067492 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:59.067428 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_3082c931-af4a-4f03-b525-968fb7f112dc/manager/6.log" Sep 18 16:56:59.068107 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:59.068074 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_3082c931-af4a-4f03-b525-968fb7f112dc/manager/7.log" Sep 18 16:56:59.089893 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:56:59.089867 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-2jckj_3082c931-af4a-4f03-b525-968fb7f112dc/copy-manifests/0.log" Sep 18 16:57:00.694169 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:00.694136 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6847d847f7-fgmrh_132cdab9-9fc4-4348-b41a-bb96fd4116c7/manager/2.log" Sep 18 16:57:00.694882 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:00.694862 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6847d847f7-fgmrh_132cdab9-9fc4-4348-b41a-bb96fd4116c7/manager/1.log" Sep 18 16:57:00.724749 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:00.724721 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wvvkm_f25112d0-8747-4ead-8f96-9b968a57d914/openshift-lws-operator/2.log" Sep 18 16:57:00.725433 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:00.725415 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wvvkm_f25112d0-8747-4ead-8f96-9b968a57d914/openshift-lws-operator/3.log" Sep 18 16:57:01.032622 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:01.032590 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-2qsjw_0962fb23-150f-4073-9710-a999268241c7/discovery/0.log" Sep 18 16:57:01.141180 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:01.141144 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-zrd98_8fccffb1-1210-4544-9243-105721024d50/payload-pre-processing/0.log" Sep 18 16:57:01.156117 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:01.156052 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-a75117-7447578cc4-9khgq_d6b62a3f-96a5-4e56-aca8-3e13af815341/payload-pre-processing/0.log" Sep 18 16:57:01.181427 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:01.181406 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w3-11725e-65d548647d-nnkl9_541268bf-2fa4-4a20-8403-3eb28672a296/payload-pre-processing/0.log" Sep 18 16:57:01.210911 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:01.210879 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsk8t_a014f89f-e0fb-4319-b6ae-6cba48d802bd/payload-processing/0.log" Sep 18 16:57:01.245645 ip-10-0-132-41 kubenswrapper[2563]: I0918 16:57:01.245618 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-11725e-5d5f859dbc-89r76_c5002529-ed7b-4d73-9154-26a20dfbff00/payload-processing/0.log"