Jul 27 15:03:18.258395 ip-10-0-135-218 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 15:03:18.258409 ip-10-0-135-218 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 15:03:18.258418 ip-10-0-135-218 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 15:03:18.258748 ip-10-0-135-218 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 15:03:27.357262 ip-10-0-135-218 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 27 15:03:27.357276 ip-10-0-135-218 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 36a1a6d0acd74e93b05346082e51aaef -- Jul 27 15:04:56.585377 ip-10-0-135-218 systemd[1]: Starting Kubernetes Kubelet... Jul 27 15:04:57.034298 ip-10-0-135-218 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 15:04:57.034298 ip-10-0-135-218 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 15:04:57.034298 ip-10-0-135-218 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 15:04:57.034298 ip-10-0-135-218 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 15:04:57.034298 ip-10-0-135-218 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 15:04:57.036845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.036745 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 15:04:57.040521 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040500 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 15:04:57.040521 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040520 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 15:04:57.040521 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040525 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040528 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040532 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040535 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040538 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040541 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040545 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040548 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040551 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040554 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040557 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040559 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040562 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040565 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040568 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040570 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040574 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040576 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040579 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040582 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 15:04:57.040625 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040584 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040587 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040589 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040592 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040595 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040597 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040600 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040602 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040606 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040608 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040613 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040617 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040620 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040623 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040627 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040630 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040633 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040635 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040639 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 15:04:57.041208 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040642 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040645 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040648 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040650 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040652 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040655 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040658 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040661 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040663 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040665 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040668 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040671 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040673 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040676 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040678 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040681 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040684 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040686 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040689 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040691 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 15:04:57.041710 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040694 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040697 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040703 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040706 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040709 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040713 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040718 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040721 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040724 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040727 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040730 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040732 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040735 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040737 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040740 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040743 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040746 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040748 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040751 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 15:04:57.042215 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040754 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040757 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040759 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040762 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040764 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040767 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040769 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040849 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040854 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040857 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040860 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040863 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040866 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040869 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040871 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040874 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040877 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040880 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.040883 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041648 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 15:04:57.042677 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041656 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041660 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041663 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041666 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041669 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041672 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041675 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041678 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041680 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041683 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041685 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041688 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041692 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041694 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041697 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041699 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041702 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041705 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041707 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041710 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 15:04:57.043216 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041713 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041715 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041718 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041720 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041723 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041726 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041730 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041732 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041735 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041738 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041740 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041743 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041745 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041748 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041750 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041753 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041755 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041758 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041760 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041763 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 15:04:57.043715 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041765 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041768 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041770 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041775 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041778 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041781 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041784 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041787 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041790 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041793 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041796 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041798 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041801 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041803 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041806 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041808 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041811 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041815 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 15:04:57.044222 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041818 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041822 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041826 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041829 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041832 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041835 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041838 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041841 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041843 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041848 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041850 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041853 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041855 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041858 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041860 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041863 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041865 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041868 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041870 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041873 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 15:04:57.044657 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041876 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041878 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041882 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041886 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041889 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041891 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041894 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041896 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041899 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041902 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041905 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041908 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041910 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041913 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041916 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041919 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041921 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041924 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041927 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.041930 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 15:04:57.045187 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042346 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042352 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042354 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042357 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042360 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042363 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042365 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042369 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042371 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042374 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042376 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042379 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042382 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042384 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042387 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042390 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042393 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042395 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042397 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042400 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 15:04:57.045667 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042403 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042405 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042409 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042411 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042414 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042418 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042422 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042425 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042427 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042430 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042433 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042436 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042438 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042441 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042443 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042446 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042448 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042451 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042455 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 15:04:57.046183 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042459 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042463 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042466 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042470 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042473 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042475 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042480 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042483 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042488 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042492 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042495 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042498 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042501 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042504 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042507 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042509 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042512 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042515 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042517 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042520 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 15:04:57.046675 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042522 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042525 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042527 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042530 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042532 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042535 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042537 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042540 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042542 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042546 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042549 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042552 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042554 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042557 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042559 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042561 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042564 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042566 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042569 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042572 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 15:04:57.047254 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042575 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042578 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042580 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042582 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042585 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042588 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042590 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042595 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042598 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042600 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042604 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042608 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042611 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042614 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042617 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042619 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042622 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042624 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042626 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.042629 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 15:04:57.047741 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042709 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042717 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042723 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042728 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042732 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042736 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042740 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042744 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042748 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042750 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042754 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042761 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042765 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042768 2553 flags.go:64] FLAG: --cgroup-root="" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042771 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042774 2553 flags.go:64] FLAG: --client-ca-file="" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042776 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042780 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042784 2553 flags.go:64] FLAG: --cluster-domain="" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042788 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042792 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042795 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042798 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042802 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 15:04:57.048282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042805 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042809 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042812 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042815 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042818 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042821 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042825 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042828 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042832 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042835 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042838 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042841 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042844 2553 flags.go:64] FLAG: --enable-server="true" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042847 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042851 2553 flags.go:64] FLAG: --event-burst="100" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042854 2553 flags.go:64] FLAG: --event-qps="50" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042856 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042859 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042862 2553 flags.go:64] FLAG: --eviction-hard="" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042867 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042870 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042874 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042877 2553 flags.go:64] FLAG: --eviction-soft="" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042880 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042883 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 15:04:57.048845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042886 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042889 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042894 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042897 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042900 2553 flags.go:64] FLAG: --feature-gates="" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042903 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042906 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042909 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042912 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042915 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042918 2553 flags.go:64] FLAG: --help="false" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042921 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042924 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042927 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042930 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042933 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042937 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042940 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042943 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042945 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042948 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042951 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042954 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042957 2553 flags.go:64] FLAG: --kube-reserved="" Jul 27 15:04:57.049492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042960 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042963 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042967 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042970 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042973 2553 flags.go:64] FLAG: --lock-file="" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042976 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042979 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042983 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042989 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042992 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042996 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.042999 2553 flags.go:64] FLAG: --logging-format="text" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043001 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043005 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043007 2553 flags.go:64] FLAG: --manifest-url="" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043010 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043015 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043018 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043022 2553 flags.go:64] FLAG: --max-pods="110" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043025 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043028 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043031 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043034 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043037 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043040 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 15:04:57.050094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043043 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043063 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043066 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043069 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043072 2553 flags.go:64] FLAG: --pod-cidr="" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043075 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043081 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043084 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043087 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043092 2553 flags.go:64] FLAG: --port="10250" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043095 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043098 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5e0671128c2ceaf" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043101 2553 flags.go:64] FLAG: --qos-reserved="" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043105 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043108 2553 flags.go:64] FLAG: --register-node="true" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043111 2553 flags.go:64] FLAG: --register-with-taints="" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043114 2553 flags.go:64] FLAG: --registry-burst="10" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043118 2553 flags.go:64] FLAG: --registry-qps="5" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043121 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043124 2553 flags.go:64] FLAG: --reserved-memory="" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043128 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043131 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043135 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043137 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043141 2553 flags.go:64] FLAG: --runonce="false" Jul 27 15:04:57.050832 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043143 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043147 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043150 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043152 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043155 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043158 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043169 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043173 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043176 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043179 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043181 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043184 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043187 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043190 2553 flags.go:64] FLAG: --system-cgroups="" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043193 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043198 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043203 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043206 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043210 2553 flags.go:64] FLAG: --tls-min-version="" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043213 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043216 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043219 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043222 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043226 2553 flags.go:64] FLAG: --v="2" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043230 2553 flags.go:64] FLAG: --version="false" Jul 27 15:04:57.051782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043237 2553 flags.go:64] FLAG: --vmodule="" Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043241 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.043245 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043364 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043368 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043371 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043376 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043380 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043383 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043386 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043389 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043392 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043394 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043397 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043400 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043403 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043406 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043409 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043411 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043414 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 15:04:57.052456 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043416 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043419 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043422 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043425 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043428 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043432 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043435 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043438 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043441 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043444 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043447 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043450 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043454 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043457 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043460 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043463 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043466 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043469 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043473 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 15:04:57.052943 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043477 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043480 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043483 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043486 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043488 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043491 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043494 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043496 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043499 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043502 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043504 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043507 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043509 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043512 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043514 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043517 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043521 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043524 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043526 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 15:04:57.053549 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043529 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043531 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043534 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043536 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043539 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043542 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043546 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043549 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043551 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043554 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043556 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043559 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043561 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043564 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043566 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043569 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043571 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043574 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043577 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043579 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 15:04:57.054008 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043582 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043584 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043587 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043589 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043592 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043594 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043597 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043599 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043602 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043605 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043608 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043610 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043613 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043616 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043618 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043620 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043623 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043626 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043630 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043633 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 15:04:57.054518 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043635 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043638 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043640 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.043643 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.044180 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.052399 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.052416 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.053837 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.053845 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053906 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053911 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053914 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053918 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053920 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053923 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 15:04:57.054999 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053926 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053928 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053931 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053934 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053936 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053939 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053941 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053944 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053947 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053949 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053953 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053956 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053959 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053961 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053964 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053967 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053969 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053972 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053975 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 15:04:57.055392 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053977 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053980 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053983 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053985 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053988 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053992 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.053996 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054000 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054002 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054005 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054008 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054012 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054015 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054018 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054021 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054024 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054027 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054030 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054034 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 15:04:57.055845 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054037 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054040 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054043 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054058 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054064 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054067 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054070 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054073 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054075 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054078 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054081 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054084 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054086 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054089 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054092 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054094 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054097 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054100 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054103 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054105 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 15:04:57.056316 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054108 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054111 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054113 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054116 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054119 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054122 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054126 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054130 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054132 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054135 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054137 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054140 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054143 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054145 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054148 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054151 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054154 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054156 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054159 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054162 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 15:04:57.056815 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054164 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054166 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054169 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054172 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054174 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054177 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054180 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054182 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054186 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054188 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054191 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054195 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054197 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054200 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054202 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 15:04:57.057353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.054208 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054328 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054333 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054336 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054339 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054342 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054344 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054348 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054351 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054353 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054356 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054359 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054362 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054366 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054368 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054371 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054373 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054377 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054382 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 15:04:57.057711 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054385 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054388 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054391 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054393 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054396 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054399 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054402 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054405 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054407 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054410 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054413 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054415 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054418 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054420 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054423 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054425 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054428 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054430 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054433 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054435 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 15:04:57.058210 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054438 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054441 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054443 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054446 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054449 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054451 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054454 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054458 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054461 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054464 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054466 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054469 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054471 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054474 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054477 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054479 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054482 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054484 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054487 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054489 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 15:04:57.058689 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054492 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054494 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054497 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054500 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054503 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054505 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054508 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054510 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054513 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054515 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054518 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054520 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054523 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054526 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054529 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054532 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054534 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054537 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054539 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054542 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 15:04:57.059195 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054545 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054548 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054551 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054553 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054555 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054558 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054561 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054564 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054566 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054570 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054573 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054576 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054579 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054581 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054588 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054591 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054593 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054597 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054601 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 27 15:04:57.059687 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054604 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:57.054607 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.054612 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.054751 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.056946 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.058356 2553 server.go:1019] "Starting client certificate rotation" Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.058463 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 15:04:57.060181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.058499 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 15:04:57.061940 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.061925 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 15:04:57.063137 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.063118 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 15:04:57.072759 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.072740 2553 log.go:25] "Validated CRI v1 runtime API" Jul 27 15:04:57.074895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.074880 2553 log.go:25] "Validated CRI v1 image API" Jul 27 15:04:57.075811 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.075795 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 15:04:57.079246 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.079227 2553 fs.go:135] Filesystem UUIDs: map[70e0116a-2012-4288-bf52-f22cee3ed58f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ff43d058-446a-4c9c-b721-8f4c0e4c06a9:/dev/nvme0n1p4] Jul 27 15:04:57.079328 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.079245 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 15:04:57.085652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.085524 2553 manager.go:216] Machine: {Timestamp:2026-07-27 15:04:57.084148277 +0000 UTC m=+0.365498342 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096798 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b5d27d77691b7845239564742204 SystemUUID:ec24b5d2-7d77-691b-7845-239564742204 BootID:36a1a6d0-acd7-4e93-b053-46082e51aaef Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:a6:a1:c6:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:a6:a1:c6:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:fd:f4:ee:89:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 15:04:57.085652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.085647 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 15:04:57.085777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.085751 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 15:04:57.085809 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.085768 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 15:04:57.088059 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.088025 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 15:04:57.088193 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.088061 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-218.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 15:04:57.088847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.088837 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 15:04:57.088876 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.088850 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 15:04:57.088876 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.088858 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 15:04:57.088876 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.088871 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 15:04:57.090161 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.090149 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 15:04:57.092950 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.092938 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 27 15:04:57.093083 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.093073 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 15:04:57.093123 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.093105 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 27 15:04:57.093123 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.093114 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 15:04:57.093204 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.093139 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 15:04:57.093204 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.093149 2553 kubelet.go:398] "Adding apiserver pod source" Jul 27 15:04:57.093204 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.093168 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 15:04:57.094562 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.094533 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 15:04:57.094672 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.094575 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 15:04:57.097888 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.097863 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 15:04:57.101279 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.101264 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 15:04:57.101356 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.101293 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 15:04:57.102395 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.102372 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 15:04:57.102395 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.102386 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 15:04:57.102634 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102623 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 15:04:57.102667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102639 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 15:04:57.102667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102645 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 15:04:57.102667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102651 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 15:04:57.102667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102660 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 15:04:57.102667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102665 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 15:04:57.102794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102673 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 15:04:57.102794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102678 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 15:04:57.102794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102685 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 15:04:57.102794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102691 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 15:04:57.102794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102699 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 15:04:57.102794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.102704 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 15:04:57.103398 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.103388 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 15:04:57.103398 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.103398 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 15:04:57.106330 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.106305 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-218.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 15:04:57.106713 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.106698 2553 server.go:1268] "Started kubelet" Jul 27 15:04:57.106774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.106728 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 15:04:57.106831 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.106757 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 15:04:57.106877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.106845 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 15:04:57.107533 ip-10-0-135-218 systemd[1]: Started Kubernetes Kubelet. Jul 27 15:04:57.107969 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.107952 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 15:04:57.111600 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.111578 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 15:04:57.111859 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.111706 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 27 15:04:57.115495 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.115479 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 15:04:57.115582 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.115500 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 15:04:57.115639 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.115584 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.116116 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116095 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 15:04:57.116116 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116095 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 15:04:57.116248 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116130 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 15:04:57.116306 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116260 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 27 15:04:57.116306 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116268 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 27 15:04:57.116460 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116441 2553 factory.go:55] Registering systemd factory Jul 27 15:04:57.116548 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.116464 2553 factory.go:223] Registration of the systemd container factory successfully Jul 27 15:04:57.117788 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.117458 2553 factory.go:153] Registering CRI-O factory Jul 27 15:04:57.117788 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.117474 2553 factory.go:223] Registration of the crio container factory successfully Jul 27 15:04:57.117788 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.117555 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 15:04:57.117788 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.117576 2553 factory.go:103] Registering Raw factory Jul 27 15:04:57.117788 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.117590 2553 manager.go:1196] Started watching for new ooms in manager Jul 27 15:04:57.118040 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.118029 2553 manager.go:349] Starting recovery of all containers Jul 27 15:04:57.118422 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.117345 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-218.ec2.internal.18c62dd574990d61 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-218.ec2.internal,UID:ip-10-0-135-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-218.ec2.internal,},FirstTimestamp:2026-07-27 15:04:57.106451809 +0000 UTC m=+0.387801875,LastTimestamp:2026-07-27 15:04:57.106451809 +0000 UTC m=+0.387801875,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-218.ec2.internal,}" Jul 27 15:04:57.121507 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.121483 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 15:04:57.121654 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.121635 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-218.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 15:04:57.129379 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.129354 2553 manager.go:354] Recovery completed Jul 27 15:04:57.134816 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.134803 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.136701 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.136681 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.136764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.136718 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.136764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.136732 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.137395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.137381 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.137476 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.137380 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 15:04:57.137476 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.137411 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 15:04:57.137476 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.137428 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 27 15:04:57.138087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.138071 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.138200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.138101 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.138200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.138118 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.138304 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.138237 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-218.ec2.internal.18c62dd57666a2ad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-218.ec2.internal,UID:ip-10-0-135-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-218.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-218.ec2.internal,},FirstTimestamp:2026-07-27 15:04:57.136702125 +0000 UTC m=+0.418052194,LastTimestamp:2026-07-27 15:04:57.136702125 +0000 UTC m=+0.418052194,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-218.ec2.internal,}" Jul 27 15:04:57.139624 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.139612 2553 policy_none.go:49] "None policy: Start" Jul 27 15:04:57.139667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.139631 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 15:04:57.139667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.139644 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 15:04:57.140980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.140969 2553 policy_none.go:47] "Start" Jul 27 15:04:57.141402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.141383 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6l59" Jul 27 15:04:57.149317 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.149254 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-218.ec2.internal.18c62dd57666f8e3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-218.ec2.internal,UID:ip-10-0-135-218.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-218.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-218.ec2.internal,},FirstTimestamp:2026-07-27 15:04:57.136724195 +0000 UTC m=+0.418074261,LastTimestamp:2026-07-27 15:04:57.136724195 +0000 UTC m=+0.418074261,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-218.ec2.internal,}" Jul 27 15:04:57.151488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.151467 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6l59" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.181987 2553 manager.go:337] "Starting Device Plugin manager" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.182029 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182041 2553 server.go:87] "Starting device plugin registration server" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182261 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182328 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182343 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182590 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182674 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.182682 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.183142 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 15:04:57.184504 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.183207 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.184928 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.184913 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 15:04:57.184976 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.184952 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 15:04:57.184976 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.184977 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 15:04:57.185062 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.185012 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 15:04:57.187420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.187403 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 15:04:57.283261 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.283216 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.284420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.284361 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.284420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.284393 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.284420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.284408 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.284570 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.284439 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.285425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.285407 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal"] Jul 27 15:04:57.285485 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.285466 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.286269 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.286245 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.286356 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.286274 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.286356 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.286287 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.287525 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.287513 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.287648 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.287634 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.287694 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.287662 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.288243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.288228 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.288243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.288238 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.288352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.288259 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.288352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.288269 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.288352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.288259 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.288352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.288347 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.289544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.289527 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.289630 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.289552 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:57.290273 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.290257 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:57.290346 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.290280 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:57.290346 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.290293 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:57.296185 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.296169 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.296247 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.296190 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-218.ec2.internal\": node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.314851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.314829 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-218.ec2.internal\" not found" node="ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.317310 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.317286 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec4ffb043a26cecc367b10a3cee4e9b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal\" (UID: \"ec4ffb043a26cecc367b10a3cee4e9b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.317417 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.317324 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec4ffb043a26cecc367b10a3cee4e9b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal\" (UID: \"ec4ffb043a26cecc367b10a3cee4e9b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.317417 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.317349 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a58aa08a3e395c7a07bb090be093243d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-218.ec2.internal\" (UID: \"a58aa08a3e395c7a07bb090be093243d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.319416 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.319395 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-218.ec2.internal\" not found" node="ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.319673 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.319655 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.418657 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.418444 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec4ffb043a26cecc367b10a3cee4e9b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal\" (UID: \"ec4ffb043a26cecc367b10a3cee4e9b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.418657 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.418664 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec4ffb043a26cecc367b10a3cee4e9b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal\" (UID: \"ec4ffb043a26cecc367b10a3cee4e9b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.418855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.418684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a58aa08a3e395c7a07bb090be093243d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-218.ec2.internal\" (UID: \"a58aa08a3e395c7a07bb090be093243d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.418855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.418709 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a58aa08a3e395c7a07bb090be093243d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-218.ec2.internal\" (UID: \"a58aa08a3e395c7a07bb090be093243d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.418855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.418533 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ec4ffb043a26cecc367b10a3cee4e9b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal\" (UID: \"ec4ffb043a26cecc367b10a3cee4e9b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.418855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.418756 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec4ffb043a26cecc367b10a3cee4e9b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal\" (UID: \"ec4ffb043a26cecc367b10a3cee4e9b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.420568 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.420549 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.521124 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.521090 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.617799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.617700 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.621168 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.621144 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.623319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.623303 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" Jul 27 15:04:57.721400 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.721363 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.821796 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.821763 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.921915 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:57.921883 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:57.954552 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:57.954526 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 15:04:58.022132 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:58.022087 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:58.057903 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.057864 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 15:04:58.058402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.058044 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 15:04:58.058402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.058108 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 15:04:58.119203 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.119174 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 15:04:58.122724 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:58.122700 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:58.155707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.155664 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 14:59:57 +0000 UTC" deadline="2027-12-24 00:07:10.420599188 +0000 UTC" Jul 27 15:04:58.155707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.155700 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12345h2m12.264902867s" Jul 27 15:04:58.223473 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:58.223442 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:58.244151 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.244118 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 15:04:58.245457 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.245441 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientMemory" Jul 27 15:04:58.245524 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.245473 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 15:04:58.245524 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.245484 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeHasSufficientPID" Jul 27 15:04:58.259599 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:58.259565 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58aa08a3e395c7a07bb090be093243d.slice/crio-f99c5f2dbb3d4471f81c481a5f0d36a1bd816050791385c31fa9297cb0e209d6 WatchSource:0}: Error finding container f99c5f2dbb3d4471f81c481a5f0d36a1bd816050791385c31fa9297cb0e209d6: Status 404 returned error can't find the container with id f99c5f2dbb3d4471f81c481a5f0d36a1bd816050791385c31fa9297cb0e209d6 Jul 27 15:04:58.265495 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.265480 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 15:04:58.325681 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:58.325645 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-218.ec2.internal\" not found" Jul 27 15:04:58.407615 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.407590 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 15:04:58.416735 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.416682 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" Jul 27 15:04:58.424537 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.424516 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 15:04:58.425527 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.425514 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" Jul 27 15:04:58.438018 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:58.437996 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 15:04:59.095214 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.095184 2553 apiserver.go:52] "Watching apiserver" Jul 27 15:04:59.102830 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.102804 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 15:04:59.104271 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.104245 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-mc7bs","kube-system/konnectivity-agent-x9x9q","openshift-cluster-node-tuning-operator/tuned-lxg7n","openshift-dns/node-resolver-bmnp2","openshift-image-registry/node-ca-l24dr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal","openshift-multus/multus-additional-cni-plugins-fnsgs","openshift-multus/multus-fb6tm","openshift-network-diagnostics/network-check-target-kcnfc","kube-system/global-pull-secret-syncer-bjd66","kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt","openshift-multus/network-metrics-daemon-cv4fx","openshift-network-operator/iptables-alerter-9zff8"] Jul 27 15:04:59.105855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.105830 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.107081 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.107048 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.107194 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.107141 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:04:59.108039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.107875 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 15:04:59.108039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.107900 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 15:04:59.108039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.108016 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.108268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.108253 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4xvv\"" Jul 27 15:04:59.108327 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.108277 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.108327 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.108307 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.108432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.108383 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 15:04:59.109721 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.109691 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.110318 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.110290 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.110318 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.110291 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gphkj\"" Jul 27 15:04:59.110477 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.110335 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.111245 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.110915 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.113155 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.112464 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.113155 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.112561 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.113155 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.112861 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mnxpm\"" Jul 27 15:04:59.115528 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.115505 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.115608 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.115531 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.115608 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.115546 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 15:04:59.115783 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.115762 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.115923 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.115773 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vwsgk\"" Jul 27 15:04:59.116412 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.116394 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 15:04:59.117355 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.117337 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.118093 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.118046 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 15:04:59.118093 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.118085 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gbpvp\"" Jul 27 15:04:59.118346 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.118309 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:04:59.118481 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.118436 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:04:59.119873 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.119852 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.120254 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.120091 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.121273 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.121254 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.121830 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.121789 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rwmd4\"" Jul 27 15:04:59.121924 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.121855 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 15:04:59.122545 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.122525 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.122649 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.122606 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:04:59.123108 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123074 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 15:04:59.123353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123331 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 15:04:59.123637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123615 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.124324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123722 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.124324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123757 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 15:04:59.124324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123883 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 15:04:59.124324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.123997 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-plp5z\"" Jul 27 15:04:59.124324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.124091 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.124324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.124108 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 15:04:59.125595 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.125576 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.126466 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.126238 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-chjkj\"" Jul 27 15:04:59.126466 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.126319 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 15:04:59.126949 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.126930 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 15:04:59.127488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127342 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 15:04:59.127488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127424 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 15:04:59.127488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127457 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 15:04:59.127707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127687 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:04:59.127770 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127727 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-systemd\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.127770 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127754 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-var-lib-kubelet\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.127886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127826 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65b2001b-a43e-4060-8dc1-e69aae55dd22-serviceca\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.127886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127857 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdfrs\" (UniqueName: \"kubernetes.io/projected/66371b5f-01b5-4329-bc7e-069d801574e9-kube-api-access-xdfrs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.127989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-system-cni-dir\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.127989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127925 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cnibin\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.127989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa7f660f-4dd4-457a-950c-07412f65b7da-tmp-dir\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.127987 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65b2001b-a43e-4060-8dc1-e69aae55dd22-host\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128012 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkkl\" (UniqueName: \"kubernetes.io/projected/65b2001b-a43e-4060-8dc1-e69aae55dd22-kube-api-access-5wkkl\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-tuned\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128086 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xxm\" (UniqueName: \"kubernetes.io/projected/3b0434b1-17f8-4bc1-8e65-aa02f5192404-kube-api-access-g7xxm\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128112 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.128179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128157 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt7hr\" (UniqueName: \"kubernetes.io/projected/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-kube-api-access-jt7hr\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128183 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysctl-conf\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128222 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b0434b1-17f8-4bc1-8e65-aa02f5192404-iptables-alerter-script\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128247 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0434b1-17f8-4bc1-8e65-aa02f5192404-host-slash\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128275 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128317 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128366 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-lib-modules\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128390 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sz6gt\"" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128398 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-host\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128426 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa7f660f-4dd4-457a-950c-07412f65b7da-hosts-file\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128475 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdz27\" (UniqueName: \"kubernetes.io/projected/fa7f660f-4dd4-457a-950c-07412f65b7da-kube-api-access-sdz27\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128480 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128505 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-modprobe-d\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.128544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128541 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysctl-d\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128579 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bab4b666-154e-4f34-aa46-ea55efe66af1-tmp\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128626 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-os-release\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128668 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128701 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysconfig\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128729 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-run\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128752 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-kubernetes\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128775 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-sys\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.129201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.128802 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wf74\" (UniqueName: \"kubernetes.io/projected/bab4b666-154e-4f34-aa46-ea55efe66af1-kube-api-access-9wf74\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.150115 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.150093 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qgxz7" Jul 27 15:04:59.157763 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.157738 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qgxz7" Jul 27 15:04:59.190487 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.190443 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" event={"ID":"a58aa08a3e395c7a07bb090be093243d","Type":"ContainerStarted","Data":"f99c5f2dbb3d4471f81c481a5f0d36a1bd816050791385c31fa9297cb0e209d6"} Jul 27 15:04:59.217473 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.217447 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 15:04:59.229292 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229257 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-k8s-cni-cncf-io\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.229475 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229318 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbhd\" (UniqueName: \"kubernetes.io/projected/97244c43-6c8f-463f-8cd7-42828e29ce00-kube-api-access-8mbhd\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.229475 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229340 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.229475 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229364 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-registration-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.229475 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229413 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:04:59.229475 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdfrs\" (UniqueName: \"kubernetes.io/projected/66371b5f-01b5-4329-bc7e-069d801574e9-kube-api-access-xdfrs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.229756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229518 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-hostroot\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.229756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229582 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.229756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229611 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-system-cni-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.229756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229673 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-cni-multus\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.229756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229701 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjx8\" (UniqueName: \"kubernetes.io/projected/c984c1f2-393e-4cc4-8b16-73d149c7726a-kube-api-access-pvjx8\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229761 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-tuned\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229788 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xxm\" (UniqueName: \"kubernetes.io/projected/3b0434b1-17f8-4bc1-8e65-aa02f5192404-kube-api-access-g7xxm\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt7hr\" (UniqueName: \"kubernetes.io/projected/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-kube-api-access-jt7hr\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229840 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-os-release\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229867 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-socket-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229893 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-systemd-units\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229917 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-run-netns\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229941 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-ovn\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.229968 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0434b1-17f8-4bc1-8e65-aa02f5192404-host-slash\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-multus-certs\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230092 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b0434b1-17f8-4bc1-8e65-aa02f5192404-host-slash\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230118 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-log-socket\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230147 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovnkube-script-lib\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230174 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-lib-modules\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230201 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa7f660f-4dd4-457a-950c-07412f65b7da-hosts-file\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.230223 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230226 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdz27\" (UniqueName: \"kubernetes.io/projected/fa7f660f-4dd4-457a-950c-07412f65b7da-kube-api-access-sdz27\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fa7f660f-4dd4-457a-950c-07412f65b7da-hosts-file\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.230325 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:04:59.730280115 +0000 UTC m=+3.011630175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230329 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-lib-modules\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230372 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-slash\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.230415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230402 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovn-node-metrics-cert\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysctl-d\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230442 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230466 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-cnibin\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-cni-netd\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230526 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-netns\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230553 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-etc-kubernetes\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230557 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysctl-d\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230578 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230607 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230629 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-kubernetes\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230646 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-sys\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230668 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-kubelet-config\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230694 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-conf-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230715 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-sys\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230719 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-daemon-config\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230722 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-kubernetes\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230749 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-systemd\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-var-lib-kubelet\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230787 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65b2001b-a43e-4060-8dc1-e69aae55dd22-serviceca\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230838 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-systemd\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230863 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-var-lib-kubelet\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230880 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-system-cni-dir\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230910 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cnibin\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230938 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-device-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230935 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-system-cni-dir\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa7f660f-4dd4-457a-950c-07412f65b7da-tmp-dir\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230986 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65b2001b-a43e-4060-8dc1-e69aae55dd22-host\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.230994 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cnibin\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231034 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkkl\" (UniqueName: \"kubernetes.io/projected/65b2001b-a43e-4060-8dc1-e69aae55dd22-kube-api-access-5wkkl\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231062 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46efa81f-a600-439b-b979-abb5028bf7b1-agent-certs\") pod \"konnectivity-agent-x9x9q\" (UID: \"46efa81f-a600-439b-b979-abb5028bf7b1\") " pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231130 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231172 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-var-lib-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.231925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231198 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-node-log\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231244 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231272 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231284 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa7f660f-4dd4-457a-950c-07412f65b7da-tmp-dir\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231291 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65b2001b-a43e-4060-8dc1-e69aae55dd22-serviceca\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231302 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-cni-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231347 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhv2\" (UniqueName: \"kubernetes.io/projected/139a73fe-3f5a-4313-a8be-8a30db9dcda5-kube-api-access-dlhv2\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231373 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65b2001b-a43e-4060-8dc1-e69aae55dd22-host\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231377 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysctl-conf\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231408 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b0434b1-17f8-4bc1-8e65-aa02f5192404-iptables-alerter-script\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231437 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231517 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-cni-bin\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231543 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-etc-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231568 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-host\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231607 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-systemd\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231672 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-cni-bin\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231694 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovnkube-config\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.232695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231713 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231806 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-modprobe-d\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231858 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231911 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-host\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232065 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysctl-conf\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.231717 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-modprobe-d\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232234 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bab4b666-154e-4f34-aa46-ea55efe66af1-tmp\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b0434b1-17f8-4bc1-8e65-aa02f5192404-iptables-alerter-script\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232262 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-os-release\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232282 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232322 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-os-release\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232339 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46efa81f-a600-439b-b979-abb5028bf7b1-konnectivity-ca\") pod \"konnectivity-agent-x9x9q\" (UID: \"46efa81f-a600-439b-b979-abb5028bf7b1\") " pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232385 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97244c43-6c8f-463f-8cd7-42828e29ce00-cni-binary-copy\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232416 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-env-overrides\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232441 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysconfig\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232466 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-run\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.233563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232510 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-dbus\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232535 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-socket-dir-parent\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232559 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-kubelet\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-sys-fs\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232615 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-kubelet\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-sysconfig\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232662 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wf74\" (UniqueName: \"kubernetes.io/projected/bab4b666-154e-4f34-aa46-ea55efe66af1-kube-api-access-9wf74\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232743 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bab4b666-154e-4f34-aa46-ea55efe66af1-run\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.234323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.232825 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.235110 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.235087 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bab4b666-154e-4f34-aa46-ea55efe66af1-etc-tuned\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.235222 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.235133 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bab4b666-154e-4f34-aa46-ea55efe66af1-tmp\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.256776 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.256653 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:04:59.256776 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.256682 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:04:59.256776 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.256695 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:04:59.256776 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.256761 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:04:59.756740878 +0000 UTC m=+3.038090937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:04:59.259319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.259293 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdz27\" (UniqueName: \"kubernetes.io/projected/fa7f660f-4dd4-457a-950c-07412f65b7da-kube-api-access-sdz27\") pod \"node-resolver-bmnp2\" (UID: \"fa7f660f-4dd4-457a-950c-07412f65b7da\") " pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.259445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.259345 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt7hr\" (UniqueName: \"kubernetes.io/projected/1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8-kube-api-access-jt7hr\") pod \"multus-additional-cni-plugins-fnsgs\" (UID: \"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8\") " pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.259824 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.259778 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wf74\" (UniqueName: \"kubernetes.io/projected/bab4b666-154e-4f34-aa46-ea55efe66af1-kube-api-access-9wf74\") pod \"tuned-lxg7n\" (UID: \"bab4b666-154e-4f34-aa46-ea55efe66af1\") " pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.259952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.259935 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkkl\" (UniqueName: \"kubernetes.io/projected/65b2001b-a43e-4060-8dc1-e69aae55dd22-kube-api-access-5wkkl\") pod \"node-ca-l24dr\" (UID: \"65b2001b-a43e-4060-8dc1-e69aae55dd22\") " pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.260124 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.260105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xxm\" (UniqueName: \"kubernetes.io/projected/3b0434b1-17f8-4bc1-8e65-aa02f5192404-kube-api-access-g7xxm\") pod \"iptables-alerter-9zff8\" (UID: \"3b0434b1-17f8-4bc1-8e65-aa02f5192404\") " pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.260180 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.260148 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdfrs\" (UniqueName: \"kubernetes.io/projected/66371b5f-01b5-4329-bc7e-069d801574e9-kube-api-access-xdfrs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.334882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334270 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbhd\" (UniqueName: \"kubernetes.io/projected/97244c43-6c8f-463f-8cd7-42828e29ce00-kube-api-access-8mbhd\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.334882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334325 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.334882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-registration-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.334882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334424 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-hostroot\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.334882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334499 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-hostroot\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.334882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.334951 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-registration-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335119 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-system-cni-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335146 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-cni-multus\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjx8\" (UniqueName: \"kubernetes.io/projected/c984c1f2-393e-4cc4-8b16-73d149c7726a-kube-api-access-pvjx8\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335214 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-os-release\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-socket-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335288 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-systemd-units\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-run-netns\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335345 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-ovn\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335389 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-multus-certs\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335420 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-log-socket\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335450 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovnkube-script-lib\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-slash\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335512 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovn-node-metrics-cert\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335544 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-cnibin\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335576 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-cni-netd\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335610 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-netns\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335643 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-etc-kubernetes\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335674 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.335730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-kubelet-config\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335771 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-conf-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335810 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-daemon-config\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335842 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-device-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335878 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46efa81f-a600-439b-b979-abb5028bf7b1-agent-certs\") pod \"konnectivity-agent-x9x9q\" (UID: \"46efa81f-a600-439b-b979-abb5028bf7b1\") " pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335914 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335944 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-var-lib-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.335973 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-node-log\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336035 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-cni-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336066 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhv2\" (UniqueName: \"kubernetes.io/projected/139a73fe-3f5a-4313-a8be-8a30db9dcda5-kube-api-access-dlhv2\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336107 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-cni-bin\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336153 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-etc-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336190 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-systemd\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336222 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-cni-bin\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovnkube-config\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.336291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336282 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46efa81f-a600-439b-b979-abb5028bf7b1-konnectivity-ca\") pod \"konnectivity-agent-x9x9q\" (UID: \"46efa81f-a600-439b-b979-abb5028bf7b1\") " pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336315 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97244c43-6c8f-463f-8cd7-42828e29ce00-cni-binary-copy\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336347 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-env-overrides\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336374 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-dbus\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336405 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-socket-dir-parent\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-kubelet\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-sys-fs\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-kubelet\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-k8s-cni-cncf-io\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336644 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-k8s-cni-cncf-io\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336698 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336762 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-system-cni-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.336810 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-cni-multus\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337164 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-os-release\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337320 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-socket-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337377 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-systemd-units\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337434 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-run-netns\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337486 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-ovn\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-multus-certs\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.337700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.337584 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-log-socket\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.338266 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.338237 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovnkube-script-lib\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.338411 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.338320 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-slash\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.338840 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.338818 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:04:59.338928 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.338892 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:04:59.838873154 +0000 UTC m=+3.120223207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:04:59.339087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339060 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-conf-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.339087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339077 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-run-netns\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.339213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339086 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-cni-netd\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339175 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-run-ovn-kubernetes\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339185 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-run-systemd\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339404 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339259 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-etc-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339404 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339395 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-etc-selinux\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.339472 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339407 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-cni-bin\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339506 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339468 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-node-log\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339538 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339527 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.339603 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339586 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-kubelet-config\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.339656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339637 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-cni-bin\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.339975 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.339954 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovnkube-config\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.340088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340051 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-socket-dir-parent\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-daemon-config\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340263 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-device-dir\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340606 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97244c43-6c8f-463f-8cd7-42828e29ce00-cni-binary-copy\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340682 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/139a73fe-3f5a-4313-a8be-8a30db9dcda5-sys-fs\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340749 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-host-var-lib-kubelet\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340795 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-host-kubelet\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340840 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-dbus\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.340856 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-multus-cni-dir\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.341011 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-etc-kubernetes\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.341319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.341300 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c984c1f2-393e-4cc4-8b16-73d149c7726a-env-overrides\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.341986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.341417 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c984c1f2-393e-4cc4-8b16-73d149c7726a-var-lib-openvswitch\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.342195 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.342164 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97244c43-6c8f-463f-8cd7-42828e29ce00-cnibin\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.342734 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.342681 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46efa81f-a600-439b-b979-abb5028bf7b1-konnectivity-ca\") pod \"konnectivity-agent-x9x9q\" (UID: \"46efa81f-a600-439b-b979-abb5028bf7b1\") " pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.342829 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.342795 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c984c1f2-393e-4cc4-8b16-73d149c7726a-ovn-node-metrics-cert\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.345735 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.345657 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46efa81f-a600-439b-b979-abb5028bf7b1-agent-certs\") pod \"konnectivity-agent-x9x9q\" (UID: \"46efa81f-a600-439b-b979-abb5028bf7b1\") " pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.346869 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.346819 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjx8\" (UniqueName: \"kubernetes.io/projected/c984c1f2-393e-4cc4-8b16-73d149c7726a-kube-api-access-pvjx8\") pod \"ovnkube-node-mc7bs\" (UID: \"c984c1f2-393e-4cc4-8b16-73d149c7726a\") " pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.348096 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.348064 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbhd\" (UniqueName: \"kubernetes.io/projected/97244c43-6c8f-463f-8cd7-42828e29ce00-kube-api-access-8mbhd\") pod \"multus-fb6tm\" (UID: \"97244c43-6c8f-463f-8cd7-42828e29ce00\") " pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.350725 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.350701 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhv2\" (UniqueName: \"kubernetes.io/projected/139a73fe-3f5a-4313-a8be-8a30db9dcda5-kube-api-access-dlhv2\") pod \"aws-ebs-csi-driver-node-v6kkt\" (UID: \"139a73fe-3f5a-4313-a8be-8a30db9dcda5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.419209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.419174 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" Jul 27 15:04:59.429577 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.429554 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l24dr" Jul 27 15:04:59.430014 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.429978 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a31ecc7_38eb_4962_9b1b_eb8b418fa2d8.slice/crio-e5d71decedfa01b446f7637f25419d7ee9b86f5d5c0480411fbe3f1de5882f4b WatchSource:0}: Error finding container e5d71decedfa01b446f7637f25419d7ee9b86f5d5c0480411fbe3f1de5882f4b: Status 404 returned error can't find the container with id e5d71decedfa01b446f7637f25419d7ee9b86f5d5c0480411fbe3f1de5882f4b Jul 27 15:04:59.437738 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.437716 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" Jul 27 15:04:59.439505 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.439321 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b2001b_a43e_4060_8dc1_e69aae55dd22.slice/crio-4b31e419cccb750d7965f2658ae4a4be55829a844bd7a6b3062a17a3fde48e69 WatchSource:0}: Error finding container 4b31e419cccb750d7965f2658ae4a4be55829a844bd7a6b3062a17a3fde48e69: Status 404 returned error can't find the container with id 4b31e419cccb750d7965f2658ae4a4be55829a844bd7a6b3062a17a3fde48e69 Jul 27 15:04:59.443098 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.443061 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bmnp2" Jul 27 15:04:59.447248 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.447217 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab4b666_154e_4f34_aa46_ea55efe66af1.slice/crio-34a950dbe1538c022bbd9ce9a3ad9e23fca219525a4485595ebc711f50525465 WatchSource:0}: Error finding container 34a950dbe1538c022bbd9ce9a3ad9e23fca219525a4485595ebc711f50525465: Status 404 returned error can't find the container with id 34a950dbe1538c022bbd9ce9a3ad9e23fca219525a4485595ebc711f50525465 Jul 27 15:04:59.451220 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.451194 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9zff8" Jul 27 15:04:59.453556 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.453456 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7f660f_4dd4_457a_950c_07412f65b7da.slice/crio-863b03104be0c02536614031a69ba7dfee0aec48988a6ed2105f4c76eabec04d WatchSource:0}: Error finding container 863b03104be0c02536614031a69ba7dfee0aec48988a6ed2105f4c76eabec04d: Status 404 returned error can't find the container with id 863b03104be0c02536614031a69ba7dfee0aec48988a6ed2105f4c76eabec04d Jul 27 15:04:59.459403 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.459341 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:04:59.460709 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.460689 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0434b1_17f8_4bc1_8e65_aa02f5192404.slice/crio-2e00daea3c83f9b686b15535e97eabc822fb0a3ce822fdd2793ace3a303613f0 WatchSource:0}: Error finding container 2e00daea3c83f9b686b15535e97eabc822fb0a3ce822fdd2793ace3a303613f0: Status 404 returned error can't find the container with id 2e00daea3c83f9b686b15535e97eabc822fb0a3ce822fdd2793ace3a303613f0 Jul 27 15:04:59.465837 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.465655 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:04:59.470152 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.470109 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc984c1f2_393e_4cc4_8b16_73d149c7726a.slice/crio-ad26ac7e785e33273faee4cc7a40a7ba80159081a9d1b4fb991b4d33a112d03e WatchSource:0}: Error finding container ad26ac7e785e33273faee4cc7a40a7ba80159081a9d1b4fb991b4d33a112d03e: Status 404 returned error can't find the container with id ad26ac7e785e33273faee4cc7a40a7ba80159081a9d1b4fb991b4d33a112d03e Jul 27 15:04:59.474109 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.474088 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fb6tm" Jul 27 15:04:59.474814 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.474794 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46efa81f_a600_439b_b979_abb5028bf7b1.slice/crio-b2df0c80de0f40938281fed6df44ce3704dbccd59d846a3415934d7dd3028493 WatchSource:0}: Error finding container b2df0c80de0f40938281fed6df44ce3704dbccd59d846a3415934d7dd3028493: Status 404 returned error can't find the container with id b2df0c80de0f40938281fed6df44ce3704dbccd59d846a3415934d7dd3028493 Jul 27 15:04:59.479499 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.479479 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" Jul 27 15:04:59.483363 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.483339 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97244c43_6c8f_463f_8cd7_42828e29ce00.slice/crio-29a3889c4588281a881a52bf5bbd89a6624a97b2f7d9e5f94d667d4e96273e2a WatchSource:0}: Error finding container 29a3889c4588281a881a52bf5bbd89a6624a97b2f7d9e5f94d667d4e96273e2a: Status 404 returned error can't find the container with id 29a3889c4588281a881a52bf5bbd89a6624a97b2f7d9e5f94d667d4e96273e2a Jul 27 15:04:59.488750 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:04:59.488725 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139a73fe_3f5a_4313_a8be_8a30db9dcda5.slice/crio-21f0e2aede93513be0f99f43e6141501620cc01f6bfe2975807c8fbea125a3d0 WatchSource:0}: Error finding container 21f0e2aede93513be0f99f43e6141501620cc01f6bfe2975807c8fbea125a3d0: Status 404 returned error can't find the container with id 21f0e2aede93513be0f99f43e6141501620cc01f6bfe2975807c8fbea125a3d0 Jul 27 15:04:59.740236 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.740201 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:04:59.740436 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.740346 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:04:59.740436 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.740411 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:00.740395277 +0000 UTC m=+4.021745331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:04:59.841087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.841042 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:04:59.841259 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:04:59.841182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:04:59.841259 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.841236 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:04:59.841465 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.841265 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:04:59.841465 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.841277 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:04:59.841465 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.841326 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:04:59.841465 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.841343 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:00.841327172 +0000 UTC m=+4.122677224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:04:59.841465 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:04:59.841397 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:05:00.841375652 +0000 UTC m=+4.122725711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:00.159940 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.159883 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 14:59:59 +0000 UTC" deadline="2028-03-10 18:51:31.568916093 +0000 UTC" Jul 27 15:05:00.159940 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.159934 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14211h46m31.408985568s" Jul 27 15:05:00.194461 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.194348 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" event={"ID":"139a73fe-3f5a-4313-a8be-8a30db9dcda5","Type":"ContainerStarted","Data":"21f0e2aede93513be0f99f43e6141501620cc01f6bfe2975807c8fbea125a3d0"} Jul 27 15:05:00.195859 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.195790 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fb6tm" event={"ID":"97244c43-6c8f-463f-8cd7-42828e29ce00","Type":"ContainerStarted","Data":"29a3889c4588281a881a52bf5bbd89a6624a97b2f7d9e5f94d667d4e96273e2a"} Jul 27 15:05:00.197163 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.197129 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"ad26ac7e785e33273faee4cc7a40a7ba80159081a9d1b4fb991b4d33a112d03e"} Jul 27 15:05:00.198476 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.198441 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9zff8" event={"ID":"3b0434b1-17f8-4bc1-8e65-aa02f5192404","Type":"ContainerStarted","Data":"2e00daea3c83f9b686b15535e97eabc822fb0a3ce822fdd2793ace3a303613f0"} Jul 27 15:05:00.199763 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.199706 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bmnp2" event={"ID":"fa7f660f-4dd4-457a-950c-07412f65b7da","Type":"ContainerStarted","Data":"863b03104be0c02536614031a69ba7dfee0aec48988a6ed2105f4c76eabec04d"} Jul 27 15:05:00.201381 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.201346 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l24dr" event={"ID":"65b2001b-a43e-4060-8dc1-e69aae55dd22","Type":"ContainerStarted","Data":"4b31e419cccb750d7965f2658ae4a4be55829a844bd7a6b3062a17a3fde48e69"} Jul 27 15:05:00.202821 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.202795 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerStarted","Data":"e5d71decedfa01b446f7637f25419d7ee9b86f5d5c0480411fbe3f1de5882f4b"} Jul 27 15:05:00.204193 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.204159 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9x9q" event={"ID":"46efa81f-a600-439b-b979-abb5028bf7b1","Type":"ContainerStarted","Data":"b2df0c80de0f40938281fed6df44ce3704dbccd59d846a3415934d7dd3028493"} Jul 27 15:05:00.205528 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.205505 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" event={"ID":"bab4b666-154e-4f34-aa46-ea55efe66af1","Type":"ContainerStarted","Data":"34a950dbe1538c022bbd9ce9a3ad9e23fca219525a4485595ebc711f50525465"} Jul 27 15:05:00.472777 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:00.472737 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4ffb043a26cecc367b10a3cee4e9b8.slice/crio-1c85edba7a40f5571e0b36b6bb26d49fe01aeeeb5cdd064dc4cd840f0f243021 WatchSource:0}: Error finding container 1c85edba7a40f5571e0b36b6bb26d49fe01aeeeb5cdd064dc4cd840f0f243021: Status 404 returned error can't find the container with id 1c85edba7a40f5571e0b36b6bb26d49fe01aeeeb5cdd064dc4cd840f0f243021 Jul 27 15:05:00.749212 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.749052 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:00.749391 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.749285 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:00.749391 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.749355 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:02.749334933 +0000 UTC m=+6.030684989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:00.850483 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.850440 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:00.850692 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:00.850541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:00.850761 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.850672 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:00.850815 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.850777 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:05:02.850757493 +0000 UTC m=+6.132107550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:00.851249 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.851227 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:05:00.851362 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.851254 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:05:00.851362 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.851267 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:00.851362 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:00.851316 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:02.851297816 +0000 UTC m=+6.132647885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:01.083242 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.082821 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 15:05:01.161152 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.161074 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 14:59:59 +0000 UTC" deadline="2028-02-06 17:05:24.568635201 +0000 UTC" Jul 27 15:05:01.161152 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.161116 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13418h0m23.407524703s" Jul 27 15:05:01.188656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.187764 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:01.188656 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:01.187910 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:01.188656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.188342 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:01.188656 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:01.188432 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:01.188656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.188503 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:01.188656 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:01.188586 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:01.214961 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.214922 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" event={"ID":"a58aa08a3e395c7a07bb090be093243d","Type":"ContainerStarted","Data":"63505ba574c66c331a6db57b9ac5153556c47dd1a05db19de14503a716b3dd67"} Jul 27 15:05:01.222656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:01.222625 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" event={"ID":"ec4ffb043a26cecc367b10a3cee4e9b8","Type":"ContainerStarted","Data":"1c85edba7a40f5571e0b36b6bb26d49fe01aeeeb5cdd064dc4cd840f0f243021"} Jul 27 15:05:02.769969 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:02.768998 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:02.769969 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.769207 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:02.769969 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.769289 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:06.769268487 +0000 UTC m=+10.050618542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:02.870060 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:02.870159 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.870314 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.870337 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.870351 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.870416 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:06.870396017 +0000 UTC m=+10.151746073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.870841 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:02.871063 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:02.870893 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:05:06.87087738 +0000 UTC m=+10.152227434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:03.187707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:03.186661 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:03.187707 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:03.186813 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:03.187707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:03.187248 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:03.187707 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:03.187348 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:03.187707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:03.187499 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:03.187707 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:03.187583 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:05.187846 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:05.185923 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:05.187846 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:05.186060 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:05.187846 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:05.186500 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:05.187846 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:05.186611 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:05.187846 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:05.186774 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:05.187846 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:05.186870 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:06.809372 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:06.809303 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:06.809852 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.809482 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:06.809852 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.809557 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:14.809534927 +0000 UTC m=+18.090884982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:06.910486 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:06.910447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:06.910690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:06.910536 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:06.910690 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.910657 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:06.910690 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.910677 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:05:06.910839 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.910703 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:05:06.910839 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.910714 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:05:14.910699404 +0000 UTC m=+18.192049457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:06.910839 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.910718 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:06.910839 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:06.910775 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:14.910757612 +0000 UTC m=+18.192107677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:07.186711 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:07.186179 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:07.186711 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:07.186210 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:07.186711 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:07.186321 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:07.186711 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:07.186395 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:07.186711 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:07.186411 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:07.186711 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:07.186497 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:09.186443 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:09.186014 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:09.186443 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:09.186014 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:09.186443 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:09.186152 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:09.186443 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:09.186257 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:09.186443 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:09.186307 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:09.186443 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:09.186406 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:11.186029 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:11.185991 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:11.186512 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:11.186008 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:11.186512 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:11.186125 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:11.186512 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:11.186178 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:11.186512 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:11.186223 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:11.186512 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:11.186357 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:13.185772 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:13.185744 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:13.186205 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:13.185751 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:13.186205 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:13.185852 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:13.186205 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:13.185873 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:13.186205 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:13.185972 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:13.186205 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:13.186072 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:14.873477 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:14.873438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:14.873933 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.873598 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:14.873933 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.873680 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:30.873662017 +0000 UTC m=+34.155012072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:14.974483 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:14.974437 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:14.974663 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:14.974501 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:14.975163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.974862 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:14.975163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.974986 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:05:30.97494874 +0000 UTC m=+34.256298804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:14.975163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.975041 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:05:14.975163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.975091 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:05:14.975163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.975107 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:14.975163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:14.975166 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:30.975149852 +0000 UTC m=+34.256499903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:15.185769 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:15.185684 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:15.185929 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:15.185699 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:15.185929 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:15.185787 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:15.185929 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:15.185699 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:15.185929 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:15.185863 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:15.185929 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:15.185924 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:17.186996 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:17.186964 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:17.186996 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:17.186995 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:17.187507 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:17.187094 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:17.187507 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:17.187164 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:17.187507 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:17.187186 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:17.187507 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:17.187274 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:19.188162 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.187374 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:19.188162 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:19.187497 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:19.188162 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.187867 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:19.188162 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:19.187952 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:19.188162 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.188010 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:19.188162 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:19.188114 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:19.266021 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.265994 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fb6tm" event={"ID":"97244c43-6c8f-463f-8cd7-42828e29ce00","Type":"ContainerStarted","Data":"b67b10e984e654c4f0c61c7e5fb8c3fe33565adafdfec2f925bae86bed335050"} Jul 27 15:05:19.268875 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.268846 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"cd5b95748bc38e9b24f70f9b6df5d277510d143710714e3513f14db0f823cc26"} Jul 27 15:05:19.271626 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.271468 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" event={"ID":"bab4b666-154e-4f34-aa46-ea55efe66af1","Type":"ContainerStarted","Data":"25fce5575677a584b25b444fd996499cfe8edfbb25a55a2e8a531c53522f10e9"} Jul 27 15:05:19.290075 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.289714 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-218.ec2.internal" podStartSLOduration=20.289694635 podStartE2EDuration="20.289694635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:01.230465821 +0000 UTC m=+4.511815901" watchObservedRunningTime="2026-07-27 15:05:19.289694635 +0000 UTC m=+22.571044709" Jul 27 15:05:19.290302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:19.290260 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lxg7n" podStartSLOduration=1.630596856 podStartE2EDuration="21.290244537s" totalImagesPullingTime="19.659647681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:19.289662914 +0000 UTC m=+22.571012989" watchObservedRunningTime="2026-07-27 15:05:19.290244537 +0000 UTC m=+22.571594615" Jul 27 15:05:20.277192 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.276785 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"be8751a7fd7af63c952c57f35f156236803373d57945bcd5d099fef026f61270"} Jul 27 15:05:20.278103 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.277288 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"4b5f9776b880b2e73a13aaf51ad89184ae2d2d16228983344f08e3125ac9b8e4"} Jul 27 15:05:20.278103 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.277338 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"d91f403562caf2410a454c870e57f1172d4ebef3c0689d1b916896d902784f46"} Jul 27 15:05:20.278103 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.277378 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"2e66d2416c8df5662bb86afb9006e95e5460be2fcdba24cccccfcf2e886c406d"} Jul 27 15:05:20.278103 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.277417 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"e3e2c3ed8f0d7b2f567d8f6a89a2648dae532bf4bc8a44f3ae62df6b461e72e4"} Jul 27 15:05:20.278320 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.278296 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bmnp2" event={"ID":"fa7f660f-4dd4-457a-950c-07412f65b7da","Type":"ContainerStarted","Data":"000187cd7fdd5e18564348654a97ed41f8191dc76de720b77227d0329a763f63"} Jul 27 15:05:20.279920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.279887 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l24dr" event={"ID":"65b2001b-a43e-4060-8dc1-e69aae55dd22","Type":"ContainerStarted","Data":"24449a6e92576279f176a11956cfa35a41ba818c6a77d6bc0ef831da26a786d1"} Jul 27 15:05:20.281611 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.281581 2553 generic.go:356] "Generic (PLEG): container finished" podID="1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8" containerID="2eed48c1c1cfd6a0ae0c288e98f050312f5ae880124645cc242e238c498147ab" exitCode=0 Jul 27 15:05:20.281745 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.281613 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerDied","Data":"2eed48c1c1cfd6a0ae0c288e98f050312f5ae880124645cc242e238c498147ab"} Jul 27 15:05:20.283353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.283333 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x9x9q" event={"ID":"46efa81f-a600-439b-b979-abb5028bf7b1","Type":"ContainerStarted","Data":"63f52e4a1bd74f9b5743d20f02b93b49721fcbcb4fa428baff3937c4805d3d54"} Jul 27 15:05:20.285103 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.285080 2553 generic.go:356] "Generic (PLEG): container finished" podID="ec4ffb043a26cecc367b10a3cee4e9b8" containerID="8d57654d8a41cbc478edec4f39ea3e5ffb5aaed9a98580a70d9e9985846640b4" exitCode=0 Jul 27 15:05:20.285198 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.285165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" event={"ID":"ec4ffb043a26cecc367b10a3cee4e9b8","Type":"ContainerDied","Data":"8d57654d8a41cbc478edec4f39ea3e5ffb5aaed9a98580a70d9e9985846640b4"} Jul 27 15:05:20.286819 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.286758 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" event={"ID":"139a73fe-3f5a-4313-a8be-8a30db9dcda5","Type":"ContainerStarted","Data":"05fe5846b981f5a1e76a8eb63807fe65cf882771e84235de048b1946d7b29490"} Jul 27 15:05:20.293366 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.293331 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bmnp2" podStartSLOduration=2.648509011 podStartE2EDuration="22.29331717s" totalImagesPullingTime="19.644808159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:20.293150076 +0000 UTC m=+23.574500162" watchObservedRunningTime="2026-07-27 15:05:20.29331717 +0000 UTC m=+23.574667245" Jul 27 15:05:20.308119 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.308043 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fb6tm" podStartSLOduration=2.63441615 podStartE2EDuration="22.308025216s" totalImagesPullingTime="19.673609066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:20.307919818 +0000 UTC m=+23.589269892" watchObservedRunningTime="2026-07-27 15:05:20.308025216 +0000 UTC m=+23.589375292" Jul 27 15:05:20.341780 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.341739 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l24dr" podStartSLOduration=2.669660227 podStartE2EDuration="22.341725035s" totalImagesPullingTime="19.672064808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:20.341572835 +0000 UTC m=+23.622922920" watchObservedRunningTime="2026-07-27 15:05:20.341725035 +0000 UTC m=+23.623075109" Jul 27 15:05:20.392592 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.392572 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:05:20.393152 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.393134 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:05:20.431372 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.431331 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x9x9q" podStartSLOduration=7.509370685 podStartE2EDuration="22.431316123s" totalImagesPullingTime="14.921945438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:20.373074839 +0000 UTC m=+23.654424904" watchObservedRunningTime="2026-07-27 15:05:20.431316123 +0000 UTC m=+23.712666197" Jul 27 15:05:20.699789 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:20.699762 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 15:05:21.185931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.185892 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:21.186306 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.185944 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:21.186306 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:21.186032 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:21.186306 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.186081 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:21.186306 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:21.186191 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:21.186306 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:21.186269 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:21.197436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.197302 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T15:05:20.699785702Z","UUID":"df16e297-d8d1-443b-bd03-b16dccc057a0","Handler":null,"Name":"","Endpoint":""} Jul 27 15:05:21.199020 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.198994 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 15:05:21.199143 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.199038 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 15:05:21.291751 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.291718 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" event={"ID":"ec4ffb043a26cecc367b10a3cee4e9b8","Type":"ContainerStarted","Data":"47f930dc836e61e4dd2b3be73dea56ee2fcbc937ea037aca5ea2de832cfa8948"} Jul 27 15:05:21.293660 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.293631 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" event={"ID":"139a73fe-3f5a-4313-a8be-8a30db9dcda5","Type":"ContainerStarted","Data":"36752dc9892d33bc6a19c610fab292e969b2e1b92ce1f1bfa39773defb9ff4fb"} Jul 27 15:05:21.295292 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.295236 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9zff8" event={"ID":"3b0434b1-17f8-4bc1-8e65-aa02f5192404","Type":"ContainerStarted","Data":"a691c7ef53e60c5eafd68bde862cd1d9b2897a2b8f832d42383152c3f1f775ca"} Jul 27 15:05:21.306895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.306855 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-218.ec2.internal" podStartSLOduration=22.306844137 podStartE2EDuration="22.306844137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:21.306722785 +0000 UTC m=+24.588072854" watchObservedRunningTime="2026-07-27 15:05:21.306844137 +0000 UTC m=+24.588194211" Jul 27 15:05:21.320303 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:21.320260 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9zff8" podStartSLOduration=3.671606412 podStartE2EDuration="23.320245654s" totalImagesPullingTime="19.648639242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:21.31975187 +0000 UTC m=+24.601101943" watchObservedRunningTime="2026-07-27 15:05:21.320245654 +0000 UTC m=+24.601595730" Jul 27 15:05:22.305428 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:22.305391 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"0015d92bbfd7c7219bee764ef727acb3405eea9e9a1de0088851b2b47c674b02"} Jul 27 15:05:22.305803 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:22.305439 2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 15:05:23.189186 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:23.189158 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:23.189186 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:23.189187 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:23.189420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:23.189195 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:23.189420 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:23.189286 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:23.189420 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:23.189345 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:23.189524 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:23.189419 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:23.310357 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:23.310310 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" event={"ID":"139a73fe-3f5a-4313-a8be-8a30db9dcda5","Type":"ContainerStarted","Data":"06bf0fdb2b32d88ad1e20a41e405cddc70bc4daad616680c251664f02dfcff3c"} Jul 27 15:05:23.333007 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:23.332951 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v6kkt" podStartSLOduration=3.131016138 podStartE2EDuration="25.332931696s" totalImagesPullingTime="22.201915558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:23.331530157 +0000 UTC m=+26.612880231" watchObservedRunningTime="2026-07-27 15:05:23.332931696 +0000 UTC m=+26.614281772" Jul 27 15:05:25.188605 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.188415 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:25.189218 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.188460 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:25.189218 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:25.188691 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:25.189218 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.188470 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:25.189218 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:25.188747 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:25.189218 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:25.188828 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:25.316780 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.316747 2553 generic.go:356] "Generic (PLEG): container finished" podID="1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8" containerID="ee365517235a06330dca2338e107b9e55c885d92bc20ad1e546393b6d91b0059" exitCode=0 Jul 27 15:05:25.316961 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.316831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerDied","Data":"ee365517235a06330dca2338e107b9e55c885d92bc20ad1e546393b6d91b0059"} Jul 27 15:05:25.320540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.320516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" event={"ID":"c984c1f2-393e-4cc4-8b16-73d149c7726a","Type":"ContainerStarted","Data":"f3db4093f62c20fd2fc7d66352eece1ae05fdfc70596b0885fe46f342b9405f8"} Jul 27 15:05:25.320836 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.320805 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:05:25.320912 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.320873 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:05:25.336158 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.336118 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:05:25.358375 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:25.358330 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" podStartSLOduration=7.657407851 podStartE2EDuration="27.358316065s" totalImagesPullingTime="19.700908214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:25.357900777 +0000 UTC m=+28.639250851" watchObservedRunningTime="2026-07-27 15:05:25.358316065 +0000 UTC m=+28.639666164" Jul 27 15:05:26.324774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.324575 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerStarted","Data":"e41be7f5ebe9930c6b7332f6c87f014cd5aa1331a1c5d8d45df637395c25c2f1"} Jul 27 15:05:26.325209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.324944 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:05:26.339699 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.339673 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjd66"] Jul 27 15:05:26.339840 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.339775 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:26.339899 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:26.339854 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:26.342158 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.342138 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:05:26.344012 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.343990 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cv4fx"] Jul 27 15:05:26.344137 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.344122 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:26.344249 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:26.344224 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:26.344800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.344769 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kcnfc"] Jul 27 15:05:26.344895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:26.344858 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:26.345012 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:26.344986 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:27.327975 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:27.327936 2553 generic.go:356] "Generic (PLEG): container finished" podID="1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8" containerID="e41be7f5ebe9930c6b7332f6c87f014cd5aa1331a1c5d8d45df637395c25c2f1" exitCode=0 Jul 27 15:05:27.328355 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:27.328010 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerDied","Data":"e41be7f5ebe9930c6b7332f6c87f014cd5aa1331a1c5d8d45df637395c25c2f1"} Jul 27 15:05:28.186267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:28.186222 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:28.186267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:28.186255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:28.186636 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:28.186258 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:28.186636 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:28.186369 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:28.186636 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:28.186475 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:28.186636 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:28.186562 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:28.332897 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:28.332867 2553 generic.go:356] "Generic (PLEG): container finished" podID="1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8" containerID="b0bece9283539b2ab699cb108dbad099df4f00fcf8628b7c46de05a668810c94" exitCode=0 Jul 27 15:05:28.333298 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:28.332952 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerDied","Data":"b0bece9283539b2ab699cb108dbad099df4f00fcf8628b7c46de05a668810c94"} Jul 27 15:05:29.285082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:29.285029 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:05:29.285268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:29.285166 2553 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 15:05:29.285629 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:29.285613 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x9x9q" Jul 27 15:05:30.186154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:30.186123 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:30.186154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:30.186147 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:30.186154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:30.186163 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:30.186681 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.186256 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjd66" podUID="21f2ae8c-8a22-46d6-ae00-414fd7d0cefe" Jul 27 15:05:30.186681 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.186396 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kcnfc" podUID="253dcffa-69df-42d7-98a1-3a9113179c30" Jul 27 15:05:30.186681 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.186498 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cv4fx" podUID="66371b5f-01b5-4329-bc7e-069d801574e9" Jul 27 15:05:30.894550 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:30.894514 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:30.894757 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.894635 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:30.894757 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.894705 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs podName:66371b5f-01b5-4329-bc7e-069d801574e9 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:02.894686294 +0000 UTC m=+66.176036351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs") pod "network-metrics-daemon-cv4fx" (UID: "66371b5f-01b5-4329-bc7e-069d801574e9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 15:05:30.994977 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:30.994932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:30.995193 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:30.995017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:30.995193 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.995125 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:30.995193 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.995161 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 15:05:30.995193 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.995187 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 15:05:30.995398 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.995202 2553 projected.go:196] Error preparing data for projected volume kube-api-access-b84h6 for pod openshift-network-diagnostics/network-check-target-kcnfc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:30.995398 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.995207 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret podName:21f2ae8c-8a22-46d6-ae00-414fd7d0cefe nodeName:}" failed. No retries permitted until 2026-07-27 15:06:02.995187192 +0000 UTC m=+66.276537243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret") pod "global-pull-secret-syncer-bjd66" (UID: "21f2ae8c-8a22-46d6-ae00-414fd7d0cefe") : object "kube-system"/"original-pull-secret" not registered Jul 27 15:05:30.995398 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:30.995253 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6 podName:253dcffa-69df-42d7-98a1-3a9113179c30 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:02.995237563 +0000 UTC m=+66.276587629 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b84h6" (UniqueName: "kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6") pod "network-check-target-kcnfc" (UID: "253dcffa-69df-42d7-98a1-3a9113179c30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 15:05:32.059643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.059362 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-218.ec2.internal" event="NodeReady" Jul 27 15:05:32.060005 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.059744 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 15:05:32.117622 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.117284 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cff8c555f-7r4j5"] Jul 27 15:05:32.120215 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.120192 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t"] Jul 27 15:05:32.120352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.120318 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.122300 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.122276 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.122811 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.122793 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mw646\"" Jul 27 15:05:32.122894 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.122860 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 15:05:32.122894 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.122876 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 15:05:32.122894 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.122886 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 15:05:32.124692 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.124668 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 15:05:32.124931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.124910 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.125456 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.125438 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.126517 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.126498 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z77zc\"" Jul 27 15:05:32.126517 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.126514 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 15:05:32.128504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.128484 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 15:05:32.129871 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.129821 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r"] Jul 27 15:05:32.133948 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.132794 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f8c885cd9-2mgwd"] Jul 27 15:05:32.133948 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.133676 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.136188 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.136166 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-52wmr"] Jul 27 15:05:32.136997 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.136975 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.137355 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.137119 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.137355 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.137279 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4vlrj\"" Jul 27 15:05:32.137355 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.137338 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 15:05:32.137553 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.137534 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.140253 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.140231 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2tn84"] Jul 27 15:05:32.140537 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.140517 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 15:05:32.142190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.141460 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.142190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.141662 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.142190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.142006 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 15:05:32.142771 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.142751 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.146039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.143065 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j7gvt\"" Jul 27 15:05:32.146039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.143520 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 15:05:32.146039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.143847 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 15:05:32.148004 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.147986 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 15:05:32.152182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.151872 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-spr6f\"" Jul 27 15:05:32.152634 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.152615 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 15:05:32.153337 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.153315 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.153511 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.153228 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql"] Jul 27 15:05:32.156067 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.156034 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 15:05:32.156295 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.156280 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f"] Jul 27 15:05:32.156438 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.156421 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.156520 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.156500 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 15:05:32.157161 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.157145 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vtp7l\"" Jul 27 15:05:32.157978 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.157948 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.158137 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.157991 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.159042 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.159022 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 15:05:32.159304 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.159285 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.159373 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.159349 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-d64q4\"" Jul 27 15:05:32.159768 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.159751 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.160405 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.160388 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 15:05:32.161527 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.161496 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-798gp"] Jul 27 15:05:32.161730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.161617 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" Jul 27 15:05:32.164072 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.163608 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7c56v"] Jul 27 15:05:32.164310 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.164289 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hxbfj\"" Jul 27 15:05:32.164777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.164574 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.165281 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.165263 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 15:05:32.165738 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.165655 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7"] Jul 27 15:05:32.166158 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.166138 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.166347 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.166330 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.166647 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.166633 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.168730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.168711 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.168891 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.168870 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.169083 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.169044 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jhqpk\"" Jul 27 15:05:32.169316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.168732 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 27 15:05:32.169740 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.169723 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.170117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.170101 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6"] Jul 27 15:05:32.170239 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.170220 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.170330 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.170240 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 15:05:32.170330 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.170288 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.170805 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.170118 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 27 15:05:32.171454 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.171433 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2c97b\"" Jul 27 15:05:32.172009 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.171985 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 27 15:05:32.172253 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.172214 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs"] Jul 27 15:05:32.172467 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.172449 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" Jul 27 15:05:32.173465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.172878 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 27 15:05:32.173465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.172976 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.173465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.173001 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4jpjn\"" Jul 27 15:05:32.173465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.173139 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.174343 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.174311 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.174625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.174606 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.174806 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.174789 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv"] Jul 27 15:05:32.174963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.174947 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.175823 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.175806 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m2bt4\"" Jul 27 15:05:32.176640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.176620 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 27 15:05:32.178239 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.178203 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 27 15:05:32.178800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.178337 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cdhk7\"" Jul 27 15:05:32.179778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.179097 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 27 15:05:32.179778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.179154 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2"] Jul 27 15:05:32.179778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.179685 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 27 15:05:32.181888 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.180081 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.182068 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.182035 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 27 15:05:32.182541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.182512 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 27 15:05:32.183334 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183311 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t"] Jul 27 15:05:32.183409 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183357 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cff8c555f-7r4j5"] Jul 27 15:05:32.183409 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183379 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r"] Jul 27 15:05:32.183409 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183407 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183428 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183448 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183462 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183480 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f8c885cd9-2mgwd"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183497 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183515 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7"] Jul 27 15:05:32.183557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183533 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-52wmr"] Jul 27 15:05:32.183826 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183579 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.183826 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183588 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-798gp"] Jul 27 15:05:32.183826 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.183614 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2"] Jul 27 15:05:32.184588 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.184570 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7c56v"] Jul 27 15:05:32.184679 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.184657 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2tn84"] Jul 27 15:05:32.185174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.185154 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:05:32.185481 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.185466 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:05:32.185744 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.185685 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:05:32.187925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.187706 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 27 15:05:32.187925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.187768 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 15:05:32.187925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.187902 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 27 15:05:32.188147 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.188014 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 27 15:05:32.188147 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.188071 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 27 15:05:32.189625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.189608 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 15:05:32.189755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.189738 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkhzc\"" Jul 27 15:05:32.189863 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.189650 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2bzfc\"" Jul 27 15:05:32.305426 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305397 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875sc\" (UniqueName: \"kubernetes.io/projected/6368b6c0-4289-433d-bdad-d502124b7802-kube-api-access-875sc\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.305568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305438 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nww6\" (UniqueName: \"kubernetes.io/projected/15e3872a-a866-4484-b425-e6f72a4967a9-kube-api-access-9nww6\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.305568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305455 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a86eae4-88cc-402b-8003-570b4b1a7aae-service-ca-bundle\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.305568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305480 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-default-certificate\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.305568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305524 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1f42dc0-6dc2-4201-8f41-b20652979554-cert\") pod \"ingress-canary-7c56v\" (UID: \"d1f42dc0-6dc2-4201-8f41-b20652979554\") " pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305570 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ed8da3-6018-4f3b-ba4b-31537e008b12-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305597 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305623 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305657 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzznl\" (UniqueName: \"kubernetes.io/projected/3434d48a-2c83-4375-89e1-de3b25813fa9-kube-api-access-rzznl\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a86eae4-88cc-402b-8003-570b4b1a7aae-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305713 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-installation-pull-secrets\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305730 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxm4\" (UniqueName: \"kubernetes.io/projected/c0d6fee3-17dd-465f-9239-cf9dcea137d7-kube-api-access-mbxm4\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305751 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e3872a-a866-4484-b425-e6f72a4967a9-tmp\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.305774 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305770 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5smd\" (UniqueName: \"kubernetes.io/projected/4a86eae4-88cc-402b-8003-570b4b1a7aae-kube-api-access-v5smd\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305796 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmx2\" (UniqueName: \"kubernetes.io/projected/1b2e370c-ec55-4f58-8ec7-8507beae1c4a-kube-api-access-bhmx2\") pod \"managed-serviceaccount-addon-agent-c5c97687b-rhsqs\" (UID: \"1b2e370c-ec55-4f58-8ec7-8507beae1c4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305819 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-image-registry-private-configuration\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305874 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7r4b\" (UniqueName: \"kubernetes.io/projected/1934808b-5939-4353-a238-33b80c7b77f4-kube-api-access-n7r4b\") pod \"volume-data-source-validator-549c6d54cf-8g85f\" (UID: \"1934808b-5939-4353-a238-33b80c7b77f4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305896 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-bound-sa-token\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305933 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305956 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-config\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.305980 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-serving-cert\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306029 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306081 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df6cb94b-d781-4eb7-87a1-b85241de9b7c-config-volume\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306127 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6cb94b-d781-4eb7-87a1-b85241de9b7c-tmp-dir\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vcg\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-kube-api-access-82vcg\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.306190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306176 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d6fee3-17dd-465f-9239-cf9dcea137d7-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306204 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d6fee3-17dd-465f-9239-cf9dcea137d7-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306242 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306307 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqs9b\" (UniqueName: \"kubernetes.io/projected/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-kube-api-access-gqs9b\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306341 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4a86eae4-88cc-402b-8003-570b4b1a7aae-snapshots\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306375 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b2e370c-ec55-4f58-8ec7-8507beae1c4a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c5c97687b-rhsqs\" (UID: \"1b2e370c-ec55-4f58-8ec7-8507beae1c4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306408 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-stats-auth\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306436 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306457 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306478 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-trusted-ca\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpph\" (UniqueName: \"kubernetes.io/projected/b45193fe-e78e-4aa1-b5ea-beb1650db5ee-kube-api-access-5lpph\") pod \"network-check-source-7d88cb55d8-mpgd6\" (UID: \"b45193fe-e78e-4aa1-b5ea-beb1650db5ee\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306573 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-hub\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306637 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcbw\" (UniqueName: \"kubernetes.io/projected/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-kube-api-access-mzcbw\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.306690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-certificates\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/15e3872a-a866-4484-b425-e6f72a4967a9-klusterlet-config\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306736 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7x5\" (UniqueName: \"kubernetes.io/projected/10ed8da3-6018-4f3b-ba4b-31537e008b12-kube-api-access-8f7x5\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306757 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-ca\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306785 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngks\" (UniqueName: \"kubernetes.io/projected/29ea7830-521c-449e-9c10-6ab80a99a36e-kube-api-access-cngks\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306820 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ea7830-521c-449e-9c10-6ab80a99a36e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306859 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306915 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ed8da3-6018-4f3b-ba4b-31537e008b12-config\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.306970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnp75\" (UniqueName: \"kubernetes.io/projected/d1f42dc0-6dc2-4201-8f41-b20652979554-kube-api-access-rnp75\") pod \"ingress-canary-7c56v\" (UID: \"d1f42dc0-6dc2-4201-8f41-b20652979554\") " pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.307001 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6368b6c0-4289-433d-bdad-d502124b7802-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.307025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dvn\" (UniqueName: \"kubernetes.io/projected/df6cb94b-d781-4eb7-87a1-b85241de9b7c-kube-api-access-l6dvn\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.307074 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a86eae4-88cc-402b-8003-570b4b1a7aae-serving-cert\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.307099 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-trusted-ca\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.307124 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a86eae4-88cc-402b-8003-570b4b1a7aae-tmp\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.307255 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.307153 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efcde43-a6d0-40de-b34d-d46787bf75d8-ca-trust-extracted\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.408135 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408094 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-certificates\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.408340 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408146 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/15e3872a-a866-4484-b425-e6f72a4967a9-klusterlet-config\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.408340 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408178 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7x5\" (UniqueName: \"kubernetes.io/projected/10ed8da3-6018-4f3b-ba4b-31537e008b12-kube-api-access-8f7x5\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.408340 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-ca\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.408340 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408224 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngks\" (UniqueName: \"kubernetes.io/projected/29ea7830-521c-449e-9c10-6ab80a99a36e-kube-api-access-cngks\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.408340 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408252 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ea7830-521c-449e-9c10-6ab80a99a36e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.408340 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408324 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ed8da3-6018-4f3b-ba4b-31537e008b12-config\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408395 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnp75\" (UniqueName: \"kubernetes.io/projected/d1f42dc0-6dc2-4201-8f41-b20652979554-kube-api-access-rnp75\") pod \"ingress-canary-7c56v\" (UID: \"d1f42dc0-6dc2-4201-8f41-b20652979554\") " pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408423 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6368b6c0-4289-433d-bdad-d502124b7802-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408449 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dvn\" (UniqueName: \"kubernetes.io/projected/df6cb94b-d781-4eb7-87a1-b85241de9b7c-kube-api-access-l6dvn\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408471 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a86eae4-88cc-402b-8003-570b4b1a7aae-serving-cert\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408493 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-trusted-ca\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.408499 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a86eae4-88cc-402b-8003-570b4b1a7aae-tmp\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efcde43-a6d0-40de-b34d-d46787bf75d8-ca-trust-extracted\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.408580 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls podName:2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b nodeName:}" failed. No retries permitted until 2026-07-27 15:05:32.908557814 +0000 UTC m=+36.189907866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls") pod "cluster-samples-operator-6748489656-2pv4r" (UID: "2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b") : secret "samples-operator-tls" not found Jul 27 15:05:32.408643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408619 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875sc\" (UniqueName: \"kubernetes.io/projected/6368b6c0-4289-433d-bdad-d502124b7802-kube-api-access-875sc\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408662 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nww6\" (UniqueName: \"kubernetes.io/projected/15e3872a-a866-4484-b425-e6f72a4967a9-kube-api-access-9nww6\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a86eae4-88cc-402b-8003-570b4b1a7aae-service-ca-bundle\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408710 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-default-certificate\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408731 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1f42dc0-6dc2-4201-8f41-b20652979554-cert\") pod \"ingress-canary-7c56v\" (UID: \"d1f42dc0-6dc2-4201-8f41-b20652979554\") " pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408753 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ed8da3-6018-4f3b-ba4b-31537e008b12-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408772 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408795 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408793 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-certificates\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408897 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efcde43-a6d0-40de-b34d-d46787bf75d8-ca-trust-extracted\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.409174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.408944 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ea7830-521c-449e-9c10-6ab80a99a36e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409305 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzznl\" (UniqueName: \"kubernetes.io/projected/3434d48a-2c83-4375-89e1-de3b25813fa9-kube-api-access-rzznl\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a86eae4-88cc-402b-8003-570b4b1a7aae-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409368 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a86eae4-88cc-402b-8003-570b4b1a7aae-service-ca-bundle\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409371 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-installation-pull-secrets\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxm4\" (UniqueName: \"kubernetes.io/projected/c0d6fee3-17dd-465f-9239-cf9dcea137d7-kube-api-access-mbxm4\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409471 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e3872a-a866-4484-b425-e6f72a4967a9-tmp\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409497 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5smd\" (UniqueName: \"kubernetes.io/projected/4a86eae4-88cc-402b-8003-570b4b1a7aae-kube-api-access-v5smd\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmx2\" (UniqueName: \"kubernetes.io/projected/1b2e370c-ec55-4f58-8ec7-8507beae1c4a-kube-api-access-bhmx2\") pod \"managed-serviceaccount-addon-agent-c5c97687b-rhsqs\" (UID: \"1b2e370c-ec55-4f58-8ec7-8507beae1c4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409551 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-image-registry-private-configuration\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7r4b\" (UniqueName: \"kubernetes.io/projected/1934808b-5939-4353-a238-33b80c7b77f4-kube-api-access-n7r4b\") pod \"volume-data-source-validator-549c6d54cf-8g85f\" (UID: \"1934808b-5939-4353-a238-33b80c7b77f4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409604 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-bound-sa-token\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.409628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409621 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10ed8da3-6018-4f3b-ba4b-31537e008b12-config\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409660 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6368b6c0-4289-433d-bdad-d502124b7802-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409662 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409717 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-config\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409740 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-serving-cert\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409796 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409821 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df6cb94b-d781-4eb7-87a1-b85241de9b7c-config-volume\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.410113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.409846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6cb94b-d781-4eb7-87a1-b85241de9b7c-tmp-dir\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.410421 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410158 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6cb94b-d781-4eb7-87a1-b85241de9b7c-tmp-dir\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.410421 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.410402 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 15:05:32.410421 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410411 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/15e3872a-a866-4484-b425-e6f72a4967a9-tmp\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.410563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vcg\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-kube-api-access-82vcg\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.410563 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.410469 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls podName:df6cb94b-d781-4eb7-87a1-b85241de9b7c nodeName:}" failed. No retries permitted until 2026-07-27 15:05:32.910450574 +0000 UTC m=+36.191800630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls") pod "dns-default-52wmr" (UID: "df6cb94b-d781-4eb7-87a1-b85241de9b7c") : secret "dns-default-metrics-tls" not found Jul 27 15:05:32.410563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410511 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d6fee3-17dd-465f-9239-cf9dcea137d7-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.410563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410543 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d6fee3-17dd-465f-9239-cf9dcea137d7-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.410756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410590 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.410756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410624 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqs9b\" (UniqueName: \"kubernetes.io/projected/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-kube-api-access-gqs9b\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.410756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410650 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4a86eae4-88cc-402b-8003-570b4b1a7aae-snapshots\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.410756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410676 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b2e370c-ec55-4f58-8ec7-8507beae1c4a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c5c97687b-rhsqs\" (UID: \"1b2e370c-ec55-4f58-8ec7-8507beae1c4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.410756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410703 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-stats-auth\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.410756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410732 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410759 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-trusted-ca\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410834 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpph\" (UniqueName: \"kubernetes.io/projected/b45193fe-e78e-4aa1-b5ea-beb1650db5ee-kube-api-access-5lpph\") pod \"network-check-source-7d88cb55d8-mpgd6\" (UID: \"b45193fe-e78e-4aa1-b5ea-beb1650db5ee\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410860 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-hub\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcbw\" (UniqueName: \"kubernetes.io/projected/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-kube-api-access-mzcbw\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.410890 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a86eae4-88cc-402b-8003-570b4b1a7aae-tmp\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.410404 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.410981 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cff8c555f-7r4j5: secret "image-registry-tls" not found Jul 27 15:05:32.411034 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.411019 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls podName:3efcde43-a6d0-40de-b34d-d46787bf75d8 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:32.911005296 +0000 UTC m=+36.192355351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls") pod "image-registry-5cff8c555f-7r4j5" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8") : secret "image-registry-tls" not found Jul 27 15:05:32.411465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.411263 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-trusted-ca\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.411896 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.411873 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a86eae4-88cc-402b-8003-570b4b1a7aae-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.412164 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.412148 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:32.412590 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.412296 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls podName:6368b6c0-4289-433d-bdad-d502124b7802 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:32.912280113 +0000 UTC m=+36.193630183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5t57t" (UID: "6368b6c0-4289-433d-bdad-d502124b7802") : secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:32.412967 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.412944 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 15:05:32.413069 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.413010 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:32.912993919 +0000 UTC m=+36.194343973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : secret "router-metrics-certs-default" not found Jul 27 15:05:32.413670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.413646 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-config\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.413768 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.413747 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10ed8da3-6018-4f3b-ba4b-31537e008b12-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.414386 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.414363 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:32.914347389 +0000 UTC m=+36.195697443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : configmap references non-existent config key: service-ca.crt Jul 27 15:05:32.414708 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.414683 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c0d6fee3-17dd-465f-9239-cf9dcea137d7-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.414895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.414874 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df6cb94b-d781-4eb7-87a1-b85241de9b7c-config-volume\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.415760 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.415737 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-trusted-ca\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.416559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.416487 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.417190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417163 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-image-registry-private-configuration\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.417324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417307 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4a86eae4-88cc-402b-8003-570b4b1a7aae-serving-cert\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.417474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-default-certificate\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.417543 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417466 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4a86eae4-88cc-402b-8003-570b4b1a7aae-snapshots\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.417599 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417548 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-ca\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.417877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417853 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-installation-pull-secrets\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.417983 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.417959 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/15e3872a-a866-4484-b425-e6f72a4967a9-klusterlet-config\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.418844 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.418825 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-serving-cert\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.420637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.420600 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.420964 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.420943 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c0d6fee3-17dd-465f-9239-cf9dcea137d7-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.422325 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.421964 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxm4\" (UniqueName: \"kubernetes.io/projected/c0d6fee3-17dd-465f-9239-cf9dcea137d7-kube-api-access-mbxm4\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7\" (UID: \"c0d6fee3-17dd-465f-9239-cf9dcea137d7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.422325 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.422037 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-stats-auth\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.423866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.423847 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b2e370c-ec55-4f58-8ec7-8507beae1c4a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c5c97687b-rhsqs\" (UID: \"1b2e370c-ec55-4f58-8ec7-8507beae1c4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.423990 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.423933 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/29ea7830-521c-449e-9c10-6ab80a99a36e-hub\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.424389 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.424362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1f42dc0-6dc2-4201-8f41-b20652979554-cert\") pod \"ingress-canary-7c56v\" (UID: \"d1f42dc0-6dc2-4201-8f41-b20652979554\") " pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.424694 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.424676 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dvn\" (UniqueName: \"kubernetes.io/projected/df6cb94b-d781-4eb7-87a1-b85241de9b7c-kube-api-access-l6dvn\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.427859 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.427811 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5smd\" (UniqueName: \"kubernetes.io/projected/4a86eae4-88cc-402b-8003-570b4b1a7aae-kube-api-access-v5smd\") pod \"insights-operator-5885fdb759-2tn84\" (UID: \"4a86eae4-88cc-402b-8003-570b4b1a7aae\") " pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.428136 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428098 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnp75\" (UniqueName: \"kubernetes.io/projected/d1f42dc0-6dc2-4201-8f41-b20652979554-kube-api-access-rnp75\") pod \"ingress-canary-7c56v\" (UID: \"d1f42dc0-6dc2-4201-8f41-b20652979554\") " pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.428136 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428109 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vcg\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-kube-api-access-82vcg\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.428743 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428487 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzznl\" (UniqueName: \"kubernetes.io/projected/3434d48a-2c83-4375-89e1-de3b25813fa9-kube-api-access-rzznl\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.428743 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428525 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmx2\" (UniqueName: \"kubernetes.io/projected/1b2e370c-ec55-4f58-8ec7-8507beae1c4a-kube-api-access-bhmx2\") pod \"managed-serviceaccount-addon-agent-c5c97687b-rhsqs\" (UID: \"1b2e370c-ec55-4f58-8ec7-8507beae1c4a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.428743 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428706 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7x5\" (UniqueName: \"kubernetes.io/projected/10ed8da3-6018-4f3b-ba4b-31537e008b12-kube-api-access-8f7x5\") pod \"service-ca-operator-5d6c75f84f-2ztql\" (UID: \"10ed8da3-6018-4f3b-ba4b-31537e008b12\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.428924 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428845 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nww6\" (UniqueName: \"kubernetes.io/projected/15e3872a-a866-4484-b425-e6f72a4967a9-kube-api-access-9nww6\") pod \"klusterlet-addon-workmgr-58d9d8dd89-6ncqv\" (UID: \"15e3872a-a866-4484-b425-e6f72a4967a9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.428983 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.428964 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngks\" (UniqueName: \"kubernetes.io/projected/29ea7830-521c-449e-9c10-6ab80a99a36e-kube-api-access-cngks\") pod \"cluster-proxy-proxy-agent-86645dd754-6c6q2\" (UID: \"29ea7830-521c-449e-9c10-6ab80a99a36e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.429113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.429093 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpph\" (UniqueName: \"kubernetes.io/projected/b45193fe-e78e-4aa1-b5ea-beb1650db5ee-kube-api-access-5lpph\") pod \"network-check-source-7d88cb55d8-mpgd6\" (UID: \"b45193fe-e78e-4aa1-b5ea-beb1650db5ee\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" Jul 27 15:05:32.429553 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.429534 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcbw\" (UniqueName: \"kubernetes.io/projected/c38c59b8-7ecf-4006-8a0e-e75eb798ef14-kube-api-access-mzcbw\") pod \"console-operator-584dcb6b54-798gp\" (UID: \"c38c59b8-7ecf-4006-8a0e-e75eb798ef14\") " pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.429642 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.429545 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-875sc\" (UniqueName: \"kubernetes.io/projected/6368b6c0-4289-433d-bdad-d502124b7802-kube-api-access-875sc\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.429689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.429676 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7r4b\" (UniqueName: \"kubernetes.io/projected/1934808b-5939-4353-a238-33b80c7b77f4-kube-api-access-n7r4b\") pod \"volume-data-source-validator-549c6d54cf-8g85f\" (UID: \"1934808b-5939-4353-a238-33b80c7b77f4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" Jul 27 15:05:32.430212 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.430192 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-bound-sa-token\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.447856 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.447831 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqs9b\" (UniqueName: \"kubernetes.io/projected/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-kube-api-access-gqs9b\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.494950 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.494915 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2tn84" Jul 27 15:05:32.502716 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.502678 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" Jul 27 15:05:32.508805 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.508783 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" Jul 27 15:05:32.515673 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.515652 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:32.523288 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.523269 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7c56v" Jul 27 15:05:32.530026 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.530005 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" Jul 27 15:05:32.536935 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.536908 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" Jul 27 15:05:32.552033 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.552010 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" Jul 27 15:05:32.559865 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.559836 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:32.573760 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.573740 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:05:32.915649 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.915612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:32.915819 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.915659 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:32.915819 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.915717 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.915819 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.915747 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:32.915819 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.915770 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 15:05:32.915819 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.915801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.915833 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.915851 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cff8c555f-7r4j5: secret "image-registry-tls" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.915832 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.915892 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.915839 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls podName:df6cb94b-d781-4eb7-87a1-b85241de9b7c nodeName:}" failed. No retries permitted until 2026-07-27 15:05:33.91581836 +0000 UTC m=+37.197168411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls") pod "dns-default-52wmr" (UID: "df6cb94b-d781-4eb7-87a1-b85241de9b7c") : secret "dns-default-metrics-tls" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:32.915979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.916009 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls podName:3efcde43-a6d0-40de-b34d-d46787bf75d8 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:33.915988562 +0000 UTC m=+37.197338630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls") pod "image-registry-5cff8c555f-7r4j5" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8") : secret "image-registry-tls" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.916027 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.916042 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:33.916029898 +0000 UTC m=+37.197379987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : configmap references non-existent config key: service-ca.crt Jul 27 15:05:32.916082 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.916073 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:33.916064262 +0000 UTC m=+37.197414314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : secret "router-metrics-certs-default" not found Jul 27 15:05:32.916459 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.916093 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls podName:6368b6c0-4289-433d-bdad-d502124b7802 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:33.916084554 +0000 UTC m=+37.197434607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5t57t" (UID: "6368b6c0-4289-433d-bdad-d502124b7802") : secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:32.916459 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:32.916118 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls podName:2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b nodeName:}" failed. No retries permitted until 2026-07-27 15:05:33.916105016 +0000 UTC m=+37.197455081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls") pod "cluster-samples-operator-6748489656-2pv4r" (UID: "2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b") : secret "samples-operator-tls" not found Jul 27 15:05:33.519620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.517203 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6"] Jul 27 15:05:33.523296 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.521803 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql"] Jul 27 15:05:33.524169 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:33.523802 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45193fe_e78e_4aa1_b5ea_beb1650db5ee.slice/crio-0219574e32f3e34b595227084fd585afcbae4fd583fdd20ac839800a2cb6281d WatchSource:0}: Error finding container 0219574e32f3e34b595227084fd585afcbae4fd583fdd20ac839800a2cb6281d: Status 404 returned error can't find the container with id 0219574e32f3e34b595227084fd585afcbae4fd583fdd20ac839800a2cb6281d Jul 27 15:05:33.534852 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.532256 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-798gp"] Jul 27 15:05:33.534852 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.532316 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2tn84"] Jul 27 15:05:33.554213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.554043 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f"] Jul 27 15:05:33.555376 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.555353 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7"] Jul 27 15:05:33.559194 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.559171 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv"] Jul 27 15:05:33.563242 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.562801 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs"] Jul 27 15:05:33.563938 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.563918 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2"] Jul 27 15:05:33.567900 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.567875 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7c56v"] Jul 27 15:05:33.572585 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:33.572386 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1934808b_5939_4353_a238_33b80c7b77f4.slice/crio-60681f2616b9b988601c4a31f6734f902f8fe2d3075fdf6baf2a39a8a1dfec3b WatchSource:0}: Error finding container 60681f2616b9b988601c4a31f6734f902f8fe2d3075fdf6baf2a39a8a1dfec3b: Status 404 returned error can't find the container with id 60681f2616b9b988601c4a31f6734f902f8fe2d3075fdf6baf2a39a8a1dfec3b Jul 27 15:05:33.572965 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:33.572933 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d6fee3_17dd_465f_9239_cf9dcea137d7.slice/crio-21471baf899d13e3fb914ffed0c0d913f4a25f72e1d92dbdb0a8638213314564 WatchSource:0}: Error finding container 21471baf899d13e3fb914ffed0c0d913f4a25f72e1d92dbdb0a8638213314564: Status 404 returned error can't find the container with id 21471baf899d13e3fb914ffed0c0d913f4a25f72e1d92dbdb0a8638213314564 Jul 27 15:05:33.574224 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:33.573991 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29ea7830_521c_449e_9c10_6ab80a99a36e.slice/crio-9914d0655919502c8d2ad57ad899aa89b41f0c5776714397586fbfe5c582d319 WatchSource:0}: Error finding container 9914d0655919502c8d2ad57ad899aa89b41f0c5776714397586fbfe5c582d319: Status 404 returned error can't find the container with id 9914d0655919502c8d2ad57ad899aa89b41f0c5776714397586fbfe5c582d319 Jul 27 15:05:33.931041 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.931003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.931042 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.931108 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.931129 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931161 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.931182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931232 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls podName:df6cb94b-d781-4eb7-87a1-b85241de9b7c nodeName:}" failed. No retries permitted until 2026-07-27 15:05:35.931211797 +0000 UTC m=+39.212561851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls") pod "dns-default-52wmr" (UID: "df6cb94b-d781-4eb7-87a1-b85241de9b7c") : secret "dns-default-metrics-tls" not found Jul 27 15:05:33.931239 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931238 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931252 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cff8c555f-7r4j5: secret "image-registry-tls" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:33.931282 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931293 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls podName:3efcde43-a6d0-40de-b34d-d46787bf75d8 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:35.931281911 +0000 UTC m=+39.212631982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls") pod "image-registry-5cff8c555f-7r4j5" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8") : secret "image-registry-tls" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931311 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:35.931302784 +0000 UTC m=+39.212652850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : configmap references non-existent config key: service-ca.crt Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931310 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931344 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931364 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:35.931347768 +0000 UTC m=+39.212697824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : secret "router-metrics-certs-default" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931344 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931392 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls podName:6368b6c0-4289-433d-bdad-d502124b7802 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:35.931378025 +0000 UTC m=+39.212728099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5t57t" (UID: "6368b6c0-4289-433d-bdad-d502124b7802") : secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:33.931573 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:33.931445 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls podName:2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b nodeName:}" failed. No retries permitted until 2026-07-27 15:05:35.931431856 +0000 UTC m=+39.212781911 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls") pod "cluster-samples-operator-6748489656-2pv4r" (UID: "2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b") : secret "samples-operator-tls" not found Jul 27 15:05:34.379836 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.379747 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7c56v" event={"ID":"d1f42dc0-6dc2-4201-8f41-b20652979554","Type":"ContainerStarted","Data":"c700a6210ba009d476478a112d45e125ab5d66ce614715bad9a33ba0679e5d20"} Jul 27 15:05:34.384347 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.383943 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" event={"ID":"c38c59b8-7ecf-4006-8a0e-e75eb798ef14","Type":"ContainerStarted","Data":"d0c06f8e53ad7652325226e685726357f068afcb7b0b2c9b8569fb37474cb3ed"} Jul 27 15:05:34.390534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.390437 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" event={"ID":"1934808b-5939-4353-a238-33b80c7b77f4","Type":"ContainerStarted","Data":"60681f2616b9b988601c4a31f6734f902f8fe2d3075fdf6baf2a39a8a1dfec3b"} Jul 27 15:05:34.393523 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.393447 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" event={"ID":"1b2e370c-ec55-4f58-8ec7-8507beae1c4a","Type":"ContainerStarted","Data":"720903bbdcb8d4784f1825cda681ba6d3e6114a2f22b231bf389dc2d7226efd9"} Jul 27 15:05:34.395316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.395257 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2tn84" event={"ID":"4a86eae4-88cc-402b-8003-570b4b1a7aae","Type":"ContainerStarted","Data":"482ff65eb9ce4d4064d627daa0c62bf93a248f0f314be66633979ec3784e4c33"} Jul 27 15:05:34.400715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.400658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" event={"ID":"15e3872a-a866-4484-b425-e6f72a4967a9","Type":"ContainerStarted","Data":"e565afe5407f74d528d1f7dad607930a45f5203b24eaab33e7d258f7a535237a"} Jul 27 15:05:34.405163 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.405041 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" event={"ID":"b45193fe-e78e-4aa1-b5ea-beb1650db5ee","Type":"ContainerStarted","Data":"0219574e32f3e34b595227084fd585afcbae4fd583fdd20ac839800a2cb6281d"} Jul 27 15:05:34.407771 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.407707 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" event={"ID":"10ed8da3-6018-4f3b-ba4b-31537e008b12","Type":"ContainerStarted","Data":"a40ec81f6801f009c0a7f0b8c2ba70b1f2cd13f2eb5b5c98790e4d2f28244e42"} Jul 27 15:05:34.412856 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.412830 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" event={"ID":"c0d6fee3-17dd-465f-9239-cf9dcea137d7","Type":"ContainerStarted","Data":"21471baf899d13e3fb914ffed0c0d913f4a25f72e1d92dbdb0a8638213314564"} Jul 27 15:05:34.425589 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.414389 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" event={"ID":"29ea7830-521c-449e-9c10-6ab80a99a36e","Type":"ContainerStarted","Data":"9914d0655919502c8d2ad57ad899aa89b41f0c5776714397586fbfe5c582d319"} Jul 27 15:05:34.425589 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.420107 2553 generic.go:356] "Generic (PLEG): container finished" podID="1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8" containerID="3eb666947559b91a2fa74c55564cc11ba7e1489ba0b8ac69cff85463f5079276" exitCode=0 Jul 27 15:05:34.425589 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:34.420139 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerDied","Data":"3eb666947559b91a2fa74c55564cc11ba7e1489ba0b8ac69cff85463f5079276"} Jul 27 15:05:35.448386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.447801 2553 generic.go:356] "Generic (PLEG): container finished" podID="1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8" containerID="6fad733fe85e4cf9ddff0a71dd7dc804329b9f806cea4f8fc6dc1bdd0bc61661" exitCode=0 Jul 27 15:05:35.448386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.447857 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerDied","Data":"6fad733fe85e4cf9ddff0a71dd7dc804329b9f806cea4f8fc6dc1bdd0bc61661"} Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.952493 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.952608 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.952671 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.952704 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.952760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:35.952825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.952987 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 15:05:35.953302 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953071 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls podName:2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.953033771 +0000 UTC m=+43.234383836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls") pod "cluster-samples-operator-6748489656-2pv4r" (UID: "2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b") : secret "samples-operator-tls" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953505 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953560 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls podName:df6cb94b-d781-4eb7-87a1-b85241de9b7c nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.953543607 +0000 UTC m=+43.234893666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls") pod "dns-default-52wmr" (UID: "df6cb94b-d781-4eb7-87a1-b85241de9b7c") : secret "dns-default-metrics-tls" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953624 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953635 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cff8c555f-7r4j5: secret "image-registry-tls" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953667 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls podName:3efcde43-a6d0-40de-b34d-d46787bf75d8 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.95365526 +0000 UTC m=+43.235005319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls") pod "image-registry-5cff8c555f-7r4j5" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8") : secret "image-registry-tls" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953718 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953748 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.9537382 +0000 UTC m=+43.235088257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : secret "router-metrics-certs-default" not found Jul 27 15:05:35.953851 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953814 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.95380417 +0000 UTC m=+43.235154225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : configmap references non-existent config key: service-ca.crt Jul 27 15:05:35.954262 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953872 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:35.954262 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:35.953902 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls podName:6368b6c0-4289-433d-bdad-d502124b7802 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.953892966 +0000 UTC m=+43.235243022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5t57t" (UID: "6368b6c0-4289-433d-bdad-d502124b7802") : secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:36.078820 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.078784 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-hcl95"] Jul 27 15:05:36.100540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.100505 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-hcl95"] Jul 27 15:05:36.100693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.100627 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.108593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.108096 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 15:05:36.108593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.108155 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hq67j\"" Jul 27 15:05:36.108593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.108406 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 15:05:36.157394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.156926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa23ab11-efac-4d9a-a0fb-77a43c481313-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.157394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.157205 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.253854 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.253688 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:36.259125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.257765 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa23ab11-efac-4d9a-a0fb-77a43c481313-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.259125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.257805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.259125 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:36.258106 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 15:05:36.259125 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:36.258170 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert podName:aa23ab11-efac-4d9a-a0fb-77a43c481313 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:36.758150826 +0000 UTC m=+40.039500881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hcl95" (UID: "aa23ab11-efac-4d9a-a0fb-77a43c481313") : secret "networking-console-plugin-cert" not found Jul 27 15:05:36.259125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.259010 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa23ab11-efac-4d9a-a0fb-77a43c481313-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.371589 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.371523 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:36.499476 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.496934 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" event={"ID":"1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8","Type":"ContainerStarted","Data":"e26aafdbc7e778312f0760dccc57c4950e450a54393e55ac00131c6bd1235c23"} Jul 27 15:05:36.512776 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.512704 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:36.523880 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.523815 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fnsgs" podStartSLOduration=7.801995422 podStartE2EDuration="38.523796744s" totalImagesPullingTime="30.721801322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:36.52132419 +0000 UTC m=+39.802674265" watchObservedRunningTime="2026-07-27 15:05:36.523796744 +0000 UTC m=+39.805146818" Jul 27 15:05:36.765077 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:36.764980 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:36.765299 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:36.765263 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 15:05:36.765399 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:36.765329 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert podName:aa23ab11-efac-4d9a-a0fb-77a43c481313 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:37.765310749 +0000 UTC m=+41.046660804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hcl95" (UID: "aa23ab11-efac-4d9a-a0fb-77a43c481313") : secret "networking-console-plugin-cert" not found Jul 27 15:05:37.156304 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.156170 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:37.234390 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.234217 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:37.506107 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.505969 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:37.513448 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.513330 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" event={"ID":"1934808b-5939-4353-a238-33b80c7b77f4","Type":"ContainerStarted","Data":"cb1a465e028db76c7f4c1b0bdb4044c8579c68aaed664c294ee4c1bbcb671917"} Jul 27 15:05:37.529942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.529875 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g85f" podStartSLOduration=26.769454833 podStartE2EDuration="29.529852465s" totalImagesPullingTime="2.760397632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:37.528203698 +0000 UTC m=+40.809553798" watchObservedRunningTime="2026-07-27 15:05:37.529852465 +0000 UTC m=+40.811202552" Jul 27 15:05:37.603082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.603021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:37.680314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.680246 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:37.780825 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:37.780041 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:37.780825 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:37.780264 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 15:05:37.780825 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:37.780331 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert podName:aa23ab11-efac-4d9a-a0fb-77a43c481313 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:39.780311555 +0000 UTC m=+43.061661609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hcl95" (UID: "aa23ab11-efac-4d9a-a0fb-77a43c481313") : secret "networking-console-plugin-cert" not found Jul 27 15:05:39.801533 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:39.801497 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:39.801905 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:39.801633 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 15:05:39.801905 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:39.801697 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert podName:aa23ab11-efac-4d9a-a0fb-77a43c481313 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:43.801680924 +0000 UTC m=+47.083030975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hcl95" (UID: "aa23ab11-efac-4d9a-a0fb-77a43c481313") : secret "networking-console-plugin-cert" not found Jul 27 15:05:40.002703 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:40.002666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:40.002703 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:40.002702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:40.002738 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:40.002786 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:40.002820 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:40.002836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002852 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.00283411 +0000 UTC m=+51.284184162 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : configmap references non-existent config key: service-ca.crt Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002852 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002905 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002916 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 15:05:40.002942 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002944 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002963 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls podName:6368b6c0-4289-433d-bdad-d502124b7802 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.002944899 +0000 UTC m=+51.284294963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5t57t" (UID: "6368b6c0-4289-433d-bdad-d502124b7802") : secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002977 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls podName:df6cb94b-d781-4eb7-87a1-b85241de9b7c nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.002971028 +0000 UTC m=+51.284321080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls") pod "dns-default-52wmr" (UID: "df6cb94b-d781-4eb7-87a1-b85241de9b7c") : secret "dns-default-metrics-tls" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002985 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.002981072 +0000 UTC m=+51.284331124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : secret "router-metrics-certs-default" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.003007 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls podName:2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.00299174 +0000 UTC m=+51.284341828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls") pod "cluster-samples-operator-6748489656-2pv4r" (UID: "2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b") : secret "samples-operator-tls" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.002920 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.003020 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cff8c555f-7r4j5: secret "image-registry-tls" not found Jul 27 15:05:40.003264 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:40.003043 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls podName:3efcde43-a6d0-40de-b34d-d46787bf75d8 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.003036308 +0000 UTC m=+51.284386363 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls") pod "image-registry-5cff8c555f-7r4j5" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8") : secret "image-registry-tls" not found Jul 27 15:05:43.839540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:43.839498 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:43.840146 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:43.839690 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 15:05:43.840146 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:43.839770 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert podName:aa23ab11-efac-4d9a-a0fb-77a43c481313 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:51.839751281 +0000 UTC m=+55.121101337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hcl95" (UID: "aa23ab11-efac-4d9a-a0fb-77a43c481313") : secret "networking-console-plugin-cert" not found Jul 27 15:05:45.051803 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:45.051775 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:45.126391 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:45.126347 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:45.245652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:45.245563 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:45.332889 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:45.332804 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:45.413015 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:45.412982 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:45.478950 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:45.478907 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:47.093168 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.093098 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:47.194768 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.194736 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:47.543621 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.543586 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2tn84" event={"ID":"4a86eae4-88cc-402b-8003-570b4b1a7aae","Type":"ContainerStarted","Data":"d34082bda52bda77984bcde5ee3fb6fdfd8fbc369a573b03d8ce8ee9771f2800"} Jul 27 15:05:47.546145 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.546116 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" event={"ID":"15e3872a-a866-4484-b425-e6f72a4967a9","Type":"ContainerStarted","Data":"812c037b15adf60fc0eb4b751782513ae0b5ea1132f5c6f7702ff02da7b3a0e9"} Jul 27 15:05:47.548511 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.547292 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:47.548511 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.548439 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" podUID="15e3872a-a866-4484-b425-e6f72a4967a9" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.15:8000/readyz\": dial tcp 10.132.0.15:8000: connect: connection refused" Jul 27 15:05:47.550630 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.550169 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" event={"ID":"b45193fe-e78e-4aa1-b5ea-beb1650db5ee","Type":"ContainerStarted","Data":"bb22b8575144ce278ad954a1a33e820246a82ddad4782d9e9928fbdf6d2fe3aa"} Jul 27 15:05:47.552979 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.552557 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" event={"ID":"10ed8da3-6018-4f3b-ba4b-31537e008b12","Type":"ContainerStarted","Data":"94b645c8f764950964707a9289d9636a8de2fdd1894226bc29a6d75d6551fc45"} Jul 27 15:05:47.579791 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.577747 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-2tn84" podStartSLOduration=25.781085845 podStartE2EDuration="39.577729173s" totalImagesPullingTime="13.796643328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:47.577140961 +0000 UTC m=+50.858491052" watchObservedRunningTime="2026-07-27 15:05:47.577729173 +0000 UTC m=+50.859079248" Jul 27 15:05:47.595112 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.593859 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" podStartSLOduration=25.763627937 podStartE2EDuration="39.593838645s" totalImagesPullingTime="13.830210708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:47.59225543 +0000 UTC m=+50.873605504" watchObservedRunningTime="2026-07-27 15:05:47.593838645 +0000 UTC m=+50.875188723" Jul 27 15:05:47.646761 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.646702 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-mpgd6" podStartSLOduration=25.804299692 podStartE2EDuration="39.646679948s" totalImagesPullingTime="13.842380256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:47.64467404 +0000 UTC m=+50.926024116" watchObservedRunningTime="2026-07-27 15:05:47.646679948 +0000 UTC m=+50.928030024" Jul 27 15:05:47.646894 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:47.646814 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" podStartSLOduration=15.851358906 podStartE2EDuration="29.646805504s" totalImagesPullingTime="13.795446598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:47.613208529 +0000 UTC m=+50.894558604" watchObservedRunningTime="2026-07-27 15:05:47.646805504 +0000 UTC m=+50.928155582" Jul 27 15:05:48.081218 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.081182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:05:48.081395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.081267 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:05:48.081395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.081319 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:05:48.081395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.081341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:05:48.081395 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081364 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:48.081395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.081393 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.081421 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081443 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls podName:6368b6c0-4289-433d-bdad-d502124b7802 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.081420428 +0000 UTC m=+67.362770496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5t57t" (UID: "6368b6c0-4289-433d-bdad-d502124b7802") : secret "cluster-monitoring-operator-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081453 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081511 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls podName:2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.081495192 +0000 UTC m=+67.362845248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls") pod "cluster-samples-operator-6748489656-2pv4r" (UID: "2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b") : secret "samples-operator-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081539 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.081521303 +0000 UTC m=+67.362871360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : configmap references non-existent config key: service-ca.crt Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081577 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081588 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cff8c555f-7r4j5: secret "image-registry-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081604 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081618 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls podName:3efcde43-a6d0-40de-b34d-d46787bf75d8 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.08160778 +0000 UTC m=+67.362957834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls") pod "image-registry-5cff8c555f-7r4j5" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8") : secret "image-registry-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081642 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs podName:3434d48a-2c83-4375-89e1-de3b25813fa9 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.081630873 +0000 UTC m=+67.362980927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs") pod "router-default-6f8c885cd9-2mgwd" (UID: "3434d48a-2c83-4375-89e1-de3b25813fa9") : secret "router-metrics-certs-default" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081682 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 15:05:48.081715 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.081711 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls podName:df6cb94b-d781-4eb7-87a1-b85241de9b7c nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.081701013 +0000 UTC m=+67.363051069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls") pod "dns-default-52wmr" (UID: "df6cb94b-d781-4eb7-87a1-b85241de9b7c") : secret "dns-default-metrics-tls" not found Jul 27 15:05:48.257092 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.256211 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dvnsj"] Jul 27 15:05:48.274795 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.274745 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvnsj"] Jul 27 15:05:48.274967 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.274888 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.276941 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.276917 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 15:05:48.277192 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.277175 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 15:05:48.277390 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.277374 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zc6mm\"" Jul 27 15:05:48.298583 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.298500 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:48.372416 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.372378 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:48.384478 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.384447 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29747912-51e0-4a07-9cdf-2c3436233245-crio-socket\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.384622 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.384487 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29747912-51e0-4a07-9cdf-2c3436233245-data-volume\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.384667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.384623 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.384667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.384658 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.384748 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.384719 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2c8\" (UniqueName: \"kubernetes.io/projected/29747912-51e0-4a07-9cdf-2c3436233245-kube-api-access-kd2c8\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.485599 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.485559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29747912-51e0-4a07-9cdf-2c3436233245-crio-socket\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.485599 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.485595 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29747912-51e0-4a07-9cdf-2c3436233245-data-volume\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.485847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.485665 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.485847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.485683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.485847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.485709 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29747912-51e0-4a07-9cdf-2c3436233245-crio-socket\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.485847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.485737 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2c8\" (UniqueName: \"kubernetes.io/projected/29747912-51e0-4a07-9cdf-2c3436233245-kube-api-access-kd2c8\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.486037 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.485867 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 15:05:48.486037 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.485941 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.985919247 +0000 UTC m=+52.267269303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : secret "insights-runtime-extractor-tls" not found Jul 27 15:05:48.486037 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.485876 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 15:05:48.486037 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.486002 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:48.985986159 +0000 UTC m=+52.267336215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : configmap "kube-rbac-proxy" not found Jul 27 15:05:48.486037 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.486024 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29747912-51e0-4a07-9cdf-2c3436233245-data-volume\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.496689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.496663 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2c8\" (UniqueName: \"kubernetes.io/projected/29747912-51e0-4a07-9cdf-2c3436233245-kube-api-access-kd2c8\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.559782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.559689 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7c56v" event={"ID":"d1f42dc0-6dc2-4201-8f41-b20652979554","Type":"ContainerStarted","Data":"044de8b5d01dfa67194490ad1fff801369c510090ab96062c4a12bf1134bb567"} Jul 27 15:05:48.562267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.562045 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/0.log" Jul 27 15:05:48.562267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.562104 2553 generic.go:356] "Generic (PLEG): container finished" podID="c38c59b8-7ecf-4006-8a0e-e75eb798ef14" containerID="020fdc0ba76db7bd73ac057f63a119a0380050e57ae1220cf091a8a846b1dcda" exitCode=255 Jul 27 15:05:48.562267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.562164 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" event={"ID":"c38c59b8-7ecf-4006-8a0e-e75eb798ef14","Type":"ContainerDied","Data":"020fdc0ba76db7bd73ac057f63a119a0380050e57ae1220cf091a8a846b1dcda"} Jul 27 15:05:48.562572 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.562552 2553 scope.go:117] "RemoveContainer" containerID="020fdc0ba76db7bd73ac057f63a119a0380050e57ae1220cf091a8a846b1dcda" Jul 27 15:05:48.565040 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.565008 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" event={"ID":"1b2e370c-ec55-4f58-8ec7-8507beae1c4a","Type":"ContainerStarted","Data":"f7f41c38006b20d474e05e011f2de4cea87626ae03c0f1626b824edd8033be85"} Jul 27 15:05:48.566677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.566652 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" event={"ID":"c0d6fee3-17dd-465f-9239-cf9dcea137d7","Type":"ContainerStarted","Data":"eae7f8421c9c84346b6b9273eb185ac1bb3ffefa25ed50e2df110b34282bc576"} Jul 27 15:05:48.568779 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.568755 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" event={"ID":"29ea7830-521c-449e-9c10-6ab80a99a36e","Type":"ContainerStarted","Data":"f429c48a7a9fbbe1ed617a5101768b7a06056cd41860675376e912c3e85a1fd9"} Jul 27 15:05:48.568877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.568792 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" event={"ID":"29ea7830-521c-449e-9c10-6ab80a99a36e","Type":"ContainerStarted","Data":"aa0235e4418fce7065bd247f4d6fb73469c322866fe7303b032f21a97ee7c5d9"} Jul 27 15:05:48.568877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.568810 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" event={"ID":"29ea7830-521c-449e-9c10-6ab80a99a36e","Type":"ContainerStarted","Data":"b781040e7186bac0a45f4b81ec70d139a4c464a5d86938e2d5b266aaa03e1302"} Jul 27 15:05:48.570122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.570104 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d9d8dd89-6ncqv" Jul 27 15:05:48.577531 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.577448 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7c56v" podStartSLOduration=2.780996703 podStartE2EDuration="16.577432879s" totalImagesPullingTime="13.796436176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:48.576767632 +0000 UTC m=+51.858117722" watchObservedRunningTime="2026-07-27 15:05:48.577432879 +0000 UTC m=+51.858782933" Jul 27 15:05:48.628481 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.628438 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" podStartSLOduration=26.829051831 podStartE2EDuration="40.628417889s" totalImagesPullingTime="13.799366058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:48.627356824 +0000 UTC m=+51.908706898" watchObservedRunningTime="2026-07-27 15:05:48.628417889 +0000 UTC m=+51.909767962" Jul 27 15:05:48.642584 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.642537 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c5c97687b-rhsqs" podStartSLOduration=16.843734681 podStartE2EDuration="30.642523228s" totalImagesPullingTime="13.798788547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:48.641159551 +0000 UTC m=+51.922509626" watchObservedRunningTime="2026-07-27 15:05:48.642523228 +0000 UTC m=+51.923873343" Jul 27 15:05:48.659566 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.659510 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" podStartSLOduration=16.839861125 podStartE2EDuration="30.65948846s" totalImagesPullingTime="13.819627335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:48.65921805 +0000 UTC m=+51.940568127" watchObservedRunningTime="2026-07-27 15:05:48.65948846 +0000 UTC m=+51.940838536" Jul 27 15:05:48.996643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.996601 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.996798 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:48.996651 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:48.996798 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.996738 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 15:05:48.996798 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.996789 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 15:05:48.996915 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.996824 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:49.996799587 +0000 UTC m=+53.278149641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : configmap "kube-rbac-proxy" not found Jul 27 15:05:48.996915 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:48.996844 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:49.996834359 +0000 UTC m=+53.278184416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : secret "insights-runtime-extractor-tls" not found Jul 27 15:05:49.318203 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.318130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:49.380414 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.380309 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:49.451886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.451858 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bmnp2_fa7f660f-4dd4-457a-950c-07412f65b7da/dns-node-resolver/0.log" Jul 27 15:05:49.548992 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.548953 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:49.574497 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.574430 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:05:49.574945 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.574927 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/0.log" Jul 27 15:05:49.575022 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.574969 2553 generic.go:356] "Generic (PLEG): container finished" podID="c38c59b8-7ecf-4006-8a0e-e75eb798ef14" containerID="f46ef1b32638616bf389c062dec081a8542c7c3e70fe625536e1eab0904d13b0" exitCode=255 Jul 27 15:05:49.575124 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.575102 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" event={"ID":"c38c59b8-7ecf-4006-8a0e-e75eb798ef14","Type":"ContainerDied","Data":"f46ef1b32638616bf389c062dec081a8542c7c3e70fe625536e1eab0904d13b0"} Jul 27 15:05:49.575183 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.575158 2553 scope.go:117] "RemoveContainer" containerID="020fdc0ba76db7bd73ac057f63a119a0380050e57ae1220cf091a8a846b1dcda" Jul 27 15:05:49.575425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.575408 2553 scope.go:117] "RemoveContainer" containerID="f46ef1b32638616bf389c062dec081a8542c7c3e70fe625536e1eab0904d13b0" Jul 27 15:05:49.575638 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:49.575617 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-798gp_openshift-console-operator(c38c59b8-7ecf-4006-8a0e-e75eb798ef14)\"" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" podUID="c38c59b8-7ecf-4006-8a0e-e75eb798ef14" Jul 27 15:05:49.618014 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:49.617989 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:50.007471 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.007436 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:50.007471 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.007477 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:50.007738 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:50.007564 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 15:05:50.007738 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:50.007581 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 15:05:50.007738 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:50.007630 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:52.007617401 +0000 UTC m=+55.288967453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : secret "insights-runtime-extractor-tls" not found Jul 27 15:05:50.007738 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:50.007643 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:52.007637385 +0000 UTC m=+55.288987436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : configmap "kube-rbac-proxy" not found Jul 27 15:05:50.019065 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.019021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:50.117376 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.117348 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:50.391810 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.391776 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:50.487969 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.487938 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:50.579345 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.579318 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:05:50.579737 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.579721 2553 scope.go:117] "RemoveContainer" containerID="f46ef1b32638616bf389c062dec081a8542c7c3e70fe625536e1eab0904d13b0" Jul 27 15:05:50.579907 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:50.579886 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-798gp_openshift-console-operator(c38c59b8-7ecf-4006-8a0e-e75eb798ef14)\"" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" podUID="c38c59b8-7ecf-4006-8a0e-e75eb798ef14" Jul 27 15:05:50.651734 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:50.651655 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l24dr_65b2001b-a43e-4060-8dc1-e69aae55dd22/node-ca/0.log" Jul 27 15:05:51.052000 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.051921 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7c56v_d1f42dc0-6dc2-4201-8f41-b20652979554/serve-healthcheck-canary/0.log" Jul 27 15:05:51.393424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.393393 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp"] Jul 27 15:05:51.399378 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.399358 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" Jul 27 15:05:51.401445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.401417 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 27 15:05:51.401543 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.401421 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5cc27\"" Jul 27 15:05:51.402238 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.402216 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 27 15:05:51.403788 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.403770 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp"] Jul 27 15:05:51.429510 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.429473 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:51.498370 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.498341 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:51.521712 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.521686 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzd79\" (UniqueName: \"kubernetes.io/projected/d603c607-eec2-4c7c-b605-89b31e3aa8e1-kube-api-access-rzd79\") pod \"migrator-6865c9967-mq8wp\" (UID: \"d603c607-eec2-4c7c-b605-89b31e3aa8e1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" Jul 27 15:05:51.616122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.616091 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-z2mm9"] Jul 27 15:05:51.618113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.618098 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.620096 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.620071 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 15:05:51.620226 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.620202 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 15:05:51.620411 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.620387 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 15:05:51.620501 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.620466 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 15:05:51.620501 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.620476 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8b7pl\"" Jul 27 15:05:51.622386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.622367 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzd79\" (UniqueName: \"kubernetes.io/projected/d603c607-eec2-4c7c-b605-89b31e3aa8e1-kube-api-access-rzd79\") pod \"migrator-6865c9967-mq8wp\" (UID: \"d603c607-eec2-4c7c-b605-89b31e3aa8e1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" Jul 27 15:05:51.628562 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.628542 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-z2mm9"] Jul 27 15:05:51.633382 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.633362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzd79\" (UniqueName: \"kubernetes.io/projected/d603c607-eec2-4c7c-b605-89b31e3aa8e1-kube-api-access-rzd79\") pod \"migrator-6865c9967-mq8wp\" (UID: \"d603c607-eec2-4c7c-b605-89b31e3aa8e1\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" Jul 27 15:05:51.709205 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.709132 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" Jul 27 15:05:51.723399 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.723373 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpjv\" (UniqueName: \"kubernetes.io/projected/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-kube-api-access-9tpjv\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.723616 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.723586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-signing-cabundle\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.723686 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.723656 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-signing-key\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.724987 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.724963 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:51.824164 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.824130 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpjv\" (UniqueName: \"kubernetes.io/projected/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-kube-api-access-9tpjv\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.824302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.824261 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-signing-cabundle\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.824302 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.824293 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-signing-key\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.825040 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.825013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:51.825213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.825042 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-signing-cabundle\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.826742 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.826718 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-signing-key\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.832879 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.832857 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp"] Jul 27 15:05:51.834623 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.834604 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpjv\" (UniqueName: \"kubernetes.io/projected/c7f2e4d1-78bd-463a-a64b-4f59c9a4943d-kube-api-access-9tpjv\") pod \"service-ca-658d7b7c78-z2mm9\" (UID: \"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d\") " pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:51.836120 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:51.836085 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd603c607_eec2_4c7c_b605_89b31e3aa8e1.slice/crio-3aa5d46d4a78766b1f054e3e16d08d892d01b0f63cc656f42877ed2d2a1e8405 WatchSource:0}: Error finding container 3aa5d46d4a78766b1f054e3e16d08d892d01b0f63cc656f42877ed2d2a1e8405: Status 404 returned error can't find the container with id 3aa5d46d4a78766b1f054e3e16d08d892d01b0f63cc656f42877ed2d2a1e8405 Jul 27 15:05:51.925498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.925453 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:05:51.925673 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:51.925599 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 15:05:51.925673 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:51.925664 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert podName:aa23ab11-efac-4d9a-a0fb-77a43c481313 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:07.925648151 +0000 UTC m=+71.206998207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-hcl95" (UID: "aa23ab11-efac-4d9a-a0fb-77a43c481313") : secret "networking-console-plugin-cert" not found Jul 27 15:05:51.927276 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:51.927256 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" Jul 27 15:05:52.026757 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.026728 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:52.026757 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.026760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:52.026953 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:52.026863 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 15:05:52.026953 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:52.026869 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 15:05:52.026953 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:52.026938 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:56.026915994 +0000 UTC m=+59.308266064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : configmap "kube-rbac-proxy" not found Jul 27 15:05:52.027077 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:52.026977 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:05:56.026961321 +0000 UTC m=+59.308311377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : secret "insights-runtime-extractor-tls" not found Jul 27 15:05:52.045843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.045815 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-z2mm9"] Jul 27 15:05:52.049318 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:05:52.049294 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f2e4d1_78bd_463a_a64b_4f59c9a4943d.slice/crio-4febfcb99c13fe58c520444d322fceeb52312583348ef3b36e8eb6ec836b6fc2 WatchSource:0}: Error finding container 4febfcb99c13fe58c520444d322fceeb52312583348ef3b36e8eb6ec836b6fc2: Status 404 returned error can't find the container with id 4febfcb99c13fe58c520444d322fceeb52312583348ef3b36e8eb6ec836b6fc2 Jul 27 15:05:52.516473 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.516329 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:52.516473 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.516398 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:05:52.516927 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.516826 2553 scope.go:117] "RemoveContainer" containerID="f46ef1b32638616bf389c062dec081a8542c7c3e70fe625536e1eab0904d13b0" Jul 27 15:05:52.517267 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:52.517195 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-798gp_openshift-console-operator(c38c59b8-7ecf-4006-8a0e-e75eb798ef14)\"" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" podUID="c38c59b8-7ecf-4006-8a0e-e75eb798ef14" Jul 27 15:05:52.550759 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.550726 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:52.589658 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.589622 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" event={"ID":"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d","Type":"ContainerStarted","Data":"a6b9afe3ff7299c3e34b63ea39eca6c672b5d4f53932ccdc70efe61e2d1f554e"} Jul 27 15:05:52.589836 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.589679 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" event={"ID":"c7f2e4d1-78bd-463a-a64b-4f59c9a4943d","Type":"ContainerStarted","Data":"4febfcb99c13fe58c520444d322fceeb52312583348ef3b36e8eb6ec836b6fc2"} Jul 27 15:05:52.591012 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.590989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" event={"ID":"d603c607-eec2-4c7c-b605-89b31e3aa8e1","Type":"ContainerStarted","Data":"3aa5d46d4a78766b1f054e3e16d08d892d01b0f63cc656f42877ed2d2a1e8405"} Jul 27 15:05:52.608144 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:52.608098 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-z2mm9" podStartSLOduration=1.6080800549999998 podStartE2EDuration="1.608080055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:52.607201061 +0000 UTC m=+55.888551137" watchObservedRunningTime="2026-07-27 15:05:52.608080055 +0000 UTC m=+55.889430129" Jul 27 15:05:53.378413 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:53.378296 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" limit=1024 Jul 27 15:05:53.598656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:53.598561 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" event={"ID":"d603c607-eec2-4c7c-b605-89b31e3aa8e1","Type":"ContainerStarted","Data":"26f58ad21169c27ee612a123694ee536a1c48f69c9cb40987350e62d6004206f"} Jul 27 15:05:53.598656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:53.598618 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" event={"ID":"d603c607-eec2-4c7c-b605-89b31e3aa8e1","Type":"ContainerStarted","Data":"3c3b808a1e15297a2c7b3629af6ce6b5b3160bd7dd8479643b44f11ac179c43e"} Jul 27 15:05:53.619099 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:53.619016 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-mq8wp" podStartSLOduration=1.248732337 podStartE2EDuration="2.618997608s" totalImagesPullingTime="1.370265271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:05:53.618242148 +0000 UTC m=+56.899592223" watchObservedRunningTime="2026-07-27 15:05:53.618997608 +0000 UTC m=+56.900347684" Jul 27 15:05:56.067843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:56.067806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:56.067843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:56.067848 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:05:56.068363 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:56.067926 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 15:05:56.068363 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:56.067957 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 15:05:56.068363 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:56.067986 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.067970761 +0000 UTC m=+67.349320817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : configmap "kube-rbac-proxy" not found Jul 27 15:05:56.068363 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:05:56.068015 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls podName:29747912-51e0-4a07-9cdf-2c3436233245 nodeName:}" failed. No retries permitted until 2026-07-27 15:06:04.067996989 +0000 UTC m=+67.349347046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvnsj" (UID: "29747912-51e0-4a07-9cdf-2c3436233245") : secret "insights-runtime-extractor-tls" not found Jul 27 15:05:58.347946 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:05:58.347913 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mc7bs" Jul 27 15:06:02.934154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:02.934121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:06:02.936395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:02.936378 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 15:06:02.947122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:02.947105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66371b5f-01b5-4329-bc7e-069d801574e9-metrics-certs\") pod \"network-metrics-daemon-cv4fx\" (UID: \"66371b5f-01b5-4329-bc7e-069d801574e9\") " pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:06:03.034702 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.034670 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:06:03.034891 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.034734 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:06:03.037321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.037298 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84h6\" (UniqueName: \"kubernetes.io/projected/253dcffa-69df-42d7-98a1-3a9113179c30-kube-api-access-b84h6\") pod \"network-check-target-kcnfc\" (UID: \"253dcffa-69df-42d7-98a1-3a9113179c30\") " pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:06:03.039691 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.039672 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 15:06:03.047729 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.047701 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21f2ae8c-8a22-46d6-ae00-414fd7d0cefe-original-pull-secret\") pod \"global-pull-secret-syncer-bjd66\" (UID: \"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe\") " pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:06:03.182556 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.182526 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkhzc\"" Jul 27 15:06:03.189993 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.189915 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2bzfc\"" Jul 27 15:06:03.190170 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.190060 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cv4fx" Jul 27 15:06:03.192674 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.192653 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjd66" Jul 27 15:06:03.197386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.197368 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:06:03.342812 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.342785 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cv4fx"] Jul 27 15:06:03.346126 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:03.346100 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66371b5f_01b5_4329_bc7e_069d801574e9.slice/crio-9e85e7f23e51adc1ebb6d073fe40eba61772b78f776322966e2334279f1e3130 WatchSource:0}: Error finding container 9e85e7f23e51adc1ebb6d073fe40eba61772b78f776322966e2334279f1e3130: Status 404 returned error can't find the container with id 9e85e7f23e51adc1ebb6d073fe40eba61772b78f776322966e2334279f1e3130 Jul 27 15:06:03.565459 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.565433 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kcnfc"] Jul 27 15:06:03.568130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.568098 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjd66"] Jul 27 15:06:03.568390 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:03.568370 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253dcffa_69df_42d7_98a1_3a9113179c30.slice/crio-eabae9f571dc8b907bc5af2afb8276d969bd02757a299b0ca47a120a3345c3d1 WatchSource:0}: Error finding container eabae9f571dc8b907bc5af2afb8276d969bd02757a299b0ca47a120a3345c3d1: Status 404 returned error can't find the container with id eabae9f571dc8b907bc5af2afb8276d969bd02757a299b0ca47a120a3345c3d1 Jul 27 15:06:03.571293 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:03.571277 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f2ae8c_8a22_46d6_ae00_414fd7d0cefe.slice/crio-95d9e8d25e3ec85b0bbb746f2b4ac943be3a1d53aaef475b30a4a32ed8550418 WatchSource:0}: Error finding container 95d9e8d25e3ec85b0bbb746f2b4ac943be3a1d53aaef475b30a4a32ed8550418: Status 404 returned error can't find the container with id 95d9e8d25e3ec85b0bbb746f2b4ac943be3a1d53aaef475b30a4a32ed8550418 Jul 27 15:06:03.632943 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.632914 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjd66" event={"ID":"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe","Type":"ContainerStarted","Data":"95d9e8d25e3ec85b0bbb746f2b4ac943be3a1d53aaef475b30a4a32ed8550418"} Jul 27 15:06:03.634028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.634001 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cv4fx" event={"ID":"66371b5f-01b5-4329-bc7e-069d801574e9","Type":"ContainerStarted","Data":"9e85e7f23e51adc1ebb6d073fe40eba61772b78f776322966e2334279f1e3130"} Jul 27 15:06:03.635368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.635348 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kcnfc" event={"ID":"253dcffa-69df-42d7-98a1-3a9113179c30","Type":"ContainerStarted","Data":"c1cffb757ceb94ed9f5ed3b70ca0cb6b6ffd92a251370d96e008c9781738e2f6"} Jul 27 15:06:03.635445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.635379 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kcnfc" event={"ID":"253dcffa-69df-42d7-98a1-3a9113179c30","Type":"ContainerStarted","Data":"eabae9f571dc8b907bc5af2afb8276d969bd02757a299b0ca47a120a3345c3d1"} Jul 27 15:06:03.635445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.635418 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:06:03.653006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:03.652962 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kcnfc" podStartSLOduration=65.652946726 podStartE2EDuration="1m5.652946726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:03.651778397 +0000 UTC m=+66.933128472" watchObservedRunningTime="2026-07-27 15:06:03.652946726 +0000 UTC m=+66.934296800" Jul 27 15:06:04.143812 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.143772 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.143852 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.143889 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.143909 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.143956 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.144003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.144099 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:04.144299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.144122 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:04.144637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.144523 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29747912-51e0-4a07-9cdf-2c3436233245-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:06:04.144850 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.144823 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3434d48a-2c83-4375-89e1-de3b25813fa9-service-ca-bundle\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:04.147392 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.147004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2pv4r\" (UID: \"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:06:04.147392 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.147108 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6368b6c0-4289-433d-bdad-d502124b7802-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5t57t\" (UID: \"6368b6c0-4289-433d-bdad-d502124b7802\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:06:04.147392 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.147264 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"image-registry-5cff8c555f-7r4j5\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:06:04.147392 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.147350 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3434d48a-2c83-4375-89e1-de3b25813fa9-metrics-certs\") pod \"router-default-6f8c885cd9-2mgwd\" (UID: \"3434d48a-2c83-4375-89e1-de3b25813fa9\") " pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:04.147625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.147446 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29747912-51e0-4a07-9cdf-2c3436233245-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvnsj\" (UID: \"29747912-51e0-4a07-9cdf-2c3436233245\") " pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:06:04.147625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.147503 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df6cb94b-d781-4eb7-87a1-b85241de9b7c-metrics-tls\") pod \"dns-default-52wmr\" (UID: \"df6cb94b-d781-4eb7-87a1-b85241de9b7c\") " pod="openshift-dns/dns-default-52wmr" Jul 27 15:06:04.188068 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.188018 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zc6mm\"" Jul 27 15:06:04.195742 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.195716 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvnsj" Jul 27 15:06:04.240736 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.240678 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mw646\"" Jul 27 15:06:04.249713 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.249648 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:06:04.255776 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.255753 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z77zc\"" Jul 27 15:06:04.263806 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.263638 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" Jul 27 15:06:04.274469 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.274440 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4vlrj\"" Jul 27 15:06:04.282490 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.282471 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" Jul 27 15:06:04.282962 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.282942 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-j7gvt\"" Jul 27 15:06:04.289755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.289734 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-spr6f\"" Jul 27 15:06:04.290754 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.290735 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:04.298606 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.298587 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-52wmr" Jul 27 15:06:04.599045 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.598953 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvnsj"] Jul 27 15:06:04.666652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.666613 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cff8c555f-7r4j5"] Jul 27 15:06:04.737261 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:04.737183 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29747912_51e0_4a07_9cdf_2c3436233245.slice/crio-ada92f96ac2fb187d65fcc4ea59886028624e4f33f76eaef76d2e377dc7ffc71 WatchSource:0}: Error finding container ada92f96ac2fb187d65fcc4ea59886028624e4f33f76eaef76d2e377dc7ffc71: Status 404 returned error can't find the container with id ada92f96ac2fb187d65fcc4ea59886028624e4f33f76eaef76d2e377dc7ffc71 Jul 27 15:06:04.737843 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:04.737820 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efcde43_a6d0_40de_b34d_d46787bf75d8.slice/crio-ced1e6e1758636c20307b8cacdb63fdce1cd075edeb21678329d5a292d3a2421 WatchSource:0}: Error finding container ced1e6e1758636c20307b8cacdb63fdce1cd075edeb21678329d5a292d3a2421: Status 404 returned error can't find the container with id ced1e6e1758636c20307b8cacdb63fdce1cd075edeb21678329d5a292d3a2421 Jul 27 15:06:04.840133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.840017 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t"] Jul 27 15:06:04.840768 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.840657 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f8c885cd9-2mgwd"] Jul 27 15:06:04.843487 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.843283 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r"] Jul 27 15:06:04.843633 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:04.843594 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6368b6c0_4289_433d_bdad_d502124b7802.slice/crio-4fb644945e72ce4aa79fdffeaf4edc6d47b8ffb35286263b8087aa255e58b654 WatchSource:0}: Error finding container 4fb644945e72ce4aa79fdffeaf4edc6d47b8ffb35286263b8087aa255e58b654: Status 404 returned error can't find the container with id 4fb644945e72ce4aa79fdffeaf4edc6d47b8ffb35286263b8087aa255e58b654 Jul 27 15:06:04.844875 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:04.844852 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3434d48a_2c83_4375_89e1_de3b25813fa9.slice/crio-11d02479c127da4bf97c46a03bcc309da4a01ec85b6ede096980caa52e43aec5 WatchSource:0}: Error finding container 11d02479c127da4bf97c46a03bcc309da4a01ec85b6ede096980caa52e43aec5: Status 404 returned error can't find the container with id 11d02479c127da4bf97c46a03bcc309da4a01ec85b6ede096980caa52e43aec5 Jul 27 15:06:04.846772 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:04.846725 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-52wmr"] Jul 27 15:06:04.851658 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:04.851629 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6cb94b_d781_4eb7_87a1_b85241de9b7c.slice/crio-bd936574f57240e53b8da00cff4a057cc1e9db31a0ac3567ae717e2ca56fe46d WatchSource:0}: Error finding container bd936574f57240e53b8da00cff4a057cc1e9db31a0ac3567ae717e2ca56fe46d: Status 404 returned error can't find the container with id bd936574f57240e53b8da00cff4a057cc1e9db31a0ac3567ae717e2ca56fe46d Jul 27 15:06:05.193866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.193832 2553 scope.go:117] "RemoveContainer" containerID="f46ef1b32638616bf389c062dec081a8542c7c3e70fe625536e1eab0904d13b0" Jul 27 15:06:05.654644 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.653657 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:06:05.654644 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.653759 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" event={"ID":"c38c59b8-7ecf-4006-8a0e-e75eb798ef14","Type":"ContainerStarted","Data":"eaec00900701814eda2330807bf924ff0ac99660e64c3546e22df24ce1641e4c"} Jul 27 15:06:05.655149 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.654968 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:06:05.659122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.658537 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cv4fx" event={"ID":"66371b5f-01b5-4329-bc7e-069d801574e9","Type":"ContainerStarted","Data":"b329491d8f2f7758ba065b98e49e44f013423e539496c5088147c1ad3f638652"} Jul 27 15:06:05.659122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.658586 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cv4fx" event={"ID":"66371b5f-01b5-4329-bc7e-069d801574e9","Type":"ContainerStarted","Data":"d1db5eb59da48b1a1c0ac1c599a286096a302558a98c0a75a252478c5b60ed7f"} Jul 27 15:06:05.662349 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.661868 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" event={"ID":"3434d48a-2c83-4375-89e1-de3b25813fa9","Type":"ContainerStarted","Data":"3fb091233d296ae61e9baec43ee637098cee2424706aabb307d42e4a94f82ec1"} Jul 27 15:06:05.662349 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.661915 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" event={"ID":"3434d48a-2c83-4375-89e1-de3b25813fa9","Type":"ContainerStarted","Data":"11d02479c127da4bf97c46a03bcc309da4a01ec85b6ede096980caa52e43aec5"} Jul 27 15:06:05.664424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.664314 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-52wmr" event={"ID":"df6cb94b-d781-4eb7-87a1-b85241de9b7c","Type":"ContainerStarted","Data":"bd936574f57240e53b8da00cff4a057cc1e9db31a0ac3567ae717e2ca56fe46d"} Jul 27 15:06:05.668086 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.668061 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" event={"ID":"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b","Type":"ContainerStarted","Data":"5c31d076aacede13b5130aa798f2a00b1f8c20d08157bce5ce0e24197e5fe02f"} Jul 27 15:06:05.671027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.671003 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" event={"ID":"6368b6c0-4289-433d-bdad-d502124b7802","Type":"ContainerStarted","Data":"4fb644945e72ce4aa79fdffeaf4edc6d47b8ffb35286263b8087aa255e58b654"} Jul 27 15:06:05.676077 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.675116 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" podStartSLOduration=43.845299903 podStartE2EDuration="57.675099903s" totalImagesPullingTime="13.8298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:05.672670295 +0000 UTC m=+68.954020370" watchObservedRunningTime="2026-07-27 15:06:05.675099903 +0000 UTC m=+68.956449979" Jul 27 15:06:05.676416 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.676388 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" event={"ID":"3efcde43-a6d0-40de-b34d-d46787bf75d8","Type":"ContainerStarted","Data":"a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9"} Jul 27 15:06:05.676505 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.676435 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" event={"ID":"3efcde43-a6d0-40de-b34d-d46787bf75d8","Type":"ContainerStarted","Data":"ced1e6e1758636c20307b8cacdb63fdce1cd075edeb21678329d5a292d3a2421"} Jul 27 15:06:05.676965 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.676946 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:06:05.680777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.680710 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvnsj" event={"ID":"29747912-51e0-4a07-9cdf-2c3436233245","Type":"ContainerStarted","Data":"ada92f96ac2fb187d65fcc4ea59886028624e4f33f76eaef76d2e377dc7ffc71"} Jul 27 15:06:05.692527 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.691132 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cv4fx" podStartSLOduration=66.275024734 podStartE2EDuration="1m7.691115927s" totalImagesPullingTime="1.416091193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:05.690747698 +0000 UTC m=+68.972097774" watchObservedRunningTime="2026-07-27 15:06:05.691115927 +0000 UTC m=+68.972466007" Jul 27 15:06:05.735534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:05.733974 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" podStartSLOduration=57.733951473 podStartE2EDuration="57.733951473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:05.710894379 +0000 UTC m=+68.992244455" watchObservedRunningTime="2026-07-27 15:06:05.733951473 +0000 UTC m=+69.015301547" Jul 27 15:06:06.165304 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:06.165067 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-798gp" Jul 27 15:06:06.185224 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:06.184832 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" podStartSLOduration=68.184807538 podStartE2EDuration="1m8.184807538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:05.73551562 +0000 UTC m=+69.016865707" watchObservedRunningTime="2026-07-27 15:06:06.184807538 +0000 UTC m=+69.466157613" Jul 27 15:06:06.291536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:06.291455 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:06.294675 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:06.294453 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:06.691581 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:06.691521 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:06.692981 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:06.692767 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f8c885cd9-2mgwd" Jul 27 15:06:07.985085 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:07.984803 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:06:07.988432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:07.988403 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa23ab11-efac-4d9a-a0fb-77a43c481313-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-hcl95\" (UID: \"aa23ab11-efac-4d9a-a0fb-77a43c481313\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:06:08.225494 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:08.225462 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hq67j\"" Jul 27 15:06:08.233538 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:08.233511 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" Jul 27 15:06:13.228531 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.228458 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-hcl95"] Jul 27 15:06:13.232821 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:13.232780 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa23ab11_efac_4d9a_a0fb_77a43c481313.slice/crio-6ef17e8024b86675420acb7da3520745536c3be8a02595a9f8d1130b406b0b9f WatchSource:0}: Error finding container 6ef17e8024b86675420acb7da3520745536c3be8a02595a9f8d1130b406b0b9f: Status 404 returned error can't find the container with id 6ef17e8024b86675420acb7da3520745536c3be8a02595a9f8d1130b406b0b9f Jul 27 15:06:13.720211 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.720121 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" event={"ID":"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b","Type":"ContainerStarted","Data":"729c88abc41fb01d5aebc3889cceb6f9a2d056ebce49e2dfc8342f68a919329e"} Jul 27 15:06:13.720211 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.720182 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" event={"ID":"2c90dfd8-8b9c-4ffd-b4c5-a03fddf20a7b","Type":"ContainerStarted","Data":"bc7b98f2477818bd633026a3c2483d55542a59a7263bf532d27bc7f7e167ae7b"} Jul 27 15:06:13.721956 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.721925 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" event={"ID":"6368b6c0-4289-433d-bdad-d502124b7802","Type":"ContainerStarted","Data":"d5eda53e78d1dcbb56c8d0821d37c7a342519568eaf649d9d949df134eb2bde7"} Jul 27 15:06:13.723615 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.723556 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjd66" event={"ID":"21f2ae8c-8a22-46d6-ae00-414fd7d0cefe","Type":"ContainerStarted","Data":"a80f2106418c78eba34d9137e1dc9671a3e1c2e21a476741a206c3d437abac9f"} Jul 27 15:06:13.725274 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.725248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvnsj" event={"ID":"29747912-51e0-4a07-9cdf-2c3436233245","Type":"ContainerStarted","Data":"8c5457cda27d4a101afd553326c716971bdde712a45e8a2f5a99b8bebf8aea51"} Jul 27 15:06:13.726416 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.726399 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" event={"ID":"aa23ab11-efac-4d9a-a0fb-77a43c481313","Type":"ContainerStarted","Data":"6ef17e8024b86675420acb7da3520745536c3be8a02595a9f8d1130b406b0b9f"} Jul 27 15:06:13.728123 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.728104 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-52wmr" event={"ID":"df6cb94b-d781-4eb7-87a1-b85241de9b7c","Type":"ContainerStarted","Data":"cf3540ede42128ca608193ba0dc805d33e2bf6e634f5af4c6fa3f2c8a1bae028"} Jul 27 15:06:13.728219 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.728138 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-52wmr" event={"ID":"df6cb94b-d781-4eb7-87a1-b85241de9b7c","Type":"ContainerStarted","Data":"48097d3c27e6809371d53ddfa95b8e89d0b7b93e699f227835db4af3edce5533"} Jul 27 15:06:13.728275 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.728237 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-52wmr" Jul 27 15:06:13.737099 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.737036 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2pv4r" podStartSLOduration=58.011960071 podStartE2EDuration="1m5.737022778s" totalImagesPullingTime="7.725062707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:13.736589253 +0000 UTC m=+77.017939326" watchObservedRunningTime="2026-07-27 15:06:13.737022778 +0000 UTC m=+77.018372852" Jul 27 15:06:13.757432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.757388 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bjd66" podStartSLOduration=66.227063751 podStartE2EDuration="1m15.757374743s" totalImagesPullingTime="9.530310992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:13.756677305 +0000 UTC m=+77.038027380" watchObservedRunningTime="2026-07-27 15:06:13.757374743 +0000 UTC m=+77.038724817" Jul 27 15:06:13.779395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.779345 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-52wmr" podStartSLOduration=33.541719299 podStartE2EDuration="41.779324904s" totalImagesPullingTime="8.237605605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:13.777399857 +0000 UTC m=+77.058749932" watchObservedRunningTime="2026-07-27 15:06:13.779324904 +0000 UTC m=+77.060674979" Jul 27 15:06:13.797368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:13.797317 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5t57t" podStartSLOduration=57.553710454 podStartE2EDuration="1m5.797301942s" totalImagesPullingTime="8.243591488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:13.796836619 +0000 UTC m=+77.078186692" watchObservedRunningTime="2026-07-27 15:06:13.797301942 +0000 UTC m=+77.078652016" Jul 27 15:06:14.638835 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.638796 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-dr2qr"] Jul 27 15:06:14.665088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.665036 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz"] Jul 27 15:06:14.665246 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.665154 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:14.667375 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.667348 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rppf5\"" Jul 27 15:06:14.667799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.667771 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 15:06:14.667984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.667968 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 15:06:14.679220 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.679198 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz"] Jul 27 15:06:14.679322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.679234 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dr2qr"] Jul 27 15:06:14.679365 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.679320 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:14.682890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.682867 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kqk5n\"" Jul 27 15:06:14.682985 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.682926 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 27 15:06:14.748771 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.748738 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/93c07ce6-2095-4d4f-ae69-ccfd19fc82bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5b9vz\" (UID: \"93c07ce6-2095-4d4f-ae69-ccfd19fc82bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:14.748771 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.748772 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmf9\" (UniqueName: \"kubernetes.io/projected/cf76b9bf-77eb-4e6c-a803-ba91e44b5c57-kube-api-access-8nmf9\") pod \"downloads-7c97dc4555-dr2qr\" (UID: \"cf76b9bf-77eb-4e6c-a803-ba91e44b5c57\") " pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:14.778437 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.778402 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5486f899-kgrps"] Jul 27 15:06:14.792381 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.792355 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.794706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.794679 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 15:06:14.794885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.794727 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 15:06:14.794978 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.794907 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 15:06:14.794978 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.794910 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b9qxj\"" Jul 27 15:06:14.794978 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.794962 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 15:06:14.795200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.795186 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 15:06:14.800981 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.800960 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5486f899-kgrps"] Jul 27 15:06:14.849997 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.849916 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-service-ca\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.850175 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850019 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvq4\" (UniqueName: \"kubernetes.io/projected/79aac610-cc21-4cc3-92c1-651f0b51b647-kube-api-access-khvq4\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.850244 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850185 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/93c07ce6-2095-4d4f-ae69-ccfd19fc82bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5b9vz\" (UID: \"93c07ce6-2095-4d4f-ae69-ccfd19fc82bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:14.850244 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850216 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-serving-cert\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.850339 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850254 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmf9\" (UniqueName: \"kubernetes.io/projected/cf76b9bf-77eb-4e6c-a803-ba91e44b5c57-kube-api-access-8nmf9\") pod \"downloads-7c97dc4555-dr2qr\" (UID: \"cf76b9bf-77eb-4e6c-a803-ba91e44b5c57\") " pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:14.850339 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850292 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-oauth-config\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.850458 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850421 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-oauth-serving-cert\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.850545 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.850487 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-console-config\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.852533 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.852516 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/93c07ce6-2095-4d4f-ae69-ccfd19fc82bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-5b9vz\" (UID: \"93c07ce6-2095-4d4f-ae69-ccfd19fc82bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:14.875852 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.875826 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmf9\" (UniqueName: \"kubernetes.io/projected/cf76b9bf-77eb-4e6c-a803-ba91e44b5c57-kube-api-access-8nmf9\") pod \"downloads-7c97dc4555-dr2qr\" (UID: \"cf76b9bf-77eb-4e6c-a803-ba91e44b5c57\") " pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:14.951383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.951349 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-service-ca\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.951383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.951385 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvq4\" (UniqueName: \"kubernetes.io/projected/79aac610-cc21-4cc3-92c1-651f0b51b647-kube-api-access-khvq4\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.951625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.951414 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-serving-cert\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.951625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.951435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-oauth-config\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.951625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.951465 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-oauth-serving-cert\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.951625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.951491 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-console-config\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.952184 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.952154 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-oauth-serving-cert\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.952325 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.952201 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-console-config\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.952325 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.952230 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-service-ca\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.953842 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.953822 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-oauth-config\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.954062 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.954031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-serving-cert\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.964784 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.964757 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvq4\" (UniqueName: \"kubernetes.io/projected/79aac610-cc21-4cc3-92c1-651f0b51b647-kube-api-access-khvq4\") pod \"console-5486f899-kgrps\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:14.975610 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.975584 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:14.987787 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:14.987764 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:15.102986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.102667 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:15.201394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.201332 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dr2qr"] Jul 27 15:06:15.204864 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:15.204831 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf76b9bf_77eb_4e6c_a803_ba91e44b5c57.slice/crio-991027ce1b8e64991a059032df135b170e32d82e64476ad406906189b386fd33 WatchSource:0}: Error finding container 991027ce1b8e64991a059032df135b170e32d82e64476ad406906189b386fd33: Status 404 returned error can't find the container with id 991027ce1b8e64991a059032df135b170e32d82e64476ad406906189b386fd33 Jul 27 15:06:15.217216 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.217192 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz"] Jul 27 15:06:15.220185 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:15.220161 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c07ce6_2095_4d4f_ae69_ccfd19fc82bc.slice/crio-ae798ce683d1d0c42086770d23b62c47cf0c9840843d845bb9cf576354803b85 WatchSource:0}: Error finding container ae798ce683d1d0c42086770d23b62c47cf0c9840843d845bb9cf576354803b85: Status 404 returned error can't find the container with id ae798ce683d1d0c42086770d23b62c47cf0c9840843d845bb9cf576354803b85 Jul 27 15:06:15.284492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.284469 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5486f899-kgrps"] Jul 27 15:06:15.287009 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:15.286984 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79aac610_cc21_4cc3_92c1_651f0b51b647.slice/crio-f60ffcc7a65089b2096b51045d444ed9bf97ceecf1c3167124f95ea8d0a7ba37 WatchSource:0}: Error finding container f60ffcc7a65089b2096b51045d444ed9bf97ceecf1c3167124f95ea8d0a7ba37: Status 404 returned error can't find the container with id f60ffcc7a65089b2096b51045d444ed9bf97ceecf1c3167124f95ea8d0a7ba37 Jul 27 15:06:15.738065 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.738005 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486f899-kgrps" event={"ID":"79aac610-cc21-4cc3-92c1-651f0b51b647","Type":"ContainerStarted","Data":"f60ffcc7a65089b2096b51045d444ed9bf97ceecf1c3167124f95ea8d0a7ba37"} Jul 27 15:06:15.739484 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.739445 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dr2qr" event={"ID":"cf76b9bf-77eb-4e6c-a803-ba91e44b5c57","Type":"ContainerStarted","Data":"991027ce1b8e64991a059032df135b170e32d82e64476ad406906189b386fd33"} Jul 27 15:06:15.741331 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.741296 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" event={"ID":"aa23ab11-efac-4d9a-a0fb-77a43c481313","Type":"ContainerStarted","Data":"6099ccf7d789e3776d125ac5bd8f4f7d8b8dcb361242fdb883aa4f307a277423"} Jul 27 15:06:15.742817 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.742790 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" event={"ID":"93c07ce6-2095-4d4f-ae69-ccfd19fc82bc","Type":"ContainerStarted","Data":"ae798ce683d1d0c42086770d23b62c47cf0c9840843d845bb9cf576354803b85"} Jul 27 15:06:15.761006 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:15.760867 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-hcl95" podStartSLOduration=37.941791456 podStartE2EDuration="39.760848339s" totalImagesPullingTime="1.819056883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:15.757563919 +0000 UTC m=+79.038913993" watchObservedRunningTime="2026-07-27 15:06:15.760848339 +0000 UTC m=+79.042198425" Jul 27 15:06:16.691701 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.691654 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75c4b55df4-c56w5"] Jul 27 15:06:16.705683 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.705645 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.714121 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.714094 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c4b55df4-c56w5"] Jul 27 15:06:16.717267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.717243 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 15:06:16.772332 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-console-config\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.772800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772341 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-oauth-serving-cert\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.772800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772489 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-trusted-ca-bundle\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.772800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772524 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrx4\" (UniqueName: \"kubernetes.io/projected/564388b9-da07-4f7b-8847-a1eafc370934-kube-api-access-mhrx4\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.772800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772591 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-serving-cert\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.772800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772631 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-oauth-config\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.772800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.772672 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-service-ca\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873104 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873069 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-serving-cert\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873104 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-oauth-config\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-service-ca\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-console-config\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873181 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-oauth-serving-cert\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873293 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-trusted-ca-bundle\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.873550 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.873339 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrx4\" (UniqueName: \"kubernetes.io/projected/564388b9-da07-4f7b-8847-a1eafc370934-kube-api-access-mhrx4\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.874104 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.874041 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-console-config\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.874240 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.874223 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-service-ca\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.874299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.874272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-oauth-serving-cert\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.874425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.874397 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-trusted-ca-bundle\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.875593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.875568 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-serving-cert\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.875691 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.875657 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-oauth-config\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:16.888926 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:16.888904 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrx4\" (UniqueName: \"kubernetes.io/projected/564388b9-da07-4f7b-8847-a1eafc370934-kube-api-access-mhrx4\") pod \"console-75c4b55df4-c56w5\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:17.020438 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.020325 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:17.222434 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.222394 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75c4b55df4-c56w5"] Jul 27 15:06:17.227040 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:17.227005 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564388b9_da07_4f7b_8847_a1eafc370934.slice/crio-2f3c15a5ffb60a45bb568ad03b00e127562bffb1f7599e0b41955b784a6b249f WatchSource:0}: Error finding container 2f3c15a5ffb60a45bb568ad03b00e127562bffb1f7599e0b41955b784a6b249f: Status 404 returned error can't find the container with id 2f3c15a5ffb60a45bb568ad03b00e127562bffb1f7599e0b41955b784a6b249f Jul 27 15:06:17.760467 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.760206 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" event={"ID":"93c07ce6-2095-4d4f-ae69-ccfd19fc82bc","Type":"ContainerStarted","Data":"eb56fc5ee4f645cba4db7630538eda9b07f2954241c3ed41781f0003687150b5"} Jul 27 15:06:17.760467 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.760309 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:17.762321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.762261 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c4b55df4-c56w5" event={"ID":"564388b9-da07-4f7b-8847-a1eafc370934","Type":"ContainerStarted","Data":"2f3c15a5ffb60a45bb568ad03b00e127562bffb1f7599e0b41955b784a6b249f"} Jul 27 15:06:17.766461 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.766436 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" Jul 27 15:06:17.777539 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:17.777483 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-5b9vz" podStartSLOduration=1.972269023 podStartE2EDuration="3.777464473s" totalImagesPullingTime="1.80519545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:17.776399986 +0000 UTC m=+81.057750062" watchObservedRunningTime="2026-07-27 15:06:17.777464473 +0000 UTC m=+81.058814548" Jul 27 15:06:18.827075 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.825424 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zps4j"] Jul 27 15:06:18.889093 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.889027 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zps4j"] Jul 27 15:06:18.889252 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.889182 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:18.904813 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.904562 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 27 15:06:18.904813 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.904562 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 27 15:06:18.905318 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.905299 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 15:06:18.905537 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.905520 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-scljk\"" Jul 27 15:06:18.998089 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.998035 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c27793e9-4c00-4698-98b5-23ac51f4e402-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:18.998282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.998126 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9vt\" (UniqueName: \"kubernetes.io/projected/c27793e9-4c00-4698-98b5-23ac51f4e402-kube-api-access-8k9vt\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:18.998282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.998161 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27793e9-4c00-4698-98b5-23ac51f4e402-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:18.998282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:18.998214 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c27793e9-4c00-4698-98b5-23ac51f4e402-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.099739 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.099166 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c27793e9-4c00-4698-98b5-23ac51f4e402-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.099739 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.099245 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9vt\" (UniqueName: \"kubernetes.io/projected/c27793e9-4c00-4698-98b5-23ac51f4e402-kube-api-access-8k9vt\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.099739 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.099279 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27793e9-4c00-4698-98b5-23ac51f4e402-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.099739 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.099333 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c27793e9-4c00-4698-98b5-23ac51f4e402-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.100353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.100324 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c27793e9-4c00-4698-98b5-23ac51f4e402-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.102887 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.102859 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c27793e9-4c00-4698-98b5-23ac51f4e402-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.103008 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.102988 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c27793e9-4c00-4698-98b5-23ac51f4e402-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.109414 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.109390 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9vt\" (UniqueName: \"kubernetes.io/projected/c27793e9-4c00-4698-98b5-23ac51f4e402-kube-api-access-8k9vt\") pod \"prometheus-operator-84d69f794d-zps4j\" (UID: \"c27793e9-4c00-4698-98b5-23ac51f4e402\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:19.204266 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:19.204231 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" Jul 27 15:06:22.884440 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:22.884359 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-zps4j"] Jul 27 15:06:22.887984 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:22.887955 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27793e9_4c00_4698_98b5_23ac51f4e402.slice/crio-426a804d380d61b455cb2c20e123d4777cb4fb2bc3af0d0462dacf7f4a798fdb WatchSource:0}: Error finding container 426a804d380d61b455cb2c20e123d4777cb4fb2bc3af0d0462dacf7f4a798fdb: Status 404 returned error can't find the container with id 426a804d380d61b455cb2c20e123d4777cb4fb2bc3af0d0462dacf7f4a798fdb Jul 27 15:06:23.734879 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:23.734845 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-52wmr" Jul 27 15:06:23.801652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:23.801609 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486f899-kgrps" event={"ID":"79aac610-cc21-4cc3-92c1-651f0b51b647","Type":"ContainerStarted","Data":"d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76"} Jul 27 15:06:23.805425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:23.805391 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" event={"ID":"c27793e9-4c00-4698-98b5-23ac51f4e402","Type":"ContainerStarted","Data":"426a804d380d61b455cb2c20e123d4777cb4fb2bc3af0d0462dacf7f4a798fdb"} Jul 27 15:06:23.808332 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:23.807994 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c4b55df4-c56w5" event={"ID":"564388b9-da07-4f7b-8847-a1eafc370934","Type":"ContainerStarted","Data":"1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48"} Jul 27 15:06:23.810837 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:23.810809 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvnsj" event={"ID":"29747912-51e0-4a07-9cdf-2c3436233245","Type":"ContainerStarted","Data":"94a5f882ac091046f50f67731d4e67ebc36d9bbf50a30e1a33d92586aac14615"} Jul 27 15:06:23.829024 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:23.828957 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5486f899-kgrps" podStartSLOduration=1.9818647550000001 podStartE2EDuration="9.828933391s" totalImagesPullingTime="7.847068636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:23.827405666 +0000 UTC m=+87.108755741" watchObservedRunningTime="2026-07-27 15:06:23.828933391 +0000 UTC m=+87.110283446" Jul 27 15:06:24.255082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:24.255027 2553 patch_prober.go:28] interesting pod/image-registry-5cff8c555f-7r4j5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 27 15:06:24.255528 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:24.255101 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" podUID="3efcde43-a6d0-40de-b34d-d46787bf75d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 15:06:24.829112 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:24.828536 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" event={"ID":"c27793e9-4c00-4698-98b5-23ac51f4e402","Type":"ContainerStarted","Data":"a067df47a9253c5e1d4af80a3c0e14e134ce41e716e630925e503f55a0868c1a"} Jul 27 15:06:25.103478 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.103367 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:25.103478 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.103438 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:25.105007 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.104970 2553 patch_prober.go:28] interesting pod/console-5486f899-kgrps container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused" start-of-body= Jul 27 15:06:25.105164 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.105040 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5486f899-kgrps" podUID="79aac610-cc21-4cc3-92c1-651f0b51b647" containerName="console" probeResult="failure" output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused" Jul 27 15:06:25.835781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.835741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" event={"ID":"c27793e9-4c00-4698-98b5-23ac51f4e402","Type":"ContainerStarted","Data":"6dc4e6ded55b1e4983ba2772ee83b97c666dd4b7e110e7386c7b0024e73e493f"} Jul 27 15:06:25.856519 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.856467 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75c4b55df4-c56w5" podStartSLOduration=4.344879552 podStartE2EDuration="9.856451792s" totalImagesPullingTime="5.51157224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:23.850805024 +0000 UTC m=+87.132155105" watchObservedRunningTime="2026-07-27 15:06:25.856451792 +0000 UTC m=+89.137801922" Jul 27 15:06:25.856679 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:25.856597 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-zps4j" podStartSLOduration=6.074314258 podStartE2EDuration="7.856586304s" totalImagesPullingTime="1.782272046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:25.854298913 +0000 UTC m=+89.135648987" watchObservedRunningTime="2026-07-27 15:06:25.856586304 +0000 UTC m=+89.137936378" Jul 27 15:06:27.020716 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:27.020680 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:27.021250 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:27.020743 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:27.022313 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:27.022285 2553 patch_prober.go:28] interesting pod/console-75c4b55df4-c56w5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.28:8443/health\": dial tcp 10.132.0.28:8443: connect: connection refused" start-of-body= Jul 27 15:06:27.022445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:27.022346 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-75c4b55df4-c56w5" podUID="564388b9-da07-4f7b-8847-a1eafc370934" containerName="console" probeResult="failure" output="Get \"https://10.132.0.28:8443/health\": dial tcp 10.132.0.28:8443: connect: connection refused" Jul 27 15:06:27.699689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:27.699653 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:06:28.232214 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.232168 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kccj2"] Jul 27 15:06:28.236320 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.236292 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.239642 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.239608 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vw8rk\"" Jul 27 15:06:28.239642 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.239627 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 15:06:28.240101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.239904 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 15:06:28.240616 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.240596 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 15:06:28.388515 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388342 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-accelerators-collector-config\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.388515 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388396 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-sys\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.388515 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388445 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-textfile\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.388515 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388485 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-metrics-client-ca\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.391076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-wtmp\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.391076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388860 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.391076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.388895 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-root\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.391076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.389075 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhc66\" (UniqueName: \"kubernetes.io/projected/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-kube-api-access-rhc66\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.391076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.389133 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-tls\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490165 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490087 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-wtmp\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490165 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490122 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490165 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490151 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-root\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490198 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhc66\" (UniqueName: \"kubernetes.io/projected/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-kube-api-access-rhc66\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490242 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-tls\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490275 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-wtmp\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490278 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-accelerators-collector-config\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490315 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-root\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490342 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-sys\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490391 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-textfile\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490680 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490428 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-metrics-client-ca\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490680 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490571 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-sys\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.490853 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.490828 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-textfile\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.491073 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.491033 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-metrics-client-ca\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.491329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.491306 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-accelerators-collector-config\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.493404 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.493378 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.493497 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.493403 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-node-exporter-tls\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.498267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.498248 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhc66\" (UniqueName: \"kubernetes.io/projected/eb57ea67-1c0c-4413-ba52-9f25c8e45e41-kube-api-access-rhc66\") pod \"node-exporter-kccj2\" (UID: \"eb57ea67-1c0c-4413-ba52-9f25c8e45e41\") " pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:28.551424 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:28.551388 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kccj2" Jul 27 15:06:33.010042 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:33.010004 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5486f899-kgrps"] Jul 27 15:06:34.645657 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:34.645624 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kcnfc" Jul 27 15:06:35.470374 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:06:35.469362 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57ea67_1c0c_4413_ba52_9f25c8e45e41.slice/crio-187b140c7d8b66e92c07d023e3c543379ce20fed0204e11119f933cb7a9d357a WatchSource:0}: Error finding container 187b140c7d8b66e92c07d023e3c543379ce20fed0204e11119f933cb7a9d357a: Status 404 returned error can't find the container with id 187b140c7d8b66e92c07d023e3c543379ce20fed0204e11119f933cb7a9d357a Jul 27 15:06:35.887206 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.887158 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvnsj" event={"ID":"29747912-51e0-4a07-9cdf-2c3436233245","Type":"ContainerStarted","Data":"832ed785dab7c5dedcd6db23bca862a0b7c447e9d35a494a4c5ba94d88c14be0"} Jul 27 15:06:35.889372 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.889342 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dr2qr" event={"ID":"cf76b9bf-77eb-4e6c-a803-ba91e44b5c57","Type":"ContainerStarted","Data":"2a849f56013425d5e09e01290fef7e9749fe8c76946ec4b523965196609f486b"} Jul 27 15:06:35.889523 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.889499 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:35.891238 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.891207 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kccj2" event={"ID":"eb57ea67-1c0c-4413-ba52-9f25c8e45e41","Type":"ContainerStarted","Data":"187b140c7d8b66e92c07d023e3c543379ce20fed0204e11119f933cb7a9d357a"} Jul 27 15:06:35.904754 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.904731 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-dr2qr" Jul 27 15:06:35.908330 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.908272 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dvnsj" podStartSLOduration=17.445618712 podStartE2EDuration="47.908254253s" totalImagesPullingTime="30.462635541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:05:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:35.906449836 +0000 UTC m=+99.187799921" watchObservedRunningTime="2026-07-27 15:06:35.908254253 +0000 UTC m=+99.189604326" Jul 27 15:06:35.926835 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:35.926783 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-dr2qr" podStartSLOduration=1.617795422 podStartE2EDuration="21.926763961s" totalImagesPullingTime="20.308968539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:35.925682471 +0000 UTC m=+99.207032546" watchObservedRunningTime="2026-07-27 15:06:35.926763961 +0000 UTC m=+99.208114035" Jul 27 15:06:37.026295 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:37.026259 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:37.031099 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:37.031075 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:06:37.904120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:37.904074 2553 generic.go:356] "Generic (PLEG): container finished" podID="eb57ea67-1c0c-4413-ba52-9f25c8e45e41" containerID="e21962a3136aee18c0c350efdb64dabc3272444be6243ad62a2bb976fe3adcbe" exitCode=0 Jul 27 15:06:37.904315 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:37.904168 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kccj2" event={"ID":"eb57ea67-1c0c-4413-ba52-9f25c8e45e41","Type":"ContainerDied","Data":"e21962a3136aee18c0c350efdb64dabc3272444be6243ad62a2bb976fe3adcbe"} Jul 27 15:06:38.912280 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:38.912241 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kccj2" event={"ID":"eb57ea67-1c0c-4413-ba52-9f25c8e45e41","Type":"ContainerStarted","Data":"08e225e6f8b8f844661ac694e70c9c11643829af09e6797ed05c0aacf1112217"} Jul 27 15:06:38.912740 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:38.912313 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kccj2" event={"ID":"eb57ea67-1c0c-4413-ba52-9f25c8e45e41","Type":"ContainerStarted","Data":"d7cafe8764b76afef9477e3fcadbd66b61302752a69ff8795bdf9ec0d2dc400b"} Jul 27 15:06:38.937281 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:38.937227 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kccj2" podStartSLOduration=9.220080084 podStartE2EDuration="10.937207854s" totalImagesPullingTime="1.614939587s" totalInitContainerRuntime="102.188183ms" isStatefulPod=true podCreationTimestamp="2026-07-27 15:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:06:38.936337531 +0000 UTC m=+102.217687605" watchObservedRunningTime="2026-07-27 15:06:38.937207854 +0000 UTC m=+102.218557929" Jul 27 15:06:42.935455 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:42.935415 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cff8c555f-7r4j5"] Jul 27 15:06:43.587242 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:43.587204 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c4b55df4-c56w5"] Jul 27 15:06:58.032448 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.032384 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5486f899-kgrps" podUID="79aac610-cc21-4cc3-92c1-651f0b51b647" containerName="console" containerID="cri-o://d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76" gracePeriod=15 Jul 27 15:06:58.326823 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.326795 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:58.365952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.365924 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-oauth-serving-cert\") pod \"79aac610-cc21-4cc3-92c1-651f0b51b647\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " Jul 27 15:06:58.366122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.365957 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-service-ca\") pod \"79aac610-cc21-4cc3-92c1-651f0b51b647\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " Jul 27 15:06:58.366122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366006 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-oauth-config\") pod \"79aac610-cc21-4cc3-92c1-651f0b51b647\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " Jul 27 15:06:58.366250 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366124 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-serving-cert\") pod \"79aac610-cc21-4cc3-92c1-651f0b51b647\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " Jul 27 15:06:58.366250 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366218 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khvq4\" (UniqueName: \"kubernetes.io/projected/79aac610-cc21-4cc3-92c1-651f0b51b647-kube-api-access-khvq4\") pod \"79aac610-cc21-4cc3-92c1-651f0b51b647\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " Jul 27 15:06:58.366343 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366245 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-console-config\") pod \"79aac610-cc21-4cc3-92c1-651f0b51b647\" (UID: \"79aac610-cc21-4cc3-92c1-651f0b51b647\") " Jul 27 15:06:58.366523 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366447 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-service-ca" (OuterVolumeSpecName: "service-ca") pod "79aac610-cc21-4cc3-92c1-651f0b51b647" (UID: "79aac610-cc21-4cc3-92c1-651f0b51b647"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:06:58.366523 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366454 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "79aac610-cc21-4cc3-92c1-651f0b51b647" (UID: "79aac610-cc21-4cc3-92c1-651f0b51b647"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:06:58.366700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366574 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-oauth-serving-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:06:58.366700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366594 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-service-ca\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:06:58.366799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.366779 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-console-config" (OuterVolumeSpecName: "console-config") pod "79aac610-cc21-4cc3-92c1-651f0b51b647" (UID: "79aac610-cc21-4cc3-92c1-651f0b51b647"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:06:58.368379 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.368355 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "79aac610-cc21-4cc3-92c1-651f0b51b647" (UID: "79aac610-cc21-4cc3-92c1-651f0b51b647"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:06:58.368591 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.368559 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "79aac610-cc21-4cc3-92c1-651f0b51b647" (UID: "79aac610-cc21-4cc3-92c1-651f0b51b647"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:06:58.368689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.368650 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79aac610-cc21-4cc3-92c1-651f0b51b647-kube-api-access-khvq4" (OuterVolumeSpecName: "kube-api-access-khvq4") pod "79aac610-cc21-4cc3-92c1-651f0b51b647" (UID: "79aac610-cc21-4cc3-92c1-651f0b51b647"). InnerVolumeSpecName "kube-api-access-khvq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:06:58.467932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.467892 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-oauth-config\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:06:58.467932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.467924 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79aac610-cc21-4cc3-92c1-651f0b51b647-console-serving-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:06:58.467932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.467934 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khvq4\" (UniqueName: \"kubernetes.io/projected/79aac610-cc21-4cc3-92c1-651f0b51b647-kube-api-access-khvq4\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:06:58.467932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.467943 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79aac610-cc21-4cc3-92c1-651f0b51b647-console-config\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:06:58.997595 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.997559 2553 generic.go:356] "Generic (PLEG): container finished" podID="79aac610-cc21-4cc3-92c1-651f0b51b647" containerID="d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76" exitCode=0 Jul 27 15:06:58.997781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.997602 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5486f899-kgrps" Jul 27 15:06:58.997781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.997644 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486f899-kgrps" event={"ID":"79aac610-cc21-4cc3-92c1-651f0b51b647","Type":"ContainerDied","Data":"d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76"} Jul 27 15:06:58.997781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.997722 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5486f899-kgrps" event={"ID":"79aac610-cc21-4cc3-92c1-651f0b51b647","Type":"ContainerDied","Data":"f60ffcc7a65089b2096b51045d444ed9bf97ceecf1c3167124f95ea8d0a7ba37"} Jul 27 15:06:58.997781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:58.997747 2553 scope.go:117] "RemoveContainer" containerID="d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76" Jul 27 15:06:59.007150 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:59.007135 2553 scope.go:117] "RemoveContainer" containerID="d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76" Jul 27 15:06:59.007440 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:06:59.007420 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76\": container with ID starting with d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76 not found: ID does not exist" containerID="d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76" Jul 27 15:06:59.007484 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:59.007450 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76"} err="failed to get container status \"d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76\": rpc error: code = NotFound desc = could not find container \"d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76\": container with ID starting with d4422cacc9d659939ed7653956f0aa5f7758ffd2e56c76d9a602101e570ece76 not found: ID does not exist" Jul 27 15:06:59.021194 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:59.021169 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5486f899-kgrps"] Jul 27 15:06:59.025018 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:59.024997 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5486f899-kgrps"] Jul 27 15:06:59.191125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:06:59.191092 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79aac610-cc21-4cc3-92c1-651f0b51b647" path="/var/lib/kubelet/pods/79aac610-cc21-4cc3-92c1-651f0b51b647/volumes" Jul 27 15:07:00.003449 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:00.003415 2553 generic.go:356] "Generic (PLEG): container finished" podID="c0d6fee3-17dd-465f-9239-cf9dcea137d7" containerID="eae7f8421c9c84346b6b9273eb185ac1bb3ffefa25ed50e2df110b34282bc576" exitCode=0 Jul 27 15:07:00.003652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:00.003492 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" event={"ID":"c0d6fee3-17dd-465f-9239-cf9dcea137d7","Type":"ContainerDied","Data":"eae7f8421c9c84346b6b9273eb185ac1bb3ffefa25ed50e2df110b34282bc576"} Jul 27 15:07:00.003857 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:00.003823 2553 scope.go:117] "RemoveContainer" containerID="eae7f8421c9c84346b6b9273eb185ac1bb3ffefa25ed50e2df110b34282bc576" Jul 27 15:07:01.010901 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:01.010865 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7" event={"ID":"c0d6fee3-17dd-465f-9239-cf9dcea137d7","Type":"ContainerStarted","Data":"72a2981eb921071970550bcab375ad25eb803a7a52f6600c7ad1db6530c6f3b6"} Jul 27 15:07:02.575152 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:02.575106 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" podUID="29ea7830-521c-449e-9c10-6ab80a99a36e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 27 15:07:07.958875 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:07.958836 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" podUID="3efcde43-a6d0-40de-b34d-d46787bf75d8" containerName="registry" containerID="cri-o://a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9" gracePeriod=30 Jul 27 15:07:08.229938 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.229873 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:07:08.351303 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351267 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-installation-pull-secrets\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351321 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efcde43-a6d0-40de-b34d-d46787bf75d8-ca-trust-extracted\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351358 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-bound-sa-token\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351376 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351400 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-trusted-ca\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351426 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-certificates\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351698 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351533 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82vcg\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-kube-api-access-82vcg\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351698 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351607 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-image-registry-private-configuration\") pod \"3efcde43-a6d0-40de-b34d-d46787bf75d8\" (UID: \"3efcde43-a6d0-40de-b34d-d46787bf75d8\") " Jul 27 15:07:08.351865 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351840 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:07:08.351865 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351853 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:07:08.352004 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351924 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-trusted-ca\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.352004 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.351944 2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-certificates\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.354131 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.354068 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:07:08.354131 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.354093 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-kube-api-access-82vcg" (OuterVolumeSpecName: "kube-api-access-82vcg") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "kube-api-access-82vcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:07:08.354321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.354158 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:07:08.354321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.354224 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:07:08.354321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.354246 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:07:08.360142 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.360122 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3efcde43-a6d0-40de-b34d-d46787bf75d8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3efcde43-a6d0-40de-b34d-d46787bf75d8" (UID: "3efcde43-a6d0-40de-b34d-d46787bf75d8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:07:08.453263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.453218 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vcg\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-kube-api-access-82vcg\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.453263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.453256 2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-image-registry-private-configuration\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.453263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.453267 2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3efcde43-a6d0-40de-b34d-d46787bf75d8-installation-pull-secrets\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.453263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.453275 2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3efcde43-a6d0-40de-b34d-d46787bf75d8-ca-trust-extracted\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.453517 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.453285 2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-bound-sa-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.453517 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.453294 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3efcde43-a6d0-40de-b34d-d46787bf75d8-registry-tls\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:08.611148 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.611089 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75c4b55df4-c56w5" podUID="564388b9-da07-4f7b-8847-a1eafc370934" containerName="console" containerID="cri-o://1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48" gracePeriod=15 Jul 27 15:07:08.871737 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.871674 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:07:08.957247 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957211 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-console-config\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957247 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957251 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhrx4\" (UniqueName: \"kubernetes.io/projected/564388b9-da07-4f7b-8847-a1eafc370934-kube-api-access-mhrx4\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957479 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957276 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-oauth-config\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957479 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957319 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-serving-cert\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957479 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957352 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-trusted-ca-bundle\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957479 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957382 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-oauth-serving-cert\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957479 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957414 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-service-ca\") pod \"564388b9-da07-4f7b-8847-a1eafc370934\" (UID: \"564388b9-da07-4f7b-8847-a1eafc370934\") " Jul 27 15:07:08.957830 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957794 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-console-config" (OuterVolumeSpecName: "console-config") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:07:08.957932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957864 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:07:08.957932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957898 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:07:08.958024 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.957940 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-service-ca" (OuterVolumeSpecName: "service-ca") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:07:08.959507 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.959475 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:07:08.959777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.959501 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:07:08.959777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:08.959522 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564388b9-da07-4f7b-8847-a1eafc370934-kube-api-access-mhrx4" (OuterVolumeSpecName: "kube-api-access-mhrx4") pod "564388b9-da07-4f7b-8847-a1eafc370934" (UID: "564388b9-da07-4f7b-8847-a1eafc370934"). InnerVolumeSpecName "kube-api-access-mhrx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:07:09.046402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.046369 2553 generic.go:356] "Generic (PLEG): container finished" podID="3efcde43-a6d0-40de-b34d-d46787bf75d8" containerID="a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9" exitCode=0 Jul 27 15:07:09.046573 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.046416 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" Jul 27 15:07:09.046573 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.046465 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" event={"ID":"3efcde43-a6d0-40de-b34d-d46787bf75d8","Type":"ContainerDied","Data":"a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9"} Jul 27 15:07:09.046573 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.046552 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cff8c555f-7r4j5" event={"ID":"3efcde43-a6d0-40de-b34d-d46787bf75d8","Type":"ContainerDied","Data":"ced1e6e1758636c20307b8cacdb63fdce1cd075edeb21678329d5a292d3a2421"} Jul 27 15:07:09.046736 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.046582 2553 scope.go:117] "RemoveContainer" containerID="a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9" Jul 27 15:07:09.047890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.047802 2553 generic.go:356] "Generic (PLEG): container finished" podID="564388b9-da07-4f7b-8847-a1eafc370934" containerID="1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48" exitCode=0 Jul 27 15:07:09.047890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.047832 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c4b55df4-c56w5" event={"ID":"564388b9-da07-4f7b-8847-a1eafc370934","Type":"ContainerDied","Data":"1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48"} Jul 27 15:07:09.047890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.047858 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75c4b55df4-c56w5" Jul 27 15:07:09.047890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.047875 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75c4b55df4-c56w5" event={"ID":"564388b9-da07-4f7b-8847-a1eafc370934","Type":"ContainerDied","Data":"2f3c15a5ffb60a45bb568ad03b00e127562bffb1f7599e0b41955b784a6b249f"} Jul 27 15:07:09.054801 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.054770 2553 scope.go:117] "RemoveContainer" containerID="a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9" Jul 27 15:07:09.055112 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:07:09.055085 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9\": container with ID starting with a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9 not found: ID does not exist" containerID="a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9" Jul 27 15:07:09.055181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.055123 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9"} err="failed to get container status \"a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9\": rpc error: code = NotFound desc = could not find container \"a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9\": container with ID starting with a8eaf738066044e4cded64cd291509f9e14014f7e6ab9c50609bcd9c9cc802f9 not found: ID does not exist" Jul 27 15:07:09.055181 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.055148 2553 scope.go:117] "RemoveContainer" containerID="1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.057975 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-serving-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.057993 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-trusted-ca-bundle\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.058004 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-oauth-serving-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.058016 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-service-ca\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.058026 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/564388b9-da07-4f7b-8847-a1eafc370934-console-config\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.058034 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrx4\" (UniqueName: \"kubernetes.io/projected/564388b9-da07-4f7b-8847-a1eafc370934-kube-api-access-mhrx4\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.058088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.058042 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/564388b9-da07-4f7b-8847-a1eafc370934-console-oauth-config\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:07:09.063379 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.063360 2553 scope.go:117] "RemoveContainer" containerID="1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48" Jul 27 15:07:09.063606 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:07:09.063589 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48\": container with ID starting with 1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48 not found: ID does not exist" containerID="1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48" Jul 27 15:07:09.063653 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.063612 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48"} err="failed to get container status \"1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48\": rpc error: code = NotFound desc = could not find container \"1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48\": container with ID starting with 1dfe9babbb78ad6ea10d67db319b1f7555c851702d8dbd0e23a87109440e9f48 not found: ID does not exist" Jul 27 15:07:09.075814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.075785 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75c4b55df4-c56w5"] Jul 27 15:07:09.078121 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.078100 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75c4b55df4-c56w5"] Jul 27 15:07:09.089182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.089162 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cff8c555f-7r4j5"] Jul 27 15:07:09.100721 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.100703 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cff8c555f-7r4j5"] Jul 27 15:07:09.191846 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.191767 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3efcde43-a6d0-40de-b34d-d46787bf75d8" path="/var/lib/kubelet/pods/3efcde43-a6d0-40de-b34d-d46787bf75d8/volumes" Jul 27 15:07:09.192167 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:09.192155 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564388b9-da07-4f7b-8847-a1eafc370934" path="/var/lib/kubelet/pods/564388b9-da07-4f7b-8847-a1eafc370934/volumes" Jul 27 15:07:12.575648 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:12.575603 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" podUID="29ea7830-521c-449e-9c10-6ab80a99a36e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 27 15:07:13.072469 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:13.072438 2553 generic.go:356] "Generic (PLEG): container finished" podID="4a86eae4-88cc-402b-8003-570b4b1a7aae" containerID="d34082bda52bda77984bcde5ee3fb6fdfd8fbc369a573b03d8ce8ee9771f2800" exitCode=0 Jul 27 15:07:13.072617 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:13.072519 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2tn84" event={"ID":"4a86eae4-88cc-402b-8003-570b4b1a7aae","Type":"ContainerDied","Data":"d34082bda52bda77984bcde5ee3fb6fdfd8fbc369a573b03d8ce8ee9771f2800"} Jul 27 15:07:13.072909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:13.072892 2553 scope.go:117] "RemoveContainer" containerID="d34082bda52bda77984bcde5ee3fb6fdfd8fbc369a573b03d8ce8ee9771f2800" Jul 27 15:07:14.080610 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:14.080571 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2tn84" event={"ID":"4a86eae4-88cc-402b-8003-570b4b1a7aae","Type":"ContainerStarted","Data":"1019e88e14d57755e318dd6d98ce67fb3231b0ad5be86229243db21ee9a08e26"} Jul 27 15:07:19.103009 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:19.102915 2553 generic.go:356] "Generic (PLEG): container finished" podID="10ed8da3-6018-4f3b-ba4b-31537e008b12" containerID="94b645c8f764950964707a9289d9636a8de2fdd1894226bc29a6d75d6551fc45" exitCode=0 Jul 27 15:07:19.103009 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:19.102991 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" event={"ID":"10ed8da3-6018-4f3b-ba4b-31537e008b12","Type":"ContainerDied","Data":"94b645c8f764950964707a9289d9636a8de2fdd1894226bc29a6d75d6551fc45"} Jul 27 15:07:19.103465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:19.103364 2553 scope.go:117] "RemoveContainer" containerID="94b645c8f764950964707a9289d9636a8de2fdd1894226bc29a6d75d6551fc45" Jul 27 15:07:20.109470 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:20.109432 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ztql" event={"ID":"10ed8da3-6018-4f3b-ba4b-31537e008b12","Type":"ContainerStarted","Data":"a623a56725c57a6ffc934496a1dd1cde84fbc0ad81f371221968b1d58a504c41"} Jul 27 15:07:22.575577 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:22.575533 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" podUID="29ea7830-521c-449e-9c10-6ab80a99a36e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 27 15:07:22.575970 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:22.575604 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" Jul 27 15:07:22.576090 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:22.576045 2553 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f429c48a7a9fbbe1ed617a5101768b7a06056cd41860675376e912c3e85a1fd9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 27 15:07:22.576134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:22.576112 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" podUID="29ea7830-521c-449e-9c10-6ab80a99a36e" containerName="service-proxy" containerID="cri-o://f429c48a7a9fbbe1ed617a5101768b7a06056cd41860675376e912c3e85a1fd9" gracePeriod=30 Jul 27 15:07:23.129224 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:23.129177 2553 generic.go:356] "Generic (PLEG): container finished" podID="29ea7830-521c-449e-9c10-6ab80a99a36e" containerID="f429c48a7a9fbbe1ed617a5101768b7a06056cd41860675376e912c3e85a1fd9" exitCode=2 Jul 27 15:07:23.129394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:23.129233 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" event={"ID":"29ea7830-521c-449e-9c10-6ab80a99a36e","Type":"ContainerDied","Data":"f429c48a7a9fbbe1ed617a5101768b7a06056cd41860675376e912c3e85a1fd9"} Jul 27 15:07:23.129394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:07:23.129282 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86645dd754-6c6q2" event={"ID":"29ea7830-521c-449e-9c10-6ab80a99a36e","Type":"ContainerStarted","Data":"02d7daa171b2810430d47cb375c1ba624428dd3ac18d099e9f91283db9b43e27"} Jul 27 15:09:57.127870 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:09:57.127835 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:09:57.137513 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:09:57.137483 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:09:57.140300 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:09:57.140280 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 15:11:44.937653 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937619 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7"] Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937931 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564388b9-da07-4f7b-8847-a1eafc370934" containerName="console" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937942 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="564388b9-da07-4f7b-8847-a1eafc370934" containerName="console" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937958 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79aac610-cc21-4cc3-92c1-651f0b51b647" containerName="console" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937964 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="79aac610-cc21-4cc3-92c1-651f0b51b647" containerName="console" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937981 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efcde43-a6d0-40de-b34d-d46787bf75d8" containerName="registry" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.937989 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efcde43-a6d0-40de-b34d-d46787bf75d8" containerName="registry" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.938042 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="79aac610-cc21-4cc3-92c1-651f0b51b647" containerName="console" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.938063 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="564388b9-da07-4f7b-8847-a1eafc370934" containerName="console" Jul 27 15:11:44.938154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.938072 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efcde43-a6d0-40de-b34d-d46787bf75d8" containerName="registry" Jul 27 15:11:44.940098 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.940082 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:44.951184 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.951163 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 15:11:44.952539 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.952517 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 15:11:44.952653 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.952620 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srgx6\"" Jul 27 15:11:44.952705 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.952649 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 15:11:44.953275 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.953259 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 15:11:44.974889 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:44.971607 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7"] Jul 27 15:11:45.024456 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.024422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d58091d9-1026-4ca3-95ea-e289b01f6cca-tmp\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.024631 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.024479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d58091d9-1026-4ca3-95ea-e289b01f6cca-webhook-cert\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.024631 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.024504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d58091d9-1026-4ca3-95ea-e289b01f6cca-manifests\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.024631 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.024556 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d58091d9-1026-4ca3-95ea-e289b01f6cca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.024631 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.024589 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9lr\" (UniqueName: \"kubernetes.io/projected/d58091d9-1026-4ca3-95ea-e289b01f6cca-kube-api-access-cz9lr\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.124972 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.124936 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d58091d9-1026-4ca3-95ea-e289b01f6cca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.124972 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.124973 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9lr\" (UniqueName: \"kubernetes.io/projected/d58091d9-1026-4ca3-95ea-e289b01f6cca-kube-api-access-cz9lr\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.125235 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.125014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d58091d9-1026-4ca3-95ea-e289b01f6cca-tmp\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.125235 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.125086 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d58091d9-1026-4ca3-95ea-e289b01f6cca-webhook-cert\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.125235 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.125125 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d58091d9-1026-4ca3-95ea-e289b01f6cca-manifests\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.125539 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.125506 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d58091d9-1026-4ca3-95ea-e289b01f6cca-manifests\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.127380 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.127357 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d58091d9-1026-4ca3-95ea-e289b01f6cca-tmp\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.127742 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.127719 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d58091d9-1026-4ca3-95ea-e289b01f6cca-webhook-cert\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.127809 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.127719 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d58091d9-1026-4ca3-95ea-e289b01f6cca-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.136584 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.136563 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9lr\" (UniqueName: \"kubernetes.io/projected/d58091d9-1026-4ca3-95ea-e289b01f6cca-kube-api-access-cz9lr\") pod \"opendatahub-operator-controller-manager-78959f89bb-d8cl7\" (UID: \"d58091d9-1026-4ca3-95ea-e289b01f6cca\") " pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.250969 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.250886 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:45.393652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.393620 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7"] Jul 27 15:11:45.397413 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:11:45.397378 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58091d9_1026_4ca3_95ea_e289b01f6cca.slice/crio-fc6d073aa6aadbaa4b7f8bd402230eb1f60f74d7399557d3036c564349550300 WatchSource:0}: Error finding container fc6d073aa6aadbaa4b7f8bd402230eb1f60f74d7399557d3036c564349550300: Status 404 returned error can't find the container with id fc6d073aa6aadbaa4b7f8bd402230eb1f60f74d7399557d3036c564349550300 Jul 27 15:11:45.399180 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:45.399161 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 15:11:46.196712 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:46.196679 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" event={"ID":"d58091d9-1026-4ca3-95ea-e289b01f6cca","Type":"ContainerStarted","Data":"fc6d073aa6aadbaa4b7f8bd402230eb1f60f74d7399557d3036c564349550300"} Jul 27 15:11:49.213037 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:49.212994 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" event={"ID":"d58091d9-1026-4ca3-95ea-e289b01f6cca","Type":"ContainerStarted","Data":"14e3236b5f80c90c26d9e69ac2bcf0ef9f304dae0dec3c8d8b2a57fa3fd8bed4"} Jul 27 15:11:50.218974 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:50.218941 2553 generic.go:356] "Generic (PLEG): container finished" podID="d58091d9-1026-4ca3-95ea-e289b01f6cca" containerID="14e3236b5f80c90c26d9e69ac2bcf0ef9f304dae0dec3c8d8b2a57fa3fd8bed4" exitCode=0 Jul 27 15:11:50.219384 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:50.218998 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" event={"ID":"d58091d9-1026-4ca3-95ea-e289b01f6cca","Type":"ContainerDied","Data":"14e3236b5f80c90c26d9e69ac2bcf0ef9f304dae0dec3c8d8b2a57fa3fd8bed4"} Jul 27 15:11:51.226405 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:51.226371 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" event={"ID":"d58091d9-1026-4ca3-95ea-e289b01f6cca","Type":"ContainerStarted","Data":"22ee9cc0502c84cd073797269e63f1f2efde0a3b1b841f9bccc604532f738f7a"} Jul 27 15:11:51.226888 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:51.226478 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:11:51.247122 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:11:51.247017 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" podStartSLOduration=2.607060929 podStartE2EDuration="7.246996698s" totalImagesPullingTime="3.585271005s" totalInitContainerRuntime="1.054664764s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:11:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:11:51.245291503 +0000 UTC m=+414.526641576" watchObservedRunningTime="2026-07-27 15:11:51.246996698 +0000 UTC m=+414.528346772" Jul 27 15:12:02.233025 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:02.232991 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-78959f89bb-d8cl7" Jul 27 15:12:14.597536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.597500 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc"] Jul 27 15:12:14.611643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.611619 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.614824 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.614800 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 27 15:12:14.614974 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.614800 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xgplx\"" Jul 27 15:12:14.615066 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.615012 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 15:12:14.615370 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.615355 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 27 15:12:14.615519 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.615508 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 15:12:14.615600 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.615583 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 27 15:12:14.640066 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.640025 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc"] Jul 27 15:12:14.677296 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.677257 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqqf\" (UniqueName: \"kubernetes.io/projected/707cc393-0db6-4cf3-beb7-d3316197803a-kube-api-access-hzqqf\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.677544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.677321 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/707cc393-0db6-4cf3-beb7-d3316197803a-manager-config\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.677544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.677403 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/707cc393-0db6-4cf3-beb7-d3316197803a-cert\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.677544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.677437 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/707cc393-0db6-4cf3-beb7-d3316197803a-metrics-cert\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.777833 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.777796 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/707cc393-0db6-4cf3-beb7-d3316197803a-manager-config\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.777833 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.777841 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/707cc393-0db6-4cf3-beb7-d3316197803a-cert\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.778130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.777857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/707cc393-0db6-4cf3-beb7-d3316197803a-metrics-cert\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.778130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.777888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqqf\" (UniqueName: \"kubernetes.io/projected/707cc393-0db6-4cf3-beb7-d3316197803a-kube-api-access-hzqqf\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.778560 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.778535 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/707cc393-0db6-4cf3-beb7-d3316197803a-manager-config\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.780403 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.780384 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/707cc393-0db6-4cf3-beb7-d3316197803a-metrics-cert\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.780462 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.780431 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/707cc393-0db6-4cf3-beb7-d3316197803a-cert\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.786655 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.786633 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqqf\" (UniqueName: \"kubernetes.io/projected/707cc393-0db6-4cf3-beb7-d3316197803a-kube-api-access-hzqqf\") pod \"lws-controller-manager-964ddd7d8-cmcvc\" (UID: \"707cc393-0db6-4cf3-beb7-d3316197803a\") " pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:14.920641 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:14.920604 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:15.081337 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:15.080278 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc"] Jul 27 15:12:15.087862 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:12:15.087824 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707cc393_0db6_4cf3_beb7_d3316197803a.slice/crio-8a2bcbfe5ca02517e97c135ea80dea8afe77a053b96a2a9f365f6bab35b0c4ee WatchSource:0}: Error finding container 8a2bcbfe5ca02517e97c135ea80dea8afe77a053b96a2a9f365f6bab35b0c4ee: Status 404 returned error can't find the container with id 8a2bcbfe5ca02517e97c135ea80dea8afe77a053b96a2a9f365f6bab35b0c4ee Jul 27 15:12:15.336703 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:15.336620 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" event={"ID":"707cc393-0db6-4cf3-beb7-d3316197803a","Type":"ContainerStarted","Data":"8a2bcbfe5ca02517e97c135ea80dea8afe77a053b96a2a9f365f6bab35b0c4ee"} Jul 27 15:12:17.689324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.689286 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n85mq"] Jul 27 15:12:17.692851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.692819 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:17.696806 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.696776 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-85d9f\"" Jul 27 15:12:17.696931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.696871 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 15:12:17.711325 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.710867 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n85mq"] Jul 27 15:12:17.726092 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.726035 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8"] Jul 27 15:12:17.732309 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.732280 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.735309 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.735004 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-g766l\"" Jul 27 15:12:17.735309 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.735248 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 15:12:17.752154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.752121 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8"] Jul 27 15:12:17.805764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.805727 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bec687-43ed-459b-b581-725638dfc717-cert\") pod \"kserve-controller-manager-d6f4b65b8-n85mq\" (UID: \"19bec687-43ed-459b-b581-725638dfc717\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:17.805764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.805772 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/601a372d-2af9-4d2b-83c1-bf0d1d6bb845-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66pd8\" (UID: \"601a372d-2af9-4d2b-83c1-bf0d1d6bb845\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.806031 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.805909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm6h7\" (UniqueName: \"kubernetes.io/projected/19bec687-43ed-459b-b581-725638dfc717-kube-api-access-rm6h7\") pod \"kserve-controller-manager-d6f4b65b8-n85mq\" (UID: \"19bec687-43ed-459b-b581-725638dfc717\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:17.806031 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.805972 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdf2\" (UniqueName: \"kubernetes.io/projected/601a372d-2af9-4d2b-83c1-bf0d1d6bb845-kube-api-access-jtdf2\") pod \"llmisvc-controller-manager-777c6bd7d5-66pd8\" (UID: \"601a372d-2af9-4d2b-83c1-bf0d1d6bb845\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.907312 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.907266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdf2\" (UniqueName: \"kubernetes.io/projected/601a372d-2af9-4d2b-83c1-bf0d1d6bb845-kube-api-access-jtdf2\") pod \"llmisvc-controller-manager-777c6bd7d5-66pd8\" (UID: \"601a372d-2af9-4d2b-83c1-bf0d1d6bb845\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.907498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.907350 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bec687-43ed-459b-b581-725638dfc717-cert\") pod \"kserve-controller-manager-d6f4b65b8-n85mq\" (UID: \"19bec687-43ed-459b-b581-725638dfc717\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:17.907498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.907378 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/601a372d-2af9-4d2b-83c1-bf0d1d6bb845-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66pd8\" (UID: \"601a372d-2af9-4d2b-83c1-bf0d1d6bb845\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.907498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.907444 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm6h7\" (UniqueName: \"kubernetes.io/projected/19bec687-43ed-459b-b581-725638dfc717-kube-api-access-rm6h7\") pod \"kserve-controller-manager-d6f4b65b8-n85mq\" (UID: \"19bec687-43ed-459b-b581-725638dfc717\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:17.910314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.910286 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bec687-43ed-459b-b581-725638dfc717-cert\") pod \"kserve-controller-manager-d6f4b65b8-n85mq\" (UID: \"19bec687-43ed-459b-b581-725638dfc717\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:17.910433 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.910370 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/601a372d-2af9-4d2b-83c1-bf0d1d6bb845-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66pd8\" (UID: \"601a372d-2af9-4d2b-83c1-bf0d1d6bb845\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.923760 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.923734 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdf2\" (UniqueName: \"kubernetes.io/projected/601a372d-2af9-4d2b-83c1-bf0d1d6bb845-kube-api-access-jtdf2\") pod \"llmisvc-controller-manager-777c6bd7d5-66pd8\" (UID: \"601a372d-2af9-4d2b-83c1-bf0d1d6bb845\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:17.924005 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:17.923981 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm6h7\" (UniqueName: \"kubernetes.io/projected/19bec687-43ed-459b-b581-725638dfc717-kube-api-access-rm6h7\") pod \"kserve-controller-manager-d6f4b65b8-n85mq\" (UID: \"19bec687-43ed-459b-b581-725638dfc717\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:18.027874 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.027788 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:18.046213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.046181 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:18.238704 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.238673 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n85mq"] Jul 27 15:12:18.239827 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:12:18.239802 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bec687_43ed_459b_b581_725638dfc717.slice/crio-24453b92e8bfecc483df4eee4f9475f779f115760581d8c4c8663772f1b747b1 WatchSource:0}: Error finding container 24453b92e8bfecc483df4eee4f9475f779f115760581d8c4c8663772f1b747b1: Status 404 returned error can't find the container with id 24453b92e8bfecc483df4eee4f9475f779f115760581d8c4c8663772f1b747b1 Jul 27 15:12:18.253270 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.253242 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8"] Jul 27 15:12:18.253888 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:12:18.253865 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod601a372d_2af9_4d2b_83c1_bf0d1d6bb845.slice/crio-a89f355c6d78e3df32d7e60524ffba677b5fb44680fbd31dec99f133ded7fef5 WatchSource:0}: Error finding container a89f355c6d78e3df32d7e60524ffba677b5fb44680fbd31dec99f133ded7fef5: Status 404 returned error can't find the container with id a89f355c6d78e3df32d7e60524ffba677b5fb44680fbd31dec99f133ded7fef5 Jul 27 15:12:18.354391 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.354362 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" event={"ID":"707cc393-0db6-4cf3-beb7-d3316197803a","Type":"ContainerStarted","Data":"23925af49db5049054fe4232424ce0dab534797aab0884299d9b27d8c426beaf"} Jul 27 15:12:18.354599 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.354460 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:18.355561 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.355538 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" event={"ID":"601a372d-2af9-4d2b-83c1-bf0d1d6bb845","Type":"ContainerStarted","Data":"a89f355c6d78e3df32d7e60524ffba677b5fb44680fbd31dec99f133ded7fef5"} Jul 27 15:12:18.356637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.356617 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" event={"ID":"19bec687-43ed-459b-b581-725638dfc717","Type":"ContainerStarted","Data":"24453b92e8bfecc483df4eee4f9475f779f115760581d8c4c8663772f1b747b1"} Jul 27 15:12:18.372439 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:18.371837 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" podStartSLOduration=1.321210595 podStartE2EDuration="4.371821605s" totalImagesPullingTime="3.05061101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:12:18.370910244 +0000 UTC m=+441.652260318" watchObservedRunningTime="2026-07-27 15:12:18.371821605 +0000 UTC m=+441.653171683" Jul 27 15:12:19.395458 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.395374 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb"] Jul 27 15:12:19.399478 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.399456 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.402511 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.402486 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xnwxf\"" Jul 27 15:12:19.402939 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.402920 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 15:12:19.414851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.414828 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb"] Jul 27 15:12:19.518449 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518415 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.518638 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518605 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.518709 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518665 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.518709 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518699 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.518802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518720 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.518802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518780 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtq2\" (UniqueName: \"kubernetes.io/projected/e0e6ad28-e2a6-4784-81fa-f14c37386753-kube-api-access-rxtq2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.518921 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.518891 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.519043 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.519022 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.519394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.519083 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e6ad28-e2a6-4784-81fa-f14c37386753-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620305 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620333 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620355 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620384 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtq2\" (UniqueName: \"kubernetes.io/projected/e0e6ad28-e2a6-4784-81fa-f14c37386753-kube-api-access-rxtq2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620420 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620480 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e6ad28-e2a6-4784-81fa-f14c37386753-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.620545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.621438 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.621984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.621936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.622932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.622202 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.622932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.622647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e6ad28-e2a6-4784-81fa-f14c37386753-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.622932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.622802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.624902 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.624873 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.625891 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.625863 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.633506 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.633479 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtq2\" (UniqueName: \"kubernetes.io/projected/e0e6ad28-e2a6-4784-81fa-f14c37386753-kube-api-access-rxtq2\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.633626 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.633569 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e6ad28-e2a6-4784-81fa-f14c37386753-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb\" (UID: \"e0e6ad28-e2a6-4784-81fa-f14c37386753\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.715208 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.715106 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:19.894764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:19.894739 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb"] Jul 27 15:12:19.896246 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:12:19.896216 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e6ad28_e2a6_4784_81fa_f14c37386753.slice/crio-f7513f52ab111477dfe890096d7209db39d722c27e9bede202770872be32fac1 WatchSource:0}: Error finding container f7513f52ab111477dfe890096d7209db39d722c27e9bede202770872be32fac1: Status 404 returned error can't find the container with id f7513f52ab111477dfe890096d7209db39d722c27e9bede202770872be32fac1 Jul 27 15:12:20.371523 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:20.371490 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" event={"ID":"e0e6ad28-e2a6-4784-81fa-f14c37386753","Type":"ContainerStarted","Data":"f7513f52ab111477dfe890096d7209db39d722c27e9bede202770872be32fac1"} Jul 27 15:12:23.002179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:23.002135 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:12:23.002530 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:23.002227 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:12:23.002530 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:23.002256 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:12:24.392844 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.392803 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" event={"ID":"601a372d-2af9-4d2b-83c1-bf0d1d6bb845","Type":"ContainerStarted","Data":"2a5580788db0fe4f16ddcd7f1177a3ef28db9329d294862f7894dac7b4ac7231"} Jul 27 15:12:24.393316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.392966 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:24.394329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.394308 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" event={"ID":"19bec687-43ed-459b-b581-725638dfc717","Type":"ContainerStarted","Data":"404c07ad445aa19944fd5bfb87651c0f94d84fbd2d3e236120f2d4569c650e88"} Jul 27 15:12:24.394434 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.394350 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:12:24.395765 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.395735 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" event={"ID":"e0e6ad28-e2a6-4784-81fa-f14c37386753","Type":"ContainerStarted","Data":"186b974b679d92771870d56fa5ee7ab7b5dd967bf3a6e968fca11fbddb6b6aee"} Jul 27 15:12:24.409779 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.409732 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" podStartSLOduration=1.941503625 podStartE2EDuration="7.409719427s" totalImagesPullingTime="5.468215802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:12:24.408372325 +0000 UTC m=+447.689722397" watchObservedRunningTime="2026-07-27 15:12:24.409719427 +0000 UTC m=+447.691069500" Jul 27 15:12:24.430863 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.430814 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" podStartSLOduration=2.32801224 podStartE2EDuration="5.430799585s" totalImagesPullingTime="3.102787345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:12:24.429012317 +0000 UTC m=+447.710362414" watchObservedRunningTime="2026-07-27 15:12:24.430799585 +0000 UTC m=+447.712149659" Jul 27 15:12:24.443540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.443487 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" podStartSLOduration=2.808491109 podStartE2EDuration="7.443472346s" totalImagesPullingTime="4.634981237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:12:24.443277323 +0000 UTC m=+447.724627420" watchObservedRunningTime="2026-07-27 15:12:24.443472346 +0000 UTC m=+447.724822419" Jul 27 15:12:24.715820 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.715742 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:24.720398 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:24.720366 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:25.401391 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:25.401355 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:25.402448 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:25.402426 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb" Jul 27 15:12:29.368303 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:29.368272 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-964ddd7d8-cmcvc" Jul 27 15:12:55.403851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:55.403780 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66pd8" Jul 27 15:12:55.406877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:12:55.406852 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:14:15.008700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.008663 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p"] Jul 27 15:14:15.011360 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.011334 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.014576 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.014549 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 15:14:15.014767 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.014744 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 15:14:15.015425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.015405 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rqmqs\"" Jul 27 15:14:15.023806 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.023784 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p"] Jul 27 15:14:15.118893 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.118855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qffcw\" (UniqueName: \"kubernetes.io/projected/ee0a4040-329f-4263-8e99-018fef9065e3-kube-api-access-qffcw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhd9p\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.118893 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.118894 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0a4040-329f-4263-8e99-018fef9065e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhd9p\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.220348 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.220315 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qffcw\" (UniqueName: \"kubernetes.io/projected/ee0a4040-329f-4263-8e99-018fef9065e3-kube-api-access-qffcw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhd9p\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.220348 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.220348 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0a4040-329f-4263-8e99-018fef9065e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhd9p\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.220702 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.220686 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0a4040-329f-4263-8e99-018fef9065e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhd9p\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.228666 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.228634 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qffcw\" (UniqueName: \"kubernetes.io/projected/ee0a4040-329f-4263-8e99-018fef9065e3-kube-api-access-qffcw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dhd9p\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.322450 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.322377 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:15.458728 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.458703 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p"] Jul 27 15:14:15.460819 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:14:15.460792 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0a4040_329f_4263_8e99_018fef9065e3.slice/crio-f4cbc469459682aa4830925edc0222d33d4bfb749e6cdaa8663d91daa2a1ed8d WatchSource:0}: Error finding container f4cbc469459682aa4830925edc0222d33d4bfb749e6cdaa8663d91daa2a1ed8d: Status 404 returned error can't find the container with id f4cbc469459682aa4830925edc0222d33d4bfb749e6cdaa8663d91daa2a1ed8d Jul 27 15:14:15.901427 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:15.901390 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" event={"ID":"ee0a4040-329f-4263-8e99-018fef9065e3","Type":"ContainerStarted","Data":"f4cbc469459682aa4830925edc0222d33d4bfb749e6cdaa8663d91daa2a1ed8d"} Jul 27 15:14:19.927373 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:19.927330 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" event={"ID":"ee0a4040-329f-4263-8e99-018fef9065e3","Type":"ContainerStarted","Data":"d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512"} Jul 27 15:14:19.927763 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:19.927443 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:19.947872 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:19.947821 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" podStartSLOduration=1.663529557 podStartE2EDuration="5.947804803s" totalImagesPullingTime="4.284275246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:14:19.945963322 +0000 UTC m=+563.227313409" watchObservedRunningTime="2026-07-27 15:14:19.947804803 +0000 UTC m=+563.229154877" Jul 27 15:14:27.403737 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.403699 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p"] Jul 27 15:14:27.404248 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.403901 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" podUID="ee0a4040-329f-4263-8e99-018fef9065e3" containerName="manager" containerID="cri-o://d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512" gracePeriod=10 Jul 27 15:14:27.406124 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.406100 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:27.659553 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.659492 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:27.832803 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.832762 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0a4040-329f-4263-8e99-018fef9065e3-extensions-socket-volume\") pod \"ee0a4040-329f-4263-8e99-018fef9065e3\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " Jul 27 15:14:27.832967 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.832910 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qffcw\" (UniqueName: \"kubernetes.io/projected/ee0a4040-329f-4263-8e99-018fef9065e3-kube-api-access-qffcw\") pod \"ee0a4040-329f-4263-8e99-018fef9065e3\" (UID: \"ee0a4040-329f-4263-8e99-018fef9065e3\") " Jul 27 15:14:27.833336 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.833313 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee0a4040-329f-4263-8e99-018fef9065e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee0a4040-329f-4263-8e99-018fef9065e3" (UID: "ee0a4040-329f-4263-8e99-018fef9065e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:14:27.835041 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.835016 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee0a4040-329f-4263-8e99-018fef9065e3-kube-api-access-qffcw" (OuterVolumeSpecName: "kube-api-access-qffcw") pod "ee0a4040-329f-4263-8e99-018fef9065e3" (UID: "ee0a4040-329f-4263-8e99-018fef9065e3"). InnerVolumeSpecName "kube-api-access-qffcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:14:27.933640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.933539 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee0a4040-329f-4263-8e99-018fef9065e3-extensions-socket-volume\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:14:27.933640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.933581 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qffcw\" (UniqueName: \"kubernetes.io/projected/ee0a4040-329f-4263-8e99-018fef9065e3-kube-api-access-qffcw\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:14:27.964594 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.964561 2553 generic.go:356] "Generic (PLEG): container finished" podID="ee0a4040-329f-4263-8e99-018fef9065e3" containerID="d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512" exitCode=0 Jul 27 15:14:27.964746 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.964605 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" Jul 27 15:14:27.964746 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.964650 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" event={"ID":"ee0a4040-329f-4263-8e99-018fef9065e3","Type":"ContainerDied","Data":"d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512"} Jul 27 15:14:27.964746 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.964699 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p" event={"ID":"ee0a4040-329f-4263-8e99-018fef9065e3","Type":"ContainerDied","Data":"f4cbc469459682aa4830925edc0222d33d4bfb749e6cdaa8663d91daa2a1ed8d"} Jul 27 15:14:27.964746 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.964726 2553 scope.go:117] "RemoveContainer" containerID="d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512" Jul 27 15:14:27.975999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.975979 2553 scope.go:117] "RemoveContainer" containerID="d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512" Jul 27 15:14:27.976309 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:14:27.976283 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512\": container with ID starting with d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512 not found: ID does not exist" containerID="d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512" Jul 27 15:14:27.976360 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.976312 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512"} err="failed to get container status \"d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512\": rpc error: code = NotFound desc = could not find container \"d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512\": container with ID starting with d59fe0d30a24d173c5c22414c2bed9e3cf1ca57fdb5cc28abf1d3def4b899512 not found: ID does not exist" Jul 27 15:14:27.989918 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.989891 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p"] Jul 27 15:14:27.999936 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:27.999909 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dhd9p"] Jul 27 15:14:29.191980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:29.191926 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee0a4040-329f-4263-8e99-018fef9065e3" path="/var/lib/kubelet/pods/ee0a4040-329f-4263-8e99-018fef9065e3/volumes" Jul 27 15:14:43.803764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.803732 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj"] Jul 27 15:14:43.804296 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.804276 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee0a4040-329f-4263-8e99-018fef9065e3" containerName="manager" Jul 27 15:14:43.804372 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.804298 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee0a4040-329f-4263-8e99-018fef9065e3" containerName="manager" Jul 27 15:14:43.804425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.804404 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee0a4040-329f-4263-8e99-018fef9065e3" containerName="manager" Jul 27 15:14:43.811075 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.811032 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.813093 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.813042 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9sgd4\"" Jul 27 15:14:43.821142 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.821114 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj"] Jul 27 15:14:43.872619 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872583 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872635 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57901768-40ee-411f-94a3-544355959d94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872667 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872691 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872707 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872723 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9crh\" (UniqueName: \"kubernetes.io/projected/57901768-40ee-411f-94a3-544355959d94-kube-api-access-z9crh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872784 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57901768-40ee-411f-94a3-544355959d94-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872826 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.872989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.872851 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57901768-40ee-411f-94a3-544355959d94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.973827 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.973784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57901768-40ee-411f-94a3-544355959d94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.973827 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.973827 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.973848 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.973960 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974005 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9crh\" (UniqueName: \"kubernetes.io/projected/57901768-40ee-411f-94a3-544355959d94-kube-api-access-z9crh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974046 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57901768-40ee-411f-94a3-544355959d94-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974260 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974291 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974318 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974324 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57901768-40ee-411f-94a3-544355959d94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974569 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974416 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974622 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974594 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974678 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974653 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.974678 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.974658 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57901768-40ee-411f-94a3-544355959d94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.976592 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.976563 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57901768-40ee-411f-94a3-544355959d94-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.976715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.976603 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57901768-40ee-411f-94a3-544355959d94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.982257 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.982234 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57901768-40ee-411f-94a3-544355959d94-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:43.982456 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:43.982434 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9crh\" (UniqueName: \"kubernetes.io/projected/57901768-40ee-411f-94a3-544355959d94-kube-api-access-z9crh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tthvj\" (UID: \"57901768-40ee-411f-94a3-544355959d94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:44.122497 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:44.122456 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:44.266333 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:44.266307 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj"] Jul 27 15:14:44.268779 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:14:44.268746 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57901768_40ee_411f_94a3_544355959d94.slice/crio-4756f2f095e61ecc66a58b1c6283ba5e2818265a5014765220d91c6d05ee5480 WatchSource:0}: Error finding container 4756f2f095e61ecc66a58b1c6283ba5e2818265a5014765220d91c6d05ee5480: Status 404 returned error can't find the container with id 4756f2f095e61ecc66a58b1c6283ba5e2818265a5014765220d91c6d05ee5480 Jul 27 15:14:44.270986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:44.270949 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:14:44.271086 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:44.271030 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:14:44.271134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:44.271090 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:14:45.044070 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:45.044024 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" event={"ID":"57901768-40ee-411f-94a3-544355959d94","Type":"ContainerStarted","Data":"c2c5b1aaaf3ea3a62ba805764ff2e972076545b24182f2dee806044918b260ec"} Jul 27 15:14:45.044541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:45.044095 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" event={"ID":"57901768-40ee-411f-94a3-544355959d94","Type":"ContainerStarted","Data":"4756f2f095e61ecc66a58b1c6283ba5e2818265a5014765220d91c6d05ee5480"} Jul 27 15:14:45.063485 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:45.063434 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" podStartSLOduration=2.063415768 podStartE2EDuration="2.063415768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:14:45.062828278 +0000 UTC m=+588.344178352" watchObservedRunningTime="2026-07-27 15:14:45.063415768 +0000 UTC m=+588.344765849" Jul 27 15:14:45.122776 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:45.122744 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:45.127525 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:45.127501 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:46.048793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:46.048761 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:46.049845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:46.049826 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:14:57.160986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:57.160961 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:14:57.163620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:14:57.163596 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:15:00.110612 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.110577 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:15:00.115465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.115441 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.117394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.117367 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 15:15:00.117517 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.117457 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 15:15:00.117562 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.117530 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 15:15:00.117903 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.117884 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nxml9\"" Jul 27 15:15:00.120881 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.120859 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:15:00.144196 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.144168 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:15:00.227293 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.227260 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99-config-file\") pod \"limitador-limitador-54d6d988d5-6kt44\" (UID: \"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.227474 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.227368 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jf2b\" (UniqueName: \"kubernetes.io/projected/b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99-kube-api-access-4jf2b\") pod \"limitador-limitador-54d6d988d5-6kt44\" (UID: \"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.328296 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.328248 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99-config-file\") pod \"limitador-limitador-54d6d988d5-6kt44\" (UID: \"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.328477 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.328345 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jf2b\" (UniqueName: \"kubernetes.io/projected/b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99-kube-api-access-4jf2b\") pod \"limitador-limitador-54d6d988d5-6kt44\" (UID: \"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.328852 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.328830 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99-config-file\") pod \"limitador-limitador-54d6d988d5-6kt44\" (UID: \"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.336160 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.336124 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jf2b\" (UniqueName: \"kubernetes.io/projected/b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99-kube-api-access-4jf2b\") pod \"limitador-limitador-54d6d988d5-6kt44\" (UID: \"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.427195 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.427103 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:00.563579 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:00.563550 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:15:00.565436 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:15:00.565411 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82dbc3d_e516_40dd_9ab4_f3f6f3bfcf99.slice/crio-a97413bfc5b64af7c590c216aaf2cd3fa40a1e6bb3f02122e87f95fde3254d0a WatchSource:0}: Error finding container a97413bfc5b64af7c590c216aaf2cd3fa40a1e6bb3f02122e87f95fde3254d0a: Status 404 returned error can't find the container with id a97413bfc5b64af7c590c216aaf2cd3fa40a1e6bb3f02122e87f95fde3254d0a Jul 27 15:15:01.123464 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:01.123421 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" event={"ID":"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99","Type":"ContainerStarted","Data":"a97413bfc5b64af7c590c216aaf2cd3fa40a1e6bb3f02122e87f95fde3254d0a"} Jul 27 15:15:04.139399 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:04.139358 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" event={"ID":"b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99","Type":"ContainerStarted","Data":"2a8a36b92bbc7cf0044c81f6c10962793bc0107550ae25bcb88bce880b0fea83"} Jul 27 15:15:04.139823 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:04.139499 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:15:04.157488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:04.157435 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" podStartSLOduration=1.633390033 podStartE2EDuration="4.157418712s" totalImagesPullingTime="2.524028679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:15:04.156133251 +0000 UTC m=+607.437483325" watchObservedRunningTime="2026-07-27 15:15:04.157418712 +0000 UTC m=+607.438768789" Jul 27 15:15:15.145022 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:15:15.144983 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6kt44" Jul 27 15:16:16.053293 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:16.053256 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:16:24.505998 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:24.505911 2553 generic.go:356] "Generic (PLEG): container finished" podID="57901768-40ee-411f-94a3-544355959d94" containerID="c2c5b1aaaf3ea3a62ba805764ff2e972076545b24182f2dee806044918b260ec" exitCode=0 Jul 27 15:16:24.505998 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:24.505982 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" event={"ID":"57901768-40ee-411f-94a3-544355959d94","Type":"ContainerDied","Data":"c2c5b1aaaf3ea3a62ba805764ff2e972076545b24182f2dee806044918b260ec"} Jul 27 15:16:24.506472 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:24.506450 2553 scope.go:117] "RemoveContainer" containerID="c2c5b1aaaf3ea3a62ba805764ff2e972076545b24182f2dee806044918b260ec" Jul 27 15:16:24.507448 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:24.507424 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:16:24.507516 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:24.507494 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:16:24.507552 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:24.507528 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:16:25.511861 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:25.511827 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" event={"ID":"57901768-40ee-411f-94a3-544355959d94","Type":"ContainerStarted","Data":"2d566a98ac8f43234539060603fbc8820426dc519cd90f4e016c737cfda3ed77"} Jul 27 15:16:26.123490 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:26.123452 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:16:26.123738 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:26.123704 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tthvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 27 15:16:26.123835 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:26.123779 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" podUID="57901768-40ee-411f-94a3-544355959d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 27 15:16:27.123542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:27.123494 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tthvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 27 15:16:27.124082 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:27.123566 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" podUID="57901768-40ee-411f-94a3-544355959d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 27 15:16:28.122837 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:28.122798 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tthvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 27 15:16:28.123150 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:28.123122 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" podUID="57901768-40ee-411f-94a3-544355959d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 27 15:16:29.123225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:29.123188 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tthvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 27 15:16:29.123697 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:29.123262 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" podUID="57901768-40ee-411f-94a3-544355959d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 27 15:16:30.128257 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:30.128224 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:16:30.128636 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:30.128575 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:16:30.129254 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:30.129236 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tthvj" Jul 27 15:16:34.240111 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:34.240075 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:16:44.235322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:44.235286 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:16:55.048139 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:16:55.048041 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:17:03.342777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:17:03.342738 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:17:19.035396 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:17:19.035363 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:17:30.438556 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:17:30.438519 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:19:18.642567 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:19:18.642532 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:19:29.139267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:19:29.139234 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:19:39.843498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:19:39.843461 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:19:57.190770 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:19:57.190689 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:19:57.191245 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:19:57.191018 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:21:22.437386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:21:22.437302 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:21:31.746761 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:21:31.746726 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:21:44.843215 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:21:44.843178 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:22:01.748394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:22:01.748357 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:22:11.535740 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:22:11.535699 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:22:20.639324 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:22:20.639288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:22:31.738549 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:22:31.738510 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:22:42.136158 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:22:42.136115 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:22:52.542764 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:22:52.542680 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:23:04.232677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:23:04.232644 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:23:13.937153 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:23:13.937115 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:23:39.135693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:23:39.135657 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:23:48.041479 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:23:48.041446 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:24:10.640360 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:10.640327 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:24:20.543544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:20.543454 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:24:37.550316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:37.550273 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:24:47.348141 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:47.348104 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:24:57.224406 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:57.224377 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:24:57.225291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:57.225262 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:24:58.448555 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:24:58.448512 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:25:07.241455 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:25:07.241420 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:25:16.246088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:25:16.246043 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:27:04.161762 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:27:04.161721 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:27:15.756902 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:27:15.756867 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:27:34.066022 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:27:34.065942 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:27:43.554659 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:27:43.554619 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:27:52.953546 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:27:52.953500 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:28:00.642927 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:28:00.642883 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:28:11.851961 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:28:11.851925 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:28:23.039836 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:28:23.039797 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:28:32.046236 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:28:32.046197 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:28:43.337508 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:28:43.337470 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:28:55.076374 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:28:55.076293 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:29:02.645814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:02.645778 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:29:19.441061 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:19.441026 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:29:29.641176 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:29.641130 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:29:42.645386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:42.645351 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:29:54.243559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:54.243517 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:29:57.252411 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:57.252379 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:29:57.255068 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:29:57.255022 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:30:00.134169 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.134132 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg"] Jul 27 15:30:00.137411 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.137389 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" Jul 27 15:30:00.139201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.139177 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgfwg\"" Jul 27 15:30:00.143076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.143036 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg"] Jul 27 15:30:00.149645 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.149623 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" Jul 27 15:30:00.284005 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.283976 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg"] Jul 27 15:30:00.286656 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:30:00.286633 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d3f4ed_900b_4cae_b59a_b5a712e3aab9.slice/crio-084ba9cd936f90899d16c7c631de98e2f99e5681842f4fd5468a5a5fc7c8a5ea WatchSource:0}: Error finding container 084ba9cd936f90899d16c7c631de98e2f99e5681842f4fd5468a5a5fc7c8a5ea: Status 404 returned error can't find the container with id 084ba9cd936f90899d16c7c631de98e2f99e5681842f4fd5468a5a5fc7c8a5ea Jul 27 15:30:00.288393 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:00.288379 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 15:30:01.227001 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:01.226964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" event={"ID":"06d3f4ed-900b-4cae-b59a-b5a712e3aab9","Type":"ContainerStarted","Data":"084ba9cd936f90899d16c7c631de98e2f99e5681842f4fd5468a5a5fc7c8a5ea"} Jul 27 15:30:05.248500 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:05.248463 2553 generic.go:356] "Generic (PLEG): container finished" podID="06d3f4ed-900b-4cae-b59a-b5a712e3aab9" containerID="7847d1aced232c747ba2321b4360a42d8309b33c5115217f30d1daba7e69eca7" exitCode=0 Jul 27 15:30:05.248899 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:05.248543 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" event={"ID":"06d3f4ed-900b-4cae-b59a-b5a712e3aab9","Type":"ContainerDied","Data":"7847d1aced232c747ba2321b4360a42d8309b33c5115217f30d1daba7e69eca7"} Jul 27 15:30:06.391444 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:06.391418 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" Jul 27 15:30:07.259743 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:07.259716 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" Jul 27 15:30:07.259889 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:07.259753 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752770-6l8gg" event={"ID":"06d3f4ed-900b-4cae-b59a-b5a712e3aab9","Type":"ContainerDied","Data":"084ba9cd936f90899d16c7c631de98e2f99e5681842f4fd5468a5a5fc7c8a5ea"} Jul 27 15:30:07.259889 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:07.259804 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="084ba9cd936f90899d16c7c631de98e2f99e5681842f4fd5468a5a5fc7c8a5ea" Jul 27 15:30:18.443913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:18.443875 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:30:32.136969 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:32.136885 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:30:39.030570 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:39.030532 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:30:48.436501 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:48.436464 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:30:56.339634 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:30:56.339601 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:31:05.933116 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:31:05.933077 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:31:15.138251 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:31:15.138212 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:31:25.036789 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:31:25.036746 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:31:34.636904 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:31:34.636864 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:31:43.647963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:31:43.647926 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:31:56.537750 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:31:56.537664 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:32:05.835284 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:32:05.835243 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:32:15.042558 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:32:15.042521 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:32:26.445647 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:32:26.445614 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:32:38.743261 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:32:38.743213 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:32:47.043341 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:32:47.043300 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:32:58.748778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:32:58.748741 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:33:10.242839 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:10.242803 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:33:17.143671 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:17.143634 2553 generic.go:356] "Generic (PLEG): container finished" podID="19bec687-43ed-459b-b581-725638dfc717" containerID="404c07ad445aa19944fd5bfb87651c0f94d84fbd2d3e236120f2d4569c650e88" exitCode=1 Jul 27 15:33:17.144106 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:17.143690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" event={"ID":"19bec687-43ed-459b-b581-725638dfc717","Type":"ContainerDied","Data":"404c07ad445aa19944fd5bfb87651c0f94d84fbd2d3e236120f2d4569c650e88"} Jul 27 15:33:17.144106 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:17.144039 2553 scope.go:117] "RemoveContainer" containerID="404c07ad445aa19944fd5bfb87651c0f94d84fbd2d3e236120f2d4569c650e88" Jul 27 15:33:18.028781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:18.028688 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:33:18.150822 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:18.150783 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" event={"ID":"19bec687-43ed-459b-b581-725638dfc717","Type":"ContainerStarted","Data":"2b275fc4a857c0cec9910f5e59132bae9b62ff47ddaaec1b69412e07f97e6af9"} Jul 27 15:33:18.151347 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:18.150881 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:33:21.178759 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:21.178667 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:33:36.287646 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:36.287611 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:33:46.544410 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:46.544371 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:33:49.158975 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:49.158940 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n85mq" Jul 27 15:33:56.632642 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:33:56.632601 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:34:06.246094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:06.242276 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:34:07.485843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.485810 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d"] Jul 27 15:34:07.486243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.486210 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d3f4ed-900b-4cae-b59a-b5a712e3aab9" containerName="cleanup" Jul 27 15:34:07.486243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.486223 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d3f4ed-900b-4cae-b59a-b5a712e3aab9" containerName="cleanup" Jul 27 15:34:07.486313 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.486299 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d3f4ed-900b-4cae-b59a-b5a712e3aab9" containerName="cleanup" Jul 27 15:34:07.489381 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.489363 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.491494 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.491467 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-654abfac-openshift-default-dockercfg-gbrrv\"" Jul 27 15:34:07.501464 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.501442 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d"] Jul 27 15:34:07.632492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632444 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-envoy\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632530 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-socket\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632563 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-credential-socket\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632608 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-certs\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632641 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-token\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632715 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089a7ad-61cb-45d7-baee-1e1f091ca204-istiod-ca-cert\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632792 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rrg\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-kube-api-access-55rrg\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632844 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-data\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.632913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.632871 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-podinfo\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734222 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734183 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-envoy\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734251 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-socket\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734284 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-credential-socket\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734328 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-certs\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734361 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-token\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089a7ad-61cb-45d7-baee-1e1f091ca204-istiod-ca-cert\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734442 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rrg\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-kube-api-access-55rrg\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734485 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-data\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734509 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-podinfo\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734882 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734780 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-credential-socket\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.734934 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.734904 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-data\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.735179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.735156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089a7ad-61cb-45d7-baee-1e1f091ca204-istiod-ca-cert\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.735312 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.735293 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-socket\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.735436 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.735346 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-certs\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.736690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.736638 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-envoy\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.736877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.736860 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-podinfo\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.744251 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.744226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-token\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.744367 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.744306 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rrg\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-kube-api-access-55rrg\") pod \"e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.799505 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.799471 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:07.942896 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.942844 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d"] Jul 27 15:34:07.945769 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:34:07.945740 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4089a7ad_61cb_45d7_baee_1e1f091ca204.slice/crio-869b39c02bd0f15dddc0ca8d097ce9ac9085a95aa14228d3f6fbdb6a9db2110a WatchSource:0}: Error finding container 869b39c02bd0f15dddc0ca8d097ce9ac9085a95aa14228d3f6fbdb6a9db2110a: Status 404 returned error can't find the container with id 869b39c02bd0f15dddc0ca8d097ce9ac9085a95aa14228d3f6fbdb6a9db2110a Jul 27 15:34:07.948401 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.948362 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:34:07.948495 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.948447 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:34:07.948495 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:07.948485 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:34:08.388805 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:08.388771 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" event={"ID":"4089a7ad-61cb-45d7-baee-1e1f091ca204","Type":"ContainerStarted","Data":"55d6cb475cedad50d866de44b7f398bc12ead9f57852019bb55159f63c374390"} Jul 27 15:34:08.388977 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:08.388825 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" event={"ID":"4089a7ad-61cb-45d7-baee-1e1f091ca204","Type":"ContainerStarted","Data":"869b39c02bd0f15dddc0ca8d097ce9ac9085a95aa14228d3f6fbdb6a9db2110a"} Jul 27 15:34:08.408955 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:08.408906 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podStartSLOduration=1.408890537 podStartE2EDuration="1.408890537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:34:08.407404864 +0000 UTC m=+1751.688754939" watchObservedRunningTime="2026-07-27 15:34:08.408890537 +0000 UTC m=+1751.690240610" Jul 27 15:34:08.800439 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:08.800351 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:09.806442 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:09.806411 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:10.398196 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:10.398166 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:10.399092 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:10.399071 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:17.231863 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:17.231823 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:34:25.462972 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:25.462938 2553 patch_prober.go:28] interesting pod/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": read tcp 10.132.0.2:53020->10.132.0.39:15021: read: connection reset by peer" start-of-body= Jul 27 15:34:25.463508 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:25.463011 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": read tcp 10.132.0.2:53020->10.132.0.39:15021: read: connection reset by peer" Jul 27 15:34:26.484461 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:26.484425 2553 generic.go:356] "Generic (PLEG): container finished" podID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerID="55d6cb475cedad50d866de44b7f398bc12ead9f57852019bb55159f63c374390" exitCode=0 Jul 27 15:34:26.484990 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:26.484481 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" event={"ID":"4089a7ad-61cb-45d7-baee-1e1f091ca204","Type":"ContainerDied","Data":"55d6cb475cedad50d866de44b7f398bc12ead9f57852019bb55159f63c374390"} Jul 27 15:34:26.485090 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:26.485010 2553 scope.go:117] "RemoveContainer" containerID="55d6cb475cedad50d866de44b7f398bc12ead9f57852019bb55159f63c374390" Jul 27 15:34:26.486038 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:26.486010 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:34:26.486164 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:26.486109 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:34:26.486164 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:26.486145 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:34:27.490899 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:27.490811 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" event={"ID":"4089a7ad-61cb-45d7-baee-1e1f091ca204","Type":"ContainerStarted","Data":"f5ef91755da5d13cbe6e8d7a31d52db5de2fa6d7e316230817d032b78860b6f3"} Jul 27 15:34:27.800278 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:27.800182 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:27.800438 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:27.800366 2553 patch_prober.go:28] interesting pod/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 15:34:27.800438 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:27.800417 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 15:34:28.800584 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:28.800549 2553 patch_prober.go:28] interesting pod/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 15:34:28.801009 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:28.800608 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 15:34:29.800541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:29.800504 2553 patch_prober.go:28] interesting pod/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 15:34:29.800729 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:29.800561 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 15:34:30.800646 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:30.800611 2553 patch_prober.go:28] interesting pod/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 27 15:34:30.800824 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:30.800667 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 27 15:34:31.803276 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:31.803243 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:31.803719 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:31.803532 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:31.804267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:31.804248 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:34:35.292430 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.292393 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst"] Jul 27 15:34:35.295853 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.295836 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:35.297848 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.297828 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgfwg\"" Jul 27 15:34:35.297951 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.297830 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 15:34:35.301473 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.301448 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst"] Jul 27 15:34:35.385776 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.385737 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733b652a-85c6-4dbd-a43c-b1f7eef25efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-654abfac-zfhst\" (UID: \"733b652a-85c6-4dbd-a43c-b1f7eef25efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:35.487017 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.486979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733b652a-85c6-4dbd-a43c-b1f7eef25efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-654abfac-zfhst\" (UID: \"733b652a-85c6-4dbd-a43c-b1f7eef25efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:35.487682 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.487656 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733b652a-85c6-4dbd-a43c-b1f7eef25efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-654abfac-zfhst\" (UID: \"733b652a-85c6-4dbd-a43c-b1f7eef25efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:35.607971 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.607941 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:35.748415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:35.743752 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst"] Jul 27 15:34:36.535667 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:36.535631 2553 generic.go:356] "Generic (PLEG): container finished" podID="733b652a-85c6-4dbd-a43c-b1f7eef25efc" containerID="3fc84be7557b1ea1bd794a579278b839b3384b0c4822d62156875843c1ca2727" exitCode=0 Jul 27 15:34:36.536043 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:36.535713 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" event={"ID":"733b652a-85c6-4dbd-a43c-b1f7eef25efc","Type":"ContainerDied","Data":"3fc84be7557b1ea1bd794a579278b839b3384b0c4822d62156875843c1ca2727"} Jul 27 15:34:36.536043 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:36.535760 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" event={"ID":"733b652a-85c6-4dbd-a43c-b1f7eef25efc","Type":"ContainerStarted","Data":"1ab8c457d23601b37837ad71339f3412a8ce81532d7e451487e662800d8ee31c"} Jul 27 15:34:37.674112 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:37.674087 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:37.808295 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:37.808205 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733b652a-85c6-4dbd-a43c-b1f7eef25efc-maas-api-service-ca\") pod \"733b652a-85c6-4dbd-a43c-b1f7eef25efc\" (UID: \"733b652a-85c6-4dbd-a43c-b1f7eef25efc\") " Jul 27 15:34:37.808583 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:37.808558 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733b652a-85c6-4dbd-a43c-b1f7eef25efc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "733b652a-85c6-4dbd-a43c-b1f7eef25efc" (UID: "733b652a-85c6-4dbd-a43c-b1f7eef25efc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:34:37.908809 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:37.908776 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/733b652a-85c6-4dbd-a43c-b1f7eef25efc-maas-api-service-ca\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:34:38.546899 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:38.546868 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" Jul 27 15:34:38.547149 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:38.546910 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst" event={"ID":"733b652a-85c6-4dbd-a43c-b1f7eef25efc","Type":"ContainerDied","Data":"1ab8c457d23601b37837ad71339f3412a8ce81532d7e451487e662800d8ee31c"} Jul 27 15:34:38.547149 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:38.546954 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ab8c457d23601b37837ad71339f3412a8ce81532d7e451487e662800d8ee31c" Jul 27 15:34:48.638815 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:48.638768 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:34:50.359166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:50.359077 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst"] Jul 27 15:34:50.361910 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:50.361879 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-654abfac-zfhst"] Jul 27 15:34:51.191321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:51.191289 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733b652a-85c6-4dbd-a43c-b1f7eef25efc" path="/var/lib/kubelet/pods/733b652a-85c6-4dbd-a43c-b1f7eef25efc/volumes" Jul 27 15:34:57.281020 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:57.280990 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:34:57.287482 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:34:57.287456 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:35:00.490223 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:00.490182 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d"] Jul 27 15:35:00.490660 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:00.490498 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" containerID="cri-o://f5ef91755da5d13cbe6e8d7a31d52db5de2fa6d7e316230817d032b78860b6f3" gracePeriod=30 Jul 27 15:35:01.500438 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.500405 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q"] Jul 27 15:35:01.500969 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.500948 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733b652a-85c6-4dbd-a43c-b1f7eef25efc" containerName="revoke-keys" Jul 27 15:35:01.501061 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.500972 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="733b652a-85c6-4dbd-a43c-b1f7eef25efc" containerName="revoke-keys" Jul 27 15:35:01.501130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.501117 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="733b652a-85c6-4dbd-a43c-b1f7eef25efc" containerName="revoke-keys" Jul 27 15:35:01.504110 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.504090 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.506040 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.506015 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-58d49f55-gw-openshift-default-dockercfg-hkg4l\"" Jul 27 15:35:01.513821 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.513795 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q"] Jul 27 15:35:01.530032 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-socket\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530041 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-certs\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530093 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-podinfo\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530164 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-token\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530199 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-credential-socket\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530215 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-data\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530250 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3781af2b-aefa-43ed-a302-232e93e65f5e-istiod-ca-cert\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530273 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjv8\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-kube-api-access-zxjv8\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.530368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.530296 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-envoy\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631644 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-socket\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631680 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-certs\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631700 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-podinfo\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-token\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631763 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-credential-socket\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631787 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-data\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.631931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631843 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3781af2b-aefa-43ed-a302-232e93e65f5e-istiod-ca-cert\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632228 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.631936 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjv8\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-kube-api-access-zxjv8\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632345 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.632329 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-envoy\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632403 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.632361 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-credential-socket\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632592 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.632545 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-data\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632718 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.632654 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-certs\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632718 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.632684 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-socket\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.632834 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.632758 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3781af2b-aefa-43ed-a302-232e93e65f5e-istiod-ca-cert\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.634462 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.634435 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-podinfo\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.634550 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.634539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-envoy\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.639290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.639263 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjv8\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-kube-api-access-zxjv8\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.639380 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.639296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-token\") pod \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:01.806097 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.805987 2553 patch_prober.go:28] interesting pod/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 15:35:01.806097 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.806073 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 15:35:01.816986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:01.816958 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:04.406687 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:04.406656 2553 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 27 15:35:04.406687 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce" Netns:"/var/run/netns/e41f7795-3cab-4112-a022-843603795b84" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:04.406687 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:04.406687 ip-10-0-135-218 kubenswrapper[2553]: > Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:04.406713 2553 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce" Netns:"/var/run/netns/e41f7795-3cab-4112-a022-843603795b84" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: > pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:04.406730 2553 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce" Netns:"/var/run/netns/e41f7795-3cab-4112-a022-843603795b84" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:04.407168 ip-10-0-135-218 kubenswrapper[2553]: > pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:04.407403 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:04.406792 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress(3781af2b-aefa-43ed-a302-232e93e65f5e)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress(3781af2b-aefa-43ed-a302-232e93e65f5e)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce\\\" Netns:\\\"/var/run/netns/e41f7795-3cab-4112-a022-843603795b84\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=3ddf3746df1fb64f3f9619611856a98106d45417c376f9029536ec1a6594e2ce;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get \\\"https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" Jul 27 15:35:04.669846 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:04.669766 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:04.670116 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:04.670103 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:05.678019 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.677990 2553 generic.go:356] "Generic (PLEG): container finished" podID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerID="f5ef91755da5d13cbe6e8d7a31d52db5de2fa6d7e316230817d032b78860b6f3" exitCode=0 Jul 27 15:35:05.678393 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.678076 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" event={"ID":"4089a7ad-61cb-45d7-baee-1e1f091ca204","Type":"ContainerDied","Data":"f5ef91755da5d13cbe6e8d7a31d52db5de2fa6d7e316230817d032b78860b6f3"} Jul 27 15:35:05.678393 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.678153 2553 scope.go:117] "RemoveContainer" containerID="55d6cb475cedad50d866de44b7f398bc12ead9f57852019bb55159f63c374390" Jul 27 15:35:05.745550 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.745525 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:35:05.770425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770333 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089a7ad-61cb-45d7-baee-1e1f091ca204-istiod-ca-cert\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770425 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770383 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-podinfo\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770632 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770439 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55rrg\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-kube-api-access-55rrg\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770632 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770467 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-envoy\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770632 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770507 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-socket\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770632 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770540 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-certs\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770632 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770568 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-token\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770632 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770603 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-credential-socket\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.770906 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.770659 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-data\") pod \"4089a7ad-61cb-45d7-baee-1e1f091ca204\" (UID: \"4089a7ad-61cb-45d7-baee-1e1f091ca204\") " Jul 27 15:35:05.771345 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771043 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:35:05.771345 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771157 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4089a7ad-61cb-45d7-baee-1e1f091ca204-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:35:05.771529 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771393 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:35:05.771529 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771422 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:35:05.771670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771639 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4089a7ad-61cb-45d7-baee-1e1f091ca204-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.771670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771661 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.771786 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771674 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.771786 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.771686 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.772357 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.772321 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-data" (OuterVolumeSpecName: "istio-data") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:35:05.773406 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.773379 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-kube-api-access-55rrg" (OuterVolumeSpecName: "kube-api-access-55rrg") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "kube-api-access-55rrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:35:05.773537 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.773518 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:35:05.773611 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.773532 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-token" (OuterVolumeSpecName: "istio-token") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:35:05.773984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.773966 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4089a7ad-61cb-45d7-baee-1e1f091ca204" (UID: "4089a7ad-61cb-45d7-baee-1e1f091ca204"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:35:05.872342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.872302 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.872342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.872333 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.872342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.872342 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.872342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.872351 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55rrg\" (UniqueName: \"kubernetes.io/projected/4089a7ad-61cb-45d7-baee-1e1f091ca204-kube-api-access-55rrg\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:05.872605 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:05.872361 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4089a7ad-61cb-45d7-baee-1e1f091ca204-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:35:06.684170 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:06.684133 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" event={"ID":"4089a7ad-61cb-45d7-baee-1e1f091ca204","Type":"ContainerDied","Data":"869b39c02bd0f15dddc0ca8d097ce9ac9085a95aa14228d3f6fbdb6a9db2110a"} Jul 27 15:35:06.684170 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:06.684153 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d" Jul 27 15:35:06.684716 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:06.684191 2553 scope.go:117] "RemoveContainer" containerID="f5ef91755da5d13cbe6e8d7a31d52db5de2fa6d7e316230817d032b78860b6f3" Jul 27 15:35:06.705316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:06.705289 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d"] Jul 27 15:35:06.709121 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:06.709097 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-654abfac-openshift-default-5cc7d78784-bfp4d"] Jul 27 15:35:07.192790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:07.192758 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" path="/var/lib/kubelet/pods/4089a7ad-61cb-45d7-baee-1e1f091ca204/volumes" Jul 27 15:35:07.258506 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:07.258469 2553 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 27 15:35:07.258506 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88" Netns:"/var/run/netns/98b696fb-ccc0-409e-9a3f-5e0f885fef35" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:07.258506 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:07.258506 ip-10-0-135-218 kubenswrapper[2553]: > Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:07.258526 2553 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88" Netns:"/var/run/netns/98b696fb-ccc0-409e-9a3f-5e0f885fef35" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: > pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:07.258560 2553 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88" Netns:"/var/run/netns/98b696fb-ccc0-409e-9a3f-5e0f885fef35" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:07.258734 ip-10-0-135-218 kubenswrapper[2553]: > pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:07.259153 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:07.258641 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress(3781af2b-aefa-43ed-a302-232e93e65f5e)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress(3781af2b-aefa-43ed-a302-232e93e65f5e)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88\\\" Netns:\\\"/var/run/netns/98b696fb-ccc0-409e-9a3f-5e0f885fef35\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=bf4d090e8a464cb9cab9f1d5f6f3084c4fe3d47ce26b034c20ff773b4578be88;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get \\\"https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" Jul 27 15:35:18.185196 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:18.185160 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:18.185581 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:18.185479 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:20.776289 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:20.776254 2553 log.go:32] "RunPodSandbox from runtime service failed" err=< Jul 27 15:35:20.776289 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0" Netns:"/var/run/netns/1f352a9c-36c3-4613-aa28-1a28ecf6b9bd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:20.776289 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:20.776289 ip-10-0-135-218 kubenswrapper[2553]: > Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:20.776311 2553 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0" Netns:"/var/run/netns/1f352a9c-36c3-4613-aa28-1a28ecf6b9bd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: > pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:20.776329 2553 kuberuntime_manager.go:1353] "CreatePodSandbox for pod failed" err=< Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0" Netns:"/var/run/netns/1f352a9c-36c3-4613-aa28-1a28ecf6b9bd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e" Path:"" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get "https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s": context deadline exceeded Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Jul 27 15:35:20.776691 ip-10-0-135-218 kubenswrapper[2553]: > pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:20.776920 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:35:20.776400 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress(3781af2b-aefa-43ed-a302-232e93e65f5e)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress(3781af2b-aefa-43ed-a302-232e93e65f5e)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q_openshift-ingress_3781af2b-aefa-43ed-a302-232e93e65f5e_0(6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0): error adding pod openshift-ingress_e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus-shim\\\" name=\\\"multus-cni-network\\\" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:\\\"6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0\\\" Netns:\\\"/var/run/netns/1f352a9c-36c3-4613-aa28-1a28ecf6b9bd\\\" IfName:\\\"eth0\\\" Args:\\\"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-ingress;K8S_POD_NAME=e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q;K8S_POD_INFRA_CONTAINER_ID=6ff3e0f49ff93f5c3f11222594517656e67622708a59f3440330eca1bfe3c2a0;K8S_POD_UID=3781af2b-aefa-43ed-a302-232e93e65f5e\\\" Path:\\\"\\\" ERRORED: error configuring pod [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q] networking: Multus: [openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q/3781af2b-aefa-43ed-a302-232e93e65f5e]: error waiting for pod: Get \\\"https://aec4395a0df5348538f12988908e86f1-d9d3b76b8a75332a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/pods/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q?timeout=1m0s\\\": context deadline exceeded\\n': StdinData: {\\\"auxiliaryCNIChainName\\\":\\\"vendor-cni-chain\\\",\\\"binDir\\\":\\\"/var/lib/cni/bin\\\",\\\"clusterNetwork\\\":\\\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\\\",\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"daemonSocketDir\\\":\\\"/run/multus/socket\\\",\\\"globalNamespaces\\\":\\\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\\\",\\\"logLevel\\\":\\\"verbose\\\",\\\"logToStderr\\\":true,\\\"name\\\":\\\"multus-cni-network\\\",\\\"namespaceIsolation\\\":true,\\\"type\\\":\\\"multus-shim\\\"}\"" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" Jul 27 15:35:36.185643 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.185603 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:36.186146 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.185969 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:36.328491 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.328464 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q"] Jul 27 15:35:36.331435 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:35:36.331406 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3781af2b_aefa_43ed_a302_232e93e65f5e.slice/crio-f55f3b7079adb0f50d544b678b012f1f1cf4f5d70ac7e9b90e111279bc9a1213 WatchSource:0}: Error finding container f55f3b7079adb0f50d544b678b012f1f1cf4f5d70ac7e9b90e111279bc9a1213: Status 404 returned error can't find the container with id f55f3b7079adb0f50d544b678b012f1f1cf4f5d70ac7e9b90e111279bc9a1213 Jul 27 15:35:36.333873 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.333831 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:35:36.333990 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.333927 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:35:36.333990 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.333981 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:35:36.829980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.829943 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" event={"ID":"3781af2b-aefa-43ed-a302-232e93e65f5e","Type":"ContainerStarted","Data":"f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d"} Jul 27 15:35:36.830176 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.829997 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" event={"ID":"3781af2b-aefa-43ed-a302-232e93e65f5e","Type":"ContainerStarted","Data":"f55f3b7079adb0f50d544b678b012f1f1cf4f5d70ac7e9b90e111279bc9a1213"} Jul 27 15:35:36.850281 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:36.850231 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podStartSLOduration=35.850214993 podStartE2EDuration="35.850214993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:35:36.848737581 +0000 UTC m=+1840.130087655" watchObservedRunningTime="2026-07-27 15:35:36.850214993 +0000 UTC m=+1840.131565066" Jul 27 15:35:37.817999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:37.817953 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:37.823081 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:37.823028 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:37.834834 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:37.834803 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:37.835547 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:37.835526 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:42.546772 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:42.546728 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:35:50.897516 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:50.897479 2553 generic.go:356] "Generic (PLEG): container finished" podID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerID="f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d" exitCode=0 Jul 27 15:35:50.897941 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:50.897553 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" event={"ID":"3781af2b-aefa-43ed-a302-232e93e65f5e","Type":"ContainerDied","Data":"f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d"} Jul 27 15:35:50.898004 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:50.897941 2553 scope.go:117] "RemoveContainer" containerID="f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d" Jul 27 15:35:50.898994 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:50.898964 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:35:50.899072 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:50.899036 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:35:50.899125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:50.899083 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:35:51.904615 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:51.904581 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" event={"ID":"3781af2b-aefa-43ed-a302-232e93e65f5e","Type":"ContainerStarted","Data":"7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d"} Jul 27 15:35:52.817401 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:52.817366 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:52.818677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:52.818647 2553 patch_prober.go:28] interesting pod/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 27 15:35:52.818905 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:52.818881 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 27 15:35:52.835505 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:52.835467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:53.817707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:53.817671 2553 patch_prober.go:28] interesting pod/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 27 15:35:53.818505 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:53.817725 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 27 15:35:54.818285 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:54.818245 2553 patch_prober.go:28] interesting pod/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 27 15:35:54.818918 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:54.818885 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 27 15:35:55.822103 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:55.822072 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:35:55.823129 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:35:55.823108 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:36:16.738573 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:16.738535 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:36:18.560941 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:18.560905 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q"] Jul 27 15:36:18.561383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:18.561193 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" containerID="cri-o://7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d" gracePeriod=30 Jul 27 15:36:23.828447 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.828381 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:36:23.956885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.956846 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-data\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.956903 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-credential-socket\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.956947 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-podinfo\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.956973 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-token\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957007 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxjv8\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-kube-api-access-zxjv8\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957093 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-socket\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957145 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3781af2b-aefa-43ed-a302-232e93e65f5e-istiod-ca-cert\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957182 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-certs\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957238 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:36:23.957415 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957258 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-envoy\") pod \"3781af2b-aefa-43ed-a302-232e93e65f5e\" (UID: \"3781af2b-aefa-43ed-a302-232e93e65f5e\") " Jul 27 15:36:23.957681 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957566 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:36:23.957681 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957649 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3781af2b-aefa-43ed-a302-232e93e65f5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:36:23.957789 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957733 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:23.957789 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957752 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:23.957789 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957767 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:36:23.957959 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.957845 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:36:23.959663 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.959641 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:36:23.959769 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.959671 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:36:23.959769 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.959751 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-kube-api-access-zxjv8" (OuterVolumeSpecName: "kube-api-access-zxjv8") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "kube-api-access-zxjv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:36:23.959851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:23.959756 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3781af2b-aefa-43ed-a302-232e93e65f5e" (UID: "3781af2b-aefa-43ed-a302-232e93e65f5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:36:24.058838 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058808 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3781af2b-aefa-43ed-a302-232e93e65f5e-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.058838 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058834 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.058838 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058843 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.059107 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058850 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.059107 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058858 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.059107 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058866 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.059107 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.058874 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxjv8\" (UniqueName: \"kubernetes.io/projected/3781af2b-aefa-43ed-a302-232e93e65f5e-kube-api-access-zxjv8\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:36:24.063216 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.063179 2553 generic.go:356] "Generic (PLEG): container finished" podID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerID="7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d" exitCode=0 Jul 27 15:36:24.063333 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.063228 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" Jul 27 15:36:24.063333 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.063242 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" event={"ID":"3781af2b-aefa-43ed-a302-232e93e65f5e","Type":"ContainerDied","Data":"7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d"} Jul 27 15:36:24.063333 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.063296 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q" event={"ID":"3781af2b-aefa-43ed-a302-232e93e65f5e","Type":"ContainerDied","Data":"f55f3b7079adb0f50d544b678b012f1f1cf4f5d70ac7e9b90e111279bc9a1213"} Jul 27 15:36:24.063333 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.063320 2553 scope.go:117] "RemoveContainer" containerID="7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d" Jul 27 15:36:24.076919 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.076893 2553 scope.go:117] "RemoveContainer" containerID="f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d" Jul 27 15:36:24.085408 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.085388 2553 scope.go:117] "RemoveContainer" containerID="7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d" Jul 27 15:36:24.085680 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:36:24.085661 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d\": container with ID starting with 7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d not found: ID does not exist" containerID="7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d" Jul 27 15:36:24.085759 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.085693 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d"} err="failed to get container status \"7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d\": rpc error: code = NotFound desc = could not find container \"7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d\": container with ID starting with 7685351189ec7adf57f606233da603a2e797c45f6b529d40de0980971491364d not found: ID does not exist" Jul 27 15:36:24.085759 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.085718 2553 scope.go:117] "RemoveContainer" containerID="f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d" Jul 27 15:36:24.086002 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:36:24.085977 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d\": container with ID starting with f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d not found: ID does not exist" containerID="f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d" Jul 27 15:36:24.086069 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.086013 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d"} err="failed to get container status \"f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d\": rpc error: code = NotFound desc = could not find container \"f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d\": container with ID starting with f074df7680c58c3480ad74280650aaa32815f6020dc582ee5a795c798b95776d not found: ID does not exist" Jul 27 15:36:24.088876 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.088856 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q"] Jul 27 15:36:24.196936 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:24.196907 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-58d49f55-gw-openshift-default-778cbcd4b4-4w87q"] Jul 27 15:36:25.192785 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:25.192750 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" path="/var/lib/kubelet/pods/3781af2b-aefa-43ed-a302-232e93e65f5e/volumes" Jul 27 15:36:29.396674 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.396552 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv"] Jul 27 15:36:29.397501 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397471 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" Jul 27 15:36:29.397501 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397501 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" Jul 27 15:36:29.397696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397521 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" Jul 27 15:36:29.397696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397531 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" Jul 27 15:36:29.397696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397567 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" Jul 27 15:36:29.397696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397575 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" Jul 27 15:36:29.397696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397589 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" Jul 27 15:36:29.397696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397597 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" Jul 27 15:36:29.397979 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397777 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" Jul 27 15:36:29.397979 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397790 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" Jul 27 15:36:29.397979 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.397799 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4089a7ad-61cb-45d7-baee-1e1f091ca204" containerName="istio-proxy" Jul 27 15:36:29.403491 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.403459 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv"] Jul 27 15:36:29.403618 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.403581 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.405525 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.405506 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3af40866-openshift-default-dockercfg-mttzc\"" Jul 27 15:36:29.507173 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-socket\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507349 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507185 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206a02c-077d-44e6-b850-9a6d144525f6-istio-podinfo\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507349 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507285 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-istio-token\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507431 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507380 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-envoy\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507431 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507410 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206a02c-077d-44e6-b850-9a6d144525f6-istiod-ca-cert\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507499 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507451 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-certs\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507499 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507478 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-data\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507502 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-credential-socket\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.507568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.507529 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccjwq\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-kube-api-access-ccjwq\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608392 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608356 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-envoy\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608392 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608395 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206a02c-077d-44e6-b850-9a6d144525f6-istiod-ca-cert\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608417 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-certs\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608432 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-data\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608459 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-credential-socket\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608485 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccjwq\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-kube-api-access-ccjwq\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608512 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-socket\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608562 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206a02c-077d-44e6-b850-9a6d144525f6-istio-podinfo\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.608656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608617 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-istio-token\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.609023 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608911 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-certs\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.609023 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.608979 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-socket\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.609203 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.609181 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-data\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.609307 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.609286 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-credential-socket\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.609376 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.609356 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206a02c-077d-44e6-b850-9a6d144525f6-istiod-ca-cert\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.610837 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.610811 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-envoy\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.611026 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.611009 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206a02c-077d-44e6-b850-9a6d144525f6-istio-podinfo\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.617698 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.617680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-istio-token\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.617798 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.617717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccjwq\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-kube-api-access-ccjwq\") pod \"e2e-ait-3af40866-openshift-default-58768868f6-x5vxv\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.715043 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.714948 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:29.863462 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.863437 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv"] Jul 27 15:36:29.865839 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:36:29.865806 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1206a02c_077d_44e6_b850_9a6d144525f6.slice/crio-de2194c83a8b3c7241553b6bb4063db8935f2a1ffa9a23497b52c2e31fdc42e6 WatchSource:0}: Error finding container de2194c83a8b3c7241553b6bb4063db8935f2a1ffa9a23497b52c2e31fdc42e6: Status 404 returned error can't find the container with id de2194c83a8b3c7241553b6bb4063db8935f2a1ffa9a23497b52c2e31fdc42e6 Jul 27 15:36:29.868176 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.868136 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:36:29.868286 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.868220 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:36:29.868286 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:29.868258 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:36:30.097825 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:30.097718 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" event={"ID":"1206a02c-077d-44e6-b850-9a6d144525f6","Type":"ContainerStarted","Data":"615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655"} Jul 27 15:36:30.097825 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:30.097784 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" event={"ID":"1206a02c-077d-44e6-b850-9a6d144525f6","Type":"ContainerStarted","Data":"de2194c83a8b3c7241553b6bb4063db8935f2a1ffa9a23497b52c2e31fdc42e6"} Jul 27 15:36:30.117634 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:30.117569 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podStartSLOduration=1.117546395 podStartE2EDuration="1.117546395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:36:30.114356802 +0000 UTC m=+1893.395706912" watchObservedRunningTime="2026-07-27 15:36:30.117546395 +0000 UTC m=+1893.398896469" Jul 27 15:36:30.715125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:30.715090 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:30.720019 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:30.719991 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:31.103053 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:31.103033 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:31.104138 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:31.104118 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:39.343466 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:39.343429 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:36:47.092355 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.092320 2553 patch_prober.go:28] interesting pod/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:51296->10.132.0.42:15021: read: connection reset by peer" start-of-body= Jul 27 15:36:47.092707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.092410 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:51296->10.132.0.42:15021: read: connection reset by peer" Jul 27 15:36:47.185655 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.185615 2553 generic.go:356] "Generic (PLEG): container finished" podID="1206a02c-077d-44e6-b850-9a6d144525f6" containerID="615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655" exitCode=0 Jul 27 15:36:47.190880 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.190850 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" event={"ID":"1206a02c-077d-44e6-b850-9a6d144525f6","Type":"ContainerDied","Data":"615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655"} Jul 27 15:36:47.191299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.191281 2553 scope.go:117] "RemoveContainer" containerID="615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655" Jul 27 15:36:47.192327 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.192299 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:36:47.192395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.192365 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:36:47.192434 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:47.192395 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:36:48.196376 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:48.196335 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" event={"ID":"1206a02c-077d-44e6-b850-9a6d144525f6","Type":"ContainerStarted","Data":"080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298"} Jul 27 15:36:48.715314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:48.715279 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:48.715547 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:48.715493 2553 patch_prober.go:28] interesting pod/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 27 15:36:48.715594 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:48.715553 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 27 15:36:49.716372 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:49.716329 2553 patch_prober.go:28] interesting pod/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 27 15:36:49.716804 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:49.716400 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 27 15:36:50.715595 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:50.715555 2553 patch_prober.go:28] interesting pod/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 27 15:36:50.715777 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:50.715616 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 27 15:36:51.715533 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:51.715494 2553 patch_prober.go:28] interesting pod/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 27 15:36:51.716028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:51.715572 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 27 15:36:52.719371 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:52.719331 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:52.719876 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:52.719723 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:36:52.720311 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:36:52.720289 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:37:10.840620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:10.840583 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:37:13.574857 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:13.574825 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv"] Jul 27 15:37:13.575279 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:13.575029 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" containerID="cri-o://080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298" gracePeriod=30 Jul 27 15:37:14.326412 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.326368 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r"] Jul 27 15:37:14.327020 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.326994 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3781af2b-aefa-43ed-a302-232e93e65f5e" containerName="istio-proxy" Jul 27 15:37:14.330141 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.330120 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.332186 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.332163 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b2ba96cb-openshift-default-dockercfg-r4n8g\"" Jul 27 15:37:14.345876 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.345845 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r"] Jul 27 15:37:14.433481 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433446 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-credential-socket\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433481 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433490 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istiod-ca-cert\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433767 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433519 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfmg\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-kube-api-access-mtfmg\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433767 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433572 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-data\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433767 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433654 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-socket\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433767 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433681 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-token\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433789 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-certs\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433827 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-envoy\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.433991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.433858 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-podinfo\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534580 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-socket\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534625 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534623 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-token\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534678 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-certs\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534694 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-envoy\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534713 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-podinfo\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534768 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-credential-socket\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534800 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istiod-ca-cert\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534823 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfmg\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-kube-api-access-mtfmg\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.534862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.534846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-data\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.535267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.535100 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-socket\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.535352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.535328 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-data\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.535487 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.535457 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-certs\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.535577 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.535529 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-credential-socket\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.535824 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.535802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istiod-ca-cert\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.537246 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.537215 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-envoy\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.537338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.537319 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-podinfo\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.543615 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.543588 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-token\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.543822 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.543804 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfmg\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-kube-api-access-mtfmg\") pod \"e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.640920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.640884 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:14.782808 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.782575 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r"] Jul 27 15:37:14.785543 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:37:14.785507 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647a510d_fc56_43d9_8a6d_a11ee39f2ff0.slice/crio-0a81752afa59d88dc6e412fd18b21b70bf9f7a225b13d3fb30f6595932e67d55 WatchSource:0}: Error finding container 0a81752afa59d88dc6e412fd18b21b70bf9f7a225b13d3fb30f6595932e67d55: Status 404 returned error can't find the container with id 0a81752afa59d88dc6e412fd18b21b70bf9f7a225b13d3fb30f6595932e67d55 Jul 27 15:37:14.787927 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.787885 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:37:14.788028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.787968 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:37:14.788028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:14.788007 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:37:15.331477 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:15.331446 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" event={"ID":"647a510d-fc56-43d9-8a6d-a11ee39f2ff0","Type":"ContainerStarted","Data":"6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c"} Jul 27 15:37:15.331646 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:15.331493 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" event={"ID":"647a510d-fc56-43d9-8a6d-a11ee39f2ff0","Type":"ContainerStarted","Data":"0a81752afa59d88dc6e412fd18b21b70bf9f7a225b13d3fb30f6595932e67d55"} Jul 27 15:37:15.350248 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:15.350201 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" podStartSLOduration=1.350185428 podStartE2EDuration="1.350185428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:37:15.349156396 +0000 UTC m=+1938.630506472" watchObservedRunningTime="2026-07-27 15:37:15.350185428 +0000 UTC m=+1938.631535502" Jul 27 15:37:15.641121 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:15.641077 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:15.646299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:15.646273 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:16.337861 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:16.337823 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:16.338861 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:16.338842 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:18.823853 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.823828 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:37:18.978589 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978530 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-socket\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978589 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978592 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206a02c-077d-44e6-b850-9a6d144525f6-istio-podinfo\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978612 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-certs\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978640 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-credential-socket\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978720 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-istio-token\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978746 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-data\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978779 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccjwq\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-kube-api-access-ccjwq\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978814 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-envoy\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.978849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978841 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206a02c-077d-44e6-b850-9a6d144525f6-istiod-ca-cert\") pod \"1206a02c-077d-44e6-b850-9a6d144525f6\" (UID: \"1206a02c-077d-44e6-b850-9a6d144525f6\") " Jul 27 15:37:18.979243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.978854 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:37:18.979243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.979151 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:18.979243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.979150 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:37:18.979243 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.979229 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:37:18.979470 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.979446 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1206a02c-077d-44e6-b850-9a6d144525f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:37:18.979700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.979677 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:37:18.980948 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.980924 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:37:18.981094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.981010 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-kube-api-access-ccjwq" (OuterVolumeSpecName: "kube-api-access-ccjwq") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "kube-api-access-ccjwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:37:18.981314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.981292 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1206a02c-077d-44e6-b850-9a6d144525f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:37:18.981394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:18.981296 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1206a02c-077d-44e6-b850-9a6d144525f6" (UID: "1206a02c-077d-44e6-b850-9a6d144525f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:37:19.079845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079798 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1206a02c-077d-44e6-b850-9a6d144525f6-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.079845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079829 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1206a02c-077d-44e6-b850-9a6d144525f6-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.079845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079839 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.079845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079848 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.080127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079857 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.080127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079866 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.080127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079874 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccjwq\" (UniqueName: \"kubernetes.io/projected/1206a02c-077d-44e6-b850-9a6d144525f6-kube-api-access-ccjwq\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.080127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.079882 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1206a02c-077d-44e6-b850-9a6d144525f6-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:19.354092 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.354045 2553 generic.go:356] "Generic (PLEG): container finished" podID="1206a02c-077d-44e6-b850-9a6d144525f6" containerID="080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298" exitCode=0 Jul 27 15:37:19.354263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.354115 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" Jul 27 15:37:19.354263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.354150 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" event={"ID":"1206a02c-077d-44e6-b850-9a6d144525f6","Type":"ContainerDied","Data":"080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298"} Jul 27 15:37:19.354263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.354203 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv" event={"ID":"1206a02c-077d-44e6-b850-9a6d144525f6","Type":"ContainerDied","Data":"de2194c83a8b3c7241553b6bb4063db8935f2a1ffa9a23497b52c2e31fdc42e6"} Jul 27 15:37:19.354263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.354227 2553 scope.go:117] "RemoveContainer" containerID="080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298" Jul 27 15:37:19.362992 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.362967 2553 scope.go:117] "RemoveContainer" containerID="615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655" Jul 27 15:37:19.371210 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.371192 2553 scope.go:117] "RemoveContainer" containerID="080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298" Jul 27 15:37:19.371476 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:37:19.371460 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298\": container with ID starting with 080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298 not found: ID does not exist" containerID="080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298" Jul 27 15:37:19.371541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.371484 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298"} err="failed to get container status \"080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298\": rpc error: code = NotFound desc = could not find container \"080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298\": container with ID starting with 080b2b1c842ff87b5a562eddae8288a107aa60a42c1d0487a76a714792276298 not found: ID does not exist" Jul 27 15:37:19.371541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.371499 2553 scope.go:117] "RemoveContainer" containerID="615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655" Jul 27 15:37:19.371725 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:37:19.371702 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655\": container with ID starting with 615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655 not found: ID does not exist" containerID="615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655" Jul 27 15:37:19.371771 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.371731 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655"} err="failed to get container status \"615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655\": rpc error: code = NotFound desc = could not find container \"615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655\": container with ID starting with 615332bd3275790b74b1f519f09733220a02a9a22ea4939a8fb9f089938cc655 not found: ID does not exist" Jul 27 15:37:19.375511 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.375488 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv"] Jul 27 15:37:19.378952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:19.378926 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3af40866-openshift-default-58768868f6-x5vxv"] Jul 27 15:37:21.192353 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:21.192316 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" path="/var/lib/kubelet/pods/1206a02c-077d-44e6-b850-9a6d144525f6/volumes" Jul 27 15:37:21.635222 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:21.635178 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:37:30.412000 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:30.411967 2553 generic.go:356] "Generic (PLEG): container finished" podID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerID="6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c" exitCode=0 Jul 27 15:37:30.412445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:30.412038 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" event={"ID":"647a510d-fc56-43d9-8a6d-a11ee39f2ff0","Type":"ContainerDied","Data":"6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c"} Jul 27 15:37:30.412487 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:30.412444 2553 scope.go:117] "RemoveContainer" containerID="6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c" Jul 27 15:37:30.413448 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:30.413418 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:37:30.413563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:30.413487 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:37:30.413563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:30.413517 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:37:31.338633 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:31.338531 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:31.418188 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:31.418147 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" event={"ID":"647a510d-fc56-43d9-8a6d-a11ee39f2ff0","Type":"ContainerStarted","Data":"2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a"} Jul 27 15:37:31.642795 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:31.642746 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:31.643023 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:31.642816 2553 patch_prober.go:28] interesting pod/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 27 15:37:31.643023 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:31.642868 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 27 15:37:32.642102 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:32.642036 2553 patch_prober.go:28] interesting pod/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 27 15:37:32.642723 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:32.642686 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 27 15:37:33.642370 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:33.642327 2553 patch_prober.go:28] interesting pod/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 27 15:37:33.642916 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:33.642396 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 27 15:37:34.641537 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:34.641492 2553 patch_prober.go:28] interesting pod/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 27 15:37:34.641808 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:34.641571 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 27 15:37:35.644896 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:35.644864 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:35.645306 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:35.645089 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:35.645847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:35.645826 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:37:41.555706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.555672 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p"] Jul 27 15:37:41.556225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.556095 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" Jul 27 15:37:41.556225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.556108 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" Jul 27 15:37:41.556225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.556181 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" Jul 27 15:37:41.556225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.556189 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" Jul 27 15:37:41.558281 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.558266 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:41.560351 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.560329 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 15:37:41.560491 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.560404 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgfwg\"" Jul 27 15:37:41.564851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.564830 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p"] Jul 27 15:37:41.589169 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.589136 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1dcb680-acf4-4f05-a61b-8bd6797ef869-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p\" (UID: \"f1dcb680-acf4-4f05-a61b-8bd6797ef869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:41.690242 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.690202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1dcb680-acf4-4f05-a61b-8bd6797ef869-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p\" (UID: \"f1dcb680-acf4-4f05-a61b-8bd6797ef869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:41.690853 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.690831 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1dcb680-acf4-4f05-a61b-8bd6797ef869-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p\" (UID: \"f1dcb680-acf4-4f05-a61b-8bd6797ef869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:41.871420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:41.871385 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:42.210141 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:42.210103 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p"] Jul 27 15:37:42.211713 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:37:42.211683 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dcb680_acf4_4f05_a61b_8bd6797ef869.slice/crio-cda315815b9b65a17e380a0f2cfb63b5d4b334beaf6bd63819ea9df0e01ec39a WatchSource:0}: Error finding container cda315815b9b65a17e380a0f2cfb63b5d4b334beaf6bd63819ea9df0e01ec39a: Status 404 returned error can't find the container with id cda315815b9b65a17e380a0f2cfb63b5d4b334beaf6bd63819ea9df0e01ec39a Jul 27 15:37:42.481907 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:42.481816 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" event={"ID":"f1dcb680-acf4-4f05-a61b-8bd6797ef869","Type":"ContainerStarted","Data":"fb5dda318a3cdd050f2a7391fc7bdb79ca7576b641608cb73ed1968eea816f50"} Jul 27 15:37:42.481907 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:42.481867 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" event={"ID":"f1dcb680-acf4-4f05-a61b-8bd6797ef869","Type":"ContainerStarted","Data":"cda315815b9b65a17e380a0f2cfb63b5d4b334beaf6bd63819ea9df0e01ec39a"} Jul 27 15:37:42.496158 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:42.496106 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" podStartSLOduration=1.496091198 podStartE2EDuration="1.496091198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:37:42.494258217 +0000 UTC m=+1965.775608291" watchObservedRunningTime="2026-07-27 15:37:42.496091198 +0000 UTC m=+1965.777441271" Jul 27 15:37:43.487449 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:43.487409 2553 generic.go:356] "Generic (PLEG): container finished" podID="f1dcb680-acf4-4f05-a61b-8bd6797ef869" containerID="fb5dda318a3cdd050f2a7391fc7bdb79ca7576b641608cb73ed1968eea816f50" exitCode=0 Jul 27 15:37:43.487896 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:43.487499 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" event={"ID":"f1dcb680-acf4-4f05-a61b-8bd6797ef869","Type":"ContainerDied","Data":"fb5dda318a3cdd050f2a7391fc7bdb79ca7576b641608cb73ed1968eea816f50"} Jul 27 15:37:44.625689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:44.625664 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:44.715733 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:44.715700 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1dcb680-acf4-4f05-a61b-8bd6797ef869-maas-api-service-ca\") pod \"f1dcb680-acf4-4f05-a61b-8bd6797ef869\" (UID: \"f1dcb680-acf4-4f05-a61b-8bd6797ef869\") " Jul 27 15:37:44.716116 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:44.716094 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dcb680-acf4-4f05-a61b-8bd6797ef869-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1dcb680-acf4-4f05-a61b-8bd6797ef869" (UID: "f1dcb680-acf4-4f05-a61b-8bd6797ef869"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:37:44.817160 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:44.817035 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1dcb680-acf4-4f05-a61b-8bd6797ef869-maas-api-service-ca\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:37:45.498290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:45.498197 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" Jul 27 15:37:45.498290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:45.498230 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p" event={"ID":"f1dcb680-acf4-4f05-a61b-8bd6797ef869","Type":"ContainerDied","Data":"cda315815b9b65a17e380a0f2cfb63b5d4b334beaf6bd63819ea9df0e01ec39a"} Jul 27 15:37:45.498290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:45.498279 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda315815b9b65a17e380a0f2cfb63b5d4b334beaf6bd63819ea9df0e01ec39a" Jul 27 15:37:55.036827 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:55.036738 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:37:56.618881 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:56.618842 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p"] Jul 27 15:37:56.621435 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:56.621410 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b2ba96cb-mkf5p"] Jul 27 15:37:56.756796 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:56.756757 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r"] Jul 27 15:37:56.757104 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:56.757072 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" containerID="cri-o://2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a" gracePeriod=30 Jul 27 15:37:57.192897 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:37:57.192864 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1dcb680-acf4-4f05-a61b-8bd6797ef869" path="/var/lib/kubelet/pods/f1dcb680-acf4-4f05-a61b-8bd6797ef869/volumes" Jul 27 15:38:02.037754 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.037726 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:38:02.180791 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180749 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-data\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.180999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180806 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-envoy\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.180999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180842 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-certs\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.180999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180893 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-credential-socket\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.180999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180915 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-socket\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.180999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180937 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-podinfo\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.180999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.180976 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtfmg\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-kube-api-access-mtfmg\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.181322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181001 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istiod-ca-cert\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.181322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181028 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-token\") pod \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\" (UID: \"647a510d-fc56-43d9-8a6d-a11ee39f2ff0\") " Jul 27 15:38:02.181322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181220 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:38:02.181322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181244 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:38:02.181322 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181207 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:38:02.181585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181495 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:38:02.181674 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181654 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.181740 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181678 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.181740 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181694 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.181885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.181860 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-data" (OuterVolumeSpecName: "istio-data") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:38:02.183372 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.183339 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-token" (OuterVolumeSpecName: "istio-token") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:38:02.183493 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.183389 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:38:02.183715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.183691 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-kube-api-access-mtfmg" (OuterVolumeSpecName: "kube-api-access-mtfmg") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "kube-api-access-mtfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:38:02.183872 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.183853 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "647a510d-fc56-43d9-8a6d-a11ee39f2ff0" (UID: "647a510d-fc56-43d9-8a6d-a11ee39f2ff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:38:02.283200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.283106 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.283200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.283139 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.283200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.283148 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfmg\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-kube-api-access-mtfmg\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.283200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.283157 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.283200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.283165 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.283200 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.283173 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647a510d-fc56-43d9-8a6d-a11ee39f2ff0-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:38:02.586028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.585940 2553 generic.go:356] "Generic (PLEG): container finished" podID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerID="2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a" exitCode=0 Jul 27 15:38:02.586028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.585988 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" Jul 27 15:38:02.586028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.586006 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" event={"ID":"647a510d-fc56-43d9-8a6d-a11ee39f2ff0","Type":"ContainerDied","Data":"2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a"} Jul 27 15:38:02.586297 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.586046 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r" event={"ID":"647a510d-fc56-43d9-8a6d-a11ee39f2ff0","Type":"ContainerDied","Data":"0a81752afa59d88dc6e412fd18b21b70bf9f7a225b13d3fb30f6595932e67d55"} Jul 27 15:38:02.586297 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.586073 2553 scope.go:117] "RemoveContainer" containerID="2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a" Jul 27 15:38:02.595389 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.595369 2553 scope.go:117] "RemoveContainer" containerID="6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c" Jul 27 15:38:02.603612 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.603593 2553 scope.go:117] "RemoveContainer" containerID="2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a" Jul 27 15:38:02.603881 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:38:02.603858 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a\": container with ID starting with 2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a not found: ID does not exist" containerID="2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a" Jul 27 15:38:02.603958 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.603889 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a"} err="failed to get container status \"2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a\": rpc error: code = NotFound desc = could not find container \"2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a\": container with ID starting with 2a867f3382b393b5720965f4f7ab7397537bdbe13fc9ca128c6c5f3bdfac444a not found: ID does not exist" Jul 27 15:38:02.603958 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.603907 2553 scope.go:117] "RemoveContainer" containerID="6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c" Jul 27 15:38:02.604166 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:38:02.604142 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c\": container with ID starting with 6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c not found: ID does not exist" containerID="6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c" Jul 27 15:38:02.604219 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.604177 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c"} err="failed to get container status \"6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c\": rpc error: code = NotFound desc = could not find container \"6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c\": container with ID starting with 6f08191f7d7aa9e3326dfad379ccf2887d38e9cdbbae3aeb8ca712fb330ef25c not found: ID does not exist" Jul 27 15:38:02.610271 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.610245 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r"] Jul 27 15:38:02.613437 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:02.613414 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b2ba96cb-openshift-default-57c4649cdd-wdb8r"] Jul 27 15:38:03.191039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:38:03.191007 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" path="/var/lib/kubelet/pods/647a510d-fc56-43d9-8a6d-a11ee39f2ff0/volumes" Jul 27 15:39:20.946202 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:20.946114 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:39:30.341795 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:30.341753 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:39:34.244107 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244068 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t"] Jul 27 15:39:34.244585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244483 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" Jul 27 15:39:34.244585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244496 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" Jul 27 15:39:34.244585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244505 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1dcb680-acf4-4f05-a61b-8bd6797ef869" containerName="revoke-keys" Jul 27 15:39:34.244585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244511 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1dcb680-acf4-4f05-a61b-8bd6797ef869" containerName="revoke-keys" Jul 27 15:39:34.244585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244534 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" Jul 27 15:39:34.244585 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244539 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1206a02c-077d-44e6-b850-9a6d144525f6" containerName="istio-proxy" Jul 27 15:39:34.244900 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244593 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" Jul 27 15:39:34.244900 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.244605 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1dcb680-acf4-4f05-a61b-8bd6797ef869" containerName="revoke-keys" Jul 27 15:39:34.247588 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.247570 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.249465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.249448 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-22f7c4a2-openshift-default-dockercfg-trs6x\"" Jul 27 15:39:34.266158 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.266128 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t"] Jul 27 15:39:34.351268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351230 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-envoy\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351552 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351532 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c52626-7608-4032-96c5-a8dcb6cae8f9-istiod-ca-cert\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351699 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351664 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtbl\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-kube-api-access-kgtbl\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351747 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-token\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-credential-socket\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351994 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351889 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-socket\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351994 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351914 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-data\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351994 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351940 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-certs\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.351994 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.351969 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-podinfo\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452566 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452524 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-envoy\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452566 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452565 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c52626-7608-4032-96c5-a8dcb6cae8f9-istiod-ca-cert\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtbl\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-kube-api-access-kgtbl\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-token\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452634 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-credential-socket\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-socket\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452698 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-data\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-certs\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.452845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.452742 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-podinfo\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.453272 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.453145 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-socket\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.453272 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.453236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-credential-socket\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.453382 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.453290 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-certs\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.453382 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.453304 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-data\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.453670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.453648 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c52626-7608-4032-96c5-a8dcb6cae8f9-istiod-ca-cert\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.455065 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.455031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-envoy\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.455291 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.455273 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-podinfo\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.461487 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.461466 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-token\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.461894 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.461872 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtbl\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-kube-api-access-kgtbl\") pod \"e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.558373 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.558281 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:34.709575 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.709546 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t"] Jul 27 15:39:34.711853 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:39:34.711821 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c52626_7608_4032_96c5_a8dcb6cae8f9.slice/crio-a8f193d4c2d98f28f785b1e73b0d5e91f493bf0792be5abaf3f1219fecf746d0 WatchSource:0}: Error finding container a8f193d4c2d98f28f785b1e73b0d5e91f493bf0792be5abaf3f1219fecf746d0: Status 404 returned error can't find the container with id a8f193d4c2d98f28f785b1e73b0d5e91f493bf0792be5abaf3f1219fecf746d0 Jul 27 15:39:34.713933 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.713897 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:39:34.714004 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.713961 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:39:34.714004 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:34.713988 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:39:35.031522 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:35.031487 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" event={"ID":"33c52626-7608-4032-96c5-a8dcb6cae8f9","Type":"ContainerStarted","Data":"fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4"} Jul 27 15:39:35.031738 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:35.031541 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" event={"ID":"33c52626-7608-4032-96c5-a8dcb6cae8f9","Type":"ContainerStarted","Data":"a8f193d4c2d98f28f785b1e73b0d5e91f493bf0792be5abaf3f1219fecf746d0"} Jul 27 15:39:35.050042 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:35.049991 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podStartSLOduration=1.049973527 podStartE2EDuration="1.049973527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:39:35.048425798 +0000 UTC m=+2078.329775872" watchObservedRunningTime="2026-07-27 15:39:35.049973527 +0000 UTC m=+2078.331323674" Jul 27 15:39:35.559127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:35.559091 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:35.563922 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:35.563899 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:36.036901 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:36.036867 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:36.037738 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:36.037714 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:42.384177 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:42.384139 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:39:51.038952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.038911 2553 patch_prober.go:28] interesting pod/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 27 15:39:51.039420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.038975 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 27 15:39:51.110602 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.110571 2553 generic.go:356] "Generic (PLEG): container finished" podID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerID="fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4" exitCode=0 Jul 27 15:39:51.110778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.110648 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" event={"ID":"33c52626-7608-4032-96c5-a8dcb6cae8f9","Type":"ContainerDied","Data":"fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4"} Jul 27 15:39:51.111126 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.111096 2553 scope.go:117] "RemoveContainer" containerID="fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4" Jul 27 15:39:51.112117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.112093 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:39:51.112176 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.112159 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:39:51.112216 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:51.112189 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:39:52.117333 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:52.117298 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" event={"ID":"33c52626-7608-4032-96c5-a8dcb6cae8f9","Type":"ContainerStarted","Data":"494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3"} Jul 27 15:39:52.558713 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:52.558634 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:52.558866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:52.558792 2553 patch_prober.go:28] interesting pod/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 27 15:39:52.558866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:52.558830 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 27 15:39:53.558871 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:53.558831 2553 patch_prober.go:28] interesting pod/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 27 15:39:53.559386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:53.558896 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 27 15:39:54.558948 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:54.558907 2553 patch_prober.go:28] interesting pod/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 27 15:39:54.559394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:54.558964 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 27 15:39:55.559544 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:55.559503 2553 patch_prober.go:28] interesting pod/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 27 15:39:55.559952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:55.559571 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 27 15:39:56.562343 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:56.562312 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:56.562748 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:56.562631 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:56.563342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:56.563325 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:39:57.310354 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:57.310324 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:39:57.317775 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:39:57.317751 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:40:02.040762 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:02.040726 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:40:10.541533 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:10.541494 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t"] Jul 27 15:40:10.542019 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:10.541789 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" containerID="cri-o://494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3" gracePeriod=30 Jul 27 15:40:11.564988 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:11.564952 2553 patch_prober.go:28] interesting pod/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 15:40:11.565400 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:11.565024 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 15:40:15.787352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.787327 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:40:15.827851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.827775 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgtbl\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-kube-api-access-kgtbl\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.827851 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.827818 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-podinfo\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828038 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.827856 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-credential-socket\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828038 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.827956 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c52626-7608-4032-96c5-a8dcb6cae8f9-istiod-ca-cert\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828038 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828003 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-certs\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828062 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-envoy\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828107 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-data\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828131 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-socket\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828148 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:40:15.828225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828191 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-token\") pod \"33c52626-7608-4032-96c5-a8dcb6cae8f9\" (UID: \"33c52626-7608-4032-96c5-a8dcb6cae8f9\") " Jul 27 15:40:15.828480 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828312 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:40:15.828480 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828353 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c52626-7608-4032-96c5-a8dcb6cae8f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:40:15.828582 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828564 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.828640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828572 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:40:15.828640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828584 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c52626-7608-4032-96c5-a8dcb6cae8f9-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.828640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.828599 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.829290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.829260 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:40:15.830464 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.830437 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:40:15.830569 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.830494 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:40:15.830569 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.830501 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-kube-api-access-kgtbl" (OuterVolumeSpecName: "kube-api-access-kgtbl") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "kube-api-access-kgtbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:40:15.830960 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.830944 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33c52626-7608-4032-96c5-a8dcb6cae8f9" (UID: "33c52626-7608-4032-96c5-a8dcb6cae8f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:40:15.929267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.929230 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.929267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.929261 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.929267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.929270 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c52626-7608-4032-96c5-a8dcb6cae8f9-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.929267 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.929279 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.929542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.929287 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgtbl\" (UniqueName: \"kubernetes.io/projected/33c52626-7608-4032-96c5-a8dcb6cae8f9-kube-api-access-kgtbl\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:15.929542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:15.929297 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c52626-7608-4032-96c5-a8dcb6cae8f9-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:40:16.242143 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.242101 2553 generic.go:356] "Generic (PLEG): container finished" podID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerID="494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3" exitCode=0 Jul 27 15:40:16.242365 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.242160 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" Jul 27 15:40:16.242365 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.242160 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" event={"ID":"33c52626-7608-4032-96c5-a8dcb6cae8f9","Type":"ContainerDied","Data":"494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3"} Jul 27 15:40:16.242365 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.242285 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t" event={"ID":"33c52626-7608-4032-96c5-a8dcb6cae8f9","Type":"ContainerDied","Data":"a8f193d4c2d98f28f785b1e73b0d5e91f493bf0792be5abaf3f1219fecf746d0"} Jul 27 15:40:16.242365 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.242317 2553 scope.go:117] "RemoveContainer" containerID="494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3" Jul 27 15:40:16.251872 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.251854 2553 scope.go:117] "RemoveContainer" containerID="fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4" Jul 27 15:40:16.260017 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.259999 2553 scope.go:117] "RemoveContainer" containerID="494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3" Jul 27 15:40:16.260262 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:40:16.260245 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3\": container with ID starting with 494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3 not found: ID does not exist" containerID="494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3" Jul 27 15:40:16.260316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.260269 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3"} err="failed to get container status \"494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3\": rpc error: code = NotFound desc = could not find container \"494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3\": container with ID starting with 494c57e73b3e5f5765b6713072ac74bc655a2961813dbce7c791ea14b7fe6cc3 not found: ID does not exist" Jul 27 15:40:16.260316 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.260285 2553 scope.go:117] "RemoveContainer" containerID="fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4" Jul 27 15:40:16.260489 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:40:16.260474 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4\": container with ID starting with fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4 not found: ID does not exist" containerID="fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4" Jul 27 15:40:16.260537 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.260491 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4"} err="failed to get container status \"fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4\": rpc error: code = NotFound desc = could not find container \"fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4\": container with ID starting with fd54adaadab3a8cbd833594cb5e3876f01f773e527005727e5cd6d94640b9cf4 not found: ID does not exist" Jul 27 15:40:16.264406 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.264381 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t"] Jul 27 15:40:16.269148 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:16.269127 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-22f7c4a2-openshift-default-88ccf85d4-fw52t"] Jul 27 15:40:17.191494 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:17.191464 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" path="/var/lib/kubelet/pods/33c52626-7608-4032-96c5-a8dcb6cae8f9/volumes" Jul 27 15:40:30.800310 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800275 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq"] Jul 27 15:40:30.800898 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800880 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" Jul 27 15:40:30.800986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800899 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" Jul 27 15:40:30.800986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800915 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" Jul 27 15:40:30.800986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800925 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" Jul 27 15:40:30.800986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800942 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" Jul 27 15:40:30.800986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.800952 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" Jul 27 15:40:30.801163 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.801038 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" Jul 27 15:40:30.801163 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.801064 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c52626-7608-4032-96c5-a8dcb6cae8f9" containerName="istio-proxy" Jul 27 15:40:30.801163 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.801076 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="647a510d-fc56-43d9-8a6d-a11ee39f2ff0" containerName="istio-proxy" Jul 27 15:40:30.804605 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.804584 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.807244 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.807215 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-44b2dff2-openshift-default-dockercfg-xdr5z\"" Jul 27 15:40:30.815379 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.815354 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq"] Jul 27 15:40:30.864373 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864337 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-podinfo\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864381 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-credential-socket\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864408 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-certs\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864435 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs45w\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-kube-api-access-qs45w\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864488 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-envoy\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-data\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864705 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864543 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc9142d-b679-4807-8c7a-ef23d95c138a-istiod-ca-cert\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864705 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864607 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-socket\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.864705 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.864642 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-token\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965272 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965237 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-socket\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965272 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965283 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-token\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965507 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965411 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-podinfo\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965507 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-credential-socket\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965507 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965469 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-certs\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965507 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965484 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs45w\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-kube-api-access-qs45w\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965532 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-envoy\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965581 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-data\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965630 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc9142d-b679-4807-8c7a-ef23d95c138a-istiod-ca-cert\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965694 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-socket\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965744 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-credential-socket\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965950 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965877 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-certs\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.965950 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.965929 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-data\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.966342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.966323 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc9142d-b679-4807-8c7a-ef23d95c138a-istiod-ca-cert\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.967905 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.967871 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-envoy\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.968010 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.967916 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-podinfo\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.976965 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.976940 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-token\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:30.977194 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:30.977177 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs45w\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-kube-api-access-qs45w\") pod \"e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:31.115885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:31.115847 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:31.256799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:31.256767 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq"] Jul 27 15:40:31.260032 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:40:31.260000 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc9142d_b679_4807_8c7a_ef23d95c138a.slice/crio-18858818b5f2687b591140875838fb331a7b3c413678d5a904c06c4e544c4f4a WatchSource:0}: Error finding container 18858818b5f2687b591140875838fb331a7b3c413678d5a904c06c4e544c4f4a: Status 404 returned error can't find the container with id 18858818b5f2687b591140875838fb331a7b3c413678d5a904c06c4e544c4f4a Jul 27 15:40:31.261998 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:31.261972 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:40:31.262080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:31.262032 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:40:31.262127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:31.262075 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:40:31.317715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:31.317688 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" event={"ID":"dfc9142d-b679-4807-8c7a-ef23d95c138a","Type":"ContainerStarted","Data":"18858818b5f2687b591140875838fb331a7b3c413678d5a904c06c4e544c4f4a"} Jul 27 15:40:32.324383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:32.324351 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" event={"ID":"dfc9142d-b679-4807-8c7a-ef23d95c138a","Type":"ContainerStarted","Data":"70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca"} Jul 27 15:40:32.358445 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:32.358382 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podStartSLOduration=2.35836521 podStartE2EDuration="2.35836521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:40:32.356167304 +0000 UTC m=+2135.637517380" watchObservedRunningTime="2026-07-27 15:40:32.35836521 +0000 UTC m=+2135.639715340" Jul 27 15:40:33.116420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:33.116384 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:33.121287 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:33.121260 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:33.329060 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:33.329007 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:33.330022 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:33.330000 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:41.250230 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:41.250193 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:40:49.360127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.360072 2553 patch_prober.go:28] interesting pod/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 15:40:49.360594 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.360176 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 15:40:49.408504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.408472 2553 generic.go:356] "Generic (PLEG): container finished" podID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerID="70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca" exitCode=0 Jul 27 15:40:49.408657 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.408543 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" event={"ID":"dfc9142d-b679-4807-8c7a-ef23d95c138a","Type":"ContainerDied","Data":"70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca"} Jul 27 15:40:49.408955 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.408942 2553 scope.go:117] "RemoveContainer" containerID="70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca" Jul 27 15:40:49.409944 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.409920 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:40:49.410011 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.409984 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:40:49.410076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:49.410017 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:40:50.415548 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:50.415517 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" event={"ID":"dfc9142d-b679-4807-8c7a-ef23d95c138a","Type":"ContainerStarted","Data":"cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01"} Jul 27 15:40:51.117097 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:51.117035 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:51.117320 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:51.117299 2553 patch_prober.go:28] interesting pod/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 27 15:40:51.117374 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:51.117348 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 27 15:40:52.116709 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:52.116666 2553 patch_prober.go:28] interesting pod/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 27 15:40:52.117231 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:52.116728 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 27 15:40:52.458352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:52.458270 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:40:53.116811 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:53.116772 2553 patch_prober.go:28] interesting pod/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 27 15:40:53.117261 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:53.116846 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 27 15:40:54.116293 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:54.116251 2553 patch_prober.go:28] interesting pod/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 27 15:40:54.116483 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:54.116307 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 27 15:40:55.120539 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:55.120504 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:55.120985 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:55.120776 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:55.121434 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:55.121412 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:40:57.396219 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:40:57.396187 2553 scope.go:117] "RemoveContainer" containerID="3fc84be7557b1ea1bd794a579278b839b3384b0c4822d62156875843c1ca2727" Jul 27 15:41:08.649266 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:41:08.649227 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:41:27.453920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:41:27.453877 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:41:35.945036 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:41:35.944996 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:42:25.272147 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:25.272040 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq"] Jul 27 15:42:25.272739 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:25.272276 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" containerID="cri-o://cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01" gracePeriod=30 Jul 27 15:42:28.943845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:28.943805 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:42:30.528225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.528199 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:42:30.547352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547279 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-credential-socket\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547331 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-envoy\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547368 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc9142d-b679-4807-8c7a-ef23d95c138a-istiod-ca-cert\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547396 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-data\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547440 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-token\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547469 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-podinfo\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547503 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs45w\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-kube-api-access-qs45w\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547565 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-socket\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547595 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:42:30.547665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547603 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-certs\") pod \"dfc9142d-b679-4807-8c7a-ef23d95c138a\" (UID: \"dfc9142d-b679-4807-8c7a-ef23d95c138a\") " Jul 27 15:42:30.548080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547890 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.548080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.547968 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc9142d-b679-4807-8c7a-ef23d95c138a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:42:30.548186 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.548143 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:42:30.548448 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.548402 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:42:30.548550 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.548514 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:42:30.550895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.550865 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-kube-api-access-qs45w" (OuterVolumeSpecName: "kube-api-access-qs45w") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "kube-api-access-qs45w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:42:30.550986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.550914 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:42:30.550986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.550948 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:42:30.551276 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.551222 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfc9142d-b679-4807-8c7a-ef23d95c138a" (UID: "dfc9142d-b679-4807-8c7a-ef23d95c138a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:42:30.648534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648494 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648523 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc9142d-b679-4807-8c7a-ef23d95c138a-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648533 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648541 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648534 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648548 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc9142d-b679-4807-8c7a-ef23d95c138a-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648556 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs45w\" (UniqueName: \"kubernetes.io/projected/dfc9142d-b679-4807-8c7a-ef23d95c138a-kube-api-access-qs45w\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648568 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.648845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.648582 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc9142d-b679-4807-8c7a-ef23d95c138a-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:42:30.908618 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.908577 2553 generic.go:356] "Generic (PLEG): container finished" podID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerID="cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01" exitCode=0 Jul 27 15:42:30.908802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.908630 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" Jul 27 15:42:30.908802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.908659 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" event={"ID":"dfc9142d-b679-4807-8c7a-ef23d95c138a","Type":"ContainerDied","Data":"cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01"} Jul 27 15:42:30.908802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.908717 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq" event={"ID":"dfc9142d-b679-4807-8c7a-ef23d95c138a","Type":"ContainerDied","Data":"18858818b5f2687b591140875838fb331a7b3c413678d5a904c06c4e544c4f4a"} Jul 27 15:42:30.908802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.908739 2553 scope.go:117] "RemoveContainer" containerID="cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01" Jul 27 15:42:30.917942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.917918 2553 scope.go:117] "RemoveContainer" containerID="70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca" Jul 27 15:42:30.926702 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.926680 2553 scope.go:117] "RemoveContainer" containerID="cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01" Jul 27 15:42:30.926967 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:42:30.926948 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01\": container with ID starting with cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01 not found: ID does not exist" containerID="cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01" Jul 27 15:42:30.927022 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.926977 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01"} err="failed to get container status \"cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01\": rpc error: code = NotFound desc = could not find container \"cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01\": container with ID starting with cd54e55ba2c5b1b3ef0cd3be00eddf8fa68fdbe722a1f486287d68097a921e01 not found: ID does not exist" Jul 27 15:42:30.927022 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.926996 2553 scope.go:117] "RemoveContainer" containerID="70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca" Jul 27 15:42:30.927289 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:42:30.927270 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca\": container with ID starting with 70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca not found: ID does not exist" containerID="70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca" Jul 27 15:42:30.927339 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.927298 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca"} err="failed to get container status \"70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca\": rpc error: code = NotFound desc = could not find container \"70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca\": container with ID starting with 70e121519a50d95052e04fac5991b154b116886a69f3bfa84a30682b910901ca not found: ID does not exist" Jul 27 15:42:30.939914 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.939884 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq"] Jul 27 15:42:30.947805 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:30.947771 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-44b2dff2-openshift-default-7786bc96d4-sclsq"] Jul 27 15:42:31.192449 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:31.192367 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" path="/var/lib/kubelet/pods/dfc9142d-b679-4807-8c7a-ef23d95c138a/volumes" Jul 27 15:42:38.945319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:42:38.945284 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:43:22.474212 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.474170 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj"] Jul 27 15:43:22.474790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.474748 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" Jul 27 15:43:22.474790 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.474763 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" Jul 27 15:43:22.474891 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.474869 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" Jul 27 15:43:22.474891 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.474883 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" Jul 27 15:43:22.486923 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.486894 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.488995 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.488968 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj"] Jul 27 15:43:22.489840 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.489818 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0d2dee-openshift-default-dockercfg-q76tn\"" Jul 27 15:43:22.510326 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510296 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istiod-ca-cert\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510483 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510335 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-envoy\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510483 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510362 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-token\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510483 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510467 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-credential-socket\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510508 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-certs\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510533 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8pq\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-kube-api-access-tn8pq\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510575 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-data\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510630 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-podinfo\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.510853 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.510665 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-socket\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611142 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istiod-ca-cert\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611190 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-envoy\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611303 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-token\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611383 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611368 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-credential-socket\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611389 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-certs\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611406 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8pq\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-kube-api-access-tn8pq\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611433 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-data\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-podinfo\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611518 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-socket\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611826 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-credential-socket\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611835 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-certs\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611876 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-socket\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.611963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611904 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-data\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.612094 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.611960 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istiod-ca-cert\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.613612 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.613586 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-envoy\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.613934 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.613916 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-podinfo\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.619765 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.619734 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-token\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.619886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.619823 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8pq\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-kube-api-access-tn8pq\") pod \"e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.799123 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.799015 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:22.937964 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.937909 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj"] Jul 27 15:43:22.940336 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:43:22.940306 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df1080a_5050_476e_8a91_2fb4e13d9b3f.slice/crio-a7ff412749d59b67a2977b39c50aa470a04d5d83dcc76418619cc717cb422b3f WatchSource:0}: Error finding container a7ff412749d59b67a2977b39c50aa470a04d5d83dcc76418619cc717cb422b3f: Status 404 returned error can't find the container with id a7ff412749d59b67a2977b39c50aa470a04d5d83dcc76418619cc717cb422b3f Jul 27 15:43:22.942349 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.942322 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:22.942416 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.942385 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:22.942462 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:22.942423 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:23.159118 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:23.159080 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" event={"ID":"5df1080a-5050-476e-8a91-2fb4e13d9b3f","Type":"ContainerStarted","Data":"a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d"} Jul 27 15:43:23.159327 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:23.159146 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" event={"ID":"5df1080a-5050-476e-8a91-2fb4e13d9b3f","Type":"ContainerStarted","Data":"a7ff412749d59b67a2977b39c50aa470a04d5d83dcc76418619cc717cb422b3f"} Jul 27 15:43:23.177115 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:23.177064 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podStartSLOduration=1.1770321240000001 podStartE2EDuration="1.177032124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:43:23.175829785 +0000 UTC m=+2306.457179871" watchObservedRunningTime="2026-07-27 15:43:23.177032124 +0000 UTC m=+2306.458382198" Jul 27 15:43:23.799680 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:23.799635 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:24.804312 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:24.804282 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:25.169503 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:25.169472 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:25.170924 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:25.170901 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:30.243471 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.243436 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6"] Jul 27 15:43:30.244088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.244011 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" Jul 27 15:43:30.244088 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.244030 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc9142d-b679-4807-8c7a-ef23d95c138a" containerName="istio-proxy" Jul 27 15:43:30.246391 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.246370 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.248348 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.248326 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-664c72-openshift-default-dockercfg-xdf48\"" Jul 27 15:43:30.258531 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.258502 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6"] Jul 27 15:43:30.260354 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.260331 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:43:30.278597 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278570 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-credential-socket\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278638 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-podinfo\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278711 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-data\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278785 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-socket\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278813 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-certs\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278838 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-envoy\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278878 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-token\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.278980 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.278921 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxf5\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-kube-api-access-ttxf5\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.279207 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.279010 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istiod-ca-cert\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.379828 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379789 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-credential-socket\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379850 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-podinfo\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379874 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-data\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379915 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-socket\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379938 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-certs\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-envoy\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.379999 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-token\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380023 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxf5\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-kube-api-access-ttxf5\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380088 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istiod-ca-cert\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380294 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-data\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380326 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-credential-socket\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380420 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-certs\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380714 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380690 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-socket\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.380866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.380845 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istiod-ca-cert\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.382494 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.382462 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-envoy\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.382588 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.382494 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-podinfo\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.387906 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.387883 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-token\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.388000 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.387961 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxf5\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-kube-api-access-ttxf5\") pod \"e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.558683 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.558586 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:30.706201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.706170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6"] Jul 27 15:43:30.708002 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:43:30.707979 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd355c3_4f13_4b88_b29f_97e1c8892e83.slice/crio-786fad78758f479cea2c91b6df61ae1600631cb5417f29bc90654736e425e0a1 WatchSource:0}: Error finding container 786fad78758f479cea2c91b6df61ae1600631cb5417f29bc90654736e425e0a1: Status 404 returned error can't find the container with id 786fad78758f479cea2c91b6df61ae1600631cb5417f29bc90654736e425e0a1 Jul 27 15:43:30.710190 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.710159 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:30.710298 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.710224 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:30.710298 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:30.710251 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:31.201466 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:31.201431 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" event={"ID":"bbd355c3-4f13-4b88-b29f-97e1c8892e83","Type":"ContainerStarted","Data":"9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3"} Jul 27 15:43:31.201669 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:31.201486 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" event={"ID":"bbd355c3-4f13-4b88-b29f-97e1c8892e83","Type":"ContainerStarted","Data":"786fad78758f479cea2c91b6df61ae1600631cb5417f29bc90654736e425e0a1"} Jul 27 15:43:31.222568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:31.222507 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podStartSLOduration=1.222489911 podStartE2EDuration="1.222489911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:43:31.221777036 +0000 UTC m=+2314.503127138" watchObservedRunningTime="2026-07-27 15:43:31.222489911 +0000 UTC m=+2314.503839985" Jul 27 15:43:31.559124 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:31.559001 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:31.564543 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:31.564513 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:32.206936 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:32.206905 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:32.208010 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:32.207984 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:38.847679 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:38.847639 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:43:39.255928 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:39.255895 2553 generic.go:356] "Generic (PLEG): container finished" podID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerID="a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d" exitCode=0 Jul 27 15:43:39.256137 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:39.255973 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" event={"ID":"5df1080a-5050-476e-8a91-2fb4e13d9b3f","Type":"ContainerDied","Data":"a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d"} Jul 27 15:43:39.256447 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:39.256432 2553 scope.go:117] "RemoveContainer" containerID="a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d" Jul 27 15:43:39.257487 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:39.257462 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:39.257541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:39.257528 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:39.257578 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:39.257557 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:40.169736 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:40.169707 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:40.262636 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:40.262595 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" event={"ID":"5df1080a-5050-476e-8a91-2fb4e13d9b3f","Type":"ContainerStarted","Data":"b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0"} Jul 27 15:43:40.799951 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:40.799917 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:40.800898 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:40.800873 2553 patch_prober.go:28] interesting pod/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 15:43:40.800999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:40.800946 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 15:43:41.800043 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:41.799997 2553 patch_prober.go:28] interesting pod/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 15:43:41.800488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:41.800081 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 15:43:42.800135 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:42.800101 2553 patch_prober.go:28] interesting pod/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 15:43:42.800613 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:42.800588 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 15:43:43.800196 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:43.800159 2553 patch_prober.go:28] interesting pod/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 15:43:43.800613 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:43.800216 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 15:43:44.804156 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:44.804119 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:44.804574 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:44.804407 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:44.805173 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:44.805151 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:43:48.121656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.121337 2553 patch_prober.go:28] interesting pod/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:44148->10.132.0.48:15021: read: connection reset by peer" start-of-body= Jul 27 15:43:48.121656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.121403 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:44148->10.132.0.48:15021: read: connection reset by peer" Jul 27 15:43:48.306167 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.306083 2553 generic.go:356] "Generic (PLEG): container finished" podID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerID="9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3" exitCode=0 Jul 27 15:43:48.306167 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.306156 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" event={"ID":"bbd355c3-4f13-4b88-b29f-97e1c8892e83","Type":"ContainerDied","Data":"9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3"} Jul 27 15:43:48.306649 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.306632 2553 scope.go:117] "RemoveContainer" containerID="9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3" Jul 27 15:43:48.307636 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.307611 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:48.307699 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.307686 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:48.307736 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.307719 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:43:48.447317 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:48.447267 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:43:49.312849 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:49.312762 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" event={"ID":"bbd355c3-4f13-4b88-b29f-97e1c8892e83","Type":"ContainerStarted","Data":"cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5"} Jul 27 15:43:49.558959 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:49.558923 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:49.559195 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:49.559171 2553 patch_prober.go:28] interesting pod/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 27 15:43:49.559247 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:49.559222 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 27 15:43:50.559404 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:50.559366 2553 patch_prober.go:28] interesting pod/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 27 15:43:50.559813 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:50.559420 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 27 15:43:51.558888 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:51.558855 2553 patch_prober.go:28] interesting pod/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 27 15:43:51.559101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:51.558908 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 27 15:43:52.559246 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:52.559211 2553 patch_prober.go:28] interesting pod/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 27 15:43:52.559734 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:52.559275 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 27 15:43:53.563089 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:53.563042 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:53.563581 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:53.563440 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:53.564164 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:53.564144 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:43:57.437685 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:43:57.437655 2553 scope.go:117] "RemoveContainer" containerID="fb5dda318a3cdd050f2a7391fc7bdb79ca7576b641608cb73ed1968eea816f50" Jul 27 15:44:01.537539 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:01.537502 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:44:18.944543 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:18.944505 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:44:48.091160 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.091122 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt"] Jul 27 15:44:48.093642 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.093625 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:48.095796 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.095775 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bgfwg\"" Jul 27 15:44:48.095917 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.095835 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 15:44:48.101455 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.101395 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt"] Jul 27 15:44:48.113341 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.113317 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc29ef65-d420-4abf-bfce-44add090f41d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt\" (UID: \"dc29ef65-d420-4abf-bfce-44add090f41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:48.214565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.214523 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc29ef65-d420-4abf-bfce-44add090f41d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt\" (UID: \"dc29ef65-d420-4abf-bfce-44add090f41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:48.215225 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.215205 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc29ef65-d420-4abf-bfce-44add090f41d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt\" (UID: \"dc29ef65-d420-4abf-bfce-44add090f41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:48.406040 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.406005 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:48.545523 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.545498 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt"] Jul 27 15:44:48.547380 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:44:48.547351 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc29ef65_d420_4abf_bfce_44add090f41d.slice/crio-a8945814a16e0bc38f8b4e496f60d2c986a0f2647d91803ee3ae242e8cf486d4 WatchSource:0}: Error finding container a8945814a16e0bc38f8b4e496f60d2c986a0f2647d91803ee3ae242e8cf486d4: Status 404 returned error can't find the container with id a8945814a16e0bc38f8b4e496f60d2c986a0f2647d91803ee3ae242e8cf486d4 Jul 27 15:44:48.611358 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:48.611326 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" event={"ID":"dc29ef65-d420-4abf-bfce-44add090f41d","Type":"ContainerStarted","Data":"a8945814a16e0bc38f8b4e496f60d2c986a0f2647d91803ee3ae242e8cf486d4"} Jul 27 15:44:49.617428 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:49.617390 2553 generic.go:356] "Generic (PLEG): container finished" podID="dc29ef65-d420-4abf-bfce-44add090f41d" containerID="0b6f7fc45daed3e6008bf89cc06ece9dcf71fe6c57b5922917a2a2069a4d5f5a" exitCode=0 Jul 27 15:44:49.617890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:49.617457 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" event={"ID":"dc29ef65-d420-4abf-bfce-44add090f41d","Type":"ContainerDied","Data":"0b6f7fc45daed3e6008bf89cc06ece9dcf71fe6c57b5922917a2a2069a4d5f5a"} Jul 27 15:44:50.757949 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:50.757923 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:50.838555 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:50.838520 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc29ef65-d420-4abf-bfce-44add090f41d-maas-api-service-ca\") pod \"dc29ef65-d420-4abf-bfce-44add090f41d\" (UID: \"dc29ef65-d420-4abf-bfce-44add090f41d\") " Jul 27 15:44:50.838877 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:50.838854 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc29ef65-d420-4abf-bfce-44add090f41d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc29ef65-d420-4abf-bfce-44add090f41d" (UID: "dc29ef65-d420-4abf-bfce-44add090f41d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:44:50.939644 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:50.939559 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc29ef65-d420-4abf-bfce-44add090f41d-maas-api-service-ca\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:44:51.628521 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:51.628489 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" Jul 27 15:44:51.629188 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:51.628489 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt" event={"ID":"dc29ef65-d420-4abf-bfce-44add090f41d","Type":"ContainerDied","Data":"a8945814a16e0bc38f8b4e496f60d2c986a0f2647d91803ee3ae242e8cf486d4"} Jul 27 15:44:51.629188 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:51.629150 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8945814a16e0bc38f8b4e496f60d2c986a0f2647d91803ee3ae242e8cf486d4" Jul 27 15:44:52.164545 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:52.164503 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:44:57.339925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:57.339893 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:44:57.348885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:44:57.348861 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:45:03.150587 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:03.150546 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt"] Jul 27 15:45:03.155972 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:03.155941 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d2dee-kx8jt"] Jul 27 15:45:03.197352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:03.197320 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc29ef65-d420-4abf-bfce-44add090f41d" path="/var/lib/kubelet/pods/dc29ef65-d420-4abf-bfce-44add090f41d/volumes" Jul 27 15:45:07.235592 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:07.235554 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:45:10.118454 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:10.118422 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj"] Jul 27 15:45:10.118885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:10.118694 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" containerID="cri-o://b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0" gracePeriod=30 Jul 27 15:45:14.807524 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:14.807488 2553 patch_prober.go:28] interesting pod/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 15:45:14.807921 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:14.807554 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 15:45:15.380985 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.380925 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:45:15.467278 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467237 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-envoy\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467278 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467281 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-token\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467526 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467331 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-data\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467526 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467363 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istiod-ca-cert\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467526 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467389 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-credential-socket\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467526 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467416 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-certs\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467727 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467536 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn8pq\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-kube-api-access-tn8pq\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467727 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467585 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-socket\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467727 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467641 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-podinfo\") pod \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\" (UID: \"5df1080a-5050-476e-8a91-2fb4e13d9b3f\") " Jul 27 15:45:15.467890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467786 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:15.467946 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467888 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:45:15.467946 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.467907 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:15.468304 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468268 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:15.468450 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468412 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:15.468615 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468593 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.468690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468620 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.468690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468634 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.468690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468648 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.468690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.468662 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.469586 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.469564 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:45:15.469816 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.469795 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:15.469896 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.469880 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:45:15.469967 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.469949 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-kube-api-access-tn8pq" (OuterVolumeSpecName: "kube-api-access-tn8pq") pod "5df1080a-5050-476e-8a91-2fb4e13d9b3f" (UID: "5df1080a-5050-476e-8a91-2fb4e13d9b3f"). InnerVolumeSpecName "kube-api-access-tn8pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:45:15.570080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.570013 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.570080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.570043 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.570080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.570077 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn8pq\" (UniqueName: \"kubernetes.io/projected/5df1080a-5050-476e-8a91-2fb4e13d9b3f-kube-api-access-tn8pq\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.570080 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.570086 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5df1080a-5050-476e-8a91-2fb4e13d9b3f-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:15.743386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.743298 2553 generic.go:356] "Generic (PLEG): container finished" podID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerID="b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0" exitCode=0 Jul 27 15:45:15.743386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.743348 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" Jul 27 15:45:15.743582 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.743386 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" event={"ID":"5df1080a-5050-476e-8a91-2fb4e13d9b3f","Type":"ContainerDied","Data":"b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0"} Jul 27 15:45:15.743582 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.743441 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj" event={"ID":"5df1080a-5050-476e-8a91-2fb4e13d9b3f","Type":"ContainerDied","Data":"a7ff412749d59b67a2977b39c50aa470a04d5d83dcc76418619cc717cb422b3f"} Jul 27 15:45:15.743582 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.743464 2553 scope.go:117] "RemoveContainer" containerID="b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0" Jul 27 15:45:15.759669 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.759647 2553 scope.go:117] "RemoveContainer" containerID="a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d" Jul 27 15:45:15.767883 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.767858 2553 scope.go:117] "RemoveContainer" containerID="b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0" Jul 27 15:45:15.768413 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:45:15.768382 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0\": container with ID starting with b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0 not found: ID does not exist" containerID="b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0" Jul 27 15:45:15.768538 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.768426 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0"} err="failed to get container status \"b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0\": rpc error: code = NotFound desc = could not find container \"b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0\": container with ID starting with b35add6d5204f6e3f670624a7f60b14b26a0535e736873179b2c25506ff1b5e0 not found: ID does not exist" Jul 27 15:45:15.768538 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.768449 2553 scope.go:117] "RemoveContainer" containerID="a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d" Jul 27 15:45:15.768713 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:45:15.768697 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d\": container with ID starting with a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d not found: ID does not exist" containerID="a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d" Jul 27 15:45:15.768771 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.768721 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d"} err="failed to get container status \"a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d\": rpc error: code = NotFound desc = could not find container \"a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d\": container with ID starting with a665f2c743387b638721c760c8d4c0e5450134b143ef6b78e0962f1e4023a57d not found: ID does not exist" Jul 27 15:45:15.770572 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.770552 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj"] Jul 27 15:45:15.774570 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:15.774547 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0d2dee-openshift-default-589c6c9c8-54jtj"] Jul 27 15:45:17.191730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:17.191697 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" path="/var/lib/kubelet/pods/5df1080a-5050-476e-8a91-2fb4e13d9b3f/volumes" Jul 27 15:45:23.640837 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:23.640745 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:45:32.046694 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.046658 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6"] Jul 27 15:45:32.047111 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.046884 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" containerID="cri-o://cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5" gracePeriod=30 Jul 27 15:45:32.927411 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927381 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r"] Jul 27 15:45:32.927773 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927759 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc29ef65-d420-4abf-bfce-44add090f41d" containerName="revoke-keys" Jul 27 15:45:32.928087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927774 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc29ef65-d420-4abf-bfce-44add090f41d" containerName="revoke-keys" Jul 27 15:45:32.928087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927784 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" Jul 27 15:45:32.928087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927791 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" Jul 27 15:45:32.928087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927867 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc29ef65-d420-4abf-bfce-44add090f41d" containerName="revoke-keys" Jul 27 15:45:32.928087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927877 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" Jul 27 15:45:32.928087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.927887 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" Jul 27 15:45:32.931120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.931102 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:32.933265 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.933246 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fcb199-openshift-default-dockercfg-sktx7\"" Jul 27 15:45:32.941179 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:32.941155 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r"] Jul 27 15:45:33.026646 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026605 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-podinfo\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026646 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026643 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-token\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026677 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwx7w\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-kube-api-access-lwx7w\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026768 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-certs\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026786 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-envoy\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026802 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-data\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026824 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istiod-ca-cert\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.026909 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-credential-socket\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.027244 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.026929 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-socket\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.127999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.127963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-podinfo\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.127999 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.127996 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-token\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128015 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwx7w\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-kube-api-access-lwx7w\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128168 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-certs\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-envoy\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128222 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-data\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128253 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istiod-ca-cert\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128456 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-credential-socket\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128565 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128504 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-socket\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128923 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128664 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-certs\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128923 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128864 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-credential-socket\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.128923 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128916 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-socket\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.129093 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128982 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istiod-ca-cert\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.129093 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.128990 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-data\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.130552 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.130526 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-envoy\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.130693 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.130606 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-podinfo\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.136821 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.136799 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwx7w\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-kube-api-access-lwx7w\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.136948 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.136925 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-token\") pod \"e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.242438 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.242344 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:33.389406 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.389382 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r"] Jul 27 15:45:33.391458 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:45:33.391432 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff9f4f4_ce26_4612_8f70_1d21c2c3c945.slice/crio-b0c3729576e6839b87147d6eff13c2ccb31df81402ad4a294da09ac57ccafb17 WatchSource:0}: Error finding container b0c3729576e6839b87147d6eff13c2ccb31df81402ad4a294da09ac57ccafb17: Status 404 returned error can't find the container with id b0c3729576e6839b87147d6eff13c2ccb31df81402ad4a294da09ac57ccafb17 Jul 27 15:45:33.393620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.393586 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:33.393732 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.393648 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:33.393732 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.393677 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:33.837720 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.837683 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" event={"ID":"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945","Type":"ContainerStarted","Data":"cbc9ff74b60c00c792a6449b3bf49f5bdcb9898a17ac75bb1a807c961f192d62"} Jul 27 15:45:33.837906 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.837735 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" event={"ID":"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945","Type":"ContainerStarted","Data":"b0c3729576e6839b87147d6eff13c2ccb31df81402ad4a294da09ac57ccafb17"} Jul 27 15:45:33.857913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:33.857865 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podStartSLOduration=1.857845562 podStartE2EDuration="1.857845562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:45:33.856571365 +0000 UTC m=+2437.137921441" watchObservedRunningTime="2026-07-27 15:45:33.857845562 +0000 UTC m=+2437.139195637" Jul 27 15:45:34.243198 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:34.243169 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:34.248092 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:34.248068 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:34.842845 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:34.842813 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:34.843828 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:34.843801 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:37.309149 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.309040 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:45:37.368502 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368467 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-credential-socket\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368515 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-token\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368543 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttxf5\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-kube-api-access-ttxf5\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368585 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-data\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368695 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368621 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-envoy\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368907 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368744 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istiod-ca-cert\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368907 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368834 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-podinfo\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.368907 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368744 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:37.368907 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368878 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-socket\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.369173 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.368918 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-certs\") pod \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\" (UID: \"bbd355c3-4f13-4b88-b29f-97e1c8892e83\") " Jul 27 15:45:37.369173 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.369040 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:45:37.369404 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.369284 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.369404 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.369388 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:37.369521 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.369414 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.369521 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.369504 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:37.369640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.369524 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:37.370905 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.370875 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:45:37.370905 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.370882 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:45:37.371076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.370918 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-kube-api-access-ttxf5" (OuterVolumeSpecName: "kube-api-access-ttxf5") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "kube-api-access-ttxf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:45:37.371076 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.370931 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbd355c3-4f13-4b88-b29f-97e1c8892e83" (UID: "bbd355c3-4f13-4b88-b29f-97e1c8892e83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:45:37.470134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470099 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.470134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470126 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.470134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470136 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.470134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470145 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.470414 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470154 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.470414 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470164 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxf5\" (UniqueName: \"kubernetes.io/projected/bbd355c3-4f13-4b88-b29f-97e1c8892e83-kube-api-access-ttxf5\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.470414 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.470174 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd355c3-4f13-4b88-b29f-97e1c8892e83-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:45:37.858750 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.858709 2553 generic.go:356] "Generic (PLEG): container finished" podID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerID="cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5" exitCode=0 Jul 27 15:45:37.858944 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.858761 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" Jul 27 15:45:37.858944 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.858776 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" event={"ID":"bbd355c3-4f13-4b88-b29f-97e1c8892e83","Type":"ContainerDied","Data":"cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5"} Jul 27 15:45:37.858944 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.858828 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6" event={"ID":"bbd355c3-4f13-4b88-b29f-97e1c8892e83","Type":"ContainerDied","Data":"786fad78758f479cea2c91b6df61ae1600631cb5417f29bc90654736e425e0a1"} Jul 27 15:45:37.858944 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.858859 2553 scope.go:117] "RemoveContainer" containerID="cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5" Jul 27 15:45:37.869819 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.869801 2553 scope.go:117] "RemoveContainer" containerID="9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3" Jul 27 15:45:37.877630 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.877612 2553 scope.go:117] "RemoveContainer" containerID="cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5" Jul 27 15:45:37.877862 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:45:37.877844 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5\": container with ID starting with cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5 not found: ID does not exist" containerID="cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5" Jul 27 15:45:37.877912 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.877871 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5"} err="failed to get container status \"cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5\": rpc error: code = NotFound desc = could not find container \"cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5\": container with ID starting with cbe02a195ea6017f2155b510fa935844fd4f617f9a42442f6d956a8be1dc1fb5 not found: ID does not exist" Jul 27 15:45:37.877912 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.877889 2553 scope.go:117] "RemoveContainer" containerID="9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3" Jul 27 15:45:37.878160 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:45:37.878144 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3\": container with ID starting with 9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3 not found: ID does not exist" containerID="9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3" Jul 27 15:45:37.878224 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.878164 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3"} err="failed to get container status \"9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3\": rpc error: code = NotFound desc = could not find container \"9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3\": container with ID starting with 9545b353be673d3c036242d9619f4c73799b6b352a5f32717c7402addf55b6b3 not found: ID does not exist" Jul 27 15:45:37.884593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.884569 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6"] Jul 27 15:45:37.905738 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:37.905710 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-664c72-openshift-default-5b9f79d75b-jx5h6"] Jul 27 15:45:38.744953 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:38.744915 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:45:39.191395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:39.191363 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" path="/var/lib/kubelet/pods/bbd355c3-4f13-4b88-b29f-97e1c8892e83/volumes" Jul 27 15:45:41.014900 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.014861 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l"] Jul 27 15:45:41.015290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015251 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" Jul 27 15:45:41.015290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015263 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df1080a-5050-476e-8a91-2fb4e13d9b3f" containerName="istio-proxy" Jul 27 15:45:41.015290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015270 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" Jul 27 15:45:41.015290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015277 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" Jul 27 15:45:41.015426 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015294 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" Jul 27 15:45:41.015426 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015300 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" Jul 27 15:45:41.015426 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015359 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" Jul 27 15:45:41.015426 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.015366 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd355c3-4f13-4b88-b29f-97e1c8892e83" containerName="istio-proxy" Jul 27 15:45:41.048952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.048916 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l"] Jul 27 15:45:41.049390 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.049370 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.052689 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.052664 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fbf2d4-openshift-default-dockercfg-l9cgw\"" Jul 27 15:45:41.102733 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102680 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a583590-7b22-4918-ae8c-653b75710677-istiod-ca-cert\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102738 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-credential-socket\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102766 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-certs\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102787 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-data\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102841 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpqd\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-kube-api-access-gjpqd\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102897 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-envoy\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.102957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-istio-token\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103661 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.103321 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a583590-7b22-4918-ae8c-653b75710677-istio-podinfo\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.103661 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.103378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-socket\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.204809 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.204777 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-socket\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.204991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.204829 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a583590-7b22-4918-ae8c-653b75710677-istiod-ca-cert\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.204991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.204873 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-credential-socket\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.204991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.204896 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-certs\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.204991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.204924 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-data\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.204991 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.204967 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpqd\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-kube-api-access-gjpqd\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-envoy\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205225 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-istio-token\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205258 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-socket\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205260 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a583590-7b22-4918-ae8c-653b75710677-istio-podinfo\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205295 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-certs\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205533 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-credential-socket\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205564 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-data\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.205797 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.205766 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a583590-7b22-4918-ae8c-653b75710677-istiod-ca-cert\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.207504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.207483 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-envoy\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.207718 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.207702 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a583590-7b22-4918-ae8c-653b75710677-istio-podinfo\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.213170 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.213150 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-istio-token\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.213283 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.213224 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpqd\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-kube-api-access-gjpqd\") pod \"e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.362726 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.362695 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:41.543078 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.540592 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l"] Jul 27 15:45:41.550157 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:45:41.550125 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a583590_7b22_4918_ae8c_653b75710677.slice/crio-6a097d07bdb7294741277f49e0d091ff7e36e672db8d26ce959c1a5fa7f7071d WatchSource:0}: Error finding container 6a097d07bdb7294741277f49e0d091ff7e36e672db8d26ce959c1a5fa7f7071d: Status 404 returned error can't find the container with id 6a097d07bdb7294741277f49e0d091ff7e36e672db8d26ce959c1a5fa7f7071d Jul 27 15:45:41.552512 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.552480 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:41.552618 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.552558 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:41.552618 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.552604 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:41.882113 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.882076 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" event={"ID":"3a583590-7b22-4918-ae8c-653b75710677","Type":"ContainerStarted","Data":"c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699"} Jul 27 15:45:41.882308 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.882129 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" event={"ID":"3a583590-7b22-4918-ae8c-653b75710677","Type":"ContainerStarted","Data":"6a097d07bdb7294741277f49e0d091ff7e36e672db8d26ce959c1a5fa7f7071d"} Jul 27 15:45:41.902755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:41.902708 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podStartSLOduration=1.902692066 podStartE2EDuration="1.902692066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:45:41.90067207 +0000 UTC m=+2445.182022145" watchObservedRunningTime="2026-07-27 15:45:41.902692066 +0000 UTC m=+2445.184042140" Jul 27 15:45:42.362986 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:42.362952 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:42.368078 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:42.368024 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:42.886895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:42.886861 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:42.887883 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:42.887862 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:45:47.914472 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:47.914433 2553 generic.go:356] "Generic (PLEG): container finished" podID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerID="cbc9ff74b60c00c792a6449b3bf49f5bdcb9898a17ac75bb1a807c961f192d62" exitCode=0 Jul 27 15:45:47.915020 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:47.914490 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" event={"ID":"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945","Type":"ContainerDied","Data":"cbc9ff74b60c00c792a6449b3bf49f5bdcb9898a17ac75bb1a807c961f192d62"} Jul 27 15:45:47.915020 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:47.914999 2553 scope.go:117] "RemoveContainer" containerID="cbc9ff74b60c00c792a6449b3bf49f5bdcb9898a17ac75bb1a807c961f192d62" Jul 27 15:45:47.916112 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:47.916083 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:47.916197 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:47.916153 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:47.916197 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:47.916186 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:48.921698 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:48.921664 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" event={"ID":"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945","Type":"ContainerStarted","Data":"b2ce7422d2130bf704e4f9f784ad9cc01dd305d4397a28a29cc6c97c06eef123"} Jul 27 15:45:49.243347 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:49.243253 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:49.244505 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:49.244477 2553 patch_prober.go:28] interesting pod/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 27 15:45:49.244612 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:49.244527 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 27 15:45:49.843014 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:49.842976 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:50.243041 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:50.242979 2553 patch_prober.go:28] interesting pod/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 27 15:45:50.243572 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:50.243152 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 27 15:45:51.243933 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:51.243894 2553 patch_prober.go:28] interesting pod/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 27 15:45:51.244422 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:51.244134 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 27 15:45:51.245008 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:51.244986 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:45:52.242974 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:52.242932 2553 patch_prober.go:28] interesting pod/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 27 15:45:52.243217 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:52.243135 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 27 15:45:53.243395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:53.243356 2553 patch_prober.go:28] interesting pod/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 27 15:45:53.244026 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:53.243995 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 27 15:45:54.251125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:54.251097 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:54.252283 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:54.252260 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:45:58.930222 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:58.930178 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 15:45:58.930732 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:58.930263 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 15:45:59.992996 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:59.992959 2553 generic.go:356] "Generic (PLEG): container finished" podID="3a583590-7b22-4918-ae8c-653b75710677" containerID="c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699" exitCode=0 Jul 27 15:45:59.993574 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:59.993034 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" event={"ID":"3a583590-7b22-4918-ae8c-653b75710677","Type":"ContainerDied","Data":"c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699"} Jul 27 15:45:59.993658 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:59.993592 2553 scope.go:117] "RemoveContainer" containerID="c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699" Jul 27 15:45:59.994622 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:59.994595 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:59.994688 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:59.994676 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:45:59.994729 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:45:59.994707 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:46:00.999765 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:00.999731 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" event={"ID":"3a583590-7b22-4918-ae8c-653b75710677","Type":"ContainerStarted","Data":"4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b"} Jul 27 15:46:01.363461 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:01.363423 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:46:01.363677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:01.363656 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 27 15:46:01.363731 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:01.363705 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 27 15:46:02.363949 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:02.363910 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 27 15:46:02.364406 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:02.363967 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 27 15:46:03.363785 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:03.363740 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 27 15:46:03.364001 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:03.363804 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 27 15:46:04.363410 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:04.363374 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 27 15:46:04.363600 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:04.363430 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 27 15:46:05.363989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:05.363953 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 27 15:46:05.364380 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:05.364012 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 27 15:46:06.366504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:06.366475 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:46:06.366929 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:06.366736 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:46:06.367567 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:06.367546 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:46:17.940994 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:17.940950 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:46:22.939128 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:22.939097 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r"] Jul 27 15:46:22.939515 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:22.939341 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" containerID="cri-o://b2ce7422d2130bf704e4f9f784ad9cc01dd305d4397a28a29cc6c97c06eef123" gracePeriod=30 Jul 27 15:46:24.254528 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:24.254484 2553 patch_prober.go:28] interesting pod/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 15:46:24.254982 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:24.254548 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 15:46:28.147558 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.147524 2553 generic.go:356] "Generic (PLEG): container finished" podID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerID="b2ce7422d2130bf704e4f9f784ad9cc01dd305d4397a28a29cc6c97c06eef123" exitCode=0 Jul 27 15:46:28.147954 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.147594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" event={"ID":"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945","Type":"ContainerDied","Data":"b2ce7422d2130bf704e4f9f784ad9cc01dd305d4397a28a29cc6c97c06eef123"} Jul 27 15:46:28.147954 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.147688 2553 scope.go:117] "RemoveContainer" containerID="cbc9ff74b60c00c792a6449b3bf49f5bdcb9898a17ac75bb1a807c961f192d62" Jul 27 15:46:28.201847 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.201823 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:46:28.273384 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273295 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-certs\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273384 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273331 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-credential-socket\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273384 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273351 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-data\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273393 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-envoy\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273435 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-podinfo\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273470 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwx7w\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-kube-api-access-lwx7w\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273528 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-token\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273581 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-socket\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273620 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istiod-ca-cert\") pod \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\" (UID: \"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945\") " Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273633 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:28.273670 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273658 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:28.274101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273871 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:28.274101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273963 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.274101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.273986 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.274101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.274004 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.274323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.274192 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:46:28.274455 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.274428 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:28.275850 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.275817 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:46:28.275976 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.275877 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-kube-api-access-lwx7w" (OuterVolumeSpecName: "kube-api-access-lwx7w") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "kube-api-access-lwx7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:46:28.275976 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.275879 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:46:28.275976 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.275886 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" (UID: "2ff9f4f4-ce26-4612-8f70-1d21c2c3c945"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:28.374920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.374892 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.374920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.374917 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwx7w\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-kube-api-access-lwx7w\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.374920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.374927 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.375187 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.374936 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.375187 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.374944 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:28.375187 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:28.374952 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:29.153497 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:29.153469 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" Jul 27 15:46:29.153963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:29.153524 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r" event={"ID":"2ff9f4f4-ce26-4612-8f70-1d21c2c3c945","Type":"ContainerDied","Data":"b0c3729576e6839b87147d6eff13c2ccb31df81402ad4a294da09ac57ccafb17"} Jul 27 15:46:29.153963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:29.153588 2553 scope.go:117] "RemoveContainer" containerID="b2ce7422d2130bf704e4f9f784ad9cc01dd305d4397a28a29cc6c97c06eef123" Jul 27 15:46:29.178920 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:29.178880 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r"] Jul 27 15:46:29.182530 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:29.182503 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fcb199-openshift-default-77bbf74bd8-md85r"] Jul 27 15:46:29.192141 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:29.192117 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" path="/var/lib/kubelet/pods/2ff9f4f4-ce26-4612-8f70-1d21c2c3c945/volumes" Jul 27 15:46:43.845942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:43.845902 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:46:49.900314 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:49.900227 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l"] Jul 27 15:46:49.900904 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:49.900516 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" containerID="cri-o://4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b" gracePeriod=30 Jul 27 15:46:50.808471 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.808434 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc"] Jul 27 15:46:50.808996 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.808979 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" Jul 27 15:46:50.808996 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.808997 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" Jul 27 15:46:50.809182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.809010 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" Jul 27 15:46:50.809182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.809019 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" Jul 27 15:46:50.809182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.809154 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" Jul 27 15:46:50.809182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.809167 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff9f4f4-ce26-4612-8f70-1d21c2c3c945" containerName="istio-proxy" Jul 27 15:46:50.812730 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.812710 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.816019 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.815994 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-f58a29-openshift-default-dockercfg-cgmb2\"" Jul 27 15:46:50.825843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.825817 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc"] Jul 27 15:46:50.874566 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874532 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-data\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874576 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmjc\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-kube-api-access-jhmjc\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874642 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-token\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874673 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-certs\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-socket\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874730 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-podinfo\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-credential-socket\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874860 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-envoy\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.874984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.874885 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fd5160-1ccf-438d-9c9f-5c432770ad17-istiod-ca-cert\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.975707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-socket\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975718 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-podinfo\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975742 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-credential-socket\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975786 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-envoy\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975804 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fd5160-1ccf-438d-9c9f-5c432770ad17-istiod-ca-cert\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975833 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-data\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975859 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmjc\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-kube-api-access-jhmjc\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975912 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-token\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.975939 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-certs\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.976170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-socket\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.976465 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-certs\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.976477 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-credential-socket\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976756 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.976629 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-data\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.976937 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.976837 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fd5160-1ccf-438d-9c9f-5c432770ad17-istiod-ca-cert\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.978066 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.978030 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-envoy\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.978521 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.978495 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-podinfo\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.983798 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.983771 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-token\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:50.983896 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:50.983860 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmjc\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-kube-api-access-jhmjc\") pod \"e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:51.126475 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.126432 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:51.269711 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.269688 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc"] Jul 27 15:46:51.271941 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:46:51.271909 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fd5160_1ccf_438d_9c9f_5c432770ad17.slice/crio-f085428d1f29a9f4872cc74e35f082aea7a6fa795e7aac252b06205209e56b77 WatchSource:0}: Error finding container f085428d1f29a9f4872cc74e35f082aea7a6fa795e7aac252b06205209e56b77: Status 404 returned error can't find the container with id f085428d1f29a9f4872cc74e35f082aea7a6fa795e7aac252b06205209e56b77 Jul 27 15:46:51.274520 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.274485 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:46:51.274601 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.274565 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:46:51.274640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.274602 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:46:51.369359 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.369328 2553 patch_prober.go:28] interesting pod/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 15:46:51.369526 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:51.369390 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 15:46:52.266489 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:52.266451 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" event={"ID":"98fd5160-1ccf-438d-9c9f-5c432770ad17","Type":"ContainerStarted","Data":"ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109"} Jul 27 15:46:52.266981 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:52.266504 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" event={"ID":"98fd5160-1ccf-438d-9c9f-5c432770ad17","Type":"ContainerStarted","Data":"f085428d1f29a9f4872cc74e35f082aea7a6fa795e7aac252b06205209e56b77"} Jul 27 15:46:52.285820 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:52.285770 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" podStartSLOduration=2.285756063 podStartE2EDuration="2.285756063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:46:52.284258331 +0000 UTC m=+2515.565608402" watchObservedRunningTime="2026-07-27 15:46:52.285756063 +0000 UTC m=+2515.567106136" Jul 27 15:46:53.126942 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:53.126903 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:53.131753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:53.131728 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:53.271371 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:53.271339 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:53.272300 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:53.272282 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:46:55.163733 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.163668 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:46:55.217961 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.217929 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-envoy\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.217981 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-certs\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218008 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-credential-socket\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218031 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-data\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218076 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-istio-token\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218105 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjpqd\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-kube-api-access-gjpqd\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218134 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a583590-7b22-4918-ae8c-653b75710677-istio-podinfo\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218166 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218160 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a583590-7b22-4918-ae8c-653b75710677-istiod-ca-cert\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218225 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-socket\") pod \"3a583590-7b22-4918-ae8c-653b75710677\" (UID: \"3a583590-7b22-4918-ae8c-653b75710677\") " Jul 27 15:46:55.218542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218337 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:55.218542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218355 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:55.218706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218578 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:55.219402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.218870 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a583590-7b22-4918-ae8c-653b75710677-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:46:55.219402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.219330 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.219402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.219350 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a583590-7b22-4918-ae8c-653b75710677-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.219402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.219365 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.219402 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.219378 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.219749 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.219478 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:55.220755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.220728 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:46:55.221007 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.220978 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:46:55.221390 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.221363 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a583590-7b22-4918-ae8c-653b75710677-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:46:55.221494 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.221371 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-kube-api-access-gjpqd" (OuterVolumeSpecName: "kube-api-access-gjpqd") pod "3a583590-7b22-4918-ae8c-653b75710677" (UID: "3a583590-7b22-4918-ae8c-653b75710677"). InnerVolumeSpecName "kube-api-access-gjpqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:46:55.283136 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.283097 2553 generic.go:356] "Generic (PLEG): container finished" podID="3a583590-7b22-4918-ae8c-653b75710677" containerID="4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b" exitCode=0 Jul 27 15:46:55.283292 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.283145 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" Jul 27 15:46:55.283292 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.283165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" event={"ID":"3a583590-7b22-4918-ae8c-653b75710677","Type":"ContainerDied","Data":"4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b"} Jul 27 15:46:55.283292 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.283219 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l" event={"ID":"3a583590-7b22-4918-ae8c-653b75710677","Type":"ContainerDied","Data":"6a097d07bdb7294741277f49e0d091ff7e36e672db8d26ce959c1a5fa7f7071d"} Jul 27 15:46:55.283292 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.283241 2553 scope.go:117] "RemoveContainer" containerID="4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b" Jul 27 15:46:55.293027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.293009 2553 scope.go:117] "RemoveContainer" containerID="c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699" Jul 27 15:46:55.301729 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.301710 2553 scope.go:117] "RemoveContainer" containerID="4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b" Jul 27 15:46:55.301997 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:46:55.301977 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b\": container with ID starting with 4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b not found: ID does not exist" containerID="4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b" Jul 27 15:46:55.302141 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.302005 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b"} err="failed to get container status \"4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b\": rpc error: code = NotFound desc = could not find container \"4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b\": container with ID starting with 4dbb190da26175a531fbddbca991a5bd51428b21bd6bd6e87a1c54c020c8a11b not found: ID does not exist" Jul 27 15:46:55.302141 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.302026 2553 scope.go:117] "RemoveContainer" containerID="c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699" Jul 27 15:46:55.302332 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:46:55.302314 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699\": container with ID starting with c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699 not found: ID does not exist" containerID="c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699" Jul 27 15:46:55.302394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.302341 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699"} err="failed to get container status \"c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699\": rpc error: code = NotFound desc = could not find container \"c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699\": container with ID starting with c6ed29a42e7727b709a3aa1f3c59e1a50c8e78d0c3c70be0de4f6da29fba4699 not found: ID does not exist" Jul 27 15:46:55.305671 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.305651 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l"] Jul 27 15:46:55.310510 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.310489 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fbf2d4-openshift-default-755b97855d-w749l"] Jul 27 15:46:55.321003 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.320953 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.321003 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.320980 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a583590-7b22-4918-ae8c-653b75710677-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.321003 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.320994 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.321003 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.321006 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjpqd\" (UniqueName: \"kubernetes.io/projected/3a583590-7b22-4918-ae8c-653b75710677-kube-api-access-gjpqd\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:55.321286 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:55.321019 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a583590-7b22-4918-ae8c-653b75710677-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:46:57.192431 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:57.192397 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a583590-7b22-4918-ae8c-653b75710677" path="/var/lib/kubelet/pods/3a583590-7b22-4918-ae8c-653b75710677/volumes" Jul 27 15:46:58.542651 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:58.542605 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:46:59.459518 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.459484 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5"] Jul 27 15:46:59.459935 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.459918 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" Jul 27 15:46:59.460027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.459938 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" Jul 27 15:46:59.460027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.459953 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" Jul 27 15:46:59.460027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.459961 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" Jul 27 15:46:59.460202 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.460102 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" Jul 27 15:46:59.460202 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.460119 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a583590-7b22-4918-ae8c-653b75710677" containerName="istio-proxy" Jul 27 15:46:59.465002 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.464976 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.467038 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.467017 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-052d16-openshift-default-dockercfg-nbhj7\"" Jul 27 15:46:59.477575 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.477550 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5"] Jul 27 15:46:59.560690 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560658 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-data\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560704 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-certs\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-token\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560798 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-podinfo\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560818 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst7k\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-kube-api-access-kst7k\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-socket\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.560967 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-envoy\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.561041 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-credential-socket\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.561209 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.561123 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istiod-ca-cert\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661599 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661567 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-socket\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661603 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-envoy\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-credential-socket\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661681 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istiod-ca-cert\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661707 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-data\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661726 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-certs\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.661799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661748 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-token\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662178 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661870 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-podinfo\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662178 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.661922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst7k\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-kube-api-access-kst7k\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662178 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.662081 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-socket\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662178 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.662102 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-credential-socket\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662400 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.662304 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-certs\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662400 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.662352 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-data\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.662587 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.662567 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istiod-ca-cert\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.664189 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.664171 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-envoy\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.664319 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.664302 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-podinfo\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.670391 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.670359 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst7k\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-kube-api-access-kst7k\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.670509 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.670422 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-token\") pod \"e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.776417 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.776329 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:46:59.965119 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.965091 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5"] Jul 27 15:46:59.966554 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:46:59.966528 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1f9561_6f62_4403_86d0_7df2e1236ffa.slice/crio-774bebf91f2af8a0dec031b9dc8cc89ea8dc7877aa0f696b09a2b98f418ccb1c WatchSource:0}: Error finding container 774bebf91f2af8a0dec031b9dc8cc89ea8dc7877aa0f696b09a2b98f418ccb1c: Status 404 returned error can't find the container with id 774bebf91f2af8a0dec031b9dc8cc89ea8dc7877aa0f696b09a2b98f418ccb1c Jul 27 15:46:59.969157 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.969125 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:46:59.969263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.969192 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:46:59.969263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:46:59.969220 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:00.310610 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:00.310510 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" event={"ID":"0a1f9561-6f62-4403-86d0-7df2e1236ffa","Type":"ContainerStarted","Data":"beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65"} Jul 27 15:47:00.310610 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:00.310578 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" event={"ID":"0a1f9561-6f62-4403-86d0-7df2e1236ffa","Type":"ContainerStarted","Data":"774bebf91f2af8a0dec031b9dc8cc89ea8dc7877aa0f696b09a2b98f418ccb1c"} Jul 27 15:47:00.333869 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:00.333801 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podStartSLOduration=1.333777209 podStartE2EDuration="1.333777209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:47:00.331185754 +0000 UTC m=+2523.612535821" watchObservedRunningTime="2026-07-27 15:47:00.333777209 +0000 UTC m=+2523.615127284" Jul 27 15:47:00.776548 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:00.776515 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:01.782329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:01.782294 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:02.326855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:02.326815 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:02.333902 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:02.333858 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:06.348393 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:06.348309 2553 generic.go:356] "Generic (PLEG): container finished" podID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerID="ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109" exitCode=0 Jul 27 15:47:06.348393 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:06.348376 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" event={"ID":"98fd5160-1ccf-438d-9c9f-5c432770ad17","Type":"ContainerDied","Data":"ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109"} Jul 27 15:47:06.348816 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:06.348748 2553 scope.go:117] "RemoveContainer" containerID="ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109" Jul 27 15:47:06.349802 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:06.349773 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:06.349921 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:06.349838 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:06.349921 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:06.349874 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:07.355581 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:07.355547 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" event={"ID":"98fd5160-1ccf-438d-9c9f-5c432770ad17","Type":"ContainerStarted","Data":"5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942"} Jul 27 15:47:08.128743 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:08.128700 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:47:08.128966 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:08.128915 2553 patch_prober.go:28] interesting pod/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 27 15:47:08.129042 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:08.128966 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 27 15:47:08.271746 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:08.271709 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:47:09.126828 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:09.126787 2553 patch_prober.go:28] interesting pod/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 27 15:47:09.127463 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:09.127425 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 27 15:47:09.751615 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:09.751573 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:47:10.127265 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:10.127227 2553 patch_prober.go:28] interesting pod/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 27 15:47:10.127893 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:10.127860 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 27 15:47:11.126927 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:11.126881 2553 patch_prober.go:28] interesting pod/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 27 15:47:11.127170 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:11.126953 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 27 15:47:12.134559 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:12.134520 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:47:12.136966 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:12.136932 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 15:47:14.488201 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.488167 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc"] Jul 27 15:47:14.491321 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.491300 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.493761 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.493740 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f58a29\"/\"kube-root-ca.crt\"" Jul 27 15:47:14.493885 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.493798 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f58a29\"/\"openshift-service-ca.crt\"" Jul 27 15:47:14.494272 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.494252 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f58a29\"/\"default-dockercfg-8mfl7\"" Jul 27 15:47:14.494338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.494259 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f58a29\"/\"auth-test-model-f58a29-kserve-self-signed-certs\"" Jul 27 15:47:14.500111 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.500089 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc"] Jul 27 15:47:14.613977 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.613938 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-dshm\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.614182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.613985 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-tmp-dir\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.614182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.614108 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-home\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.614182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.614127 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvkn\" (UniqueName: \"kubernetes.io/projected/766ae3d7-a831-46e7-a972-a67c02644a88-kube-api-access-zlvkn\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.614182 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.614149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766ae3d7-a831-46e7-a972-a67c02644a88-tls-certs\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.614331 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.614209 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-model-cache\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.714856 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.714820 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-model-cache\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.714872 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-dshm\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.714904 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-tmp-dir\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.715013 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-home\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715208 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.715041 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvkn\" (UniqueName: \"kubernetes.io/projected/766ae3d7-a831-46e7-a972-a67c02644a88-kube-api-access-zlvkn\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715208 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.715094 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766ae3d7-a831-46e7-a972-a67c02644a88-tls-certs\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715208 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.715203 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-model-cache\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715356 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.715318 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-tmp-dir\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.715417 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.715371 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-home\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.717028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.717007 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-dshm\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.717558 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.717542 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766ae3d7-a831-46e7-a972-a67c02644a88-tls-certs\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.724734 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.724700 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvkn\" (UniqueName: \"kubernetes.io/projected/766ae3d7-a831-46e7-a972-a67c02644a88-kube-api-access-zlvkn\") pod \"auth-test-model-f58a29-kserve-5445c8df47-spmdc\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.802581 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.802508 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:14.940437 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.940412 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc"] Jul 27 15:47:14.941958 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:47:14.941930 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod766ae3d7_a831_46e7_a972_a67c02644a88.slice/crio-4dfddf65b7d7a24af365e49c767105610561ca6b81d78824b1449fa6c2606c35 WatchSource:0}: Error finding container 4dfddf65b7d7a24af365e49c767105610561ca6b81d78824b1449fa6c2606c35: Status 404 returned error can't find the container with id 4dfddf65b7d7a24af365e49c767105610561ca6b81d78824b1449fa6c2606c35 Jul 27 15:47:14.944143 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:14.944123 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 15:47:15.399568 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:15.399526 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" event={"ID":"766ae3d7-a831-46e7-a972-a67c02644a88","Type":"ContainerStarted","Data":"4dfddf65b7d7a24af365e49c767105610561ca6b81d78824b1449fa6c2606c35"} Jul 27 15:47:16.405709 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:16.405669 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" event={"ID":"766ae3d7-a831-46e7-a972-a67c02644a88","Type":"ContainerStarted","Data":"858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100"} Jul 27 15:47:16.425597 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:16.425534 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" podStartSLOduration=1.288369153 podStartE2EDuration="2.425511742s" totalImagesPullingTime="1.137142589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:47:16.423470506 +0000 UTC m=+2539.704820581" watchObservedRunningTime="2026-07-27 15:47:16.425511742 +0000 UTC m=+2539.706861827" Jul 27 15:47:17.686932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:17.686896 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:53268->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 27 15:47:17.687364 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:17.686968 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:53268->10.132.0.53:15021: read: connection reset by peer" Jul 27 15:47:18.417294 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.417259 2553 generic.go:356] "Generic (PLEG): container finished" podID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerID="beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65" exitCode=0 Jul 27 15:47:18.417472 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.417326 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" event={"ID":"0a1f9561-6f62-4403-86d0-7df2e1236ffa","Type":"ContainerDied","Data":"beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65"} Jul 27 15:47:18.417797 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.417782 2553 scope.go:117] "RemoveContainer" containerID="beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65" Jul 27 15:47:18.418843 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.418818 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:18.418914 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.418900 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:18.418952 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.418936 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:47:18.943640 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:18.943591 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:47:19.424432 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:19.424394 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" event={"ID":"0a1f9561-6f62-4403-86d0-7df2e1236ffa","Type":"ContainerStarted","Data":"b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f"} Jul 27 15:47:19.777408 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:19.777308 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:19.777573 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:19.777525 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 27 15:47:19.777617 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:19.777577 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 27 15:47:20.777338 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:20.777294 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 27 15:47:20.777954 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:20.777350 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 27 15:47:21.776912 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:21.776875 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 27 15:47:21.777117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:21.776930 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 27 15:47:22.777276 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:22.777236 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 27 15:47:22.777677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:22.777296 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 27 15:47:23.777684 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:23.777642 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 27 15:47:23.778189 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:23.777747 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 27 15:47:24.781203 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:24.781173 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:24.781613 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:24.781482 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:24.782186 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:24.782168 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 15:47:24.803153 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:24.803121 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:24.803153 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:24.803161 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:24.815782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:24.815751 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:47:25.474548 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:47:25.474513 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:48:07.578160 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:48:07.578124 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:48:33.651953 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:48:33.651866 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:48:54.470140 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:48:54.470102 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:49:12.260522 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:49:12.260485 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:49:32.875734 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:49:32.875692 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:49:46.463360 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:49:46.463319 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:49:57.371069 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:49:57.370969 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:49:57.380586 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:49:57.380561 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:49:59.166119 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:49:59.166082 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:50:16.076892 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:50:16.076855 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:50:29.848096 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:50:29.848046 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:50:50.346099 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:50:50.346038 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:50:57.521569 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:50:57.521534 2553 scope.go:117] "RemoveContainer" containerID="0b6f7fc45daed3e6008bf89cc06ece9dcf71fe6c57b5922917a2a2069a4d5f5a" Jul 27 15:51:07.764997 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:51:07.764959 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:51:29.450394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:51:29.450298 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:51:46.852540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:51:46.852499 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:52:14.840605 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:14.840567 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:52:19.663139 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:19.663100 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc"] Jul 27 15:52:30.254720 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:30.254686 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:52:34.682461 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.682416 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" podUID="766ae3d7-a831-46e7-a972-a67c02644a88" containerName="main" containerID="cri-o://858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100" gracePeriod=45 Jul 27 15:52:34.939091 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.939014 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:52:34.989109 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.989036 2553 generic.go:356] "Generic (PLEG): container finished" podID="766ae3d7-a831-46e7-a972-a67c02644a88" containerID="858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100" exitCode=0 Jul 27 15:52:34.989109 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.989096 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" event={"ID":"766ae3d7-a831-46e7-a972-a67c02644a88","Type":"ContainerDied","Data":"858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100"} Jul 27 15:52:34.989329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.989106 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" Jul 27 15:52:34.989329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.989140 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc" event={"ID":"766ae3d7-a831-46e7-a972-a67c02644a88","Type":"ContainerDied","Data":"4dfddf65b7d7a24af365e49c767105610561ca6b81d78824b1449fa6c2606c35"} Jul 27 15:52:34.989329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.989161 2553 scope.go:117] "RemoveContainer" containerID="858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100" Jul 27 15:52:34.997862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.997842 2553 scope.go:117] "RemoveContainer" containerID="858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100" Jul 27 15:52:34.998163 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:52:34.998142 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100\": container with ID starting with 858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100 not found: ID does not exist" containerID="858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100" Jul 27 15:52:34.998264 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:34.998167 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100"} err="failed to get container status \"858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100\": rpc error: code = NotFound desc = could not find container \"858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100\": container with ID starting with 858795631fb00bcdcd8c1cc4de2b61bc2c44aeffdac481a6f62cea8fdedbb100 not found: ID does not exist" Jul 27 15:52:35.077504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077464 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-tmp-dir\") pod \"766ae3d7-a831-46e7-a972-a67c02644a88\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " Jul 27 15:52:35.077504 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077509 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlvkn\" (UniqueName: \"kubernetes.io/projected/766ae3d7-a831-46e7-a972-a67c02644a88-kube-api-access-zlvkn\") pod \"766ae3d7-a831-46e7-a972-a67c02644a88\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " Jul 27 15:52:35.077755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077558 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766ae3d7-a831-46e7-a972-a67c02644a88-tls-certs\") pod \"766ae3d7-a831-46e7-a972-a67c02644a88\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " Jul 27 15:52:35.077755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077578 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-model-cache\") pod \"766ae3d7-a831-46e7-a972-a67c02644a88\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " Jul 27 15:52:35.077755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077616 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-home\") pod \"766ae3d7-a831-46e7-a972-a67c02644a88\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " Jul 27 15:52:35.077755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077700 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-dshm\") pod \"766ae3d7-a831-46e7-a972-a67c02644a88\" (UID: \"766ae3d7-a831-46e7-a972-a67c02644a88\") " Jul 27 15:52:35.077946 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077806 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "766ae3d7-a831-46e7-a972-a67c02644a88" (UID: "766ae3d7-a831-46e7-a972-a67c02644a88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:52:35.077946 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077850 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-model-cache" (OuterVolumeSpecName: "model-cache") pod "766ae3d7-a831-46e7-a972-a67c02644a88" (UID: "766ae3d7-a831-46e7-a972-a67c02644a88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:52:35.077946 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.077909 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-home" (OuterVolumeSpecName: "home") pod "766ae3d7-a831-46e7-a972-a67c02644a88" (UID: "766ae3d7-a831-46e7-a972-a67c02644a88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:52:35.078143 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.078017 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-tmp-dir\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:52:35.078143 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.078031 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-model-cache\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:52:35.078143 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.078039 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-home\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:52:35.079982 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.079958 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-dshm" (OuterVolumeSpecName: "dshm") pod "766ae3d7-a831-46e7-a972-a67c02644a88" (UID: "766ae3d7-a831-46e7-a972-a67c02644a88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:52:35.080085 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.080014 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766ae3d7-a831-46e7-a972-a67c02644a88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "766ae3d7-a831-46e7-a972-a67c02644a88" (UID: "766ae3d7-a831-46e7-a972-a67c02644a88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:52:35.080085 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.080027 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766ae3d7-a831-46e7-a972-a67c02644a88-kube-api-access-zlvkn" (OuterVolumeSpecName: "kube-api-access-zlvkn") pod "766ae3d7-a831-46e7-a972-a67c02644a88" (UID: "766ae3d7-a831-46e7-a972-a67c02644a88"). InnerVolumeSpecName "kube-api-access-zlvkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:52:35.178757 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.178718 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766ae3d7-a831-46e7-a972-a67c02644a88-dshm\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:52:35.178757 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.178751 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvkn\" (UniqueName: \"kubernetes.io/projected/766ae3d7-a831-46e7-a972-a67c02644a88-kube-api-access-zlvkn\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:52:35.178757 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.178763 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766ae3d7-a831-46e7-a972-a67c02644a88-tls-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:52:35.305800 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.305762 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc"] Jul 27 15:52:35.309776 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:35.309747 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f58a29/auth-test-model-f58a29-kserve-5445c8df47-spmdc"] Jul 27 15:52:37.192150 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:37.192114 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="766ae3d7-a831-46e7-a972-a67c02644a88" path="/var/lib/kubelet/pods/766ae3d7-a831-46e7-a972-a67c02644a88/volumes" Jul 27 15:52:40.459719 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:40.459683 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:52:59.482388 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.482356 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954"] Jul 27 15:52:59.482804 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.482790 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="766ae3d7-a831-46e7-a972-a67c02644a88" containerName="main" Jul 27 15:52:59.482852 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.482809 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="766ae3d7-a831-46e7-a972-a67c02644a88" containerName="main" Jul 27 15:52:59.482893 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.482878 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="766ae3d7-a831-46e7-a972-a67c02644a88" containerName="main" Jul 27 15:52:59.487007 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.486990 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.489342 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.489317 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-052d16\"/\"kube-root-ca.crt\"" Jul 27 15:52:59.489455 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.489317 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-052d16\"/\"sub-test-model-052d16-kserve-self-signed-certs\"" Jul 27 15:52:59.490031 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.490008 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-052d16\"/\"openshift-service-ca.crt\"" Jul 27 15:52:59.490101 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.490033 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-052d16\"/\"default-dockercfg-4l8kt\"" Jul 27 15:52:59.500808 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.500782 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954"] Jul 27 15:52:59.603890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.603853 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fccd98-57e7-4543-815c-465dc55ca30b-tls-certs\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.604117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.603892 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjk5g\" (UniqueName: \"kubernetes.io/projected/94fccd98-57e7-4543-815c-465dc55ca30b-kube-api-access-pjk5g\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.604117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.603921 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-dshm\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.604117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.603991 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-home\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.604117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.604028 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-tmp-dir\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.604117 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.604073 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-model-cache\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705204 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705169 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fccd98-57e7-4543-815c-465dc55ca30b-tls-certs\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705214 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjk5g\" (UniqueName: \"kubernetes.io/projected/94fccd98-57e7-4543-815c-465dc55ca30b-kube-api-access-pjk5g\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705235 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-dshm\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705269 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-home\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705290 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-tmp-dir\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-model-cache\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705779 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705753 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-model-cache\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705905 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705792 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-home\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.705905 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.705809 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-tmp-dir\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.707575 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.707546 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-dshm\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.707862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.707842 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fccd98-57e7-4543-815c-465dc55ca30b-tls-certs\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.714635 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.714613 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjk5g\" (UniqueName: \"kubernetes.io/projected/94fccd98-57e7-4543-815c-465dc55ca30b-kube-api-access-pjk5g\") pod \"sub-test-model-052d16-kserve-b947c568c-jc954\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.796621 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.796526 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:52:59.928933 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:52:59.928902 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954"] Jul 27 15:52:59.931043 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:52:59.931018 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94fccd98_57e7_4543_815c_465dc55ca30b.slice/crio-b125f74acc66248baf6e3423e77e3989d369725cc53887be08f4c17712fec757 WatchSource:0}: Error finding container b125f74acc66248baf6e3423e77e3989d369725cc53887be08f4c17712fec757: Status 404 returned error can't find the container with id b125f74acc66248baf6e3423e77e3989d369725cc53887be08f4c17712fec757 Jul 27 15:53:00.111394 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:00.111359 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" event={"ID":"94fccd98-57e7-4543-815c-465dc55ca30b","Type":"ContainerStarted","Data":"86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7"} Jul 27 15:53:00.111567 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:00.111402 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" event={"ID":"94fccd98-57e7-4543-815c-465dc55ca30b","Type":"ContainerStarted","Data":"b125f74acc66248baf6e3423e77e3989d369725cc53887be08f4c17712fec757"} Jul 27 15:53:00.128855 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:00.128801 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" podStartSLOduration=1.128785288 podStartE2EDuration="1.128785288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:53:00.127599101 +0000 UTC m=+2883.408949176" watchObservedRunningTime="2026-07-27 15:53:00.128785288 +0000 UTC m=+2883.410135362" Jul 27 15:53:03.147890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:03.147852 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:53:09.797550 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:09.797510 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:53:09.798036 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:09.797575 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:53:09.809596 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:09.809570 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:53:10.173130 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:10.173105 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:53:34.845963 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:34.845925 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:53:54.540213 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:53:54.540174 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:54:01.399536 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:01.399503 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954"] Jul 27 15:54:14.137083 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:14.137033 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:54:16.419423 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.419357 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" podUID="94fccd98-57e7-4543-815c-465dc55ca30b" containerName="main" containerID="cri-o://86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7" gracePeriod=45 Jul 27 15:54:16.677950 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.677887 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:54:16.801028 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.800995 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-home\") pod \"94fccd98-57e7-4543-815c-465dc55ca30b\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " Jul 27 15:54:16.801253 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801071 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-dshm\") pod \"94fccd98-57e7-4543-815c-465dc55ca30b\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " Jul 27 15:54:16.801253 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801118 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fccd98-57e7-4543-815c-465dc55ca30b-tls-certs\") pod \"94fccd98-57e7-4543-815c-465dc55ca30b\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " Jul 27 15:54:16.801253 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801150 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-model-cache\") pod \"94fccd98-57e7-4543-815c-465dc55ca30b\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " Jul 27 15:54:16.801253 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801232 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-tmp-dir\") pod \"94fccd98-57e7-4543-815c-465dc55ca30b\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " Jul 27 15:54:16.801469 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801278 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjk5g\" (UniqueName: \"kubernetes.io/projected/94fccd98-57e7-4543-815c-465dc55ca30b-kube-api-access-pjk5g\") pod \"94fccd98-57e7-4543-815c-465dc55ca30b\" (UID: \"94fccd98-57e7-4543-815c-465dc55ca30b\") " Jul 27 15:54:16.801469 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801343 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-home" (OuterVolumeSpecName: "home") pod "94fccd98-57e7-4543-815c-465dc55ca30b" (UID: "94fccd98-57e7-4543-815c-465dc55ca30b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:54:16.801623 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801487 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-model-cache" (OuterVolumeSpecName: "model-cache") pod "94fccd98-57e7-4543-815c-465dc55ca30b" (UID: "94fccd98-57e7-4543-815c-465dc55ca30b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:54:16.801623 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801497 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94fccd98-57e7-4543-815c-465dc55ca30b" (UID: "94fccd98-57e7-4543-815c-465dc55ca30b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:54:16.801700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801623 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-model-cache\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:54:16.801700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801641 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-tmp-dir\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:54:16.801700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.801656 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-home\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:54:16.803488 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.803464 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fccd98-57e7-4543-815c-465dc55ca30b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94fccd98-57e7-4543-815c-465dc55ca30b" (UID: "94fccd98-57e7-4543-815c-465dc55ca30b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 15:54:16.803603 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.803501 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-dshm" (OuterVolumeSpecName: "dshm") pod "94fccd98-57e7-4543-815c-465dc55ca30b" (UID: "94fccd98-57e7-4543-815c-465dc55ca30b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:54:16.803603 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.803519 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fccd98-57e7-4543-815c-465dc55ca30b-kube-api-access-pjk5g" (OuterVolumeSpecName: "kube-api-access-pjk5g") pod "94fccd98-57e7-4543-815c-465dc55ca30b" (UID: "94fccd98-57e7-4543-815c-465dc55ca30b"). InnerVolumeSpecName "kube-api-access-pjk5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:54:16.902171 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.902134 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94fccd98-57e7-4543-815c-465dc55ca30b-dshm\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:54:16.902171 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.902165 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94fccd98-57e7-4543-815c-465dc55ca30b-tls-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:54:16.902171 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:16.902178 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjk5g\" (UniqueName: \"kubernetes.io/projected/94fccd98-57e7-4543-815c-465dc55ca30b-kube-api-access-pjk5g\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:54:17.505125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.505023 2553 generic.go:356] "Generic (PLEG): container finished" podID="94fccd98-57e7-4543-815c-465dc55ca30b" containerID="86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7" exitCode=0 Jul 27 15:54:17.505125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.505099 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" Jul 27 15:54:17.505629 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.505119 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" event={"ID":"94fccd98-57e7-4543-815c-465dc55ca30b","Type":"ContainerDied","Data":"86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7"} Jul 27 15:54:17.505629 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.505167 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954" event={"ID":"94fccd98-57e7-4543-815c-465dc55ca30b","Type":"ContainerDied","Data":"b125f74acc66248baf6e3423e77e3989d369725cc53887be08f4c17712fec757"} Jul 27 15:54:17.505629 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.505187 2553 scope.go:117] "RemoveContainer" containerID="86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7" Jul 27 15:54:17.513938 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.513920 2553 scope.go:117] "RemoveContainer" containerID="86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7" Jul 27 15:54:17.514235 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:54:17.514216 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7\": container with ID starting with 86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7 not found: ID does not exist" containerID="86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7" Jul 27 15:54:17.514300 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.514243 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7"} err="failed to get container status \"86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7\": rpc error: code = NotFound desc = could not find container \"86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7\": container with ID starting with 86c42c0a5f8fcae094f165a776394aac5390d11273f67f1581defb2f85c48aa7 not found: ID does not exist" Jul 27 15:54:17.522363 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.522340 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954"] Jul 27 15:54:17.528662 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:17.528641 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-052d16/sub-test-model-052d16-kserve-b947c568c-jc954"] Jul 27 15:54:19.191647 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:19.191615 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94fccd98-57e7-4543-815c-465dc55ca30b" path="/var/lib/kubelet/pods/94fccd98-57e7-4543-815c-465dc55ca30b/volumes" Jul 27 15:54:47.873618 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:47.873581 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:54:57.400732 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:57.400699 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:54:57.414541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:54:57.414515 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:55:20.491666 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:55:20.491628 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:55:33.570377 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:55:33.570334 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:55:57.079538 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:55:57.079462 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:56:05.762205 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.762169 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-140536443373104"] Jul 27 15:56:05.762603 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.762556 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fccd98-57e7-4543-815c-465dc55ca30b" containerName="main" Jul 27 15:56:05.762603 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.762568 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fccd98-57e7-4543-815c-465dc55ca30b" containerName="main" Jul 27 15:56:05.762680 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.762641 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fccd98-57e7-4543-815c-465dc55ca30b" containerName="main" Jul 27 15:56:05.765685 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.765669 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:05.767866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.767845 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lcckw\"" Jul 27 15:56:05.773700 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.773678 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443373104"] Jul 27 15:56:05.865403 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.865373 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pct9k\" (UniqueName: \"kubernetes.io/projected/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0-kube-api-access-pct9k\") pod \"test-curl-7741-140536443373104\" (UID: \"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0\") " pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:05.966470 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.966432 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pct9k\" (UniqueName: \"kubernetes.io/projected/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0-kube-api-access-pct9k\") pod \"test-curl-7741-140536443373104\" (UID: \"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0\") " pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:05.975510 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:05.975485 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pct9k\" (UniqueName: \"kubernetes.io/projected/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0-kube-api-access-pct9k\") pod \"test-curl-7741-140536443373104\" (UID: \"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0\") " pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:06.076569 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:06.076463 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:06.218241 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:06.218207 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443373104"] Jul 27 15:56:06.218838 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:06.218808 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e2323cf_da7f_4610_bf7f_0e13f3ae55b0.slice/crio-75489ec7752683168717ce77d4ed426ad85184248c0c3be5b72a8ab72fb2191a WatchSource:0}: Error finding container 75489ec7752683168717ce77d4ed426ad85184248c0c3be5b72a8ab72fb2191a: Status 404 returned error can't find the container with id 75489ec7752683168717ce77d4ed426ad85184248c0c3be5b72a8ab72fb2191a Jul 27 15:56:06.220251 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:06.220230 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 15:56:07.027841 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:07.027809 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443373104" event={"ID":"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0","Type":"ContainerStarted","Data":"2e7c1057da12259d70c540e144c28ee50fb4d6ee4b6cbae3f63b958778c904c1"} Jul 27 15:56:07.028169 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:07.027851 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443373104" event={"ID":"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0","Type":"ContainerStarted","Data":"75489ec7752683168717ce77d4ed426ad85184248c0c3be5b72a8ab72fb2191a"} Jul 27 15:56:07.046466 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:07.046419 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7741-140536443373104" podStartSLOduration=1.3054207 podStartE2EDuration="2.046396202s" totalImagesPullingTime="740.975502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:56:07.045719941 +0000 UTC m=+3070.327070015" watchObservedRunningTime="2026-07-27 15:56:07.046396202 +0000 UTC m=+3070.327746275" Jul 27 15:56:07.206677 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:07.206643 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7741-140536443373104_0e2323cf-da7f-4610-bf7f-0e13f3ae55b0/test-curl-7741-140536443373104/0.log" Jul 27 15:56:08.034706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:08.034666 2553 generic.go:356] "Generic (PLEG): container finished" podID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" containerID="2e7c1057da12259d70c540e144c28ee50fb4d6ee4b6cbae3f63b958778c904c1" exitCode=0 Jul 27 15:56:08.035123 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:08.034730 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443373104" event={"ID":"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0","Type":"ContainerDied","Data":"2e7c1057da12259d70c540e144c28ee50fb4d6ee4b6cbae3f63b958778c904c1"} Jul 27 15:56:09.181711 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.181685 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:09.196716 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.196692 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pct9k\" (UniqueName: \"kubernetes.io/projected/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0-kube-api-access-pct9k\") pod \"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0\" (UID: \"0e2323cf-da7f-4610-bf7f-0e13f3ae55b0\") " Jul 27 15:56:09.199295 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.199261 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0-kube-api-access-pct9k" (OuterVolumeSpecName: "kube-api-access-pct9k") pod "0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" (UID: "0e2323cf-da7f-4610-bf7f-0e13f3ae55b0"). InnerVolumeSpecName "kube-api-access-pct9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:56:09.220645 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.220580 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-140536443373104"] Jul 27 15:56:09.222456 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.222433 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-140536443373104"] Jul 27 15:56:09.298414 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.298323 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pct9k\" (UniqueName: \"kubernetes.io/projected/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0-kube-api-access-pct9k\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:56:09.343862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.343830 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-140536443374640"] Jul 27 15:56:09.344244 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.344230 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" containerName="test-curl-7741-140536443373104" Jul 27 15:56:09.344299 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.344246 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" containerName="test-curl-7741-140536443373104" Jul 27 15:56:09.344336 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.344309 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" containerName="test-curl-7741-140536443373104" Jul 27 15:56:09.347252 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.347237 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:09.355984 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.355957 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443374640"] Jul 27 15:56:09.398913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.398882 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49p5p\" (UniqueName: \"kubernetes.io/projected/837917c1-58d7-4e0c-b714-cddf787b4f24-kube-api-access-49p5p\") pod \"test-curl-7741-140536443374640\" (UID: \"837917c1-58d7-4e0c-b714-cddf787b4f24\") " pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:09.500212 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.500177 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49p5p\" (UniqueName: \"kubernetes.io/projected/837917c1-58d7-4e0c-b714-cddf787b4f24-kube-api-access-49p5p\") pod \"test-curl-7741-140536443374640\" (UID: \"837917c1-58d7-4e0c-b714-cddf787b4f24\") " pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:09.508288 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.508262 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49p5p\" (UniqueName: \"kubernetes.io/projected/837917c1-58d7-4e0c-b714-cddf787b4f24-kube-api-access-49p5p\") pod \"test-curl-7741-140536443374640\" (UID: \"837917c1-58d7-4e0c-b714-cddf787b4f24\") " pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:09.656995 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.656950 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:09.785528 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:09.785347 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443374640"] Jul 27 15:56:09.787731 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:09.787699 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod837917c1_58d7_4e0c_b714_cddf787b4f24.slice/crio-5c60cc8a7c030af66ecfa2965411028852a297368ba23cd688c7c4d07a537358 WatchSource:0}: Error finding container 5c60cc8a7c030af66ecfa2965411028852a297368ba23cd688c7c4d07a537358: Status 404 returned error can't find the container with id 5c60cc8a7c030af66ecfa2965411028852a297368ba23cd688c7c4d07a537358 Jul 27 15:56:10.046011 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:10.045937 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:10.046011 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:10.045986 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75489ec7752683168717ce77d4ed426ad85184248c0c3be5b72a8ab72fb2191a" Jul 27 15:56:10.047189 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:10.047155 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443374640" event={"ID":"837917c1-58d7-4e0c-b714-cddf787b4f24","Type":"ContainerStarted","Data":"5c60cc8a7c030af66ecfa2965411028852a297368ba23cd688c7c4d07a537358"} Jul 27 15:56:10.062975 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:56:10.062953 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-140536443373104\" is forbidden: User \"system:node:ip-10-0-135-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-218.ec2.internal' and this object" podUID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:11.054152 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:11.054115 2553 generic.go:356] "Generic (PLEG): container finished" podID="837917c1-58d7-4e0c-b714-cddf787b4f24" containerID="dde20b38cda266910effe1dda110ff8975ea1061635cc9eda8cebf1d0bedbf7c" exitCode=0 Jul 27 15:56:11.054540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:11.054160 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443374640" event={"ID":"837917c1-58d7-4e0c-b714-cddf787b4f24","Type":"ContainerDied","Data":"dde20b38cda266910effe1dda110ff8975ea1061635cc9eda8cebf1d0bedbf7c"} Jul 27 15:56:11.056172 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:56:11.056144 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-140536443373104\" is forbidden: User \"system:node:ip-10-0-135-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-218.ec2.internal' and this object" podUID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" pod="opendatahub/test-curl-7741-140536443373104" Jul 27 15:56:11.192091 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:11.192036 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e2323cf-da7f-4610-bf7f-0e13f3ae55b0" path="/var/lib/kubelet/pods/0e2323cf-da7f-4610-bf7f-0e13f3ae55b0/volumes" Jul 27 15:56:11.755694 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:11.755661 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:56:12.196753 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.196729 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:12.223407 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.223299 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7741-140536443374640_837917c1-58d7-4e0c-b714-cddf787b4f24/test-curl-7741-140536443374640/0.log" Jul 27 15:56:12.226030 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.226000 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49p5p\" (UniqueName: \"kubernetes.io/projected/837917c1-58d7-4e0c-b714-cddf787b4f24-kube-api-access-49p5p\") pod \"837917c1-58d7-4e0c-b714-cddf787b4f24\" (UID: \"837917c1-58d7-4e0c-b714-cddf787b4f24\") " Jul 27 15:56:12.228926 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.228862 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837917c1-58d7-4e0c-b714-cddf787b4f24-kube-api-access-49p5p" (OuterVolumeSpecName: "kube-api-access-49p5p") pod "837917c1-58d7-4e0c-b714-cddf787b4f24" (UID: "837917c1-58d7-4e0c-b714-cddf787b4f24"). InnerVolumeSpecName "kube-api-access-49p5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:56:12.247420 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.247390 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-140536443374640"] Jul 27 15:56:12.251128 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.251105 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-140536443374640"] Jul 27 15:56:12.327591 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.327499 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49p5p\" (UniqueName: \"kubernetes.io/projected/837917c1-58d7-4e0c-b714-cddf787b4f24-kube-api-access-49p5p\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:56:12.416759 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.416728 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:12.417129 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.417116 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837917c1-58d7-4e0c-b714-cddf787b4f24" containerName="test-curl-7741-140536443374640" Jul 27 15:56:12.417194 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.417130 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="837917c1-58d7-4e0c-b714-cddf787b4f24" containerName="test-curl-7741-140536443374640" Jul 27 15:56:12.417194 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.417187 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="837917c1-58d7-4e0c-b714-cddf787b4f24" containerName="test-curl-7741-140536443374640" Jul 27 15:56:12.420013 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.419998 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:12.423821 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.423794 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:12.529146 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.529110 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmmr\" (UniqueName: \"kubernetes.io/projected/12320aa4-1a60-4ee8-bc20-164f6ca8247b-kube-api-access-4dmmr\") pod \"test-curl-7741-140536443248832\" (UID: \"12320aa4-1a60-4ee8-bc20-164f6ca8247b\") " pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:12.629579 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.629545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmmr\" (UniqueName: \"kubernetes.io/projected/12320aa4-1a60-4ee8-bc20-164f6ca8247b-kube-api-access-4dmmr\") pod \"test-curl-7741-140536443248832\" (UID: \"12320aa4-1a60-4ee8-bc20-164f6ca8247b\") " pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:12.638074 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.638024 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmmr\" (UniqueName: \"kubernetes.io/projected/12320aa4-1a60-4ee8-bc20-164f6ca8247b-kube-api-access-4dmmr\") pod \"test-curl-7741-140536443248832\" (UID: \"12320aa4-1a60-4ee8-bc20-164f6ca8247b\") " pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:12.731289 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.731250 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:12.866465 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:12.866433 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:12.867377 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:12.867348 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12320aa4_1a60_4ee8_bc20_164f6ca8247b.slice/crio-171ef57bffa7ce1f4ff74a4414e5d146b33920c644343d27a7e38a15d3fdb028 WatchSource:0}: Error finding container 171ef57bffa7ce1f4ff74a4414e5d146b33920c644343d27a7e38a15d3fdb028: Status 404 returned error can't find the container with id 171ef57bffa7ce1f4ff74a4414e5d146b33920c644343d27a7e38a15d3fdb028 Jul 27 15:56:13.064540 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:13.064508 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443248832" event={"ID":"12320aa4-1a60-4ee8-bc20-164f6ca8247b","Type":"ContainerStarted","Data":"171ef57bffa7ce1f4ff74a4414e5d146b33920c644343d27a7e38a15d3fdb028"} Jul 27 15:56:13.065706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:13.065681 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:13.065831 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:13.065681 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c60cc8a7c030af66ecfa2965411028852a297368ba23cd688c7c4d07a537358" Jul 27 15:56:13.075872 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:56:13.075842 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-140536443374640\" is forbidden: User \"system:node:ip-10-0-135-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-218.ec2.internal' and this object" podUID="837917c1-58d7-4e0c-b714-cddf787b4f24" pod="opendatahub/test-curl-7741-140536443374640" Jul 27 15:56:13.193545 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:13.193513 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837917c1-58d7-4e0c-b714-cddf787b4f24" path="/var/lib/kubelet/pods/837917c1-58d7-4e0c-b714-cddf787b4f24/volumes" Jul 27 15:56:14.077715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:14.077676 2553 generic.go:356] "Generic (PLEG): container finished" podID="12320aa4-1a60-4ee8-bc20-164f6ca8247b" containerID="5eb8333a796ea444ee794041dfe5b19e7ba8f068f49adb58f82bc9e7905be7e1" exitCode=0 Jul 27 15:56:14.078120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:14.077737 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443248832" event={"ID":"12320aa4-1a60-4ee8-bc20-164f6ca8247b","Type":"ContainerDied","Data":"5eb8333a796ea444ee794041dfe5b19e7ba8f068f49adb58f82bc9e7905be7e1"} Jul 27 15:56:15.227260 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.227231 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:15.248323 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.248264 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7741-140536443248832_12320aa4-1a60-4ee8-bc20-164f6ca8247b/test-curl-7741-140536443248832/0.log" Jul 27 15:56:15.278134 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.278091 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:15.280622 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.280594 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:15.353349 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.353328 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dmmr\" (UniqueName: \"kubernetes.io/projected/12320aa4-1a60-4ee8-bc20-164f6ca8247b-kube-api-access-4dmmr\") pod \"12320aa4-1a60-4ee8-bc20-164f6ca8247b\" (UID: \"12320aa4-1a60-4ee8-bc20-164f6ca8247b\") " Jul 27 15:56:15.355268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.355244 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12320aa4-1a60-4ee8-bc20-164f6ca8247b-kube-api-access-4dmmr" (OuterVolumeSpecName: "kube-api-access-4dmmr") pod "12320aa4-1a60-4ee8-bc20-164f6ca8247b" (UID: "12320aa4-1a60-4ee8-bc20-164f6ca8247b"). InnerVolumeSpecName "kube-api-access-4dmmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:56:15.414446 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.414414 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:15.414785 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.414773 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12320aa4-1a60-4ee8-bc20-164f6ca8247b" containerName="test-curl-7741-140536443248832" Jul 27 15:56:15.414833 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.414787 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="12320aa4-1a60-4ee8-bc20-164f6ca8247b" containerName="test-curl-7741-140536443248832" Jul 27 15:56:15.414866 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.414855 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="12320aa4-1a60-4ee8-bc20-164f6ca8247b" containerName="test-curl-7741-140536443248832" Jul 27 15:56:15.417890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.417876 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:15.421746 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.421720 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:15.453970 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.453937 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88ft\" (UniqueName: \"kubernetes.io/projected/f8da3410-99a9-4be9-80fb-c17a3aa881eb-kube-api-access-h88ft\") pod \"test-curl-7741-140536443248832\" (UID: \"f8da3410-99a9-4be9-80fb-c17a3aa881eb\") " pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:15.454160 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.454126 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dmmr\" (UniqueName: \"kubernetes.io/projected/12320aa4-1a60-4ee8-bc20-164f6ca8247b-kube-api-access-4dmmr\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:56:15.554860 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.554823 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88ft\" (UniqueName: \"kubernetes.io/projected/f8da3410-99a9-4be9-80fb-c17a3aa881eb-kube-api-access-h88ft\") pod \"test-curl-7741-140536443248832\" (UID: \"f8da3410-99a9-4be9-80fb-c17a3aa881eb\") " pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:15.563669 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.563636 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88ft\" (UniqueName: \"kubernetes.io/projected/f8da3410-99a9-4be9-80fb-c17a3aa881eb-kube-api-access-h88ft\") pod \"test-curl-7741-140536443248832\" (UID: \"f8da3410-99a9-4be9-80fb-c17a3aa881eb\") " pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:15.728848 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.728763 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:15.862180 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:15.862152 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:15.863104 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:15.863076 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8da3410_99a9_4be9_80fb_c17a3aa881eb.slice/crio-2e3c5ef9468855b2804f5bcc6194381937cd7dbe407e9cd57628cd2e3d205cfd WatchSource:0}: Error finding container 2e3c5ef9468855b2804f5bcc6194381937cd7dbe407e9cd57628cd2e3d205cfd: Status 404 returned error can't find the container with id 2e3c5ef9468855b2804f5bcc6194381937cd7dbe407e9cd57628cd2e3d205cfd Jul 27 15:56:16.088829 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:16.088736 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443248832" event={"ID":"f8da3410-99a9-4be9-80fb-c17a3aa881eb","Type":"ContainerStarted","Data":"2e3c5ef9468855b2804f5bcc6194381937cd7dbe407e9cd57628cd2e3d205cfd"} Jul 27 15:56:16.089932 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:16.089914 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:16.090038 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:16.089917 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="171ef57bffa7ce1f4ff74a4414e5d146b33920c644343d27a7e38a15d3fdb028" Jul 27 15:56:16.102931 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:16.102893 2553 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7741-140536443248832" oldPodUID="12320aa4-1a60-4ee8-bc20-164f6ca8247b" podUID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" Jul 27 15:56:16.279381 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:56:16.279350 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8da3410_99a9_4be9_80fb_c17a3aa881eb.slice/crio-conmon-7c3eee71285829956931507b6c0921ed46c9d6730de47989f0a4660fffd743d1.scope\": RecentStats: unable to find data in memory cache]" Jul 27 15:56:17.095725 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:17.095694 2553 generic.go:356] "Generic (PLEG): container finished" podID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" containerID="7c3eee71285829956931507b6c0921ed46c9d6730de47989f0a4660fffd743d1" exitCode=0 Jul 27 15:56:17.095911 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:17.095759 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443248832" event={"ID":"f8da3410-99a9-4be9-80fb-c17a3aa881eb","Type":"ContainerDied","Data":"7c3eee71285829956931507b6c0921ed46c9d6730de47989f0a4660fffd743d1"} Jul 27 15:56:17.191929 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:17.191895 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12320aa4-1a60-4ee8-bc20-164f6ca8247b" path="/var/lib/kubelet/pods/12320aa4-1a60-4ee8-bc20-164f6ca8247b/volumes" Jul 27 15:56:18.241721 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.241690 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:18.264296 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.264238 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7741-140536443248832_f8da3410-99a9-4be9-80fb-c17a3aa881eb/test-curl-7741-140536443248832/0.log" Jul 27 15:56:18.278750 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.278722 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h88ft\" (UniqueName: \"kubernetes.io/projected/f8da3410-99a9-4be9-80fb-c17a3aa881eb-kube-api-access-h88ft\") pod \"f8da3410-99a9-4be9-80fb-c17a3aa881eb\" (UID: \"f8da3410-99a9-4be9-80fb-c17a3aa881eb\") " Jul 27 15:56:18.280814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.280790 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8da3410-99a9-4be9-80fb-c17a3aa881eb-kube-api-access-h88ft" (OuterVolumeSpecName: "kube-api-access-h88ft") pod "f8da3410-99a9-4be9-80fb-c17a3aa881eb" (UID: "f8da3410-99a9-4be9-80fb-c17a3aa881eb"). InnerVolumeSpecName "kube-api-access-h88ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:56:18.294541 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.294512 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:18.296290 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.296268 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-140536443248832"] Jul 27 15:56:18.379697 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.379663 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88ft\" (UniqueName: \"kubernetes.io/projected/f8da3410-99a9-4be9-80fb-c17a3aa881eb-kube-api-access-h88ft\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:56:18.417710 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.417676 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:18.418070 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.418042 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" containerName="test-curl-7741-140536443248832" Jul 27 15:56:18.418124 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.418072 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" containerName="test-curl-7741-140536443248832" Jul 27 15:56:18.418160 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.418135 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" containerName="test-curl-7741-140536443248832" Jul 27 15:56:18.420941 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.420923 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:18.426781 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.426734 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:18.481418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.481365 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjdm\" (UniqueName: \"kubernetes.io/projected/89ad1e82-cb67-4a12-9b53-1e08449a05be-kube-api-access-brjdm\") pod \"test-curl-7741-140536443226128\" (UID: \"89ad1e82-cb67-4a12-9b53-1e08449a05be\") " pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:18.582127 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.582089 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjdm\" (UniqueName: \"kubernetes.io/projected/89ad1e82-cb67-4a12-9b53-1e08449a05be-kube-api-access-brjdm\") pod \"test-curl-7741-140536443226128\" (UID: \"89ad1e82-cb67-4a12-9b53-1e08449a05be\") " pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:18.590720 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.590693 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjdm\" (UniqueName: \"kubernetes.io/projected/89ad1e82-cb67-4a12-9b53-1e08449a05be-kube-api-access-brjdm\") pod \"test-curl-7741-140536443226128\" (UID: \"89ad1e82-cb67-4a12-9b53-1e08449a05be\") " pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:18.731212 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.731121 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:18.866034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:18.866002 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:18.867810 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:18.867785 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89ad1e82_cb67_4a12_9b53_1e08449a05be.slice/crio-29ce53a3825670a0c7d04d5ac4ca55cf019bee8d5ec8c0f153c57b24b2727610 WatchSource:0}: Error finding container 29ce53a3825670a0c7d04d5ac4ca55cf019bee8d5ec8c0f153c57b24b2727610: Status 404 returned error can't find the container with id 29ce53a3825670a0c7d04d5ac4ca55cf019bee8d5ec8c0f153c57b24b2727610 Jul 27 15:56:19.106989 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:19.106953 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:19.107191 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:19.106966 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e3c5ef9468855b2804f5bcc6194381937cd7dbe407e9cd57628cd2e3d205cfd" Jul 27 15:56:19.108035 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:19.108012 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443226128" event={"ID":"89ad1e82-cb67-4a12-9b53-1e08449a05be","Type":"ContainerStarted","Data":"29ce53a3825670a0c7d04d5ac4ca55cf019bee8d5ec8c0f153c57b24b2727610"} Jul 27 15:56:19.121871 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:56:19.121837 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-140536443248832\" is forbidden: User \"system:node:ip-10-0-135-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-218.ec2.internal' and this object" podUID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" pod="opendatahub/test-curl-7741-140536443248832" Jul 27 15:56:19.192556 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:19.192529 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8da3410-99a9-4be9-80fb-c17a3aa881eb" path="/var/lib/kubelet/pods/f8da3410-99a9-4be9-80fb-c17a3aa881eb/volumes" Jul 27 15:56:20.114680 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:20.114641 2553 generic.go:356] "Generic (PLEG): container finished" podID="89ad1e82-cb67-4a12-9b53-1e08449a05be" containerID="389faff0fc451758d4f0bf50f17bae7ae98a1a34115ace7a3bc19d0201ccc2d3" exitCode=0 Jul 27 15:56:20.115079 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:20.114707 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443226128" event={"ID":"89ad1e82-cb67-4a12-9b53-1e08449a05be","Type":"ContainerDied","Data":"389faff0fc451758d4f0bf50f17bae7ae98a1a34115ace7a3bc19d0201ccc2d3"} Jul 27 15:56:21.263068 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.263018 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:21.283274 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.283245 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7741-140536443226128_89ad1e82-cb67-4a12-9b53-1e08449a05be/test-curl-7741-140536443226128/0.log" Jul 27 15:56:21.307922 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.307890 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brjdm\" (UniqueName: \"kubernetes.io/projected/89ad1e82-cb67-4a12-9b53-1e08449a05be-kube-api-access-brjdm\") pod \"89ad1e82-cb67-4a12-9b53-1e08449a05be\" (UID: \"89ad1e82-cb67-4a12-9b53-1e08449a05be\") " Jul 27 15:56:21.308760 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.308732 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:21.309893 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.309866 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ad1e82-cb67-4a12-9b53-1e08449a05be-kube-api-access-brjdm" (OuterVolumeSpecName: "kube-api-access-brjdm") pod "89ad1e82-cb67-4a12-9b53-1e08449a05be" (UID: "89ad1e82-cb67-4a12-9b53-1e08449a05be"). InnerVolumeSpecName "kube-api-access-brjdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:56:21.312702 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.312677 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:21.409600 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.409506 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brjdm\" (UniqueName: \"kubernetes.io/projected/89ad1e82-cb67-4a12-9b53-1e08449a05be-kube-api-access-brjdm\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:56:21.448886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.448852 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:21.449263 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.449249 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ad1e82-cb67-4a12-9b53-1e08449a05be" containerName="test-curl-7741-140536443226128" Jul 27 15:56:21.449305 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.449265 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ad1e82-cb67-4a12-9b53-1e08449a05be" containerName="test-curl-7741-140536443226128" Jul 27 15:56:21.449359 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.449348 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ad1e82-cb67-4a12-9b53-1e08449a05be" containerName="test-curl-7741-140536443226128" Jul 27 15:56:21.452416 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.452400 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:21.457418 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.457386 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:21.510358 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.510324 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzv7p\" (UniqueName: \"kubernetes.io/projected/a764b5d7-fdab-47be-a171-4fba1b688b2b-kube-api-access-xzv7p\") pod \"test-curl-7741-140536443226128\" (UID: \"a764b5d7-fdab-47be-a171-4fba1b688b2b\") " pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:21.611332 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.611299 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzv7p\" (UniqueName: \"kubernetes.io/projected/a764b5d7-fdab-47be-a171-4fba1b688b2b-kube-api-access-xzv7p\") pod \"test-curl-7741-140536443226128\" (UID: \"a764b5d7-fdab-47be-a171-4fba1b688b2b\") " pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:21.623870 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.623834 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzv7p\" (UniqueName: \"kubernetes.io/projected/a764b5d7-fdab-47be-a171-4fba1b688b2b-kube-api-access-xzv7p\") pod \"test-curl-7741-140536443226128\" (UID: \"a764b5d7-fdab-47be-a171-4fba1b688b2b\") " pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:21.762620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.762529 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:21.905258 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:21.905222 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:21.908149 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:21.908125 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda764b5d7_fdab_47be_a171_4fba1b688b2b.slice/crio-bdbabf24cc54bb27f357963bf58dc19970c115af2c8ec35c7fa4b1187e04a685 WatchSource:0}: Error finding container bdbabf24cc54bb27f357963bf58dc19970c115af2c8ec35c7fa4b1187e04a685: Status 404 returned error can't find the container with id bdbabf24cc54bb27f357963bf58dc19970c115af2c8ec35c7fa4b1187e04a685 Jul 27 15:56:22.126313 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:22.126277 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443226128" event={"ID":"a764b5d7-fdab-47be-a171-4fba1b688b2b","Type":"ContainerStarted","Data":"bdbabf24cc54bb27f357963bf58dc19970c115af2c8ec35c7fa4b1187e04a685"} Jul 27 15:56:22.127482 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:22.127459 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:22.127482 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:22.127479 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ce53a3825670a0c7d04d5ac4ca55cf019bee8d5ec8c0f153c57b24b2727610" Jul 27 15:56:22.140363 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:22.140336 2553 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7741-140536443226128" oldPodUID="89ad1e82-cb67-4a12-9b53-1e08449a05be" podUID="a764b5d7-fdab-47be-a171-4fba1b688b2b" Jul 27 15:56:23.134254 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:23.134217 2553 generic.go:356] "Generic (PLEG): container finished" podID="a764b5d7-fdab-47be-a171-4fba1b688b2b" containerID="574fb934b93ce16817065a9b450410bcb8d7b5ec53d1049ba6ae74b6e8a6dc68" exitCode=0 Jul 27 15:56:23.134665 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:23.134298 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7741-140536443226128" event={"ID":"a764b5d7-fdab-47be-a171-4fba1b688b2b","Type":"ContainerDied","Data":"574fb934b93ce16817065a9b450410bcb8d7b5ec53d1049ba6ae74b6e8a6dc68"} Jul 27 15:56:23.192128 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:23.192092 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ad1e82-cb67-4a12-9b53-1e08449a05be" path="/var/lib/kubelet/pods/89ad1e82-cb67-4a12-9b53-1e08449a05be/volumes" Jul 27 15:56:24.293485 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.293462 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:24.313376 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.313348 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7741-140536443226128_a764b5d7-fdab-47be-a171-4fba1b688b2b/test-curl-7741-140536443226128/0.log" Jul 27 15:56:24.340485 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.340454 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:24.344583 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.344562 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7741-140536443226128"] Jul 27 15:56:24.438327 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.438243 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzv7p\" (UniqueName: \"kubernetes.io/projected/a764b5d7-fdab-47be-a171-4fba1b688b2b-kube-api-access-xzv7p\") pod \"a764b5d7-fdab-47be-a171-4fba1b688b2b\" (UID: \"a764b5d7-fdab-47be-a171-4fba1b688b2b\") " Jul 27 15:56:24.440288 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.440250 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a764b5d7-fdab-47be-a171-4fba1b688b2b-kube-api-access-xzv7p" (OuterVolumeSpecName: "kube-api-access-xzv7p") pod "a764b5d7-fdab-47be-a171-4fba1b688b2b" (UID: "a764b5d7-fdab-47be-a171-4fba1b688b2b"). InnerVolumeSpecName "kube-api-access-xzv7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:56:24.539576 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:24.539543 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzv7p\" (UniqueName: \"kubernetes.io/projected/a764b5d7-fdab-47be-a171-4fba1b688b2b-kube-api-access-xzv7p\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:56:25.136801 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.136768 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2"] Jul 27 15:56:25.137384 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.137367 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a764b5d7-fdab-47be-a171-4fba1b688b2b" containerName="test-curl-7741-140536443226128" Jul 27 15:56:25.137431 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.137389 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a764b5d7-fdab-47be-a171-4fba1b688b2b" containerName="test-curl-7741-140536443226128" Jul 27 15:56:25.137489 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.137478 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a764b5d7-fdab-47be-a171-4fba1b688b2b" containerName="test-curl-7741-140536443226128" Jul 27 15:56:25.141018 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.140996 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.143350 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.143329 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c4dbeb-openshift-default-dockercfg-lwtk8\"" Jul 27 15:56:25.146533 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.146511 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdbabf24cc54bb27f357963bf58dc19970c115af2c8ec35c7fa4b1187e04a685" Jul 27 15:56:25.146533 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.146531 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:25.152120 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.152096 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2"] Jul 27 15:56:25.175318 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:56:25.175278 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7741-140536443226128\" is forbidden: User \"system:node:ip-10-0-135-218.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-218.ec2.internal' and this object" podUID="a764b5d7-fdab-47be-a171-4fba1b688b2b" pod="opendatahub/test-curl-7741-140536443226128" Jul 27 15:56:25.196862 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.196828 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a764b5d7-fdab-47be-a171-4fba1b688b2b" path="/var/lib/kubelet/pods/a764b5d7-fdab-47be-a171-4fba1b688b2b/volumes" Jul 27 15:56:25.246393 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246341 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-podinfo\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246604 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246453 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5be7f1f-3131-4e75-907c-f197afa09cf8-istiod-ca-cert\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246684 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246648 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-token\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246745 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246730 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-socket\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246803 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246761 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-credential-socket\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246803 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246792 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-certs\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246846 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9h2\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-kube-api-access-tw9h2\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.246925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-data\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.247034 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.246923 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-envoy\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.347776 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.347737 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5be7f1f-3131-4e75-907c-f197afa09cf8-istiod-ca-cert\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.348374 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.348348 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-token\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.348875 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.348854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-socket\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349027 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349009 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-credential-socket\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349622 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-certs\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349656 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349646 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-socket\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.348613 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5be7f1f-3131-4e75-907c-f197afa09cf8-istiod-ca-cert\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349427 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-credential-socket\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349716 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9h2\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-kube-api-access-tw9h2\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349780 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-data\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.349814 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-envoy\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.350177 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349862 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-podinfo\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.350177 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.349953 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-certs\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.350486 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.350449 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-data\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.352605 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.352581 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-envoy\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.352808 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.352790 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-podinfo\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.356590 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.356571 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-token\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.357691 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.357674 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9h2\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-kube-api-access-tw9h2\") pod \"e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.454793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.454704 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:25.592362 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.592326 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2"] Jul 27 15:56:25.594727 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:25.594696 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5be7f1f_3131_4e75_907c_f197afa09cf8.slice/crio-bd116862b5316a75d5ed0e41337224b5aa3d91fa9fc65e7668287abb5c3c3107 WatchSource:0}: Error finding container bd116862b5316a75d5ed0e41337224b5aa3d91fa9fc65e7668287abb5c3c3107: Status 404 returned error can't find the container with id bd116862b5316a75d5ed0e41337224b5aa3d91fa9fc65e7668287abb5c3c3107 Jul 27 15:56:25.596934 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.596904 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:25.597039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.596964 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:25.597039 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:25.597001 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:26.153435 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:26.153396 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" event={"ID":"e5be7f1f-3131-4e75-907c-f197afa09cf8","Type":"ContainerStarted","Data":"8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7"} Jul 27 15:56:26.153602 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:26.153459 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" event={"ID":"e5be7f1f-3131-4e75-907c-f197afa09cf8","Type":"ContainerStarted","Data":"bd116862b5316a75d5ed0e41337224b5aa3d91fa9fc65e7668287abb5c3c3107"} Jul 27 15:56:26.176706 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:26.176651 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podStartSLOduration=1.176636393 podStartE2EDuration="1.176636393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:56:26.174638723 +0000 UTC m=+3089.455988798" watchObservedRunningTime="2026-07-27 15:56:26.176636393 +0000 UTC m=+3089.457986467" Jul 27 15:56:26.455816 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:26.455716 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:26.460810 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:26.460782 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:27.159012 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:27.158980 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:27.159964 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:27.159939 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:36.742320 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:36.742280 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:56:43.869901 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.869867 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m"] Jul 27 15:56:43.873782 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.873754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.876895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.876839 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-809b00-openshift-default-dockercfg-r9bf7\"" Jul 27 15:56:43.884890 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.884854 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m"] Jul 27 15:56:43.928262 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928227 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-credential-socket\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928435 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928281 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-data\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928435 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928346 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27c743-80b0-49de-8391-8a97d8e63e8d-istiod-ca-cert\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928435 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928401 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-socket\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928435 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928433 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-token\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928647 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928477 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-envoy\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928647 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928517 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtkj\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-kube-api-access-nqtkj\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928647 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928594 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-podinfo\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:43.928779 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:43.928647 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-certs\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.029630 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-socket\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.029630 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029634 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-token\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.029886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029750 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-envoy\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.029886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029802 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtkj\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-kube-api-access-nqtkj\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.029886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-podinfo\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.029886 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029875 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-certs\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.029957 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-credential-socket\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030001 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-data\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030032 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27c743-80b0-49de-8391-8a97d8e63e8d-istiod-ca-cert\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030083 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-socket\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030357 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030338 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-credential-socket\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030464 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030439 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-data\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030582 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030559 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-certs\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.030762 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.030712 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27c743-80b0-49de-8391-8a97d8e63e8d-istiod-ca-cert\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.032542 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.032516 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-envoy\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.032635 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.032538 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-podinfo\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.038718 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.038694 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-token\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.039557 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.039534 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtkj\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-kube-api-access-nqtkj\") pod \"e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.191482 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.191385 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:44.354838 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.354817 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m"] Jul 27 15:56:44.357809 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:56:44.357780 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27c743_80b0_49de_8391_8a97d8e63e8d.slice/crio-7cdc06dd5576f3ba73da52fbb73838cc06da0f652f8333938b6f62504adce2cb WatchSource:0}: Error finding container 7cdc06dd5576f3ba73da52fbb73838cc06da0f652f8333938b6f62504adce2cb: Status 404 returned error can't find the container with id 7cdc06dd5576f3ba73da52fbb73838cc06da0f652f8333938b6f62504adce2cb Jul 27 15:56:44.360356 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.360320 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:44.360451 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.360430 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:44.360498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:44.360486 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:45.264673 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:45.264637 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" event={"ID":"4f27c743-80b0-49de-8391-8a97d8e63e8d","Type":"ContainerStarted","Data":"b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4"} Jul 27 15:56:45.265125 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:45.264697 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" event={"ID":"4f27c743-80b0-49de-8391-8a97d8e63e8d","Type":"ContainerStarted","Data":"7cdc06dd5576f3ba73da52fbb73838cc06da0f652f8333938b6f62504adce2cb"} Jul 27 15:56:45.289171 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:45.289116 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podStartSLOduration=2.289101185 podStartE2EDuration="2.289101185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:56:45.2871129 +0000 UTC m=+3108.568462975" watchObservedRunningTime="2026-07-27 15:56:45.289101185 +0000 UTC m=+3108.570451260" Jul 27 15:56:46.191715 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.191678 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:46.197189 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.197160 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:46.271447 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.271413 2553 generic.go:356] "Generic (PLEG): container finished" podID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerID="8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7" exitCode=0 Jul 27 15:56:46.271921 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.271494 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" event={"ID":"e5be7f1f-3131-4e75-907c-f197afa09cf8","Type":"ContainerDied","Data":"8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7"} Jul 27 15:56:46.271921 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.271797 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:46.272140 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.272071 2553 scope.go:117] "RemoveContainer" containerID="8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7" Jul 27 15:56:46.272948 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.272826 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:56:46.273177 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.273151 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:46.273288 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.273219 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:46.273288 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:46.273248 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:56:47.279062 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:47.279029 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" event={"ID":"e5be7f1f-3131-4e75-907c-f197afa09cf8","Type":"ContainerStarted","Data":"d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477"} Jul 27 15:56:47.455511 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:47.455475 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:47.455696 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:47.455676 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 15:56:47.455750 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:47.455723 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 15:56:48.455799 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:48.455766 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 15:56:48.456231 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:48.455819 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 15:56:49.455937 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:49.455894 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 15:56:49.456368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:49.455956 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 15:56:50.455865 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:50.455829 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 15:56:50.456087 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:50.455885 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 15:56:51.456138 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:51.456094 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 15:56:51.456553 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:51.456160 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 15:56:52.455741 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:52.455703 2553 patch_prober.go:28] interesting pod/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 15:56:52.455925 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:52.455757 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 15:56:53.459397 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:53.459359 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:53.459787 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:53.459633 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:53.460247 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:53.460224 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 15:56:55.646241 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:56:55.646204 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:57:05.381979 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:05.381891 2553 generic.go:356] "Generic (PLEG): container finished" podID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerID="b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4" exitCode=0 Jul 27 15:57:05.381979 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:05.381965 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" event={"ID":"4f27c743-80b0-49de-8391-8a97d8e63e8d","Type":"ContainerDied","Data":"b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4"} Jul 27 15:57:05.382492 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:05.382407 2553 scope.go:117] "RemoveContainer" containerID="b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4" Jul 27 15:57:05.383498 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:05.383471 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:57:05.383569 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:05.383556 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:57:05.383613 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:05.383589 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:57:06.389257 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:06.389218 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" event={"ID":"4f27c743-80b0-49de-8391-8a97d8e63e8d","Type":"ContainerStarted","Data":"b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde"} Jul 27 15:57:07.191554 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:07.191519 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:57:07.191767 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:07.191744 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 15:57:07.191835 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:07.191793 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 15:57:08.191743 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:08.191706 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 15:57:08.192194 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:08.191761 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 15:57:09.191792 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:09.191758 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 15:57:09.192234 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:09.191815 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 15:57:10.192307 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:10.192269 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 15:57:10.192721 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:10.192322 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 15:57:11.196317 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:11.196284 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:57:11.196807 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:11.196584 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:57:11.197332 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:11.197313 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 15:57:24.845620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:24.845540 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:57:54.038778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:57:54.038736 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:58:05.057812 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.057776 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw"] Jul 27 15:58:05.070563 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.070528 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw"] Jul 27 15:58:05.071012 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.070884 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.075357 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.075334 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-4cd197-openshift-default-dockercfg-s2krh\"" Jul 27 15:58:05.219793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.219755 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-token\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.219793 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.219797 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.219988 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.219866 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-envoy\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.219988 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.219901 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-socket\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.220086 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.220007 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-podinfo\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.220131 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.220096 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-certs\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.220174 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.220127 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-credential-socket\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.220211 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.220182 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-data\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.220256 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.220224 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgxs\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-kube-api-access-mkgxs\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321401 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321295 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-socket\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321401 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321381 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-podinfo\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-certs\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321460 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-credential-socket\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-data\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321547 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgxs\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-kube-api-access-mkgxs\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321602 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-token\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321637 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321630 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321983 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321689 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-envoy\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.321983 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321740 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-socket\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.322188 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.321991 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-certs\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.322282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.322247 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-data\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.322282 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.322274 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-credential-socket\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.322928 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.322902 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.324395 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.324365 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-envoy\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.324707 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.324687 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-podinfo\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.330176 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.330154 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-token\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.330417 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.330398 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgxs\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-kube-api-access-mkgxs\") pod \"e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.382913 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.382883 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:05.526752 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.526717 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw"] Jul 27 15:58:05.528616 ip-10-0-135-218 kubenswrapper[2553]: W0727 15:58:05.528587 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a24e843_ebe3_40c8_bd6f_e7fbd13e32e6.slice/crio-25938251dcbaf4804c0e5a67ecdaab6253933433091d3c83be19f1475d001f0d WatchSource:0}: Error finding container 25938251dcbaf4804c0e5a67ecdaab6253933433091d3c83be19f1475d001f0d: Status 404 returned error can't find the container with id 25938251dcbaf4804c0e5a67ecdaab6253933433091d3c83be19f1475d001f0d Jul 27 15:58:05.530784 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.530753 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:58:05.530848 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.530838 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:58:05.530892 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.530870 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:58:05.695329 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.695292 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" event={"ID":"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6","Type":"ContainerStarted","Data":"4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e"} Jul 27 15:58:05.695535 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.695354 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" event={"ID":"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6","Type":"ContainerStarted","Data":"25938251dcbaf4804c0e5a67ecdaab6253933433091d3c83be19f1475d001f0d"} Jul 27 15:58:05.718368 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:05.718307 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podStartSLOduration=0.718284992 podStartE2EDuration="718.284992ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:58:05.715397144 +0000 UTC m=+3188.996747219" watchObservedRunningTime="2026-07-27 15:58:05.718284992 +0000 UTC m=+3188.999635067" Jul 27 15:58:06.383961 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:06.383926 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:06.389137 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:06.389109 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:06.700893 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:06.700815 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:06.701679 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:06.701658 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:14.147522 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:14.147481 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:58:22.040593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.039794 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:37748->10.132.0.64:15021: read: connection reset by peer" start-of-body= Jul 27 15:58:22.040593 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.039865 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:37748->10.132.0.64:15021: read: connection reset by peer" Jul 27 15:58:22.786962 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.786928 2553 generic.go:356] "Generic (PLEG): container finished" podID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerID="4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e" exitCode=0 Jul 27 15:58:22.787154 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.786979 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" event={"ID":"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6","Type":"ContainerDied","Data":"4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e"} Jul 27 15:58:22.787411 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.787398 2553 scope.go:117] "RemoveContainer" containerID="4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e" Jul 27 15:58:22.788522 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.788495 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:58:22.788577 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.788564 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:58:22.788614 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:22.788600 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 27 15:58:23.794755 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:23.794724 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" event={"ID":"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6","Type":"ContainerStarted","Data":"3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162"} Jul 27 15:58:24.383911 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:24.383877 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:24.384133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:24.384077 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 15:58:24.384133 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:24.384121 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 15:58:25.384114 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:25.384079 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 15:58:25.384620 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:25.384132 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 15:58:26.383347 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:26.383311 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 15:58:26.383519 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:26.383375 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 15:58:27.384132 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:27.384090 2553 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 27 15:58:27.384638 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:27.384158 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 27 15:58:28.386694 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:28.386666 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:28.387147 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:28.386903 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:28.387654 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:28.387636 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:40.350794 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:40.350757 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:58:45.583478 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:45.583444 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw"] Jul 27 15:58:45.583895 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:45.583708 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" containerID="cri-o://3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162" gracePeriod=30 Jul 27 15:58:50.832971 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.832944 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:50.890859 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.890823 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-token\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891083 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.890868 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-socket\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891083 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.890904 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-credential-socket\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891083 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.890951 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-envoy\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891091 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istiod-ca-cert\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891121 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkgxs\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-kube-api-access-mkgxs\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891154 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-data\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891180 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-certs\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891212 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-podinfo\") pod \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\" (UID: \"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6\") " Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891225 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:58:50.891268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891245 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:58:50.891628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891473 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:58:50.891628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891491 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 15:58:50.891741 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891635 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.891741 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891702 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.891741 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891718 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.891741 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.891731 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.892119 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.892092 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:58:50.893301 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.893275 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:58:50.893386 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.893346 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 15:58:50.893440 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.893426 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 15:58:50.893497 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.893479 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-kube-api-access-mkgxs" (OuterVolumeSpecName: "kube-api-access-mkgxs") pod "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" (UID: "0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6"). InnerVolumeSpecName "kube-api-access-mkgxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 15:58:50.937628 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.937595 2553 generic.go:356] "Generic (PLEG): container finished" podID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerID="3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162" exitCode=0 Jul 27 15:58:50.937778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.937649 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" Jul 27 15:58:50.937778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.937680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" event={"ID":"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6","Type":"ContainerDied","Data":"3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162"} Jul 27 15:58:50.937778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.937735 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw" event={"ID":"0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6","Type":"ContainerDied","Data":"25938251dcbaf4804c0e5a67ecdaab6253933433091d3c83be19f1475d001f0d"} Jul 27 15:58:50.937778 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.937760 2553 scope.go:117] "RemoveContainer" containerID="3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162" Jul 27 15:58:50.947268 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.947250 2553 scope.go:117] "RemoveContainer" containerID="4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e" Jul 27 15:58:50.955751 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.955732 2553 scope.go:117] "RemoveContainer" containerID="3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162" Jul 27 15:58:50.956020 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:58:50.956003 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162\": container with ID starting with 3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162 not found: ID does not exist" containerID="3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162" Jul 27 15:58:50.956090 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.956029 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162"} err="failed to get container status \"3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162\": rpc error: code = NotFound desc = could not find container \"3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162\": container with ID starting with 3a37b6d39e745bf2261f327e69be6e41a3a9ef1b89d5436c26be915d1d5c7162 not found: ID does not exist" Jul 27 15:58:50.956090 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.956045 2553 scope.go:117] "RemoveContainer" containerID="4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e" Jul 27 15:58:50.956294 ip-10-0-135-218 kubenswrapper[2553]: E0727 15:58:50.956275 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e\": container with ID starting with 4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e not found: ID does not exist" containerID="4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e" Jul 27 15:58:50.956335 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.956300 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e"} err="failed to get container status \"4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e\": rpc error: code = NotFound desc = could not find container \"4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e\": container with ID starting with 4934d489a1f491aaba653483554d812eb64b8e8bf67aa6c87c646adc5634858e not found: ID does not exist" Jul 27 15:58:50.959026 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.959006 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw"] Jul 27 15:58:50.964123 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.964101 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4cd197-openshift-default-8656db6f68-kcwmw"] Jul 27 15:58:50.992904 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.992874 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.992904 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.992901 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.993126 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.992912 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkgxs\" (UniqueName: \"kubernetes.io/projected/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-kube-api-access-mkgxs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.993126 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.992923 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:50.993126 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:50.992934 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 15:58:51.193352 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:58:51.193253 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" path="/var/lib/kubelet/pods/0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6/volumes" Jul 27 15:59:05.241929 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:59:05.241897 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:59:20.548652 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:59:20.548615 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 15:59:57.437066 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:59:57.437021 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:59:57.451526 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:59:57.451500 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 15:59:58.637891 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:59:58.637850 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2"] Jul 27 15:59:58.638334 ip-10-0-135-218 kubenswrapper[2553]: I0727 15:59:58.638158 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" containerID="cri-o://d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477" gracePeriod=30 Jul 27 16:00:03.896233 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.896208 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 16:00:03.999146 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999017 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw9h2\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-kube-api-access-tw9h2\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999146 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999105 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-credential-socket\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999146 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999149 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-podinfo\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999453 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999165 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-certs\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999453 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999186 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-envoy\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999453 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999271 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-socket\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999453 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999335 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5be7f1f-3131-4e75-907c-f197afa09cf8-istiod-ca-cert\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999453 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999372 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-token\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999453 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999411 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-data\") pod \"e5be7f1f-3131-4e75-907c-f197afa09cf8\" (UID: \"e5be7f1f-3131-4e75-907c-f197afa09cf8\") " Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999514 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999500 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999566 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999744 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5be7f1f-3131-4e75-907c-f197afa09cf8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999811 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999832 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:03.999838 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999844 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5be7f1f-3131-4e75-907c-f197afa09cf8-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.000263 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:03.999852 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.000483 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.000457 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:04.001568 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.001543 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 16:00:04.001568 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.001560 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:00:04.001699 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.001546 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:04.001699 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.001579 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-kube-api-access-tw9h2" (OuterVolumeSpecName: "kube-api-access-tw9h2") pod "e5be7f1f-3131-4e75-907c-f197afa09cf8" (UID: "e5be7f1f-3131-4e75-907c-f197afa09cf8"). InnerVolumeSpecName "kube-api-access-tw9h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:00:04.101279 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.101239 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9h2\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-kube-api-access-tw9h2\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.101279 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.101271 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.101279 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.101280 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.101279 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.101289 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.101562 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.101297 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5be7f1f-3131-4e75-907c-f197afa09cf8-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:04.316844 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.316755 2553 generic.go:356] "Generic (PLEG): container finished" podID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerID="d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477" exitCode=0 Jul 27 16:00:04.316844 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.316816 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" Jul 27 16:00:04.317021 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.316842 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" event={"ID":"e5be7f1f-3131-4e75-907c-f197afa09cf8","Type":"ContainerDied","Data":"d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477"} Jul 27 16:00:04.317021 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.316929 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2" event={"ID":"e5be7f1f-3131-4e75-907c-f197afa09cf8","Type":"ContainerDied","Data":"bd116862b5316a75d5ed0e41337224b5aa3d91fa9fc65e7668287abb5c3c3107"} Jul 27 16:00:04.317021 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.316958 2553 scope.go:117] "RemoveContainer" containerID="d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477" Jul 27 16:00:04.327171 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.327152 2553 scope.go:117] "RemoveContainer" containerID="8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7" Jul 27 16:00:04.338218 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.338197 2553 scope.go:117] "RemoveContainer" containerID="d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477" Jul 27 16:00:04.338497 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:04.338474 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477\": container with ID starting with d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477 not found: ID does not exist" containerID="d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477" Jul 27 16:00:04.338558 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.338506 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477"} err="failed to get container status \"d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477\": rpc error: code = NotFound desc = could not find container \"d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477\": container with ID starting with d35c0c85be5940174244e5a83e0e682480d0dfe4bbab7ebcc14cd539a7da8477 not found: ID does not exist" Jul 27 16:00:04.338558 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.338523 2553 scope.go:117] "RemoveContainer" containerID="8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7" Jul 27 16:00:04.338752 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:04.338723 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7\": container with ID starting with 8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7 not found: ID does not exist" containerID="8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7" Jul 27 16:00:04.338872 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.338759 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7"} err="failed to get container status \"8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7\": rpc error: code = NotFound desc = could not find container \"8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7\": container with ID starting with 8e33828888fc6262168a22cb1242fdb6cad7e0d065b59f896f1a6fc1927d27f7 not found: ID does not exist" Jul 27 16:00:04.338959 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.338940 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2"] Jul 27 16:00:04.343016 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:04.342995 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c4dbeb-openshift-default-7b67fc547d-5nzx2"] Jul 27 16:00:05.191753 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:05.191718 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" path="/var/lib/kubelet/pods/e5be7f1f-3131-4e75-907c-f197afa09cf8/volumes" Jul 27 16:00:13.843245 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:13.843213 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 16:00:21.053369 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:21.053273 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m"] Jul 27 16:00:21.053944 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:21.053518 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" containerID="cri-o://b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde" gracePeriod=30 Jul 27 16:00:26.105670 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:26.105634 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27c743_80b0_49de_8391_8a97d8e63e8d.slice/crio-b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f27c743_80b0_49de_8391_8a97d8e63e8d.slice/crio-conmon-b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde.scope\": RecentStats: unable to find data in memory cache]" Jul 27 16:00:26.198626 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.198592 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 27 16:00:26.198787 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.198650 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 27 16:00:26.316144 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.316083 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 16:00:26.417140 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417102 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27c743-80b0-49de-8391-8a97d8e63e8d-istiod-ca-cert\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417334 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417158 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-podinfo\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417334 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417182 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-data\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417334 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417217 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-credential-socket\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417334 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417241 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-envoy\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417515 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417334 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-certs\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417515 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417376 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-socket\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417515 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417477 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqtkj\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-kube-api-access-nqtkj\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417675 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417521 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-token\") pod \"4f27c743-80b0-49de-8391-8a97d8e63e8d\" (UID: \"4f27c743-80b0-49de-8391-8a97d8e63e8d\") " Jul 27 16:00:26.417675 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417632 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:26.417675 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417667 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f27c743-80b0-49de-8391-8a97d8e63e8d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:00:26.417836 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417674 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:26.417836 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417737 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:26.417939 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417886 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.417939 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417905 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.417939 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417921 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.417939 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.417934 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f27c743-80b0-49de-8391-8a97d8e63e8d-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.418322 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.418297 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:26.419519 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.419495 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:26.419635 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.419620 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 16:00:26.420034 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.420014 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-kube-api-access-nqtkj" (OuterVolumeSpecName: "kube-api-access-nqtkj") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "kube-api-access-nqtkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:00:26.420034 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.420028 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f27c743-80b0-49de-8391-8a97d8e63e8d" (UID: "4f27c743-80b0-49de-8391-8a97d8e63e8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:00:26.433355 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.433323 2553 generic.go:356] "Generic (PLEG): container finished" podID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerID="b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde" exitCode=0 Jul 27 16:00:26.433454 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.433397 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" Jul 27 16:00:26.433454 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.433404 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" event={"ID":"4f27c743-80b0-49de-8391-8a97d8e63e8d","Type":"ContainerDied","Data":"b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde"} Jul 27 16:00:26.433531 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.433456 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m" event={"ID":"4f27c743-80b0-49de-8391-8a97d8e63e8d","Type":"ContainerDied","Data":"7cdc06dd5576f3ba73da52fbb73838cc06da0f652f8333938b6f62504adce2cb"} Jul 27 16:00:26.433531 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.433480 2553 scope.go:117] "RemoveContainer" containerID="b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde" Jul 27 16:00:26.442598 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.442583 2553 scope.go:117] "RemoveContainer" containerID="b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4" Jul 27 16:00:26.450352 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.450336 2553 scope.go:117] "RemoveContainer" containerID="b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde" Jul 27 16:00:26.450614 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:26.450594 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde\": container with ID starting with b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde not found: ID does not exist" containerID="b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde" Jul 27 16:00:26.450675 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.450623 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde"} err="failed to get container status \"b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde\": rpc error: code = NotFound desc = could not find container \"b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde\": container with ID starting with b5304a5bbac04330aed168f66954170fe58bc1e29f616915c0c7750469c51dde not found: ID does not exist" Jul 27 16:00:26.450675 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.450640 2553 scope.go:117] "RemoveContainer" containerID="b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4" Jul 27 16:00:26.450880 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:26.450861 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4\": container with ID starting with b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4 not found: ID does not exist" containerID="b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4" Jul 27 16:00:26.450929 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.450886 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4"} err="failed to get container status \"b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4\": rpc error: code = NotFound desc = could not find container \"b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4\": container with ID starting with b668efce6aadf5b1767fb1b5d24f1930093fc98f18d69b80d36b97474e2638e4 not found: ID does not exist" Jul 27 16:00:26.455702 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.455680 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m"] Jul 27 16:00:26.461770 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.461745 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-809b00-openshift-default-78486bc4c8-p2l2m"] Jul 27 16:00:26.519445 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.519409 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.519445 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.519441 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqtkj\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-kube-api-access-nqtkj\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.519627 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.519452 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.519627 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.519467 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:26.519627 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:26.519476 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f27c743-80b0-49de-8391-8a97d8e63e8d-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:27.192612 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:27.192581 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" path="/var/lib/kubelet/pods/4f27c743-80b0-49de-8391-8a97d8e63e8d/volumes" Jul 27 16:00:34.056794 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:34.056754 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 16:00:43.212687 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:43.212648 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc"] Jul 27 16:00:43.213203 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:43.212896 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" containerID="cri-o://5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942" gracePeriod=30 Jul 27 16:00:48.479484 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.479412 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 16:00:48.527836 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.527798 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-socket\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.527836 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.527840 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-data\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528117 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.527876 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmjc\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-kube-api-access-jhmjc\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528117 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.527923 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-envoy\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528117 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.527948 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-credential-socket\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528117 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.527972 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fd5160-1ccf-438d-9c9f-5c432770ad17-istiod-ca-cert\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528117 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528030 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-certs\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528117 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528092 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-podinfo\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528411 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528172 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-token\") pod \"98fd5160-1ccf-438d-9c9f-5c432770ad17\" (UID: \"98fd5160-1ccf-438d-9c9f-5c432770ad17\") " Jul 27 16:00:48.528495 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528167 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:48.528561 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528275 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:48.528561 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528362 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:48.528660 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528563 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fd5160-1ccf-438d-9c9f-5c432770ad17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:00:48.528808 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.528781 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-data" (OuterVolumeSpecName: "istio-data") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:48.530522 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.530494 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-token" (OuterVolumeSpecName: "istio-token") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:00:48.530652 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.530516 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 16:00:48.530720 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.530658 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-kube-api-access-jhmjc" (OuterVolumeSpecName: "kube-api-access-jhmjc") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "kube-api-access-jhmjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:00:48.530720 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.530658 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98fd5160-1ccf-438d-9c9f-5c432770ad17" (UID: "98fd5160-1ccf-438d-9c9f-5c432770ad17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:00:48.548850 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.548819 2553 generic.go:356] "Generic (PLEG): container finished" podID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerID="5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942" exitCode=0 Jul 27 16:00:48.549003 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.548868 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" Jul 27 16:00:48.549003 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.548900 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" event={"ID":"98fd5160-1ccf-438d-9c9f-5c432770ad17","Type":"ContainerDied","Data":"5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942"} Jul 27 16:00:48.549003 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.548962 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc" event={"ID":"98fd5160-1ccf-438d-9c9f-5c432770ad17","Type":"ContainerDied","Data":"f085428d1f29a9f4872cc74e35f082aea7a6fa795e7aac252b06205209e56b77"} Jul 27 16:00:48.549003 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.548993 2553 scope.go:117] "RemoveContainer" containerID="5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942" Jul 27 16:00:48.559115 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.559098 2553 scope.go:117] "RemoveContainer" containerID="ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109" Jul 27 16:00:48.567685 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.567665 2553 scope.go:117] "RemoveContainer" containerID="5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942" Jul 27 16:00:48.567974 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:48.567954 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942\": container with ID starting with 5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942 not found: ID does not exist" containerID="5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942" Jul 27 16:00:48.568025 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.567985 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942"} err="failed to get container status \"5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942\": rpc error: code = NotFound desc = could not find container \"5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942\": container with ID starting with 5c061df4d942da3514c6865531f78dd97250d2f89e7ec342560bf8ed10ea7942 not found: ID does not exist" Jul 27 16:00:48.568025 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.568003 2553 scope.go:117] "RemoveContainer" containerID="ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109" Jul 27 16:00:48.568257 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:00:48.568237 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109\": container with ID starting with ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109 not found: ID does not exist" containerID="ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109" Jul 27 16:00:48.568294 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.568266 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109"} err="failed to get container status \"ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109\": rpc error: code = NotFound desc = could not find container \"ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109\": container with ID starting with ffa51642638d50aa659ec57f8f759ceae985f0aa9a360e7a8f1c580eccb54109 not found: ID does not exist" Jul 27 16:00:48.574033 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.574010 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc"] Jul 27 16:00:48.577751 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.577729 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-f58a29-openshift-default-57456d4488-7fdlc"] Jul 27 16:00:48.629441 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629407 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629441 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629437 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629441 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629448 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmjc\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-kube-api-access-jhmjc\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629670 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629458 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629670 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629467 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629670 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629475 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fd5160-1ccf-438d-9c9f-5c432770ad17-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629670 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629483 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fd5160-1ccf-438d-9c9f-5c432770ad17-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629670 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629490 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:48.629670 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:48.629497 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fd5160-1ccf-438d-9c9f-5c432770ad17-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:00:49.191487 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:49.191454 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" path="/var/lib/kubelet/pods/98fd5160-1ccf-438d-9c9f-5c432770ad17/volumes" Jul 27 16:00:57.044201 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:00:57.044161 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6kt44"] Jul 27 16:01:05.552032 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:05.551995 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5"] Jul 27 16:01:05.552569 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:05.552309 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" containerID="cri-o://b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f" gracePeriod=30 Jul 27 16:01:09.783843 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:09.783805 2553 patch_prober.go:28] interesting pod/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 16:01:09.784259 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:09.783867 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 16:01:10.805892 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.805867 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 16:01:10.933367 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933330 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-envoy\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933367 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933374 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-credential-socket\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933397 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-podinfo\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933433 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istiod-ca-cert\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933469 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-data\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933500 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-socket\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933546 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-certs\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933576 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kst7k\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-kube-api-access-kst7k\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.933628 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933620 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-token\") pod \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\" (UID: \"0a1f9561-6f62-4403-86d0-7df2e1236ffa\") " Jul 27 16:01:10.934002 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933867 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:01:10.934002 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933919 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:01:10.934002 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933961 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:01:10.934002 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.933958 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 16:01:10.934460 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.934431 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:01:10.935658 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.935634 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 16:01:10.935742 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.935695 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 16:01:10.935803 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.935782 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:01:10.935900 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:10.935879 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-kube-api-access-kst7k" (OuterVolumeSpecName: "kube-api-access-kst7k") pod "0a1f9561-6f62-4403-86d0-7df2e1236ffa" (UID: "0a1f9561-6f62-4403-86d0-7df2e1236ffa"). InnerVolumeSpecName "kube-api-access-kst7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 16:01:11.034464 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034422 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-certs\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034464 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034456 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kst7k\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-kube-api-access-kst7k\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034464 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034469 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-token\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034483 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-envoy\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034495 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-credential-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034507 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-podinfo\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034517 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istiod-ca-cert\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034527 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-istio-data\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.034728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.034538 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1f9561-6f62-4403-86d0-7df2e1236ffa-workload-socket\") on node \"ip-10-0-135-218.ec2.internal\" DevicePath \"\"" Jul 27 16:01:11.669331 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.669296 2553 generic.go:356] "Generic (PLEG): container finished" podID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerID="b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f" exitCode=0 Jul 27 16:01:11.669537 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.669340 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" Jul 27 16:01:11.669537 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.669390 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" event={"ID":"0a1f9561-6f62-4403-86d0-7df2e1236ffa","Type":"ContainerDied","Data":"b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f"} Jul 27 16:01:11.669537 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.669465 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5" event={"ID":"0a1f9561-6f62-4403-86d0-7df2e1236ffa","Type":"ContainerDied","Data":"774bebf91f2af8a0dec031b9dc8cc89ea8dc7877aa0f696b09a2b98f418ccb1c"} Jul 27 16:01:11.669537 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.669493 2553 scope.go:117] "RemoveContainer" containerID="b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f" Jul 27 16:01:11.678493 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.678477 2553 scope.go:117] "RemoveContainer" containerID="beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65" Jul 27 16:01:11.687316 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.687292 2553 scope.go:117] "RemoveContainer" containerID="b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f" Jul 27 16:01:11.687628 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:01:11.687605 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f\": container with ID starting with b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f not found: ID does not exist" containerID="b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f" Jul 27 16:01:11.687723 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.687645 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f"} err="failed to get container status \"b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f\": rpc error: code = NotFound desc = could not find container \"b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f\": container with ID starting with b829d5fa438e54afe29265af8d38436c6dc11e869f927e4a675b6420caffeb2f not found: ID does not exist" Jul 27 16:01:11.687723 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.687669 2553 scope.go:117] "RemoveContainer" containerID="beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65" Jul 27 16:01:11.687723 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.687610 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5"] Jul 27 16:01:11.687989 ip-10-0-135-218 kubenswrapper[2553]: E0727 16:01:11.687972 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65\": container with ID starting with beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65 not found: ID does not exist" containerID="beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65" Jul 27 16:01:11.688029 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.687994 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65"} err="failed to get container status \"beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65\": rpc error: code = NotFound desc = could not find container \"beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65\": container with ID starting with beaea5b543b3cb3a2baf4dce095d93eff09d7fed7cb4004c70f274a2c8feaf65 not found: ID does not exist" Jul 27 16:01:11.692455 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.692431 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-052d16-openshift-default-66ff8fdd5b-gkzd5"] Jul 27 16:01:11.801129 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.801101 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n85mq_19bec687-43ed-459b-b581-725638dfc717/manager/1.log" Jul 27 16:01:11.927301 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:11.927226 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-66pd8_601a372d-2af9-4d2b-83c1-bf0d1d6bb845/manager/0.log" Jul 27 16:01:12.415590 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:12.415561 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78959f89bb-d8cl7_d58091d9-1026-4ca3-95ea-e289b01f6cca/copy-manifests/0.log" Jul 27 16:01:12.428972 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:12.428941 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78959f89bb-d8cl7_d58091d9-1026-4ca3-95ea-e289b01f6cca/manager/0.log" Jul 27 16:01:13.153506 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:13.153447 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752770-6l8gg_06d3f4ed-900b-4cae-b59a-b5a712e3aab9/cleanup/0.log" Jul 27 16:01:13.190946 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:13.190913 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" path="/var/lib/kubelet/pods/0a1f9561-6f62-4403-86d0-7df2e1236ffa/volumes" Jul 27 16:01:15.523414 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:15.523381 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6kt44_b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99/limitador/0.log" Jul 27 16:01:16.005840 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:16.005807 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb_e0e6ad28-e2a6-4784-81fa-f14c37386753/istio-proxy/0.log" Jul 27 16:01:16.519557 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:16.519519 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/1.log" Jul 27 16:01:16.901194 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:16.901167 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c885cd9-2mgwd_3434d48a-2c83-4375-89e1-de3b25813fa9/router/0.log" Jul 27 16:01:23.485325 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485285 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6fjkv/must-gather-6hp7p"] Jul 27 16:01:23.485730 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485678 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" Jul 27 16:01:23.485730 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485690 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" Jul 27 16:01:23.485730 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485711 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" Jul 27 16:01:23.485730 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485716 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" Jul 27 16:01:23.485730 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485727 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" Jul 27 16:01:23.485730 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485733 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485741 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485748 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485754 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485760 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485779 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485784 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485790 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485796 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485802 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485807 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485861 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485870 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485877 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485883 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485889 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5be7f1f-3131-4e75-907c-f197afa09cf8" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485895 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f27c743-80b0-49de-8391-8a97d8e63e8d" containerName="istio-proxy" Jul 27 16:01:23.485924 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485900 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485957 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485963 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485981 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.485987 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.486063 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fd5160-1ccf-438d-9c9f-5c432770ad17" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.486071 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a24e843-ebe3-40c8-bd6f-e7fbd13e32e6" containerName="istio-proxy" Jul 27 16:01:23.486462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.486079 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a1f9561-6f62-4403-86d0-7df2e1236ffa" containerName="istio-proxy" Jul 27 16:01:23.490499 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.490479 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.492674 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.492650 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6fjkv\"/\"default-dockercfg-65s6n\"" Jul 27 16:01:23.492781 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.492703 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6fjkv\"/\"openshift-service-ca.crt\"" Jul 27 16:01:23.492781 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.492723 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6fjkv\"/\"kube-root-ca.crt\"" Jul 27 16:01:23.500433 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.500410 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fjkv/must-gather-6hp7p"] Jul 27 16:01:23.547404 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.547371 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2346d358-0e6d-45c1-b2bf-fcdc21f3ae27-must-gather-output\") pod \"must-gather-6hp7p\" (UID: \"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27\") " pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.547568 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.547416 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xc7\" (UniqueName: \"kubernetes.io/projected/2346d358-0e6d-45c1-b2bf-fcdc21f3ae27-kube-api-access-g7xc7\") pod \"must-gather-6hp7p\" (UID: \"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27\") " pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.648699 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.648663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xc7\" (UniqueName: \"kubernetes.io/projected/2346d358-0e6d-45c1-b2bf-fcdc21f3ae27-kube-api-access-g7xc7\") pod \"must-gather-6hp7p\" (UID: \"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27\") " pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.648886 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.648759 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2346d358-0e6d-45c1-b2bf-fcdc21f3ae27-must-gather-output\") pod \"must-gather-6hp7p\" (UID: \"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27\") " pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.649096 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.649079 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2346d358-0e6d-45c1-b2bf-fcdc21f3ae27-must-gather-output\") pod \"must-gather-6hp7p\" (UID: \"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27\") " pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.658564 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.658538 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xc7\" (UniqueName: \"kubernetes.io/projected/2346d358-0e6d-45c1-b2bf-fcdc21f3ae27-kube-api-access-g7xc7\") pod \"must-gather-6hp7p\" (UID: \"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27\") " pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.800957 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.800875 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fjkv/must-gather-6hp7p" Jul 27 16:01:23.942139 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.942114 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fjkv/must-gather-6hp7p"] Jul 27 16:01:23.944783 ip-10-0-135-218 kubenswrapper[2553]: W0727 16:01:23.944751 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2346d358_0e6d_45c1_b2bf_fcdc21f3ae27.slice/crio-8274cfbccf46e5675e4fe6f12c85a4f38e6b739644c942f69cc2313bda4ee487 WatchSource:0}: Error finding container 8274cfbccf46e5675e4fe6f12c85a4f38e6b739644c942f69cc2313bda4ee487: Status 404 returned error can't find the container with id 8274cfbccf46e5675e4fe6f12c85a4f38e6b739644c942f69cc2313bda4ee487 Jul 27 16:01:23.946685 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:23.946665 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 16:01:24.738385 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:24.738350 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fjkv/must-gather-6hp7p" event={"ID":"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27","Type":"ContainerStarted","Data":"8274cfbccf46e5675e4fe6f12c85a4f38e6b739644c942f69cc2313bda4ee487"} Jul 27 16:01:28.770478 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:28.770430 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fjkv/must-gather-6hp7p" event={"ID":"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27","Type":"ContainerStarted","Data":"1baec90a35b36e34514a6ee73fadacf2ff28bcc534ff35cfe07c29df9d8d02c1"} Jul 27 16:01:28.770995 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:28.770488 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fjkv/must-gather-6hp7p" event={"ID":"2346d358-0e6d-45c1-b2bf-fcdc21f3ae27","Type":"ContainerStarted","Data":"6672adde52c23434feaf5d079af3fe8512a23e08936f68f181d9db593e9c8dd2"} Jul 27 16:01:28.788343 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:28.788283 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6fjkv/must-gather-6hp7p" podStartSLOduration=1.788429192 podStartE2EDuration="5.78826639s" totalImagesPullingTime="3.999837198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:01:28.785889525 +0000 UTC m=+3392.067239652" watchObservedRunningTime="2026-07-27 16:01:28.78826639 +0000 UTC m=+3392.069616461" Jul 27 16:01:29.616010 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:29.615977 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bjd66_21f2ae8c-8a22-46d6-ae00-414fd7d0cefe/global-pull-secret-syncer/0.log" Jul 27 16:01:29.784041 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:29.784011 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x9x9q_46efa81f-a600-439b-b979-abb5028bf7b1/konnectivity-agent/0.log" Jul 27 16:01:29.822290 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:29.822263 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-218.ec2.internal_a58aa08a3e395c7a07bb090be093243d/haproxy/0.log" Jul 27 16:01:35.634499 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:35.634462 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6kt44_b82dbc3d-e516-40dd-9ab4-f3f6f3bfcf99/limitador/0.log" Jul 27 16:01:37.915982 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:37.915945 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-5t57t_6368b6c0-4289-433d-bdad-d502124b7802/cluster-monitoring-operator/0.log" Jul 27 16:01:38.234951 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:38.234874 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kccj2_eb57ea67-1c0c-4413-ba52-9f25c8e45e41/node-exporter/0.log" Jul 27 16:01:38.265064 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:38.265023 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kccj2_eb57ea67-1c0c-4413-ba52-9f25c8e45e41/kube-rbac-proxy/0.log" Jul 27 16:01:38.298988 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:38.298956 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kccj2_eb57ea67-1c0c-4413-ba52-9f25c8e45e41/init-textfile/0.log" Jul 27 16:01:38.755406 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:38.755371 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-zps4j_c27793e9-4c00-4698-98b5-23ac51f4e402/prometheus-operator/0.log" Jul 27 16:01:38.797206 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:38.797169 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-zps4j_c27793e9-4c00-4698-98b5-23ac51f4e402/kube-rbac-proxy/0.log" Jul 27 16:01:38.834914 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:38.834884 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-5b9vz_93c07ce6-2095-4d4f-ae69-ccfd19fc82bc/prometheus-operator-admission-webhook/0.log" Jul 27 16:01:40.329069 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:40.329034 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-hcl95_aa23ab11-efac-4d9a-a0fb-77a43c481313/networking-console-plugin/0.log" Jul 27 16:01:40.854325 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:40.854286 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/1.log" Jul 27 16:01:40.862690 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:40.862666 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-798gp_c38c59b8-7ecf-4006-8a0e-e75eb798ef14/console-operator/2.log" Jul 27 16:01:41.384898 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.384869 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-dr2qr_cf76b9bf-77eb-4e6c-a803-ba91e44b5c57/download-server/0.log" Jul 27 16:01:41.498245 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.498202 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn"] Jul 27 16:01:41.503014 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.502984 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.512234 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.512204 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn"] Jul 27 16:01:41.632422 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.632383 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xwq\" (UniqueName: \"kubernetes.io/projected/80124d83-1375-458e-8f30-8872a72eecd1-kube-api-access-h6xwq\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.632599 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.632481 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-proc\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.632599 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.632516 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-podres\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.632599 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.632557 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-sys\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.632728 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.632601 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-lib-modules\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.733288 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.733200 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-proc\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.734586 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.734559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-podres\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.734948 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.734930 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-sys\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.735171 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.733407 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-proc\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.735273 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.734856 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-podres\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.735273 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.735104 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-sys\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.735273 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.735189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-lib-modules\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.735403 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.735278 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80124d83-1375-458e-8f30-8872a72eecd1-lib-modules\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.735403 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.735281 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xwq\" (UniqueName: \"kubernetes.io/projected/80124d83-1375-458e-8f30-8872a72eecd1-kube-api-access-h6xwq\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.744918 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.744890 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xwq\" (UniqueName: \"kubernetes.io/projected/80124d83-1375-458e-8f30-8872a72eecd1-kube-api-access-h6xwq\") pod \"perf-node-gather-daemonset-vhzsn\" (UID: \"80124d83-1375-458e-8f30-8872a72eecd1\") " pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:41.820576 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:41.820540 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:42.004120 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:42.003206 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn"] Jul 27 16:01:42.015994 ip-10-0-135-218 kubenswrapper[2553]: W0727 16:01:42.015950 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80124d83_1375_458e_8f30_8872a72eecd1.slice/crio-e3144daae1b79dc4875b1e2ad72b3f78b3a1aa029e9e3208ad6cf7c4d96a4359 WatchSource:0}: Error finding container e3144daae1b79dc4875b1e2ad72b3f78b3a1aa029e9e3208ad6cf7c4d96a4359: Status 404 returned error can't find the container with id e3144daae1b79dc4875b1e2ad72b3f78b3a1aa029e9e3208ad6cf7c4d96a4359 Jul 27 16:01:42.067217 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:42.067189 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-8g85f_1934808b-5939-4353-a238-33b80c7b77f4/volume-data-source-validator/0.log" Jul 27 16:01:42.864558 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:42.864520 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" event={"ID":"80124d83-1375-458e-8f30-8872a72eecd1","Type":"ContainerStarted","Data":"313ad8787b70653a284594b65beec9b6996d15fd1e3c3a9be909f028ac308726"} Jul 27 16:01:42.865012 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:42.864575 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" event={"ID":"80124d83-1375-458e-8f30-8872a72eecd1","Type":"ContainerStarted","Data":"e3144daae1b79dc4875b1e2ad72b3f78b3a1aa029e9e3208ad6cf7c4d96a4359"} Jul 27 16:01:42.865012 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:42.864619 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:42.883462 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:42.883405 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" podStartSLOduration=1.883384125 podStartE2EDuration="1.883384125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:01:42.879828373 +0000 UTC m=+3406.161178448" watchObservedRunningTime="2026-07-27 16:01:42.883384125 +0000 UTC m=+3406.164734200" Jul 27 16:01:43.118922 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:43.118838 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-52wmr_df6cb94b-d781-4eb7-87a1-b85241de9b7c/dns/0.log" Jul 27 16:01:43.141068 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:43.141033 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-52wmr_df6cb94b-d781-4eb7-87a1-b85241de9b7c/kube-rbac-proxy/0.log" Jul 27 16:01:43.288015 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:43.287990 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bmnp2_fa7f660f-4dd4-457a-950c-07412f65b7da/dns-node-resolver/0.log" Jul 27 16:01:43.930808 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:43.930781 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l24dr_65b2001b-a43e-4060-8dc1-e69aae55dd22/node-ca/0.log" Jul 27 16:01:44.896122 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:44.896093 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb_e0e6ad28-e2a6-4784-81fa-f14c37386753/istio-proxy/0.log" Jul 27 16:01:45.149188 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:45.149094 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/0.log" Jul 27 16:01:45.182031 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:45.181987 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/1.log" Jul 27 16:01:45.272079 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:45.272032 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c885cd9-2mgwd_3434d48a-2c83-4375-89e1-de3b25813fa9/router/0.log" Jul 27 16:01:46.090516 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:46.090478 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7c56v_d1f42dc0-6dc2-4201-8f41-b20652979554/serve-healthcheck-canary/0.log" Jul 27 16:01:46.626735 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:46.626704 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2tn84_4a86eae4-88cc-402b-8003-570b4b1a7aae/insights-operator/0.log" Jul 27 16:01:46.628193 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:46.628046 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2tn84_4a86eae4-88cc-402b-8003-570b4b1a7aae/insights-operator/1.log" Jul 27 16:01:46.719219 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:46.719190 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvnsj_29747912-51e0-4a07-9cdf-2c3436233245/kube-rbac-proxy/0.log" Jul 27 16:01:46.744282 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:46.744248 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvnsj_29747912-51e0-4a07-9cdf-2c3436233245/exporter/0.log" Jul 27 16:01:46.768006 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:46.767981 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvnsj_29747912-51e0-4a07-9cdf-2c3436233245/extractor/0.log" Jul 27 16:01:48.881232 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:48.881197 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6fjkv/perf-node-gather-daemonset-vhzsn" Jul 27 16:01:48.994284 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:48.994250 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n85mq_19bec687-43ed-459b-b581-725638dfc717/manager/1.log" Jul 27 16:01:48.994284 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:48.994286 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n85mq_19bec687-43ed-459b-b581-725638dfc717/manager/0.log" Jul 27 16:01:49.248354 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:49.248274 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-66pd8_601a372d-2af9-4d2b-83c1-bf0d1d6bb845/manager/0.log" Jul 27 16:01:49.633859 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:49.633826 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78959f89bb-d8cl7_d58091d9-1026-4ca3-95ea-e289b01f6cca/manager/0.log" Jul 27 16:01:49.652542 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:49.652516 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78959f89bb-d8cl7_d58091d9-1026-4ca3-95ea-e289b01f6cca/copy-manifests/0.log" Jul 27 16:01:50.047990 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:50.047900 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb_e0e6ad28-e2a6-4784-81fa-f14c37386753/istio-proxy/0.log" Jul 27 16:01:50.289293 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:50.289258 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/0.log" Jul 27 16:01:50.316908 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:50.316825 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/1.log" Jul 27 16:01:50.375761 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:50.375724 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c885cd9-2mgwd_3434d48a-2c83-4375-89e1-de3b25813fa9/router/0.log" Jul 27 16:01:51.129013 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:51.128978 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-964ddd7d8-cmcvc_707cc393-0db6-4cf3-beb7-d3316197803a/manager/0.log" Jul 27 16:01:52.036081 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:52.036004 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vjtwb_e0e6ad28-e2a6-4784-81fa-f14c37386753/istio-proxy/0.log" Jul 27 16:01:52.250962 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:52.250921 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/0.log" Jul 27 16:01:52.288396 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:52.288312 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tthvj_57901768-40ee-411f-94a3-544355959d94/istio-proxy/1.log" Jul 27 16:01:52.351446 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:52.351410 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f8c885cd9-2mgwd_3434d48a-2c83-4375-89e1-de3b25813fa9/router/0.log" Jul 27 16:01:56.329686 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:56.329655 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mq8wp_d603c607-eec2-4c7c-b605-89b31e3aa8e1/migrator/0.log" Jul 27 16:01:56.369423 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:56.369391 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-mq8wp_d603c607-eec2-4c7c-b605-89b31e3aa8e1/graceful-termination/0.log" Jul 27 16:01:56.797937 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:56.797891 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7_c0d6fee3-17dd-465f-9239-cf9dcea137d7/kube-storage-version-migrator-operator/1.log" Jul 27 16:01:56.799038 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:56.799016 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-zx8x7_c0d6fee3-17dd-465f-9239-cf9dcea137d7/kube-storage-version-migrator-operator/0.log" Jul 27 16:01:58.212195 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.212155 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/kube-multus-additional-cni-plugins/0.log" Jul 27 16:01:58.239866 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.239839 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/egress-router-binary-copy/0.log" Jul 27 16:01:58.278617 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.278591 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/cni-plugins/0.log" Jul 27 16:01:58.308770 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.308742 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/bond-cni-plugin/0.log" Jul 27 16:01:58.350332 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.350310 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/routeoverride-cni/0.log" Jul 27 16:01:58.389369 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.389336 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/whereabouts-cni-bincopy/0.log" Jul 27 16:01:58.443821 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.443797 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fnsgs_1a31ecc7-38eb-4962-9b1b-eb8b418fa2d8/whereabouts-cni/0.log" Jul 27 16:01:58.730801 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.730766 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fb6tm_97244c43-6c8f-463f-8cd7-42828e29ce00/kube-multus/0.log" Jul 27 16:01:58.851251 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.851217 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cv4fx_66371b5f-01b5-4329-bc7e-069d801574e9/network-metrics-daemon/0.log" Jul 27 16:01:58.873863 ip-10-0-135-218 kubenswrapper[2553]: I0727 16:01:58.873835 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cv4fx_66371b5f-01b5-4329-bc7e-069d801574e9/kube-rbac-proxy/0.log"